Paper Title: Generative Sentiment Analysis via Latent Category Distribution and Constrained Decoding.









Generative Sentiment Analysis via LCD and CD







Jun Zhou et al.




    Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, China 
    
    Beijing Rigour Technology Co., Ltd., China
    School of Cyberspace, Hangzhou Dianzi University, China
    North China Power Engineering Co., Ltd. of China Power Engineering Consulting Group, China

    
    
    
    
    dhji@whu.edu.cn
    
    
    

    
    



J. Zhou et al.









Generative Sentiment Analysis via Latent Category Distribution and Constrained Decoding
    Jun Zhou1 Dongyang Yu2 Kamran Aziz 1 Fangfang Su3 Qing Zhang 4 Fei Li 1 Donghong Ji 1()
    
    July 2024
================================================================================================






	
	F. Author et al.
	
	
	
	Princeton University, Princeton NJ 08544, USA Springer Heidelberg, Tiergartenstr. 17, 69121 Heidelberg, Germany
	lncs@springer.com

	<http://www.springer.com/gp/computer-science/lncs> ABC Institute, Rupert-Karls-University Heidelberg, Heidelberg, Germany

	{abc,lncs}@uni-heidelberg.de
	
	Generative Sentiment Analysis via Latent Category Distribution and Constrained Decoding
    First Author10000-1111-2222-3333 Second Author2,31111-2222-3333-4444 Third Author32222–3333-4444-5555
    July 2024
=========================================================================================================


	



§ ABSTRACT 

Fine-grained sentiment analysis involves extracting and organizing sentiment elements from textual data. However, existing approaches often overlook issues of category semantic inclusion and overlap, as well as inherent structural patterns within the target sequence. This study introduces a generative sentiment analysis model. To address the challenges related to category semantic inclusion and overlap, a latent category distribution variable is introduced. By reconstructing the input of a variational autoencoder, the model learns the intensity of the relationship between categories and text, thereby improving sequence generation. Additionally, a trie data structure and constrained decoding strategy are utilized to exploit structural patterns, which in turn reduces the search space and regularizes the generation process. Experimental results on the Restaurant-ACOS and Laptop-ACOS datasets demonstrate a significant performance improvement compared to baseline models. Ablation experiments further confirm the effectiveness of latent category distribution and constrained decoding strategy.















§ INTRODUCTION



Sentiment analysis, especially the fine-grained Aspect-Based Sentiment Analysis (ABSA), plays a vital role in Natural Language Processing (NLP) <cit.>. ABSA aims to convert unstructured text into structured records to achieve a profound comprehension of sentiment semantics.
For instance, in the comment “as far as gaming performance, the m370x does quite well”, the ABSA quadruple extraction task <cit.> seeks to extract aspect-level sentiment elements, including aspect, category, opinion, and sentiment polarity, represented as a structured quadruple ⟨m370x, GRAPHICS#OPERATION_PERFORMANCE, well, POS⟩. 


Given its ability to offer opinions and sentiments regarding specific aspects, this task has been widely used in domains such as marketing and conversational agents <cit.>.







Sentiment extraction models are currently categorized into two main groups: extractive methods and generative methods <cit.>. Extractive methods typically frame the task as a sequence labeling or multi-classification problem <cit.>, necessitating task-specific classification networks <cit.>. 
The introduction of generative pre-trained language models such as T5 <cit.> and BART <cit.> has introduced a new paradigm for information extraction, shifting tasks towards sequence generation <cit.>. This generative approach has also demonstrated promising results in sentiment quadruple extraction tasks <cit.>.




However, existing sentiment extraction models often overlook the challenges posed by category inclusion and overlap, which refer to the semantic inclusion or overlapping of categories within the dataset.
For example, the category “GRAPHICS” is semantically encompassed by categories such as “HARDWARE” and “LAPTOP” (where “GRAPHICS” is a semantic subset of “HARDWARE”, and “HARDWARE” is a semantic subset of “LAPTOP”). Similarly, categories like “MEMORY”, “KEYBOARD”, and “BATTERY” exhibit similar semantic inclusion relationships with “HARDWARE” and “LAPTOP”. Additionally, “GRAPHICS” and “DISPLAY” share a semantic overlap, as do “WARRANTY” and “SUPPORT”. 
These instances of semantic inclusion and overlap complicate the accurate extraction of elements, a concern often neglected by current models.








On the other hand, current generative models typically employ greedy decoding <cit.>, selecting the word with the highest probability from the entire target vocabulary at each time step. Nonetheless, this strategy fails to effectively utilize pattern knowledge inherent in the target sequence, which could otherwise constrain the search space <cit.>. To elaborate, datasets typically define restricted sets of categories, subcategories, and sentiment polarities. Greedy decoding based on the entire vocabulary may lead to the generation of invalid categories, misaligned subcategories, as well as invalid sentiment polarities. Despite these challenges, current generative models often neglect the significance of pattern knowledge in the target sequence.







Motivated by recent advances in generative frameworks, this study proposes a T5-based model for sentiment quadruple extraction. The model treats extraction as a generative process, producing sentiment elements in a uniform, natural language format. 
To address category semantic inclusion and overlap issues, a Latent Category Distribution (LCD) variable is introduced. 
This variable, learned through reconstructing the input of a Variational AutoEncoder (VAE) <cit.>, captures the relationship intensity between text and categories, supplementing target sequence generation.
Furthermore, to leverage the inherent pattern knowledge within the target sequence, a Constrained Decoding (CD) strategy based on a trie data structure <cit.> is adopted. 
This strategy introduces structural pattern knowledge during the decoding stage to reduce the search space and enhance sequence generation.
Evaluation on the Restaurant-ACOS <cit.> and Laptop-ACOS <cit.> datasets show that the proposed model outperforms baseline models, and demonstrate the efficacy of the LCD and CD.



The primary contributions of this study include:

	
  * Addressing category semantic inclusion and overlap challenges via a latent category distribution variable learning the relationship intensity between text and categories.
	
  * Regulating the generation process through a constrained decoding strategy leveraging structural pattern knowledge within the target sequence.
	
  * Experimental results on the Restaurant-ACOS and Laptop-ACOS datasets demonstrate the effectiveness of the proposed model.




§ RELATED WORKS












In recent years, there has been a growing interest in fine-grained sentiment analysis <cit.>, involving the extraction and organization of sentiment-related elements such as aspects, categories, opinions, and sentiment polarities from text. Initially focusing on individual elements such as aspect terms <cit.>, categories <cit.>, and sentiment polarities <cit.>. Early research has expanded to include paired extractions <cit.>. Subsequent studies have advanced the field by proposing tasks like triplet extraction <cit.>, involving aspects, opinions, and sentiment polarities. More recently, quadruple extraction <cit.> has been introduced to extract aspects, categories, opinions, and sentiment polarities concurrently, including implicit elements <cit.>. This task has garnered significant attention in the research community.







Quadruple extraction models can be broadly categorized into two main approaches <cit.>: extractive and generative methods. Extractive methods decompose the task into distinct subtasks, addressing them separately <cit.>. For instance, Cai et al. <cit.> employ either the Double Propagation (DP) <cit.> algorithm or the JET <cit.> model to extract aspect-opinion-sentiment triplets, and then predict the category of these triplets. Alternatively, they might use the TAS-BERT <cit.> model for the joint extraction of aspect-opinion pairs, followed by category and sentiment polarity prediction. Zhu et al. <cit.> capture local and contextual features and then utilize a grid labeling scheme along with its decoding method to extract quadruples. 
Extractive methods often transform the task into sequence labeling or multi-classification problems, requiring the development of task-specific classification networks <cit.>.



The emergence of generative language models <cit.> has introduced a novel paradigm to information extraction by unifying tasks into sequence generation problems, eliminating the need for task-specific network designs. 
This shift simplifies model design and facilitates the comprehensive use of correlation information among different subtask modules <cit.>. 
Generative extraction has been applied to various tasks <cit.>. Efforts have recently also focused on transforming fine-grained sentiment analysis tasks into sequence generation challenges. For example, the BARTABSA <cit.> model reformulates sentiment element extraction as a problem of generating indices for target elements. The Paragraph <cit.> model employs predefined templates to reframe quadruple extraction as a template-based generation problem. The GAS <cit.> model proposes annotation-based and extraction-based approaches to model the task as a text generation problem.




However, the models discussed overlook challenges posed by category semantic overlap and inclusion, and neglect to utilize structural pattern knowledge to constrain decoding. This study, inspired by the generative paradigm, adopts a generative model as its core framework. Unlike existing generative models, it addresses semantic overlap and inclusion by incorporating a latent category distribution, and utilizes pattern knowledge to guide decoding during generation. Experimental results validate the effectiveness of the proposed model.





§ TASK DEFINITION


Given a comment X = [x_1, …, x_N], the objective of sentiment quadruple extraction is to extract structured quadruples ⟨ a, c, o, s ⟩ from the comment <cit.>:
	
    X ⇒ [⟨ a_1, c_1, o_1, s_1 ⟩, ⟨ a_2, c_2, o_2, s_2 ⟩, …]

	Here, a represents the aspect described in the comment, c denotes the category to which the aspect belongs, o signifies the opinion on the aspect, and s indicates the sentiment polarity of the opinion. Given that a comment may encompass multiple aspects, the extraction may result in multiple quadruples being identified.


The objective of sentiment quadruple extraction is to extract structured quadruples ⟨ a, c, o, s ⟩ from a given comment X = [x_1, …, x_N] <cit.>:

    X ⇒ [⟨ a_1, c_1, o_1, s_1 ⟩, ⟨ a_2, c_2, o_2, s_2 ⟩, …]

where a represents the aspect, c denotes the category to which the aspect belongs, o signifies the opinion, and s indicates the sentiment polarity of the opinion. 


Since this study transforms the extraction task into a generative manner, specifically, given the comment X, this study utilizes a Sequence-to-Sequence (Seq2Seq) framework to produce a linearized augmented language sequence <cit.> Y:
	
    X ⇒ Y = [y_1, y_2, …]

	Here, the augmented language linear sequence Y comprises natural language words and encompasses the essential elements of the sentiment quadruples. Parsing the generated augmented sequence Y allows for the extraction of sentiment quadruples.
Since this study adopts a generative manner to the extraction task, using a Sequence-to-Sequence (Seq2Seq) framework to convert X into a linearized augmented language sequence Y <cit.>, the task is specifically transformed into:


    X ⇒ Y = [y_1, y_2, …]



where Y is a word sequence consisting of the essential elements of the quadruples, which can be extracted by parsing Y.






§ MODEL STRUCTURE







 §.§ Latent Category Distribution




Latent Category Distribution (LCD) module (refer to <Ref>) models category distribution within text following Fei et al. <cit.>, using a latent variable instead of explicit measurement. It reconstructs the variational autoencoder input to learn the LCD representation Z.



  §.§.§ Encoder

The input to the LCD module consists of the Category Bag of Words (CBoW) features X_CBoW∈ℝ^L, following the approach outlined by Fei et al. <cit.>. 
	The construction of the CBoW features involves the following steps: 
	1) Eliminating stopwords, meaningless words, and sentiment-related words from the training set sentences, while retaining words related to categories and aspects as much as possible, to construct a CBoW vocabulary of length L. 
	2) Calculating the frequency of words in the vocabulary that appear in the sentence to derive the CBoW features of the sentence, resulting in a L-dimensional vector. 
	The objective of the first step is twofold: firstly, to capture the latent category distribution, ensuring that the latent distribution representation contains comprehensive category information rather than general semantic information; secondly, reducing the size of the BoW vocabulary aids in training the variational autoencoder <cit.>.


The input to the LCD module is the Category Bag of Words (CBoW) features <cit.> X_CBoW∈ℝ^L. The construction of CBoW features involves: 
1) Removing stopwords, meaningless words, and sentiment-related words from texts while preserving category- and aspect-related words to construct a CBoW vocabulary of length L. This step aims to ensure that representation Z contains comprehensive category information rather than general semantic information, while decreasing the BoW vocabulary size is beneficial for VAE training <cit.>. 
2) Calculating the occurrence frequency of words in the vocabulary appearing in a sentence to obtain the sentence's CBoW features, forming a L-dimensional vector. 




The encoder of the LCD module comprises two Feedforward Neural Networks (FFNs), denoted as f_μ(·) and f_σ(·), respectively. Each of these networks includes a hidden layer with a dimensionality of 256, which is responsible for transforming the features X_CBoW into the prior parameters μ and σ:

    μ = f_μ ( X_CBoW )


    logσ = f_σ ( X_CBoW )


Following Bowman et al. <cit.>, a latent variable Z' = μ + σ·ϵ∈ℝ^K is defined to represent the latent category distribution, where K is the number of categories, ϵ is a Gaussian noise variable sampled from 𝒩(0,1). Z' is then normalized as follows:

    Z = softmax(Z')


Consequently, the latent category distribution can be reflected by variable Z.


  §.§.§ Decoder


The LCD module employs variational inference <cit.> to approximate a posterior distribution over Z. It begins by applying a linear transformation f_lcd(·) to Z:


    R^lcd = f_lcd(Z, W_lcd)

where W_lcd∈ℝ^K × dim, dim is the encoding dimension. R^lcd will be fed into the generative framework (<Ref>) with category distribution information to enhance the  generation.
Subsequently, a FFNs f_R(·) is employed to further decode R^lcd into R:

    R = f_R(R^lcd, W_R)

where W_R ∈ℝ^L × dim.
Finally, the CBoW features are reconstructed by:

    X̂_CBoW = softmax(R)




  §.§.§ Loss Function

Parameters of LCD module are learned through the optimization of the variational lower bound on the marginal log likelihood of features:

    log p_φ(X) = 𝔼_Z∼ q_ϕ(Z|X)[ log p_φ(X|Z) ] - KL(q_ϕ(Z|X)  p(Z))

where ϕ and φ are parameters of the encoder and decoder, respectively. The KL divergence term ensures that the distribution q_ϕ(Z|X) approaches the prior probability p(Z), while p_φ(X|Z) represents the decoding process.



 §.§ Generative Framework






  §.§.§ Encoder

The generative framework (refer to <Ref>) employs a T5 encoder <cit.> for input encoding. Given a comment X = [x_1,⋯ ,x_N], the encoder first converts the token sequence into a high-dimensional vector representation E∈ℝ^N × dim using an internal embedding layer, where N is the sequence length, and dim is the dimension of the representation.

Subsequently, E is passed through the T5 encoder which comprises multiple layers of Transformer <cit.> structure, to calculate context-aware representations:

    H^enc = T5Encoder(E)

where H^enc∈ℝ^N × dim.




  §.§.§ Category-Text Attention

After encoding the input sequence into representation H^enc, the model computes attention score of the latent category distribution over the input sequence, to establish the relationship between them. The attention score is calculated as follows:

    e_j = dot(R^lcd, H_j^enc)/√(dim)

where dot(·) is the dot product, and j ∈ [1, N]. Following this, e is fed to a softmax function to calculate the attention weights:

    a = softmax(e)

a is then applied as weights to the representation of the input sequence to obtain the output vector V of the encoder:

    V_j = a_j H_j^enc

The vector V, representing the input text and incorporating LCD information, is then fed into the decoder to provide contextual information for decoding.




  §.§.§ Decoder


The model employs a T5 decoder <cit.> to sequentially generate the output sequence Y=[y_1, …, y_i, …] in an auto-regressive manner. When predicting the output at step i, the decoder uses the context vector V and the previous outputs y_<i as inputs to calculate the current hidden state of the decoder:


    H_i^dec = T5Decoder(V, y_<i)


Here, each T5Decoder(·) layer comprises a Transformer structure and a cross-attention mechanism <cit.> for enhancing the representation of H_i^dec. The probability distribution over target vocabulary at step i is calculated as follows:

    P(y_i|y_<i,V) = Softmax(WH_i^dec + b)

where W∈ℝ^|𝒱| × dim, 𝒱 is the target vocabulary, and b is a bias vector.


The decoder repeats this procedure until it encounters the end token “⟨ eos ⟩” or reaches the maximum specified output length.



  §.§.§ Loss Function


The loss function of the generative framework is defined as the cross-entropy loss between the generated sequence and the target sequence:

    L(θ) = -∑_i=1^Mlog P(y_i|y_<i, V)

where θ represents the parameters of the generative framework, and M is the length of the target sequence.



 §.§ Constrained Decoding





  §.§.§ Target Sequence Pattern Knowledge




In generative models, a commonly used decoding approach is greedy decoding <cit.>, where the model calculates the probability distribution over the entire vocabulary at each decoding step, and selects the word with the highest predicted probability as the output.

However, this decoding strategy may lead to invalid structure patterns. 
For instance, 
1) due to the limited number of category types, greedy decoding using the entire vocabulary may yield invalid categories,
2) since each category corresponds to distinct subcategories, greedy decoding might generate subcategories that do not align with that category,
3) as sentiment polarity is finite, greedy decoding could result in invalid sentiment polarities,
4) incomplete sequence structures, such as the failure to generate pairs of brackets, may occur. 
Following Lu et al. <cit.>, information regarding the structural patterns of the target sequence, which can be used to regulate its generation process, is referred to as target sequence pattern knowledge in this paper.



  §.§.§ Constrained Decoding Strategy





Pattern knowledge of the target sequence can constrain the decoding process, reducing the search space and enhancing target generation. 
To leverage this knowledge, this study employs a trie-based constrained decoding strategy <cit.>. 
A trie, also referred to as a prefix tree, is a type of k-ary search tree that serves as a data structure for locating specific keys within a set <cit.>.
With this structure, constrained decoding strategy dynamically selects and prunes candidate vocabulary based on the current generation state. The candidate vocabulary 𝒱' includes the following three types:

	
  * Structure pattern: category set 𝒞_1 (see <Ref>), their respective subcategory set 𝒞_2 (see <Ref>), and sentiment polarities set 𝒮 (see <Ref>).
	
  * Text span: aspects 𝒜 and opinions 𝒪, representing text spans in the input.
	
  * Structure indicators: “[”,  “]”, “⟨”, and “⟩” (see <Ref>), utilized to combine structure patterns and text spans.





















<Ref> depicts constrained decoding using a trie of linearized sentiment quadruples. Starting with the initial token “⟨ bos ⟩”, candidate vocabulary 𝒱' at each step is confined to child nodes stemming from the previously generated node. 

For example, after generating the sequence “⟨ bos ⟩ [”, 𝒱' is limited to the child node set {“⟨”, “]”} of node “[”.






When generating the category, decoding involves searching the trie structure of category 𝒞_1 (<Ref>). Similarly, when generating the subcategory, decoding searches the corresponding subtree, constraining the search space based on the category-subcategory dependency. For example, when generating the subcategory of category “HARDWARE”, the candidate vocabulary 𝒱' is the child node set of “HARDWARE”, namely, {“PORTABILITY”, ⋯, “USABILITY”}.

To generate sentiment polarity, decoding is confined to searching the sentiment polarity trie 𝒮 (<Ref>). For aspect words and opinion words generation, the candidate vocabulary 𝒱' is the entire T5 vocabulary. This generation process repeats until the termination token “⟨ eos ⟩” is produced. Finally, the output sequence is converted into quadruples to serve as the final extraction results.



§ EXPERIMENTS



 §.§ Datasets





Two datasets, Restaurant-ACOS <cit.> and Laptop-ACOS <cit.>, are used to evaluate model performance. Each sample comprises a review text and annotated sentiment quadruples, as summarized in <Ref>.



These datasets also introduce the concept of implicit elements, where aspects or opinions within the review are considered implicit if they are not explicitly stated. 
For example, in the review “apparently apple isn't even trying anymore”, the annotated quadruple ⟨apple, COMPANY#GENERAL, Null, NEG⟩ indicates that the opinion towards the aspect “apple” is not explicitly stated in the review and is considered implicit, labeled as “Null” in the annotation.



The statistics of implicit elements is shown in <Ref>. Here, EA&EO represents quadruples with only explicit aspects and opinions, IA&EO indicates those with implicit aspects and explicit opinions, EA&IO covers quadruples with explicit aspects and implicit opinions, and IA&IO denotes quadruples with only implicit aspects and opinions.







 §.§ Experimental Setup










A GeForce RTX 3090 (24GB) was utilized for computation. T5-base model was employed with a training batch size of 16, and encoding and hidden layer dimensions set to 768. The AdamW optimizer was employed with a learning rate of 2e-5, and dropout was applied at a rate of 0.2. Experiments were initialized five times randomly, and the average result was considered the final outcome.







While it was possible to train the model solely with the generation framework loss (<Ref>), practical experiments revealed slow convergence and poor performance. To tackle this issue, the LCD module was initially trained with the reconstruction loss (<Ref>) until convergence. Subsequently, the entire model was trained using the generation loss (<Ref>). Once the generation loss approached convergence, further training was applied to the LCD module until convergence. This process was repeated until the model demonstrated improved performance.





Evaluation employed Precision, Recall, and F1 score, with a quadruple being deemed correctly predicted only if all its four elements were predicted accurately.



 §.§ Baselines


The subsequent models were employed for comparative analysis:

Extractive:
DP-ACOS <cit.>: Utilizes the DP algorithm to extract sentiment triplets ⟨ a,o,s ⟩ and then determines the category of each triplet.
JET-ACOS <cit.>: Employs JET <cit.> to extract triplets ⟨ a,o,s ⟩ and then predicts category of each triplet.
TB-ACOS <cit.>: Utilizes TAS-BERT <cit.> for aspect-opinion joint extraction based on category-sentiment, then filters out invalid pairs.
EC-ACOS <cit.>: Performs aspect-opinion joint extraction, then predicts category-sentiment pairs.
SGTS <cit.>: Captures high-dimensional features of sentences, then uses grid labeling scheme and its decoding method for extraction.

Generative:
BARTABSA <cit.>: A model based on BART <cit.>, generates sentiment quadruples by predicting the indices of target words and sentiment categories.
GAS <cit.>: Using T5, GAS models sentiment analysis and various subtasks as generative tasks.
Paraphrase <cit.>: Transforms the quadruple extraction task into a paraphrase generation process using predefined templates.
Seq2Path <cit.>: Formalizes the generated content as a tree, with sentiment tuples representing the paths of the tree.



§ EXPERIMENTAL RESULTS AND ANALYSES



 §.§ Main Results














<Ref> presents the experimental results of the proposed model and baseline models on two datasets. The results highlight the superiority of generative models over extractive models. This advantage can be attributed primarily to two factors: 1) treating the extraction task as an end-to-end sequence generation problem reduces error propagation; 2) uniformly modeling the generation of each quadruple element facilitates more effective utilization of semantic knowledge in pretrained models, enhancing semantic sharing and decision interaction.

Furthermore, the results in <Ref> suggest that the proposed model, utilizing LCD and CD, outperforms the comparing generative models on both datasets. Specifically, on the Restaurant-ACOS dataset, compared to the baseline model GAS, the proposed model shows a decrease of 0.004 in Precision, but an increase of 0.038 in Recall, and an increase of 0.017 in F1. On the Laptop-ACOS dataset, the proposed model exhibits an increase of 0.031 in Precision, 0.035 in Recall, and 0.033 in F1. These results highlight the superior performance of the proposed model in sentiment quadruple extraction.

Despite a notable improvement in performance compared to the baseline models, the overall performance remains modest. This can be attributed to two main factors. 
First, the inherent complexity of the task necessitates the extraction and precise matching of four distinct types of sentiment elements, thereby increasing the difficulty of the task. 
Second, special expressions in language such as the presence of implicit elements within the text presents a significant challenge, potentially impeding the model's comprehensive semantic understanding, and consequently leading to suboptimal performance. 
The following subsection will delve into the extraction of implicit elements.







 §.§ Implicit Element Extraction Results

As shown in <Ref>, both datasets contain a certain proportion of implicit elements <cit.>. Since these elements are not explicitly mentioned in the original text, the model lacks sufficient clues to extract them, thus requiring a deeper understanding of the text, which has also been a focus of attention in the field of sentiment analysis in recent years <cit.>.
	
	To evaluate the performance of the proposed model and the comparison models in extracting implicit elements, this section divides the test set into four subsets to examine the performance of the models on each subset. The four subsets are divided as follows: (1) EA&EO indicates that this subset contains quadruples with only explicit aspect words and opinion words; (2) IA&EO indicates that this subset contains quadruples with implicit aspect words and explicit opinion words; (3) EA&IO indicates that this subset contains quadruples with explicit aspect words and implicit opinion words; (4) IA&IO indicates that this subset contains quadruples with only implicit aspect words and opinion words. The proposed model and baseline models are experimented on these four subsets, and the results are shown in <Ref>, where the comparison model results are from the original literature.
	
	From <Ref>, it can be seen that on the Restaurant-ACOS dataset, the proposed model not only performs best in the EA&EO subset but also outperforms other models in subsets containing implicit elements, such as IA&EO and IA&IO. On the Laptop-ACOS dataset, the proposed model also achieves the best performance in the EA&EO subset and outperforms other models in subsets containing implicit elements, such as EA&IO and IA&IO. These results demonstrate that the proposed model's ability to deeply understand and generate implicit elements is superior to that of the comparison models.












As outlined in <Ref>, both datasets exhibit a notable presence of implicit elements <cit.>. Due to their non-explicit nature in the text, these elements present a challenge for models, requiring a deeper understanding of the text. This challenge has garnered significant attention in recent sentiment analysis studies <cit.>.
	
	To assess the capacity of models in extracting implicit elements, the test set was divided into four subsets, each targeting specific types of quadruples. These subsets, namely EA&EO, IA&EO, EA&IO, and IA&IO, are elaborated in <Ref>.
	<Ref> illustrates that the model's performance in extracting explicit elements is superior to its performance in extracting implicit elements. For instance, on the Restaurant-ACOS dataset, the proposed model achieves an F1 score of 66.12 on the subset EA&EO, which contains only explicit elements, outperforming the F1 scores of 54.35, 44.20, and 41.25 on the other three subsets containing implicit elements. Similar results are observed on the Laptop-ACOS dataset. This performance gap reflects the challenges associated with extracting implicit elements.
	
	The results presented in <Ref> indicate that, on the Restaurant-ACOS dataset, the proposed model excels not only on the EA&EO subset but also on subsets that include implicit elements (IA&EO and IA&IO). Similarly, on the Laptop-ACOS dataset, the proposed model shows superior performance on the EA&EO subset and surpasses on subsets containing implicit elements (EA&IO and IA&IO). These findings highlight the superior capability of the proposed model in deep semantic understanding and generating implicit elements.










	




As noted in <Ref>, both datasets include implicit elements <cit.>, which are not explicitly stated in the reviews. These elements necessitate a comprehensive understanding of the text and present a challenge for models, thus attracting considerable attention in recent sentiment analysis research <cit.>.

To assess the capacity of models in extracting implicit elements, the test set was divided into four subsets: EA&EO, IA&EO, EA&IO, and IA&IO, each targeting specific type of quadruples, whose meanings were introduced in <Ref>. Results in <Ref> indicate that models excel in extracting explicit elements compared to implicit ones. For instance, on the Restaurant-ACOS dataset, the proposed model achieves an F1 score of 0.661 on the EA&EO subset, which is comprised solely of explicit elements, significantly outperforming scores of 0.544, 0.442, and 0.413 on the other three subsets containing implicit elements. Similar trends are also observed on the Laptop-ACOS dataset. These results highlight the challenge of extracting implicit elements.

Moreover, <Ref> shows that, on Restaurant-ACOS, the proposed model not only outperforms baseline models on the EA&EO subset, but also on subsets containing implicit elements (IA&EO and IA&IO). 
Similarly, on Laptop-ACOS, the proposed model also outperforms on subsets comprising implicit elements (EA&IO and IA&IO). These findings highlight the capability of the proposed model in deep semantic comprehension and generation of implicit elements.












 §.§ Ablation Study





To assess the effectiveness of the proposed LCD module and CD strategy, ablation experiments were conducted on the datasets. Results in <Ref> indicate the impact of removing the LCD module (w/o LCD), excluding the CD strategy (w/o CD), and simultaneously excluding both components (w/o LCD&CD).

As shown in <Ref>, performance deteriorates when LCD and CD are excluded. Specifically, on the Restaurant-ACOS dataset, the removal of LCD alone resulted in an F1 decrease of 0.009, while excluding CD alone led to an F1 decrease of 0.012. Simultaneous removal of both components resulted in an F1 decrease of 0.014. Similar trends were also observed on the Laptop-ACOS dataset. 

These results suggest that the proposed LCD module enhances the extraction performance by capturing latent category features, while the CD strategy regulates and guides the generation process, mitigating the generation of erroneous patterns.






 §.§ Visualization of Latent Category Distribution





As introduced in <Ref>, the LCD module learns latent category distribution representation Z for a sample. To visually illustrate the effectiveness of this module, five samples were selected with their learned LCD representations visualized in <Ref>. The x-axis denotes the total 23 categories, while the y-axis indicates the sample indices. 
Each row in the figure corresponds to a LCD representation vector of a sample across categories, with color intensity reflecting the magnitude of the vector in respective category.
The five samples, along with their respective standard categories in parentheses, are as follows:

	
  * the product is great, but the customer support is horrible. (LAPTOP, SUPPORT)
	
  * second issue is with scaling of the ui. (SOFTWARE)
	
  * asus, has a horrible reputation. (COMPANY)
	
  * hdmi out doesn't work right. (PORTS)
	
  * now i have to deal with warranty stuff and sending it back. (WARRANTY)



 



 
As depicted in <Ref>, the LCD module effectively captures latent category distribution information for samples, including those with complex category inclusion (sample 4) or overlap (sample 5) relationships. 
For example, while the standard category of sample 4 is “PORTS”, it is highly predictable as “HARDWARE” or “LAPTOP” due to semantic inclusion relationships, where “PORTS” is semantically included by both “HARDWARE” and “LAPTOP”. 
For sample 5, although its standard category is “WARRANTY”, it is highly predictable as “SUPPORT” due to semantic overlap between the two categories. <Ref> demonstrates the accurate capture of latent category distributions by LCD.



§ CONCLUSION

This study introduces a generative model for extracting sentiment quadruples. To address challenges related to semantic inclusion and overlap among categories, the model incorporates a latent category distribution to capture the strength of the relationship between text and categories. Moreover, a constrained decoding strategy based on a trie structure is employed to exploit knowledge of target sequence patterns, thereby reducing the search space and enhancing the generation process. Experimental results demonstrate performance enhancements compared to baseline models, with ablation studies verifying the efficacy of the latent category distribution and constrained decoding strategy.
Despite advancements, performance remains moderate, primarily due to task complexity and special expressions in language, including implicit elements.

Future research could focus on enhancing feature extraction and representation, as well as improving contextual understanding, and integrating coreference resolution and background knowledge, to improve the extraction process, including implicit elements.















	


  §.§.§ Sample Heading (Third Level)
 Only two levels of
	headings should be numbered. Lower level headings remain unnumbered;
	they are formatted as run-in headings.
	
	

  
Sample Heading (Fourth Level)
	The contribution should contain no more than four levels of
	headings. Table <ref> gives a summary of all heading levels.
	
	
	
	
	Displayed equations are centered and set on a separate
	line.
	
    x + y = z

	Please try to avoid rasterized images for line-art diagrams and
	schemas. Whenever possible, use vector graphics instead (see
	Fig. <ref>).
	
	
	
	
	This is a sample theorem. The run-in heading is set in bold, while
	the following text appears in italics. Definitions, lemmas,
	propositions, and corollaries are styled the same way.
	
	
	
	
	
	
	Proofs, examples, and remarks have the initial word in italics,
	while the following text appears in normal font.
	
	For citations of references, we prefer the use of square brackets
	and consecutive numbers. Citations using labels or the author/year
	convention are also acceptable. The following bibliography provides
	a sample reference list with entries for journal
	articles <cit.>, an LNCS chapter <cit.>, a
	book <cit.>, proceedings without editors <cit.>,
	and a homepage <cit.>. Multiple citations are grouped
	<cit.>,
	<cit.>.






  §.§.§ 


This work was supported by the National Natural Science Foundation of China (No. 62176187), the National Key Research and Development Program of China (No. 2022YFB3103602, No. 2017YFC1200500), the Research Foundation of Ministry of Education of China (No. 18JZD015).




  §.§.§ 

	It is now necessary to declare any competing interests or to specifically
	state that the authors have no competing interests. Please place the
	statement with a bold run-in heading in small font size beneath the
	(optional) acknowledgments[If EquinOCS, our proceedings submission
	system, is used, then the disclaimer can be provided directly in the system.],
	for example: The authors have no competing interests to declare that are
	relevant to the content of this article. Or: Author A has received research
	grants from Company W. Author B has received a speaker honorarium from
	Company X and owns stock in Company Y. Author C is a member of committee Z.









splncs04


8
	ref_article1
	Author, F.: Article title. Journal 2(5), 99–110 (2016)
	
	ref_lncs1
	Author, F., Author, S.: Title of a proceedings paper. In: Editor,
	F., Editor, S. (eds.) CONFERENCE 2016, LNCS, vol. 9999, pp. 1–13.
	Springer, Heidelberg (2016). 10.10007/1234567890
	
	ref_book1
	Author, F., Author, S., Author, T.: Book title. 2nd edn. Publisher,
	Location (1999)
	
	ref_proc1
	Author, A.-B.: Contribution title. In: 9th International Proceedings
	on Proceedings, pp. 1–2. Publisher, Location (2010)
	
	ref_url1
	LNCS Homepage, <http://www.springer.com/lncs>, last accessed 2023/10/25
	





 Paper Title: Need of AI in Modern Education: in the Eyes of Explainable AI (xAI).






    



    



    







    



[
     R. Zamini[Corresponding author: rahelehzamini1361@yahoo.com] ^1,  S. Ghafouri^2  and F. Goodarzi^3

	^1  Department  of Mathematics, Faculty of Mathematical Sciences and  Computer, Kharazmi University, Thehran, Iran. 

	^2   Department of Mathematics, Faculty of Science, Arak University, Arak, 384817758, Iran.

^3   Department of Statistics, Faculty of Mathematical Sciences, University of Kashan, Kashan, Iran.

    
===================================================================================================================================================================================================================================================================================================================================================================================================================================













§ ABSTRACT

     Modern Education is not Modern without AI. However, AI's complex nature makes understanding and fixing problems challenging. Research worldwide shows that a parent's income greatly influences a child's education. This led us to explore how AI, especially complex models, makes important decisions using Explainable AI tools. Our research uncovered many complexities linked to parental income and offered reasonable explanations for these decisions. However, we also found biases in AI that go against what we want from AI in education: clear transparency and equal access for everyone. These biases can impact families and children's schooling, highlighting the need for better AI solutions that offer fair opportunities to all. This chapter tries to shed light on the complex ways AI operates, especially concerning biases. These are the foundational steps towards better educational policies, which include using AI in ways that are more reliable, accountable, and beneficial for everyone involved.




§ INTRODUCTION


codes: https://colab.research.google.com/drive/149rwtHbcBqbKFuNWbXz7PhAmV_cdSpzW?usp=sharing.notebook
The trajectory of the education system has traversed a remarkable journey from medieval times to the modern era, reflecting societal advancements, technological innovations, and evolving pedagogical paradigms. Initially characterized by rudimentary pedagogical methods, limited accessibility, and a rigid curriculum, the education landscape has undergone transformative changes, albeit with persistent challenges.

Historically, the medieval education system was predominantly characterized by religious institutions, emphasizing classical literature, theology, and rote memorization. Access was largely restricted to elite echelons of society, with pedagogical approaches rooted in tradition and conformity.

Fast-forwarding to the modern era, the education system has witnessed democratization, technological integration, and a paradigm shift towards personalized learning. However, contemporary challenges persist, necessitating strategic interventions:


    
  * Better Recommendation of Courses:
Traditional systems often adopt a one-size-fits-all approach, overlooking individual learning preferences and aptitudes. AI-powered recommender systems being employed in book selection  <cit.>, utilizing collaborative filtering algorithms, analysing student data to generate personalized course recommendations, and optimizing learning outcomes and engagement. Contemporary software solutions, such as Coursera's recommendation engine1[https://www.coursera.org/1. https://www.coursera.org/].
 and platforms like Knewton Alta 2[https://www.wiley.com/en-us/education/alta2. https://www.wiley.com/en-us/education/alta]
 leverage advanced algorithms to tailor learning pathways based on individual student profiles.


  * SWOT Analysis of Students:
Comprehensive student evaluation transcends academic performance, necessitating a nuanced understanding of individual strengths, weaknesses, opportunities, and threats. Machine learning models, employing clustering analysis algorithms  <cit.>, facilitate intricate SWOT analyses, enabling educators to devise tailored learning strategies and interventions. Software applications like IBM WatsonAnalytics3[https://www.ibm.com/watson3. https://www.ibm.com/watson]
 and Tableau4[https://www.tableau.com/4. https://www.tableau.com/]
 harness sophisticated algorithms to decode complex student profiles, fostering personalized academic trajectories.


  * Transparency for Students:
The ambiguity surrounding grading criteria, learning objectives, and assessment methodologies undermines student engagement and comprehension. Natural Language Processing (NLP) techniques, coupled with sentiment analysis algorithms  <cit.>, elucidate grading metrics and curricular objectives, fostering transparent and collaborative learning environments. Modern platforms like Turnitin5[https://www.turnitin.com/5. https://www.turnitin.com/]
 and Grammarly6[https://www.grammarly.com/6. https://www.grammarly.com/]
 employ AI-driven NLP tools to provide actionable feedback, enhancing transparency and academic integrity.


  * Better Performance Indicators with Personalized Feedback:
Conventional performance metrics often lack granularity, overshadowing individual accomplishments and areas of improvement. AI-driven data analytics platforms, harnessing predictive analytics algorithms, empower educators to deliver personalized performance indicators and actionable insights, optimizing learning trajectories and student engagement. Software solutions such as Blackboard Analytics7[https://www.blackboard.com7. https://www.blackboard.com] and Moodle8[https://moodle.org/8. https://moodle.org/] leverage advanced algorithms to analyze student data, facilitating targeted interventions and pedagogical enhancements.





 §.§   The Point of This Book Chapter:


Numerous studies have underscored the profound potential of Artificial Intelligence (AI) in augmenting and revitalizing the contemporary education system  <cit.>. However, the integration of 'Modern' education with AI is not devoid of challenges, particularly concerning financial implications and accessibility disparities.

A plethora of global research, spanning developed nations such as Norway  <cit.>, Japan  <cit.>, developing nations such as China  <cit.> to least developed African countries  <cit.> like Ghana  <cit.>, elucidates a direct correlation between parental income and the quality of education availed by their progeny. In the 21st century landscape, while AI embodies an indispensable cornerstone of educational advancement, its implementation entails substantial costs encompassing resource allocation, hiring, cross-testing, verification, licensing, and deployment  <cit.>. Consequently, the escalating financial exigencies associated with modern education exacerbate accessibility barriers9,[https://www.thehindubusinessline.com/opinion/columns/c-p-chandrasekhar/the-alarming-rise-in-education-costs-in-new-india/article33215181.ece9. https://www.thehindubusinessline.com/opinion/columns/c-p-chandrasekhar/the-alarming-rise-in-education-costs-in-new-india/article33215181.ece]
10[https://timesofindia.indiatimes.com/india/rising-cost-of-education-worries-parents-survey-shows/articleshow/17946981.cms?from=mdr10. https://timesofindia.indiatimes.com/india/rising-cost-of-education-worries-parents-survey-shows/articleshow/17946981.cms?from=mdr]
 <cit.>, constraining students hailing from economically disadvantaged backgrounds. Regrettably, instances abound wherein academically meritorious students, despite securing admissions into prestigious global universities, confront insurmountable financial hurdles, precluding educational attainment due to parental income constraints.

We acknowledge the fact that parental income substantially influences the attainability of educational opportunities for students. In this study, we want to investigate this aspect of parental income. How the income of an individual is dependent on which factors. How can we visualise those dependencies? Moreover, are those dependencies fair? If not, how can we investigate the unfairness? Is the Notion of feature importance enough to recognise a biased model? This book chapter endeavours to delve deeper into these pivotal nexus. We use several methodologies for visualising the feature importance from a model and then we investigate whether are they enough for investigating (un)fairness. We use the adult census dataset in this study to work on this binary classification problem of predicting whether an individual (can) earn more than 50k USD a year, given the other attribute.  This is the primary indicator for our study where we're presuming (based on the earlier studies mentioned above) that the individual, as a parent, enhances the chance of availing better education for his progeny. Through rigorous analysis and interpretation, this chapter aspires to furnish insights, foster dialogue, and catalyze informed interventions addressing the confluence of AI, education, and socio-economic disparities, steering towards an inclusive and equitable educational landscape. 

In this book chapter, we have extensively scrutinized several recent research papers, tutorials, official documentation, and internet articles to gather information and reproduce the relevant results from current research. This book chapter is meant to be suitable for both technical as well as non-technical readers. We have, furthermore, considered numerous ways of structuring this chapter inspired by our scrutinization. The links to relevant resources are kept in the footnotes below. 

Finally, Our study, by no means, is meant to be devised as an Oracle but we fundamentally believe in asking for explanations for high-stakes decisions taken by AI models for the sake of trustworthiness. In the education sector, this has been stressed in recent times <cit.>, and in this chapter, we are demonstrating the same on one of the fundamental aspects of availing 'Modern' education: parental income using state-of-the-art xAI techniques.
[resources:]
[<https://bit.ly/3u2v0ay>]
[<https://rb.gy/dd2dcx>]
[<https://bit.ly/3S1S9lo>]
[<https://rb.gy/ekghfm>]
[<https://rb.gy/qusxjj>]
[<https://rb.gy/fq4ysa>]
[<https://shorturl.at/dhAO7>]
[<https://shorturl.at/ntvBH>]
[<https://bit.ly/425kmg4>]
[<https://bit.ly/3O2DBko>]
[<https://bit.ly/3SjuPRR>]




§ INHERENT COMPLEXITY: A DOUBLE-EDGED SWORD


As discussed, while AI harbours immense potential to revolutionize education, fostering personalized learning, enhancing administrative efficiency, and facilitating data-driven decision-making, its inherent complexity poses formidable challenges, particularly concerning transparency and interpretability in deducing decisions which concurrently introduces nuanced challenges that necessitate vigilant scrutiny and strategic navigation. Sophisticated AI software including Llms <cit.>, SaaS and commercial software such as <cit.>, and virtual assistants <cit.>, while endowed with remarkable capabilities, often manifests as convoluted enigmas, presenting two underlying problems:


    
  * Explanation Gap: AI models, characterized by intricate hyperspaces defined by multifaceted weights and attributes, inherently lack traceability and debuggability. The stochastic nature of the training phase, coupled with the intricacies of the hyperspace, obfuscates the retrieval and comprehension of model parameters and rationales. Consequently, elucidating the 'why' behind AI predictions remains a daunting endeavor, devoid of straightforward exploration avenues.
    
    
  * Trust in the `Black-Box': For seasoned machine learning practitioners well-versed in model architectures and training phases, navigating the intricacies of AI may seem feasible. However, for end-users devoid of specialized expertise, AI often emerges as an opaque 'black box', necessitating dual proficiency as AI and domain experts to engender trust. While domain expertise may be commonplace, it scarcely equates to an unwavering confidence in results generated by intricate AI systems, perpetuating scepticism and apprehension.



 §.§ Explainable AI (XAI): Bridging the Trust Gap


In the pursuit of deciphering the intricacies of 'black-box' AI systems and fostering trust among end-users, scientists have embarked on a relentless quest, culminating in the emergence of Explainable AI (XAI) as an active area of research in this decade. XAI transcends traditional AI paradigms, endeavouring to illuminate the opaque nature of complex machine learning models by enabling:


    
  * Enhanced Explainability: XAI endeavours to cultivate machine learning techniques that aim to yield more explainable models without compromising learning performance or prediction accuracy <cit.>. By elucidating model rationale and inherent strengths and weaknesses, XAI augments user comprehension and trust in several high stakes (for example: <cit.>), pivotal for fostering collaborative partnerships between humans and AI entities.
    
    
  * Human-Centric Design: XAI advocates for the integration of state-of-the-art human-computer interface techniques, facilitating the translation of intricate models into comprehensible and actionable insights for end-users <cit.>. By crafting intuitive explanation dialogues and interfaces, XAI empowers users to navigate, interpret, and leverage AI capabilities effectively, transcending the complexities of underlying algorithms and architectures.
    
    
  * Diverse Methodological Portfolio: Recognizing the multifaceted challenges inherent in achieving explainability without compromising performance, XAI adopts a multifaceted approach, exploring and integrating diverse techniques across the performance-versus-explainability trade space <cit.>. By curating a portfolio of methodologies, XAI equips developers with versatile design options, facilitating tailored solutions aligned with specific application domains and user requirements.





§ METHODOLOGY: EXPLAINABILITY, INTERPRETABILITY AND MORE

Before jumping into the technical and mathematical intricacies of xAI, it is critical to distinguish between explainability and interpretability in machine learning applications for our objectives. We regard interpretability as a means of obtaining explainability. Explainable AI (or XAI) refers to models whose results can be comprehended by ordinary humans. However, this does not imply that the model is required to be interpretable. That is, there are two types of models: interpretable models and non-interpretable models. Linear regression, logistic regression, decision trees etc are examples of interpretable models. Practitioners can simply determine how the model outputs either its label or value. Then there are non-interpretable models for image processing and natural language processing, such as random forests, feed-forward neural networks, and deep learning architectures <cit.>.
In order to make sense of the process by which the intricate model that was employed to generate the forecast came to be, we employ post hoc explanations <cit.> for these. This is frequently accomplished by examining the model's gradients or using stand-in models to simulate behavior in specific areas.

We believe that the line that separates interpretability from explainability also follows the processes of the human mind. For example, human beings occasionally use deliberate, logical thinking to arrive at a conclusion. As a result, the logic can be examined in detail.
In other cases, people base their initial conclusions on intuition after analyzing complex facts over extended periods of time. Then, just as we use post hoc explanations on black box models, the people look for an explanation for a decision. Determining which strategy is superior for explainable AI is probably going to need major advancements in human cognitive science and our comprehension of the functioning of the human brain.


 §.§ Post-hoc Explainations

In regards to the realistic scenario <cit.>, in which the end user looks at the details after a study has been concluded and the data collected, our goal is to provide context for the predictions of a machine learning model. In order to do this, we must first define what it means to produce an explanation. An explanation typically states that some features are more significant than others or explains how an input feature can influence the output's magnitude in a positive or negative way.  It usually links the feature values of an instance to its model prediction in a way that is easily understood by humans. Not all explanations must be focused on the significance of a feature. They could also be particular cases or groups of influential instances. They could be real English languages or collections of rules. In fact, difficulties with homogeneous judgment arise from a variety of reasons. In our current study, we focus primarily on the importance of the feature.

Post hoc approaches might be model-agnostic or model-dependent, global or local.
Model-dependent interpretation techniques are focused on a particular model or set of models. Model-agnostic tools, on the other hand, can be used on any machine learning model, no matter how complex. These agnostic approaches typically analyze feature input and output pairs. Model-agnostic methods frequently profit from the reality that certain practitioners may not have access to the original model's specific weights and parameters. In our current work, we have been focused on local `Post-hoc' explanation methods in model-agnostic fashion. Below is the description of the methods we used in study for generating post-hoc explanation.

    
  * LIME <cit.>:
One of the most well-known post hoc methods is LIME or local interpretable model-agnostic explanations. LIME's concept is to build a surrogate model in a local area at a certain target point to explain the significance of the input features. Because it hypothesized that sophisticated black boxes tend to demonstrate more linear or simple behavior in local neighborhoods. We build new samples for LIME by perturbing for the target sample. We then query the black-bo model to obtain the label of the perturbed data points, scores them using a kernel, and train a sparse linear model using the data-points and labels.

  * KernalSHAP <cit.>:
	Shapley values <cit.> are based on the idea that a prediction may be explained by imagining each feature value of the instance as a 'player' in a game where the prediction is the payoff. Shapley values, a strategy from coalitional game theory, teach us how to allocate the "payout" among the characteristics in a fair manner.

Shapley regression values serve as indicators of feature importance in linear models, particularly when confronted with multicollinearity. This approach mandates the model's retraining across all conceivable feature subsets S ⊆ F, where F denotes the complete set of features. Each feature is assigned an importance value, signifying the impact of its inclusion on the model prediction. The computation involves training a model f_S ∪{i} with the specific feature and another model f_S without it. Subsequently, the discrepancy in the predictions for the current input is assessed as f_S ∪{i}(x_S ∪{i})-f_S(x_S), where x_S represents the input feature values within the set S. As the influence of excluding a feature is contingent on other model features, these differences are computed for all feasible subsets S ⊆ F \{i}. Subsequently, the Shapley values are calculated and used as feature attributions, representing a weighted average of all potential differences <cit.>:


    ϕ_i = ∑_S ⊆ F \{i}|S| !(|F|-|S|-1) !/|F| ![f_S ∪{i}(x_S ∪{i}) - f_S(x_S)]

shaply value calculation





 































However, both LIME and Kernel SHAP share the same foundation but choose different kernels and loss functions to calculate the importance of features <cit.>.
With this, we are now moving to Definitions. Thoughts have been mainly drawn from  <cit.>

An algorithm's interpretability is its ability to give users enough expressive data to comprehend how the algorithm operates. In this case, human-readable text or graphics may be considered interpretable domains. "If something is interpretable, it is possible to find its meaning or to find a particular meaning in it," according to the Cambridge Dictionary.



Interpretation is the process of reducing a complicated domain—like the outputs of a machine learning model—to meaningful, rational, and human-understandable ideas.



An explanation is extra metadata that describes the feature importance or relevance of an input instance to a certain output classification. It can be produced by the machine learning model itself or by an external method.





An xAI post hoc explanation (here, LIME & KernelShaP) works by assuming that the model architecture is a black box.  Most post hoc explainers build a single loss function using heuristics, gradients, game theory, or some other method <cit.>. However, the black-box functions are based on the feature set's global domain. These explainers, on the other hand, mimic functions in the local domain perturbing a few examples.




















    






§ EXPERIMENTAL SETUP

The study follows the guidelines given in <cit.> for experimentation and implementation particulars. It uses the Adult Census Income dataset from the UCI Machine Learning Repository. We start our experiments by training an XGBoost <cit.>  model.
The following performance metrics were achieved for the model: 
    Accuracy     Precision     Recall     F1     AUC 

Train     0.88     0.82    0.65    0.73     0.95

Test     0.87     0.78    0.62    0.69    0.92
We start our investigation on the trained XGBoost based on easy to complex hypothesis gradually for calculating feature importance. We start with the ELI5 library<cit.>. We used the library's `Permutation Importance' (PI) method to compute the feature importance. It works on the hypothesis that the importance of a feature can be quantified by the drop in the intended metrics (accuracy, precision etc) when removed. 
In this study, we use AUC <cit.> as the performance metric to evaluate the trained XGBoost model. We report the PI for both train and test counterparts in Figure <ref>: the left column shows the PI in the train set and the one in the right is for the test.
Despite the fact that the order of the most significant traits varies, it appears that the most crucial one () remains the same in both folds. Furthermore, the six most important variables based on PI are retained in training and testing. We anticipate their difference in relative ordering are mostly due to the sample dispersion between the folds. 

Next, we use a more sophisticated method named SHAP <cit.> for visualizing the feature importance. We also want to compare the feature importance from the trained XGBoost with the ones obtained from SHAP. We use Gain as the default feature importance for the XGBoost model. Gain of a feature is computed by taking its relative contribution in each tree for the whole model. In the comparison between the SHAP feature importance and XGBoost's counterpart, we take the top 20 features in both of them.
In Figure <ref>, we report the XGBoost feature importance. We compare the same against the mean of the absolute SHAP values presented in Figure <ref>. Finally, in Figure <ref> we report the SHAP values of all these features to demonstrate their impact on the model.









It is clear from the comparison that the most important feature () is common in both the comparison. Next,  is there within the top-3 features in both feature importance ranking. However, the rest of the features including `' are either not in common or ranked differently based on their importance in the comparisons. 
Next, we move to the summary plots produced by SHAP for a nuanced overview of feature importance.


 §.§    Summary Plot SHAP

The SHAP Summary Plot (Figure <ref> ) offers more information than the conventional plot. The significance of features is arranged according to decreasing feature relevance.

Impact on Prediction: The horizontal axis's position reflects how much or how little the values of the dataset instances for each feature affect the model's output.

Original Value: The colour designates a high or low value (within each feature's range) for each characteristic.

Correlation: A feature's colour (or range of values) and the effect on the horizontal axis are usually used to assess how well it correlates with the model output. 

Notably, SHAP identifies the variable  as having significant interaction, suggesting that individuals who are married and possess higher educational qualifications are more inclined to earn above $50,000.














Next, we move to Dependency Plots made by SHAP.


 §.§ Dependency plots

 Dependency plot not only provides the marginal influence the feature has on the model's output, but it also illustrates the relationship with the feature with which it most interacts by colour. This illustrates the model's reliance on the provided feature.  The vertical spread of data points indicates interaction effects. We presented some dependency plots (Figure <ref> with different pairs of features to present how the interaction is between characteristics and how one influences another. 


  

  




As depicted in Figure <ref>, there's a discernible upward trend in marriages between the ages of 20 to nearly 50 years. Meanwhile, Figure <ref> reveals a pronounced rise in the education.num concerning capital gains ranging from 0 to 20k USD. Figure <ref> further elucidates a consistent progression between educational attainment and marital status. Lastly, Figure <ref> underscores that married individuals tend to exhibit higher capital gains compared to their unmarried counterparts.

Now, in the next section, we'll discuss post-hoc local explanations using LIME and SHAP.


§ DISCOVERING LOCAL SCALES

Local surrogate models <cit.> are interpretable models that are used to explain individual predictions of black-box machine learning models.


 §.§ LIME

LIME <cit.> stands for Local Interpretable Model-independent Explanations. LIME investigates what occurs in model predictions when the input data is changed. It creates a new dataset containing permuted samples and the old model's related predictions. LIME trains interpretable models (Logistic Regression, Decision Tree, LASSO Regression, etc.) on this synthetic set, which are then weighted by the closeness of the sampled examples to the instance of interest.

The explanation, for example, X will be that of the surrogate model that minimises the loss function (performance measure for example, MSE- between the surrogate model's forecast and the prediction of the original model), while keeping the model's complexity low.

However, LIME has a distinct problem; it is inherently unstable. As illustrated in Figure <ref>, LIME often produces varied explanations for identical instances when executed multiple times. This inconsistency, while not always prevalent, arises from the inherent randomness in generating the surrogate models employed by its linear models. This is not the case with SHAP as it enjoys the uniqueness of SHAP values for a given instance <cit.>.  There are various ways to visualise SHAP explanations. This chapter will go over two of them: the Decision & Force plot <cit.>.


  * Force Plot:
It shows the effect that each attribute had on the forecast. The output value (model prediction for the instance) and the base value (average prediction for the whole dataset) are the two important numbers to pay attention to. Greater influence is shown by a larger bar, and the colour shows whether the feature value shifted the forecast from the base value to 1 (red) or 0 (blue). 

We run the same experiment 10 times to show that the explanation is unique and does not exhibit randomness like LIME <ref>.

  * Decision Plots:
The Decision Plot: (Figure <ref>) displays the same information as the Force Plot. The gray vertical line represents the baseline, and the red line indicates whether each characteristic increased the output value higher or lower than the average forecast. Sometimes the information representation depends upon the end user's intention due to which various representations have been engineered under SHAP library. 






























































So far, we have investigated and extensively visualised the feature importance using several representations. Next, we move to mimicking the model globally rather than mere feature importance estimation. We use global surrogate models for globally explaining the model.


 §.§  Global Explanations

An interpretable model that has been trained to match the predictions of the black-box model to a very large extent is a global surrogate model. By analysing the surrogate model, we may make inferences about the black box model. Though it, by any means, is not a way to predict the internal architecture of the black box, rather it is meant to deduce some insights regarding the prediction.
Using a Decision Tree and a Logistic Regression as global surrogate models, we will attempt to approximate the XGBoost.For both the Train and Test sets, R-squared is negative in the case of Logistic Regression (-1.40 , -1.42 respectively). This occurs when the fit is poorer than using the mean alone. As a result, it is argued that Logistic Regression is NOT an adequate surrogate model.

However, we have obtained 0.77 & 0.79 for R-Square scores in train and test respectively in the case of decision tree, so we can proceed with the same if it mimicks the trained XGBoost well.

 
    Accuracy     Precision     Recall     F1     AUC 

Train     0.85     0.75     0.55     0.64     0.868197 

Test     0.85     0.75     0.55     0.64     0.865062
The Decision Tree approximates the decisions of the  XGBoost model quite well as per the scores obtained, hence it may be used as a surrogate model for evaluating the main model. However,  it is not guaranteed that the decision tree uses the features in the same way as the XGBoost. It is possible that the tree approximates the XGBoost properly in certain portions of the input space but acts erratically in others as the models are themselves different. We can see in Figure <ref> that  is the most important feature followed by  and .  In the importance of the XGBoost feature, these were the top-3 features and in the SHAP dependency plots as well, these were important features to consider. 

    
    < g r a p h i c s >

  See the tree!
  

Next, we use another method named SP-LIME for generating global explanation.



 §.§ Submodularity:









 The submodular picking <cit.> is important as end users may not have time to examine a large number of explanations. It is critical to choose which instances to explain with care. It seeks to provide explanations for machine learning models by attributing their predictions to human-understandable features. To achieve this objective effectively, it's essential to execute the explanation model across a varied yet representative collection of instances. By doing so, LIME aims to produce a non-redundant set of explanations that collectively offer a comprehensive understanding of the model's behaviour on a global scale. SP LIME algorithm selects a collection of explanations for the user that are diverse and representative, meaning they do not repeat themselves, while also demonstrating how the model works in general as shown in <cit.>. Technically, each instance being a collection of features is a subset of the whole feature set. Collecting a minimal or almost minimal number of subsets to cover the whole or almost the whole set is the main agenda of employing the SP LIME algorithm.
This problem is a derivation of the set cover problem <cit.> which is NP-complete. LIME has its own greedy picking <cit.> algorithm which we have demonstrated in Figure <ref>. More technical details can be found at  <cit.>.


    
    < g r a p h i c s >

    Submodular pic
    

In Figure <ref>, capital.gain is shown to be the most important attribute in all individual interpretations that distinguish the classes. Following that, depending on the situation, the most important determinants are married1, education.num, age, and sex. These features align with those pinpointed by the algorithms as globally significant.













































































































So far, we have investigated how the model's features interplay and exhibit patterns. However, whether this interplay is fair & legitimate is what we investigate in the next section.


§ ALL IS WELL, BUT IS ALL FAIR?

In machine learning, "fairness" refers to the several ways that algorithmic bias in machine learning-based automated decision processes is attempted to be corrected. If computer decisions after a machine-learning process were predicated on factors deemed sensitive, they may be viewed as unjust. These variables can be related to gender, ethnicity, sexual orientation, disability, and other factors. Definitions of prejudice and justice are inherently contentious, as is the case with many ethical notions. Fairness and prejudice are typically taken into consideration when making decisions that have an influence on people's lives. The issue of algorithmic bias in machine learning is generally recognised and researched. Many factors have the potential to distort the results, making them unfair to particular persons or groups.

'FairML' <cit.>, the library for fairness in Python helps to investigate these intricacies.

The underlying idea behind FairML (and many other attempts to audit or understand model behaviour) is to change the inputs of a model to quantify its dependency on them. The model is sensitive to a feature if a modest change to an input feature drastically impacts the output.
 In order to quantify how much each characteristic influences the prediction model, FairML projects the input orthogonally.

Let F(x, y) be a model trained on two characteristics, x and y. The change in output resulting from a modified input, where x' perturbs x, can be quantified to measure the model's reliance on x. We express this as:

    Δ F(x', y) = F(x', y) - F(x, y)
caputing the change in output
Here, Δ F(x', y) represents the change in the model output, and the perturbation in x' renders the other feature y orthogonal to x'.

However, linear transformation is an orthogonal projection. FairML uses a basis expansion along with greedy search in the case of non-linear relationships. But, in auditing a model, the determination of fairness is often nuanced and context-dependent. The sensitivity of features can vary across different contexts; what may be considered sensitive in one scenario may not hold the same significance in another. The definition of fairness is multifaceted, influenced by user experience, and cultural, social, historical, political, legal, and ethical considerations. Identifying appropriate fairness criteria involves navigating trade-offs among these factors.


The visualization: Figure <ref> uses red bars to signify a contribution to an output of  Income > USD 50K; light blue bars stand for the opposite. FairML reveals a significant dependence on sensitive features like , , and . Specifically, the model indicates a strong bias: according to its predictions, an individual is more likely to make more than USD 50k if the individual's nationality is of the United States,  race is white, and sex is male.
Notably, the algorithm's orthogonal projection brings out the relevance of features like , which did not appear in other interpretation methods like SHAP’s Feature Importance. This underscores the importance of investigating for bias along with feature importance. As we observe the popular methods for feature importance may not always be capable of uncovering the intrinsic bias and unfairness in trained models.
Now, in the case of availing better education, we started by looking at how parental income influences the education of children and these types of severe bias are what we've found out. This type of societal bias directly impacts the household and the education of the children.
In our assessment, the principles of fairness and transparency appear conspicuously absent. All is simply, not Fair <cit.>.


§ CONCLUDING REMARK:





Fairness in machine learning models is a nuanced and evolving concept. Fairness considerations need to account for diverse perspectives, cultural nuances, and the dynamic societal landscape. Moreover, the interpretability of fairness metrics remains an open question. Defining fairness is not a one-size-fits-all endeavour, and achieving fairness often involves navigating complex trade-offs between different dimensions of fairness <cit.>.


Our comprehensive analysis delved deeply into the critical interplay between parental income and its consequential impact on educational opportunities, harnessing advanced xAI tools such as LIME, SHAP, and the FairML library. By scrutinizing predictive models derived from the Adult Census data, our deliberate preference for sophisticated algorithms like XG Boost and Decision Trees over simpler models aimed to mirror the inherent complexities typical of real-world black-box systems. However, our findings unveil a disconcerting pattern: pervasive biases persistently permeate these models. While FairML identified these biases as direct orthogonal projections, SHAP analyses sometimes remained oblivious to them. Intriguingly, LIME's volatility introduced yet another layer of complexity, further complicating our interpretative landscape

In conclusion, we want to revisit the objectives: AI came to education for speed and transparency, a strong pillar for availing education is parental income and when that is being scrutinised, it is showing heavy bias under several tests mimicking the real world. The effort we put here to investigate the situation is solely to question the fundamentals: In society, modern education is the pillar to move the nation ahead whereas the Scope of availing education is deliberately getting shrunk due to several biases and education, as the fundamental right to every citizen is being heavily exclusive to a particular direction where the Bias lends itself. Biases present in the training data may be inadvertently perpetuated or even amplified by xAI algorithms, reinforcing existing societal biases <cit.>. This is why refining fairness definitions, enhancing the transparency of complex models, and mitigating biases in training data are key challenges that warrant continued attention and collaborative efforts from the research and practitioner communities. Through constructive criticism and iterative refinement, xAI can evolve into a more robust and ethically grounded field <cit.>, fostering trust and accountability in the deployment of artificial intelligence systems.


§ FURTHER WORK

We are consistently striving for improved policies for Modern education, emphasizing values of Transparency, Equal Opportunity, & Accessibility. Our primary focus lies in harnessing the potential of xAI to enhance policymaking. Operating within the realm of public policy, our endeavour centres on exploring innovative avenues to render education policies more accessible, starting from the grassroots level. Leveraging advanced algorithms, we aim to bring about positive transformations that align with our commitment to creating a more transparent, equitable, and accessible educational landscape.


99
all exp in oneHan T, Srinivas S, Lakkaraju H. Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations. Advances in neural information processing systems. 2022 Dec 6;35:5256-68.
LIMERibeiro MT, Singh S, Guestrin C. " Why should i trust you?" Explaining the predictions of any classifier. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining 2016 Aug 13 (pp. 1135-1144).
SHAP Lundberg SM, Lee SI. A unified approach to interpreting model predictions. Advances in neural information processing systems. 2017;30.
SHAPLY valuesShapley LS. Notes on the n-person game—ii: The value of an n-person game.

closed form solutionHill D, Masoomi A, Ghimire S, Torop M, Dy J. Explanation Uncertainty with Decision Boundary Awareness. arXiv preprint arXiv:2210.02419. 2022 Oct 5.

fairMLAdebayo JA. FairML: ToolBox for diagnosing bias in predictive modeling (Doctoral dissertation, Massachusetts Institute of Technology).

xAI BookMolnar C. Interpretable machine learning. Lulu. com; 2020.

SubmodularityKrause A, Golovin D. Submodular function maximization. Tractability. 2014 Feb 6;3(71-104):3.

set coverFeige U. A threshold of ln n for approximating set cover. Journal of the ACM (JACM). 1998 Jul 1;45(4):634-52.


def1Arrieta AB, Díaz-Rodríguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, García S, Gil-López S, Molina D, Benjamins R, Chatila R. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion. 2020 Jun 1;58:82-115.
def2Bellucci M, Delestre N, Malandain N, Zanni-Merk C. Towards a terminology for a fully contextualized XAI. Procedia Computer Science. 2021 Jan 1;192:241-50.
def3Adadi A, Berrada M. Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE access. 2018 Sep 16;6:52138-60.

xgbChen T, Guestrin C. Xgboost: A scalable tree boosting system. InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining 2016 Aug 13 (pp. 785-794).
el5Kawakura S, Hirafuji M, Ninomiya S, Shibasaki R. Adaptations of Explainable Artificial Intelligence (XAI) to Agricultural Data Models with ELI5, PDPbox, and Skater using Diverse Agricultural Worker Data. European Journal of Artificial Intelligence and Machine Learning. 2022 Dec 2;1(3):27-34.

foolSlack D, Hilgard S, Jia E, Singh S, Lakkaraju H. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society 2020 Feb 7 (pp. 180-186).

r1Deshpande A, Sharp H. Responsible AI Systems: Who are the Stakeholders?. InProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society 2022 Jul 26 (pp. 227-236).
r2Soklaski R, Goodwin J, Brown O, Yee M, Matterer J. Tools and practices for responsible AI engineering. arXiv preprint arXiv:2201.05647. 2022 Jan 14.
r3McElhaney K, Smith G, Rustagi I, Groth OJ. Responsible AI: Tackling Tech’s Largest Corporate Governance Challenges. The Berkeley-Haas Case Series. University of California, Berkeley. Haas School of Business; 2023.
r4Amarasinghe K, Rodolfa KT, Lamba H, Ghani R. Explainable machine learning for public policy: Use cases, gaps, and research directions. Data & Policy. 2023;5:e5.
f1Amodei D, Olah C, Steinhardt J, Christiano P, Schulman J, Mané D. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. 2016 Jun 21.
study1Machebe CH, Ezegbe BN, Onuoha J. The Impact of Parental Level of Income on Students' Academic Performance in High School in Japan. Universal Journal of Educational Research. 2017;5(9):1614-20.
study2Lin T, Lv H. The effects of family income on children’s education: An empirical analysis of CHNS data. Research on Modern Higher Education. 2017 Apr;4(02002).
study3Wiborg ØN, Grätz M. Parents’ income and wealth matter more for children with low than high academic performance: Evidence from comparisons between and within families in egalitarian Norway. Research in social stratification and mobility. 2022 Jun 1;79:100692.
study4Glick P, Sahn DE. Schooling of girls and boys in a West African country: the effects of parental education, income, and household structure. Economics of education review. 2000 Feb 1;19(1):63-87.
study5Ani-Asamoah Marbuah D. Influence of Parental Income and Educational Attainment on Children’s Years of Schooling: Case of Ghana.
edu1Woolf B. AI in Education. University of Massachusetts at Amherst, Department of Computer and Information Science; 1991.
edu2Zhai X, Chu X, Chai CS, Jong MS, Istenic A, Spector M, Liu JB, Yuan J, Li Y. A Review of Artificial Intelligence (AI) in Education from 2010 to 2020. Complexity. 2021 Apr 20;2021:1-8.
edu3Beck J, Stern M, Haugsjaa E. Applications of AI in Education. XRDS: Crossroads, The ACM Magazine for Students. 1996 Sep 1;3(1):11-5.
edu4Tahiru F. AI in education: A systematic literature review. Journal of Cases on Information Technology (JCIT). 2021 Jan 1;23(1):1-20.
trust in educationVincent-Lancrin S, Van der Vlies R. Trustworthy artificial intelligence (AI) in education: Promises and challenges.
ai recomCho E, Han M. AI powered book recommendation system. InProceedings of the 2019 ACM Southeast Conference 2019 Apr 18 (pp. 230-232).
clus algoXu D, Tian Y. A comprehensive survey of clustering algorithms. Annals of Data Science. 2015 Jun;2:165-93.
sentiMedhat W, Hassan A, Korashy H. Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal. 2014 Dec 1;5(4):1093-113.
cost of aiLuccioni AS, Jernite Y, Strubell E. Power Hungry Processing: Watts Driving the Cost of AI Deployment?. arXiv preprint arXiv:2311.16863. 2023 Nov 28.
cost of higher educationLi H. The rising cost of higher education: A supply and demand analysis. NY University. 2013 May.
llamaTouvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S, Bikel D. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023 Jul 18.
h2oMalohlava M, Hava J, Mehta N. Machine learning with sparkling water: H2o+ spark. H2O. ai Inc. 2016 Apr.
article-1Pretto, F. (2020, July 30). Uncovering the magic: Interpreting machine learning black-box models. Medium. https://towardsdatascience.com/uncovering-the-magic-interpreting-machine-learning-black-box-models-3154fb8ed01a 
article-2Pretto, F. (2020b, August 12). Inclusive machine learning: Addressing model fairness. Medium. https://towardsdatascience.com/inclusive-machine-learning-addressing-model-fairness-532884ecd859 
einsteinShrivastava M. Learning Salesforce Einstein. Packt Publishing Ltd; 2017 Jun 28.
geminiTeam G, Anil R, Borgeaud S, Wu Y, Alayrac JB, Yu J, Soricut R, Schalkwyk J, Dai AM, Hauth A, Millican K. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. 2023 Dec 19.
gptWu T, He S, Liu J, Sun S, Liu K, Han QL, Tang Y. A brief overview of ChatGPT: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica. 2023 May 1;10(5):1122-36.
alexa et allKepuska V, Bohouta G. Next-generation of virtual personal assistants (microsoft cortana, apple siri, amazon alexa and google home). In2018 IEEE 8th annual computing and communication workshop and conference (CCWC) 2018 Jan 8 (pp. 99-103). IEEE.
perf vs expCrook B, Schlüter M, Speith T. Revisiting the performance-explainability trade-off in explainable artificial intelligence (XAI). In2023 IEEE 31st International Requirements Engineering Conference Workshops (REW) 2023 Sep 4 (pp. 316-324). IEEE.
haiHorvatić D, Lipic T. Human-centric AI: The symbiosis of human and artificial intelligence. Entropy. 2021 Mar 11;23(3):332.
xai trustMahbooba B, Timilsina M, Sahal R, Serrano M. Explainable artificial intelligence (XAI) to enhance trust management in intrusion detection systems using decision tree model. Complexity. 2021 Jan 28;2021:1-1.
xai defArrieta AB, Díaz-Rodríguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, García S, Gil-López S, Molina D, Benjamins R, Chatila R. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion. 2020 Jun 1;58:82-115.
postJesus S, Belém C, Balayan V, Bento J, Saleiro P, Bizarro P, Gama J. How can I choose an explainer? An application-grounded evaluation of post-hoc explanations. InProceedings of the 2021 ACM conference on fairness, accountability, and transparency 2021 Mar 3 (pp. 805-815).
surrogateNóbrega C, Marinho L. Towards explaining recommendations through local surrogate models. InProceedings of the 34th ACM/SIGAPP Symposium on Applied Computing 2019 Apr 8 (pp. 1671-1678).
AUCBradley AP. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition. 1997 Jul 1;30(7):1145-59.





 Paper Title: Distributionally Robust Optimization as a Scalable Framework to Characterize Extreme Value Distributions.


Energy-lowering symmetry breaking creates a flat-band insulator in paramagnetic Nb_3Cl_8
    Alex Zunger
    August 5, 2024
========================================================================================



§ ABSTRACT 
The goal of this paper is to develop distributionally robust optimization (DRO) estimators, specifically for multidimensional Extreme Value Theory (EVT) statistics. EVT supports using semi-parametric models called max-stable distributions built from spatial Poisson point processes. While powerful, these models are only asymptotically valid for large samples. However, since extreme data is by definition scarce, the potential for model misspecification error is inherent to these applications, thus DRO estimators are natural. In order to mitigate over-conservative estimates while enhancing out-of-sample performance, we study DRO estimators informed by semi-parametric max-stable constraints in the space of point processes. We study both tractable convex formulations for some problems of interest (e.g. CVaR) and more general neural network based estimators. Both approaches are validated using synthetically generated data, recovering prescribed characteristics, and verifying the efficacy of the proposed techniques. Additionally, the proposed method is applied to a real data set of financial returns for comparison to a previous analysis. We established the proposed model as a novel formulation in the multivariate EVT domain, and innovative with respect to performance when compared to relevant alternate proposals. 




§ INTRODUCTION



Modeling rare and extreme events is an important task in many disciplines such as finance, climate science, and medicine <cit.>. Estimating distributions of rare events from data is difficult due to the lack of observations within this region, making it challenging to understand the risks deep in the tail of a distribution. Extreme Value Theory (EVT) studies the class of distributions arising as the possible distributional limits that can be used to estimate multivariate distributions in distant (relative to the origin) regions (i.e., tails) which by their nature witness very few observations (or none at all). These distributional limits are derived as the possible asymptotic statistical laws of shifted and re-scaled data as the sample size increases. It turns out that such possible distributional limits form a semi-parametric class called max-stable distributions, which are constructed in terms of a spatial Poisson point process. 

Naturally, because of the lack of data in extremal regions and because of the asymptotic nature of max-stable models, their use in inferential tasks involving tails is exposed to high variance due to model misspecification. Moreover, when using max-stable models, the assumptions that the data converges to the distributions specified by EVT must be made. This leads to an important question: How can we robustify against scenarios deep in the tails while observing potentially sub-asymptotic data where the assumptions of EVT may be violated? To answer this question, we propose a solution based on distributionally robust optimization (DRO). DRO involves a zero-sum game in which the statisticians play against an adversary that perturbs (in a non-parametric way) the nominal / baseline distribution assumed by the statistician. Building on classical DRO, we carefully design constraints to retain the extrapolation properties of EVT for the robustified distribution.

Since we are interested in robustifying the tails, we will consider max-stable baseline distributions given by extreme value distributions (EVDs), as presented by <cit.>. Max-stability roughly states that the distribution of the maximum of independently and identically distributed (i.i.d.) samples belongs to  the same distribution up to a change in the location and scale parameters. This means that the “shape” of the distribution is preserved under the max operation, and it is this property that allows for extrapolating to regions outside the observation domain. In our robustification framework, we therefore wish to preserve this max-stability property. We do this by searching over the space of distributions that are also max-stable, in effect constraining our search to only distributions that extrapolate according to EVT, which form a semi-parametric class. We do this by carefully designing the uncertainty set such that the necessary max-stable properties are preserved. 

To illustrate our desired result, we refer to Figure <ref>. This figure visualizes how a completely unconstrained adversary (solid line) may be too conservative and may not consider the extrapolating properties of the distribution. A well constrained adversary to a max-stable distribution (dashed line) provides an appropriate balance of coverage while maintaining underlying structural properties. This figure demonstrates a case when even if we consider two adversarial formulations that achieve a similar minimum error value, the properly constrained adversary (dashed line) is preferable because the size of uncertainty is very hard to calibrate. Therefore, a curve that is "flatter" around the minimum as a function of the uncertainty size parameter is preferable.




Our work focuses on the case of multivariate extreme value distributions (MEVs) which characterize the joint risk between different variables. Unlike univariate EVDs which have a fully parametric form, MEVs are semi-parametric, where the dependence structure is an infinite dimensional object, and much harder to estimate. We focus on robustifying the dependence structure while preserving the MEV character. This representation leads to additional difficulties which we overcome through the proposed DRO framework.

In view of the fact that the lack of data will naturally induce model error, we introduce an approach to quantify model misspecification based on optimal transport DRO. We select the Wasserstein metric for optimal transport because this approach, together with moment constraints, encompasses most DRO formulations as demonstrated by <cit.>.

The cost structure in the optimal transport discrepancy allows one to balance various objectives when modeling data, specifically the trade-off between tractability and control of pessimism in extremal behavior. A too-powerful adversary makes the size of uncertainty hard to calibrate since a slight increase may result in adversarial policies that perturb a distribution in ways that are too pessimistic and may not be consistent with the constraints imposed by EVT.

We mitigate these concerns by studying DRO formulations which constrain the adversarial perturbations to explore non-parametric models that induce robust estimators while preserving MEV characteristics. One of the approaches we employ is based on optimal transport for point processes, while the other is based on a neural network architecture. We illustrate the performance of the neural network architecture in the context of a Conditional Value at Risk (CVaR) metric applied to a multi-variable extreme value distributed data set. The network is evaluated across several synthetic data sets, specifically constructed to challenge the assumptions associated with EVT. The neural network architecture uses the tractability of the optimal transport distance metric to parameterize model uncertainty. Furthermore, our analysis is extended to a real data set of financial returns as a baseline comparison, similar to the data proposed by <cit.>.This experiment demonstrates precisely the anticipated behavior shown in Figure <ref>, where our methodology leverages a properly constrained adversarial estimator.

Related Work
A number of related research directions exist that consider both estimating EVDs from data as well as robustifying them using DRO. Since our focus is on multivariate EVDs, we will review a few of the estimators from the literature. For estimating multivariate EVDs, copula based approaches have been developed in a variety of instances including work by  <cit.>. Samplers for MEVs have also been considered as demonstrated by  <cit.>. <cit.> provides a flexible framework that uses neural networks to estimate and sample from multivariate EVDs irrespective of dimension, which we use in the computational component of this work. However, all of these methods only consider the case where the model is well-specified and do not consider uncertainty associated with the model class. We build upon these works with the addition of the DRO perspective. 

With regards to the DRO literature, <cit.> described the general framework for Wasserstein DRO that we will use throughout this work. Additional discussion is available in  <cit.> and  <cit.>. Other DRO frameworks for EVT have been considered, but only in the univariate case, by  <cit.>. <cit.> considers DRO under the Kullback-Liebler divergence, which is appropriate for the univariate analysis but may not be appropriate for the multivariate case where supports are likely to be disjoint.

 Finally, <cit.> considers DRO estimators specifically for MEV distributions leveraging extremal coefficient constraints. 
 In this work, upper and lower bounds are established on a Value at Risk (VaR) loss and applied to a real data set of financial returns. 
 These bounds are established over the infinite domain of spectral measures, using a finite set of constraints formulated as a linear semi-infinite program. 
 This is a limited subset of application problems when compared to our investigation. 
 While the general goal of <cit.> is similar to our framework, our method extends to more general risk measures and considers a flexible uncertainty set specified by the Wasserstein distance. 
 Furthermore, we achieve a single robust loss (upper bound), as opposed to less precise (upper and lower) boundaries. 
 In Section <ref> we generate a similar data set to <cit.> and employ our proposed method for comparison. 
 In summary, the main contributions of this paper are: 


  * We introduce a framework to produce DRO estimators for MEV distributions based on optimal transport.


  * We provide tractable DRO formulations for various estimators of interest when adversaries live in the (infinite dimensional) space of point processes.


  * We test the performance of our estimators with the goal of showing that our MEV-constrained adversaries improve performance in the sense of Figure <ref>, and compare to a previous work for baseline analysis.



§ BACKGROUND AND PROBLEM FORMULATION
 
In this section we focus our discussion on concepts critical to our proposed results and we introduce the framework for distributionally robust estimators in MEV distributions.
We first provide a brief overview of EVT and describe how it is used to extrapolate beyond the observed data. 
We then introduce the multivariate counterpart, which we use throughout this work, to describe EVT in multi-dimensional settings.
Finally, we discuss the distributionally robust optimization framework we use based on the Wasserstein distance. 



 §.§ Extreme Value Theory Background


We begin by reviewing concepts surrounding MEV distributions. Consider a sequence of n i.i.d. random vectors {X^(1), , X^(n)}, with X^(i)∈ℝ^d and i = 1, , n and denote the maxima over each dimension as M_k,n:=max_i =1^n X_k^(i), where k ∈{ 1, …, d }. Similarly to univariate EVT analysis, we consider MEV distributions G, where P((M_1,n- b_1,n)/a_1, n≤ z_1, , (M_d,n- b_d,n)/a_d, n≤ z_d) → G(z_1, , z_d), for some normalizing constants a_k, n > 0 and b_k, n, as the number of observations n increases to infinity. 





 §.§ Spectral Representation of Component-wise Maxima and Asymptotic Characterization


We will now introduce specific properties of MEV distributions that we will exploit in our framework. Consider i.i.d. random vectors {X^(1), , X^(n)}, with X^(i)∈ℝ^d and i = 1, , n, with unit Frechét margins such that F_X_k(x) = exp(- 1 /x), with x > 0 for all k = 1, …, d. Following <cit.>, let a sequence N_1, , N_n be a point process, where N_n(·)=∑_i=1^n1_X^(i)/n(·) with N_n(·)d→ N(·) as n →∞ with d denoting convergence in distribution and N is a Poisson point process. We will apply this result further in Section <ref> to define the proposed robustificaiton of the Poisson point process.



  
Decomposing Max-Stable Random Variables
Max-stable distributions can be decomposed according to the radial and spectral decomposition <cit.>. Specifically, if we let Y^(n)∈ℝ^d ∼ H be a sample from the spectral distribution and A^(n) to be the n^th arrival of a unit rate Poisson point process, then a max-stable random variable is represented as


    M = max_n≥ 1Y^(n)/A^(n).


Under the condition that 𝔼[Y_k] =1, the variable M is distributed with unit Frechét margins. This decomposition provides a semi-parametric class of distributions whose structure we will use throughout the rest of the text. With the spectral decomposition of the MEV we are now able to analyze an MEV distribution. Additionally, we introduce Lemma <ref> which allows for the explicit transformation of the MEV cumulative distribution function (CDF). 


    For X^(k) with standard Frechét marginal distributions, we have:
    
    P   (M_1,n/n≤ x_1,⋯, M_d,n/n≤ x_d ) 
       →exp(-V(x_1,…,x_d)),

    where
    
    V(x_1,⋯,x_d)=d∫_Δ_d-1max_k = 1^dw_k/x_kH(dw),

    Δ_d-1 represents the unit d-dimensional simplex and H(·) satisfies: ∫ w_k H(dw)=d^-1 for all k ∈{1, …, d}.


When dealing with MEVs, the important component in estimation is the spectral measure H. When placed in its standardized form, H is a measure with support over the simplex that describes the dependence between the covariates, specifically which components become extreme simultaneously. When a face of the simplex has H non-zero, it implies that covariates associated with the vertices of the face experience extreme events simultaneously. For more details on estimating the spectral measure, we refer the reader to <cit.>. In Section <ref>, we describe how H plays the role of an intensity function when taken in the perspective of the point process.



 §.§ Wasserstein DRO Formulations for EVDs

Referencing the framework to quantify model uncertainty via DRO described by <cit.>, we define the probability space (Ω, ℱ, P), where a candidate robust distribution P is feasible such that d(P, P_0) ≤δ. P_0 is a baseline distribution and d(·) is a distance measure, constrained by the parameter δ. Two methods are commonly employed to quantify model uncertainty when constructing distributional ambiguity sets. The first considers the corruption of the likelihood baseline model to be misspecified, which is addressed via ϕ-divergence ambiguity sets. The second method perturbs the actual data, which leads to the use of Wasserstein distances to quantify model misspecification.  

A recent investigation by <cit.> has demonstrated that both considerations of likelihood and perturbations of data may be unified under the Wasserstein distance, where d(·) = W_c(·). Consider a loss function ℓ : ℝ^d →ℝ_+ and the Wasserstein distance transport cost c: ℝ^d ×ℝ^d →ℝ_+. We define the primal optimization problem as follows:

    max_P : W_c(P, P_0) ≤δ𝔼_X ∼ P[ℓ(X)]

In the context of our problem, P_0 could be the estimated distribution from available samples, but one may not have enough coverage since rare events may not have been observed within the data collection period. Following <cit.>, the dual form of (<ref>) is given by the following problem

    min_λ≥ 0 [ λδ + 𝔼_X ∼ P_0[ max_Zℓ(Z) - λ c(X, Z)  ]  ].

As we will discuss, the dual form is particularly amenable for computational purposes, especially when one can exploit specific properties of c, ℓ, and P. Our investigation will describe two perspectives of (<ref>) for EVDs. The first we present in Section <ref>, and is based on interpreting an MEV through a point process and the second is based on the interpretation using a copula. Each perspective has different properties that are useful for computational purposes, which we will discuss in Section <ref>. 


§ ROBUSTIFICATION OF THE POISSON POINT PROCESS


In this section, we provide a more general formulation for DRO problems using a characterization with a point process. 
For the probability space (Ω, ℱ, P), we denote a counting measure N(·) on a Polish space (S,d). 
We define N(·)=∑_i=1^∞1_x_i(·) and N'(·)=∑_i=1^∞1_y_i(·) as two random counting measures. 
Additionally, we define scaling functional κ: ℤ→ℝ_≥0, and a lower semi-continuous cost function c:S× S→ℝ_≥0. 
Let σ(·) denote the permutation function defined on the support of N(·), and we then define the distance as:

    c(N(·), N'(·))=   ∞1_N(S)≠N'(S) or N(S)∨ N'(S)=∞
       + κ(N(S))inf_σ(·)∑_i=1^N(S)c(x_i, y_σ(i)).


This distance is also similar to the ones presented by <cit.>. Additionally, we define the 1-Wasserstein distance between two random counting measure by:

    W_c(μ,ν)=inf_γ∈Γ(μ,ν)𝔼_(N, N')∼γc(N, N'),

Here μ is the measure corresponding to N(·) while ν is the measure corresponding to N'(·). Analogously to standard DRO problems, we can write down a new formulation for this DRO problem with point process as follows:


Let N(·) denote a Poisson point process, f(·) denote a Borel function, and W_c(P, P_0) denote the Wasserstein distance between measures induced by N' and N under cost metric c. For a distance δ≥ 0, the following problems are equivalent:

    sup_ W_c(P, P_0) ≤δ𝔼_N' ∼ P[ℓ(N'(f))] = 
    inf_λ≥ 0   {λδ  + 𝔼_N∼ P_0 [sup_N' [ ℓ(N'(f) ) - λc(N, N') ] ] },

where N(f):=∫ f(x)N(dx).

    This follows from the application of <cit.>. 


As mentioned in the background material in (<ref>), MEV observations are given by the product of  radial (A^(n)) and spectral (Y^(n)) components. In the perspective of the point process, the associated measures for each of these components forms the intensity of the point process. Specifically, we let the atoms of the point process be given by (A^(n), Y^(n)) where A^(n) can be thought of as a time coordinate (since A^(n) is the n^th arrival time) and Y^(n) as a space (or the mark) component. Define N(da, dv) = ∑_n=1^∞1_((A^(n), Y^(n)) (da, dv). This provides a useful characterization in terms of the dual problem insofar as we can optimize over the boundaries of the integrals to form our adversary point process. As we will illustrate, this is equivalent to finding a new point process with modified intensity. 
 
With regards to the cost in the Wasserstein distance, we place an infinite cost on changes in the arrival times of the events. This has the interpretation that our uncertainty is not on the arrival time of the tail events but rather on the dependence. We formalize this through the following cost function between two points:

    c((s, u), (t,v)) = ∞1_s ≠ t + |u - v| 1_s = t

Our Wasserstein cost is then between the spectral component while fixing the radial component. We formalize this case in the following corollary.

    
    For the given setting in Theorem <ref>, if the point process satisfies N(x)=lim_n∑_i=1^n1_X^(i)/n(x), where X^(i) are i.i.d. standard Frechét margins. Then, we can rewrite the primal problem as:
    
    sup_W_c(P, P_0)≤δ𝔼_N∼ P[ℓ(N(f∘ T^-1)],

    where N(da, dv)=∑_n1_(A^(n), Y^(n))(da, dv) and T(X)=(1/X_1, X/X_1). 
    Under P_0, A^(n) is the n-th arrival of the Poisson point process with intensity 1 and Y^(n) follows the spectral measure H(·). 
    Here we introduce the composition of the Borel function f on the transformation T.

 



§ OPTIMIZATIONS FOR SPECIFIC LOSS FUNCTIONS


Having introduced the different perspectives of the optimization, we provide specific loss functions where each perspective is particularly conducive towards specific robustification cases.


 §.§ Robustifying the CDF


Consider the cumulative density function (CDF) of M∈ℝ^d satisfying M_i=max_n=1^∞Y_i^(n)/A^(n), where Y^(n)∈ℝ^d are i.i.d. random vectors and A^(n) is the n-th arrival time of a unit Poisson point process. 
The CDF of M satisfies:

    P(M_1 ≤ x_1^-1,    …,  M_d ≤ x_d^-1 ) = 
       exp ( - d∫_Δ^d-1max_i=1^d x_i w_i H(dw)).

Here, Equation <ref> is derived from Lemma <ref> by using the max stable decomposition of the random variable, as demonstrated in Equation <ref>. 
We will now exploit specific geometric properties of the point process to derive a dual form. 
Given the definition of M_i, we may make a transformation of the CDF algebraically as follows:

    P     (M_1 ≤ x_1^-1, …,  M_d ≤ x_d^-1 )= 
        P(max_k=1^d Y_k^(n)x_k≤ A^(n),∀ n ) :=P(V_x^(n)≤ A^(n),∀ n).

In this case, we can reduce our analysis to this two dimensional space and consider the robustification of the point process on this space, rather than the full d dimensional space of the Y^(n)'s. 
A schematic of this representation before robustification is given in Figure <ref>. Using this form leads to a dual form given by the following theorem:


Consider the setting in Theorem <ref> and the loss function ℓ(X) = 1_X≥ 1, the point process N(·):=∑_n=1^∞1_(A^(n),V^(n)_x)(·), the function f(a,v)=1_𝒞(a,v) and 𝒞={(a,v)∈ℝ^2 | a≥0,a≤ v}. Then, the dual objective is:

    inf_W_c(P, P_0)≤δP'(M_1 ≤ x_1^-1, …,  M_d ≤ x_d^-1 )
    
    =   1-inf_λ≥ 0λδ + 𝔼_P_0 [1 - λmin_n≥ 1( A^(n)- V_x^(n))^+  ].



See Appendix <ref>.


This formulation is advantageous in many ways. First, we reduced the original optimization problem over measures to one over a scalar variable λ. Second, this formulation includes the relevant constraints such that the perturbed distribution maintains the max-stability property of EVT. Third, the optimal solution is given by a shift in the rate function, as illustrated in the following corollary: 


The minimizer of the problem in Theorem <ref> has the following form:

    inf_P:W_c(P,P_0)≤δP(X_1 ≤ x_1^-1, …, X_d ≤ x_d^-1) 
       = exp ( - 𝔼_N∼ P_0 [N( 𝒞_1/λ^⋆)]  )

where 𝒞_1/λ^⋆ is the region 
{(a, v) : V_x^(n) > A^(n) - 1/λ^⋆}
and λ^⋆ is the minimizer of (<ref>).
 
 
     See Appendix <ref>.
 

The ideas behind Corollary <ref> are illustrated in Figure <ref> where the region over which the expectation is taken is shifted by 1/λ^⋆.

 

 §.§ Probability an Event in a Rare Set Occurs

In this part, we describe a generalization of the CDF case where we are interested in robustifying the probability that an event in the set A ⊂ℝ^d_+ occurs, i.e. P(N(A) ≥ 1). Our primal problem in this case is:
 
    sup_W_c(P, P_0) ≤δ P(N(A) ≥ 1) = sup_W_c(P, P_0) ≤δ𝔼_X ∼ P[1_X ∈ A]

 and we let the distance be any ℓ_p-norm.
 We can write the dual form and obtain the following simplification:
 
    Consider the setting in Theorem <ref> and the loss function ℓ(X)=1_X≥1, the point process N(·)=∑_n=1^∞1_X^(n)(·), and the function f(x)=1_A(x). Then the dual objective is:
    
    sup_W_c(P, P_0) ≤δ P(N(A) ≥ 1) 
    
        =   inf_λ≥ 0{λδ +  𝔼_P_0 ( 1- λinf_ninf_y ∈ A y - X^(n))^+ } .

    
 
  
 See Appendix <ref>.
 

 Relatedly, we can also compute the expected number of events to occur within a set using the following theorem:
 
     Consider the setting in Theorem <ref> and the loss function ℓ(X)=X, the point process N(·)=∑_n=1^∞1_X^(n)(·), and the function f(x)=1_A(x). The the dual objective is:
     
    sup_W_c(P, P_0) ≤δ𝔼_N ∼ P[N(A)]
    
             =   min_λ≥ 0 [ λδ + 𝔼_P_0∑_n=1^∞ ( 1 - λinf_y ∈ A y - X^(n))^+  ].

 
 

 §.§ Conditional Value at Risk

Conditional value at risk (CVaR) is a commonly used risk measure in finance. Often there is uncertainty around the calculation, so we derive an efficient form for the calculation. In the DRO case, we can write the CVaR problem with the given level α as:
 
    sup_P: W_c(P,P_0)≤δinf_z𝔼_P((∫⋁_k=1^d x_k N(dx)-z)^+/1-α+z).

 By exchanging the sup-inf to inf-sup, we can write the dual formulation as the following theorem:

 
      Consider the setting in Theorem <ref> and the loss function ℓ(X)=(X-z)^+/1-α+z, the point process N(·)=∑_n=1^∞1_X^(n)(·), and the function f(x)=x_∞. The the dual objective is:
    
    inf_zsup_P: W_c(P,P_0)≤δ𝔼_P((∫⋁_k = 1^d x_i N(dx)-z)^+/1-α+z)
    
             =   δ/1-α + 𝔼_P_0 [ ∑_n=1^∞X^(n)_∞ | ∑_n=1^∞X^(n)_∞ > q_α. ],

    where P_0(∑_n=1^∞X^(n)_∞≤ q_α)=α and q_1<∞.
 
 
 See Appendix <ref>.
 
Having introduced these simplifications, we see that the optimization problems for commonly used losses under the point process interpretation of EVT can be made feasible.
 



§ EXPERIMENTS AND RESULTS


We now consider empirical validation of the proposed optimization schemes using two experiments. The first uses a synthetic data set constructed to pathologically challenge our model and represent real world risk scenarios. 
The second validation is a baseline comparison using a real data set of financial returns similar to the DRO MEV experiment proposed by <cit.>. 
For the synthetic and real data experiments the observations are extreme; however, the standard EVD model is shown to not provide appropriate coverage for computing the worst-case risks. 
For both data sets, we provide empirical results validating that the proposed method satisfies the properties that we desired and outlined in the introduction, i.e. those of sufficient coverage while not being too conservative.




 §.§ Computational Implementations

For losses that have a reduced dual problem such as those in Section <ref>, we use the empirical data to represent the expectations and optimize over λ.
However, for general losses we use the equivalence stated in Corollary  <ref> to estimate the adversarial distribution.
Our procedure follows the method in <cit.> where we first estimate the spectral measure from the observations using a generative neural network and then optimize over the space of networks within some ball of the initial distribution.
The case for arbitrary ℓ is illustrated in Algorithm <ref>.







 §.§ Experiments with Synthetic EVT Data


We evaluate our proposed methodology on two synthetic two-dimensional EVD data sets.
 
We hypothesize that a properly formulated DRO approach, where max-stability is enforced via constraints, demonstrates a greater invariance to increased model misspecification relative to a non-constrained models. Our methodology is designed to account for the complexities of real data and produce more accurate and reliable predictions. 
Details concerning computational parameters, runtime, and code base reference is available in Appendix <ref>.



  §.§.§ Synthetic Datasets


For our synthetic data experiments, we consider mixture distributions where one mixture component with a more concentrated risk appears less frequently than the other mixture component. This is illustrated in Figures <ref> and  <ref>, where the ten thousand observations of the two datasets are visualized. In Figure <ref> the component has a smaller tail index and more realizations with smaller magnitude whereas in Figure <ref> the component has a larger tail index, but is more rare.  
For more information see Appendix <ref>.

A standard EVD approach would be sufficient when modeling a non-mixture Symmetric Logistic (SL) or Asymmetric Logistic (ASL) set of data. However, this mixture inserts a pathological modeling issue, which emulates the practical concerns often observed in real data where extrapolating to out-of-sample extreme values is difficult given the scarcity of data in these scenarios. 
These generated mixture distributions exhibit different tail behavior, which make it difficult for standard approaches to capture the dependencies accurately, and thus we turn to the proposed DRO estimator and evaluate our ability to recover the true risk.








  §.§.§ Synthetic Experiment Results



We now illustrate the results of the computational method for the loss corresponding to CVaR of the ℓ^1-norm of the observations.
Specifically, we aim to robustify the calculation of 1/α𝔼[X_1 1_X_1 ≤ x_α] where x_α = inf{ x : P(X_1 ≤ x) ≥α}.
As α→ 0, this corresponds to the CVaR for values deep in the tail.
We approximate this region as a MEV under both the SL and ASL distributions described previously. 
Our goal in this experiment is to understand the behavior of the error, defined as the difference between the true risk and the risk computed using the different methods, as a function of the budget δ.
We compare 3 different robustifications: (1) a totally unconstrained robustification where P can be in any class of distributions; (2) an EVT constrained case where P must also be an MEV and shares the same A^(n)'s; (3) an EVT constrained case where the margins are assumed to be well calibrated and only the dependence is modified (such that w ∼ P,   s.t.  𝔼[w_i] = 1/d).
In terms of standard EVT methods, (3) corresponds to the case where the adversarial distribution has a different dependence structure but maintains the same margins.
Specifically, this would suggest estimating the margins is easier and therefore should be conserved, but the dependence structure should be perturbed.
Figures <ref> and <ref> illustrates the error of using a DRO approach with max-stability constraints compared to the unconstrained approach, producing a distribution which is not of the MEV model class as functions of δ. 
This allows for the understanding of performance based on the behavior of the loss function as δ changes. 



  
Synthetic Experiment Results Analysis
In Figures <ref> and <ref>, we visualize that the models which are constrained to be max-stable, specifically P_⋆ and P_⋆ (𝔼[w_i] ≈1/d), demonstrate a lower and robust (positive) error when compared to the unconstrained model P_⋆ (unc). Additionally, we observe a slight improvement in the error of the constrained P_⋆ model for an increase of model uncertainty (δ) and subsequent monotonically increasing error as  uncertainty increases. This behavior confirms our conceptual illustration of the DRO max-stable constrained approach in Figure <ref>.




  §.§.§ Convergence of CDF as Distance is Increased

We now consider an experiment that corresponds to Theorem <ref> described in Section <ref>.
We illustrate the behavior in Figure <ref> where we see an improvement in the error for a wide range of values of δ. 





 §.§ Experiment on High Dimensional Financial Data


We consider a dataset on financial returns of equities in the S&P 500 index similar to the observations used in the validation experiments proposed by <cit.>. 
The dataset is composed of all current members of the S&P 500 who have been members since at least January 1983 to provide forty years of equity price data. 
We then average returns across companies, where each company is uniquely assigned to one of eleven industrial sectors (industrial, health care, consumer staples, etc.), leading to an eleven dimensional dataset. 
Maximum daily returns are taken both annually and weekly across the forty year data set, where each of these annual and weekly maximum data sets now constitute extreme observations.

This dataset is in the dimension of the number of industries considered, in this case d = 11. 
A visualization of this extreme data for highest weekly and annual returns is provided in Appendix <ref>. 
Obtaining a ground truth risk for real data is nearly impossible since we may never observe realizations deep in the tails, so validating against a ground truth risk is difficult.
To circumvent this, we consider training on less extreme data given by maxima over shorter time scales and use maxima over longer time scales as the ground truth risk.
Specifically, we fit the models using weekly maxima and test on yearly maxima.
The reason for this is that large events are more likely to occur over the longer time scale (the year) than over the day, leading to a more representative set of measurements deeper in the tails. 

We use the exact same methodology outlined in Section <ref> and illustrate the results of this experiment in  Figure <ref>.



  
Real Data Experiment Results Analysis
We observe that when δ = true risk, a minimal error is achieved between the true and calculated risk.
Additionally, this experiment demonstrates that for a real extreme data set the models constrained by EVT achieve a lower error level when compared to the unconstrained model, as anticipated.
We note that the comparable formulation in <cit.> does not provide a modifiable parameter such as our formulation does with δ, to adjust model uncertainty. 
This demonstrates the additional value of our proposed formulation over a comparable method. 



§ DISCUSSION

In this paper, we introduced a framework for robustifying MEV distributions according to the point process viewpoint. 
We provided a novel framework and discussed simplifications that are possible under certain cost functions. 
We numerically demonstrated that, as we proposed, adding additional constraints from EVT improves the robustification without being too conservative. 
The framework is evaluated on a synthetic data set which is constructed to challenge the model and a high-dimensional real world data set derived from a similar investigation, demonstrating expected and improved behavior. 
The proposed methodology has a variety of applications in many fields, particularly in risk sensitive settings where model misspecification may be present. 

There are many avenues for extending the proposed methodology.
One important extension involves the case where one must specify a policy to mitigate a worst-case risk.
This applies in, for example, portfolio optimization problems where the optimal portfolio minimizes the conditional value at risk over a distribution of portfolio losses.
Additionally, further exploration of the proposed formulation leveraging max-stable processes extended to a more general max-infinitely division process may be insightful. 



  
Limitations
There exist a number of limitations associated with the proposed framework. 
With regards to the computational method, there may be numerical instabilities associated with the optimization particularly in cases of δ being small. 
This is because the λ parameter must grow large, which leads to unstable optimization schemes.



 
Material in this paper is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-20-1-0397.










Distributionally Robust Optimization as a Scalable 

           Framework to Characterize Extreme Value Distributions 
 (Supplementary Material)
    Alex Zunger
    August 5, 2024
================================================================================================================================================





§ PROOFS






 §.§ Proof of Equivalence in Representations


Let us first recall the two problems that we are solving. 
On one hand, we take an explicit point process view point in:

    max_ W_c(P, P_0) ≤δ𝔼_N ∼ P[ℓ(N(f))],
    Point Process

where N(dx)=∑_nδ_X^(n)(dx) is a non-homogeneous Poisson process. By the properties of EVT, the intensity satisfies

    𝔼[N(f)] = ∫ f(T^-1(r,w)) r^-2dr H(dw).

By change of variables, we could decompose X^(n)=Y^(n)/A^(n), where A^(n) is the n-th arrival of Poisson process with intensity 1 and Y^(n)∼ H(·). Thus, we could rewrite the problem with:

    max_W_c(P, P_0)≤δ𝔼_N'∼ P[ℓ(N'(f∘ T^-1)],

where N'(dt, dy)=∑_n1_(A^(n), Y^(n))(dt, dy) and T(X)=(1/X_1, X/X_1).



 §.§ CDF proof




We now detail the proofs for specific loss functions, beginning with the cumulative distribution function (CDF), followed by the conditional value at risk (CVaR), probability, and expected number of events in rare sets. 


 
 Define the upper triangular region as 𝒰 = { (a, v) :  a ≥ 0, v ≥ a }.
 To compute the CDF, we want to condition on the Poisson process having no points within 𝒰, i.e. N(𝒰) = 0. 
 We can compute the rate of the point process according to 𝔼[N(𝒰)]. 
 Since this is a Poisson point process, the probability that no points occur within 𝒰 is given by
 exp(- 𝔼[V_x]).
 Having written the CDF in terms of the rate of the Poisson point process, we now follow through to describe the simplified form written in (<ref>).
 First, we write the dual problem when directly applying <cit.> as
 
    min_λ≥ 0{λδ + 𝔼_N ∼ P_0 [max_N'1_N'(𝒰) ≥ 1 - λc(N, N')  ] }.

Conditioning directly on the first inequality, we can rewrite this as
 
    min_λ≥ 0{λδ + 𝔼_N ∼ P_0 [1- λmin_N'(𝒰) ≥ 1c(N, N')  ]^+ }.

 Here we also set c((s,u), (v,t))=|u-v|1_s=t+∞1_s≠t. Moreover, we notice the equivalent representation between X^(n) and (A^(n), V_n(x)), which means we can write N(dx) = ∑_n1_X^(n)(dx)=∑_n1_(A^(n), V_n(x))(dt, dv). Thus, if N(𝒰)≥ 1, it means ∃ n s.t. V_n(x)-A_n≥0, we can simply set N'(·)=N(·) to obtain the minimum of min_N'(𝒰) ≥ 1 c(N, N'). On the other hand, if N(𝒰)=0, it means ∀ n, V_n(x)-A_n<0. Then we denote δ=min_n(A^(n)- V_n(x)) and n^*∈min_n(A^(n)- V_n(x)) and set N'(dt,dv)=1_(A^(n^*), V_n^*(x)+δ)(dt,dv)+∑_n≠n^*1_(A^(n), V_n(x))(dt,dv), which guarantees N'(𝒰)≥ 1, and the minimum of min_N'(𝒰) ≥ 1 c(N, N') is exactly δ.  To conclude, we have the following equivalence holding:
 
    min_N'(𝒰) ≥ 1 c(N, N') = min_n≥ 1 [ A^(n) - V^(n)_x]^+.

Substituting, we obtain the desired result of (<ref>):
 
    min_λ≥ 0λδ + 𝔼_N∼ P_0 [1 - λmin_n≥ 1( A^(n)- V_x^(n))^+  ]^+






  §.§.§ Minimizer of CDF Proof


 
 Following where we ended in the proof of Theorem <ref>, 
Denote f(A,V_x):=min_n≥ 1( A^(n)- V_x^(n))^+. By first order condition of the Equation (<ref>), we have:
 
    δ=𝔼_N∼ P_01(f(A,V_x)≤1/λ^*)f(A,V_x).

 Substituting it to objective, we have:
 
    min_λ≥ 0λδ + 𝔼_N∼ P_0 [1 - λmin_n≥ 1( A^(n)- V_x^(n))^+  ]^+
       =P_0(f(A,V_x)≤1/λ^*).

 Thus, we have:
 
    min_P:W_c(P,P_0)≤δP(X_1 ≤ x_1^-1, …, X_d ≤ x_d^-1)
       =1-P_0(f(A,V_x)≤1/λ^*)
       =P_0(f(A,V_x)>1/λ^*)
       =P_0(N(𝒞_1/λ^*)=0)
       =exp(-𝔼_P_0N(𝒞_1/λ^*)  )

 



 §.§ Proof of Rare Region Probability

 

 
     We write the primal problem as 
     
    sup_D(P, P_0) ≤δ P(N(A) ≥ 1)

     and writing it in the dual form we get
     
    inf_λ≥ 0{λδ + 𝔼_P_0 [sup_Z^(n)( sup_n=1^∞1_Z^(n)∈ A - λ∑_n=1^∞ Z^(n) - X^(n) )  ]}.

     Replacing the inner maximization, we can write
     
    sup_Z^(n)( sup_n1_Z^(n)∈ A - λ∑_n=1^∞ Z^(n) - X^(n) ) = ( 1- λinf_ninf_z ∈ Az - X^(n))^+,

     which is due to:
     
    sup_Z^(n)( sup_n1_Z^(n)∈ A - λ∑_n=1^∞ Z^(n) - X^(n) ) = ( 1- λinf_∃ n, Z^(n)∈ A∑_n=1^∞Z^(n) - X^(n))^+.

     And solving inf_∃ n, Z^(n)∈ A∑_n=1^∞Z^(n) - X^(n), we have it equals inf_ninf_z ∈ Az - X^(n) as the region only requires one sample z^(n)∈ A and the others can be equal to corresponding X^(n).
 

 

 §.§ Proof of Number of Rare Region Occurrences


    We begin with the dual definition
    
    inf_λ≥ 0{λδ + 𝔼_N ∼ P_0 [sup_ Z^(1)⋯ Z^(N) ( ∑_n=1^∞1_Z^(n)∈ A  - λ∑_n=1^∞ Z^(n) - X^(n))  ]}.

    Rewriting the part within the sup, we obtain the desired result, since
    
    sup_Z^(1)⋯ Z^(N) ( ∑_n=1^∞1_Z^(n)∈ A  - λ∑_n=1^∞ Z^(n) - X^(n)) = ∑_n=1^∞( 1- λinf_z ∈ Az - X^(n))^ +.

    



 §.§ CVaR proof

 

 

At first, we consider the finite point case, where N(·)=∑_n=1^N δ_X^(n)(·). The CVaR problem is given as
 
    min_z max_D(P, P_0) ≤δ𝔼_P (  ( ∫⋁_i=1^d x_i N(dt, dx) - z )^+ /1- α+ z )_primal problem.

 Taking the primal problem component, we can write the dual problem as:
      
    min_λ≥ 0{λδ + 𝔼_P_0 [ max_Z^(n)( ∑_n=1^∞ Z^(n)_∞ - z )^+ - λ∑_n=1^∞ Z^(n) - X^(n)_∞ ] }
       = min_λ≥ 1{λδ + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞ -z)^+  }   (Lemma <ref>)
       = δ + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞ -z)^+ .

     Adding the minimization over z, we obtain
     
    min_z [ δ + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞ -z)^+/1-α + z   ] .

Taking the minimum, we get

    δ/1-α + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞|∑_n=1^N X^(n)_∞≥ q_α )

where P_0 ( ∑_n=1^N  X^(n)_∞≤ q_α) = α. Finally, as q_1<∞, the summation converges a.s., we can safely let N→∞ and obtain the final result.
 

    


    
        For x∈ℝ^d_+ and λ≥0, we have:
        
    max_y_1⋯ y_N∈ℝ^d_+(∑_n=1^N y_n_∞-z)^+-λ∑_n=1^Ny_n-x_n_∞=
                ∞,    when λ∈[0,1),
    (∑_n=1^N x_n_∞-z)^+,   when λ∈[1,∞).

    
    
        For λ∈[0,1), we claim the objective value would diverge to infinity. In fact, we let y_n=x_n for n≥2. The objective becomes:
        
    max_y_1∈ℝ^d_+(y_1_∞+∑_n=2^Nx_n_∞-z)^+-λy_1_∞.

        Then, we can pick y_1 such that y_1_∞≥ z-∑_n=2^Nx_n_∞ and letting it goes infinity, we find the objective will also go infinity.

        For λ∈[1,∞), we have:
        
    (∑_n=1^N y_n_∞-z)^+-λ∑_n=1^Ny_n-x_n_∞   ≤(∑_n=1^N y_n-x_n_∞+x_n_∞-z)^+-λ∑_n=1^Ny_n-x_n_∞
       =(1-λ)∑_n=1^Ny_n-x_n_∞+(∑_n=1^N x_n_∞-z)^+.

        Thus, maximizing over y, we have the upper bound for the objective is (∑_n=1^N x_n_∞-z)^+. And this value is attained when y_n=x_n for n=1,⋯,N.
    
 

 



§ DATA DESCRIPTION


Here we provide more details on the data distributions used.
The symmetric logistic (SL) and asymmetric logistic (ASL) distributions are commonly used parametric models that have the property of max-stability.
We describe some of their properties and how they are applied in the mixture model.
We use the sampling algorithms described in <cit.> to generate all the datasets.

As mentioned, for computing the cumulative distribution, the central object of interest is the spectral measure. 
In each of these datasets, the spectral measure is given by a mixture of two spectral measures.
Our baseline model P_0 is fit to the mixture without consideration that the model contains two components and results in model misspecification.



 §.§ Symmetric Logistic Mixture

For these experiments, we consider a mixture model with two components: one that is almost completely dependent and another that is almost completely independent. 
The component with dependence is sampled with lower probability than the one with independence. 
Therefore, a naive estimator would largely concentrate on the component with independence than the one with dependence. 
The dependence function for this distribution is given by

    A_SL(w) =  (∑_i=1^d w_i^1/α )^α,   α∈ (0, 1].

Notice that the influence α is equal for all components of the vector.
We specifically sample the distribution with large dependence (α=0.1) with probability 0.1 whereas we sample the distribution with small dependence (α = 0.9) with probability 0.9.



 §.§ Asymmetric Logistic Mixture

While the SL distribution is symmetric and all components have the same level of dependence, the ASL can impose specific dependencies between any subset of variables.
Denoting 𝒫_d as the power set of {1, …, d }, the dependence function for this family of distributions is given by

    A_ASL(w) = ∑_b ∈𝒫_d (∑_i ∈ b ( λ_i, b w_i)^1/α_b )^α_b,    w ∈Δ_d-1, α∈ (0, 1), λ_b ∈Δ_|b|-1.


We then consider a mixture of two different components, one again with a heavier tail and another with a lighter tail.
This is given by setting λ to be the same random point on the simplex for both mixture components but then considering α = 0.1 and α=0.9 for the components with high dependence and low dependence, respectively. 



 §.§ Financial Data and Visualization


Below we've provided a visualization of the maximum daily returns taken over weekly blocks, and maximum daily returns taken over annual blocks. The returns are provided for the eleven industries, where the values are averages taken across selected companies uniquely belonging to each industry. To compute the average of each industry, we first selected the 93 members of the S&P 500 that have been listed on the index since January 1983, in order to provide forty years of financial data. Each of these company's daily returns were averaged across their assigned industry to create the average daily returns for each industry over forty years.





§ ALGORITHM DETAILS

Here we provide additional details that were left out of the main manuscript. 
Specifically, we go over the Maximize and Minimize steps in greater detail.
The basic idea is we have an iterative training algorithm similar to the min-max problems in generative adversarial networks.
λ is initialized according to 1/δ + ε for some small ε. 
P is initialized according to the same parameters as P_0.
For each of the K iterations, we

Maximization step:
We fix λ to be the value from the last iteration.

    
  * Sample N points from the base distribution X^(n)∼ P_0,    n=1,…,N to obtain the X points from the base distribution.
    
  * Sample N points from the current estimate of the adversarial distribution P to get X̃^(n)∼ P,    n = 1 … N.
    
  * Compute ℒ for each X^(n), X̃^(n) which results in a N × N matrix of values, i.e. 
    
    𝐋 :=
        [ ℒ_λ(X̃^(1), X^(1))                  ⋯ ℒ_λ(X̃^(1), X^(n));                  ⋮                  ⋱                  ⋮; ℒ_λ(X̃^(n), X^(1))                  ⋯ ℒ_λ(X̃^(n), X^(n));                    ]

    
  * Take the max over the X̃^(n) dimension, i.e. max_i𝐋_i, ·
    
  * Average over the remaining Y^(n) dimension, i.e. R = 1/N∑_j=1^N max_i=1^N 𝐋_i, j.
    
  * Compute the gradient of R with respect to the parameters of P, update the parameters of P via gradient descent.

Adding different constraints to the problem results in different forms of the Ỹ, which we describe in the next section.


Minimization step:
Now we fix the parameters of P and only minimize over λ.

    
  * Compute R̂ = λδ + 1/N∑_j=1^N max_i=1^N 𝐋.
    
  * Compute gradient of R̂ with respect to λ, update according to gradient descent.

We next describe a few algorithmic considerations of the maximization step.



 §.§ Unconstrained Case

When sampling from an MEV, one samples according to A^(n), Y^(n),    n=1,…, N, N ≫ 0.
Then samples are generated according to max_n=1^N Y^(n)/A^(n) (e.g. see <cit.> for additional considerations).
In the unconstrained problem, we let the X^(i) in (<ref>) be the MEV, i.e. X^(i) = max_n=1^N Y^(i,n)/A^(i,n). 

Then, P does not use any of the factorization of the Y^(n)/A^(n) and is free to be whatever distribution maximizes the quantity.
The problem is unconstrained because it does not consider any of the constraints given by EVT, specifically that the factorization in terms of the radial A and spectral component Y must hold.


 §.§ Constrained Case

On the other hand, we consider the explicit factorization in the constrained case.
Here we share the A^(n) across both distributions of P, P_0. 
Then, X remains as described in the unconstrained case but X̃ is given by X̃^(i) = max_n=1^N Ỹ^(i,n)/A^(i,n) where the Ỹ^(i,n) are sampled from the estimated P distribution.
Note that the A^(n) are the same for the calculation of X and X̃.
The constraints then inform the adversarial distribution that it must be max-stable and that the radial component is consistent between both.



 §.§ Computational Details and Code Reference


The code base for the DRO MEV model, written in python, is available at the following github link:  <https://github.com/patrick-kuiper/mev_dro>. Below, in Table <ref>, we provide the run time for each of the synthetic data experiments: Unconstrained, Constrained Extreme Value Distributed (EVD), and Constrained Extreme Value Distributed: with Unit Margin (EVD-SM). We chose to highlight the synthetic data sets as they represent the most challenging experiments computationally. Each of these experiments were run individually on NVIDIA GeForce RTX 3090 GPUs with CUDA Version 12.2. Each experiment was run for 2001 epochs.  






 Paper Title: Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances.


MIST: A Simple and Scalable End-To-End 3D Medical Imaging Segmentation Framework
    
Adrian Celaya1,2 Evan Lim1 Rachel Glenn1 Brayden Mi1 Alex Balsells1,2 Tucker Netherton1 Caroline Chung1 Beatrice Riviere2 David Fuentes1

    August 5, 2024
==============================================================================================================================================

§ ABSTRACT 
This paper introduces a novel approach to emotion detection in speech using Large Language Models (LLMs). We address the limitation of LLMs in processing audio inputs by translating speech characteristics into natural language descriptions. Our method integrates these descriptions into text prompts, enabling LLMs to perform multimodal emotion analysis without architectural modifications or addition. We evaluate our approach on two datasets: IEMOCAP and MELD, demonstrating significant improvements in emotion recognition accuracy, particularly for high-quality audio data. Our LLM-finetuning experiments show that incorporating speech descriptions yields a more than 2 percentage point increase in average weighted F1 score on IEMOCAP (from 70.111% to 72.596%). We also compare various LLM architectures and explore the effectiveness of different feature representations. Our findings highlight the potential of this approach in enhancing emotion detection capabilities of LLMs and underscore the importance of audio quality in speech-based emotion recognition tasks. We'll release the source code on Github. 






§ INTRODUCTION

Emotion detection in speech is a crucial component of human-computer interaction, with applications ranging from customer service to mental health monitoring. While Large Language Models (LLMs) have shown remarkable capabilities in processing and understanding text, they lack the ability to directly process audio inputs. This limitation has hindered their potential in multimodal emotion recognition tasks that involve both textual and audio data.

Our research addresses this challenge by introducing a novel approach that enables LLMs to leverage speech characteristics for emotion detection without requiring architectural modifications. We propose a method that translates audio features into natural language descriptions, which can be seamlessly integrated into text prompts for LLMs. This approach bridges the gap between audio and text modalities, allowing LLMs to perform multimodal emotion analysis.

In this paper, we evaluate our method using two widely recognized datasets in the field of emotion recognition: IEMOCAP <cit.> and MELD <cit.>. We explore the effectiveness of various feature representations and compare the performance of different LLM architectures. Our study aims to demonstrate the potential of this approach to enhance the emotion detection capabilities of LLMs and to provide insights into the factors that influence its success.

By enabling LLMs to consider both textual and audio-derived information, we open up new possibilities for more nuanced and accurate emotion recognition systems. This research not only contributes to the field of affective computing but also provides a framework for incorporating non-textual information into language-model-based systems, with potential implications for a wide range of multimodal AI applications.



§ RELATED WORK



 §.§ LLM for emotion recognition

Recent advancements in Emotion Recognition in Conversation (ERC) have leveraged Large Language Models (LLMs) to enhance performance and generalizability. Our work builds upon these developments, particularly the InstructERC framework <cit.>, which reformulated ERC as a generative task using LLMs. InstructERC introduced a retrieval template module and additional emotion alignment tasks, achieving state-of-the-art results on common ERC datasets. We extend this approach by incorporating speech descriptions into the template, aiming to capture crucial paralinguistic information.

Similar extensions to InstructERC have been explored in recent literature. BiosERC <cit.> enhances the model by extracting and injecting "biographical information" of speakers into the system, demonstrating improved performance across multiple datasets. CKERC <cit.> takes a different approach, incorporating commonsense knowledge about interlocutors' reactions and intentions to deepen the understanding of speaker information. DialogueLLM <cit.> incorporates  visual descriptions generated from GPT-4 into the prompt. All of these methods, like our approach, aim to enrich the input to LLMs with additional contextual information.

Our work differentiates itself by focusing specifically on the integration of speech characteristics, translated into natural language descriptions. This approach bridges the gap between audio and text modalities in LLM-based ERC systems, potentially capturing emotion cues that are present in speech but not in text alone. By doing so, we aim to combine the strengths of LLMs in processing textual data with the rich emotional information contained in speech, offering a novel perspective in the evolving landscape of ERC research.



 §.§ Speech Incorporation in LLM

The integration of speech features into LLM-based systems for emotion recognition and related tasks has been explored in various recent works, each offering unique approaches and insights. SECap <cit.> introduces a framework that uses LLaMA as a text decoder to generate coherent speech emotion captions instead of labels. While SECap demonstrates the potential of LLMs in describing speech emotions, it differs from our approach by requiring an additional audio encoder (HuBERT) and a bridging network (Q-Former) to process speech inputs. Similarly, <cit.> proposes an approach to integrate speech into LLMs for depression detection by adding acoustic landmark tokens to the input prompt. This method shares our goal of incorporating speech information directly into LLM inputs but necessitates additional parameters and pre-training steps to introduce audio information.

LanSER <cit.> takes a different approach by leveraging LLMs to infer weak emotion labels from speech transcripts, enabling the use of unlabeled data in SER tasks. Building upon this concept, <cit.> explores using LLMs for emotional speech annotation by incorporating both conversation context and textual acoustic feature descriptors. While this approach shares our interest in combining textual and acoustic information in the prompt, it primarily uses LLMs for weak label generation rather than direct emotion recognition training.

Recent research has also explored the application of LLMs in Automatic Speech Recognition (ASR), demonstrating the versatility of LLMs in speech-processing tasks. Seed-ASR <cit.> introduces an audio-conditioned LLM framework that incorporates continuous speech representations by developing a unique audio encoder. <cit.> proposes a speech prefix-tuning approach with RNNT loss to optimize speech prefixes for LLMs in ASR tasks. SLAM-ASR <cit.> takes a simpler approach, combining off-the-shelf speech encoders and LLMs with only a trainable linear projector, achieving state-of-the-art performance on the Librispeech benchmark.

Further extending the capabilities of LLMs in speech processing, AudioChatLlama <cit.> presents an end-to-end model that integrates general-purpose speech abilities into the Llama-2 model. This approach enables the model to handle audio prompts, perform spoken question answering, speech translation, and audio summarization, among other tasks. Similarly, Audio-LLM <cit.> introduces a method for encoding audio data into embedded representations that LLMs can comprehend, enabling tasks such as automatic speech recognition, emotion recognition, and music captioning. These advancements highlight the growing trend of integrating comprehensive speech-processing capabilities into LLMs.

Our work distinguishes itself from these approaches in several key aspects: (a) Direct Integration: We achieve seamless integration without requiring additional neural network components (e.g., audio encoders or projection layers) to process speech inputs, thereby reducing computational complexity. (b) End-to-End Training: We train the LLM directly on combined textual and speech description inputs, avoiding the need for weak label generation and additional training stages. (c) Flexibility and Generalizability: Our method of translating speech features into natural language descriptions allows for easy integration as a plug-in for other approaches and tasks, offering a versatile solution for incorporating speech information into LLM-based systems.





§ METHODOLOGY

Our novel approach incorporates speech characteristic descriptions in natural language into the prompt for LLMs. Previous LLM-based approaches have primarily focused on textual information, neglecting audio signals due to LLMs' inability to process direct audio inputs. This omission results in the loss of crucial emotional cues, as the same sentence can convey different emotions through variations in audio features such as pitch, volume, and intonation. Our method bridges this gap by translating audio signals into natural language descriptions of speech characteristics that are instrumental in emotion detection.

Our approach offers several advantages in the field of emotion detection. First, it enables multimodal analysis by seamlessly integrating speech characteristics into text prompts, allowing LLMs to process audio-derived information without requiring architectural modifications. This method is highly compatible with existing LLM infrastructures, facilitating easy implementation and integration into current systems. Moreover, by preserving paralinguistic information such as volume, pitch, and speaking
rate in our natural language descriptions, we capture crucial emotional cues that are 

often lost in text-only analysis. This enhanced contextual understanding empowers the LLM to grasp the full spectrum of the utterance, potentially leading to more accurate and nuanced emotion classification.



 §.§ Large Language Model Prompt Template




Our methodology utilizes a comprehensive LLM Prompt Template designed to enhance emotion detection capabilities. This template consists of four key components:


  
  * Instruction: This component sets the stage by positioning the LLM as an expert in sentiment and emotion analysis, priming it for the task ahead.
  
  * Context: A crucial element that provides the conversational background, allowing the LLM to understand the broader situation and interpersonal dynamics at play. We also add speech features behind the last three utterances to provide additional speech context.
  
  * Speech Descriptions: This innovative addition translates audio signals into natural language descriptions of speech characteristics. By including details such as volume, pitch, and speaking rate, we bridge the gap between audio and text modalities, enabling the LLM to consider paralinguistic features crucial for accurate emotion detection. 
  
  * Question: This component presents the specific task to the LLM, asking it to select an emotional label for a target utterance from a predefined set of options. Importantly, it explicitly instructs the model to consider both the contextual information and the audio features described.


This structured approach ensures that the LLM has access to both textual and audio-derived information, enabling a more nuanced and accurate emotion detection process. By incorporating speech characteristics into the prompt, we allow the LLM to perform multimodal analysis without requiring direct audio input processing capabilities.




 §.§ Audio Features

In our approach to emotion detection, we leverage five intuitive audio features that are easily interpretable by both humans and LLMs. These features capture essential aspects of speech that convey emotional information. volume is represented by two sub-features: average volume, which indicates the overall loudness of speech, and volume variation, which captures dynamic changes in intensity. Pitch, another crucial indicator of emotion, is similarly divided into average pitch and pitch variation, allowing us to detect both the general tone and the modulation of a speaker's voice. The fifth feature, speaking rate, represents the speed of speech, offering insights into the speaker's emotional state and urgency. By focusing on these fundamental yet comprehensive audio characteristics, we ensure that the extracted features are not only rich in emotional content but also readily understandable by LLMs when described in natural language. 



  §.§.§ Audio Feature Processing

In our approach to audio feature processing, we transform raw numerical features into categorical representations to enhance interpretability and facilitate natural language descriptions. This process involves four key steps:

a) Threshold Calculation: We compute thresholds for each audio feature (such as average volume, pitch variation, and speaking rate) using quantile-based segmentation. Depending on the desired granularity, we divide the feature space into 3, 4, 5, or 6 classes, with thresholds calculated using appropriate quantiles (e.g., 25th and 75th percentiles for 3 classes, 10th, 25th, 75th, and 90th percentiles for 5 classes).

b) Speaker-Specific Normalization: Recognizing the individual differences in speech patterns, we calculate speaker-specific thresholds, falling back to overall thresholds for less frequent speakers. This approach allows for a more accurate categorization of speech features relative to each speaker's typical patterns.

c) Categorization: Each numerical feature is then categorized based on these thresholds. For instance, in a 5-class system, a feature value might be classified as 'very low', 'low', 'medium', 'high', or 'very high'. This categorization provides a more intuitive representation of the audio features, bridging the gap between numerical values and human-interpretable descriptions.

d) Feature-Specific Descriptions: For each key audio feature (volume, pitch, speaking rate), we create descriptive phrases based on their categorical values. For example, "high volume with moderate variation" or "low pitch with high variation".




  §.§.§ Audio Impression



To make the audio features more accessible and meaningful for both human analysis and LLM processing, we generate natural language impressions based on the categorized features. Going beyond mere description, we generate interpretative impressions that suggest potential emotional or psychological states associated with the observed speech patterns. For instance, "speaking loudly with significant volume changes" might be interpreted as indicating "excitement, confidence, or urgency".

To account for the inherent uncertainty in interpreting speech patterns, we incorporate confidence-adjusting phrases (e.g., "likely", "may") based on the feature's proximity to category thresholds. This nuanced approach helps prevent overconfident interpretations of borderline cases.

The final impression synthesizes information about pitch, volume, and speaking rate into a coherent, flowing sentence. This integration provides a nuanced description of the speaker's vocal characteristics and their potential implications, offering valuable context for emotion detection tasks.



§ EXPERIMENTS




 §.§ Dataset

For our experiments, we utilized two widely recognized datasets in the field of emotion recognition: IEMOCAP and MELD.

IEMOCAP (Interactive Emotional Dyadic Motion Capture): IEMOCAP consists of dyadic conversations between actors, providing a rich set of emotional expressions in a controlled setting. It contains approximately 12 hours of audiovisual data from 10 actors, split into 5 sessions. Each utterance is annotated with one of nine emotion labels: anger, happiness, excitement, sadness, frustration, fear, surprise, neutral, and other. IEMOCAP is particularly valued for its high-quality audio recordings and motion capture data, making it ideal for multimodal emotion analysis. The dataset includes both scripted and improvised scenarios, offering a balance between controlled emotional expressions and more naturalistic interactions.

MELD (Multimodal EmotionLines Dataset): MELD is derived from the popular TV series "Friends," making it particularly suitable for analyzing emotions in conversational contexts. It contains 13,708 utterances from 1,433 dialogues, each annotated with one of seven emotion categories: anger, disgust, fear, joy, neutral, sadness, and surprise. MELD is multimodal, providing audio, visual, and textual data for each utterance. The dataset is pre-split into training (9,989 utterances), validation (1,109 utterances), and test (2,610 utterances) sets. Its multi-party conversational nature and situational diversity make it an excellent resource for evaluating emotion detection in complex, realistic scenarios.

For our experiments, we focused on the audio modality from both datasets, extracting and processing acoustic features as outlined in our methodology. In the case of the MELD dataset, we implemented speaker-based standardization of audio features to account for individual vocal characteristics. For the IEMOCAP dataset, we applied gender-based standardization, recognizing the physiological differences that can impact vocal features. Additionally, for IEMOCAP, we confined our analysis to six emotion categories (anger, happiness, excitement, sadness, frustration, and neutral), aligning with common practices in the field.

Our utilization of both MELD and IEMOCAP datasets was strategically designed to evaluate our approach across diverse audio recording environments. MELD, with its origin in television shows, presents more chaotic and naturalistic scenarios, while IEMOCAP offers a controlled, studio-based environment. This dual-dataset approach enables a comprehensive assessment of our emotion detection method across varied recording conditions.



 §.§ Primary Results




Our results from Table <ref> demonstrate that incorporating speech features into the text input yields performance improvements across both datasets, albeit to varying degrees. For the IEMOCAP dataset, we observe a substantial improvement of nearly 2 percentage points when adding speech descriptions to the text input (from 70.111% to 72.021%). By providing additional speech features for context, the model improves to 72.596%. (Detailed experiments are in Appendix <ref>)  This significant gain underscores the value of integrating acoustic information in emotion recognition tasks, particularly for high-quality audio recordings.

In contrast, the MELD dataset does not show improvemnet using speech descriptions and a modest improvement with the addition of speech impression, increasing from 67.44% to 67.604%. This disparity in performance between the two datasets can be attributed to the differences in audio quality and recording conditions. IEMOCAP, recorded in a controlled studio environment, provides clean and consistent audio, allowing for more accurate extraction of speech features. On the other hand, MELD, derived from the TV show "Friends," presents more challenging audio conditions. The presence of background laughter, varying audio quality, overlapping speakers, and short utterance durations (averaging 2-3 seconds) in MELD likely contribute to less reliable speech feature extraction, thereby limiting the potential benefits of speech descriptions.

Interestingly, the use of speech impressions does not yield further improvements over speech descriptions for either dataset. For IEMOCAP, the performance (71.542%) falls between that of text-only and text with speech descriptions, while for MELD, it performs similarly to the text-only baseline (67.023%). This suggests that our hard-coded impressions, despite being more interpretative, may introduce noise or inaccuracies that the model struggles to leverage effectively. The lack of improvement with speech impressions indicates that allowing the model to learn directly from more objective speech descriptions may be a more robust approach.

These findings highlight the potential of integrating speech characteristics into text-based emotion recognition models, particularly when working with high-quality audio data. However, they also underscore the challenges posed by real-world audio conditions and the need for careful consideration when designing multimodal emotion recognition systems.



 §.§ Model Comparison



To evaluate the effectiveness of our approach across different language models, we conducted experiments using various popular open-sourced LLMs, including LLaMA-2, LLaMA-3, and Phi-3, and a powerful closed-sourced model, Claude Sonnet 3.5. Table <ref> presents the emotion recognition accuracy on the IEMOCAP dataset for these models under different input conditions: text-only, text with speech descriptions, and text with speech impressions.

Our results demonstrate a consistent improvement in performance when incorporating either speech descriptions or impressions across all models. This underscores the value of integrating audio-derived features in emotion recognition tasks, regardless of the underlying language model. For most models, speech descriptions yield slightly better results than speech impressions, although the difference is often marginal. This suggests that more objective, feature-based descriptions may be more reliable for emotion detection tasks.

Interestingly, increasing model size does not necessarily lead to improved performance in this task. Despite their stronger performance on public benchmarks, larger models like LLaMA-3 and Phi-3 do not consistently outperform the smaller LLaMA-2 model when fine-tuned for emotion detection. In fact, Phi-3, despite being the largest model tested (13B parameters), performs worse than both LLaMA-2 and LLaMA-3 across all input conditions. This indicates that model size alone is not a determining factor for success in this specific task. While LLaMA-3-8b-instruct achieves the highest accuracy (72.098%) among all tested models when provided with speech descriptions, the improvement over LLaMA-2-7b-base (72.021%) is minimal. Additionally, we observe that using the instruct or base version of these models does not result in significant performance differences for this task. These findings suggest that while there may be some benefit to the architectural improvements in newer models, the gains are not as significant as one might expect given the increase in model size and the advancements claimed in public benchmarks. Certain features of the LLaMA architecture may be particularly well-suited for emotion recognition tasks, but the task-specific nature of emotion detection may limit the impact of general language model improvements.

The substantial performance gap between the zero-shot Claude model and the fine-tuned models highlights the importance of task-specific fine-tuning for emotion recognition. However, it is noteworthy that even in a zero-shot setting, Claude benefits from the addition of speech features, further emphasizing the value of multimodal inputs in emotion detection.

These findings emphasize the complexity of model selection for specific tasks like emotion recognition. While larger, more recent models often excel in general language understanding tasks, our results show that task-specific fine-tuning and the integration of multimodal features can sometimes yield superior results with smaller, specialized models. This underscores the importance of empirical testing and careful model selection in developing effective emotion recognition systems.






 §.§ Evaluation of Speech Features Using Large Language Model (LLM)






We compared the performance between numerical and categorical acoustic features using simple ML models in Appendix <ref>. Converting numerical features to categorical ones didn't significantly reduce performance and the speech features of IEMOCAP performed significantly better than random guessing. We found that audio quality is crucial for effective speech-based emotion recognition. High-quality audio (IEMOCAP) allows for extraction of informative speech features. Low-quality audio (MELD) makes speech features unreliable for emotion detection.

The results in Table <ref> demonstrate the potential of LLMs in leveraging speech features for emotion classification. As a reference point, recall that the 5-class categorical representation with ML models achieved a weighted F1 score of 31.333%, significantly outperforming the random guess baseline of 16.67%. When using LLMs with natural language descriptions of speech characteristics, we observe a slight decrease in performance compared to the ML approach. The 5-class categorical representation with LLMs yields a weighted F1 score of 27.895%, while the 3-class representation achieves 27.602%.

This performance gap between ML models and LLMs can be attributed to the inherent differences in how these models process information. ML models, designed specifically for discriminative tasks, can directly leverage the structured, one-hot encoded categorical features. In contrast, LLMs receive the information as natural language descriptions, which introduces a layer of abstraction and potential for ambiguity. Despite this, the LLM's performance remains substantially above the random guess baseline, indicating that it successfully extracts meaningful emotional cues from the speech descriptions.

The relatively small difference between 3-class and 5-class representations in LLM performance (27.602% vs. 27.895%) suggests that increasing the granularity of speech feature categorization does not significantly impact the LLM's ability to discern emotions. This could imply that LLMs are adept at interpreting broader emotional cues rather than fine-grained acoustic details.

Notably, we also examined the performance of LLMs using only speech impressions, which are derived from the 5-class categorical speech descriptions but provide a more interpretative representation of the audio features. This approach yields a weighted F1 score of 27.794%, slightly lower than 5-class categorical descriptions. This finding aligns with our earlier results when combining speech features with textual information, where speech impressions also showed a marginal decrease in performance compared to direct speech descriptions. The consistency of this pattern across different experimental setups suggests that, while speech impressions offer a more human-interpretable format, they may introduce a level of abstraction or subjectivity that slightly reduces the model's ability to accurately classify emotions. This underscores the delicate balance between feature interpretability and classification performance in emotion recognition tasks.

While there is some loss of information when transitioning from raw categorical features to natural language descriptions, the LLM's ability to maintain performance well above the baseline demonstrates its potential in multimodal emotion recognition tasks. This approach opens up possibilities for more flexible and interpretable emotion recognition systems that can integrate textual and acoustic information in a unified framework.



§ CONCLUSION


Our study demonstrates that integrating speech characteristics into LLM-based emotion detection systems significantly improves performance, especially for high-quality audio data. By translating audio features into natural language descriptions, we enable LLMs to perform multimodal emotion analysis without architectural changes.

Key findings show that incorporating speech descriptions enhances emotion recognition accuracy across different LLM architectures. However, the effectiveness of this approach heavily depends on audio quality, as evidenced by the contrasting results between IEMOCAP and MELD datasets.

This research opens up new possibilities for developing more flexible and interpretable emotion recognition systems that seamlessly integrate textual and acoustic information. Future work should focus on improving feature extraction methods for noisy, real-world audio data and exploring more sophisticated ways of representing speech characteristics in natural language.

Our approach not only enhances the emotion detection capabilities of LLMs but also provides a framework for incorporating non-textual information into language model-based systems, with potential implications for a wide range of multimodal AI applications.




§ ACKNOWLEDGMENTS

This research is supported in part by the Defense Advanced Research Projects Agency (DARPA), via the CCU Program contract HR001122C0034. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.





§ LIMITATION

Our approach to enhancing LLM-based emotion detection shows promise, but it is important to acknowledge its limitations. A significant constraint is the method's heavy reliance on audio quality. As evidenced by the contrasting results between the IEMOCAP and MELD datasets, our system's performance degrades considerably with noisy or low-quality audio input.

Another limitation lies in the current feature extraction process. Our approach utilizes a limited set of audio features, which may not capture the full spectrum of emotional nuances present in speech. More complex emotional cues embedded in prosody, rhythm, or spectral characteristics could be overlooked, potentially limiting the depth and accuracy of our emotional analysis. This simplification of audio features may result in a loss of subtle emotional information that humans can easily discern.

The nature of our training datasets presents another challenge. Our study primarily relies on acted (IEMOCAP) or scripted (MELD) emotional expressions, which may not fully represent the complexity and subtlety of emotions in natural, spontaneous speech. This dataset bias could limit the generalizability of our findings to real-world scenarios where emotions are often more nuanced, mixed, or ambiguously expressed. The gap between performed emotions in our datasets and the diverse, spontaneous emotional expressions in everyday life may impact the real-world effectiveness of our emotion recognition system.

Lastly, the computational resources required for this approach pose a practical limitation. Fine-tuning large language models demands significant computational power, which may restrict the implementation of this method in resource-constrained environments. This requirement could limit the accessibility and widespread adoption of our approach, particularly in settings where high-performance computing resources are not readily available.

Addressing these limitations in future research will be crucial for developing more robust, versatile, and widely applicable emotion recognition systems that can handle the complexities of real-world emotional expressions across various audio conditions and computational environments.














§ APPENDIX





 §.§ Speech features for context





In our exploration of incorporating speech features into context utterances, we found that selective application of these features yielded better results than a blanket approach. As shown in Table <ref>, adding speech descriptions to only the last three utterances in the context proved more effective than applying them to all utterances. This suggests that recent context carries more weight in emotion recognition. Notably, pitch features outperformed volume features, with pitch information from the last three utterances providing a 0.6% improvement over non-contextual speech descriptions, achieving a weighted F1 score of 72.595%. Interestingly, combining all speech features for context utterances did not enhance performance, likely due to information overload. These findings highlight the importance of judicious feature selection and context consideration in multimodal emotion recognition tasks.




 §.§ Evaluation of Speech Features Using Simple Machine Learning Models


To evaluate the effectiveness of our extracted speech features in capturing emotion-related information, we conducted experiments using various machine learning models on both numerical and categorical representations of these features. Table <ref> and Table <ref> presents the performance of Random Forest (RF), Support Vector Machine (SVM), and Multi-Layer Perceptron (MLP) models on the IEMOCAP and MELD datasets.






Our analysis of the IEMOCAP dataset reveals that the simple five speech features contain substantial emotion-related information. Both weighted F1 and macro F1 scores (approximately 0.32 and 0.30, respectively) are nearly double the random guess baseline of 0.167. This significant improvement over the baseline indicates that our extracted speech features are effectively capturing emotional cues. Interestingly, the conversion of numerical features into categorical features does not result in a substantial loss of emotion-related information. Moreover, dividing the numerical features into five classes yields better performance than other granularity levels, with weighted F1 and macro F1 scores of 0.313 and 0.293, respectively. This suggests that a moderate level of discretization can reserve clear emotional patterns in speech features.

The results for the MELD dataset highlight one crucial aspects of speech-based emotion recognition: the importance of audio quality. While the weighted F1 scores (around 0.36) are higher than the random guess baseline of 0.143, the macro F1 scores (approximately 0.15) are only marginally better than the baseline. This discrepancy is crucial to understand: MELD is highly imbalanced, with neutral emotions comprising about 50% of the dataset. The high-weighted F1 score is misleading as it primarily reflects the model's ability to predict the dominant neutral class, rather than its capacity to discern emotions from speech features. In contrast, the near-baseline macro F1 score, which treats all classes equally, reveals that the extracted speech features provide minimal informative value across emotion categories. This lack of substantial improvement in macro F1 scores indicates that the key factor behind this poor performance is likely the low quality and challenging nature of the audio in MELD. This finding underscores the critical importance of audio quality in obtaining reliable speech features for emotion recognition tasks. It also explains the modest gains observed when incorporating speech descriptions in our earlier language model experiments on MELD, contrasting sharply with the significant improvements seen in IEMOCAP.

In conclusion, these results not only validate the informativeness of our extracted speech features for emotion detection but also emphasize the crucial role of audio quality in this process. The stark contrast between IEMOCAP and MELD results highlights that high-quality audio data, like that in IEMOCAP, allows for the extraction of informative speech features that significantly enhance emotion recognition. Conversely, challenging audio conditions, as in MELD, can render speech features unreliable and minimally informative. This underscores the need for robust feature extraction methods or pre-processing techniques when dealing with real-world, noisy audio data. Despite these challenges, the performance of categorical representations for IEMOCAP supports our strategy of using discretized speech descriptions in language model prompts for emotion recognition tasks, providing a foundation for future improvements in handling diverse audio qualities.































 Paper Title: Towards Explainable and Interpretable Musical Difficulty Estimation: A Parameter-efficient Approach.






Substrate stabilization of Jahn–Teller distortion in a single molecule
    Andrea Donarini
    August 5, 2024
======================================================================



§ ABSTRACT 
Estimating music piece difficulty is important for organizing educational music collections. This process could be partially automatized to facilitate the educator's role. Nevertheless, the decisions performed by prevalent deep-learning models are hardly understandable, which may impair the acceptance of such a technology in music education curricula. Our work employs explainable descriptors for difficulty estimation in  symbolic music representations. Furthermore, through a novel parameter-efficient white-box model, we outperform previous efforts while delivering interpretable results. These comprehensible outcomes emulate the functionality of a rubric, a tool widely used in music education.
Our approach, evaluated in piano repertoire categorized in 9 classes, achieved  
41.4% accuracy independently, with a mean squared error (MSE) of 1.7, showing precise difficulty estimation. 
Through our baseline, we
illustrate how building on top of past research can  offer alternatives for music difficulty assessment which are explainable and interpretable. With this, we aim to promote a more effective communication between the Music Information Retrieval (MIR) community and the music education one.








§ INTRODUCTION



Estimating the difficulty of music pieces aids in 
organizing large collections for music education purposes. 
However, manually assigning difficulty levels is laborious and might lead to
subjective errors <cit.>. To address this, 
Music Information Retrieval (MIR) research has focused on automating this process for piano works represented in 
various modalities <cit.> as well as repertoires from 
other instruments <cit.>. Furthermore, the interest of companies like Muse Group <cit.> and Yousician <cit.> highlights the industry's recognition of the importance of the task.

Previous work in this field has mainly focused on processing machine-readable symbolic  scores <cit.>.  
These, unlike acoustic features extracted from audio whose understanding depends on signal processing knowledge, are both analyzable by computers and interpretable by humans. Musicians find also easier to understand symbolic features since 
based on music theory knowledge. Initial 
works towards interpretable difficulty assessment focused on visualization 
<cit.>, with Chiu and Chen
 <cit.> making the first attempt to classify difficulty in the piano repertoire with 
explainable descriptors. 




Still, the continually increasing trend towards deep-learning based solutions <cit.>, whose lack of transparency  limits  users'  understanding and therefore leads to an  eventual non-acceptance  in real life applications <cit.>, can impair a fruitful implementation of such technologies in  music educational  practices.





With this background, we propose a white-box <cit.> model (cf. Figure <ref>), 
which 
through the concept of a 

rubric, 

an evaluation instrument from 
music education  used  to support objective assessment  <cit.>,  allows a transparent interpretation of music difficulty. From this point forward, the white-box model will be denoted by . 
Furthermore, to gain a profound understanding of what 
music difficulty means from an explainable perspective, we build upon the 
descriptors  of  
Chiu and Chen
 <cit.> by proposing a new one focusing on music repetitive patterns. 


We also provide
an interactive companion page[At: <https://pramoneda.github.io/rubricnet>] to visualize 
the evaluated 
data and 
scrutinize the results 
in light of its 
interpretability from a musical point of view. 







Through 
eXplainable Artificial Intelligence (XAI), we aim to contribute to music education by 
facilitating the understanding of measurable 
factors that determine a piece's difficulty. 
Our work builds on methods from 
music education, 
where 
objectively assessing  abstract competences through measuring concrete criteria 
is consolidated by employing rubrics, tools which, unlike a black-box, break down complex concepts into simpler ones <cit.>.







Our interpretable methodology aims to bridge the gap between computational models and practical music education needs, enabling educators to make facilitated, but also informed decisions about curriculum development based on the difficulty levels of pieces. 

We release all the code and models of this research[At: <https://github.com/pramoneda/rubricnet>], in order to offer a baseline
for further research in music difficulty assessment.









§ RELATED WORKS


Previous research aiming to  automatically assess  the difficulty of piano repertoire
examined the link between 
fingering patterns and the pieces'
difficulty level <cit.>. 
Recent studies <cit.> have also made significant contributions. 
In <cit.>, representations are used to feed three deep learning models—covering music notation, physical gestures, and expressiveness—to emulate Cook's dimensions <cit.>. These models' predictions are merged using an ensemble method to estimate the scores' difficulty. 
While we appreciate their musicology-inspired approach, its lack of interpretability harms its usability.






Difficulty estimation of 
piano pieces has also been investigated through 
hybrid methods that merge features with deep learning models <cit.>. However, the absence of publicly shared data and code complicates performing comparative analyses these works. 
In <cit.>, the authors  combine the methods from Chiu and Chen
 <cit.> with deep learning models trained using piano roll as input. 
In a similar vein, <cit.> 
uses JSymbolic features <cit.> and deep learning models on a proprietary dataset. 



In the study by Chiu and Chen
<cit.>, 159 pieces from the 8notes website were used, whereas <cit.> utilized 1800 MIDI files from the same source. The categorization of these pieces, provided by users of 8notes, raises concerns about their reliability. Unfortunately, neither study provides access to their data or details on how they were segmented. Other work <cit.> has attempted to understand the effectiveness of various features, including those proposed in <cit.> for categorizing the grade levels of a specific piano curriculum. Recent efforts by Zhang et al. <cit.> and Ramoneda et al. <cit.> have focused on compiling datasets with difficulty annotations from the established piano publisher Henle Verlag, with the latter's dataset not only being the most extensive but also the only one made publicly available. Therefore, for our comparative analysis, we will use the  open-source datasets presented in <cit.>, namely Can I Play It? (CIPI), which has 9 levels of difficulty, and Mikrokosmos-difficulty (MKD), which includes 3 levels.
 


Finally, in order to validate our approach, we consider a different and established feature set, i.e., the standard music symbolic features available through Music21 library <cit.>, which includes (amongst others) established JSymbolic features <cit.>,  
thus facilitating a meaningful comparison with our proposed descriptors. In addition, 
we also contrast the results achieved with our novel descriptors with those obtained with the features by Chiu and Chen
 <cit.>, which are also reimplemented and open-sourced in this study.
Note that none
of the approaches previously mentioned 
has focused on the interpretability of the descriptors, which is a key contribution of our work.




§ INTERPRETABLE 


The  model (cf. Figure <ref>) is designed to provide interpretability akin to a rubric, enabling its analysis and results to be intuitively aligned with established practices in music education. This approach ensures that the model's logic and outcomes are easily comprehensible, facilitating their usage in music education 
along to traditional 
tools.



 §.§ Model Architecture



The network, 
comprising a series of linear layers dedicated to process individual input descriptors and followed by a nonlinear activation function,  
is formulated as follows:

Given a set of N input descriptors, each descriptor x_i is first processed through its dedicated linear layer with weight 
w_i and bias b_i, followed by a hyperbolic tangent activation function to yield:

    s_i = tanh(w_i · x_i + b_i)

where s_i represents the processed score for the i-th descriptor. 
Scores are then aggregated  
in a single 
score S_agg:

    S_agg = ∑_i=1^n s_i


The aggregated score S_agg is then 
passed through a final linear layer to obtain the logits for the class predictions, which are  
mapped to probabilities with a 
sigmoid function:

    P⃗ = σ(S_agg·w_f  + b_f)

where σ denotes the sigmoid function, 
w_f and b_f are the weight and bias of the final linear layer, respectively. 








 §.§ Ordinal Optimization


This model applies an ordinal optimization approach <cit.>, predicting ordered categorical outcomes, difficulty levels such as beginner (1), intermediate (2), and  advanced (3), 
through logits. These logits, computed using a mean squared error (MSE) loss, indicate the model's predictions on the ordinal scale.

Difficulty level is then obtained as:


    max{ i  where  P_i ≥ 0.5  and  P_j ≥ 0.5, ∀ j < i}







 §.§ Interpretability




In , the descriptors (automatically computed from the data) are, to some extent, comparable to the formalized evaluation criteria defined in traditional rubrics; similarly, the aggregated score, might be comparable to a final grade/mark assigned in an educational scenario. Given the correspondences between both, we could consider the model a “white-box” approach, able to promote 
transparency and interpretability, similarly to a  
rubric. 


It uses independent linear transformations on input descriptors to generate scores between -1 and 1, which directly influence the regression output, S_agg. 
Since negative scores, might be not fully understood in terms of difficulty level, 
we normalize scores between 0 and 1, rescaling  between 0 and 12. 
This approach mirrors rubric's ability to provide objective and 
structured feedback, with the simplicity of these transformations aiding in understanding the impact of features in predictions.


The interpretability of the model lies in its ability to dissect each descriptors' influence on a piece's difficulty level. 
Consequently, analyzing each descriptor's scores might reveal its overall importance on the prediction.
Lastly, S_agg is a continuous-ordered scalar with rank correlation to difficulty. 
Therefore, from S_agg, we retrieve ordered and discrete categories  with clear decision boundaries.









§ EXPLAINABLE DESCRIPTORS








From codified musical scores, we extracted numeric features which are feed to a classification
algorithm. 
We re-implemented a set of features from the literature <cit.> while proposing
a novel one,
Pitch Set LZ.
In addition to explaining the features (cf. Table <ref>), we will provide their technical descriptions and analyze their relevance to difficulty and interdependencies using the data.





 §.§ Descriptors


In our work, we analyze music sheets encoded in symbolic format, focusing on extracting pitch and timing. Following the approach suggested by Chiu and Chen <cit.>, we process left and right hand parts separately to clarify pedagogical aspects of musical difficulty. Our primary analysis involves sequences of pitch set events, each characterized by a pitch set S and onset time T. Pitch sets, represented by sets of MIDI numbers, are defined over the alphabet of all pitch sets 𝐒 that occurred in a score part, while onset times are calculated in seconds from the performance start by the music21 library <cit.> marked tempo information. This method emphasizes the timing of note attacks, duration and rests. Additionally, we consider a collection of pitch events, each defined by pitch P over the alphabet of all pitches 𝐏. Our analysis started with the five features identified by Chiu and Chen <cit.> as most relevant to understanding musical difficulty.



Pitch Entropy.
The entropy of pitches in the pitch events:

    -∑_i∈𝐏 p(P = i)log_2p(P = i)



Pitch Range.
The distance between the minimum and maximum MIDI pitches in a score part.



Average Pitch.
The average MIDI pitch in a music sheet.





Displacement Rate. 
Initially proposed by <cit.>, it quantifies the extent of hand movement across the
keyboard during the performance of a score. It analyzes maximum pitch distances between consecutive pitch set events and is calculated as a weighted average of three categories: distances less than 7 semitones (assigned a weight of zero); distances over 7 semitones but under an octave (assigned a weight of one); and distances of an octave or larger (assigned a weight of two to emphasize larger movements).



Average IOI:
Average Inter Onset Interval.
A concept similar to the “Playing speed”  
introduced by
 <cit.>, a term we consider deceptive since it actually decreases as the hand's “speed” increases. This is an average time in seconds between onsets of two consecutive pitch set events. Let's denote i^th onset time with T_i, then the value is:

    ∑_1≤ i ≤ N^events - 1(T_i+1 - T_i)/N^events-1

In 23% of the scores, information about the recommended performance tempo is missing.

We then assume the tempo is 100 beats per minute (bpm). 
Thus, in cases of missing bpm, the Average IOI feature might not be relevant.



Pitch Set LZ. Lempel-Ziv complexity of pitch set sequence. Before introducing our proposed descriptor,
it is crucial to provide context and motivation.
Pitch Entropy, as emphasized by Chiu and Chen <cit.>,
is particularly relevant—a conclusion supported by the analysis of correlations between difficulty
and features in the following section, as well as by informal experiments.
As Sayood discusses <cit.>, there's a link between entropy of a task
and the cognitive load it imposes on the performer, a concept that may also apply to music performance <cit.>.
However, music is often perceived in terms of larger structures like phrases
and sections, not just isolated pitches, prompting us to seek a descriptor that captures the “repetitiveness”
of music on a broader scale. To this end, we employ LZ-complexity, a measure of redundancy introduced by Lempel and Ziv <cit.>.
In context of music research, it was used for binary encoded rhythm analysis by Shmulevich and Povel
<cit.>. We apply LZ-complexity to sequence of pitch sets:
scan a score part, identify all subsequences of pitch sets that cannot be reproduced
from preceding material through a recursive copying procedure.
The number of such unique subsequences is defined as the LZ-complexity of the part.
This approach allows us to assess the structural complexity and redundancy of a musical piece,
highlighting the cognitive demands placed on performers.




 §.§ Feature Analysis





We assume that, for easier interpretability,
features must on average change monotonically with the difficulty level.
To measure this quality, 

we use the τ_c version of Kendall rank correlation coefficient due to 
its ability to deal with
“heavily tied” rankings <cit.> (many musical pieces have the same difficulty, hence, we have multiple ties in the ranking by difficulty). 
τ_c is equal to 1 when feature and difficulty rankings are perfectly aligned in the same direction, -1 if they are aligned in opposite directions. As the number of nonconcordant cases increases, the coefficient approaches zero. 
In <ref>,
the results show that
the features related to pitch organization are the most correlated to difficulty.
Hand displacement and Inter-onset intervals are less correlated,
while average pitch seems almost irrelevant.


In addition, we aim to uncover dependencies among the features themselves while mitigating the influence of difficulty, with whom  most features are correlated.
To achieve this,
we calculate conditional τ_c correlations for all feature pairs given a fixed difficulty level, 
and average the coefficients across all difficulty levels.
We then convert these coefficients into a distance matrix
and apply hierarchical agglomerative clustering based on average distance to identify clusters of correlated features.
From the resulting dendrogram (cf. <ref>), we observe that features correlated with
difficulty—namely Pitch Entropy, Pitch Set LZ, and Pitch Range—are also interrelated.
This is remarkable because the three most correlated features are not inherently dependent:
 one could envision a music piece with any of them maximized while maintaining low values for the others.
However, pieces in CIPI
typically exhibit coordinated values in these descriptors.
Thus, we mostly observe the combined effect of these features,
making it challenging to reliably decompose “difficulty” into an aggregate of independent components.


























§ EXPERIMENTS



 §.§ Experimental Setup


To evaluate the effectiveness of our proposed method, we utilized the Mikrokosmos-difficulty (MKD), and Can I play it? (CIPI) datasets <cit.>.
For fair comparison,
we use the 5-fold cross-validation approach defined in <cit.>. 
In each split, 60% of the data is used as a train set, while the remaining  is equally divided into  validation and test sets.
 

As in <cit.>,  
we employ 
mean squared error (MSE) and accuracy within n classes (-n) for evaluation. These metrics are chosen for their applicability to ordinal classification challenges, with -n assessing the model's accuracy for n classes from the true labels, and MSE measuring the average squared prediction error across classes.
The effects of dataset imbalances and a fair evaluation across classes are mitigated by macro-averaged metrics.





We optimize the models during training through Adam optimizer with a learning rate of 10^-2.
The training process incorporates early stopping, based on the -n and MSE metrics from the validation set, to prevent overfitting. Through 
Ordinal Loss, 
we frame difficulty prediction as an ordinal classification task, as mentioned in Section 3. 
We apply a standard scaler and dropout to the features to prevent individual ones 
from dominating. 
For each experiment, we look for the best hyperparameters using Bayesian optimization <cit.>: batch size within the range from 16 to 128, dropout rate between 0.1 and 0.5, learning rate decay from 0.1 to 0.9, and the learning rate itself, tested over a logarithmic scale from 1e-5 to 1e-1. This 
approach allows us to systematically explore the hyperparameter space and identify the optimal settings for our models; thus, enabling a
fair comparison between experiments.








 §.§ Experimental Results





In Table <ref>, the results from the comparison between the performance of our novel approach  with the presented descriptors  (cf. Sections 3 and 4) and the results achieved by three previous models from the literature (argnn <cit.>, virtuoso <cit.>, pitch) as well as their collective ensemble, are shown. 
Our model 
achieves the highest
Acc-9 score of 41.4 (±3.1) in CIPI, surpassing the ensemble's 39.5 (±3.4), 
while displaying the second lower 
MSE of 1.7 (±0.5), only overtaken by 
the ensemble's 1.1 (±0.2). 
With an Acc-3 score of 79.6 (±8.8) in the MKD dataset, our approach is superior to previous ones but with a higher standard deviation.



























































In the following, we 

examine the impact of various feature and model configurations on  performance (cf.  Table <ref>).
As baseline for comparison, we consider the configuration previously discussed (cf. Ours in Table <ref>).


Employing only the five Chiu and Chen <cit.> descriptors, excluding Pitch Set LZ, leads to a decrease in Acc-9 by -5.2, reflecting a performance drop from the baseline. The use of Music21 <cit.> descriptors, which include JSymbolic <cit.> and other descriptors widely used in the community, results in a decrease in Acc-9 by -4.7 and a decrease in MSE by -0.4, showing slight improvements in MSE but not in accuracy. However, note that a larger number of descriptors could decrease the explainability.
Combining all the descriptors  
slightly decreases Acc-9 and MSE:   -2.5 and -0.4, respectively; with the accuracy results  still under the baseline. 
These results indicate that 
the  descriptors discussed 
in Section 4 constitute the best option for difficulty estimation on CIPI. 
Since average pitch showed no relation to difficulty in the feature analysis, we repeated the experiments without this feature. This lead, however, to non-significant worsening of the results.








Concerning the impact of different model configurations, we replace the tanh by sigmoid non-linearities to guarantee positive scores. The obtained MSE rate is similar but the accuracy drops by -3.1. This means that negative scores could aid in training, which is why we keep them, but normalize the scores after the training phase. Besides, substituting the ordinal encoding used in the baseline 
with a traditional one-hot encoding with cross-entropy loss, results in a decrease in Acc-9 by -5.2 and an increase in MSE by +0.4, highlighting the importance of ordinal regression in achieving lower MSE rates. Lastly, logistic regression with ordinal loss 
decreases the Acc-9 by -1.4 while showing a decrease of MSE by -0.2. This offers a compromise for both metrics
but without beating our setup and to our understanding, being less interpretable.  












Overall, the gains offered by Rubricnet with the features proposed are relatively modest compared to the baselines. However, having a 
smaller feature set is necessary for explainability. The novelty of our approach lies in aligning the interpretability of music education with rubric-like interpretability feedback. This alignment is essential for a successful 
application 
of our model in 
practice, as we will discuss in further sections. 




 §.§ Decision Boundaries


In , the input features are combined into a single scalar before performing
the final ordinal classification. Analysis of the results shows that the final layer
defines optimized decision boundaries, setting thresholds for S_agg that progressively increase along with difficulty levels.
Because of the final sigmoid activation, once  exceeds a boundary, the 
corresponding difficulty level will always be active, which guarantees the ordinality of the predictions.




By examining the decision boundaries (cf. Figure <ref>), we observe that the trends are similar across splits, displaying shorter valid ranges around intermediate levels. Note that in split 2, there are only 8 classes because the model ignored the last class. This can happen as we use numeric optimization, which sometimes falls into local minima. These minima might seem optimal based on the validation metrics but do not meet our overall performance expectations.









§ DISCUSSION AND LIMITATIONS



























Now, we analyze whether  is interpretable from a musical point of view. 

To understand  how features impact the final level suggested by the model, we evaluate the contribution of each descriptor to the aggregated score. 

Since learning to play an instrument is a progressive process,  relative contributions of features to different levels the grade 1 are displayed instead of  absolute values.
These contributions are averaged across splits on the test set and shown in  <ref>.









We observe a trend of higher contributions when the level
increases for every descriptor. 

This observation is consistent 

with the fact that  value increases for higher levels (cf. <ref>).
The most discriminative features are 
pitch entropy and 
pitch range, 
as well as the LZ descriptor for 
higher levels. 

Conversely, some features, average IOI 
or the average pitch,  
have low contribution 
to the model's decisions, as shown by their relatively constant and small values across grades. 



The latter is expected, since 
very different pieces could have the same average pitch, not disclosing anything about difficulty.


The former might be explained by the averaging, which
can remove information, especially when a piece can alternate between fast and slow parts. Besides, as mentioned before, tempo is often poorly annotated in the dataset.









To better understand the explainable capabilities of the proposed descriptors, in the following, we provide a musical examination of two concrete samples, by this demonstrating the interpretability of our approach.
Nocturne op. 9, no. 3 by F. Chopin, is labelled as level 7, but classified as level 2. All the descriptors are below the grade average, as shown in the rubric (cf. Grade Divergence in Figure <ref>).
Our hypothesis is that this nocturne contains many challenges that go beyond the descriptors used. There are constant changes in dynamics, a variety of articulations, and as a key difficulty aspect, many types of polyrhythms between the right and left hands. Further research should address all the types of difficulty challenges, probably underrepresented in the existing datasets.




The piece Berceuse in D-flat major, Op.57 by F. Chopin, shown in <ref> is appropriately classified as grade 7. This is because it maintains a left-hand accompaniment with few changes, in contrast to the higher virtuosity of the right hand. 
The left hand has scores below average in most descriptors because of its few changes: Pitch Range (-0.41), Average IOI (-0.04), Pitch Set LZ (-0.34), Average Pitch (-0.01), and Pitch Entropy (-0.52). In contrast, the right hand shows more virtuosity, with higher than average scores for all the right hand features.
These scores collectively contribute to a final cumulative score that accurately reflects the overall difficulty.


































Finally, it should  be noted that our approach primarily focuses on
descriptors
related to pitch sequences and
onsets, while disregarding others. 
Still, the ablation study showed that other features sets (those from music21), even  covering aspects like rhythm variety,  
do not  enhance our classifier's performance either. In addition, expressive elements <cit.> such as dynamics, tempo changes, and articulation, since often left to performers' interpretation, 
are  not always captured in musical notation <cit.>, 
and therefore is 
a dimension our score-based model does not consider.







§ CONCLUSION AND FUTURE WORK



In our study, we proposed a novel white-box 
parameter-efficient model aligned with the music education community tools, rubrics, which outperforms previous approaches on difficulty estimation. 
In addition, we  
created an interactive companion page for visualizing CIPI and MKD datasets.
In summary,   
we showed that analyzing explainable 
descriptors, unlike deep learning models, 
offers clarity, 
which  gives both teachers and students specific insights into pieces.   
This approach not only underscores the importance of explainable artificial intelligence (XAI) in understanding music difficulty, but also emphasizes the potential for such technologies to contribute to the broader field of music education.
For future research, we consider interesting to creating a dataset based on technical challenges like finger fluency and polyphonic complexity, as well as user studies for understanding the perception of interpretable feedback by music education community.















§ ETHICS STATEMENT


The system presented in this paper aims at obtaining the difficulty of a musical
piece through several descriptors. In previous work, descriptors were not available, limiting access to the area. This situation underscores the need for open science practices. Therefore, we open our implementation, to facilitate access for new researchers. Besides, the dataset used for this study is available upon request for non-profit and academic research purposes.
While this limits its use in commercial applications, it ensures the reproducibility of the results. The data consists of open-source scores of music that
is no longer copyrighted, its use for open research can thus be considered fair.


The proposed work belongs to the area of assisted music learning. One might
argue that such a tool can have a detrimental impact on music teaching jobs. 
While this is a valid concern, we think that an eventual solution of the addressed task,  would not endanger music educators profession, whose role naturally goes much beyond than  categorizing music in difficulty levels.  Instead, this technology should be seen as a way to support them in the own   teaching practices, for instance, by alleviating their burden on some duties, such as exploring large collections, and by this enabling them to easily discover forgotten  musical works from our cultural heritage which fit students' needs. 
Moreover, through this research, we also aim to convey the message that the path to advancement does not solely lie in acquiring more data or creating larger models. 
By highlighting what drives its decisions, our proposed model aligns with the goals of eXplainable AI, something crucial for its acceptance in music education.  
Although our efforts in making the system interpretable and explainable will partly answer  the common criticisms made to black-box approaches, the real impact of our system remains to be verified by  its future use in real scenarios. 





§ ACKNOWLEDGEMENTS


We want to thank Alia Morsi's previous work on difficulty estimation from a feature engineering perspective, which encouraged research in tabular data <cit.>. The team would also acknowledge Marius Miron for continuously insisting on aligning difficulty estimation with explainability. He also highlighted the direction of using a rubric-like explainability feedback, pointing out Ustun et al.'s work <cit.>. We also thank Roser Batlle-Roca for helping to discuss some concepts between interpretability and explainability based on her research <cit.>.

This work was supported by “IA y Música: Cátedra en Inteligencia Artificial y Música’' (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial, and the European Union-Next Generation EU, under the program “Cátedras ENIA 2022 para la creación de cátedras universidad-empresa en IA’'. This work was also supported by the ANR project TABASCO (ANR-22-CE38-0001) and the travel grant MERMOZ2-012047. 
Finally, this work was also possible through the support of the Hightech Agenda Bayern, funded by the  Free State of Bavaria (Germany).






































 Paper Title: Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks.


A Cooperation Control Framework Based on Admittance Control and Time-varying Passive Velocity Field Control for Human–Robot Co-carrying Tasks

    Dang Van Trong, Sumitaka Honji, and Takahiro Wada


        
The authors are with the Graduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma, Nara 630-0192, Japan (e-mail: van_trong.dang.ve6@naist.ac.jp, honji.sumitaka@naist.ac.jp, and t.wada@is.naist.jp).



    August 5, 2024
===========================================================================================================================================================================================================================================================================================================





§ ABSTRACT 
    
    Reinforcement Learning (RL) has been widely used to solve tasks where the environment consistently provides a dense reward value. However, in real-world scenarios, rewards can often be poorly defined or sparse. Auxiliary signals are indispensable for discovering efficient exploration strategies and aiding the learning process. In this work, inspired by intrinsic motivation theory, we postulate that the intrinsic stimuli of novelty and surprise can assist in improving exploration in complex, sparsely rewarded environments. We introduce a novel sample-efficient method able to learn directly from pixels, an image-based extension of TD3  with an autoencoder called NaSA-TD3. The experiments demonstrate that NaSA-TD3 is easy to train and an efficient method for tackling complex continuous-control robotic tasks, both in simulated environments and real-world settings. NaSA-TD3 outperforms existing state-of-the-art RL image-based methods in terms of final performance without requiring pre-trained models or human demonstrations.






§ INTRODUCTION
 

    Motivation could be seen as an internal force or energy to encourage someone to execute or avoid a task.
    It is related to several internal or external factors, such as personality, own beliefs, culture, the surrounding environment, or earlier experiences.
    Motivation permits agents, organisms such as an animal or artificial organisms such as a robot, to play, explore, and discover new behaviours without external reward signals <cit.>. 
    Psychologists distinguish two groups of motivation, intrinsic and extrinsic <cit.>. When you do something for its satisfaction, i.e. without any immediate obvious intention, it is considered intrinsic motivation <cit.>. In contrast,  extrinsic motivation is when you do something to obtain positive feedback associated with that behaviour and the environment.
    Inside the intrinsic motivation group, novelty and surprise play an essential role in the agent's learning process. They stimulate interest and exploration of new stimuli <cit.>. As mentioned in <cit.>, only novel, unexpected and challenging circumstances make an organism intrinsically motivated. Specialized literature  <cit.> claims that exploration of new environments enriches plasticity in the brain to stimulate learning. This phenomenon is easy to recognize in young human infants, who show significant preferences for situations or stimuli never experienced before, even when these stimuli do not provide an associated direct reward <cit.>. 
    
    The underlying idea is simple: a novel situation or a surprise emotion encourages the agent to explore and learn better solutions faster than an agent using only external reward signals <cit.>. This is particularly important in the real world, especially in robotics, where most basic tasks do not have an immediate external reward or are absent. Furthermore, not only are intrinsic and extrinsic reward signals required to learn to solve tasks, but rich and abundant information is also needed, information that can describe and provide key features of the environment and the task itself.  This is particularly important in RL applications, where the state space representation plays a critical role during the learning process. 
    
    Learning from images has proven to be a feasible choice in RL <cit.>. Images encapsulate valuable information that may not always be readily available in other representations, such as proprioceptive states of the dynamics. Moreover, learning directly from pixels offers scalability for real-world applications. Utilizing cameras to gather environmental information proves to be more accurate, cost-effective, and straightforward compared to other sensor types, especially in complex setups featuring high dynamics, such as robots with multiple degrees of freedom (DoF).
    
    Nonetheless, learning directly from high-dimensional images presents a considerable challenge. Most of the state-of-the-art RL algorithms struggle when the input state is pure pixels <cit.>, as well as the computational cost increase <cit.>. Thus, having an accurate but simplified compact representation of the image that the agent can use to learn a control policy might help reduce the complexity and cost. However, creating a compact and valuable representation of an image with all relevant information about the environment and the task from which to learn is a challenge <cit.>. 
    
    Mapping images into a useful encoding usually demands a large training dataset, which is not a particular feature in RL. Ideally, mapping the images into a compressed representation and learning a control policy should co-occur and depend on each other. This interdependence resembles a chicken-and-egg dilemma: an optimal policy necessitates an effective representation, while an effective representation relies on meaningful gradient information derived from the policy. Therefore, achieving synergy between representation learning and policy optimization is paramount.
    
    
    Thus, in this study, aiming to address the challenges of sparse reward environments and image-based learning representations, we introduce a novel RL algorithm called Novelty and Surprise Autoencoder-TD3, NaSA-TD3. We extend and adapt the TD3 framework proposed by <cit.> from a conventional deterministic actor-critic method to a comprehensive image-based actor-critic approach. Drawing inspiration from intrinsic motivation theory, we translate human motivational behaviours into a computational framework by representing novel and surprising stimuli. We leverage these stimuli to uncover more effective exploration strategies to enhance the learning process. Additionally, to support and facilitate the endeavour of learning directly from images, we employ an autoencoder network in conjunction with an actor-critic methodology. This combination enables learning control policies tailored for executing complex robotic tasks, such as dexterous manipulations. The paper's contributions can be summarized as follows:
    
    
        
  * A computational representation of human intrinsic stimuli novelty and surprise.  
        
  * Evaluation of the influence of intrinsic stimuli novelty and surprise, introduced as a reward bonus, on RL for complex robotics tasks.
        
  * An image-based extension of TD3 <cit.> with an autoencoder for learning control policies for continuous action problems.
    
    


§ DEFINITIONS
 
    
    Novelty and surprise are closely related to each other. Their definitions are often incorrectly interchanged or interpreted in the same way; however, there are precise attributes that make them distinguishable <cit.>. 
    
    It is important to mention that there are several other detailed concepts relevant to motivation and intrinsic stimulus from a psychological perspective, along with extensive neuroscience literature; however, they are beyond this article's scope, and we do not address them because it would put us in a category too far from our primary goal. Nevertheless, for more in-depth definitions, we encourage the reader to review <cit.> and <cit.>. This work uses the following definitions.
    


 §.§  What is Novelty?


    The Oxford  English Dictionary defines novelty as “the quality of being new, different, and interesting" <cit.>. Psychologists, psychiatrists, and scientists differ in describing and measuring novelty depending if the stimuli are physical, visual, or behavioural <cit.>. In all cases, however, the common word is different <cit.>. We define novelty as something that we have not seen before; in other words, the process of recognizing new stimuli that are distinguishable from anything known before. 
    Detecting novelty involves digging into the contents of the agent's memory to associate or recognize if an experience has previously been seen or attended. Consequently, whether an observation is novel depends on whether a representation of that observation can be found, partial or identical, in the memory <cit.>.
    For instance, in dexterous manipulation using a robotic gripper, a novel stimulus might manifest as a joint configuration or object state that the agent has not previously encountered.
    


 §.§ What is Surprise?


    A surprise could be defined as a discrepancy, presented as a feeling or emotion excited, between an expected observation and an actual observation <cit.>. As mentioned in <cit.>, “Surprise requires an internal world model that formulates an expectation about the future". Measuring surprise involves a mechanism of prediction and comparison between expectation and actual observation.  Besides, here we do not examine the current observation with past observations; i.e. surprise does not directly correlate with the memory, even though a predicted world model is built using previous experiences <cit.>. 
    In dexterous manipulation with a robot gripper, a surprise stimulus can be conceptualized as the disparity between the expected position of the gripper and the target and the actual positions after taking an action. With surprise, time is important since prediction involves a specific time based on the current observation <cit.>, while novelty can not be associated with predictions since we can not predict situations or events never been experienced <cit.>. 
    


§ RELATED WORK


    The idea of using intrinsic motivation has been used in RL to encourage the agent's exploration. 
    In low-domain environments, a common method for computing exploration bonus signals is having a tabular record of how often a state has been visited, i.e. a count-based strategy. Previous proposals using count-based approaches, such as <cit.>, have shown satisfactory results and demonstrated tackling the exploration problem. However, maintaining state-action visitation count is impractical, inefficient and expensive in complex tasks with continuous state spaces,  making count-based approaches unscalable <cit.>. For example, it would be impractical in a robotics application to try to save all possible states when we deal with an infinite state space. 
    
    To avoid a tabular record of every visited state, <cit.> proposes EX2, a pseudo-count exploration method based on a discriminative exemplar model where the authors use a K-exemplar conditioned network. This approach, however, acts as a binary classifier that may tend to overfit due to the nature of the classifier itself. In <cit.>, the authors present a method using self-organizing maps, a CNN autoencoder and a predictive ensemble model to estimate an intrinsic reward. This method was tested in a simple environment using a single actuator for a learning-to-reach task. While work by <cit.> presents a concept that uses an agent's ability to reconstruct observations by giving their “contexts" (a downsample or noise version of the input image) to generate an intrinsic bonus that is decomposed into fast and slow rewards signals. Due to the need to balance the importance of the intrinsic signals, this method can only be applied to on-policy RL methods. No experiments in real-world or continuous action spaces are presented. Learning without extrinsic rewards has also been examined in <cit.>, where the novelty of a particular state is calculated as the distance of the current behaviour features to the k-nearest neighbours in a memory of behaviour features. However, the agent can end in “procrastination" on the main task instead. 
    
    
    On the other hand, <cit.> proposes a curiosity-based method to help the agent explore the environment more by generating an intrinsic signal based on an inverse dynamics model. In this proposal, the authors transform the observation input into a feature space with the relevant information and only consider the predictions that can affect the agent and ignore the rest. In <cit.>, a goal-based curiosity method is presented, where the authors decompose a task into several easier sub-tasks (called goals) given an observation. The agent samples K goals and predicts the probability that each goal can be achieved from the current observation using a predictor network. The number of samples needed to train the policy is too high to scale this method to real-world setups.

    
    
    Most closely related to our method is the work presented in <cit.>, which introduces a curiosity-based approach coupled with a comprehensive analysis of state representation learning (SRL) to enhance the sample efficiency of image-based RL. In this study, two methods are proposed: one utilizing a regularized autoencoder (RAE), and the other employing a contrastive learning approach. However, this work differs from ours in that it does not integrate intrinsic signals as part of the reward; instead, a separate curious policy based on the SRL error is trained to foster interest in problematic states. This approach is applicable only to off-policy methods and has yet to be validated in real-world environments.
    
    
    Despite the promising results demonstrated in these studies, intrinsic motivation remains underutilized as a potential solution in RL. While related works have illustrated that employing intrinsic stimuli as bonus rewards or additional policy promotes agent exploration, only a subset of these approaches are viable for real-world deployment due to limitations in sample efficiency or computational cost. Moreover, many related works have been evaluated in simplistic real-world environments or simulations, often relying on intricate calculations, predefined initial conditions, or pre-trained models <cit.>. Addressing the majority of these challenges, we present a scalable and robust method capable of tackling control tasks in continuous action spaces. Our algorithm leverages human-inspired stimuli of novelty and surprise to enhance RL learning directly from images. In comparison to traditional heuristic approaches, our method has the ability to solve complex tasks across both simulated and real-world environments.
    
    


§ NASA-TD3
 
    
    The goal of an agent in RL is to find the best action or decision for each state to maximize future rewards R; nevertheless, in some complex tasks, the reward R  may not be present or is sparse, complicating the search for an optimal policy. However, by employing intrinsic motivation stimuli, i.e. augmenting the reward function to deliver a bonus for visiting interesting, novel or surprising states, we can encourage exploration and improve the policy in an environment where the reward is sparse. Thus, intrinsic stimuli encourage the agent to explore more than an agent using extrinsic reward signals only. 

       
    
    The total R signal given a particular state s and an action a can be presented as:
    
    
    R_total(s, a)    = R_ext + R_int
    
            R_int(s, a)      = α * R_novel + β * R_surprise

    
    where R_ext is the extrinsic reward from the environment, R_int is the intrinsic reward formed by R_novel and R_surprise, which are functions designed to capture the novelty and surprise of a given state-action pair and α and β are the weights to balance the degree each term. In our experiments, we fix α =1 and β=1.



 §.§ Novelty and Surprise Detection

    
    Studies <cit.> have demonstrated that even after viewing a thousand pictures, each for a moment and single exposure, we humans can remember with high precision details we have seen in the picture. Recognizing whether we have seen an image is at some degree equivalent to detecting visual novelty, i.e. if we cannot remember seeing something, it can be considered a novel stimulus. As presented in <cit.>, a familiarity memory system involves the medial temporal lobe, perirhinal cortex, and inputs from the visual system to learn and recognize objects without paying attention to details. Consequently, we do not need to look for identical aspects or details in each image. Inspired by this idea, we present a novelty detection method using an unsupervised representation learning with images and an Autoencoder (AE). We determine if an observation looks “familiar" to what we have seen previously, mimicking the human brain. 

                

    At each step, we pass the current agent's observation (an RGB image from the camera) into the AE. If the output of the AE, representing a reconstruction of the observation, deviates significantly from the original input, we infer that the observation is novel and has not been encountered previously. Although we do not explicitly compare the input image with every image stored in memory, we postulate that the AE inherently possesses a memory representation of all previously seen states within its internal layers. This implicit memory mechanism reduces computational costs and eliminates the necessity of comparing each incoming state with the entire stored memory. To quantify novelty, we propose calculating the Structural Similarity Index (SSIM) between the input observation and its reconstruction from the AE, as illustrated in Fig <ref>. Essentially, we measure the disparity between the reconstructed and original images, utilizing this difference as the intrinsic reward bonus. Our choice of SSIM stems from its ability to identify similarities within pixels with better consistency compared to other metrics, such as Mean Square Error (MSE) or Peak Signal to Noise Ratio (PSNR), which are commonly used. This selection sets our approach apart from previous works, such as <cit.>. Thus, at each time step for each observation s_t,  the intrinsic reward signal for novelty is calculated by:    
     
    R_novel (s_t) = 1 - SSIM (s_t, Dec(Enc(s_t))

    
    
    Likewise, we aim to encourage agent exploration by providing a bonus signal for surprise events, i.e., to promote curiosity in unexpected situations. We measure surprise as the discrepancy between what is expected by the agent and what is actually obtained. In other words, the difference between the predicted observation and the true observation. As mentioned earlier, surprise requires an internal world model; however, predicting and generating high-dimensional observations, e.g. images, can be difficult and expensive.  Instead, we use a model to predict the next encoded image, i.e. the latent representation. This simplifies the computational cost compared to the full image reconstruction, presenting another distinctive aspect of our approach compared to previous proposals.
    
    From this, z_t+1 denotes the encoding of the image state s_t+1, and z'_t+1 the predicted representation of the next encoding state given the previous encoding image state z_t and action a_π using a predictive model P_ρ. The intrinsic reward signal for surprise is calculated by:  
    
    
    R_surprise(s_t, a_t)    = MSE(z_t+1, z'_t+1)
     
            z'_t+1   = P_ρ(z_t, a_t)

    
    Additionally, inspired by our previous work <cit.>, we use an ensemble of predictive models. This simple but effective combination helps improve the predictions and keeps stability <cit.> and makes the model less sensitive to noise, especially in the real world. Fig <ref> illustrates the configuration of the predictive model.

    
    
    
    
    It is important to underscore the distinctiveness in our approach regarding intrinsic signals and their respective computation methodologies. While previous studies, such as <cit.>, commonly adopt a unified intrinsic signal strategy, our research clearly emphasises delineating between novelty and surprise, employing separate methodologies for their assessment.
    
    
    
    


 §.§ Image-based policy learning


    Inspired by <cit.>, we introduce our Auto-Encoder Actor-Critic Off-Policy method. Our approach builds upon TD3 <cit.> and adheres to its training dynamics. The selection of TD3 is motivated by its simplicity and stability during training, as well as its ability to mitigate overestimation bias through the utilization of two critic networks. However, what sets our proposal apart is the integration of an AE within the main networks. This addition empowers our method to directly process high-dimensional observations, enabling the agent to learn directly from them. The complete architecture, including the number of layers and activation functions, is illustrated in Fig <ref>.

    
    
    The Actor-Critic network learns and evaluates a policy. This network receives as input the low-dimensional vector z from the encoder, which attempts to extract relevant features from the s observations at time t. Here, both the encoder and decoder are trained by minimizing the L2 reconstruction loss between the input observation and the reconstructed observation. To determine the optimal latent space size, we conducted experiments training the agent under identical conditions while varying only the latent space size. The results of this analysis are presented in Fig. <ref>. Based on these findings, a latent size of z=200 was selected for the remainder of the experiments. 
    
    In terms of policy learning methodology, we adhere closely to the established process of the base TD3 approach. However, instead of utilizing the true observation state s, we employ the latent representation vector z. Both the AE and the policy π are trained simultaneously. Notably, the encoder and decoder networks employed for novelty detection are identical to those utilized for policy learning. A graphic representation illustrating how and where we update the gradients can be seen in the broken lines in Fig <ref>. The actor's gradients are not used to update the encoder networks, while the critic's gradients update the critic networks and the encoder at each training step. 
    
    One of the primary distinctions between our proposal and previous works, such as <cit.> or <cit.>, lies in the comprehensive update of the entire encoder at each step, encompassing both the convolutional and linear layers. We observed that this simple modification enhances the stability of the networks. We attribute this improvement to the fact that the linear layers play a crucial role in constructing the image representation. Furthermore, we maintain consistency by utilizing the same encoder for both the target and main networks (actor and critic) throughout the training process. Consequently, there is no requirement to update the encoder weights of the target networks at a faster rate than the rest of the network's parameters, as observed in <cit.>.  




 §.§ Simulated Environments


    

    Repeating the nominal action is a popular technique used in RL. Several proposals <cit.> claim that action repetition positively affects the learning process. However, suppose the number of repetitions is not set adequately. In that case, it can cause instability in the control dynamics and lead to sub-optimal behaviour <cit.>, making choosing the correct number of repetitions a tuning hyperparameter that depends on each environment <cit.>. Likewise, repetition might not be suitable for real-world applications where servo-motors or linear actuators have fixed movements where action repetitions won't have any effect. Since our goal is an easily scalable algorithm capable of working in a wide range of applications, including real robots, we are not using this technique in our method; instead, we use novelty and surprise signals to obtain the same performance as the algorithms that use action repetition. More details of the training process as well as a list of hyperparameters, full source code and videos of each task can be found on the paper’s website at: <https://sites.google.com/aucklanduni.ac.nz/nasa-td3-pytorch/>
    
    


§ EXPERIMENTAL RESULTS
 
     We assess our algorithm's performance across diverse, complex continuous control robotics tasks in both virtual and real-world scenarios. Additionally, we conduct comparative analyses against SAC-AE <cit.>, utilizing the official source-code implementation provided by the author [<https://github.com/denisyarats/pytorch_sac_ae>]. We choose challenging continuous control tasks of the DeepMind Control Suite <cit.>. 
    
        
    
    This test-bed platform provides complex environments for continuous control tasks with normalized actions and image state observations. We choose six tasks with different complexity and rewards; see Fig <ref>. 
    A third-person camera image of size (84×84) pixels with three channels is used for all tasks. A stack of three consecutive full-colour images is used as the state observation. We run each environment for 1×10^6 training steps with the same hyperparameters for five independently seeded runs. All parameters are trained with the Adam optimizer, and we perform G=5 (based on a trial error experimentation) gradient updates per environment step. We regularly evaluate the agent's performance in each environment during the training; after every 1 × 10^4 training steps, we compute the average reward over 10 episodes. During the evaluation, no gradient updates are performed.

    
    


 §.§ Dexterous Robotics Manipulation in the Real World

    To test the applicability and efficiency of our proposed method to solve real-world tasks, we describe an experiment involving a 4-DoF robot gripper <cit.>; see Fig <ref>. The gripper has two identical fingers equipped with Dynamixel XL-320 servomotors and a standard webcam (C270 Logitech with 720p) on top of the structure. The task requires turning a valve tap to an arbitrary target orientation. Despite the apparent simplicity, the dexterous manipulation task poses significant challenges due to its requirement for visual perception and physical coordination of 4-DoF. Specifically, the robot must learn to synchronize the movements of its two fingers to prevent cancellation of rotational movements, while also mastering a complex finger gait <cit.>. The resulting solution needs to account for real-world factors such as noise, control inaccuracies, and environmental conditions like lighting dynamics and friction. It is worth noting that we are conducting this task without the use of commercially expensive hardware platforms, and we aim to learn directly from raw images without modelling or system identification.
    
    The extrinsic reward for this task is computed as the normalized negative L2-norm distance between the current valve's position and the desired angle. 
    The intrinsic reward is calculated with the novelty and surprise strategy mentioned above. The horizontal episode is limited to 50 steps where we use an action space that outputs a four-element vector ℝ^4 with the desired position of each servomotor. We maintain the same image size as in the previous section to ensure consistency across experiments. Thus, the state space consists of the RGB images from the camera (840×640) downscaled to an image of size (84×84) pixels with three channels. 

    
    
    The robotic task has two scenarios at the end of the episode. In the first case, the valve orientation is left in place between episodes with no resetting or reinitialisation. In the second case, the valve orientation is reinitialised to a random angle (using a servomotor attached to the valve) once the episode ends. Previous related tasks rotate the valve to the same initial orientation at the end of each episode <cit.>. Reinitialising the valve to any random orientation between [0, 2 π) increases the complexity of the task as it requires the agent to learn to use the information from the images to perceive the current valve's orientation. 
    We refrain from comparing this task with SAC-AE <cit.> due to the necessity of modifying its original code. Our primary objective is to showcase that our proposal is readily applicable to real-world applications. We train the model for a total of 5 × 10^4 environment steps, which amounts to approximately 11 hours of real-world training time. Five independently seeded runs are used to confirm the stability of our proposal. All hyperparameters used in this experiment and task-solved videos are listed on our website.



§ RESULTS AND DISCUSSION 

    Fig. <ref> shows the average evaluation reward comparing our NaSA-TD3 proposal, TD3-AE (which is our image-based extension of TD3 without any intrinsic rewards), SAC-AE <cit.> and Pixel-TD3 on the six continuous control tasks under identical conditions[For a fair comparison with our proposal with the exact number of steps for an episode, we present the result of SAE-AE passing action repetition parameter equal to 1]. Utilizing the novelty and surprise signals as intrinsic rewards significantly enhances task performance, with instances of achieving maximum reward values (corresponding to a 1 × 10^3 max return) observed in certain environments. In contrast, some other image-based approaches fail to reach the maximum value or struggle to solve the task entirely, as exemplified by Pixel-TD3. 

        
    
    It is worth noting that AE-TD3’s performance can match or improve SAC-AE in some environments. By further incorporating novelty and surprise stimuli as intrinsic rewards, NaSA-TD3 outperforms all variations. Notably, our method does not require a complex tuning of hyperparameters or complicated training steps as other image-based RL algorithms. Nevertheless, in the spirit of transparency regarding the results, it is worth noting that the novelty and surprise signals do not significantly impact the final results when the task is not highly complex, or the extrinsic reward is not sparse. This can be seen more clearly in environments such as  Walker or Finger Spin, where the agent performed similarly to SAC-AE. Presumably, this is because the intrinsic reward is not required, implying that the problem is easy enough to be solved with an extrinsic reward. In contrast, when the environment is complex or has sparse rewards, as in the Reacher or Cheetah environments, the intrinsic signals help significantly improve the performance.
    
    Fig <ref> shows the average evaluation reward for the real-world manipulation experiments. The results show the effect of novelty and surprise signals in executing the dexterous manipulation task. The superiority of our proposal over extrinsic-reward-only methods is even more evident. Our proposal solves the task and gets the highest reward in both tested scenarios. Pixel-TD3 failed to solve the tasks, while AE-TD3 struggled when the valve was reinitialised to a random orientation. These results underscore the effectiveness of intrinsic stimuli in encouraging the agent to explore a broader range of state-action pairs in complex environments, ultimately leading to faster and improved performance in real-world scenarios.
    
    Our proposal does have some limitations, with one of the most significant being the amount of RAM required. Utilizing pure images as input and storing them in a replay buffer can be computationally demanding, necessitating at least 64 GB of RAM to run a simulation environment for 1 million environment steps. However, this challenge can be mitigated by tuning the buffer size. Our experimentation revealed that other related proposals, such as <cit.> and <cit.>, require similar or even greater amounts of RAM, as they often train models for more extensive training steps. Notably, this issue is not explicitly addressed in their papers. Furthermore, despite our method demonstrating superior and faster convergence, the computational time for each environment step is higher compared to other methods such as SAC-AE <cit.>. This is primarily due to the additional gradient updates for the autoencoder and policy in each step, as well as the computation of  SSIM and MSE for novelty and surprise values, which involve additional calculations and resources.
    


§ CONCLUSION

    
    In this work, we set out to investigate the use of intrinsic rewards as a mechanism for improving the performance of RL in the execution of complex tasks such as dexterous robotic manipulation. We formulated reward metrics inspired by novelty and surprise, using the predictability of patterns across image sequences. A key novelty was the use of Autoencoders, which reduced our system's computational and memory needs to support real-world computation. Our findings show the ability to learn directly from raw images in simulation and real-world robots. This opens the door to scaling this proposal to more complex scenarios and tasks where the only information needed can come from a camera. Likewise, we verify that when the reward is sparse, auxiliary intrinsic signals such as novelty and surprise can help to discover efficient exploration strategies and assist the learning process.
    
    Future work will seek to include other intrinsic signals such as boredom, frustration, and pleasure. Furthermore, we will seek to improve the efficiency in the number of samples necessary, perhaps with the adaption of model-based reinforcement learning algorithms, plus a more comprehensive analysis of other autoencoder architectures and their effect on policy learning.

    































                        








§ ACKNOWLEDGEMENTS

    This research was supported by New Zealand's Science For Technical Innovation (SfTI) on contract UoA3727019. We want to thank the Human Aspects in Software Engineering Lab (HASEL) from the University of Auckland. 


ieeetr





 


 Paper Title: FreqTSF: Time Series Forecasting Via Simulating Frequency Kramer-Kronig Relations.


Decentralized and Uncoordinated Learning of Stable Matchings: A Game-Theoretic Approach
    S. Rasoul Etesami† and R. Srikant

†Department of Industrial and Systems Engineering, Department of Electrical and Computer Engineering, and Coordinated Science Laboratory, University of Illinois Urbana-Champaign,  Urbana, IL 61801 (Email: etesami1@illinois.edu).
Department of Electrical and Computer Engineering and Coordinated Science Laboratory, University of Illinois Urbana-Champaign,  Urbana, IL 61801 (Email: rsrikant@illinois.edu).

    August 5, 2024
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
	Time series forecasting (TSF) is immensely important in extensive applications, such as electricity transformation, financial trade, medical monitoring, and smart agriculture. Although Transformer-based methods can handle time series data, their ability to predict long-term time series is limited due to the “anti-order" nature of the self-attention mechanism. To address this problem, we focus on frequency domain to weaken the impact of order in TSF and propose the FreqBlock, where we first obtain frequency representations through the Frequency Transform Module. Subsequently, a newly designed Frequency Cross Attention is used to obtian enhanced frequency representations between the real and imaginary parts, thus establishing a link between the attention mechanism and the inherent Kramer-Kronig relations (KKRs). Our backbone network, FreqTSF, adopts a residual structure by concatenating multiple FreqBlocks to simulate KKRs in the frequency domain and avoid degradation problems. On a theoretical level, we demonstrate that the proposed two modules can significantly reduce the time and memory complexity from 𝒪(L^2) to 𝒪(L) for each FreqBlock computation. Empirical studies on four benchmark datasets show that FreqTSF achieves an overall relative MSE reduction of 15% and an overall relative MAE reduction of 11% compared to the state-of-the-art methods. The code will be available soon.








§ INTRODUCTION


Time series forecasting (TSF) has become increasingly widespread in real-world applications, such as energy <cit.>, finance <cit.>, medical treatment <cit.>, and agriculture <cit.>. Due to its significant practical value in production and daily life, TSF has received considerable attention.


Significantly benefiting from the robust nonlinear modeling capabilities of deep models, numerous methods have emerged to analyze the complex variations in real-world time series data. Among these methods, approaches based on recurrent neural networks (RNNs) have achieved remarkable results <cit.>. However, due to problems such as the vanishing or exploding gradient <cit.>, RNNs often face performance limitations, which render them unable to capture long-term dependencies. Inspired by the recent successes in NLP and CV <cit.>, Transformer has been introduced to TSF, aiming to address gradient-related issues and capture long-term dependencies <cit.>. Unfortunately, Transformer methods fail to capture regularities in the temporal dimension due to the “anti-order" nature of self-attention <cit.>. Additionally, finding reliable dependencies directly from scattered time points is challenging for attention mechanisms because temporal dependencies may be deeply hidden in complex temporal patterns <cit.>. Some approaches tend to use the simplest multilayer perceptron (MLP) to avoid problems caused by attention <cit.>. While there are numerous approaches <cit.> that currently shift the focus from the time domain to the frequency domain, they typically exploit attention mechanisms after Fast Fourier Transforms (FFT), thereby overlooking intra- and inter-variable variations. Here, as depicted in Figure <ref>, intra-variable variations refer to the changes within each variable over time, while inter-variable variations refer to the mutual influence between different variables.

r0.5
    
    < g r a p h i c s >
 
    Intra- and inter-variable variations. 
    
   

In this paper, we adopt two approaches to capture the long-term dependencies of time series data and overcome the problems of context information loss caused by the “anti-order" nature of attention mechanisms. Firstly, inspired by the significant advantages of frequency-domain methods in TSF <cit.>, 
we shift our perspective to the frequency domain to handle intricate temporal patterns and weaken the impact of order in TSF, which constitute the inherent mechanisms of long-term dependencies in time series data. Recognizing that real-world time series often exhibit multi-periodicity, such as daily and annual variations in electricity transformations and regular weekly and quarterly changes in exchange rates, we use the Short-Time Fourier Transform (STFT) with predefined windows of different sizes. 
STFT decouples overlapping and interacting periodic signals from the time domain into the frequency domain for better deal with intra-variable and inter-variable variations. 
Our second approach is to make the Transformer suitable for feature extraction in the frequency domain. 
We introduce Kramer-Kronig relations (KKRs), 
which are mathematical mechanisms in the frequency domain, into the attention mechanism. We redesign the attention-based structure to extract real and imaginary parts, which simulates KKRs in the frequency domain. Relevant experiments have confirmed that these two representations contribute to TSF tasks. By combining these two approaches, we propose FreqTSF as a new model for TSF. 

Our contributions are four-fold:


    
  * We propose a time series forecasting framework called FreqTSF, formed by residual connections of FreqBlocks, to better capture the global KKR properties of time series in the frequency domain. 
    
    
  * We introduce the Frequency Cross Attention into the FreqBlock structure to capture enhanced mechanisms within time series through frequency domain mapping. The Frequency Cross Attention builds the ability to capture the cross-temporal interactions of time series signals. 
    
  * In particular, we effectively reduce the time complexity and memory complexity from 𝒪(L^2) to 𝒪(L) for each FreqBlock computation, as verified both theoretically and empirically.
    
  * Experimental results show that FreqTSF consistently achieves the state-of-the-art TSF performance with around 15% MSE reduction. 




§ RELATED WORK


In this section, related work includes the problem statement (Section <ref>), the summarization of TSF methods (Section <ref>), the standard method for transforming time domain information into frequency domain information - the Short Time Fourier Transform ( Section <ref>), and Kramers-Kronig relations between the real and imaginary parts of causal systems in the frequency domain (Section <ref>), which form the theoretical foundation of our work.



 §.§ Problem Statement


Given data points X={x^t_1,...,x^t_D}^L_t=1∈ℝ^D× L in a lookback window of time series as input, where L is the size of the window, D≥ 1 is the number of variables, and x^t_i is the value of the i-th variable at the t-th time step. 
The task of TSF is to predict the forecasting horizon X̂={x̂^t_1,...,x̂^t_D}^L+T_t=L+1∈ℝ^D× T. In this paper, our research is based on multi-step TSF, i.e., T>1.



 §.§ Time Series Forecasting


Data-driven time series forecasting allows researchers to understand the evolution of systems without the need to construct precise physical laws <cit.>. With the development of deep neural networks, RNNs, specialized for sequence data analysis, have been proposed and achieved remarkable results. However, due to the gradient problems, RNN <cit.>, LSTM <cit.>, GRU <cit.> and LSSL <cit.> still need help adapting to long-term time series forecasting tasks.

Transformer has been developed and applied to time series forecasting. As a core component of Transformer, the attention mechanism plays a crucial role in identifying dependencies between different variables at various time points. Many improvements have been proposed to make the attention mechanism suitable for time series forecasting at lower complexity <cit.>. However, complex temporal patterns may mask such correlations and are challenging to manifest as significant trends or periodicities in the time domain <cit.>. Therefore, some researches aim to use the simple multilayer perceptron (MLP) to deal with the above problems <cit.>. Other methods shift the perspective from time domain to frequency domain, and then discover the inherent trends or periodicities of time series <cit.>. However, most of the SOTA models <cit.> still process time series data in the time domain without modeling the potential frequency domain causal mechanisms of time series data, facing challenges in dealing with complex temporal variations. More discussions about TSF methods are in the Appendix.



 §.§ Short-Time Fourier Transform 


A commonly used method for transforming time series data from the time domain to the frequency domain is the Short-Time Fourier Transform (STFT) <cit.>. 
The fundamental component of STFT, consisting of multiple windows of different sizes, is the Fourier transform, which can be accelerated using FFT. STFT initially divides more extended time series into shorter segments of equal length and then computes the Fourier transform separately for each of these shorter segments. 
As the window slides along the time axis until it reaches the end, the STFT of the time series is computed, resulting in a two-dimensional representation of the series. Moreover, different window sizes form multi-scale STFT results.
Mathematically, this process can be expressed as:


    STFT{X}(ω,τ)=∑_t=1^L x^t w(t-τ)e^-iω t,


where w(τ) represents the window function, usually a Hann window, X denotes the time series to be transformed and ω is the angular frequency with i^2=-1. The STFT{X} is essentially the Fourier transform of x^t w(t-τ), representing a function that captures the variations of the signal concerning both time domain and frequency domain.



 §.§ Kramers-Kronig Relations


In control theory, a causal system is defined as one in which the output depends on past and present inputs but not on future ones <cit.>. 
Such a relationship is also consistent with the inherent generation mechanism of time series data, where the data unfolds gradually over time, and future data points do not precede those with a time delay. Therefore, in this study, we approximate time series data as a causal system for analysis. In the following, we will elucidate a theoretical constraint on a causal system in the frequency domain, specifically regarding the interdependence of the real and imaginary parts in terms of Kramers-Kronig relations <cit.>. The constraints on the relationship between the real and imaginary parts can guide the design and training of FreqTSF, ensuring the mapping of the time domain to the frequency domain without altering the intrinsic mechanisms.


    If X(t) is a causal time function with respect to time t, i.e. , X(t) = 0,     t < 0 
X(t),     t≥ 0, and does not contain the continuous-time unit impulse function δ(t) = 0,     t 0 
∞,     t=0 and derivatives at t=0. Suppose the Fourier transform ℱ of X(t) is represented in the form of the real part Re and the imaginary part Im, i.e., X(t)↔ F(ω)=Re(ω)+iIm(ω). Then, we have:

    
    Re(ω)=1/π∫_-∞^∞Im(σ)/ω-σdσ,
       Im(ω)=-1/π∫_-∞^∞Re(σ)/ω-σdσ.



The proof for Theorem <ref> is shown in the Appendix. The Equation <ref> <cit.> states that for any causal time function, the real and imaginary parts of its Fourier transform satisfy the KKRs. These relations represent bidirectional mathematical connections between the real and imaginary parts of complex functions. From the Equation <ref>, it can be intuitively observed that the representations of the real and imaginary parts exhibit periodic characteristics based on ω. In addition, the real and imaginary parts themselves can reflect the trend characteristics of time series, making the expression of the two parts based on KKRs conducive to effective forecasting.

Furthermore, in the frequency domain, we can obtain a multi-scale representation and multi-frequency components of time series through the STFT. This transform not only improves the computational efficiency but also effectively reduces the computational cost of time series analysis <cit.>. However, the significance of KKRs formed by the real and imaginary parts in the frequency domain, which characterize the properties of time series variables in the frequency domain, has not been recognized. To the best of our knowledge, our proposed method is the first work to achieve TSF by utilizing real and imaginary parts of KKRs in the frequency domain to improve the accuracy and reliability of the model.





§ FREQTSF


In this section, we propose the FreqTSF, depicted in Figure <ref>, with a modular architecture to model the KKRs of time series in the frequency domain derived from different variables. More details about the overview of the framework are in the Appendix. To capture intra-variable and inter-variable variations, shown in Figure <ref>, we design a set of residual-connected FreqBlocks within the FreqTSF, which adopt the Frequency Transform Module, followed by the Frequency Cross Attention, to adaptively allocate attention weights to the real and imaginary parts of the intra-variable variations. Simultaneously, it also integrates inter-variable information through a parameter-efficient inception block.



 §.§ Frequency Transform Module


The Frequency Transform Module transforms the input into frequency components using the STFT:


    C_i,p,k=STFT{X_i}(ω_k,τ_p)=∑_t=1^L x_i^t w_p(t-τ_p)e^-iω_k t,


where X_i is the i-th time series, w_p(τ_p) represents the p-th window function with center τ_p (totally P window functions with the set of window functions denoted as W={w_p}_p=1...P), and ω_k is the k-th angular frequency. C_i,p,k denotes the k-th result of the STFT using the p-th predefined window function for the i-th time series (including the real and imaginary parts). The Equation <ref> is further decomposed into the real and imaginary parts based on e^-iω_k t=cos(-ω_k t)+isin(-ω_k t)=cos(ω_k t)-isin(ω_k t) as follows:


    C^Re_i,p,k   =∑_t=1^L x_i^t w_p(t-τ_p) cosω_k t,
    
            C^Im_i,p,k   =-∑_t=1^L x_i^t w_p(t-τ_p) sinω_k t,


where C^Re_i,p,k and C^Im_i,p,k represent the k-th real and imaginary components, respectively, of the i-th time series obtained by decomposition using the p-th window. Considering the sparsity of the frequency domain, we select only the top-M amplitude values to reduce the computational burden of the attention mechanism. Then we obtain the most significant real components C^Re_i,p,1,...,C^Re_i,p,M and imaginary components C^Im_i,p,1,...,C^Im_i,p,M, where M is a hyperparameter. Additionally, we concatenate the two parts obtained through all P windows to get Re∈ℝ^D× P× M and Im∈ℝ^D× P× M with Amp(·) denoting the amplitude values:


    {C^Re_i,p,1,...,C^Re_i,p,M},{C^Im_i,p,1,...,C^Im_i,p,M}= k∈{1,...,M}Top-M(Amp(C^Re_i,p,k,C^Im_i,p,k)),
       Re =i∈{1,...,D},p∈{1,...,P},m∈{1,...,M}Concatenate(C^Re_i,p,m),
       Im =i∈{1,...,D},p∈{1,...,P},m∈{1,...,M}Concatenate(C^Im_i,p,m).




 §.§ Frequency Cross Attention


Based on the Theorem <ref>, we can obtain the corresponding imaginary part from the real part based on the KKRs, and vice versa. The key idea is to compute an importance score between the real and imaginary parts of each variable, which can characterize the KKRs. Therefore, we design the Frequency Cross Attention, which includes two parts, B^Re and B^Im, where B^Re characterizes the ability of the real part to preserve the imaginary part, while B^Im establishes the ability of the imaginary part to transform into the real part, as illustrated in Figure <ref> right. 

We consider the above establishments as extensions of the relationships in the frequency domain. Therefore, to learn the correct representation of the real part, we need to enrich the real part Re with the information from the imaginary part Im, resulting in the KKR-compliant real part R̃ẽ and vice versa. To allow the model to jointly attend to information from different representation subspaces, we use H heads of cross-attention mechanisms. For the h-th head, we use linear mappings to transform the real part Re into Q^Re_h, K^Re_h, and V^Re_h, and the imaginary part Im into Q^Im_h, K^Im_h, and V^Im_h:


    Q_h^Re=Re^TW^Re,Q_h + b^Re,Q_h,    Q_h^Im=Im^TW^Im,Q_h + b^Im,Q_h,
     
        K_h^Re=Re^TW^Re,K_h + b^Re,K_h,    K_h^Im=Im^TW^Im,K_h + b^Im,K_h,
     
        V_h^Re=Re^TW^Re,V_h + b^Re,V_h,    V_h^Im=Im^TW^Im,V_h + b^Im,V_h.


Here, W^*,Q/K_h ∈ℝ^D × d_K, W^*,V_h ∈ℝ^D × d_V, b^*,Q/K_h∈ℝ^d_K, and b^*,V_h ∈ℝ^d_V are learnable parameters. 
Therefore, Q_h^*, K_h^*∈ℝ^P × M × d_K, V_h^*∈ℝ^P × M × d_V. Then, when calculating Attn^Im_h, the representation of the real part Re is enriched with the imaginary part Im. In contrast, when calculating Attn^Re_h, the representation of the real part Re needs to be filtered by learnable weights to select information that can establish the KKRs with the imaginary part Im:


    Attn^Im_h(Q_h^Im,K_h^Im,V_h^Re)   =softmax(Q_h^Im(K^Im_h)^T/√(d_K))V_h^Re,
    Attn^Re_h(Q_h^Re,K_h^Re,V_h^Im)   =softmax(Q_h^Re(K^Re_h)^T/√(d_K))V_h^Im.


After computing the attention weight matrices for all H real attention heads and imaginary attention heads, we concatenate all Attn^Im_h and Attn^Re_h and then apply linear transformations to map them to the real part R̃ẽ∈ℝ^P× M× d_hidden and imaginary part Ĩm̃∈ℝ^P× M× d_hidden with implicit KKRs.


    R̃ẽ   =MultiHead(Q^Im,K^Im,V^Re)= Concat(Attn^Im_1,...,Attn^Im_H)W^Re,
    Ĩm̃   =MultiHead(Q^Re,K^Re,V^Im)= Concat(Attn^Re_1,...,Attn^Re_H)W^Im.


Here, H denotes the number of parallel attention heads, and the projections are parameter matrices W^Re, W^Im∈ℝ^Hd_V× d_hidden. Thus, we achieve adaptive attention weight allocation for the real and imaginary parts within the variables by Frequency Cross Attention while determining the KKRs.



 §.§ FreqBlock


As shown in Figure <ref>, we organize the FreqBlock in a residual way <cit.>. Specifically, for the length-L input time series X ∈ℝ^D× L, we first apply a normalization block at the very beginning to deal with the nonstationary characteristics arising from varying mean and standard deviation. Additionally, a denormalization block is applied at the very end to transform the model outputs back to the original statistics <cit.>. Subsequently, for the normalized input time series X' ∈ℝ^D× L, we utilize the linear embedding layer Embed to project the column inputs into deep features X^0∈ℝ^E× L. For the nth layer of the FreqTSF (containing a total of N layers) with the input X^n-1∈ℝ^E× L:


    X^n=FreqBlock^n(X^n-1)+X^n-1.


As depicted in Figure <ref>, in the nth FreqBlock, the process consists of two successive phases in the frequency domain: capturing intra-variable causal variations and integrating inter-variable information.



  §.§.§ Capturing Intra-variable Causal Variations


In each FreqBlock, we apply the Frequency Transform Module and the Frequency Cross Attention in sequence. The deep features X^n-1∈ℝ^E× L undergo the Frequency Transform Module to obtain enhanced real and imaginary parts in the frequency domain, which are selectively Top-M enhanced in the frequency domain. Using the Frequency Cross Attention, we connect the real part R̃ẽ and the imaginary part Ĩm̃ with the KKRs. This process allows us to efficiently derive informative causal representations using a parameter-efficient inception block <cit.>. The formalized procedure is:


    Re^n,Im^n = Frequency Transform Module^n(X^n-1),
       R̃ẽ^n,Ĩm̃^n=Frequency Cross Attention^n(Re^n,Im^n),
       X̂^n = Inception^n(Linear^n(Concat(R̃ẽ^n,Ĩm̃^n))),


where Re^n,Im^n∈ℝ^E× P× M are the enhanced real and imaginary parts obtained by encoding X^n-1 through the n-th Frequency Transform Module, where P is the number of windows in the STFT, and M is a predefined parameter as mentioned in Section <ref>. R̃ẽ^n,Ĩm̃^n∈ℝ^P× M× d_hidden represent the real and imaginary parts with potential KKRs.



  §.§.§ Integrating Inter-variable Information


After the Frequency Cross Attention, we concatenate the real and imaginary parts with KKRs and then process them through a linear mapping Linear^n with learnable parameters W^n∈ℝ^d_hidden× E and b^n∈ℝ^E. Subsequently, we pass the result through a parameter-efficient inception block denoted as Inception^n(·), which incorporates multi-scale kernels and is widely recognized as a prominent architectural component in computer vision. The learned representations serve as the output of the n-th FreqBlock, combining inter-variable information that captures correlations between variables through their mutual influence. 



 §.§ Efficient computation


For each FreqBlock, the time complexity is the sum of the time complexity of the STFT in the Frequency Transform Module and the attention mechanism in the Frequency Cross Attention. The time complexity of the STFT with a window size of ||w_p|| is typically expressed as 𝒪(||w_p||log ||w_p||). This is because applying the FFT to each window requires computational effort of 𝒪(||w_p|| log ||w_p||), and for a time series data with length L and variable number D, the total time complexity for P windows is 𝒪(∑_p=1^PLD ||w_p|| log ||w_p||). Considering that W, D, P are usually predefined constants and are typically much smaller than L, the time complexity is finally 𝒪(L).

On the other hand, the memory complexity of FreqTSF depends on the requirements of storing intermediate results in the STFT and attention weight matrices in the Frequency Cross Attention. For the STFT, the FFT results must be stored for each window, resulting in a cost of 𝒪(∑_p=1^P DL). Therefore, the memory complexity of the Frequency Cross Attention is mainly due to the storage of attention weights and intermediate results. However, since the Frequency Transform Module freezes the input size for Frequency Cross Attention, i.e., Re^n,Im^n∈ℝ^E× P× M, where E, P, M are predefined hyperparameters, the complexity of Frequency Cross Attention in this work is 𝒪(EPM)=𝒪(1). Furthermore, for D, P, which are generally predefined constants and typically much smaller than L, the memory complexity of FreqTSF is 𝒪(L).



§ EXPERIMENTS


In this section, we compare the well-acknowledged and advanced models in time series forecasting, including the RNN-based models: LSTM <cit.> and LSSL <cit.>; CNN-based Model: TimesNet <cit.>; MLP-based models: LightTS <cit.> and DLinear <cit.>; Transformer-based models: Reformer <cit.>, Informer <cit.>, Pyraformer <cit.>, Autoformer <cit.>, FEDformer <cit.>, Non-stationary Transformer <cit.> and ETSformer <cit.>. Overall, 12 baselines are discussed in the Appendix and included for a comprehensive comparison. 

To thoroughly evaluate the performance of FreqTSF, we evaluate the proposed FreqTSF on two real-world benchmarks and two widely acknowledged time series simulators, covering mainstream applications in TSF: energy (the ETT dataset <cit.>), economics (the Exchange dataset <cit.>), medicine (the Type 1 Diabetes Mellitus Simulator (T1DMs) <cit.>), and agriculture (the Lintul3 <cit.>).

Similar to TimesNet <cit.>, We follow standard protocol and split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets. We use the mean square error (MSE) and mean absolute error (MAE) to evaluate the accuracy of TSF. Details about datasets and metrics are shown in the Appendix.






 §.§ Main Results


For better comparison, we follow the experimental settings of TimesNet <cit.> and Autoformer <cit.>. Expressly, for the ETT, Exchange and T1DMs datasets, the lookback window length is consistently set to 96, and the forecasting lengths for both training and evaluation are fixed to be 96, 192, 336, and 720, respectively. Furthermore, in the case of the Lintul3 dataset, the fixed lookback window length is 30, with corresponding forecasting lengths for both training and evaluation set to 7, 14, 21, and 30. 

The results of TSF are presented in Table <ref>. We count the number of first predictions achieved by each model across various datasets, which serves as a comprehensive metric for evaluating model performance. Experimental results show that FreqTSF achieves the best overall performance across all benchmark datasets. Specifically, FreqTSF achieves state-of-the-art performance in over 30% of the cases. Compared to TimesNet and Dlinear, FreqTSF exhibits an overall relative reduction of 15% in MSEs and 11% in MAEs. Notably, on specific datasets, such as ETTh1, the improvement exceeds 30%. While FreqTSF performs slightly worse than TimesNet, ETSformer, LightTS, and DLinear on the Exchange dataset, the results of these five methods are close together, indicating that FreqTSF remains highly competitive on this dataset.
On the T1DMs dataset, the systematic variations in human blood glucose levels contribute to a highly regular pattern characterized by a sharp postprandial rise followed by a gradual decline to a stable baseline. Consequently, all baselines, including FreqTSF, exhibit remarkably low MSEs and MAEs due to the predictable nature of these glycemic fluctuations.
FreqTSF demonstrates competitive MAE performance compared to the optimal LSTM on the Lintul3 dataset. However, the relative MSEs are more considerable, possibly due to abrupt numerical variations in the Lintul3 dataset. FreqTSF, together with the Frequency Transform Module, may discard such variations as noise, leading to inaccurate fitting of certain variables. Nevertheless, overall, this confirms the effectiveness of FreqTSF in TSF. This improvement is attributed to the design of the Frequency Transform Module and Frequency Cross Attention, which will be further verified in the subsequent ablation experiments.




 §.§ Model Efficiency Analysis


To summarize the model performance and efficiency, we conduct a comparison of parameter scale and GPU memory size on the ETTh1 dataset using five baseline models that achieved the first MSE and MAE in TSF (i.e. TimesNet, ETSformer, LightTS, Dlinear, and FEDformer). Among these models, our proposed FreqTSF and TimesNet exhibit nearly identical parameter sizes and GPU memory, with parameter sizes of 4.6MB and 4.5MB and GPU memory sizes of 1703MiB for both. In contrast, the MLP-based models, such as LightTS and Dlinear, have relatively smaller parameter sizes and GPU memory, averaging around 0.04MB and 1699MiB, respectively. However, these models may struggle to simulate time series data effectively due to their small parameter size. On the other hand, ETSformer and FEDformer have parameter sizes and GPU memory similar to MLP-based models, but they fail to capture the extreme values in time series data. We attribute the parameter size of our model to the adoption of the inception block at the end of the FreqBlock, a design choice shared with TimesNet. Despite the increased parameters, FreqTSF achieves the best overall performance within an acceptable single-GPU model size.



§ DISCUSSION


This study presents, for the first time, an attention-based algorithm for simulating the KKRs in the frequency domain. Major findings include: (1) our proposed FreqTSF demonstrates significantly superior performance compared to baseline TSF algorithms in terms of both accuracy and computational complexity; (2) through ablation experiments on modules, shown in the Appendix, the combination of the two proposed modules (the Frequency Transform Module and the Frequency Cross Attention) leads to a significant improvement in model performance; (3) ablation experiments on the frequency domain filter in the Appendix confirm the importance of M, which affects the final model performance; (4) ablation experiments on hyperparameter sensitivity in the Appendix show the robustness and reproducibility of FreqTSF. However, it is worth noting that Table <ref> presents the performances of FreqTSF compared to state-of-the-art baselines. Nevertheless, for Exchange and T1DMs, the performances of FreqTSF do not exhibit statistical significance when compared to other methods. Therefore, we will perform a more thorough analysis to determine the reasons for this result.

Looking more closely at FreqTSF's performance on the Exchange dataset, while the relative gaps between FreqTSF and the best-performing models range from 7% to 25%, the absolute gap averages only 0.15. For the T1DMs dataset, while FreqTSF has larger relative gaps compared to the best-performing models, the absolute gap ranges from 0.01 to 0.16. We argue that based on these observations, FreqTSF remains competitive as its absolute gaps with state-of-the-art baselines are not substantial.

A possible reason for these observations is that the Exchange dataset contains a significant number of mutations. FreqTSF first transforms time series data from the time domain to the frequency domain within each FreqBlock, automatically categorizing mutations as noise and consequently ignoring them. This disregard may contribute significantly to the poor performance of FreqTSF on the Exchange dataset. In addition, for the T1DMs dataset, physiological indicators exhibit periodic variations due to the fixed timing of daily carbohydrate intake (three meals per day). This simple periodicity and regular fluctuation is easily captured by Pyraformer, which uses pyramidal attention to capture hierarchical multi-scale temporal dependencies in the time domain. Conversely, our frequency domain transformation module uses STFT with windows of different sizes, which can capture such periodic changes. However, by transforming the time domain to the frequency domain, the effectiveness in capturing these simple patterns may be reduced.



§ CONCLUSION, LIMITATIONS AND FUTURE WORKS


In this paper, we propose FreqTSF that leverages FreqBlocks connected in a residual way, resulting in state-of-the-art performance. Each FreqBlock employs the STFT for the multi-period signal analysis of variables, facilitating the acquisition of intra-variable time series KKRs from a frequency perspective. The proposed Frequency Cross Attention fully exploits the KKRs of the real and imaginary parts in the frequency domain of causal systems, demonstrating its effectiveness in time series forecasting. Extensive experiments demonstrate that FreqTSF achieves superior forecasting performance with minimal resource consumption on four benchmark datasets, outperforming 12 state-of-the-art algorithms. Furthermore, FreqTSF reduces the time and memory complexity of each FreqBlock computation from 𝒪(L^2) to 𝒪(L).

While FreqTSF demonstrates exceptional performance, there are still some limitations. For example, FreqTSF may incorrectly categorize mutations in time-series data as noise and overlook the accurate modeling of simple periodic curves. In addition, FreqTSF currently relies solely on attention to learn the KKR representation, but the use of neural ordinary differential equations (ODEs) <cit.> could potentially provide a superior solution. Therefore, to further improve and overcome the limitations of FreqTSF, in our future work, we intend to utilize both time- and frequency-domain information to improve model performance. In addition, we aim to explore large-scale pre-training methods specifically tailored to time series applications, using FreqTSF as the underlying architecture. This approach has the potential to bring significant benefits to a wide range of downstream tasks.



§ ACKNOWLEDGMENTS


This study was supported in part by a grant from the National Key Research and Development Program of China [2021ZD0110900] and the National Natural Science Foundation of China [62006063].

unsrtnat
 














§ APPENDIX




 §.§ Time Series Forecasting


Many classical methods, such as ARIMA <cit.>, follow a Markovian process by constructing autoregressive TSF models based on predefined time patterns. However, this autoregressive process becomes ineffective in real-world nonlinear and non-stationary conditions.

With the flourishing development of deep neural networks, RNNs, specialized for sequence data analysis, have been proposed and achieved remarkable results. Vanilla RNN <cit.>, as the simplest form in the RNN family with a single hidden layer and recurrent connections, often suffers from the problems of vanishing or exploding gradient, which makes it challenging for long sequence prediction <cit.>. To address the gradient problems, Long Short-Term Memory (LSTM) <cit.> and Gated Recurrent Unit (GRU) <cit.> employ gated structures to control the flow of information, allowing better handling of dependencies in long sequences. 
LSSL <cit.> conceptually unifies the strengths of RNNs and maps a sequence by simply simulating a linear continuous-time state-space representation.
However, due to inherent design flaws, recurrent models still need help adapting to long-term TSF tasks.

Transformer has been developed and applied to TSF. As a core component of the Transformer, the attention mechanism plays a crucial role in identifying dependencies between different variables at various time points. Reformer <cit.> addresses the problem of quadratic computational complexity in the attention mechanism. It replaces dot-product attention with locality-sensitive hashing to reduce the time complexity of attention computation and employs reversible residual layers to reduce memory complexity. Informer <cit.> proposes the probspare self-attention to reduce the time and memory complexity to 𝒪(Llog L).
Pyraformer <cit.> adopts pyramidal attention to capture hierarchical multi-scale temporal dependencies, thereby reducing the computational cost. 
Non-stationary Transformer <cit.> focuses on improving the forecasting ability to model the non-stationary data. It introduces series stationarization and de-stationary attention and demonstrates its effectiveness in forecasting non-stationary data prediction through both theoretical analysis and experiments.
However, relying on the attention mechanism in the above Transformer-based methods to directly discover correlations between variables at different time points is unreliable. Complex temporal patterns may mask such correlations and are challenging to manifest as significant trends or periodicities in the time domain <cit.>. Furthermore, the primary capability of the Transformer comes from its attention mechanism, which inevitably leads to the loss of temporal information <cit.>.

Some researches aim to use the simple multilayer perceptron (MLP) to deal with the above problems.
LightTS <cit.> applies an MLP-based structure on top of two delicate down-sampling strategies, including interval and continuous sampling.  
Dlinear <cit.> introduces an embarrassingly simple one-layer linear model and employs direct multistep forecasting, achieving high accuracy. 
Other approaches shift the focus from the time domain to the frequency domain.
Autoformer <cit.> proposes autocorrelation mechanisms which first employ period-based dependencies and then aggregate time delay using FFT, for accelerated computation. 
Fedformer <cit.> adopts an expert mixture design to improve trend and periodicity decomposition and proposes a sparse attention mechanism in the frequency domain, ensuring efficiency while improving prediction accuracy.
ETSformer <cit.> employs exponential smoothing attention and frequency attention mechanisms to replace the self-attention mechanism in the vanilla Transformer, improving both accuracy and efficiency. TimesNet <cit.> provides a new approach to TSF by decomposing time series into multiple periods using predefined windows. Through the interaction of intra-period and inter-period information, it connects generic inception blocks to complete TSF. However, models mentioned above process time series data without modeling intra- and inter-variable variations. As a result, they still face challenges in dealing with complex temporal variations.



 §.§ Theoretical Proof



    If X(t) is a causal time function with respect to time t, i.e. , X(t) = 0,     t < 0 
X(t),     t≥ 0, and does not contain the continuous-time unit impulse function δ(t) = 0,     t 0 
∞,     t=0 and derivatives at t=0. Suppose the Fourier transform ℱ of X(t) is represented in the form of the real part Re and the imaginary part Im, i.e., X(t)↔ F(ω)=Re(ω)+iIm(ω). Then, we have:

    
    Re(ω)=1/π∫_-∞^∞Im(σ)/ω-σdσ,
       Im(ω)=-1/π∫_-∞^∞Re(σ)/ω-σdσ.





  X(t) can be expressed as X(t)=X(t)ϵ(t) by introducing the unit step function ϵ(t) = 1,     t>0 
0,     t≤ 0. Taking the Fourier transform of both sides of X(t)=X(t)ϵ(t), based on the frequency domain convolution theorem <cit.>, we have:
  
    ℱ{X(t)}=Re(ω)+iIm(ω),
       ℱ{X(t)ϵ(t)}=1/2π[Re(ω)+iIm(ω)]× [πδ(ω)+1/iω].


  Using ℱ{X(t)}=ℱ{X(t)ϵ(t)} and then we have:

  
    Re(ω)+iIm(ω)
                 =1/2π[Re(ω)+iIm(ω)]× [πδ(ω)+1/iω]
       =[Re(ω)/2+1/2πIm(ω)×1/ω]+i[Im(ω)/2-1/2πRe(ω)×1/ω].

  
  Equating the real and imaginary parts on both sides of the equation and rearranging, we get:
  
    Re(ω)=1/πIm(ω)×1/ω=1/π∫_-∞^∞Im(σ)/ω-σdσ,
       Im(ω)=-1/πRe(ω)×1/ω=-1/π∫_-∞^∞Re(σ)/ω-σdσ.


  So far, the Equation <ref> has been proven.





 §.§ General Overview of FreqTSF


The input lookback window, X ∈ℝ^D× L, undergoes normalization and embedding modules to obtain X^0 ∈ℝ^E× L. It is then fed into a stack of residual FreqBlocks, and the output of the Nth-layer FreqBlock, after denormalization, serves as the forecasting horizon X̂∈ℝ^D× T. The nth FreqBlock is defined as X^n=FreqBlock^n(X^n-1)+X^n-1. First, the FreqBlock utilizes Equation <ref> to apply the STFT with windows of various sizes to the time series data, obtaining the corresponding real and imaginary parts in the frequency domain, which follows the Top-M frequency enhancement (Equation <ref>). Next, the FreqBlock establishes intra-variable KKRs by applying the Frequency Cross Attention module between the enhanced real and imaginary parts. Finally, the FreqBlock integrates inter-variable information through a parameter-efficient inception block. The Frequency Cross Attention module leverages KKR-based attention to construct mathematical mechanisms, and concatenates the real part R̃ẽ and the imaginary part Ĩm̃ of the attention output (Equation <ref>), representing the hidden information with inherent KKRs in the frequency domain for output.





 §.§ Datasets


To thoroughly evaluate the performance of FreqTSF, we evaluate the proposed FreqTSF on two real-world benchmarks and two widely acknowledged time series simulators, covering mainstream applications in TSF: energy (the ETT dataset <cit.>), economics (the Exchange dataset <cit.>), medicine (the Type 1 Diabetes Mellitus Simulator (T1DMs) <cit.>), and agriculture (the Lintul3 <cit.>).


    
  * The ETT dataset <cit.> comprises data collected from electricity transformers, recorded every 15 minutes from July 2016 to July 2018. The dataset contains seven time-series variables, including load and oil temperature. Specifically, ETTh1 and ETTh2 are variant datasets with hourly granularity.
    
  * The Exchange dataset <cit.> contains daily exchange rates for eight countries recorded from 1990 to 2016, encompassing eight time-series variables.
    
  * The Type 1 Diabetes Mellitus Simulator (T1DMs) <cit.> elucidates the temporal evolution of patient states in response to different distributions of exogenous insulin intervention doses. This simulator has been accepted by the FDA as a viable alternative to preclinical animal testing for the evaluation of novel treatment modalities for Type 1 Diabetes Mellitus. In this study, simulations are performed using default settings tailored for adult patients to generate temporal data sequences.
    
  * Lintul3 <cit.> is a simple crop growth model designed to simulate growth under water and nitrogen-limited conditions. In this experiment, we simulate the growth process of spring wheat in Northeast China from January 2002 to August 2002. The dataset includes 20 time-series variables, including two dimensions of action variables and 18 dimensions of crop trait variables.


We follow standard protocol and split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets. 



 §.§ Evaluation Metrics


We use the mean square error (MSE) and mean absolute error (MAE) to evaluate the accuracy of TSF, which is calculated as follows:


    MSE(X̂,X)   =1/D× T∑_d=1^D∑_t=L+1^L+T(x̂_d^t-x_d^t)^2
    MAE(X̂,X)   =1/D× T∑_d=1^D∑_t=L+1^L+T|x̂_d^t-x_d^t|


Here, X̂∈ℝ^D× T represents the estimated time series, while X denotes the corresponding ground-truth.



 §.§ Implementation details


Similar to TimesNet <cit.>, our proposed FreqTSF is trained using the L2 loss and optimized using the Adam optimizer with an initial learning rate of 10^-3. The batch size is set to 32. All experiments are repeated five times, implemented in PyTorch, and conducted on a single NVIDIA RTX TITAN 24GB GPU.

Note that we set the embedding dimension E to 32 and map the input X∈ℝ^D× L to X'∈ℝ^32× L. Subsequently, we employ N=4 FreqBlocks with residual connections to avoid the degradation issues. In each FreqBlock, we utilize P=3 windows with sizes W={100,50,20}, all of which are Hann windows. We make the top-M=10 selection for the STFT result in each window to reduce computational complexity. The dimensions d_K, d_V, and d_hidden in the Frequency Cross Attention are all set to 64. Within the inception block, we use six kernels of different sizes to fuse inter-variable information from different STFT windows. We find that the dimensions of the hidden layers and the number of kernels are not critical factors affecting the performance of FreqTSF. Therefore, we conduct ablation experiments on the number of FreqBlocks N, the embedding dimension E, the window W, and the frequency domain filter M, which will be discussed in Section <ref> and <ref>.





 §.§ Showcases


To dissect the contributions of each module within our proposed FreqTSF, we conduct a comparative analysis of prediction results on the ETTh1 dataset using five baseline models (TimesNet, ETSformer, LightTS, DLinear, and FEDformer). These baselines have previously demonstrated proficiency with first MSE and MAE performance on the ETT and Exchange real-world datasets. In Figure <ref>, it is evident that the two modules enhance the forecasting capabilities of the Transformer model from different perspectives, particularly in capturing intricate details of the prediction curve, which are conducted on the ETTh1 dataset.

FreqTSF, TimesNet, and FEDformer exhibit higher forecasting accuracy among these six models. In contrast, other models suffer from deficiencies in capturing trends and periodicities, leading to imprecise predictions. Additionally, we observe that the prediction curves of TimesNet and FEDformer are relatively smoother, failing to capture abrupt changes in the time series at certain time points. ETSformer struggles with predicting extreme points, as evidenced by inaccurate predictions of both the lowest and highest values in the figure. Moreover, LightTS and DLinear, both MLP-based methods, exhibit significant cumulative errors that result in increasingly inaccurate predictions over time. In comparison, FreqTSF stands out by effectively capturing both trends and periodicities, as well as abrupt changes in the time series at specific time points. We believe that this phenomenon is attributed to the Frequency Cross Attention, which provides a consolidated representation of the real and imaginary parts in the frequency domain, thereby making enhanced trend and periodicity features causal. This module, followed by the inception blocks, enables adaptive integration of information between variables, resulting in precise predictive performance. This result underscores the effectiveness of FreqTSF in simultaneously capturing various characteristics of time series, offering a comprehensive and accurate forecasting solution.





 §.§ Ablation Studies


In this section, two types of ablation experiments are conducted. The module ablation studies are designed to validate the effectiveness of the two proposed modules. Moreover, the ablation studies for M aim to assess the significance of the critical parameter M, which determines the extent to which noise introduced by meaningless frequencies is mitigated in the context of this study.



  §.§.§ Ablation Studies for Modules


In this section, we conduct ablation experiments on ETTh1 dataset to validate the effectiveness of the Frequency Transform Module and the Frequency Cross Attention. The current state-of-the-art results of FreqTSF, which utilizes both the Frequency Transform Module and the Frequency Cross Attention, serve as the baseline. Three ablation variants of FreqTSF are tested: 1) FreqTSF-FTM-FCA: neither the Frequency Transform Module nor the Frequency Cross Attention is used; 2) FreqTSF-FCA: the Frequency Transform Module, and the vanilla attention mechanism are used to capture frequency domain information; 3) FreqTSF-FTM: the Frequency Cross Attention is used without the Frequency Transform Module. The ablated versions of FreqTSF and the SOTA model are compared in Table <ref>. 


As shown in Table <ref>, FreqTSF, including the Frequency Transform Module and the Frequency Cross Attention, shows nearly 100% improvement over FreqTSF-FTM-FCA, demonstrating improvement in all four cases. When the Frequency Transform Module is added, FreqTSF-FCA achieves an additional 4% improvement over FreqTSF-FTM-FCA. We posit that the Frequency Transform Module can effectively remove extraneous frequency information (such as noise) from the frequency domain, Transform functional frequency components, and thereby facilitate the model's understanding of trend-related information in time series data, ultimately improving accuracy. With the addition of Frequency Cross Attention, FreqTSF-FTM achieves a 6% improvement over FreqTSF-FTM-FCA. We attribute this improvement to the ability of Frequency Cross Attention to simulate time series modeling based on the inherent mechanics of causal systems through the establishment of real and imaginary KKRs. This improvement contributes to the reliability of the model. As for the combination of the Frequency Transform Module and Frequency Cross Attention it not only achieves denoising effects but also captures the causal characteristics of the system, resulting in state-of-the-art performance, which validates the effectiveness of the designed Frequency Transform Module and Frequency Cross Attention in capturing frequency domain information.



  §.§.§ Ablation Studies for Frequency Domain Filter




We introduce a hyperparameter M in the Equation <ref> to eliminate irrelevant information in the frequency domain. In this section, M=∞ denotes the absence of the Top-M selection. The ablation analysis for M is presented in Figure <ref>. Experimental results on ETTh1 dataset show that, when T={96,192,336,720} respectively, MSEs and MAEs tend to initially decrease (or remain constant) and then increase with increasing M. We posit that the choice of M significantly affects the final model performance, possibly because M directly influences the amount of information within deep representations. Furthermore, when M=∞, an excessive amount of information in the frequency domain results in Frequency Cross Attention being unable to capture critical information. Conversely, when M=5, insufficient frequency domain information hinders accurate predictions. Hence, in this study, considering both efficiency and performance, we set M=10 for TSF.



 §.§ Hyperparameter Sensitivity 


We introduce a hyperparameter W in the Equation <ref> to predefine the sizes of the windows in the STFT, enriching the information obtained from the Fourier transform through different dimensions. Therefore, we conduct ablation experiments on ETTh1 dataset for the embedding dimension E and the number of FreqBlocks N. Table <ref> provides the sensitivity analysis for the above three hyperparameters.
Besides, we can also find out the following observations:

The number of FreqBlocks N: We observe that N should be increased with the growth of the forecasting length T, but its impact on the model accuracy is not substantial. In the experiments with T=96 and T=192, across the above eight comparison experiments with fixed E and W, when N=2 it leads to lower MSEs and MAEs in 7/8 cases. However, in the experiments with T=336 and T=720, it achieves lower MSEs and MAEs in 6/8 cases when N=4. Nevertheless, the choice of N does not significantly reduce the MSEs and MAEs. Thus, we believe that the model performance is more robust to the choice of N.

The embedding dimension E: We find that E should also increase with the growth of the forecasting length T, and E has a notable impact on the accuracy of FreqTSF. In the experiments with T=96, we observe that holding N and W constant, the MSEs and MAEs are lower when E=32 compared to E=64. However, in the experiments with T=192, T=336, and T=720, FreqTSF achieves lower MSEs and MAEs when E=64. This effect can be attributed to the fact that E directly affects the amount of information in deep representations.

The window sizes W: We find that the effect of the window sizes on the FreqTSF decreases as T increases. In the experiments with T=96 and T=192, keeping N and E constant, we observe that it leads to lower MSEs and MAEs in 6/8 cases with window lengths of {100, 50, 20}. However, in the experiments with T=336 and T=720, changes in the STFT window size have little effect on the MSEs and MAEs. Thus, we claim that the model performance is more robust to the choice of W.






 Paper Title: AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning.
























  Baylor University
  One Bear Place
  Waco
  Texas
maisha_rashid1@baylor.edu



  Baylor University
  One Bear Place
  Waco
  Texas
pablo_rivas@baylor.edu













§ ABSTRACT 
  Multimodal machine learning models that combine visual and textual data are increasingly being deployed in critical applications, raising significant safety and security concerns due to their vulnerability to adversarial attacks. This paper presents an effective strategy to enhance the robustness of multimodal image captioning models against such attacks. By leveraging the Fast Gradient Sign Method (FGSM) to generate adversarial examples and incorporating adversarial training techniques, we demonstrate improved model robustness on two benchmark datasets: Flickr8k and COCO. Our findings indicate that selectively training only the text decoder of the multimodal architecture shows performance comparable to full adversarial training while offering increased computational efficiency. This targeted approach suggests a balance between robustness and training costs, facilitating the ethical deployment of multimodal AI systems across various domains. 







<ccs2012>
 <concept>
  <concept_id>10010147.10010178.10010219.10010220</concept_id>
  <concept_desc>Computing methodologies Image representations</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>10010147.10010257.10010321</concept_id>
  <concept_desc>Computing methodologies Language resources</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>10002978.10003022.10003023</concept_id>
  <concept_desc>Security and privacy Adversarial learning</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
 <concept>
  <concept_id>10010147.10010257.10010293.10010294</concept_id>
  <concept_desc>Computing methodologies Neural networks</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
 <concept>
  <concept_id>10010147.10010257.10010282</concept_id>
  <concept_desc>Computing methodologies Cross-validation</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
</ccs2012>


[500]Computing methodologies Image representations
[500]Computing methodologies Language resources
[300]Security and privacy Adversarial learning
[300]Computing methodologies Neural networks
[100]Computing methodologies Cross-validation

























AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning
    Pablo Rivas
    
======================================================================================





§ INTRODUCTION

Multimodal machine learning is an advanced area of artificial intelligence that integrates and processes multiple types of data inputs, such as text, images, and audio, to perform tasks that mimic human cognitive abilities <cit.>. This integration allows the model to leverage the strengths of each modality, leading to richer interpretations and more accurate predictions that could be achieved by models processing the data in isolation. One of the most compelling applications of multimodal learning involves the combination of visual and textual data, commonly referred to as image-text pairs. This pairing is particularly significant as it mirrors the way humans often receive and interpret information, making the study of these models not only interesting but also aligned with natural human communication patterns.

The increasing deployment of multimodal models in critical applications also raises significant safety and security concerns. These models, like all machine learning systems, are susceptible to adversarial attacks which means intentional inputs designed to confuse the model and provoke incorrect outputs <cit.>. The robustness of multimodal models, therefore, becomes a critical area of focus. Ensuring these models can withstand adversarial attacks is not just a technical necessity but a safety imperative, particularly when these models are employed in sensitive contexts such as autonomous driving, healthcare, and content moderation. 

Making multimodal machine learning models more robust against adversarial attacks is a key challenge <cit.>. These models, which combine different types of data like text and images, can be especially vulnerable because of the way these data types interact. Traditional methods that focus on making each type of data robust on its own can be very demanding and might not address all the issues.

In this paper, we look at a simpler and more efficient approach. Instead of trying to make the entire model robust, we focus on improving just one part of it. We apply adversarial training techniques specifically to the text decoder part of our image captioning model. Our experiments with the Flickr8k and COCO datasets show that this method works well. Training only the text decoder, while keeping the image encoder fixed, gives us results almost as good as training the whole model. However, when we fix the text decoder and train only the image encoder, the performance drops significantly. This shows that the text decoder plays a crucial role in making the model robust against attacks.

This paper helps improve ethical AI by showing a focused way to make multimodal models stronger. This method not only improves the technical side of AI but also helps build public trust in AI systems used in different areas.

This paper is organized as follows: Section 2 reviews related work on adversarial attacks and defenses in multimodal machine learning. Section 3 describes our model architecture and the adversarial training method. Section 4 presents the experimental setup and results on the Flickr8k and COCO datasets. Section 5 discusses what our findings mean for developing robust and ethical AI systems. Finally, Section 6 concludes the paper and suggests future research directions.



§ BACKGROUND

In <cit.>, the authors investigated the vulnerability of multimodal deep learning models to adversarial attacks, finding that even if only one input modality is attacked, the overall performance degrades significantly. The authors highlight the need to explore ways to obtain robust features from multimodal data to achieve useful information from each modality and improve adversarial defense mechanism. To develop adversarial defense mechanisms we explored different adversarial attacks in machine learning models. 
One of the first and best-known adversarial attack techniques is the Fast Gradient Sign Method (FGSM), which involves applying perturbations to the input data in the direction of the gradient of the loss function with respect to the input data to produce adversarial examples <cit.>. Another adversarial attack similar to FGSM, which creates the attack iteratively, is Projected Gradient Descent (PGD) <cit.>. PGD generates adversarial examples by iteratively taking small steps toward the direction of maximum loss function and projecting the outcome back onto the allowed data range at each step. Other well-known adversarial attack strategies are Jacobian Saliency Map Adversary (JSMA) and C&W Attack.
JSMA is an iterative adversarial attack technique that computes a saliency map based on the Jacobian matrix to identify the most significant pixels to perturb in order to mislead neural network classifiers <cit.>. The C&W attack, proposed by Carlini and Wagner, is an powerful and efficient optimization-based adversarial attack that finds quasi-imperceptible perturbations to input examples that reliably cause misclassification in neural networks <cit.>.

The authors in <cit.> discussed about adversarial attack in multimodal machine learning model like CLIP in their paper. They presented new types of adversarial attacks against the multi-modal neural network model CLIP, which integrates the detection of visual objects with text reading. Basic typographical manipulations like misspellings and font changes are included in the attacks, along with conceptual ones that use contradicting text and image inputs to trick the model. Adversarial training is one of the most common defense mechanism for adversarial attack in machine learning models. Adversarial training is using adversarial examples as samples in the training phase of the model. The authors in <cit.> proposed a fast adversarial training algorithm that produces robust models at a low computational cost by recycling the gradient information computed during training. The method is able to train robust image classifiers for the ImageNet dataset in a short time on a modest hardware setup.


§ METHODOLOGY



 §.§ Model Architecture

This work uses an architecture for the image captioning task that combines GPT-2 and Vision Transformer (ViT) models <cit.>, with the GPT-2 <cit.> acting as a decoder and the ViT as an encoder. The ViT model divides an input image of 224 x 224 pixels into 16 x 16 pixel patches. Then, these patches are flattened and projected linearly into a space with dimensions of 768. The transformer encoder receives the image patch sequence and a [CLS] token. The encoder consists of multiple layers of position-wise feedforward and multi-headed self-attention networks. The  token, which compiles data from all patches, represents the whole image. This token's final hidden state at the ViT encoder's output captures the contextualized global aspects of the image, which are essential for generating a caption.  

The decoder obtains the encoded image features from the ViT's [CLS] token, which uses them as the starting point for caption generation. Through many layers of masked multi-headed self-attention, the GPT-2 model processes this input, allowing each point in the output sequence to attend only to earlier positions in the sequence. This arrangement preserves the autoregressive characteristic by ensuring that each word in the caption is generated solely based on the words that came before it. The GPT-2 model produces a sequence of tokens that comprise the image's caption. Every token is created one after the other, and the model predicts the subsequent token by analyzing the preceding tokens and the contextual data extracted from the image.  



 §.§ Adversarial Attack

In our work, we used the adversarial example technique to create adversarial attacks. Specifically, we employed the Fast Gradient Sign Method (FGSM) to generate these examples. FGSM is a method for crafting adversarial examples, which are inputs to machine learning models that have been intentionally modified to cause the model to make incorrect predictions <cit.>.

Given an input 𝐱 and its true label y, we define the loss function J(θ, 𝐱, y), where θ represents the parameters of the model. The goal is to find a small perturbation η such that the perturbed input 𝐱' = 𝐱 + η leads to a misclassification. FGSM achieves this by using the gradient of the loss function with respect to the input 𝐱.

First, we compute the gradient of the loss function:

    𝐠 = ∇_𝐱 J(θ, 𝐱, y).


Then, the adversarial example 𝐱' is generated by adding a perturbation in the direction of the sign of this gradient:

    𝐱' = 𝐱 + ϵ·sign(𝐠).


Here, ϵ is a small constant that controls the magnitude of the perturbation. The sign function, sign(𝐠), is applied element-wise to the gradient, ensuring that each element of the perturbation has the same magnitude, ϵ.




Table <ref> shows examples of the adversarial perturbations generated using FGSM on the Flickr8k and COCO datasets. Despite its simplicity, FGSM is effective in generating adversarial examples that are close to the original inputs but cause the model to make incorrect predictions. This method is computationally efficient and serves as a useful tool for testing and improving the robustness of machine learning models. This table shows how adversarial attacks are imperceptible to human eyes, as shown in the second column. The third column demonstrates the perturbations by showing the difference between the original and perturbed images. In the 'Difference' column, we can see square or patch-like patterns. This happens because of ViT's architecture, which processes input images by dividing them into patches. FGSM leverages the gradients computed for each patch, generating perturbations that exploit the vulnerabilities within these regions. Table <ref> underscores the necessity for robust models capable of defying such attacks. By incorporating adversarial examples into the training process, we can enhance the model's ability to defend against such attacks and improve its overall robustness.



 


 §.§ Dataset

In this study, we leverage two widely-used and benchmark datasets for the image captioning task: Flickr8k and COCO (Common Objects in Context).


  §.§.§ Flickr8k Dataset

The Flickr8k dataset comprises 8,000 images, each accompanied by five human-annotated captions. The images are sourced from Flickr, a popular online photo-sharing platform, and are categorized into 20 distinct classes.


  §.§.§ COCO Dataset

The COCO dataset is a larger and more challenging resource for image captioning, containing 123,287 images with five captions per image. Unlike the Flickr8k dataset, the images in COCO are sourced from various online platforms, including Flickr, and are categorized into 80 classes. 


 §.§ Evaluation Metric

To evaluate the performance for image captioning task, we used Bilingual Evaluation Understudy (BLEU) score <cit.>. Comparing the generated captions to the reference captions in the dataset generates a similarity score known as the BLEU score. Higher scores correspond to a closer match. The BLEU score computes the n-gram overlap between the model's output and the ground truth. There is a perfect match between the generated and reference captions when the BLEU score is 1, from 0 to 1. 


 §.§ Experiment

We used a multi-phased approach incorporating adversarial training to increase the robustness of our multimodal image captioning model, which combines a Vision Transformer (ViT) encoder and a GPT-2 decoder. We conducted our experiment in five phases as follows:

    
  * We trained the ViT-GPT-2 architecture on the Flickr8k dataset without considering adversarial factors, creating a baseline model that served as a benchmark for assessing the following experiments.
    
  * We applied the FGSM to create adversarial examples for the training and test sets. We used these samples to train the model and evaluated the performance of the model. 
    
  * To improve robustness, we performed adversarial training on the model using both the original data and adversarial examples.
    
  * We investigated a more focused strategy to further improve the model's robustness. Initially, we used only the GPT-2 decoder for adversarial training while freezing the ViT encoder.
    
  * We reversed the process by freezing the GPT-2 decoder and using the ViT encoder alone for adversarial training.



We conducted steps four and five three times each and calculated the average of these trials.  
Focusing on a single modality at a time allowed us to implement more effective adversarial defense mechanisms without having to scale them across the entire model. We repeated these five steps of experiment with COCO dataset as well to access the robustness of our model. For both the datasets we have used the same perturbation magnitude, ϵ = 0.1.



§ RESULTS

After conducting experiments on both the Flickr8k and COCO datasets, we evaluated the performance of our multimodal machine learning models using the BLEU score metric. The goal of these experiments was to find an efficient way to increase the robustness of these models against adversarial attacks. As shown in Table <ref>, our baseline model, trained on the original data, initially achieved the best performance. However, when we generated adversarial examples using the FGSM and trained the model with these samples, its performance decreased significantly. This suggests that our multimodal model is not robust against adversarial attacks, as generating a single adversarial example for each image in the dataset was sufficient to degrade its performance.

To enhance the model's robustness, we trained it with both the original data and the adversarial examples. This adversarial training approach improved the model's performance, indicating that incorporating adversarial examples during training can help mitigate the impact of such attacks.

Aiming to find an efficient way to increase the robustness of our multimodal model, we investigated whether training only one of its components (either the image model or the text model) would be sufficient. First, we froze the image model (ViT) and trained the text model with both the original data and the adversarial examples. As shown in Table <ref>, the model's performance in this case was lower than with full adversarial training, although the difference was not significant. Alternatively, we froze the text model (GPT) and trained the image model; in this case, we can see the performance is much lower than the full adversarial training's performance. 


We conducted similar experiments using the COCO dataset, and the results are presented in Table <ref>. The baseline performance on the COCO dataset was higher than on Flickr8k. However, when the model was trained solely on adversarial examples generated from the COCO dataset, the performance decreased significantly, as expected. In the third phase, where we trained the model on both the original data and the adversarial samples, the result was comparable to the baseline model's performance.

When we froze the image model (ViT) and trained only the text model, the difference in BLEU score compared to the full adversarial training approach was relatively small. This observation aligns with our findings from the Flickr8k dataset.





§ DISCUSSION

Through experiments on both the Flickr8k and COCO datasets, we can conclude that while the adversarial training technique improves the robustness of our multimodal models against adversarial attacks, their performance does not fully match the baseline levels achieved with clean data. However, the adversarially trained models showed performance close to the baseline, suggesting that this approach can effectively mitigate the impact of adversarial attacks.

Our findings suggest that performing adversarial training only in the text model can achieve comparable performance to full adversarial training. However, freezing the text model or the GPT-2 model during training significantly degrades performance. When the ViT encoder is frozen and adversarial training is performed solely on the GPT-2 decoder, the model still benefits from the ViT encoder's robust feature extraction capabilities. The ViT encoder continues to deliver consistent and reliable image features, enabling the GPT-2 decoder to concentrate on learning to generate text robustly. In contrast, when the GPT-2 decoder is frozen and adversarial training is applied solely to the ViT encoder, the model's capacity to generate coherent and contextually relevant text is hampered. Since the parameters of the GPT-2 decoder remain fixed, it cannot adjust to the alterations in the image features produced by the ViT encoder during adversarial training. Consequently, this lack of adaptability in the text generation process results in a decline in performance. In conclusion, this finding presents an opportunity to increase computational efficiency by focusing the training efforts on a text modality, with a modest trade-off in terms of performance compared to training the entire multimodal model.



§ CONCLUSION

Our research has shown important steps forward in making multimodal machine learning models more robust, especially for tasks like image captioning. We combined Vision Transformer (ViT) and GPT-2 architectures and used adversarial training methods to focus on specific parts of the model. This focused approach helps protect against adversarial attacks without needing to train the entire model, which saves time and resources. Through our experiments with the Flickr8k and COCO datasets, we found that applying adversarial training to just the text decoder can make the models much more robust. While the performance of these models with adversarial training is slightly lower than those trained only on clean data, the trade-off is minimal. This means we can improve model safety without losing much accuracy. Freezing the image encoder and training the text decoder helps balance performance and robustness efficiently. On the other hand, training the image encoder alone does not provide the same benefits, highlighting the text decoder's key role in maintaining robustness.

Future work will explore changes in activation functions for additional gains in adversarial robustness <cit.>. Current and future work supports the goals of Ethical AI by making AI systems safer and reliable. By focusing on specific parts of the model, we can make the adversarial defense process more efficient. This targeted approach can lead to the development of AI systems that are both resource-efficient and secure for public use. Enhancing the robustness of AI models against adversarial attacks builds public trust and confidence in AI systems, especially in critical applications like healthcare, autonomous driving, and content moderation. 




§ ACKNOWLEDGMENT

This research work was part of a project funded by the National Science Foundation under grants CNS-2210091, CHE-1905043, and CNS-2136961.

ACM-Reference-Format








 Paper Title: Image Super-Resolution with Taylor Expansion Approximation and Large Field Reception.


























Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals
Image Super-Resolution with Taylor Expansion Approximation and Large Field Reception
    Jiancong Feng, Yuan-Gen Wang, Mingjie Li, and Fengchuang Xing




Feng, J., Wang, Y.-G., Li, M., and Xing F. are with the School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou 510006, China (e-mail: fengjiancong@e.gzhu.edu.cn; wangyg@gzhu.edu.cn; limingjie@gzhu.edu.cn; xfchuang@e.gzhu.edu.cn). 




    August 5, 2024
===========================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Self-similarity techniques are booming in blind super-resolution (SR) due to accurate estimation of the degradation types involved in low-resolution images. However, high-dimensional matrix multiplication within self-similarity computation prohibitively consumes massive computational costs. We find that the high-dimensional attention map is derived from the matrix multiplication between Query and Key, followed by a softmax function. This softmax makes the matrix multiplication between Query and Key inseparable, posing a great challenge in simplifying computational complexity. To address this issue, we first propose a second-order Taylor expansion approximation (STEA)  to separate the matrix multiplication of Query and Key, resulting in the complexity reduction from 𝒪(N^2) to 𝒪(N). Then, we design a multi-scale large field reception (MLFR) to compensate for the performance degradation caused by STEA. Finally, we apply these two core designs to laboratory and real-world scenarios by constructing LabNet and RealNet, respectively. Extensive experimental results tested on five synthetic datasets demonstrate that our LabNet sets a new benchmark in qualitative and quantitative evaluations. Tested on the RealWorld38 dataset, our RealNet achieves superior visual quality over existing methods. Ablation studies further verify the contributions of STEA and MLFR towards both LabNet and RealNet frameworks.





Blind Super-Resolution, Self-similarity, Taylor Expansion, High-dimensional matrix multiplication 










§ INTRODUCTION



Single-Image Super-Resolution (SISR) <cit.> has obtained widespread application in various fields, such as remote sensing imagery, security surveillance, and medical imaging. SISR methods are generally divided into two categories: non-blind SR and blind SR. The non-blind SR <cit.> assumes that the degradation type is known, which degradation function is typically represented as:

    𝐱_lr = 𝐱_hr↓_s,

where 𝐱_hr and 𝐱_lr denote the HR and LR images respectively, and ↓_s denotes the bicubic downsampling operation with a scale factor of s. However, the non-blind SR represented by Eq. (<ref>) fails to accurately approximate the real-world degradation due to the inaccessibility of the reference image.



Subsequently, researchers have developed blind SR methods, which assume that the LR image is obtained from the HR image through a degradation function with a complex and unknown degradation type, and this degradation function is typically assumed to follow the equation:

    𝐱_lr = (𝐱_hr⊗𝐊𝐞𝐫𝐧𝐞𝐥) ↓_s + N,

where ⊗ denotes the convolutional operation, 𝐊𝐞𝐫𝐧𝐞𝐥 represents the unknown blur kernel, ↓_s signifies the downsampling operation with a scale factor of s, and N accounts for sophisticated noise. According to Eq. (<ref>), we can clearly see that a single HR image corresponds to infinitely many possible LR images, which is typically an ill-posed problem. Nonetheless, a multitude of blind SR methods has been proposed to overcome this problem. They can be broadly divided into two main types: methods without blur kernel estimation <cit.> and methods with blur kernel estimation <cit.>.

As for methods without blur kernel estimation, SRMD <cit.> is the first to combine the preset blur kernel, noise, and LR image as the input to the SR network. After that, <cit.> incorporated Maximum A Posteriori Probability into iterative optimization. For these methods, the selection of parameters for the preset blur kernel is crucial for achieving effective results in practical applications. However, it is challenging to manually choose appropriate preset blur kernel parameters in the real world. To overcome this challenge, researchers have developed some methods to estimate the blur kernel. IKC <cit.> proposed to iteratively estimate the blur kernel, which divides the SR reconstruction process into two steps: 1) estimating the blur kernel from the LR image and 2) combining the estimated blur kernel and the LR image to reconstruct the SR image. However, the independence between the training of the blur kernel estimation model and the SR model leads to incompatibility issues. To address this issue, DANv1 <cit.> proposed an alternating optimization strategy between the blur kernel estimation and SR models, which makes the two models form a closed loop that can be trained in an end-to-end manner. Although DANv1 has achieved impressive performance in blind SR tasks, NLCUNet <cit.> has made significant advancements by incorporating the Non-Local Sparse Attention (NLSA) <cit.> module. It screens global features in the feature map by calculating self-similarities, enabling implicit estimation of the blur kernel. Therefore, the self-similarity technique has received the widest application in blind SR tasks. However, blind SR models that incorporate the calculation of self-similarity suffer from high computational complexity, making them unsuitable for resource-constrained devices. It is worth noting that the commonly used module for calculating self-similarity is Non-Local Attention (NLA) <cit.>.

In this paper, we go deep into the execution process of NLA. We find that during the self-similarity calculation, a matrix multiplication operation between the high-dimensional attention map and the Value is required, leading to high computational complexity. The attention map is obtained by multiplying the Query matrix with the Key matrix. Then the softmax function is further performed, which hinders the separation of Query and Key. Based on this finding, we propose a second-order Taylor expansion approximation (STEA) to significantly reduce the computational complexity. Compared to self-similarity calculations, STEA loses the ability to filter global features. Hence, we design a multi-scale large field reception (MLFR) to compensate for the performance loss caused by STEA. Finally, we apply our two core designs to laboratory and real-world scenarios by constructing LabNet and RealNet, respectively. Extensive experiments validate the effectiveness of the proposed method. The major contributions of this paper are summarized as follows.

    
  * We propose the second-order Taylor expansion approximation (STEA) strategy, which significantly reduces the computational complexity in self-similarity-based blind SR tasks.
    
  * We design a multi-scale large field reception (MLFR) to compensate for the performance loss caused by STEA, showcasing competitive performance over the state-of-the-art.
    
  * We construct the LabNet and the RealNet by integrating our two core designs (STEA and MLFR) to respectively verify the effectiveness of our designs in the laboratory and real-world scenarios. Extensive experimental results demonstrate that our LabNet achieves state-of-the-art performance while maintaining linear computational complexity, and our RealNet effectively handles various real-world degradations and achieves superior performance in diverse blind SR scenarios.






The structure of this paper is organized as follows: Section II provides a comprehensive overview of the proposed methodology. Specifically, Section II-A presents a detailed explanation of the intricate aspects of STEA. Section II-B elucidates the specific details of MLFR. Section II-C outlines the implementation specifics of LabNet, while Section II-D delves into the implementation details of RealNet. Subsequently, Section III encompasses the dataset and evaluation metrics (Section III-A), implementation details of LabNet (Section III-B), implementation details of RealNet (Section III-C), comparison with state-of-the-art techniques (Section III-D), ablation studies and analysis of LabNet (Section III-E), ablation study and analysis of RealNet (Section III-F) and adjustable parameter details in RealNet (Section III-G). Section IV concludes the paper.




§ METHODOLOGY

In this section, we provide a detailed explanation of the STEA and MLFR algorithms, as well as their deployment to the LabNet (testing for laboratory settings) and RealNet (testing for real-world scenarios).











 §.§ Second-Order Taylor Expansion Approximation (STEA)

The self-similarity has been widely applied to the blind SR task, which is commonly implemented by a Non-Local Attention (NLA) <cit.>. However, we found that the NLA brings extremely high computational complexity. This mainly arises from the matrix multiplication between the high-dimensional Attention Map and the V matrix, as highlighted by the red dashed box of Fig. <ref>. Without loss of generality, Q, K, V ∈ℛ^N× d denote query, key, and value matrix, respectively. Then, we know the computational complexity of NLA is 𝒪(N^2× d), where N and d represent the dimension of the feature map and the number of channels, respectively. In this part, we propose an STEA algorithm to significantly reduce the computational complexity of NLA. To this end, we first analyze the execution process of the NLA module. As shown in Fig. <ref>, the Attention Map is derived from the matrix multiplication of Q and K^T. Due to the limitation imposed by the softmax operation, the matrix multiplication between Q and K^T cannot be separated, resulting in a 𝒪(N^2) complexity. To address this challenge, we propose a Second-Order Taylor Expansion to approximate the NLA module. Here, we mathematically rewrite the execution process of the NLA as follows:

    Q = XW_q, K = XW_k, V = XW_v,

where W_q, W_k and W_v stand for the parameters of three 1× 1 convolutions, respectively. Q, K and V represent the Query, Key and Value, respectively. X denotes the input feature map. Next, we perform the matrix multiplication between Q and K^T, followed by the softmax operation, to obtain the Attention Map:

    AttMap = SoftMax(QK^T),

where AttMap represents the Attention Map, and SoftMax(·) denotes the softmax function. By substituting Eq. (<ref>) to Eq. (<ref>), we have

    AttMap = SoftMax(XW_qW_k^TX^T).

We denote W_qk = W_qW_k^T, resulting in the below expression:

    AttMap = SoftMax(XW_qkX^T).

In the original NLA module, the multiplication between Q and K^T followed by the softmax operation gives rise to softmax attention. Although softmax attention can effectively enhance the performance of the SR network by making the matrix full-rank, it introduces high computational complexity since such a high-dimensional Attention Map (AttMap) must be multiplied with V in the subsequent computation. For that, we propose to simplify the softmax function as an exponential form:

    AttMap = SoftMax(QK^T) = 1/k× e^QK^T = 1/k× e^XW_qkX^T,

where k is a learnable parameter used to approximate the denominator of softmax. By now, we can see that the operation between QK^T and V is still nonlinear due to the presence of an exponential function, which disables us from separating Q and K from QK^T. In subsequent multiplication by V, the basic operation of the matrix cannot be flexibly used, failing to reduce the high computational complexity. For this purpose, we exploit a second-order Taylor expansion as a substitute for the exponential function, achieving a good trade-off between



efficiency and accuracy. 

We know that in principle, the more remainder terms kept in the Taylor expansion of the exponential function, the more the expansion approximates the raw function. According to our empirical results and considering a good trade-off between accuracy and complexity, we adopt the second-order Taylor expansion on Eq. (<ref>) as follows:

    AttMap = 1/k× e^XW_qkX^T
     = 1/k× (I_n + XW_qkX^T + (XW_qkX^T)^2/2! + O(XW_qkX^T)),

where I_n stands for identity matrix and O(·) stands for the high-order remainder of the Taylor expansion. By ignoring the constant and remainder terms of Eq.(<ref>), we achieve:

    AttMap= 1/k× (I_n + XW_qkX^T + 
     XW_qkX^TXW_qkX^T).

Then, we multiply AttMap by V and refer to Eq.(<ref>) to obtain:

    Output = AttMap× V = 1/k× (XW_v + 
     XW_qkX^TXW_v + XW_qkX^TXW_qkX^TXW_v).

Eq. (<ref>) clearly indicates that the constraint of the softmax function has been removed, and therefore we can apply the associative property of the matrix operation in Eq. (<ref>), which allows us to calculate X^TX first. Considering that X^TX is a symmetric matrix, we can transform X^TX into a diagonalization expression as follows:  

    X^TX = ZBZ^T,

where B is a diagonal matrix with its diagonal elements being the eigenvalues of X^TX, and Z is an orthogonal matrix whose columns are the eigenvectors of X^TX corresponding to the eigenvalues. By substituting Eq. (<ref>) into Eq. (<ref>), we obtain:

    Output = 1/k× (XW_v + XW_qkZBZ^TW_v + 
     XW_qkZBZ^TW_qkZBZ^TW_v).

According to Eq. (<ref>), we can see that obtaining B through the matrix eigen-decomposition operation is computationally expensive. If we randomly initialize a learnable diagonal matrix B to avoid the high computational complexity from the eigen-decomposition operation of X^TX. Unfortunately, this strategy makes the matrix B independent of the input feature maps (X). Besides, in the random initialization, the matrix B is required to be a fixed size, making X have to be the fixed size. This disables the input LR image to be an arbitrary size. To reduce the complexity of the matrix eigen-decomposition and implement the SR task on the LR images with an arbitrary size, we design two separative eigenvalue extraction modules to respectively obtain two diagonal matrices B_1 and B_2, as shown in Fig. <ref>. Our design avoids randomly initializing a single B, and meanwhile enhances the representation ability of the model. Therefore, we can obtain a learning-based approximation expression as follows: 

    Output = 1/k× (XW_v + XW_qkZB_1Z^TW_v + 
     XW_qkZB_1Z^TW_qkZB_2Z^TW_v).

It is observed from Fig. <ref> that our design modules can dynamically extract global information from the matrices, which can well match the matrix eigen-decomposition where global information is exploited dynamically. 

Similarly, we adopt learnable parameters to substitute Z, solving the high computational complexity of the eigen-decomposition of X^TX. Next, we let W_1 = W_qkZ, W_2 = Z^TW_qkZ, and W_3 = Z^TW_v. Hence, we obtain:

    Output = 1/k× (XW_v + XW_1B_1W_3 + XW_1B_1W_2B_2W_3).

By now, we equate the softmax function to an exponential function and perform a Taylor expansion, making the operations of Q, K, and V more flexible. It is worth noting that the softmax function can transform the resulting matrix derived from Q and K^T into a full-rank matrix, while the matrix obtained by direct multiplication of Q and K^T is not a full-rank matrix, which significantly reduces the feature diversity, in turn, leads to a degradation of the model's performance <cit.>. Therefore, as done in <cit.>, we perform a depthwise convolution (DWC) operation on XW_v of Eq. (<ref>) to compensate for the performance degradation. Finally, we obtain:

    FinalOutput = 1/k× [DWC(XW_v) + 
     XW_1B_1W_3 + XW_1B_1W_2B_2W_3].

As shown in Fig. <ref>, we design the same network block to learn W_1, W_2, and W_3, which first reduces the number of channels to a quarter and then restores it to the original channel number. Meanwhile, an efficient Conv_1×1 operation is adopted to approximate W_v. 

As shown in Fig. <ref>, N and d are a multiplication of the width and height of a feature map and the number of channels. In general, N is far larger than d. According to Eq. (<ref>), our STEA strategy reduces the complexity of the original NLA from 𝒪(N^2× d) to 𝒪(N× d^2). The subsequent experimental results validate that our STEA is more efficient compared to the NLA module, however, exhibits a slight performance degradation due to the ignoring of remaining terms of Taylor extension and the use of the learning-based matrix. To compensate for such performance degradation, we design a Multi-Scale Large Field Reception (MLFR), which will be detailedly described in the following.





 §.§ Multi-Scale Large Field Reception (MLFR)

As mentioned in Section II-A, compared to the NLA module, our STEA achieves higher efficiency but sacrifices



the ability to extract global features from the feature maps. To this end, we design a Multi-Scale Large Field Reception (MLFR) strategy to obtain a large receptive field, as shown in Fig. <ref>. For the unfilled parts of the dilated convolution, we use normal convolution to fill in the dilation. Specifically, we process the input feature map X through three separate branches. In the first branch, the input is processed by a dilated convolution with an expansion rate of 6 and a depth-wise convolution with a kernel size of 7×7, respectively. The output results are concatenated along the channel dimension. Simultaneously, in the second branch, the input is processed by a dilated convolution with an expansion rate of 4 and a depth-wise convolution with a kernel size of 5×5, respectively. Similarly, the output results are concatenated along the channel dimension. In the last branch, the input is processed by a dilated convolution with an expansion rate of 2 and a depth-wise convolution with a kernel size of 3×3, respectively. Again, the output results are concatenated along the channel dimension. We represent the above execution process using the following equation: 

    Output_(i,j)=Concat(DDWC(X)_dr=i, 
    DWC(X)_ks=j)_i=2,4,6
    j=3,5,7,

where Concat(·,·) stands for concatenate operation, DDWC(·) stands for dilation convolution operation on the input, dr is the dilation rate, and DWC(·) stands for depth-wise convolution operation on the input, ks is the convolution kernel size. Furthermore, each value of i corresponds uniquely to one value of j. Subsequently, we employ 1×1 convolutions to perform channel-wise feature fusion on the outputs of the three branches, effectively reducing the dimension of the channels:

    Fusion_(i,j) = Conv_1×1(Output_(i,j))_i=2,4,6
    j=3,5,7,

where Fusion_(i,j) represents the output obtained after channel-wise feature fusion of the input, and Conv_1×1(·) denotes the 1×1 operation performed on the input. Similarly, each value of i corresponds uniquely to one value of j. In addition, the input X undergoes a 1×1 bypass convolution in addition to the aforementioned three branches:

    Bypass = Conv_1×1(X).

The purpose of the 1×1 convolution in Eq. (<ref>) is to perform subtle adjustments to the channels of the input X. Finally, we concatenate the outputs of the three branches with the output of the bypass 1×1 convolution along the channel dimension and perform channel-wise feature fusion using a 1×1 convolution to reduce the channel dimension:

    FinalOutput=Conv_1×1(Concat(Fusion_(i,j), 
     Bypass)_i=2,4,6
    j=3,5,7).

In summary, we utilize multiple dilated convolutions with different scales to extract features with various receptive fields. For the unpadded regions in dilated convolutions, we employ depth-wise convolutions to fill in the dilation. Additionally, we utilize 1×1 convolutions to perform subtle adjustments on the input as bypass convolutions. In the final step of feature fusion, we concatenate all the branches along the channel dimension and then employ a 1×1 convolution for channel-wise feature fusion, resulting in the final output as shown in Eq. (<ref>).







 §.§ Application to Laboratory Scenario (LabNet)

To show the effectiveness of our STEA and MLFR designs in a laboratory scenario, we construct a novel network termed LabNet for testing. As shown in Fig. <ref>, our LabNet is an end-to-end blind SR framework composed of three parts: shallow feature extraction, deep feature extraction, and HR image reconstruction. For the shallow feature extraction part, we employ a 3×3 convolution to extract shallow features from the input LR image. Subsequently, the extracted shallow features are fed into the UNet for deep feature extraction. Each LSTEA module in the UNet consists of multiple LSTEA blocks. When the input features pass through an LSTEA block, they undergo a channel-wise LayerNorm operation, followed by two branches. One branch utilizes multiple depth-wise convolutions for local feature extraction, while the other branch employs the STEA and MLFR modules for large-scale feature extraction. The channel features are then fused using a 1×1 convolution and filtered using the channel attention module. Subsequently, the features pass through the GDFN <cit.> module, which performs a nonlinear mapping operation on the input to enhance network performance. Finally, for the HR image reconstruction part, the channel dimension is adjusted using a 3×3 convolution. The H and W dimensions are then expanded to the target size using a pixel shuffle operation, and a final refinement is performed using a 3×3 convolution to obtain the ultimate output results.



 §.§ Application to Real-world Scenario (RealNet)

To show the effectiveness of our STEA and MLFR designs in real-world scenarios, we construct a novel network termed RealNet for testing. As illustrated in Fig. <ref>, our RealNet comprises three key components: shallow feature extraction, deep feature extraction, and HR image reconstruction. Firstly, we perform shallow feature extraction on the input LR image using two separate 3×3 convolutions. Subsequently, the extracted features are separately subjected to deep feature extraction in the denoising branch and the deblurring branch. Both the Denoising and Deblurring modules are composed of multiple LSTEA blocks. The features extracted from the Denoising module and the Deblurring module are then fused in proportion using Adapter modules. Finally, we employ a Fusion module for HR image reconstruction.

Adapter Module. For images contaminated by noise, there are numerous detailed features within the image that can be learned by the deblurring branch. Similarly, for blurry images, there are many low-frequency features within the image that can be learned by the denoising branch. Therefore, as depicted in Fig. <ref>, we propose the Adapter module. The parameters α and β control the proportions of denoising and deblurring, respectively. Analyzing the denoising branch, we take the features extracted by the deblurring branch in the previous step, apply a 3×3 convolution, and perform a channel-wise concatenation with the features extracted by the denoising branch. Subsequently, a 1×1 convolution is employed for channel-wise feature fusion, followed by a channel attention module for channel-wise feature selection. Similarly, a similar processing approach is utilized for the deblurring branch.

Fusion Module. To adjust the proportions of the outputs from the final denoising module and deblurring module, we also utilize two adjustable parameters (α and β). After adjusting the two sets of output feature maps, we perform a channel-wise concatenation and then apply a 1×1 convolution for channel information fusion and adjustment of channel dimensions. Finally, the output feature maps are resized to match the target size using a pixel shuffle operation and fine-tuned with a 3×3 convolution.

RealNet-GAN. To achieve better visual results, we employ a GAN training strategy for the proposed RealNet. Specifically, we use RealNet as the generator and utilize a U-Net as the discriminator <cit.>. The training process consists of two parts. Firstly, we train a PSNR-oriented model using the ℓ_1 loss, which is then used as the generator in the GAN training strategy. We then train a GAN-oriented model using a combination of ℓ_1 loss <cit.>, perceptual loss <cit.>, and adversarial loss <cit.>. The overall loss function of our network is

    ℒ_entire = ℒ_1 + ℒ_perc + 0.1 ×ℒ_adv,

where ℒ_1, ℒ_perc, and ℒ_adv represent the ℓ_1 loss, perceptual loss, and adversarial loss, respectively. According to our empirical study, their weights are set to be 1, 1, and 0.1, respectively. 
We utilize the {conv1,..., conv5} feature maps before activation in the pre-trained VGG19 <cit.> model (with weights {0.1, 0.1, 1, 1, 1}) as the perceptual loss.







§ EXPERIMENTS





 §.§ Datasets and Evaluation Metrics

We train all our proposed SR models using the DF2K dataset (DIV2K <cit.> + Flickr2K <cit.>). There are 3450 2K-resolution images in the dataset. For the quantitative evaluation, we conducted tests on five standard super-resolution image benchmark datasets: Set5 <cit.>, Set14 <cit.>, BSD100 <cit.>, Urban100 <cit.>, and Manga109 <cit.>. To conduct a qualitative evaluation, we compared the visual results of our approach with previous methods on the RealWorld38 <cit.> dataset. The quantitative performance metrics we use for comparison are the Peak Signal-to-Noise Ratio (PSNR) <cit.> and Structural Similarity (SSIM) <cit.>.





































 §.§ Implementation Details of LabNet

Our LabNet is built on the foundation of the LSTEA module, forming a U-Net. Each LSTEA module consists of multiple LSTEA blocks, as shown in Fig. <ref>, the number of LSTEA blocks in each LSTEA module from left to right is: {6, 8, 10, 6, 8, 8}. During the training process, we set the size of the input image to 64 × 64, and we use the Mean Absolute Error (MAE) as the loss function. The total iterations are 1.2×10^6. The initial learning rate is set to 4×10^-4 and is halved every 3×10^5 iterations. We use Adam <cit.> as the optimizer with β_1 = 0.9 and β_2 = 0.99. All models are trained on Tesla T4 GPUs.

Data Preprocessing of LabNet. For the training dataset, a blur operation with a blur kernel size of 21 × 21 is applied. For different scaling factors (×2, ×3, and ×4), the variance of the blur kernel is uniformly sampled during the training process from the ranges [0.2, 2.0], [0.2, 3.0], and [0.2, 4.0], respectively. For the test dataset, we uniformly select 8 blur kernels from the ranges [0.80, 1.60], [1.35, 2.40], and [1.8, 3.2] for different scaling factors (×2, ×3, and ×4), respectively.



 §.§ Implementation Details of RealNet

Our RealNet is constructed by two branches: denoising and deblurring. As illustrated in Fig. <ref>, both the denoising branch and the deblurring branch consist of 4 modules, each composed of 5 LSTEA modules. The entire network is trained for 30 epochs. The initial learning rate is set to 1× 10^-3 and is halved every 15 epochs. We use Adam as the optimizer with β_1 = 0.9 and β_2 = 0.99. All models are trained on Tesla T4 GPUs.

Data Preprocessing of RealNet. As shown in Fig. <ref>, we first independently and in parallel apply second-order noise and second-order blur processing to the HR image, resulting in two corresponding LR images. Then, these two LR images are passed through the denoising and deblurring branches of RealNet, respectively. Finally, a fusion operation is performed to obtain the restored SR image.






 §.§ Comparisons with State-of-the-art

Quantitative Evaluations in Lab Scenarios. From Table <ref>, it can be observed that our LabNet significantly outperforms other state-of-the-art deep SR models on nearly all benchmarks and scaling factors. For instance, compared to NLCUNet <cit.> with a scale factor of ×2, our LabNet achieves performance improvements of 0.03 dB, 0.05 dB, 0.11 dB, and 0.02 dB on the Set14, BSD100, Urban100, and Manga109 datasets, respectively. Similarly, compared to NLCUNet with a scale factor of ×4, our LabNet achieves performance improvements of 0.04 dB, 0.02 dB, and 0.11 dB on the Set14, BSD100, and Manga109 datasets, respectively. Our LabNet consistently outperforms DCLS <cit.> across all scale factors. Additionally, as shown in Table <ref>, when the input image size is (1,3,64,64) and the scaling factor is ×4, our LabNet achieves an inference time of 1.18 seconds on a GPU, which is 1.05 seconds faster than NLCUNet but slightly slower than DCLS. Furthermore, our LabNet has significantly fewer GFLOPs compared to NLCUNet and is comparable to DCLS. In summary, our LabNet not only achieves comparable performance to NLCUNet but also exhibits higher efficiency. This can be attributed to the utilization of LSTEA in our LabNet, which combines STEA and MLFR to capture multi-scale features with a large receptive field while maintaining linear complexity.


Qualitative Evaluations in Lab Scenarios. As shown in Fig. <ref>, by comparing the reconstruction result of the 'zebra' in Fig. <ref>, we can observe that the black and white stripes on the legs of the zebra generated by our LabNet are close to the GT image. Furthermore, by comparing the reconstruction result of 'YamatoNoHane', we can observe that our LabNet performs better in recovering the textual content in the image compared to other methods.


Quantitative Evaluation in Real-World Scenarios. As commonly observed, GAN-based SR methods often generate lower values in terms of PSNR and SSIM metrics while producing visually captivating results. As shown in Table <ref>, Real-ESRGAN and BSRGAN perform significantly lower than non-GAN-based SR methods on the five benchmark test datasets. However, Fig. <ref> reveals the impressive visual outcomes achieved by Real-ESRGAN and BSRGAN. Additionally, due to the absence of corresponding GT images for the degraded images in the RealWorld38 dataset, we only conducted visual comparisons between our RealNet and other blind SR methods.


Qualitative Evaluation in Real-World Scenarios. The experimental results are presented in Fig. <ref>. We evaluated the generalization capability of our model using the RealWorld38 dataset, which consists of real-world images. Our RealNet outperforms previous methods for reconstructing LR images in real-world scenarios. Among all the methods, the non-deep learning bicubic interpolation method performs the worst, followed by DANv2, DCLS, and NLCUNet. These methods only employ Gaussian blur and downsampling during training, resulting in visually unsatisfactory outcomes. BSRGAN <cit.> and Real-ESRGAN achieve favorable visual results due to their training on datasets with complex degradations. However, upon comparing the 'OST_009' image in Fig. <ref>, we observe that the result obtained by BSRGAN exhibits a significant color deviation from the LR input, while the result produced by Real-ESRGAN evokes a strong sense of discordance. In contrast, our RealNet reconstruction result closely resembles the color of the input LR image and demonstrates improved overall coherence.





 §.§ Ablation Study and Analysis of LabNet



  §.§.§ Impact of STEA

Table <ref> showcases the effect of employing first-order Taylor expansion approximation (FTEA) and second-order Taylor expansion approximation (STEA) on the performance of blind SR tasks.
By comparing the PSNR and SSIM values of the first and second rows in Table <ref>, we observed an improvement in performance across all test datasets when DWC(·) was applied to the Value. Furthermore, comparing the PSNR and SSIM values of the second and third rows in Table <ref>, we found that utilizing STEA for the exponential function outperformed FTEA across all test datasets. Particularly on the Urban100 dataset, the PSNR was higher by 0.11 dB when using a STEA compared to a FTEA. Furthermore, we found that when the order of Taylor expansion is greater than 2, the performance does not show a significant improvement. 

Efficiency Analysis. As shown in Table <ref>, when applying DWC(·) to Value, there is a slight increase in the number of parameters and a slight rise in MFLOPs. By comparing rows 2 to 4 in Table <ref>, as the order of Taylor expansion increases, the third-order Taylor expansion approximation (TTEA) parameter count and MFLOPs are greater than STEA, which are greater than FTEA. Ultimately, we used STEA to build our LabNet and RealNet.



  §.§.§ Impact of MLFR

Table <ref> highlights the influence of incorporating multi-scale dilated convolutions in MLFR on blind SR performance. By comparing the PSNR and SSIM values of the first to third rows in Table <ref>, we found that in MLFR, when utilizing dilated convolutions with dilations of 2, 4, and 6 simultaneously, the performance was the best across all test datasets. Particularly on the Urban100 dataset, the PSNR was 0.12 dB higher when using dilated convolutions with dilations of 2, 4, and 6 compared to dilated convolutions with dilations of 4 and 6. 

Efficiency Analysis. As shown in Table <ref>, in MLFR, the maximum number of parameters and MFLOPs are achieved when utilizing dilated convolutions with Dilated = 6, 4, and 2 simultaneously.




  §.§.§ Impact of STEA and MLFR

As shown in Table <ref>, LabNet achieves the best performance on all five benchmarks when both STEA and MLFR are used simultaneously.



 §.§ Ablation Study and Analysis of RealNet

The experimental results from Table <ref> highlight the following insights: Our RealNet without NLA has the fastest inference speed, processing a 64 × 64 input image in just 0.40 seconds. In contrast, Our RealNet with NLA and NLSA have much higher computational costs, with inference times of 4.21 seconds and 5.15 seconds, respectively. Our RealNet with the STEA, achieves high efficiency by processing an input image in only 0.48 seconds. Our RealNet with the STEA and the MLFR strikes a favorable balance between efficiency and visual result, processing an input image in 0.53 seconds.



 §.§ Details of adjustable parameters in RealNet

Adjusting a single parameter: Users can adjust the values of the denoising and deblurring parameters in RealNet to achieve the desired effects. In the visualization shown in Fig. <ref>, Increasing α_1 slightly enhances image details, but excessive increase may lead to distorted edges. Gradually increasing α_3 improves the distinction between objects and the background, but setting it too high can negatively impact the image. α_2 progressively enhances image visualization as it increases, but setting it to extremely large values can damage the image. Careful adjustment of α_4 is necessary to prevent color distortion, as it is related to the deep layer. It is advisable to avoid setting β_1 to β_4 to excessively large values to prevent color distortion in the resulting image. 

Joint modification of multiple parameters: Users can still fine-tune the denoising and deblurring ratios (i.e., α_i and β_i) according to their preferences, even after the training process. In the visualization shown in Fig. <ref>, the second and third rows display the restored results using different ratios for denoising and deblurring. This demonstrates that RealNet is capable of restoring rich texture information from LR images, catering to different user preferences.



§ CONCLUSION

In this paper, we presented a second-order Taylor expansion approximation (STEA) to handle the high computational complexity of self-similarity techniques. Then, we designed a multi-scale large-field reception (MLFR) to compensate for the super-resolution (SR) performance loss caused by STEA. To confirm the superiority of our designed STEA and MLFR, we respectively constructed the LabNet for testing in laboratory scenarios and the RealNet for testing in real-world scenarios. Extensive experimental results demonstrated the effectiveness of our method in blind SR tasks. The ablation study further validates their contribution to the overall framework.
IEEEbib



























 Paper Title: From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification.








Journal of  Class Files, Vol. 14, No. 8, December 2023
Shell et al.: A Sample Article Using IEEEtran.cls for IEEE Journals




From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification
    

        
        Fanzhi Jiang, 
        Su Yang1, 
        Mark W. Jones, 
        Liumei Zhang1
        
        
        1Corresponding author
        
This paper was produced by the IEEE Publication Technology Group. They are in Piscataway, NJ.
Manuscript received April 20, 2024; revised August 16, 2024.
        
    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 

  Text-based person re-identification (Re-ID) is a challenging topic in the field of complex multimodal analysis, its ultimate aim is to recognize specific pedestrians by scrutinizing attributes/natural language descriptions. Despite the wide range of applicable areas such as security surveillance, video retrieval, person tracking, and social media analytics, there is a notable absence of comprehensive reviews dedicated to summarizing the text-based person Re-ID from a technical perspective. To address this gap, we propose to introduce a taxonomy spanning Evaluation, Strategy, Architecture, and Optimization dimensions, providing a comprehensive survey of the text-based person Re-ID task. We start by laying the groundwork for text-based person Re-ID, elucidating fundamental concepts related to attribute/natural language-based identification. Then a thorough examination of existing benchmark datasets and metrics is presented. Subsequently, we further delve into prevalent feature extraction strategies employed in text-based person Re-ID research, followed by a concise summary of common network architectures within the domain. Prevalent loss functions utilized for model optimization and modality alignment in text-based person Re-ID are also scrutinized. To conclude, we offer a concise summary of our findings, pinpointing challenges in text-based person Re-ID. In response to these challenges, we outline potential avenues for future open-set text-based person Re-ID and present a baseline architecture for text-based pedestrian image generation-guided re-identification (TBPGR).



Person Re-Identification, Text, Natural Language, Attributes, Diffusion Model.





§ INTRODUCTION

THE demands in public safety and the subsequently installed surveillance networks make the manual tracking and identifying individuals increasingly challenging. 
Automatic person re-identification (Re-ID) has emerged as a popular research area in computer vision to address this issue. Also known as person search  <cit.>, person Re-ID specifically refers to recognizing and tracking specific individuals using non-overlapping images and videos captured by cameras, determining whether a specific query person appears persistently or momentarily across different times and locations  <cit.>. 
It finds important applications in scenarios like searching for characters in movies, locating missing children, and tracking criminals  <cit.>. 
The source data collection for these applications is usually from CCTV footage on the street. Given the constraints of the equipment and the environment, a number of challenges arise. 
These challenges include lighting variations, occlusions, background changes, pose variations, and differences in camera resolutions. 
A typical person re-identification system involves conducting image queries and searching for corresponding individuals in a gallery of images or a video pool. 
It involves extracting features directly from pedestrian appearance images and performing direct matching, then ranking against existing appearance images in the gallery. 
These scenarios assume that visual examples of individual identities are always available as queries. However, some special cases where the visualization samples of personal identity are not available, we perform retrieval based on textual descriptions only, which is called text-based person re-identification, as shown in Figure <ref>.






 §.§ Attribute-based Person Re-ID

Text-based person re-identification is a special form of person re-identification that, instead of using image data, relies on descriptions to reflect a person's appearance  <cit.>. In the task of text-based person Re-ID, a natural language description and an image are provided as input, and the output is the identification of the person matching the description  <cit.>. There has been a desire to perform high-level semantic person search using free-form natural language descriptions.  However, early research initially started with the attribute-based person re-identification <cit.>. 
Attribute-based methods (Pedestrian Attributes Recognition, PAR) stem from the high correlation between attributes and pedestrian images. Pedestrian attributes such as gender, age, clothing type, clothing color, etc. are arranged and combined to make the pedestrian identity appear distinguishable <cit.>.
Extracting attribute information from pedestrian images can be achieved by using pre-trained classifiers or attribute detectors. These classifiers or detectors can recognize and locate different attributes(body part) in the images. These extracted pedestrian attributes information needs to be encoded into a feature vectors pattern matching. This often involves transforming discrete attributes into continuous feature representations, common methods include one-hot encoding and embedding encoding <cit.>. During the recognition process, matching is conducted by comparing the attribute feature vectors of query images and gallery images. Common matching methods include computing distances or similarities between feature vectors (such as Euclidean distance, cosine similarity, etc.). Finally, based on the results of attribute matching, the final recognition result is determined through fusion of matching scores from different attributes or by adopting decision strategies.
The attribute-based person Re-ID approach improves recognition accuracy and retrieval interpretability. This is because these attributes can be considered as high-level semantic information that is robust to viewpoint changes and different viewing conditions <cit.>.

Previous work on attribute-based person retrieval has attempted to reduce modality gaps by aligning each person category and its corresponding images in a joint embedding space through modal adversarial training <cit.>, or by enhancing the expressive power of embedding vectors for person categories and images hierarchically <cit.>. While these pioneering studies reveal important but less explored methods for person retrieval, there remains significant room for further improvement. Firstly, due to their adversarial learning strategy, they exhibit instability and high computational cost during training <cit.>. For other, they incur high inference costs due to the need for additional networks to match high-dimensional embedding vectors <cit.>. More importantly, these methods treat person categories as independent labels of person images and overlook their relationships, such as how many attributes differ between them, despite the potential rich supervisory signals for learning better representations of person categories and images. Dong et al.  <cit.> address this issue by capturing rich information of two modalities through hierarchical embedding. However, their model entails high computational complexity as it computes high-dimensional embeddings and deploys additional networks for matching. Yin et al.  <cit.> and Cai et al.  <cit.> learn a joint embedding space where person categories and images are directly matched. To bridge the modality gap, their embedding space is trained in a modal adversarial manner; however, due to the nature of min-max optimization, this often leads to unstable and slow convergence. Furthermore, all these methods suffer from a limitation where person categories are treated as separate labels and the important relationships among them are overlooked. In the joint embedding space of the two modalities, the loss pulls images of the same person category closer to achieve modal alignment. <cit.> introduces a novel loss, Adaptive Semantic Margin Regularizer (ASMR), for learning cross-modal embeddings in the context of attribute-based person retrieval.





 §.§ Natural Language-based Person Re-ID

Attribute-based person retrieval <cit.> provides a method for person re-identification based on free-form descriptions. 
However, there is an awareness that the leverage of attributes to describe a person's appearance could be limited in some circumstances. For instance, the PETA dataset <cit.> defines 61 binary and 4 multi-class person attributes, along with hundreds of phrases used to describe a person's appearance.  In addition, even with a constrained set of attributes, labeling them for large-scale person image datasets is challenging and expensive. Using free-form natural language (NL) descriptions may offer more flexibility and robustness compared to these predefined attributes. 

Attribute-based person retrieval also has limitations in terms of the results robustness. For instance, shuffling and flipping some attributes while using a basic fact set for retrieval can lead to significant performance degradation of the results. Compared to manually assigned sets of attributes, textual descriptions can be seen as an unstructured form of annotation. In general, most attributes fall into the category of noun phrases, and natural language includes higher-level semantic words and complex events than the attributes. In principle, natural language processing (NLP) techniques can facilitate all attributes with any possible words and lengths, capturing unique details through natural language descriptions. Additionally, descriptions can include indicators of certainty or fuzziness to retain as much information as possible, potentially allow sets of multi-class attribute assignments. Both attribute-based and NLP-based person re-identification are referred to as text-based person re-identification in this paper.

Li et al. <cit.> introduced the first dataset for NL-based person Re-ID, evaluated and compared various possible models, and proposed an RNN with a gated neural attention mechanism. Jing et al.  <cit.> proposed to focus on mining fine-grained local parts and performing fine-grained visual-textual matching. These methods can learn distinctive part representations and achieve 81.23% accuracy, but they are troubled by the problem of ambiguous embeddings. Image regions and words are correlated at different semantic granularities <cit.>. The discriminative parts of a person usually appear with different granularities, which was overlooked in the previous methods and ambiguated embeddings. To alleviate this issue, words (based on NL) as guidance have been integrated into the related image regions. However, irrelevant words may mislead the model and even exacerbate the ambiguous embeddings problem. 

Cross-domain embeddings technique is one of the main trends in the Re-ID community. Work proposed in <cit.> added supplementary annotations to extract meaningful image regions independently, these regions, however, are not always available in the real-world scenes. Work in <cit.> proposed to learn cross-modal embeddings to improve matching results using a shared attention mechanism. <cit.> designed a cross-modal projection matching loss and a cross-modal projection classification loss for learning discriminative image-text embeddings. <cit.> proposed a matrix factorization-based dictionary learning algorithm to eliminate the impact of style and pedestrian pose information on cross-domain Re-ID. <cit.> proposed a patch-word matching model and designed an adaptive threshold mechanism in the model, it attempted to establish global and local image-language associations to enhance the semantic consistency between local visual and language features. 

Deep adversarial graph attention convolutional network was also proposed to address the problem of text-based person re-identification<cit.>. Niu et al. <cit.>proposed a Multi-level Image-Text Alignment (MIA) model, which introduced cross-modal adaptive attention in three different granularities, integrating these three similarities to comprehensively judge the affinity of image-sentence pairs and better distinguish interested pedestrians. Zhu et al. <cit.>achieved an environment-person separation (DSSL) method under the mutual exclusion constraint, and also annotated another dataset for person search using language called Real Scene Text-based Person Re-ID (RSTPReID). Wu et al. <cit.> designed a twin task, image coloring and text completion, and explicitly built fine-grained cross-modal associations bidirectionally based on color reasoning, their LapsCore method achieved 63.4% rank@1 retrieval accuracy on the CUHK-PEDES dataset. <cit.> proposed a novel cross-modal matching method named Cross-modal Co-occurrent Attribute Alignment (C2A2), which can better handle noise and significantly improve retrieval performance in person search by language. Relying on matrix factorization to construct visual and textual attribute dictionaries and cross-modal alignment using denoising reconstruction features addresses noise generated by extraneous elements of pedestrians. Zheng et al. <cit.> made the first attempt to perform NL-based person search within unsegmented complete images.



 §.§ Person Re-Identification previous survey

In order to provide researchers with a comprehensive understanding of the current status and research directions in the domain of text-based person re-identification, we conducted an in-depth investigation of text-based person re-identification methodologies and synthesized recent research achievements. Prior to this effort, several researchers have conducted reviews in the field of person re-identification <cit.> and we summarize the primary contributions of these studies in the Table <ref>. 
Among them, certain surveys <cit.> introduced previous challenges in the field of person Re-ID. <cit.> summarized deep learning based pedestrian re-recognition methods. <cit.> investigated the available image/video-based person Re-ID datasets. <cit.> surveyed image-based or video-based pedestrian re-recognition systems. <cit.> proposed metric learning based classification method for pedestrian re-recognition. 
<cit.> presented a pedestrian re-recognition taxonomy based on representation learning. Huang et al.<cit.> provided a visible-infrared cross-modal categorization of pedestrian re-identification studies. Wang et al.<cit.> reviewed cross-modality based person Re-ID research from four aspects, including sketch, text, low resolution (LR) and infrared (IR), as Figure <ref> is introduced later.

However, these surveys exhibit areas for improvement, such as lacking a systematic categorization and analysis of text-based deep learning methods in person re-identification, and omitting a comprehensive discussion of text-based person re-identification. Therefore, in comparison to the aforementioned reviews, this paper delves into an in-depth analysis of recent research in text-based person re-identification from dimensions including datasets, strategies, architectures, and optimizations. We comprehensively review existing deep learning-based approaches and discuss their strengths and limitations.








§ EVALUATION

The evaluation of text-based person Re-ID tasks, typically involves publicly available benchmark datasets and common performance metrics. This provides a standardized testing platform, enabling a quantified and objective comparative assessment of the effectiveness of different experimental algorithms.



 §.§ Datasets

























  §.§.§ attribute-based Person Re-ID Datasets

Attribute-based Person Re-identification (Re-ID) often involves the utilization of semantic attributes that describe appearance characteristics of individuals, such as gender, age, clothing type, clothing texture, and clothing color <cit.>. However, these attributes frequently encompass low-level semantic attributes that can be directly associated with image regions, such as clothing type, clothing texture, and clothing color, as well as high-level semantic attributes that directly correspond to people, such as gender and age. By leveraging these attributes, the negative impact of variations in viewpoint and appearance on person re-identification can be mitigated, resulting in more reliable recognition and tracking performance <cit.>. To facilitate research in Attribute-based Person Re-ID, researchers have initiated the construction of several datasets specifically tailored for this purpose. Notable publicly available datasets in this domain include PETA <cit.>, UAV-Human <cit.>, RAP  <cit.>, RAP2.0 <cit.>, PA-100K <cit.>, Market1501-Attribute <cit.>, and DukeMTMC-Attribute <cit.>. 

These datasets typically consist of a substantial collection of images capturing individuals from surveillance cameras, each accompanied by attribute labels. The dataset creation process generally involves selecting a subset of images from existing Person Re-ID datasets. Subsequently, attribute labels are assigned to each image either manually or through automated annotation. Ultimately, the annotated images along with their corresponding attribute labels are organized into a unified dataset, which serves as a resource for researchers in the field, as shown in Table dataset.



  §.§.§ Natural Language-based Person Re-ID Datasets

Publicly available NL-based person re-identification datasets have significant practical implications for benchmarking, algorithm improvement, security and surveillance applications, privacy protection, and advancing multimodal research. Currently, there are three large publicly available datasets, CUHK-PEDES, RSTPReid, and ICFG-PEDES. The details of these datasets will be described. 



  
CUHK-PEDES
CUHK-PEDES is the first description dataset introduced by The Chinese University of Hong Kong <cit.>, designed for training and evaluating person description and image retrieval tasks in computer vision. The dataset was collected by aggregating person images from five existing and well-known pedestrian re-identification evaluation datasets, namely CUHK03 <cit.>, Market-1501 <cit.>, SSM <cit.>, VIPER <cit.>, CUHK01 <cit.>, and subsequently annotated with descriptive text by crowdsourced workers from Amazon Mechanical Turk (AMT). It consists of over 40,206 images with 80,412 person descriptions, each containing at least 23 words, across 13,003 unique person identities. The training set includes 34,054 images, 11,003 unique persons, and 68,108 textual descriptions. The validation set contains 3,078 images, 1,000 unique persons, and 6,158 textual descriptions, while the test set comprises 3,074 images, 1,000 unique persons, and 6,156 textual descriptions. The individuals in the dataset were captured in street and public places in the Hong Kong area. CUHK-PEDES provides diverse person descriptions encompassing appearance, actions, postures, and interactions, with each person description linked to the corresponding image ID. The release of the CUHK-PEDES dataset promotes interdisciplinary research between computer vision and natural language processing, providing researchers in the fields of person description and image retrieval with the first benchmark dataset.



  
RSTPReid
The RSTPReid dataset is a pedestrian re-identification dataset introduced by Nanjing University of Science and Technology <cit.>, designed for training and evaluating pedestrian re-identification tasks in computer vision. To address the issue in the CUHK-PEDES dataset where each specific pedestrian is captured by the same camera under the same time-space conditions, which does not reflect real-world scenarios, the authors constructed the Real Scenarios Text-based Person Reidentification (RSTPReid) dataset based on MSMT17 <cit.>. The RSTPReid dataset contains 20,505 images of 4,101 individuals captured by 15 independent cameras from different viewpoints, lighting conditions, locations, and weather conditions. Each individual has 5 corresponding images taken by different cameras, and each image is accompanied by 2 text descriptions with no fewer than 23 words. The training set includes 3,701 identities, the validation set has 200 identities, and the test set also consists of 200 identities. The dataset includes a relatively small number of pedestrian identities but covers both indoor and outdoor scenarios. Each pedestrian appears in multiple images with different shooting angles and lighting conditions, making this dataset more challenging and realistic.



  
ICFG-PEDES
The Identity-Centric and Fine-Grained Person Description Dataset (ICFG-PEDES) is a person description dataset introduced by South China University of Technology <cit.>, which also serves as a new benchmark dataset for research in person description and image retrieval domains. Similar to CUHK-PEDES, this dataset contains a large number of person descriptions paired with corresponding image IDs. ICFG-PEDES consists of 54,522 pedestrian images of 4,102 distinct identities, all collected from the MSMT17 database <cit.>. The training set comprises 34,674 image-text pairs of 3,102 pedestrians, while the test set contains 19,848 image-text pairs of 1,000 pedestrians. Each image is associated with only one text description, with an average of 37.2 words per description. Compared to the CUHK-PEDES dataset, ICFG-PEDES emphasizes more on fine-grained person descriptions and reduces some irrelevant action and background information. Additionally, it addresses the issue of consistent backgrounds in the former dataset by emphasizing more on appearance variations. Consequently, the ICFG-PEDES dataset can be employed for more challenging image retrieval and person description tasks.



 §.§ Metrics




  §.§.§ Mean Average Precision(mAP)


For each query i, we define a precision P_i(j) as the proportion of correct matches in the top j matches, and then for each positive instance, we calculate the average of the proportions of all positive instances before this one, which is the Average Precision AP_i:


    AP_i = 1/M_i∑_j=1^M_i P_i(j) · I(rank_i = j).


Where M_i is the number of positive instances for query i, and ∑ is the summation over all positive instances. The symbol I(rank_i = j) usually denotes an indicator function. The indicator function takes the value of 1 under certain conditions, otherwise it takes the value of 0. Specifically:

    I(rank_i = j)
 indicates that the indicator function I(rank_i = j) takes the value of 1 when the first i query's first j result is a correct match, otherwise it takes the value of 0. In other words, it determines whether or not there is a correct match in the ranked position of j.
Then, mAP is the average of the Average Precision of all queries:


    mAP = 1/N∑_i=1^N AP_i.


Where N is the total number of queries, and ∑ is the summation over all queries.



  §.§.§ Rank-N Accuracy

For each query i, we define an indicator function I(rank_i ≤ n), which is 1 if the correct match is within the top n matches, and 0 otherwise. Then, Rank-n accuracy is the average of this function:


    Rank-n = 1/N∑_i=1^N I(rank_i ≤ n).


Where N is the total number of queries, and ∑ is the summation over all queries.







§ STRATEGY

In the task of text-based person re-identification, image data is typically captured through CCTV in open environments [45]. This data not only contains key descriptive features of individuals but also includes a significant amount of background noise. Early works <cit.> typically directly extract global image text features for simple cross-modal matching. However, due to the uniqueness of the person re-identification task, which differs from existing image-text retrieval <cit.>, the categories to be retrieved all belong to highly similar individuals. Identification based solely on distinguishing features such as clothing and posture increases the challenges of the text-based person re-identification task.



 §.§ Stripe Segmentation

Some work suggests that various parts of the human body are evenly arranged in images, and Stripe Segmentation can be employed as a guiding model to achieve multimodal local alignment supervision <cit.>. Stripe Segmentation is a commonly used technique in pedestrian re-identification, aiming to mitigate the impact of pose, occlusion, and lighting variations on re-identification performance within pedestrian images <cit.>, <cit.>. The fundamental idea is to partition pedestrian images into multiple vertical stripes, also referred to as person parts. Each stripe can be considered a subregion of the pedestrian image (head, upper body, lower body, foot). The advantage of this approach lies in dividing pedestrian images into multiple local regions, effectively capturing local features of pedestrians to enhance recognition accuracy <cit.>. For instance, local texture, color, and shape features can be extracted within each stripe, facilitating better differentiation between different parts of pedestrians. Stripes of different heights can correspond to different scales of pedestrian parts, providing richer scale information to address scale variation issues in pedestrian images. Ding et al. <cit.> proposed a Semantic Self-Alignment Network (SSAN) capable of automatically extracting part-level textual features for corresponding visual regions. They designed a multi-view non-local network to capture relationships between body parts, thereby establishing better correspondences between body parts and noun phrases. Li et al <cit.> proposed to vertically divide a character image into multiple regions using two different approaches, including overlapping slices and key-point-based slices. Niu et al. <cit.> also emphasized local-local alignment, employing Stripe Segmentation in the proposed Bidirectional Fine-Grained Matching (BFM) module to match visual human body parts with noun phrases. However, this strategy is fragile and sensitive to conditional changes. For example, the regions of image samples do not always contain a complete human body, and at times, the human head does not consistently appear in the first stripe, significantly impacting the robustness of existing methods.



 §.§ Multi-scale Fusion

Due to the small inter-class variance in both pedestrian images and descriptions, comprehensive information is required for pedestrian retrieval to coordinate visual and textual clues across all scales. Subsequent to the introduction of local feature extraction strategies other than stripe segmentation, the utilization of fused features is referred to as multi-scale feature fusion <cit.>, as illustrated in Figure  <ref>. Image Structure Graph Network (A-GANet) <cit.> employs residual modules for visual representation extraction and utilizes designed graph attention convolutional layers for structured high-level visual semantic feature extraction. Specifically, it captures relationship features of different pedestrian parts through graph convolutional networks. Wang et al. <cit.> proposed the Visual Text Attribute Alignment model (ViTAA), which learns to decompose the person's feature space into subspaces corresponding to attributes using an optically assisted attribute segmentation layer. On the image side, a segmentation layer is employed to divide pedestrian images into full body, head, upper clothes, Lower clothes, shoes, and bag parts. On the text side, corresponding phrases of different segmented parts are extracted. 

Ji et al. <cit.> introduced an Asymmetric Cross-Scale Alignment (ACSA) method. Global text representation and local phrase representation are employed for text extraction. The global representation is divided into four regions as local visual representations, namely Head, Upper body, Lower body, and foot. These features are used afterwards for feature concatenation and computation. The partition strategy does not involve additional computational costs but can better retain prominent body parts for fine-grained matching. Wang et al.<cit.> proposed a novel Multi-Granularity Embedding Learning (MGEL) model. It generates multi-granularity embeddings of partial human bodies from coarse to fine by revisiting person images at different spatial scales. Chen et al.<cit.> adopted a multi-branch representation in the learning path, enabling text-adaptive matching of corresponding visual local representations, thereby aligning text and visual local representations. Additionally, a multi-stage cross-modal matching strategy was proposed to eliminate modality gaps from low-level, local, and global features, gradually narrowing the feature gap between image and text domains. 

Niu et al.<cit.> presented a Multi-Granularity Image-Text Alignment (MIA) framework to address cross-modal fine-grained issues. It consists of three modules corresponding to three granularities. The Global Contrast (GC) module is used for global-global alignment. The Relation-Guided Global-Local Alignment (RGA) module filters global-local relationships. The Bidirectional Fine-Grained Matching (BFM) module aligns local to local based on trained fine-grained local components. Zheng et al. <cit.> proposed a Hierarchical Adaptive Matching model to learn subtle feature representations from three different granularities (i.e., word-level, phrase-level, and sentence-level) for fine-grained image-text retrieval tasks. Wang et al. <cit.> addressed the issue that existing methods typically learn similar mappings between local parts of images and text or embed entire images and text into a unified embedding space, resulting in a problem of local-global correlation. They designed a Divide and Merge Embedding (DME) learning framework for text-based person search. It models the relationship between local parts and global embedding, merging local details into the global embedding. Li et al.<cit.> constructed features for text and person image matching, different from other methods. The latter acquires global features by aggregating aligned local features, leading to misalignment between text and visual features. They proposed placing features with the same semantics in the same spatial position to construct the final features for person-text-image matching, achieving semantic consistency and interpretability of global features. 

Yan et al.<cit.> proposed an Implicit Local Alignment module, adaptively aggregating image and text features into a set of modality-shared semantic topic centers, implicitly learning local fine-grained correspondence between images and text without additional supervision information and complex cross-modal interaction. Additionally, global alignment is introduced as a supplement to the local perspective. Gao et al. <cit.> introduced a method called Non-Local Alignment on Full Scale (NAFS), capable of adaptively aligning image and text features at all scales. First, a novel ladder network structure is proposed to extract full-size image features with better locality. Secondly, a language model, Bidirectional Encoder Representations from Transformers (BERT), with local constraint attention is proposed to obtain description representations at different scales. Then, instead of aligning features separately at each scale, a novel context non-local attention mechanism is applied to simultaneously discover potential alignment at all scales. 

In summary, applying target detection or additional branch networks to detect significant regions and then extract local features for obtaining multi-scale features has certain accuracy advantages. However, due to the increased external network overhead, these methods often result in higher computational costs.



 §.§ Attention Mechanism

Cross-modal alignment is inherently challenging in achieving fine-grained matching between text and images. In addition to implicit alignment strategies achieved through the automatic fusion of multi-scale features, another key approach involves the use of explicit alignment strategies. For instance, employing attention mechanisms enhances the capability of feature extraction by capturing discriminative feature representations related to both language descriptions and visual appearances. This aids in intuitively understanding and controlling the alignment process. Attention models can establish correspondences between body parts and words <cit.>, offering the advantage of not relying on external cues for aligning feature embeddings. These attention strategies can be categorized into spatial attention, channel attention, mixed attention, non-local attention, and positional attention.

Liu et al. <cit.> proposed an attention-based deep neural network capable of capturing multiple attention features from low-level to semantic layers to learn comprehensive features for fine-grained pedestrian representation. Wang et al. <cit.> utilized a multi-head self-attention module to extract embeddings for different granularity portions of the text stream, employing adaptive filtering to remove interference and obtain fine-grained features. Lee et al. <cit.> introduced a Stacked Cross Attention Mechanism (SCAN) that compares attention information for text and image in two stages, determining the importance of image regions and words to achieve potential alignment between the two modalities. Spatial attention-based methods often focus solely on local discriminative features of pedestrians, neglecting the impact of feature diversity on pedestrian retrieval.

Li et al. <cit.> proposed a cubic attention convolutional neural network that combines spatial and channel attention to maximize complementary information from different scale attention features, addressing the challenges of cross-modal alignment in pedestrian re-identification. Wang et al. <cit.> designed a Feature Division Network (FDN) that embeds inputs into K locally guided semantic representations using self-attention, each representing different parts of a person. They then introduced a Correlation-based Subspace Projection (RSP) method to merge different local representations into a compact global embedding. Yan et al. <cit.> presented an Efficient Joint Information and Semantic Alignment Network (ISANet) for text-based person search. Specifically, they designed an image-specific information suppression module that suppresses image background and environmental factors through relationship-guided localization and channel attention filtering, effectively alleviating information inequality and achieving information alignment between text and image.

Considering the indistinguishability of features extracted from first-order attention (e.g., spatial and channel attention) in complex camera views and pose-changing scenarios, Farooq et al. <cit.> applied non-local attention directly after calculating interactions between text features to model long-term dependencies between different phrases in the text. Inspired by self-attention, Gao et al. <cit.> proposed a Context Non-Local Attention to enable cross-modal features to align with each other in a coarse-to-fine manner based on their semantics, rather than relying solely on predefined and fixed rules (e.g., local-local, global-global). Wang et al. <cit.> introduced a Part-based Multi-scale Attention Network (PMAN), consisting of a dual-path feature extraction framework with attention-based branches for a multi-scale cross-modal matching module. This module extracts visual semantic features from different scales and matches them with text features.

Addressing concerns that utilizing soft attention mechanisms to infer semantic alignment between image regions and corresponding words in a sentence may lead to the fusion of unrelated multimodal features and result in redundant matching, Wang et al. <cit.> proposed an IMG-Net model. This model combines intra-modal self-attention and cross-modal hard region attention with a fine-grained model to extract multi-granularity semantic information. Zheng et al. <cit.> introduced a Hierarchical Gumbel Hard Attention Module, using the Gumbel top-k reparameterization algorithm as a low-variance, unbiased gradient estimator to select strongly semantically related regions for all regions of the image and corresponding words or phrases, enhancing precise alignment and similarity calculation results.

While these techniques often bring about noticeably superior retrieval performance, they frequently come at the cost of efficiency. Specifically, for M galleries and N queries, their complexity is O(M+N). In contrast, the complexity of attention-based cross-modal approaches rises to O(MN) due to pairwise inputs. To address this issue, Yang et al. <cit.> initially employ cross-modal non-attention features to quickly identify candidates and then deploy attention-based modules to refine the final ranking scores.



 §.§ External Auxiliary

For Text-based Person Re-Identification (Re-ID) tasks, several auxiliary strategies such as image segmentation, random mask, human body keypoints, attribute prediction, clustering analysis, and color extraction can be employed to extract advanced semantics, thereby improving cross-modal retrieval performance. <cit.> proposed a dual-part simultaneous alignment representation scheme using attention mechanisms to capture distinguishing information beyond body parts by leveraging complementary information from accurate human parts and noisy parts to update representations.

To leverage multi-level visual content, Jing et al. <cit.> introduced a Pose-guided Multi-granularity Attention Network (PMA). They initially proposed a Coarse Alignment network (CA) that uses similarity-based attention to select relevant image regions for global description. Pose information can be utilized to learn potential semantic alignments between visual body parts and textual noun phrases. Shu et al. <cit.> introduced Bidirectional Mask Modeling (BMM) without the need for additional manual labeling. They applied random masks to images and corresponding text keywords, forcing the model to explore more useful matching clues. This approach increases data diversity and enhances model generalization. 

Wang et al. <cit.> proposed a novel Visual-Textual Attribute Alignment model (ViTAA). For semantic feature extraction, segmentation labels are used to drive the learning of attribute-aware features from input images. Aggarwal et al. <cit.> introduced a method for text-based person search by learning attribute-driven spatial and class-driven spatial information. The goal is to create semantically retained embeddings through an additional task of attribute prediction. Suo et al. <cit.> proposed a novel Simple Robust Correlated Filtering (SRCF) framework, which effectively extracts key clues and adaptively aligns multimodal features by constructing a set of universal semantic templates (filters). This framework differs from previous attention-based methods, focusing on computing similarities between templates and inputs. Two types of filtering modules (denoising filter and dictionary filter) were designed to extract key features and establish multimodal mappings.

Zhao et al. <cit.> introduced a Cross-Modal Mutual Training (CMMT) framework. Specifically, to alleviate intra-class variation, clustering methods generate pseudo-labels for visual and text instances. To further refine clustering results, CMMT provides a mutual pseudo-label refinement module, which utilizes clustering results in one modality to refine clustering results in another modality constrained by text-image pairwise relationships. Chen et al. <cit.> integrated information theory and adversarial learning into an end-to-end framework, exploring information theory to reduce heterogeneity gaps in cross-modal retrieval. This method is beneficial for building a shared space to further learn commonalities between cross-modal features, applicable to video-text matching. Additionally, regularization terms based on KL-divergence and temperature scaling were introduced to address the issue of data imbalance. 

Li et al. <cit.> proposed a novel Joint Label and Feature Alignment Framework (TFAF) to gradually reduce inter-modal and intra-class gaps. Firstly, a dual-path feature learning network is constructed to extract features and align them to reduce inter-modal gaps. 
Secondly, a text generation module is designed to generate label sequences using visual features, followed by label alignment to reduce intra-class gaps. Finally, a fusion interaction module is introduced, utilizing a multi-stage feature fusion strategy to further eliminate modality heterogeneity. Chen et al. <cit.> introduced the concept of knowledge distillation to image and text networks, aiming to improve the matching capabilities of two networks with different modalities by balancing information from both networks. 
Wang et al. <cit.> addressed the color-dependency issue in cross-modal pedestrian retrieval by proposing a Color-Aware Information Beyond Color(CAIBC) architecture through joint optimization of a multi-branch structure. 
CAIBC includes three branches(RGB, Grayscale, and Color) employing mutual learning mechanisms for knowledge exchange across branches. 
It achieved an accuracy of 88.37% in top-10 on the CUHK-PEDES dataset for weakly supervised text-to-person retrieval.








§ ARCHITECTURE

When mapping features from a specific modality to a common manifold, the feature distribution of other modalities remains imperceptible. This implies that embedding and aligning multimodal features in the common manifold entirely depend on the model's own experience rather than the actual data distribution. In other words, a major challenge in cross-modal Person Re-ID is ensuring that the feature distribution in the common manifold accurately reflects the feature distribution in the original modalities. This necessitates the model to possess sufficient capability to capture and understand the relationships between different modalities, which typically requires abundant data and carefully designed model structures.



 §.§ Convolutional Neural Network

Text-based person Re-ID can be viewed as a multimodal pedestrian retrieval problem, with the primary challenge lying in the extraction of features from both visual and textual data. Due to the capability of neural networks to automatically learn and extract features from pedestrian images, the cumbersome process of manual feature design can be avoided. Most existing efforts consider employing network architectures designed for image classification as the backbone of visual networks. Consequently, convolutional neural networks (CNNs) have been utilized extensively in text-based pedestrian re-identification, primarily leveraging CNNs <cit.>, and their major variants such as VGG-16 <cit.>, MobileNet <cit.>, ResNet50 <cit.>, ResNet101 <cit.>, ResNet-152 <cit.>, etc., for extracting crucial information in visual features. These networks are sometimes also employed for text feature extraction <cit.>. The combination of pedestrian images extracted by CNNs and textual description information is achieved, aligning low and high-level semantic information across modalities in an implicit space using matching methods. This facilitates the recognition and matching of pedestrian identities.

Two independently pre-trained ResNet-50 models on ImageNet serve as the visual backbone in study <cit.>. Addressing the challenge of manual annotation for existing cross-modal data, Jing et al. propose a Matrix Alignment Network (MAN) to tackle cross-modal cross-domain person search tasks, applying the single-modal cross-domain adaptive idea to the problem of image-text-based cross-modal person re-identification <cit.>. Farooq et al. introduce the Deep Vision Net model based on ResNet-50 to construct the visual feature extraction backbone <cit.>. Deviating from the original architecture, the proposed network includes two fully connected layers before the classifier layers, applies batch normalization after the two FC layers, and employs dropout before the final layer. They further present a two-stream deep convolutional neural network framework supervised by cross-entropy loss <cit.>. The language network is modified to resemble the visual branch in terms of the number of layers in deep residual networks, connecting the class probability weights between the second-to-last layer and the final layer, i.e., sharing logits of the softmax layer across both networks. AXM-Net, proposed by <cit.>, is a novel architecture based on convolutional neural networks (CNN) that does not rely on external cues for explicit alignment of feature embeddings and is capable of learning semantically aligned cross-modal feature representations. Most of the aforementioned works initialize pre-trained ResNet-50 <cit.> and ResNet-101 <cit.> backbones on large datasets to better map modal-corresponding information. Additionally, the application of large models such as CLIP in some fine-grained visual tasks has demonstrated effective paradigm transfer, fully migrating visual-text large models based on contrastive learning to text-based person re-identification tasks <cit.>. Presently, many studies commonly regard representation learning as a classification method for person re-identification, believing that feature embedding is a core module for text-based person re-identification tasks, crucial for narrowing the semantic gap between modalities. CNNs exhibit significant advantages in perceptual localization of local features in pedestrian vision and end-to-end learning. However, they still have certain limitations in handling long-term dependent data relationships.



 §.§ Recurrent Neural Network

Beyond visual features, textual descriptions directly impact the accuracy of text-based Person Re-ID. Efficient extraction of textual features is crucial under existing textual data descriptions <cit.>. Recurrent Neural Networks (RNNs) and their variants, as neural networks for processing sequential events, including text, have been widely accepted in the research community <cit.>. Early on, in this field, word embedding techniques were initially employed to map words to vector spaces. Subsequently, variants such as LSTM <cit.>, Bi-LSTM <cit.> attempted to capture long dependencies of sentences for semantic feature extraction from text sequences. Later, with the emergence of large language models like Bidirectional Encoder Representations from Transformers (BERT)<cit.>, the paradigm shift of "pre-training + fine-tuning" has been triggered in the Text-based Person Re-ID domain. Utilizing pre-trained models to learn general features of natural language and obtaining high-quality word embeddings, followed by inputting into RNNs variants to capture contextual dependencies.

Niu et al. <cit.> first proposed a concise and effective framework for image-text alignment to address visual-semantic differences. Innovatively integrating two opposing directions, source to target and target to source, for cross-domain adaptation. Wang et al. <cit.> introduced a novel Adversarial Multi-space Embedding Network (AMEN) to learn and match embeddings in multiple spaces. Bi-GRU is used for text feature extraction, and the paradigms of inter-modal and intra-modal reconstruction collaborate to embed features correctly into the opposite modal space while learning a robust common space. <cit.> introduces a Text-Graph Attention Network consisting of a Bi-LSTM layer, a text scene graph module, and a joint embedding layer. Bi-LSTM units gradually take the embeddings of each word in the text as input, capturing the historical and future context information of processed words. Yan et al. <cit.> employed an LSTM-based Recurrent Feature Aggregation Network to accumulate discriminative features from the first LSTM node to the deepest LSTM node, effectively alleviating interference caused by occlusion, background noise, and detection failures. Chen et al. <cit.> decomposed video sequences into multiple segments and used LSTM to learn to detect the segments where the images are located in both time and space features. This approach reduces variations in the same person across samples, favoring the learning of similarity features. Both of the above methods independently process each video frame. Ding et al. <cit.> employed Bidirectional Long Short-Term Memory Network (Bi-LSTM) to process each text description, capturing relationships between words. With contextual cues, the Word Attention Module (WAM) based on word representations infers word part correspondences, referring to word part correspondences to obtain original part-level texture features. Wang et al. <cit.> proposed a new algorithm called LBUL to learn a Consistent Cross-Modal Common Manifold (C3M) for text-based person retrieval. For textual samples, the proposed method embeds the text modality into C3M after considering the distribution features of cross-modalities comprehensively to achieve more reliable cross-modal distribution consensus, rather than blind predictions. However, RNN and its variants also have some issues in the Text-based Person Re-ID task. For instance, the embedding quality of pedestrian text inputs is often very sensitive, features extracted are typically influenced by the length of data sequences, and RNNs only establish temporal correlations on high-level features, thus failing to capture the temporal clues describing the local details of images.



 §.§ Autoencoder

Sometimes, text-based Person Re-ID tasks also involve the utilization of Autoencoder network architectures. Commonly, these models map images or textual descriptions into a low-dimensional embedding space and employ this embedding space for person re-identification tasks <cit.>. In this manner, pedestrian images and their corresponding textual descriptions are mapped to a low-dimensional vector space, which encapsulates crucial semantic information, as both image and text provide pivotal details about the identity of the pedestrian. Autoencoders, particularly those with atypical encoder components, exhibit several advantages in Text-based Person Re-ID tasks. Firstly, they can overcome issues prevalent in traditional approaches arising from the lack of semantic information in image-text pairs. Secondly, Autoencoder models proficiently capture essential information from textual descriptions and compress it, facilitating effective data compression. Additionally, this approach boasts good interpretability, as relationships between pedestrian images and textual descriptions can be visualized based on vectors within the embedding space. Wang et al. <cit.> introduced an innovative Adversarial Multi-space Embedding Network (AMEN) for learning and matching embeddings across multiple spaces. Following an encoder-decoder paradigm, the inter-modal reconstruction paradigm collaborates with the intra-modal reconstruction paradigm to embed features accurately into the opposite modal space while learning a robust common space. <cit.> proposed an Improved Embedding Learning through Virtual Attribute Decoupling (iVAD) model. This method enhances embedding learning by using an encoder-decoder structure to decompose attribute information in images and text. Pedestrian attributes are treated as hidden vectors, and attribute-related embeddings are obtained. Diverging from previous works that separate attribute learning from image-text embedding learning, a hierarchical feature embedding framework is introduced. The Attribute-Enhanced Feature Embedding (AEFE) module merges attribute-related embeddings into the learned image-text embeddings, leveraging attribute information to enhance feature discriminability. However, a drawback of this approach is its reliance on a substantial amount of training data to train the Autoencoder model, and meticulous tuning of model parameters and embedding space dimensions is necessary to achieve optimal performance.



 §.§ Graph Neural Network

Utilizing Graph Neural Networks (GNN) for text-based pedestrian re-identification has become a noteworthy research direction in recent years <cit.>. GNN is a type of deep learning method based on graph structures, suitable for handling unstructured data and relationships, and applicable to various data types such as images, text, and structured data. In text-based Person Re-ID tasks, GNN is often employed to construct a graph structure, where each node represents a pedestrian body part along with its corresponding phrase description. Edges between nodes represent semantic relationships between different body parts and descriptions, such as verb (wear), noun (women), adjective (yellow), and others. The methods utilizing GNN for text-based Person Re-ID typically involve the following steps. First, images and corresponding text descriptions for each pedestrian are extracted from the dataset. Second, two graph convolution structures are established based on auxiliary strategies, such as attention mechanisms, image segmentation, etc. Afterwards, Graph Convolutional Networks (GCN) are employed to learn the relationships between body nodes in the graph structure and transform these relationships into feature vectors. Finally, feature vectors of body structures undergo cross-modal asymmetric alignment, and the text modality is input into a classifier for Person Re-ID.

The methods employing GNN for text-based Person Re-ID offer several advantages. Firstly, they can capture semantic relationships between different pedestrian images and text descriptions, facilitating future inference-based retrieval, i.e., recognizing a pedestrian engaged in complex events. Secondly, GNN can handle text descriptions of varying lengths and structures, providing the advantage of data flexibility. Numerous studies have explored GNN-based Person Re-ID methods, including those using Graph Convolutional Networks (GCN) and methods based on graph self-attention mechanisms. Liu et al. <cit.> proposed a novel deep adversarial graph attention convolutional network (A-GANet) for text-based person search. A-GANet utilizes text and visual scene graphs, including object properties and relationships, from pedestrian text queries and gallery images to learn information-rich text and visual representations. It learns an effective joint text-visual latent feature space through adversarial learning, bridging the modality gap and promoting person matching. Li et al. <cit.> presented a fundamental framework using graph convolution as a feature representation. They simulate adversarial attacks caused by text and image diversity on feature extraction by embedding additional attack nodes in graph convolution layers to enhance the model's robustness to textual and visual diversity. However, when graph neural networks compute edges for pedestrian body nodes and entity relationships, it may result in high computational complexity and memory requirements. Moreover, GNNs are more susceptible to overfitting compared to general network structures.



 §.§ Transformer

Due to the success of Transformers in multimodal tasks, it has become a popular architecture widely employed in text-based Person Re-ID tasks <cit.>. Currently, a commonly used approach involves a dual-stream network architecture, where one stream is dedicated to processing pedestrian images, and the other handles the corresponding text descriptions. In the image stream, convolutional neural networks <cit.> or vision transformers <cit.> are utilized to extract visual features from pedestrian images. Simultaneously, in the text stream, Long Short-Term Memory (LSTM) <cit.> or transformers <cit.> are applied to extract feature vectors from the corresponding text descriptions. Li et al. <cit.> introduced a transformer-based Person Re-ID architecture, evaluating the shared attention between language reference terms and visual features through transformer blocks. Apart from achieving outstanding search performance, the proposed method also provides interpretability by visualizing the attention between image parts in person pictures and their corresponding referential terms. Existing works often overlook the granularity differences between features of the two modalities, where visual features tend to be fine-grained while text features are coarse-grained, leading to significant modality gaps. Shao et al. <cit.> proposed an end-to-end framework named Learning Granularity-Unified Representations (LGUR), based on the transformer, aiming to map visual and text features to a unified granularity space. Addressing alignment deficiencies between image region features and text features in weakly supervised learning methods, resulting in poor feature distribution, Gao et al. <cit.> presented a novel method called Conditional Feature Learning based Transformer (CFLT). CFLT maps sub-regions and phrases to a unified latent space and dynamically adjusts features from one modality based on data from another modality by constructing explicit conditional embeddings.

With the significant progress achieved by large models, Sarafianos et al. <cit.> demonstrated for the first time the effectiveness of BERT as an openly accessible word embedding extraction language model in the domain of image-text matching. Subsequently, they combined two feature vectors using a cross-attention mechanism and applied them to pedestrian re-identification tasks. Li et al. [70] proposed a semantically aligned feature aggregation network that adaptively aggregates unit features with the same semantics into different part-perceptive features. Initially introducing Transformer-based backbones ViT <cit.> and BERT into visual and text modalities, Ji et al. <cit.> extracted robust feature representations for text-based person search. They proposed an Asymmetric Cross-Scale Alignment (ACSA) method, employing Swin Transformer for visual representation extraction and BERT for text and phrase representation in the text domain. In contrast to previous methods that use two separate networks for feature extraction, Wang et al. <cit.> introduced an Implicit Visual-Text (IVT) framework, a single-stream network architecture using only a single transformer. Text and image feature extraction utilizes a backbone network, optimizing network shared parameters with cross-modal data to facilitate learning a common space mapping. Recently, Vision-Language Pre-training (VLP) models in visual language pre-training have demonstrated the transferability of knowledge for downstream Text-Picture Search (TPS) tasks, enhancing performance effectively. An approach called CLIP-driven Fine-grained Information Mining (CFine) <cit.> was proposed to leverage CLIP's visual knowledge for person re-identification, incorporating fine-grained information mining to explore intra-modal discriminative clues and inter-modal correspondences. However, the lack of VLP text representations may compromise the valuable cross-modal alignment of the two encoders. Hence, Wang et al. <cit.> introduced a model with two pre-training modes, TP-TPS, where both text and image encoders are based on transformers, and all backbones are initialized with pre-training weights from CLIP. They introduced Multi-Integrity Description Constraints (MIDC) to enhance the robustness of the text modality by combining different components of a fine-grained corpus during training. The advantage of the Transformer model lies in its ability to effectively learn sequence information, making it capable of capturing information in text descriptions efficiently. Additionally, this approach exhibits good interpretability in the field of multi-scale learning, as it allows the visualization of regions of interest in pedestrian images and keywords in text descriptions based on attention weights. Finally, it plays a crucial role in unified modality architectures.








§ OPTIMIZATION

Deep metric learning plays a paramount optimization role in cross-modal person re-identification <cit.>. 
For instance, it enhances the discriminative power of feature representation, addresses a multitude of noise and variations such as diverse lighting conditions, poses, and occlusions, adapts the distance metric accordingly, embeds more semantically meaningful spatial structures, and leverages domain adaptation to tackle the challenges posed by multiple viewing angles. Of particular significance is the optimization of loss functions, thereby improving the matching performance in cross-modal person re-identification tasks <cit.>. This approach effectively addresses a spectrum of challenges, elevating the network's robustness and generalization capacity, consequently achieving more accurate person matching across distinct camera scenes and conditions. 
This study places emphasis on prevalent loss functions used in text-based person re-identification tasks, including identity loss, verification loss, contrastive loss, triplet loss, quadruplet loss, and adversarial loss, as shown in Figure <ref>.





 §.§ Identity Loss

Zheng et al. <cit.> introduced the ID Discriminative Embedding (IDE) network, which treats each pedestrian as a separate category and employs the pedestrian's ID as a classification label to train a deep neural network. This allows the network to learn the capability of predicting the identity given a pedestrian image. The classification loss is typically computed using the cross-entropy loss function. During training, the classifier optimizes its weight parameters by minimizing the classification loss. A fully connected layer (FC) for classification is appended to the end of the network, and the softmax activation function maps the features to a probability space representing identity labels <cit.>. The cross-entropy loss for the multi-class pedestrian Re-ID task can be expressed as:


    L_ID = -1/N∑_i=1^Ny_ilog(p_i).


Here, x_i denotes the input image, y_i represents the corresponding label, p_i is the probability that x_i is recognized as y_i after the softmax function, and N indicates the batch size. Identity loss is widely adopted in pedestrian re-identification <cit.>, due to its automatic hard sample mining and ease of training. However, as the labeled data increases significantly, issues related to decreased efficiency in classifier training might arise, especially when dealing with datasets containing a large number of similar pedestrians. In cross-modal pedestrian re-identification, the classification loss is often used in conjunction with other loss functions <cit.>, such as contrastive loss and triplet loss <cit.>, to enhance performance of the model.



 §.§ Verification Loss

The verification loss is employed to quantify the similarity between two pedestrian images <cit.>. In pedestrian re-identification, it is often necessary to compare two images to determine if they belong to the same individual. The verification loss aims to achieve this objective by comparing the feature vectors of two images to compute their similarity <cit.>. Specifically, the verification loss is calculated by inputting the feature vectors of two images into a similarity metric function. This similarity metric function can be Euclidean distance, cosine similarity, or other similar measures. If two images belong to the same person, their feature vectors should exhibit high similarity, resulting in a high similarity score <cit.>. Thus, the verification loss is minimized to enhance the similarity between two images of the same person. The specific formula is as follows:


    L_veri = -1/N∑_i=1^Ny_ilog(p_i) + (1-y_i)log(1-p_i).


Here, N is the number of sample pairs, y_i is a label indicating whether the sample pair x_i and x_j belongs to the same category (1 denotes the same category, 0 denotes different categories), and the softmax function is utilized to calculate the probability p_i of having the same label for both samples. Due to the necessity of pairwise computation, the verification loss tends to have lower efficiency in re-identification tasks. Therefore, it is often used in combination with other losses, such as identity loss, to leverage the advantages of both methods, thereby enhancing recognition speed and accuracy.



 §.§ Contrastive Loss

Contrastive Loss was originally introduced by Sumit Chopra et al. in the context of face verification tasks <cit.>, aiming to train neural networks to determine whether two inputs belong to the same facial identity. In the context of person re-identification, the principle of encouraging similar identity feature representations to be closer in the feature space while dispersing feature representations of different identities is leveraged for model training within Siamese networks <cit.>. The Contrastive Loss not only considers the distances between images of the same identity but also takes into account the distances between images of different identities. The function can be expressed as follows:


    L_con = 1/2N∑_n=1^Ny_nD_n^2 + (1-y_n)max(margin - D_n, 0)^2.


Here, N is the number of sample pairs, y_n is a label indicating whether sample pairs x_i and x_j belong to the same class (1 for the same class, 0 for different classes), D_n represents the distance between feature representations (Euclidean distance, cosine distance, etc.), and margin is a threshold indicating that when sample pairs belong to different classes, their distance should be greater than this value. For sample pairs belonging to the same class, the loss is the square of the distance D_n, while for sample pairs belonging to different classes, the loss is the square of the difference between the distance and the threshold value margin. This framework aims to bring feature representations of sample pairs from the same class closer together and push feature representations of sample pairs from different classes further apart. By adjusting the value of margin and the weight of the loss, the distance range between sample pairs can be controlled, influencing the learning of feature representations <cit.>.



 §.§ Triplet Loss

The Triplet Loss is a widely utilized loss function in pedestrian re-identification, aiming to learn discriminative feature representations by comparing the distances between three images <cit.>. Its fundamental concept involves learning feature representations with discrimination, such that the distance between images of the same person is minimized, while the distance between images of different individuals is maximized <cit.>, <cit.>, achieved by minimizing the Triplet Loss during training. Specifically, given an anchor image, a positive sample image, and a negative sample image, the objective of Triplet Loss is to minimize the distance between the anchor and the positive sample while simultaneously maximizing the distance between the anchor and the negative sample <cit.>, <cit.>. The formal definition of Triplet Loss is as follows:


    L_triplet =    ∑_n=1^Nmax(||f(x_n^a) - f(x_n^p)||_2^2
          - ||f(x_n^a) - f(x_n^n)||_2^2 + margin, 0).


Here, f is a function that maps an image x to a feature vector, x_n^a, x_n^p and x_n^n are the anchor, positive, and negative sample images of the n-th instance, respectively. ||·||_2 represents the Euclidean distance between two embedding vectors, max(·) represents taking the maximum value between 0 and ·, margin is a hyperparameter controlling the difference in distances between images of the same person and images of different people. It is worth noting that in practice, Triplet Loss has encountered certain issues, such as sample selection and slow learning speed. Consequently, improved versions have been proposed <cit.>. Wang et al. <cit.> proposed a novel mutually connected classification loss (MCCL) based on triplet loss to fully exploit identity-level information. It not only incorporates identification information into both images and language descriptions but also encourages similar cross-modal classification probabilities for instances belonging to the same identity. These improved versions prove to address certain existing issues more effectively and have demonstrated excellent performance in person re-identification tasks.



 §.§ Quadruplet Loss

The Quadruplet Loss is an extension of the Triplet Loss <cit.>. The Triplet Loss is commonly employed to train a neural network to embed images of the same person into close proximity within the embedding space, while images of different individuals are embedded far from the space occupied by the same person. However, the Triplet Loss may suffer from issues such as gradient vanishing or exploding, which could render it less stable in practical usage. The Quadruplet Loss enhances the stability of the Triplet Loss by introducing a fourth sample.

In specific terms, the Quadruplet Loss computes pairs of embedding distances for two images of the same person and two images of different individuals, and employs these four distances to formulate the loss function. In this regard, the two images of the same person should be closer, while the two images of different individuals should be farther apart. By separately using the two images of the same person and the two images of different individuals to calculate distances, the Quadruplet Loss can yield more stable training, thus enhancing the performance of person re-identification. The formula for the Quadruplet Loss is as follows:


    L_quad = ∑_n=1^Nmax(   ||f(x_n^a) - f(x_n^p)||_2^2 - ||f(x_n^a) - f(x_n^n1)||_2^2
        + α, 0)+ max(||f(x_n^a) - f(x_n^n1)||_2^2
       - ||f(x_n^n1) - f(x_n^n2)||_2^2 + β, 0).


Here, f denotes the embedding function of the neural network, x_n^a, x_n^p, x_n^n1 and x_n^n2 respectively represent the anchor image, positive sample image, first negative sample image, and second negative sample image for the n-th sample. ||·||_2 represents the Euclidean distance between two embedding vectors. max(·) represents taking the maximum value between 0 and ·, The hyperparameters α and β control the difference in distances between the two images of the same person and the two images of different individuals.

The Quadruplet Loss effectively reduces the within-class variance and improves the discriminative performance of the model by introducing an additional negative sample. However, the increase in parameters also requires the computation of more pairs of samples, leading to a significant increase in computational complexity and longer training time. Therefore, suitable strategies are required to select effective quadruples in practical applications.



 §.§ Adversarial Loss

The adversarial loss primarily leverages adversarial training to learn more robust pedestrian feature representations <cit.>. Its fundamental concept involves mapping pedestrian feature embeddings from the training data into a similarity space, where the distribution of generated image features is made closer to that of real image features, enhancing the robustness of the network in handling multi-modal information <cit.>. Specifically, adversarial loss employs adversarial samples during training, where the model seeks to minimize the adversarial loss. The computation of the adversarial loss involves introducing perturbations to the model input, making the input pedestrian image appear visually similar to the original image while introducing significant disparities in the embedding space. These perturbations are generated by a generator network, which attempts to produce images similar in appearance to the input image but with substantial deviations in the embedding space. The discriminator seeks to maximize the adversarial loss to effectively discriminate between these two types of features. The general formula for adversarial loss is as follows:


    L_adv =    𝔼_x∼ p_data(x)[log D(x)] 
       + 𝔼_z∼ p_z(z)[log(1 - D(G(z)))].


Here, G is the generator, D is the discriminator, x represents real image features, z represents generated image features, log D(x) denotes the logarithm of the discriminator's predicted probability for real image features, and log(1 - D(G(z))) denotes the logarithm of the discriminator's predicted probability for generated image features. Liu et al. <cit.> proposed the application of adversarial principles in pedestrian re-identification to acquire more robust feature representations. They contend that the distributions of text and image features should be modality-invariant and semantically discriminative, while maintaining latent cross-modal semantic similarities. These features exhibit enhanced resistance to adversarial attacks. Subsequently, an adversarial learning module was devised to optimize the representations extracted from image graph attention networks and text graph attention networks. The objective of the modality discriminator is to reliably distinguish between modalities of samples as much as possible, given a well-trained feature transformer, until the point where discrimination becomes infeasible. Adversarial loss is employed to classify the modality labels (belonging to either image or text) of input samples. Despite its relatively recent application in pedestrian re-identification, adversarial loss has demonstrated significant efficacy in some recent studies <cit.>.








§ CHALLENGE


In this paper, we first give an overview of text-based person Re-ID in two parts: attribute-based and natural language-based person Re-ID. Secondly, the datasets based on the two parts are summarized and described respectively. Moreover, the commonly used technical strategies in text-based person Re-ID are analyzed. In addition, an overview of the deep learning architectures that usually appear in text-based person Re-ID is made. Further, the loss functions (alignment optimization methods) commonly used in text-based person Re-ID are summarized and described in detail. Finally, we analyze the challenges and possible future directions of text-based person Re-ID from the perspectives of feature extraction, data scarcity, pre-trained models, and generative models, respectively.

Although text-based person Re-ID technology offers numerous advantages in fields such as video surveillance, security, social media, and virtual reality<cit.>, it still faces various challenges. The most critical issues include the ambiguity and granularity of natural language. Identical descriptions may be incomplete, ambiguous, or contain ambiguities, potentially leading to incorrect identity matches. On the other hand, photographs available in real-world galleries pertaining to pedestrians often lack high resolution, ample lighting, and freedom from obstruction<cit.>. 
Zhang et al. <cit.> demonstrated recently that gait information may be the basis for general person Re-ID, and that its features may be more robust compared to the highly homogeneous pedestrian appearance feature signature. However, existing gait recognition methods are not able to meet the requirements of less controlled scenarios, e.g., most of the studies are based on well controlled scenarios such as narrow corridors. If one wants to improve the efficiency of person Re-ID by combining gait features with appearance features, it may be necessary to obtain more coherent and high-quality gait data, while the appearance is recognizable. 
Therefore, integrating text descriptions with image/video information requires more efficient and robust cross-modal fusion techniques for the system to better understand and match entities. Pre-trained large-scale multimodal models may represent a reasonable direction for achieving improved encoding effects. Furthermore, text-based person re-identification necessitates more high-quality data to enable the model to learn relationships between various descriptions and identities <cit.>. It is noteworthy that the recent popularity of text-to-image generation models, as a novel approach in pedestrian re-identification, holds significant potential, as these models generate graphics from textual descriptions for retrieval. Given that cameras in the real world do not always capture images from an available query library, customizing the generation of corresponding target entities based on eyewitness text is meaningful. However, due to the insufficient maturity of text-to-image generation model controllability, we remain optimistic about this direction.








§ FORESIGHT




 §.§ A Novel Baseline

Since the current text-based person Re-ID tasks being based on disparate datasets, specialized models are separately trained on large samples. However, in the real world, the lack of a uniform style (resolution, size, angle, brightness) in pedestrian Re-ID data is often due to the constraints imposed by different specifications of data collection devices (cameras). Retrieving arbitrary pedestrian targets in an open-world scenario becomes quite challenging for dedicated models. There is an urgent need for a new approach that enables zero-shot retrieval of targets from any gallery. On the other hand, text-to-image retrieval fundamentally relies on comparing cross-modal data distributions from small datasets through contrastive learning. If the gallery itself lacks target pedestrians, this results in ineffective retrieval, contradicting the concept of open-world pedestrian recognition. Inspired by the controllable generation retrieval in text-to-image for pedestrians, we propose a novel baseline model for text-based person re-identification, which involves re-retrieving target images from text to image.

As illustrated in Figure <ref>, the entire text-based pedestrian image generation guided Re-ID baseline architecture is demonstrated. The initial step involves encoding natural language descriptions of pedestrians and utilizing a diffusion model to generate corresponding pedestrian images. Subsequently, these generated pedestrian images are employed in the task of pedestrian re-identification. Given that current text-to-image generation models primarily focus on coarse-grained overall image generation tasks, emphasizing inter-class coordination among different objects rather than concentrating on intra-class distinctiveness in human images, and while simultaneously considering authenticity, a strategy based on human body key-point assistance is adopted. This strategy involves extracting features from human body key-points to assist in controlling the pedestrian generation process, ensuring compliance with the requirement for the presence of the majority of the human body in the pedestrian re-identification task. The generated pedestrian images serve as queries for retrieval and are input into the backbone network for visual feature extraction. To enhance retrieval accuracy, emphasis is placed on restoring features with the highest relevance to the original image. For instance, in comparison to generated images, sensitivity to channel attention may be more pronounced in original images. Various attention mechanisms are employed to assess their impact on retrieval effectiveness. However, due to the coarser granularity of linguistic descriptions compared to images, the generation of pedestrian images may not be sufficiently detailed in terms of the style and material of the pedestrian's clothing. For example, the textual description may be as broad as "black coat", but the image may be as specific as "short black leather jacket", which results in feature learning inaccuracy. In order to improve the accuracy of localized feature generation for pedestrians in a realistic open set, an approach using cross-attention based image editing is recommended. After using natural language instructions, the local features of pedestrians are controlled to change, and the witness's memory of the local features for the target person is refined and specialized. Finally, the results of pedestrian retrieval are obtained through different optimization strategies.







 §.§ Experiment & Results

For A) Text-based pedestrian generation module, We employed the Frozen CLIP Text Encoder <cit.> to encode pedestrian descriptions, followed by inputting them into stable diffusion <cit.> to generate pedestrian images. The denoising module utilized the cross-attention U-Net <cit.> backbone, which proves beneficial for the generation of high-quality pedestrian images. For pedestrian pose assistance, we adopt ControlNet <cit.> for Pedestrian integrity control, which ensures that the generated pedestrian image has full body. Specifically, OpenPose <cit.> was employed for detecting key points on the pedestrian body. Subsequently, pose features were extracted through skip connections and zero convolutions, facilitating their incorporation into pedestrian generation. 

In the part of B) pedestrian local feature refinement, we input the local image editing instructions and then use GPT-3 <cit.> to rewrite and replace the pedestrian image generation instructions. Re-inputting the rewritten text cue and the pedestrian image before rewriting into the stable diffusion model, the visual and textual features of pedestrians are fused using the cross-attention layer, and the image can be edited by editing the text only to realize the mapping from the source image to the target image <cit.>. As a comparative analysis, we also attempted fine-tuning the baseline model using a subset of text-image pairs data based on text-driven pedestrian re-identification. In the retrieval module, the generated pedestrian images were employed as queries. 

In the C) pedestrian Re-ID system, ResNet-50 <cit.> served as the visual backbone, incorporating self-attention mechanisms. The Part-based Convolutional Baseline (PCB)<cit.> strategy was employed for segmenting human bodies to extract feature maps, and random horizontal image flipping was applied for data augmentation. The training process involved the use of Instance Batch Normalization (IBN) <cit.>strategy, with a batch size set to 32, droprate at 0.5, erasing probability at 0.2, ins-gamma set to 32, learning rate at 0.05, and weight decay at 0.0005. Furthermore, the training amalgamated ID loss, Contrastive loss, and Triplet loss as the loss functions. The optimization utilized SGD optimizera with momentum, and the model was trained for 60 epochs.


As depicted in the Table  experiment results, we evaluated the proposed Text-Based Pedestrian Image Generation Guided Re-ID (TBPGR) baseline architecture on three widely used text-based person Re-ID datasets (CUHK-PEDES, RSTPReid, ICFG-PEDES). Specifically, retrieval of pedestrian images was generated using the generation modules with and without fine-tuned Stable Diffusion, respectively. We report the performance of visual representation modules DenseNet, EfficientNet Swin-Transformer and ResNet on the task for each dataset and each condition of with and without fine tuning. 
ResNet generally shows better performance. After the image generation in the intermediate step, we qualitatively analyzed the content of the images. It was found that the quality of the pedestrian images generated before fine-tuning was higher, and the quality of the images generated after fine-tuning became lower, but the retrieval efficiency was slightly improved, conjecturing that the quality of the pedestrian image text data pairs had some effect on the data distribution. 
Besides, since the CUHK-PEDES dataset retrieval effect is better than other datasets, it is hypothesized that it is because the generation of pedestrian images seldom involves the factors such as lens occlusion and illumination changes, which have some implications on the pedestrian retrieval performance, as shown in Figure  <ref>.

Although our baseline model(TBPGR) is still not powerful enough in comparison to traditional training approaches based on implicit spatial alignment of text and images, we believe that such an open-retrieval paradigm is far more valuable in future Open Set text-based person Re-ID scenarios than just pursuing the accuracy of the present moment.









§ ACKNOWLEDGMENTS

The authors would like to thank all the anonymous reviewers and associate editors for their valuable comments to improve the paper. The authors would also like to thank the authors of the three publicly available datasets.

















 







1
IEEEtran

aggarwalTextbasedPersonSearch2020a
S. Aggarwal, R. V. Babu, and A. Chakraborty, “Text-based Person Search via
  Attribute-aided Matching,” in 2020 IEEE Winter Conference on
  Applications of Computer Vision (WACV), pp. 2606–2614, Mar.
  2020.

ALIGNScalingVisual
“ALIGN: Scaling Up Visual and Vision-Language Representation
  Learning With Noisy Text Supervision – Google AI Blog.”
  https://ai.googleblog.com/2021/05/align-scaling-up-visual-and-vision.html.

baiRaSaRelationSensitivity2023
Y. Bai, M. Cao, D. Gao, Z. Cao, C. Chen, Z. Fan, L. Nie, and M. Zhang,
  “RaSa: Relation and Sensitivity Aware Representation Learning
  for Text-based Person Search,” May 2023.

bedagkar-galaSurveyApproachesTrends2014b
A. Bedagkar-Gala and S. K. Shah, “A survey of approaches and trends in
  person re-identification,” Image and Vision Computing, vol. 32,
  pp. 270–286, Apr. 2014.

birdNLTKNaturalLanguage2006
S. Bird, “NLTK: The natural language toolkit,” in Proceedings of the
  COLING/ACL on Interactive Presentation Sessions, COLING-ACL
  '06, (USA), pp. 69–72, Association for Computational Linguistics, July
  2006.

caiParallelDataAugmentation2022a
H.-Q. Cai, X. Li, Y. Ji, Y. Li, and C.-P. Liu, “Parallel Data Augmentation
  for Text-based Person Re-identification,” in 2022 International
  Joint Conference on Neural Networks (IJCNN), pp. 1–8, July 2022.

caoSymbioticAdversarialLearning2020
Y.-T. Cao, J. Wang, and D. Tao, “Symbiotic Adversarial Learning for
  Attribute-Based Person Search,” in Computer Vision
  – ECCV 2020: 16th European Conference, Glasgow,
  UK, August 23–28, 2020, Proceedings, Part XIV,
  (Berlin, Heidelberg), pp. 230–247, Springer-Verlag, Aug. 2020.

CascadeAttentionNetwork2018
“Cascade Attention Network for Person Search: Both Image and
  Text-Image Similarity Selection.”
  https://deepai.org/publication/cascade-attention-network-for-person-search-both-image-and-text-image-similarity-selection,
  Sept. 2018.

chaharStudyDeepConvolutional2017
H. Chahar and N. Nain, “A Study on Deep Convolutional Neural Network
  Based Approaches for Person Re-identification,” in Pattern
  Recognition and Machine Intelligence (B. U. Shankar, K. Ghosh, D. P.
  Mandal, S. S. Ray, D. Zhang, and S. K. Pal, eds.), Lecture Notes in
  Computer Science, (Cham), pp. 543–548, Springer International
  Publishing, 2017.

changpinyoConceptual12MPushing2021
S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, “Conceptual 12M:
  Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual
  Concepts,” in Proceedings of the IEEE/CVF Conference on
  Computer Vision and Pattern Recognition, pp. 3558–3568, 2021.

chechikLargeScaleOnline2010
G. Chechik, V. Sharma, U. Shalit, and S. Bengio, “Large Scale Online
  Learning of Image Similarity Through Ranking,” The Journal of
  Machine Learning Research, vol. 11, pp. 1109–1135, Mar. 2010.

chenCrossModalKnowledgeAdaptation2021
Y. Chen, R. Huang, H. Chang, C. Tan, T. Xue, and B. Ma, “Cross-Modal
  Knowledge Adaptation for Language-Based Person Search,” IEEE
  Transactions on Image Processing, vol. 30, pp. 4057–4069, 2021.

chenCrossModalRetrievalHeterogeneous2022
D. Chen, M. Wang, H. Chen, L. Wu, J. Qin, and W. Peng, “Cross-Modal
  Retrieval with Heterogeneous Graph Embedding,” in Proceedings of
  the 30th ACM International Conference on Multimedia, MM '22,
  (New York, NY, USA), pp. 3291–3300, Association for Computing Machinery,
  Oct. 2022.

chenEndtoEndUnsupervisedVisionandLanguage2022
C. Chen, P. Li, M. Sun, and Y. Liu, “End-to-End Unsupervised
  Vision-and-Language Pre-training with Referring Expression Matching,”
  in Conference on Empirical Methods in Natural Language
  Processing, 2022.

chengPersonReidentificationMultiChannel2016
D. Cheng, Y. Gong, S. Zhou, J. Wang, and N. Zheng, “Person
  Re-identification by Multi-Channel Parts-Based CNN with Improved
  Triplet Loss Function,” in 2016 IEEE Conference on Computer
  Vision and Pattern Recognition (CVPR), pp. 1335–1344, June 2016.

chenImprovingDeepVisual2018
D. Chen, H. Li, X. Liu, Y. Shen, J. Shao, Z. Yuan, and X. Wang, “Improving
  Deep Visual Representation for Person Re-identification by Global
  and Local Image-language Association,” in Proceedings of the
  European Conference on Computer Vision (ECCV), pp. 54–70, 2018.

chenImprovingTextBasedPerson2018
T. Chen, C. Xu, and J. Luo, “Improving Text-Based Person Search by
  Spatial Matching and Adaptive Threshold,” in 2018 IEEE Winter
  Conference on Applications of Computer Vision (WACV),
  pp. 1879–1887, Mar. 2018.

chenIntegratingInformationTheory2021
W. Chen, Y. Liu, E. M. Bakker, and M. S. Lew, “Integrating information theory
  and adversarial learning for cross-modal retrieval,” Pattern
  Recognition, vol. 117, p. 107983, Sept. 2021.

chenKeypointMessagePassing2022
D. Chen, A. Doering, S. Zhang, J. Yang, J. Gall, and B. Schiele, “Keypoint
  Message Passing for Video-Based Person Re-identification,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36,
  pp. 239–247, June 2022.

chenTIPCBSimpleEffective2022
Y. Chen, G. Zhang, Y. Lu, Z. Wang, and Y. Zheng, “TIPCB: A simple but
  effective part-based convolutional baseline for text-based person search,”
  Neurocomputing, vol. 494, pp. 171–181, July 2022.

chenTripletLossDeep2017
W. Chen, X. Chen, J. Zhang, and K. Huang, “Beyond triplet loss: A deep
  quadruplet network for person re-identification,” Apr. 2017.

chenUNITERUNiversalImageTExt2020
Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu,
  “UNITER: UNiversal Image-TExt Representation Learning,” in Computer Vision – ECCV 2020 (A. Vedaldi, H. Bischof,
  T. Brox, and J.-M. Frahm, eds.), Lecture Notes in Computer Science,
  (Cham), pp. 104–120, Springer International Publishing, 2020.

chenyangTexttoImagePersonSearch2022a
Z. Chenyang, F. Jun, and W. Jiaqing, “Text-to-Image Person Search Based on
  SSAN Model and Re-rank Post-Processing,” in 2022
  International Conference on Computer Engineering and Artificial
  Intelligence (ICCEAI), pp. 409–413, July 2022.

chopraLearningSimilarityMetric2005
S. Chopra, R. Hadsell, and Y. Lecun, Learning a Similarity Metric
  Discriminatively, with Application to Face Verification, vol. 1.
July 2005.

cormierUPARChallengePedestrian2023
M. Cormier, A. Specker, J. C. S. Jacques, L. Florin, J. Metzler, T. B.
  Moeslund, K. Nasrollahi, S. Escalera, and J. Beyerer, “UPAR Challenge:
  Pedestrian Attribute Recognition and Attribute-based Person Retrieval
  - Dataset, Design, and Results,” 2023 IEEE/CVF Winter
  Conference on Applications of Computer Vision Workshops (WACVW),
  pp. 166–175, Jan. 2023.

dengImageNetLargescaleHierarchical2009b
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet:
  A large-scale hierarchical image database,” in 2009 IEEE
  Conference on Computer Vision and Pattern Recognition,
  pp. 248–255, June 2009.

dengPedestrianAttributeRecognition2014a
Y. DENG, P. Luo, C. C. Loy, and X. Tang, “Pedestrian Attribute Recognition
  At Far Distance,” in Proceedings of the 22nd ACM International
  Conference on Multimedia, MM '14, (New York, NY, USA),
  pp. 789–792, Association for Computing Machinery, Nov. 2014.

devlinBERTPretrainingDeep2019
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training
  of Deep Bidirectional Transformers for Language Understanding,” May
  2019.

dingSemanticallySelfAlignedNetwork2021
Z. Ding, C. Ding, Z. Shao, and D. Tao, “Semantically Self-Aligned Network
  for Text-to-Image Part-aware Person Re-identification,” ArXiv,
  July 2021.

dongPersonSearchText2019
Q. Dong, X. Zhu, and S. Gong, “Person Search by Text Attribute Query As
  Zero-Shot Learning,” 2019 IEEE/CVF International Conference on
  Computer Vision (ICCV), pp. 3651–3660, Oct. 2019.

dosovitskiyImageWorth16x162023
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai,
  T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit,
  and N. Houlsby, “An Image is Worth 16x16 Words: Transformers
  for Image Recognition at Scale,” in International
  Conference on Learning Representations, Mar. 2023.

duanDeepLocalizedMetric2018
Y. Duan, J. Lu, J. Feng, and J. Zhou, “Deep Localized Metric Learning,”
  IEEE Transactions on Circuits and Systems for Video Technology,
  vol. 28, pp. 2644–2656, Oct. 2018.

engilbergeFindingBeansBurgers2018
M. Engilberge, L. Chevallier, P. Perez, and M. Cord, “Finding Beans in
  Burgers: Deep Semantic-Visual Embedding with Localization,” in
  2018 IEEE/CVF Conference on Computer Vision and Pattern
  Recognition, (Salt Lake City, UT), pp. 3984–3993, IEEE, June 2018.

faghriVSEImprovingVisualSemantic2017
F. Faghri, D. J. Fleet, J. Kiros, and S. Fidler, “VSE++: Improving
  Visual-Semantic Embeddings with Hard Negatives,” in British
  Machine Vision Conference, July 2017.

farooqAXMNetImplicitCrossModal2022
A. Farooq, M. Awais, J. Kittler, and S. S. Khalid, “AXM-Net: Implicit
  Cross-Modal Feature Alignment for Person Re-identification,” in Proceedings of the AAAI Conference on Artificial Intelligence,
  vol. 36, pp. 4477–4485, June 2022.

farooqConvolutionalBaselinePerson2020
A. Farooq, M. Awais, F. Yan, J. Kittler, A. Akbari, and S. S. Khalid, “A
  Convolutional Baseline for Person Re-Identification Using Vision and
  Language Descriptions,” Feb. 2020.

farooqCrossModalPerson2020
A. Farooq, M. Awais, J. Kittler, A. Akbari, and S. S. Khalid, “Cross Modal
  Person Re-identification with Visual-Textual Queries,” in 2020
  IEEE International Joint Conference on Biometrics (IJCB),
  pp. 1–8, Sept. 2020.

frolovAdversarialTexttoImageSynthesis2021
S. Frolov, T. Hinz, F. Raue, J. Hees, and A. Dengel, “Adversarial
  Text-to-Image Synthesis: A Review,” Neural Networks, vol. 144,
  pp. 187–209, Dec. 2021.

galiyawalaPersonRetrievalSurveillance2021
H. Galiyawala and M. S. Raval, “Person retrieval in surveillance using textual
  query: A review,” Multimedia Tools and Applications, vol. 80,
  pp. 27343–27383, July 2021.

gaoConditionalFeatureLearning2022
C. Gao, G. Cai, X. Jiang, F. Zheng, J. Zhang, Y. Gong, F. Lin, X. Sun, and
  X. Bai, “Conditional Feature Learning Based Transformer for Text-Based
  Person Search,” IEEE Transactions on Image Processing, vol. 31,
  pp. 6097–6108, 2022.

gaoContextualNonLocalAlignment2021
C. Gao, G. Cai, X. Jiang, F. Zheng, J. Zhang, Y. Gong, P. Peng, X.-W. Guo, and
  X. Sun, “Contextual Non-Local Alignment over Full-Scale
  Representation for Text-Based Person Search,” ArXiv, Jan. 2021.

geVisualTextualAssociationHardest2019a
J. Ge, G. Gao, and Z. Liu, “Visual-Textual Association with Hardest
  and Semi-Hard Negative Pairs Mining for Person Search,” Dec. 2019.

gomezChapterSelfSupervisedLearning2019
R. Gomez, L. Gomez, J. Gibert, and D. Karatzas, “Chapter 9 - Self-Supervised
  Learning from Web Data for Multimodal Retrieval,” in Multimodal Scene Understanding (M. Y. Yang, B. Rosenhahn, and V. Murino,
  eds.), pp. 279–306, Academic Press, Jan. 2019.

grayEvaluatingAppearanceModels2007
D. Gray, S. Brennan, and H. Tao, “Evaluating appearance models for
  recognition, reacquisition, and tracking,” Proc. IEEE Int. Workshop
  Vis. Surveill. Perform. Eval. Tracking Surveill., Oct., Jan. 2007.

guLookImagineMatch2018
J. Gu, J. Cai, S. Joty, L. Niu, and G. Wang, “Look, Imagine and Match:
  Improving Textual-Visual Cross-Modal Retrieval with Generative
  Models,” June 2018.

hanFashionViLFashionFocusedVisionandLanguage2022
X. Han, L. Yu, X. Zhu, L. Zhang, Y.-Z. Song, and T. Xiang, “FashionViL:
  Fashion-Focused Vision-and-Language Representation Learning,” arXiv,
  2022.

hanTextBasedPersonSearch2021a
X. Han, S. He, L. Zhang, and T. Xiang, “Text-Based Person Search with
  Limited Data,” in BMVC 2021, arXiv, Oct. 2021.

heDeepResidualLearning2016
K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image
  Recognition,” in 2016 IEEE Conference on Computer Vision and
  Pattern Recognition (CVPR), pp. 770–778, June 2016.

hermansDefenseTripletLoss2017
A. Hermans, L. Beyer, and B. Leibe, “In Defense of the Triplet Loss
  for Person Re-Identification,” Nov. 2017.

huangDeepLearningVisibleinfrared2023
N. Huang, J. Liu, Y. Miao, Q. Zhang, and J. Han, “Deep learning for
  visible-infrared cross-modality person re-identification: A comprehensive
  review,” Information Fusion, vol. 91, pp. 396–411, Mar. 2023.

huangSeeingOutBOx2021
Z. Huang, Z. Zeng, Y. Huang, B. Liu, D. Fu, and J. Fu, “Seeing Out of
  tHe bOx: End-to-End Pre-training for Vision-Language Representation
  Learning,” 2021 IEEE/CVF Conference on Computer Vision and Pattern
  Recognition (CVPR), pp. 12971–12980, June 2021.

iodiceTextAttributeAggregation2020
S. Iodice and K. Mikolajczyk, “Text Attribute Aggregation and Visual
  Feature Decomposition for Person Search,” in 2020 British
  Machine Vision Conference, 2020.

islamPersonSearchNew2020
K. Islam, “Person search: New paradigm of person re-identification: A
  survey and outlook of recent works,” Image and Vision Computing,
  vol. 101, p. 103970, Sept. 2020.

jeongASMRLearningAttributeBased2021
B. Jeong, J. Park, and S. Kwak, “ASMR: Learning Attribute-Based Person
  Search with Adaptive Semantic Margin Regularizer,” 2021 IEEE/CVF
  International Conference on Computer Vision (ICCV), pp. 11996–12005, Oct.
  2021.

jiangCrossModalImplicitRelation2023b
D. Jiang and M. Ye, “Cross-Modal Implicit Relation Reasoning and
  Aligning for Text-to-Image Person Retrieval,” in Proceedings of
  the IEEE/CVF Conference on Computer Vision and Pattern
  Recognition, pp. 2787–2797, 2023.

jiaScalingVisualVisionLanguage2021
C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung,
  Z. Li, and T. Duerig, “Scaling Up Visual and Vision-Language
  Representation Learning With Noisy Text Supervision,” in Proceedings
  of the 38th International Conference on Machine Learning,
  pp. 4904–4916, PMLR, July 2021.

jiAsymmetricCrossScaleAlignment2022
Z. Ji, J. Hu, D. Liu, L. Y. Wu, and Y. Zhao, “Asymmetric Cross-Scale
  Alignment for Text-Based Person Search,” IEEE Transactions on
  Multimedia, pp. 1–11, 2022.

jingCrossModalCrossDomainMoment2020
Y. Jing, W. Wang, L. Wang, and T. Tan, “Cross-Modal Cross-Domain Moment
  Alignment Network for Person Search,” 2020 IEEE/CVF Conference on
  Computer Vision and Pattern Recognition (CVPR), pp. 10675–10683, June 2020.

jingPoseGuidedJointGlobal2018
Y. Jing, C. Si, J. Wang, W. Wang, L. Wang, and T. Tan, “Pose-Guided Joint
  Global and Attentive Local Matching Network for Text-Based Person
  Search,” Sept. 2018.

jingPoseGuidedMultiGranularityAttention2020a
Y. Jing, C. Si, J. Wang, W. Wang, L. Wang, and T. Tan, “Pose-Guided
  Multi-Granularity Attention Network for Text-Based Person Search,”
  Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34,
  pp. 11189–11196, Apr. 2020.

karanamSystematicEvaluationBenchmark2019
S. karanam, M. Gou, Z. Wu, A. Rates-Borras, O. Camps, and R. J. Radke, “A
  Systematic Evaluation and Benchmark for Person Re-Identification:
  Features, Metrics, and Datasets,” IEEE Transactions on
  Pattern Analysis and Machine Intelligence, vol. 41, pp. 523–536, Mar. 2019.

laviSurveyReliableDeep2020
B. Lavi, I. Ullah, M. Fatan, and A. Rocha, “Survey on Reliable Deep
  Learning-Based Person Re-Identification Models: Are We There Yet?,”
  Apr. 2020.

leeStackedCrossAttention2018
K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He, “Stacked Cross Attention for
  Image-Text Matching,” July 2018.

liAlignFuseVision2021
J. Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, and S. Hoi,
  “Align before Fuse: Vision and Language Representation Learning
  with Momentum Distillation,” in Neural Information Processing
  Systems, July 2021.

liaoPersonReidentificationLocal2015
S. Liao, Y. Hu, Xiangyu Zhu, and S. Z. Li, “Person re-identification by
  Local Maximal Occurrence representation and metric learning,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR), (Boston, MA, USA), pp. 2197–2206, IEEE, June 2015.

liDiversityRegularizedSpatiotemporal2018
S. Li, S. Bak, P. Carr, and X. Wang, “Diversity Regularized Spatiotemporal
  Attention for Video-based Person Re-identification,” Mar. 2018.

liHumanReidentificationTransferred2013
W. Li, R. Zhao, and X. Wang, “Human Reidentification with Transferred
  Metric Learning,” in Computer Vision – ACCV 2012
  (K. M. Lee, Y. Matsushita, J. M. Rehg, and Z. Hu, eds.), Lecture Notes in
  Computer Science, (Berlin, Heidelberg), pp. 31–44, Springer, 2013.

liHybridAttentionNetwork2020
Y. Li, H. Xu, and J. Xiao, “Hybrid Attention Network for Language-Based
  Person Search,” Sensors, vol. 20, p. 5279, Jan. 2020.

liIdentityAwareTextualVisualMatching2017
S. Li, T. Xiao, H. Li, W. Yang, and X. Wang, “Identity-Aware Textual-Visual
  Matching With Latent Co-Attention,” in Proceedings of the IEEE
  International Conference on Computer Vision, pp. 1890–1899, 2017.

liJointTokenFeature2022a
S. Li, A. Lu, Y. Huang, C. Li, and L. Wang, “Joint Token and Feature
  Alignment Framework for Text-Based Person Search,” IEEE Signal
  Processing Letters, vol. 29, pp. 2238–2242, 2022.

liLearningSemanticAlignedFeature2021
S. Li, M. Cao, and M. Zhang, “Learning Semantic-Aligned Feature
  Representation for Text-based Person Search,” Dec. 2021.

linImprovingPersonReidentification2019
Y. Lin, L. Zheng, Z. Zheng, Y. Wu, Z. Hu, C. Yan, and Y. Yang, “Improving
  person re-identification by attribute and identity learning,” Pattern
  Recognition, vol. 95, pp. 151–161, Nov. 2019.

linPersonSearchChallenges2021a
X. Lin, P. Ren, Y. Xiao, X. Chang, and A. Hauptmann, “Person search challenges
  and solutions: A survey,” in Proceedings of the Thirtieth
  International Joint Conference on Artificial Intelligence,
  pp. 4500–4507, Association for the Advancement of Artificial Intelligence
  (AAAI), 2021.

liPedestrianReidentificationBased2021
C. Li, X. Yang, K. Yin, Y. Chang, Z. Wang, and G. Yin, “Pedestrian
  re-identification based on attribute mining and reasoning,” IET Image
  Processing, vol. 15, no. 11, pp. 2399–2411, 2021.

liPersonSearchNatural2017
S. Li, T. Xiao, H. Li, B. Zhou, D. Yue, and X. Wang, “Person Search With
  Natural Language Description,” in Proceedings of the IEEE
  Conference on Computer Vision and Pattern Recognition,
  pp. 1970–1979, 2017.

liPersonTextImageMatching2022
F. Li, H. Zhou, H. Li, Y. Zhang, and Z. Yu, “Person Text-Image Matching
  via Text-Feature Interpretability Embedding and External Attack Node
  Implantation,” 2022.

liRichlyAnnotatedDataset2016a
D. Li, Z. Zhang, X. Chen, H. Ling, and K. Huang, “A Richly Annotated
  Dataset for Pedestrian Attribute Recognition,” Apr. 2016.

liRichlyAnnotatedPedestrian2019
D. Li, Z. Zhang, X. Chen, and K. Huang, “A Richly Annotated Pedestrian
  Dataset for Person Retrieval in Real Surveillance Scenarios,” IEEE Transactions on Image Processing, vol. 28, pp. 1575–1590, Apr. 2019.

liTransformerBasedLanguagePersonSearch2021
H. Li, J. Xiao, M. Sun, E. Lim, and Y. Zhao, “Transformer-Based
  Language-Person Search With Multiple Region Slicing,” IEEE
  Transactions on Circuits and Systems for Video Technology, Apr. 2021.

liUAVHumanLargeBenchmark2021
T. Li, J. Liu, W. Zhang, Y. Ni, W. Wang, and Z. Li, “UAV-Human: A Large
  Benchmark for Human Behavior Understanding with Unmanned Aerial
  Vehicles,” Aug. 2021.

liuDeepAdversarialGraph2019
J. Liu, Z.-J. Zha, R. Hong, M. Wang, and Y. Zhang, “Deep Adversarial Graph
  Attention Convolution Network for Text-Based Person Search,” in Proceedings of the 27th ACM International Conference on Multimedia,
  MM '19, (New York, NY, USA), pp. 665–673, Association for Computing
  Machinery, Oct. 2019.

liuEfficientTextbasedPerson2022
T.-Y. Liu, C. Zhu, and L. Yang, “Efficient Text-based Person Search via
  Single-stage Identity-guided Attribute Parsing and Alignment,” 2022 26th International Conference on Pattern Recognition (ICPR),
  pp. 4111–4117, Aug. 2022.

liuGraphStructuredNetwork2020a
C. Liu, Z. Mao, T. Zhang, H. Xie, B. Wang, and Y. Zhang, “Graph Structured
  Network for Image-Text Matching,” in 2020 IEEE/CVF
  Conference on Computer Vision and Pattern Recognition (CVPR),
  pp. 10918–10927, June 2020.

liuHydraPlusNetAttentiveDeep2017
X. Liu, H. Zhao, M. Tian, L. Sheng, J. Shao, S. Yi, J. Yan, and X. Wang,
  “HydraPlus-Net: Attentive Deep Features for Pedestrian
  Analysis,” Sept. 2017.

liuSwinTransformerHierarchical2021
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin
  Transformer: Hierarchical Vision Transformer using Shifted
  Windows,” Aug. 2021.

liuTS2NetTokenShift2022
Y. Liu, P. Xiong, L. Xu, S. Cao, and Q. Jin, “TS2-Net: Token Shift and
  Selection Transformer for Text-Video Retrieval,” in Computer
  Vision – ECCV 2022: 17th European Conference, Tel
  Aviv, Israel, October 23–27, 2022, Proceedings,
  Part XIV, (Berlin, Heidelberg), pp. 319–335, Springer-Verlag, Oct.
  2022.

luoSpectralFeatureTransformation2018
C. Luo, Y. Chen, N. Wang, and Z. Zhang, “Spectral Feature Transformation
  for Person Re-identification,” Nov. 2018.

manningStanfordCoreNLPNatural2014
C. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. Bethard, and D. McClosky,
  “The Stanford CoreNLP Natural Language Processing Toolkit,” in Proceedings of 52nd Annual Meeting of the Association for
  Computational Linguistics: System Demonstrations, (Baltimore,
  Maryland), pp. 55–60, Association for Computational Linguistics, June
  2014.

mathurBriefSurveyDeep2020
N. Mathur, S. Mathur, D. Mathur, and P. Dadheech, “A Brief Survey of
  Deep Learning Techniques for Person Re-identification,” in 2020
  3rd International Conference on Emerging Technologies in Computer
  Engineering: Machine Learning and Internet of Things
  (ICETCE), pp. 129–138, Feb. 2020.

mingDeepLearningbasedPerson2022
Z. Ming, M. Zhu, X. Wang, J. Zhu, J. Cheng, C. Gao, Y. Yang, and X. Wei, “Deep
  learning-based person re-identification methods: A survey and outlook of
  recent works,” Image and Vision Computing, vol. 119, p. 104394, Mar.
  2022.

niExpandingLanguageImagePretrained2022
B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, and H. Ling,
  “Expanding Language-Image Pretrained Models for General Video
  Recognition,” in Computer Vision – ECCV 2022
  (S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, eds.),
  Lecture Notes in Computer Science, (Cham), pp. 1–18, Springer
  Nature Switzerland, 2022.

niuCrossmodalCooccurrenceAttributes2022
K. Niu, L. Huang, Y. Huang, P. Wang, L. Wang, and Y. Zhang, “Cross-modal
  Co-occurrence Attributes Alignments for Person Search by
  Language,” in Proceedings of the 30th ACM International
  Conference on Multimedia, MM '22, (New York, NY, USA),
  pp. 4426–4434, Association for Computing Machinery, Oct. 2022.

niuFusingTwoDirections2019a
K. Niu, Y. Huang, and L. Wang, “Fusing Two Directions in Cross-Domain
  Adaption for Real Life Person Search by Language,” in 2019
  IEEE/CVF International Conference on Computer Vision Workshop
  (ICCVW), pp. 1815–1818, Oct. 2019.

niuImprovingDescriptionBasedPerson2020
K. Niu, Y. Huang, W. Ouyang, and L. Wang, “Improving Description-Based
  Person Re-Identification by Multi-Granularity Image-Text Alignments,”
  IEEE Transactions on Image Processing, vol. 29, pp. 5542–5556, 2020.

niuTextualDependencyEmbedding2020
K. Niu, Y. Huang, and L. Wang, “Textual Dependency Embedding for Person
  Search by Language,” Proceedings of the 28th ACM International
  Conference on Multimedia, pp. 4032–4040, Oct. 2020.

PapersCodeUAVHuman
“Papers with Code - UAV-Human: A Large Benchmark for Human
  Behavior Understanding with Unmanned Aerial Vehicles.”
  https://paperswithcode.com/paper/uav-human-a-large-benchmark-for-human.

parkLearningDiscriminativePart2020a
J. Park, B. Jeong, J. Shin, J. Lee, and S. Kwak, “Learning Discriminative
  Part Features Through Attentions For Effective And Scalable Person
  Search,” in 2020 IEEE International Conference on Image
  Processing (ICIP), pp. 2376–2380, Oct. 2020.

pengDeepLearningBasedOccluded2023
Y. Peng, J. Wu, B. Xu, C. Cao, X. Liu, Z. Sun, and Z. He, “Deep
  Learning-Based Occluded Person Re-Identification: A Survey,” ACM Transactions on Multimedia Computing, Communications, and Applications,
  July 2023.

perwaizUbiquitousVisionTransformers2023
N. Perwaiz, M. Shahzad, and M. M. Fraz, “Ubiquitous vision of transformers for
  person re-identification,” Machine Vision and Applications, vol. 34,
  p. 27, Feb. 2023.

radfordLearningTransferableVisual2021
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry,
  A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning
  Transferable Visual Models From Natural Language Supervision,” in Proceedings of the 38th International Conference on Machine
  Learning, pp. 8748–8763, PMLR, July 2021.

reedLearningDeepRepresentations2016
S. Reed, Z. Akata, B. Schiele, and H. Lee, “Learning Deep Representations
  of Fine-grained Visual Descriptions,” May 2016.

sarafianosAdversarialRepresentationLearning2019
N. Sarafianos, X. Xu, and I. A. Kakadiaris, “Adversarial Representation
  Learning for Text-to-Image Matching,” in Proceedings of the
  IEEE/CVF International Conference on Computer Vision,
  pp. 5814–5824, 2019.

scheirerMultiattributeSpacesCalibration2012
W. J. Scheirer, N. Kumar, P. N. Belhumeur, and T. E. Boult, “Multi-attribute
  spaces: Calibration for attribute fusion and similarity search,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition,
  pp. 2933–2940, June 2012.

schusterBidirectionalRecurrentNeural1997
M. Schuster and K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Transactions on Signal Processing, vol. 45, pp. 2673–2681, Nov. 1997.

shahNaturalLanguagePerson2018
A. Shah and T. Vuong, “Natural Language Person Search Using Deep
  Reinforcement Learning,” Sept. 2018.

shaoLearningGranularityUnifiedRepresentations2022
Z. Shao, X. Zhang, M. Fang, Z. Lin, J. Wang, and C. Ding, “Learning
  Granularity-Unified Representations for Text-to-Image Person
  Re-identification,” in Proceedings of the 30th ACM International
  Conference on Multimedia, (Lisboa Portugal), pp. 5566–5574, ACM,
  Oct. 2022.

shuSeeFinerSee2023
X. Shu, W. Wen, H. Wu, K. Chen, Y. Song, R. Qiao, B. Ren, and X. Wang, “See
  Finer, See More: Implicit Modality Alignment for Text-Based
  Person Retrieval,” in Computer Vision – ECCV 2022
  Workshops (L. Karlinsky, T. Michaeli, and K. Nishino, eds.), Lecture
  Notes in Computer Science, (Cham), pp. 624–641, Springer Nature
  Switzerland, 2023.

singhComprehensiveSurveyPerson2022
N. K. Singh, M. Khare, and H. B. Jethva, “A comprehensive survey on person
  re-identification approaches: Various aspects,” Multimedia Tools and
  Applications, vol. 81, pp. 15747–15791, May 2022.

songDeepMetricLearning2015
H. O. Song, Y. Xiang, S. Jegelka, and S. Savarese, “Deep Metric Learning
  via Lifted Structured Feature Embedding,” Nov. 2015.

sunPartModelsPerson2018
Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, “Beyond Part Models:
  Person Retrieval with Refined Part Pooling (and A Strong
  Convolutional Baseline),” in Proceedings of the European
  Conference on Computer Vision (ECCV), pp. 480–496, 2018.

sunSVDNetPedestrianRetrieval2017
Y. Sun, L. Zheng, W. Deng, and S. Wang, “SVDNet for Pedestrian
  Retrieval,” Aug. 2017.

suoSimpleRobustCorrelation2022a
W. Suo, M. Sun, K. Niu, Y. Gao, P. Wang, Y. Zhang, and Q. Wu, “A Simple
  and Robust Correlation Filtering Method for Text-Based Person
  Search,” in Computer Vision - ECCV 2022 - 17th European
  Conference, Tel Aviv, Israel, October 23-27, 2022,
  Proceedings, Part XXXV (S. Avidan, G. J. Brostow, M. Cissé,
  G. M. Farinella, and T. Hassner, eds.), vol. 13695 of Lecture Notes
  in Computer Science, pp. 726–742, Springer, 2022.

tangImprovingPedestrianAttribute2019
C. Tang, L. Sheng, Z. Zhang, and X. Hu, “Improving Pedestrian Attribute
  Recognition With Weakly-Supervised Multi-Scale Attribute-Specific
  Localization,” Oct. 2019.

vaqueroAttributebasedPeopleSearch2009
D. A. Vaquero, R. S. Feris, D. Tran, L. Brown, A. Hampapur, and M. Turk,
  “Attribute-based people search in surveillance environments,” in 2009
  Workshop on Applications of Computer Vision (WACV),
  pp. 1–8, Dec. 2009.

variorSiameseLongShortTerm2016
R. R. Varior, B. Shuai, J. Lu, D. Xu, and G. Wang, “A Siamese Long
  Short-Term Memory Architecture for Human Re-Identification,” July
  2016.

vinyalsShowTellNeural2015
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and Tell: A Neural
  Image Caption Generator,” in Proceedings of the IEEE Conference
  on Computer Vision and Pattern Recognition, pp. 3156–3164, 2015.

wangAMENAdversarialMultispace2021
Z. Wang, J. Xue, A. Zhu, Y. Li, M. Zhang, and C. Zhong, “AMEN:
  Adversarial Multi-space Embedding Network for Text-Based Person
  Re-identification,” in Pattern Recognition and Computer
  Vision: 4th Chinese Conference, PRCV 2021, Beijing, China,
  October 29 – November 1, 2021, Proceedings, Part
  II, (Berlin, Heidelberg), pp. 462–473, Springer-Verlag, Oct. 2021.

wangCAIBCCapturingAllround2022
Z. Wang, A. Zhu, J. Xue, X. Wan, C. Liu, T. Wang, and Y. Li, “CAIBC:
  Capturing All-round Information Beyond Color for Text-based Person
  Retrieval,” in Proceedings of the 30th ACM International
  Conference on Multimedia, MM '22, (New York, NY, USA),
  pp. 5314–5322, Association for Computing Machinery, Oct. 2022.

wangCrossDomainPersonReidentification2021
Y. Wang, S. Yang, S. Liu, and Z. Zhang, “Cross-Domain Person
  Re-identification: A Review,” in Artificial Intelligence in
  China (Q. Liang, W. Wang, J. Mu, X. Liu, Z. Na, and X. Cai, eds.),
  Lecture Notes in Electrical Engineering, (Singapore), pp. 153–160,
  Springer, 2021.

wangDivideandMergeEmbeddingSpace2021
C. Wang, Z. Luo, Z. Zhong, and S. Li, “Divide-and-Merge the embedding
  space for cross-modality person search,” Neurocomputing, vol. 463,
  pp. 388–399, Nov. 2021.

wangExploitingTextualPotential2023
G. Wang, F. Yu, J. Li, Q. Jia, and S. Ding, “Exploiting the Textual
  Potential from Vision-Language Pre-training for Text-based Person
  Search,” Mar. 2023.

wangImageForeignLanguage2022
W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K.
  Mohammed, S. Singhal, S. Som, and F. Wei, “Image as a Foreign Language:
  BEiT Pretraining for All Vision and Vision-Language Tasks,” Aug.
  2022.

wangIMGNetInnercrossmodalAttentional2020e
Z. Wang, A. Zhu, Z. Zheng, J. Jin, Z. Xue, and G. Hua, “IMG-Net:
  Inner-cross-modal attentional multigranular network for description-based
  person re-identification,” Journal of Electronic Imaging, vol. 29,
  Aug. 2020.

wangImprovingEmbeddingLearning2022
C. Wang, Z. Luo, Y. Lin, and S. Li, “Improving embedding learning by virtual
  attribute decoupling for text-based person search,” Neural Computing
  and Applications, vol. 34, pp. 5625–5647, Apr. 2022.

wangInteractEmbedEnlargE2022
Z. Wang, C. Li, A. Zheng, R. He, and J. Tang, “Interact, Embed, and
  EnlargE: Boosting Modality-Specific Representations for Multi-Modal
  Person Re-identification,” Proceedings of the AAAI Conference on
  Artificial Intelligence, vol. 36, pp. 2633–2641, June 2022.

wangIntramodalitySurveyHeterogeneous2020
Z. Wang, Z. Wang, Y. Zheng, Y. Wu, W. Zeng, and S. Satoh, “Beyond
  Intra-modality: A Survey of Heterogeneous Person
  Re-identification,” Apr. 2020.

wangLanguagePersonSearch2019
Y. Wang, C. Bo, D. Wang, S. Wang, Y. Qi, and H. Lu, “Language Person
  Search with Mutually Connected Classification Loss,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics,
  Speech and Signal Processing (ICASSP), pp. 2057–2061, May 2019.

wangLearningFineGrainedImage2014
J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and
  Y. Wu, “Learning Fine-Grained Image Similarity with Deep Ranking,”
  in 2014 IEEE Conference on Computer Vision and Pattern
  Recognition, (Columbus, OH, USA), pp. 1386–1393, IEEE, June 2014.

wangLookYouLeap2022
Z. Wang, A. Zhu, J. Xue, X. Wan, C. Liu, T. Wang, and Y. Li, “Look Before
  You Leap: Improving Text-based Person Retrieval by Learning A
  Consistent Cross-modal Common Manifold,” Proceedings of the 30th ACM
  International Conference on Multimedia, pp. 1984–1992, Oct. 2022.

wangPartBasedMultiScaleAttention2022a
Y. Wang, D. Qi, and C. Zhao, “Part-Based Multi-Scale Attention Network
  for Text-Based Person Search,” in Pattern Recognition and
  Computer Vision (S. Yu, Z. Zhang, P. C. Yuen, J. Han, T. Tan, Y. Guo,
  J. Lai, and J. Zhang, eds.), Lecture Notes in Computer Science,
  (Cham), pp. 462–474, Springer International Publishing, 2022.

wangPersonReIdentificationCascaded2018
Y. Wang, Z. Chen, F. Wu, and G. Wang, “Person Re-Identification With
  Cascaded Pairwise Convolutions,” in Proceedings of the IEEE
  Conference on Computer Vision and Pattern Recognition,
  pp. 1470–1478, 2018.

wangPoseGuidedFeatureDisentangling2022
T. Wang, H. Liu, P. Song, T. Guo, and W. Shi, “Pose-Guided Feature
  Disentangling for Occluded Person Re-identification Based on
  Transformer,” Proceedings of the AAAI Conference on Artificial
  Intelligence, vol. 36, pp. 2540–2549, June 2022.

wangSUMSerializedUpdating2022
Z. Wang, A. Zhu, J. Xue, D. Jiang, C. Liu, Y. Li, and F. Hu, “SUM:
  Serialized Updating and Matching for text-based person retrieval,”
  Knowledge-Based Systems, vol. 248, p. 108891, July 2022.

wangSurveyPersonReidentification2018
K. Wang, H. Wang, M. Liu, X. Xing, and T. Han, “Survey on person
  re-identification based on deep learning,” CAAI Transactions on
  Intelligence Technology, vol. 3, no. 4, pp. 219–227, 2018.

wangTextbasedPersonSearch2021
C. Wang, Z. Luo, Y. Lin, and S. Li, “Text-based Person Search via
  Multi-Granularity Embedding Learning,” in Proceedings of the
  Thirtieth International Joint Conference on Artificial Intelligence,
  (Montreal, Canada), pp. 1068–1074, International Joint Conferences on
  Artificial Intelligence Organization, Aug. 2021.

wangViTAAVisualTextualAttributes2020a
Z. Wang, Z. Fang, J. Wang, and Y. Yang, “ViTAA: Visual-Textual
  Attributes Alignment in Person Search by Natural Language,” in
  Computer Vision – ECCV 2020 (A. Vedaldi,
  H. Bischof, T. Brox, and J.-M. Frahm, eds.), Lecture Notes in Computer
  Science, (Cham), pp. 402–420, Springer International Publishing, 2020.

weiFineGrainedImageAnalysis2021
X.-S. Wei, Y.-Z. Song, O. Mac Aodha, J. Wu, Y. Peng, J. Tang, J. Yang, and
  S. Belongie, “Fine-Grained Image Analysis with Deep Learning: A
  Survey,” Nov. 2021.

weiPersonTransferGAN2018a
L. Wei, S. Zhang, W. Gao, and Q. Tian, “Person Transfer GAN to Bridge
  Domain Gap for Person Re-identification,” in 2018 IEEE/CVF
  Conference on Computer Vision and Pattern Recognition, pp. 79–88,
  June 2018.

wuDeepLearningbasedMethods2019
D. Wu, S.-J. Zheng, X.-P. Zhang, C.-A. Yuan, F. Cheng, Y. Zhao, Y.-J. Lin,
  Z.-Q. Zhao, Y.-L. Jiang, and D.-S. Huang, “Deep learning-based methods for
  person re-identification: A comprehensive review,” Neurocomputing,
  vol. 337, pp. 354–371, Apr. 2019.

wuLapsCoreLanguageGuidedPerson2021
Y. Wu, Z. Yan, X. Han, G. Li, C. Zou, and S. Cui, “LapsCore:
  Language-Guided Person Search via Color Reasoning,” in Proceedings of the IEEE/CVF International Conference on Computer
  Vision, pp. 1624–1633, 2021.

wuOverviewDeepLearning2022
D. Wu, H. Huang, Q. Zhao, S. Zhang, J. Qi, and J. Hu, “Overview of deep
  learning based pedestrian attribute recognition and re-identification,” Heliyon, vol. 8, p. e12086, Dec. 2022.

xiaoJointDetectionIdentification2016b
T. Xiao, S. Li, B. Wang, L. Lin, and X. Wang, “Joint Detection and
  Identification Feature Learning for Person Search,” Apr. 2016.

xiaoJointDetectionIdentification2017a
T. Xiao, S. Li, B. Wang, L. Lin, and X. Wang, “Joint Detection and
  Identification Feature Learning for Person Search,” in 2017
  IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR), pp. 3376–3385, July 2017.

xuE2EVLPEndtoEndVisionLanguage2021
H. Xu, M. Yan, C. Li, B. Bi, S. Huang, W. Xiao, and F. Huang, “E2E-VLP:
  End-to-End Vision-Language Pre-training Enhanced by Visual
  Learning,” Proceedings of the 59th Annual Meeting of the Association
  for Computational Linguistics and the 11th International Joint Conference on
  Natural Language Processing (Volume 1: Long Papers), pp. 503–513, 2021.

xuMultilevelLanguageVision2019
H. Xu, K. He, B. A. Plummer, L. Sigal, S. Sclaroff, and K. Saenko, “Multilevel
  Language and Vision Integration for Text-to-Clip Retrieval,”
  Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33,
  pp. 9062–9069, July 2019.

yaghoubiSSSPRShortSurvey2021
E. Yaghoubi, A. Kumar, and H. Proença, “SSS-PR: A short survey of
  surveys in person re-identification,” Pattern Recognition Letters,
  vol. 143, pp. 50–57, Mar. 2021.

yanCLIPDrivenFinegrainedTextImage2022
S. Yan, N. Dong, L. Zhang, and J. Tang, “CLIP-Driven Fine-grained Text-Image
  Person Re-identification,” 2022.

yanCrossdomainPersonReidentification2022b
S. Yan, Y. Zhang, M. Xie, D. Zhang, and Z. Yu, “Cross-domain person
  re-identification with pose-invariant feature decomposition and hypergraph
  structure alignment,” Neurocomputing, vol. 467, pp. 229–241, Jan.
  2022.

yangSurveyUnsupervisedTechniques2021a
C. Yang, F. Qi, and H. Jia, “Survey on Unsupervised Techniques for
  Person Re-Identification,” in 2021 2nd International Conference
  on Computing and Data Science (CDS), pp. 161–164, Jan. 2021.

yangUnifiedTextbasedPerson2023a
S. Yang, Y. Zhou, Y. Wang, Y. Wu, L. Zhu, and Z. Zheng, “Towards Unified
  Text-based Person Retrieval: A Large-scale Multi-Attribute and
  Language Search Benchmark,” Aug. 2023.

yanImageSpecificInformationSuppression2022
S. Yan, H. Tang, L. Zhang, and J. Tang, “Image-Specific Information
  Suppression and Implicit Local Alignment for Text-based Person
  Search,” 2022.

yanImageSpecificInformationSuppression2023
S. Yan, H. Tang, L. Zhang, and J. Tang, “Image-Specific Information
  Suppression and Implicit Local Alignment for Text-based Person
  Search,” July 2023.

yanPersonReIdentificationVision2018
F. Yan, J. Kittler, and K. Mikolajczyk, “Person Re-Identification with
  Vision and Language,” in 2018 24th International Conference
  on Pattern Recognition (ICPR), pp. 2136–2141, Aug. 2018.

yeDeepLearningPerson2022
M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S. C. H. Hoi, “Deep
  Learning for Person Re-Identification: A Survey and
  Outlook,” IEEE Transactions on Pattern Analysis and Machine
  Intelligence, vol. 44, pp. 2872–2893, June 2022.

yeSpecificPersonRetrieval2015
M. Ye, C. Liang, Z. Wang, Q. Leng, J. Chen, and J. Liu, “Specific Person
  Retrieval via Incomplete Text Description,” in Proceedings of the
  5th ACM on International Conference on Multimedia Retrieval,
  ICMR '15, (New York, NY, USA), pp. 547–550, Association for Computing
  Machinery, June 2015.

yeVisibleThermalPerson2018
M. Ye, Z. Wang, X. Lan, and P. C. Yuen, “Visible Thermal Person
  Re-Identification via Dual-Constrained Top-Ranking,” in Proceedings of the Twenty-Seventh International Joint Conference on
  Artificial Intelligence, (Stockholm, Sweden), pp. 1092–1099,
  International Joint Conferences on Artificial Intelligence Organization,
  July 2018.

yinAdversarialAttributeimagePerson2018
Z. Yin, W.-S. Zheng, A. Wu, H.-X. Yu, H. Wan, X. Guo, F. Huang, and J. Lai,
  “Adversarial attribute-image person re-identification,” in Proceedings
  of the 27th International Joint Conference on Artificial
  Intelligence, IJCAI'18, (Stockholm, Sweden), pp. 1100–1106, AAAI
  Press, July 2018.

yuCascadeTransformersEndtoEnd2022
R. Yu, D. Du, R. LaLonde, D. Davila, C. Funk, A. Hoogs, and B. Clipp, “Cascade
  Transformers for End-to-End Person Search,” Mar. 2022.

zahraPersonReidentificationRetrospective2023
A. Zahra, N. Perwaiz, M. Shahzad, and M. M. Fraz, “Person re-identification:
  A retrospective on domain specific open challenges and future trends,”
  Pattern Recognition, vol. 142, p. 109669, Oct. 2023.

zellersRecognitionCognitionVisual2019
R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi, “From Recognition to
  Cognition: Visual Commonsense Reasoning,” in Proceedings of the
  IEEE/CVF Conference on Computer Vision and Pattern
  Recognition, pp. 6720–6731, 2019.

zhangDeepCrossModalProjection2018a
Y. Zhang and H. Lu, “Deep Cross-Modal Projection Learning for Image-Text
  Matching,” in Computer Vision – ECCV 2018
  (V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, eds.), Lecture
  Notes in Computer Science, (Cham), pp. 707–723, Springer
  International Publishing, 2018.

zhangGLIPv2UnifyingLocalization2022
H. Zhang, P. Zhang, X. Hu, Y.-C. Chen, L. H. Li, X. Dai, L. Wang, L. Yuan,
  J.-N. Hwang, and J. Gao, “GLIPv2: Unifying Localization and
  Vision-Language Understanding,” in Advances in Neural Information
  Processing Systems, Oct. 2022.

zhangGlobalLocalMultipleGranularity2021
L. Zhang, G. Du, F. Liu, H. Tu, and X. Shu, “Global-Local Multiple
  Granularity Learning for Cross-Modality Visible-Infrared Person
  Reidentification,” IEEE Transactions on Neural Networks and Learning
  Systems, pp. 1–11, 2021.

zhangTextbasedPersonSearch2021
S. Zhang, D. Long, Y. Gao, L. Gao, Q. Zhang, K. Niu, and Y. Zhang, “Text-based
  Person Search in Full Images via Semantic-Driven Proposal
  Generation,” ArXiv, Sept. 2021.

zhaoWeaklySupervisedTextBased2021
S. Zhao, C. Gao, Y. Shao, W.-S. Zheng, and N. Sang, “Weakly Supervised
  Text-Based Person Re-Identification,” in Proceedings of the
  IEEE/CVF International Conference on Computer Vision,
  pp. 11395–11404, 2021.

zhengDiscriminativelyLearnedCNN2018
Z. Zheng, L. Zheng, and Y. Yang, “A Discriminatively Learned CNN Embedding
  for Person Re-identification,” ACM Transactions on Multimedia
  Computing, Communications, and Applications, vol. 14, pp. 1–20, Feb. 2018.

zhengDualpathConvolutionalImageText2020
Z. Zheng, L. Zheng, M. Garrett, Y. Yang, M. Xu, and Y.-D. Shen, “Dual-path
  Convolutional Image-Text Embeddings with Instance Loss,” ACM
  Transactions on Multimedia Computing, Communications, and Applications,
  vol. 16, pp. 51:1–51:23, May 2020.

zhengHierarchicalGumbelAttention2020
K. Zheng, W. Liu, J. Liu, Z.-J. Zha, and T. Mei, “Hierarchical Gumbel
  Attention Network for Text-based Person Search,” Proceedings of
  the 28th ACM International Conference on Multimedia, pp. 3441–3449, Oct.
  2020.

zhengPersonReidentificationMeets2015
L. Zheng, L. Shen, L. Tian, S. Wang, J. Bu, and Q. Tian, “Person
  Re-identification Meets Image Search,” Feb. 2015.

zhengPersonReidentificationPresent2016
L. Zheng, Y. Yang, and A. G. Hauptmann, “Person Re-identification:
  Past, Present and Future,” Oct. 2016.

zhengReIdentificationConsistentAttentive2019
M. Zheng, S. Karanam, Z. Wu, and R. J. Radke, “Re-Identification With
  Consistent Attentive Siamese Networks,” in Proceedings of the
  IEEE/CVF Conference on Computer Vision and Pattern
  Recognition, pp. 5735–5744, 2019.

zhouAttentionBasedNaturalLanguage2017
T. Zhou, M. Chen, J. Yu, and D. Terzopoulos, “Attention-Based Natural
  Language Person Retrieval,” in 2017 IEEE Conference on Computer
  Vision and Pattern Recognition Workshops (CVPRW), pp. 27–34, July
  2017.

zhuDSSLDeepSurroundingsperson2021
A. Zhu, Z. Wang, Y. Li, X. Wan, J. Jin, T. Wang, F. Hu, and G. Hua, “DSSL:
  Deep Surroundings-person Separation Learning for Text-based Person
  Retrieval,” in Proceedings of the 29th ACM International
  Conference on Multimedia, MM '21, (New York, NY, USA),
  pp. 209–217, Association for Computing Machinery, Oct. 2021.

zhuDualCrossAttentionLearning2022
H. Zhu, W. Ke, D. Li, J. Liu, L. Tian, and Y. Shan, “Dual Cross-Attention
  Learning for Fine-Grained Visual Categorization and Object
  Re-Identification,” in 2022 IEEE/CVF Conference on Computer
  Vision and Pattern Recognition (CVPR), pp. 4682–4692, June 2022.

zhuIdentityGuidedHumanSemantic2020
K. Zhu, H. Guo, Z. Liu, M. Tang, and J. Wang, “Identity-Guided Human
  Semantic Parsing for Person Re-identification,” Computer Vision
  – ECCV 2020, vol. 12348, pp. 346–363, 2020.

zouPersonReidentificationBased2021
G. Zou, G. Fu, X. Peng, Y. Liu, M. Gao, and Z. Liu, “Person re-identification
  based on metric learning: A survey,” Multimedia Tools and
  Applications, vol. 80, pp. 26855–26888, July 2021.


dengPedestrianAttributeRecognition2014
Y. DENG, P. Luo, C. C. Loy, and X. Tang, “Pedestrian Attribute Recognition
  At Far Distance,” in Proceedings of the 22nd ACM International
  Conference on Multimedia, MM '14, (New York, NY, USA),
  pp. 789–792, Association for Computing Machinery, Nov. 2014.

hirzerPersonReidentificationDescriptive2011
M. Hirzer, C. Beleznai, P. M. Roth, and H. Bischof, “Person re-identification
  by descriptive and discriminative classification,” in Proceedings of
  the 17th Scandinavian Conference on Image Analysis, SCIA'11,
  (Berlin, Heidelberg), pp. 91–102, Springer-Verlag, May 2011.

laynePersonReidentificationAttributes2012
R. Layne, T. M. Hospedales, and S. Gong, “Person Re-identification by
  Attributes,” in British Machine Vision Conference, BMVC
  2012, Surrey, UK, September 3-7, 2012, pp. 1–11, BMVA Press,
  2012.

liDeepReIDDeepFilter2014
W. Li, R. Zhao, T. Xiao, and X. Wang, “DeepReID: Deep Filter Pairing
  Neural Network for Person Re-identification,” in Proceedings of
  the 2014 IEEE Conference on Computer Vision and Pattern
  Recognition, CVPR '14, (USA), pp. 152–159, IEEE Computer Society,
  June 2014.

linImprovingPersonReidentification2019
Y. Lin, L. Zheng, Z. Zheng, Y. Wu, Z. Hu, C. Yan, and Y. Yang, “Improving
  person re-identification by attribute and identity learning,” Pattern
  Recognition, vol. 95, pp. 151–161, Nov. 2019.

wangPedestrianAttributeRecognition2019
X. Wang, S. Zheng, R. Yang, B. Luo, and J. Tang, “Pedestrian Attribute
  Recognition: A Survey,” Jan. 2019.

xiongPersonReIdentificationUsing2014
F. Xiong, M. Gou, O. Camps, and M. Sznaier, “Person Re-Identification Using
  Kernel-Based Metric Learning Methods,” in Computer Vision
  – ECCV 2014 (D. Fleet, T. Pajdla, B. Schiele, and
  T. Tuytelaars, eds.), (Cham), pp. 1–16, Springer International
  Publishing, 2014.

  caoOpenPoseRealtimeMultiPerson2019
  Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, “OpenPose:
    Realtime Multi-Person 2D Pose Estimation using Part Affinity
    Fields,” May 2019.
  
  panTwoOnceEnhancing2020
  X. Pan, P. Luo, J. Shi, and X. Tang, “Two at Once: Enhancing Learning
    and Generalization Capacities via IBN-Net,” Mar. 2020.
  
  radfordLearningTransferableVisual2021a
  A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry,
    A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning
    Transferable Visual Models From Natural Language Supervision,” Feb.
    2021.
  
  rombachHighResolutionImageSynthesis2022
  R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer,
    “High-Resolution Image Synthesis with Latent Diffusion Models,”
    Apr. 2022.
  
  ronnebergerUNetConvolutionalNetworks2015
  O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional
    Networks for Biomedical Image Segmentation,” May 2015.

  zhangadding2023
  L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to
      text-to-image diffusion models,” arXiv preprint arXiv:2302.05543,
      2023.
    
  brown2020language
    T. B. Brown, “Language models are few-shot learners,” arXiv preprint
      arXiv:2005.14165, Jul 2020.
    
  hertz2022prompt
    A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or,
      “Prompt-to-prompt image editing with cross attention control,” arXiv
      preprint arXiv:2208.01626, Aug 2022.
    
  zhang2022realgait
    S. Zhang, Y. Wang, T. Chai, A. Li, and A. K. Jain, “Realgait: Gait recognition
      for person re-identification,” arXiv preprint arXiv:2201.04806, 2022.






§ APPENDIX

This appendix provides tables and figures that supplement the data and analyses presented in the main body of the paper. 
These tables and figures provide more detailed information and further support the findings discussed. 
They are referenced throughout the paper where relevant.


  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
      

 
  
  
  


  









 






















