Paper Title: Multimodal Fusion and Coherence Modeling for Video Topic Segmentation.


Few-shot Defect Image Generation based on Consistency Modeling
    Qingfeng Shi1,2 0009-0000-8480-9314Jing Wei1,2 0000-0001-7697-8070Fei Shen1,2,3()0000-0001-9263-4489 Zhengtao Zhang1,2,30000-0003-1659-7879
    August 5, 2024
===============================================================================================================================================

§ ABSTRACT 
The video topic segmentation (VTS) task segments videos into intelligible, non-overlapping topics, facilitating efficient comprehension of video content and quick access to specific content. VTS is also critical to various downstream video understanding tasks. Traditional VTS methods using shallow features or unsupervised approaches struggle to accurately discern the nuances of topical transitions. Recently, supervised approaches have achieved superior performance on video action or scene segmentation over unsupervised approaches. In this work, we improve supervised VTS by thoroughly exploring multimodal fusion and multimodal coherence modeling. Specifically, (1) we enhance multimodal fusion by exploring different architectures using cross-attention and mixture of experts. (2) To generally strengthen multimodality alignment and fusion, we pre-train and fine-tune the model with multimodal contrastive learning. (3) We propose a new pre-training task tailored for the VTS task, and a novel fine-tuning task for enhancing multimodal coherence modeling for VTS. We evaluate the proposed approaches on educational videos, in the form of lectures, due to the vital role of topic segmentation of educational videos in boosting learning experiences. Additionally, we introduce a large-scale Chinese lecture video dataset to augment the existing English corpus, promoting further research in VTS. Experiments on both English and Chinese lecture datasets demonstrate that our model achieves superior VTS performance compared to competitive unsupervised and supervised baselines[The code and model checkpoints will be released upon acceptance].




§ INTRODUCTION

The proliferation of digital video content over the last few decades has underscored the importance of efficient content navigation and comprehension. As the unstructured nature of videos poses significant challenges for users seeking to quickly grasp or reference specific topics, Video Topic Segmentation (VTS) has emerged as a vital tool in addressing these demands. By delineating videos into coherent non-overlapping topics, VTS not only facilitates intuitive understanding of video content but also enables swiftly pinpointing and accessing topics of interest. This is particularly pertinent for the furtherance of various video understanding tasks, where VTS serves as a foundational component.

Traditional VTS techniques predominantly hinge on shallow features <cit.> and unsupervised methods <cit.> due to the scarcity of labeled data. These methods often fall short in capturing the semantic cues that signal topical shifts in video streams, hence suffer from limited precision. Recent advancements in supervised learning paradigms, benefiting from the availability of large-scale labeled data, have achieved notable performance improvements in various video segmentation tasks, such as video action segmentation <cit.>, scene segmentation <cit.>, and topic segmentation <cit.>, surpassing unsupervised methods. Performance can be further enhanced by pre-training on vast volumes of unlabeled data <cit.> or initializing models from pre-trained models <cit.> and then fine-tuning the model. Hence, in this work, we focus on further improving supervised methods for VTS. 

Compared to text topic segmentation <cit.>, videos contain rich and diverse multimodal contextual information. Fully utilizing multimodal information, such as visual cues and textual data (e.g., screen text and subtitles), could facilitate more detailed content understanding and in turn more accurate semantic segmentation than relying on text only. Also, coherence is essential for understanding logical structures and semantics. Enhancing coherence modeling has achieved significant improvements in long document topic segmentation <cit.>. Therefore, we improve supervised VTS methods by thoroughly exploring multimodal fusion and multimodal coherence modeling. We enhance multimodal fusion from the perspectives of model architecture and pre-training and fine-tuning tasks. Specifically, we compare various multimodal fusion architectures built upon Cross-Attention and Mixture-of-Experts (MoE). We investigate the effect of multi-modal contrastive learning for general pre-training and fine-tuning for strengthening cross-modal alignment. For enhancing multimodal coherence modeling, we propose a new pre-training task tailored for the VTS task, and a novel fine-tuning task by elevating intra-topic multimodal feature similarity and inter-topic multimodal feature differences. The proposed approaches are extensively evaluated on educational videos, in the form of lectures, due to the pivotal contributions of topic segmentation of educational videos in bolstering the learning experiences.

Our contributions can be summarized as follows.

    
  * We propose a supervised multimodal sequence labeling model for VTS, denoted MMVTS model. We compare various multimodal fusion architectures.
    In addition to applying multimodal contrastive learning for strengthening cross-modal alignment, we propose a new self-supervised pre-training task tailored to the VTS task
    and a novel fine-tuning task for enhancing multimodal coherence modeling.
    
  * We introduce a large-scale Chinese Lecture Video Topic Segmentation dataset (CLVTS) to promote the research of VTS.
    
  * Experiments show that our model sets new state-of-the-art (SOTA) performance on both English and Chinese lecture video datasets, outperforming competitive unsupervised and supervised baselines. Comprehensive ablation study further confirms the effectiveness of our approaches.






§ RELATED WORK

Text Topic Segmentation
The objective of text topic segmentation is to automatically partition text into topically consistent, non-overlapping segments <cit.>. By automatically mining clues of topic shifts from large amounts of labeled data <cit.>, contemporary supervised models <cit.> have demonstrated superior performance compared to unsupervised approaches <cit.>. Notably, supervised models that excel at modeling long sequences <cit.> are capable of capturing longer contextual nuances and thereby achieve better topic segmentation performance, compared to models that model local sentence pairs or block pairs <cit.>. In addition, recent works <cit.> have demonstrated that strengthening coherence modeling can improve the topic segmentation performance. Therefore, in this work, we explore enhancing coherence modeling for video topic segmentation under the multimodal configurations. 



Video Topic Segmentation
Distinct from text topic segmentation, video topic segmentation is inherently characterized by its multimodal data context; hence, comprehensive utilization of the multimodal information could improve semantic understanding capabilities of a model. Specifically,  <cit.> introduce an unsupervised method that employs the TWFINCH algorithm <cit.> to cluster video clips into distinct topics based on learned visual and text features, which are derived from self-supervised tasks of matching the narration with the temporally aligned visual content.  <cit.> simply concatenate the sampled encoded visual features and text embedding as the input to a pre-trained language model, which may cause inferior fine-tuning due to the discrepancy between the pre-training stage of the language model and the fine-tuning stage combining textual and visual inputs. Other works related to VTS are summarized and compared in Appendix <ref>.




§ THE CLVTS DATASET


Collection Procedure
The CLVTS dataset is primarily sourced from Public Video Platforms[<https://study.163.com/>][<https://www.bilibili.com/v/knowledge>]. Videos are first transcribed by a competitive automatic speech recognition (ASR) system[<https://tingwu.aliyun.com/home>]. We then ask annotators to combine visual and textual (ASR 1-best) information to mark the timestamp (in seconds) of the end of each topic. We ensure high accuracy and reliability of annotations from three aspects, including annotator training, hierarchical topic labeling, and a multi-annotator strategy. Details of these strategies and the rigorous quality assessment and quality control we conduct can be found in Appendix <ref>.

Dataset Statistics
To measure the extent of consensus for VTS annotations, following <cit.>, we compute the F_1@k score (described in Appendix <ref>) based on the absolute distance between two topic boundary sequences, varying the threshold k from 0 to 8 seconds with a step size of 2 seconds, where 8 seconds are approximately the average duration of a clip. By averaging the F_1 scores across all three pairs of annotated topic boundaries from three annotators on the same video, we obtain the consistency score. The more similar the annotations from all annotators on the same video are, the higher the consistency score is. Figure <ref> shows that the consistency scores of the majority of videos exceed 0.5, indicating a decent degree of consensus for VTS annotations <cit.>.

Table <ref> compares our CLVTS dataset against existing VTS datasets. Among annotated videos in CLVTS, 47% are presentations showing slides, 34% are blackboard demonstrations, and 19% are miscellaneous types. We also collect 1027 hours of unlabeled videos from the same sources for pre-training. Besides the linguistic distinctness, CLVTS is characterized by its natural and uninterrupted long videos, a stark contrast to the English lecture dataset, AVLecture <cit.>, since nearly two-thirds of AVLecture are reassembled pre-segmented short videos. Figure <ref> and <ref> show a diverse distribution of video and topic durations and a broad spectrum of subjects in CLVTS.



§ METHODOLOGY

Figure <ref> depicts the architecture of our VTS model. Section <ref> presents the problem definition of multimodal VTS and the overall architecture. We enhance multimodal fusion from the perspectives of model architecture, pre-training, and fine-tuning tasks. Specifically, we compare different architectures built upon Merge- and Cross-Attention, and Mixture-of-Experts for multimodal fusion (Section <ref>). We explore multimodal contrastive learning for cross-modality alignment and propose a new pre-training task tailored for VTS (Section <ref>). For fine-tuning, we also propose a novel task for multimodal coherence modeling (Section <ref>).






 §.§ Multi-Modal Video Topic Segmentation




  
Overall Architecture Following prior works <cit.>, we define video topic segmentation as a clip-level sequence labeling task and propose our MultiModal Video Topic Segmentation (MMVTS) model. As illustrated in Figure <ref>, we apply unimodal pre-trained encoders for each modality and then fuse multimodal information at the intermediate representation level (i.e., `middle fusion' <cit.>) through Multimodal Fusion Layers. Given a video, we transcribe it with a competitive automatic speech recognition (ASR) system[<https://tingwu.aliyun.com/home>] and use ASR 1-best as the text modality. We then divide the video into n clips (c^v_i, c^t_i)_i=1^n, with clips segmented at the sentence boundaries predicted on ASR 1-best. c^v_i = {f^i_1,..., f^i_k} denotes evenly sampled k frames from the i-th clip and is fed to a visual encoder E_v to extract visual features. c^t_i = {w^i_1,...,w^i_s_i+1} denotes the sequence of words from ASR 1-best within the i-th clip, where w^i_1 is the inserted special token [BOS] and s_i denotes the number of words of i-th clip. c^t_i is fed to a text encoder E_t for extracting textual features. After extracting unimodal features, we first apply trainable projection matrices W_v and W_t to convert unimodal features into the same dimension, as visual feature sequence v={v_1,...,v_n} and textual feature sequence t={t_1,...,t_n} (Eq. <ref>). Then we fuse the multimodal information with M Multimodal Fusion Layers MFL_M and obtain the updated visual features h^v={h^v_1,...,h^v_n} and textual features h^t={h^t_1,...,h^t_n} (Eq. <ref>), which are then concatenated into the multimodal features m={m_1,...,m_n} (Eq. <ref>). Finally, the multimodal features m are fed into the predictor composed of a linear layer W_p to obtain the probability of binary classification p={p_1,...,p_n} (Eq. <ref>), where p_i indicates whether the i-th clip is at the boundary of the topic that it belongs to. We use the standard binary cross-entropy loss (Eq. <ref>) to train the model, where y_i∈{0, 1} is the label. The last clip is excluded from loss computation.

Compared to `late fusion' where no cross-modal interaction happens until after the classifiers, middle fusion and `early fusion' are found to generally outperform late fusion <cit.>, probably because early and middle fusion aligns better with human perception where multimodal fusion happens early in sensory processing. On the other hand, compared to early fusion, middle fusion yields superior or comparable performance <cit.> and is much less computationally expensive since we could freeze partially strong pre-trained unimodal encoders and only train a small number of parameters in the Multimodal Fusion Layers.


    v = W_v·E_v({c^v_1,...,c^v_n})
    t = W_t·E_t({c^t_1,...,c^t_n})


    h^v; h^t = MFL_M(v;t)


    m_i = h^v_i;h^v_i


    p = W_p·m


    l_vts = -∑_i=1^n-1[y_iln p_i+(1-y_i)ln(1-p_i)]




  
Multimodal Fusion Layer (MFL) We compare four different cross-modal interaction mechanisms for multimodal fusion layers. We investigate the Merge-Attention and Co-Attention multimodal fusion layers proposed in  <cit.>. With Merge-Attention (Figure <ref>), features from unimodal encoders are sequentially concatenated and then input into a standard transformer encoder layer <cit.>, which shares attention parameters across modalities. A feed forward layer is added on top to produce the final output representation. In contrast, with Co-Attention (Figure <ref>),  features from each unimodal encoder first go through self-attention with modality-specific attention parameters, then we perform symmetrical cross-attention to integrate information from all other modalities to enhance the representation of the considered modality, followed by a feed forward layer.

Inspired by <cit.>, which interleaves MoE encoder layers and standard dense encoder layers for image-text multimodal models, we also investigate two new architectures by replacing the traditional single feed-forward layers in Figure <ref> and <ref> with a Mixture-of-Experts (MoE) <cit.> module respectively, depicted by Figure <ref> and <ref>. The motivation is that adding MoE on top of the fused representations may facilitate deeper cross-modal integration of information and improve model capacity without a proportional increase in computational complexity. Specifically, experts are MLPs activated depending on the input. Firstly, we concatenate fused features output from self-attention or cross-attention. Then, we implement the Noisy Top-k Gating mechanism <cit.> to select K experts from a total of E candidates (Eq. <ref> - <ref>), where SN() denotes stand normal distribution, W_n denotes tunable Gaussian noise to help load balancing, W_g is a trainable weight matrix, K and E are hyper-parameters. Finally, the outputs of the K activated experts are linearly combined according to the learned gating weights  (Eq. <ref>). For the MoE training objective, we sum importance loss and load loss <cit.> to balance expert utilization as Eq. <ref>.


    G(x) = Softmax(KeepTopK(H_x, k))


    H(x)_i = (W_g· x)_i + SN() · Softplus((W_n· x)_i)


    KeepTopK(x, k)_i =
        
            x_i if x_i is in top-k.
    
            -∞ otherwise.


    MoE(x) = ∑^K_e=1 G(x)_e· MLP_e(x)


    l_balance = l_importance + l_load







 §.§ Pre-training with Unlabeled Data


Prior works have demonstrated that standard self-supervised denoising pre-training (even using only the downstream task data) <cit.> or pre-training adapted to the downstream task <cit.> often perform substantially better than randomly initializing the parameters. Therefore, to better initialize the parameters of the Multimodal Fusion Layers, we explore pre-training with unlabeled video data before fine-tuning on the labeled training set. Firstly, we introduce a general cross-modality alignment pre-training task to learn the multi-modal representation. We use contrastive learning loss to adjust the features learned by the model fusion layer, by maximizing the cosine similarity of the visual features and textual features of the same clip, while reducing the similarity of the modality features between different clips, as show in Eq. <ref>, where ϵ is used to prevent division by 0 and τ is a temperature hyper-parameter to scale the cosine similarity.


    l_cma = -1/n∑_i=1^n e^sim(h^v_i, h^t_i)/∑_i=1^n∑_j=1^n e^sim(h^v_i, h^t_j)+ϵ


    sim(x_1, x_2) = x_1^T· x_2/x_1·x_2 / τ

Secondly, we introduce a novel pre-training task tailored for the downstream VTS task, focusing on utilizing unlabeled data for learning pseudo topic boundaries and also enhancing modality alignment. Initially, a Kernel Density Estimation (KDE) <cit.> model is employed to approximate the topic duration distribution of the training set; then, videos are split into segments matching durations sampled from the KDE-derived distribution. Subsequently, for every resulting segment, we undertake one of three actions with an equal probability: inserting segments randomly selected from other videos either before or after the current segment, replacing the current segment with a segment randomly selected from other videos, or no change. The resulting segments within the modified video are then treated as distinct topics, thereby enabling the model to learn pseudo topic boundaries during pre-training. This task-adaptive pre-training task has the same l_vts objective as shown in Eq. <ref>. The overall pre-training objective is shown in Eq. <ref>, where α and β are hyper-parameters to adjust the loss weights.

    l_pretrain = l_vts + α l_cma + β l_balance




 §.§ Fine-tuning with Multimodal Coherence Modeling


For fine-tuning, we introduce two auxiliary tasks to enhance multimodal coherence modeling. The cross modal alignment task is the same task in Eq. <ref> used in pre-training. This continuity ensures that the modalities retain their coherence through both pre-training and fine-tuning, fostering a consistent interplay between different modalities. 
In addition, we adapt the Contrastive Semantic Similarity Learning (CSSL) task proposed by <cit.>, which leverages the inherent characteristics of topic-related coherence, to the multimodal context. We adopt the same strategy for selecting positive and negative sample pairs <cit.>, but extend the features to the multimodal representations, as shown in Eq. <ref>, where k_1 and k_2 are hyper-parameters that determine the number of positive and negative pairs. For each clip's multimodal representation m_i,  m_i,j^+ denotes the multimodal representation of the j-th similar clip in the same topic as clip i, while m_i,j^- denotes the multimodal representation of the j-th dissimilar clip in a different topic from clip i. We hypothesize that this extension could improve multimodal representation learning by identifying relative consistency relations within and across topics.

    l_mcssl=-1/n∑_i=1^nlog ∑_j=1^k_1 e^sim(m_i, m_i,j^+)/∑_j=1^k_1 e^sim(m_i, m_i,j^+)+∑_j=1^k_2 e^sim(m_i, m_i,j^-)

The overall fine-tuning objective combines Eq. <ref>,  <ref>,  <ref>, and  <ref>,  as shown in Eq. <ref>, where σ, θ, and γ are hyper-parameters to adjust loss contribution. When the multimodl fusion layer does not contains MoE structure, β in Eq. <ref> and σ are set to zero.

    l_finetune=l_vts + σ l_balance + θ l_mcssl + γ l_cma





§ EXPERIMENTS



 §.§ Experimental Setup

Datasets We partition the labeled data within AVLecture into 70% for training, 10% for validation, and 20% for test sets. We also split the CLVTS dataset into train, validation, and test sets with identical ratios.  The unlabeled data of AVLectures and CLVTS are used for pre-training on each dataset, respectively. 

Baselines
We carefully select the following representative baselines:

- UnsupAVLS <cit.> is an unsupervised approach that clusters video clips into discrete topics based on visual and text embeddings learned from matching the narration with temporally aligned visual content. The actual number of topics is used as the desired number of clusters.

- Visual-only BaSSL <cit.> is initially proposed for video scene segmentation. We use their released checkpoints to initialize our model and fine-tune on the VTS task to evaluate the performance of a visual-only model. 

- Text-only LongFormer is evaluated on long document topic segmentation by <cit.>. We fine-tune LongFormer and LongFormer_cssl in <cit.> to evaluate the performance of a text-only model on the VTS task.

- Llama-3-8B[<https://huggingface.co/meta-llama/Meta-Llama-3-8B>] Our pre-training (Section <ref>) is conducted on the relatively limited unlabeled videos of AVLectures and CLVTS datasets. To investigate the effect of fine-tuning a powerful pre-trained text large language model (LLM) on VTS, we fine-tune Llama-3-8B with two prompting methods (see Appendix <ref> for details).

- SWST_seq is our adapted version of the multimodal video scene and topic segmentation model <cit.> to the sequence labeling VTS task, for comparing performance between their early fusion and our middle fusion strategy on VTS. SWST_seq uses a pre-trained LongFormer <cit.> as backbone.

Evaluation Metrics We adopt four commonly used metrics for VTS, including positive F_1 <cit.>(denoted as F_1 to simplify notations), BS@k <cit.>, mIoU <cit.>, and F_1@k. Definitions of the four metrics are in Appendix <ref>. We follow  <cit.> and set k to 30 seconds. Additionally, we use the average score (Avg) of these four metrics to measure the overall performance of a model.

Implementation Details
We use the same maximum sequence length 2048 as  <cit.> for a fair comparison with the text-only models. All results are the mean values over three runs with different random seeds. More implementation details are in Appendix <ref>.



 §.§ Results and Analysis

Table <ref> reports the performance of baselines (the first group) and our MMVTS model variants (the second and third group) on the AVLecture and CLVTS test sets.

Unimodal performance. The text-only Longformer (Row3) substantially outperforms the visual-only BaSSL (Row2) on BS@30 by a large gain (+25.31), and also surpasses the unsupervised approach UnsupAVLS by a notable gain (+13.25). Such results are expected since the textual modality inherently conveys more precise and richer information than the visual modality for VTS. 

Mutimodal performance. As can be seen from Table <ref>, Avg of SWST_seq closely aligns with that of the text-only LongFormer, suggesting that more data may be necessary to mitigate the discrepancy between fine-tuning with early fusion and pre-training the language model, in order to fully exploit the potential of the early fusion strategy. Our MMVTS Baseline_1,2,3 simply concatenate unimodal features to predict topic boundaries. Without pre-training on unlabeled data (PT, Eq. <ref>) and the two auxiliary fine-tuning tasks to enhance multimodal coherence modeling (FT-Coh, Eq. <ref>), on F_1, MMVTS Baseline_1 outperforms the text-only Longformer by 2.28 and 2.90 on AVLecture and CLVTS respectively, while on Avg score,  MMVTS Baseline_1 outperforms Longformer by 1.1 on AVLecture yet slightly underperforms Longformer by 0.65 on CLVTS. These results suggest that simply concatenating unimodal features to predict topic boundaries does not bring consistent gains over unimodality. The third group in Table <ref> compares the four Multimodal Fusion Layer architectures with pre-training (PT) and fine-tuning (FT-Coh). Overall, compared to all the competitive unsupervised and supervised baselines, after pre-training and fine-tuning, our MMVTS model using Co-Attention with MoE as Multimodal Fusion Layers achieves the best Avg, BS@30, and F_1@30 results and sets new SOTA on AVLecture, and achieves the best F_1 and a Avg score comparable to the best result on CLVTS. The gains on Avg from our Co-Attention MoE with PT and FT-Coh over LongFormer_cssl and MMVTS Baseline_1 are statistically significant (p-values are less than 0.05). It is also notable that the performance of models on CLVTS is generally much lower than that on AVLecture, with the best Avg score between AVLecture and CLVTS differing by 17.50 (68.84 versus 51.34), indicating a greater challenge posed by our CLVTS dataset than the AVLecture dataset.

Table <ref> also shows that on CLVTS, fine-tuning the powerful pre-trained Llama-3-8B with our Discrete prompt achieves the best Avg score 51.34, although very close to the performance of Merge-Attn with MoE and Co-Attn with MoE after PT and FT-Coh. However, on AVLecture, the performance of fine-tuning Llama-3-8B is much lower than Multimodal Fusion Layer with PT and FT-Coh; particularly, F_1 on both AVLecture and CLVTS are much lower because the model's predicted boundaries often have a clip offset, but its F_1@30 and mIoU results are still best on CLVTS. 


We conduct ablation studies and analysis to validate effectiveness of the proposed multimodal fusion layers, pre-training and fine-tuning tasks.


(1) Effect of Multimodal Fusion Layers. Comparing the third group and Baseline_3 in Table <ref> shows that with the same pre-training and fine-tuning, the best performing architecture using Multimodal Fusion Layers always substantially outperforms Baseline_3. Specifically, with PT and FT-Coh, on AVLecture, both Co-Attention and Co-Attention with MoE notably outperform MMVTS Baseline_3 by 1.32 on Avg; on CLVTS, all four Multimodal Fusion Layer architectures achieve remarkable gains on Avg over MMVTS Baseline_3, from 2.65 to 4.18. These results demonstrate that deep cross-modal interaction has notable advantage for multimodal fusion over simple unimodal feature concatenation for VTS. Moreover, adding MoE on top consistently improves Co-Attention on both AVLecture and CLVTS, by 0.47 and 1.07 on Avg; whereas, the effect of MoE on top of Merge-Attention is inconsistent, with a slight degradation on AVLecture and 1.53 gain on Avg on CLVTS. In addition, we conduct more analysis of the effect of Co-Attn with MoE with different numbers of multimodal fusion layers in Appendix <ref>.

(2) Effect of Pre-training tasks. Table <ref> shows that for simple concatenation of unimodal features, pre-training before fine-tuning (as Baseline_3) outperforms Baseline_2 (w/o PT). We conduct ablation studies of the proposed pre-training and fine-tuning tasks on the AVLecture dataset. We apply the same fine-tuning with multimodal coherence modeling (FT-Coh, Eq. <ref>) and compare (a) random initialization of parameters of Multimodal Fusion Layers (w/o pre-training) (b) pre-training the model on unlabeled data. Table <ref> shows that w/o pre-training,  Co-Attn slightly improves Avg over Merge-Attn by 0.4 pts, and MoE further improves Avg by 0.87 pts. Pre-training improves the performance on all four Multimodal Fusion Layer architectures, with the average Avg score increased by 1.36 pts (66.29 → 67.65). Pre-training also improves stability of a model as the standard deviations of all w/ PT experiments are less than 1. Table <ref> in Appendix further compares the fine-tuning performance after applying different pre-training tasks. Compared to using both pre-training tasks, removing loss l_cma or l_vts leads to Avg reduction of 0.4 and 1.64 respectively,  indicating that the pretraining task more closely aligned with the downstream task, i.e., l_vts, yields greater benefits.


(3) Effect of Fine-tuning tasks. Table <ref> compares the efficacy of different fine-tuning tasks after pre-training. Compared to the standard l_vts, adding the two auxiliary losses l_cma and l_mcssl notably improves  Avg by 1.12 pts, while decreases F_1 by 0.73. 

These results suggest that while multimodal coherence modeling may slightly compromise the precision of exact matches, it enhances the overall contextual comprehension of a model for VTS. Adding l_cma or l_mcssl individually improves Avg by 0.53 and 0.57 respectively, with improvements mainly on BS@30 and mIoU, suggesting that feature alignment at different granularities may improve fuzzy matching of a model. 



§ CONCLUSION

In conclusion, we propose a supervised MultiModal Video Topic Segmentation (MMVTS) model to advances VTS by meticulously investigating multimodal fusion and coherence modeling, coupled with an innovative pre-training task tailored for VTS. Furthermore, we introduce a large-scale Chinese Lecture Video Topic Segmentation (CLVTS) dataset to promote research in VTS.




§ LIMITATIONS

It is essential to acknowledge that our methodology solely capitalizes on the multimodal information derived from visual and textual data, with the parameters of the visual encoder fixed due to consideration about computational complexity. This specific limitation potentially results in a less-than-optimal exploitation of the vast visual information contained within the dataset. Concurrently, features from additional modalities, such as audio, are not incorporated. Additionally, we will conduct more investigations and study approaches to make Co-Attn with MoE more stable in future work. Finally, the integration of general multimodal pre-trained models and large language models could be investigated, offering a more holistic and effective exploration of multimodal information for VTS.












§ RELATED WORK TO VTS


 <cit.> utilized visual, audio and textual information, and focus on hierarchical modeling techniques to simultaneously learn the boundaries of scenes, stories and topics, which suggests that the segmentation can be benefited by introducing the hierarchical structures of videos. Nevertheless, their approach did not delve into optimizing the integration of multimodal information, which is one of the motivations of our work. The integration of audio features represents a direction for our future work. <cit.> encode the image and text sequences with respective encoders and use an asymmetric cross-modal cross-attention mechanism to produce text-aware visual representations, followed by a BiLSTM, to fuse multimodal features. Our approaches are drastically different from  <cit.> as we investigate symmetric Cross-Modal Cross-Attention and also Mixture-of-Experts mechanism to fuse multimodal information. <cit.> also investigate cross-modal and sentence level intra-modal contrastive learning to empower unsupervised domain adaptation, whereas we explore extending the topic level Contrastive Semantic Similarity Learning (CSSL) task proposed by <cit.> to the multimodal setting.



§ DETAILS OF ANNOTATIONS FOR THE CLVTS DATASET



For the CLVTS dataset, we ensure high accuracy and reliability of annotations from three aspects. 
Firstly, before the actual annotation process, the annotators take two rounds of training. Each annotator needs to annotate 5 videos in each round; at the end of each round, we assess the annotation quality, provide feedback, and ensure that the annotators address the issues and understand the annotation guideline clearly at the end of training. Secondly, during annotation, to help the annotators thoroughly understand the course content, we ask the annotators to annotate topics hierarchically, that is, they label both coarse-grained topic (large topic) and fine-grained topic (small topic) boundaries while the large topic boundaries are a subset of the small topic boundaries. We take the small topic boundaries as the final topic boundary labels. 
Thirdly, we employ a multi-annotator strategy. All data is annotated in batches, with two annotators annotating each sample independently. The third annotator reviews the annotations by the first two annotators, rectifies errors, and provides the final annotations. After the annotations of each batch, we randomly select 5% of a batch for quality assessment. If the unacceptable rate (the proportion of the wrongly annotated topic boundaries) is less than 10%, the data are deemed satisfactory and accepted; otherwise, after communicating quality assessment results and possible reasons for errors, the annotators are requested to re-annotate the batch based on the feedback. This quality control process is repeated until the unacceptable rate is lower than 10% for all batches; in this work, we could finish quality control within 3 iterations.



§ TRAINING DETAILS


Our experiments are implemented with transformers package[< https://github.com/huggingface/transformers>]. For video with the number of clips greater than the max sequence length, we use sliding window and take the last clip of the prior sample as the start clip of the next sample. All supervised models use a threshold strategy, where clips with scores above a threshold 0.5 are predicted as topic boundaries. Following <cit.>, for each video clip, we extract three visual feature types: OCR, 2D and 3D. Specifically, the OCR feature is derived by encoding the textual output obtained from the OCR API[<https://help.aliyun.com/zh/viapi/developer-reference/api-sy75xq>] of the clip's central image. This encoding task is performed using the BERT-based sentence transformer model[<https://www.sbert.net/docs/sentence_transformer/pretrained_models.html>], where all-mpnet-base-v2 and paraphrase-multilingual-mpnet-base-v2 models are employed for English and Chinese language experiments, respectively. The 3D feature are extracted using the same video feature extraction pipeline as in  <cit.>, while the 2D feature is extracted by visual encoder of CLIP <cit.> which is pre-trained to predict if an image and a text snippet are paired together. Specifically, the images from AVLecture and CLVTS are processed to extract 2D features using CLIP_ViT-B/16[<https://github.com/openai/CLIP>] and CN-CLIP_ViT-B/16[<https://github.com/OFA-Sys/Chinese-CLIP>], respectively.

After extracting the visual features, we concatenate them as shown in Eq. <ref> to get v_i, which then will be fed into following projection layer. During pre-training and fine-tuning, the parameters of visual encoders are kept fixed. The learning rate is 5e-5 and dropout probability is 0.1. AdamW <cit.> is used for optimization. The batch size is 8 and the epoch for pre-training and fine-tuning is 1 and 5, respectively. The loss weight α and γ for l_cma is 0.5, β and σ for l_balance is 1.0 when MoE is in multimodal fusion layer. θ for l_mcssl is 0.5. k_1 and k_2 of Eq. <ref> is 1 and 3 following <cit.>. We select one multimodal fusion layer to comprehensively compare different types of fusion structure. We also investigate the performance of different number of multimodal fusion layer in Figure <ref>. As for the MoE layer in multimodal fusion layer, we choose 4 candidate experts and activate 2 experts for each input feature, the intermediate size of expert is 3072. Therefore, the total number of trainable parameters is shown in Table <ref>.

    v_i = v^2d_i;v^3d_i;v^ocr_i






§ EVALUATION METRICS


F_1 is a metric used to evaluate the accuracy of text topic segmentation <cit.>, which focus on the performance of exact matching of the positive class and balances precision and recall rate.

BS@k <cit.> is the average number of predicted boundaries matching with the ground truth boundaries within a k second interval, which can be regard as the recall rate based on fuzzy matching.

F_1@k denotes the F_1 score calculated based on matching predicted and ground truth boundaries within k seconds. Considering subjectivity and uncertainty in VTS annotations, we introduce F_1@k as a supplement to BS@k to enabling a more comprehensively assessment of model performance in dealing with ambiguously defined boundaries.

mIoU is commonly used in video action segmentation <cit.> and video scene segmentation <cit.>. While the F_1, BS@k and F_1@k focus on the accuracy of positive predictions (either exact match or fuzzy match), the mIoU measures the overlapping area between predicted and ground truth segments, providing a generalized assessment of how well the model's predicted segments match the actual segments in the segmentation task.

Our implementation of BS@30 draws upon the code publiced by <cit.>[<https://github.com/Darshansingh11/AVLectures/>], while the approach to implement mIoU is guided by <cit.>[<https://github.com/kakaobrain/bassl>]. We have relied on the scikit-learn package[<https://scikit-learn.org/stable/>] to compute  F_1, following the implementation by <cit.>. The definitions provided in the aforementioned sources also inform our implementation of F_1@k.






§ ARTIFACT USE CONSISTENT WITH INTENDED USE


All of our use of existing artifacts is consistent with their intended use, provided that it was specified. For the CLVTS data set we created, its license will be for research purpose only.




§ FINE-TUNE LLAMA-3-8B FOR VIDEO TOPIC SEGMENTATION


Considering the computational complexity and data volume, we employ LoRA <cit.> for fine-tuning Llama-3-8B[<https://huggingface.co/meta-llama/Meta-Llama-3-8B>] with a maximum sequence length of 2048. Our training configuration includes a batch size of 32 and a total of 6 epochs, utilizing a learning rate of 5e-5 and cosine scheduler <cit.>. Table <ref> shows two prompts we used to fine-tuning Llama-3-8B on text data of VTS. We first try the Generative type that <cit.> has experimented with due to the better zero-shot and one-shot performance than the Discriminative type. However, compared with Longformer, the Avg score of Generative prompting Llama-3-8B is lower 8.3 and 6.27 on AVLecture and CLVTS respectively. We suspect that this is due to the issue of sparse labels in binary classification, leading us to refine the Discriminative into the Discrete prompt as outlined in the Table <ref>. Although the F_1 on AVLecture and CLVTS is much lower (the prediction boundary of the model usually has a clip offset), the Avg score of Llama-3-8B on AVLecture is merely 2.26 points behind LongFormer. Moreover, Llama-3-8B_Discrete secures the highest Avg score on CLVTS, surpassing LongFormer by 4.53 points. which suggests that the Discrete prompt is more suitable for stimulating Llama-3-8B and future work can further explore how to better use large language models for VTS.




§ PERFORMANCE OF MFLS ACROSS VARIED LAYER NUMBERS


Utilizing Co-Attention with Mixture of Experts (MoE) as the multimodal fusion layer, we investigated the Avg performance of models featuring various numbers of Multimodal Fusion Layers (MFLs) on AVLecture, as depicted in Figure <ref>. The findings indicate that when the model is fine-tuned directly on labeled data, a single MFL yields the best performance, surpassing the no-layer configuration by 2.34 points. However, adding more layers leads to diminished performance and training instability, particularly noticeable with three layers. By incorporating a pre-training phase followed by fine-tuning through a coherence modeling task, performance enhancements of 4.0, 2.87, 2.37, and 13.51 points are observed, respectively. These results clearly demonstrate the substantial benefits of our pre-training and coherence modeling strategies in boosting the model's effectiveness. Notably, they facilitate the convergence of a model with three layers, achieving optimal results that marginally exceed the performance of a single-layer model by 0.1 points.




 Paper Title: Understanding Public Safety Trends in Calgary through data mining.






Understanding Public Safety Trends in Calgary through data mining


    Zack Dewis
Dept. of Geomatics 

University of Calgary

Calgary, Alberta 

zachary.dewis@ucalgary.ca 

30091912
Apratim Sen
Dept. of Geomatics 

University of Calgary

Calgary, Alberta 

apratim.sen@ucalgary.ca 

30202853
Jeffrey Wong
Dept. of Geomatics 

University of Calgary

Calgary, Alberta 

chunkit.wong@ucalgary.ca 

30200496
Yujia Zhang
Dept. of Geomatics 

University of Calgary

Calgary, Alberta 

yujia.zhang2@ucalgary.ca 

30221253

    August 5, 2024
================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
This paper utilizes statistical data from various open datasets in Calgary to to uncover patterns and insights for community crimes, disorders, and traffic incidents. Community attributes like demographics, housing, and pet registration were collected and analyzed through geospatial visualization and correlation analysis. Strongly correlated features were identified using the chi-square test, and predictive models were built using association rule mining and machine learning algorithms. The findings suggest that crime rates are closely linked to factors such as population density, while pet registration has a smaller impact. This study offers valuable insights for city managers to enhance community safety strategies.


Keywords—Statistical data, predictive model, crime, disorder, traffic incidents, community characteristics, demographics, housing, pet registration, geospatial visualization, correlation analysis, chi-square test, association rule mining, machine learning algorithms, population density, community safety strategies




§ INTRODUCTION



 §.§ Research Background and Objectives

With the acceleration of urbanization,community crimes, disorders, and traffic incidents have become increasingly prominent issues, severely impacting residents' quality of life and the sustainable development of cities. Data mining techniques provide new perspectives for addressing these problems. This study, using Calgary, Canada, as an example, aims to utilize multi-source urban data including streetlights, trees, traffic accidents, crimes, pet registrations, census data, community security, and traffic camera data to explore the key factors affecting community safety and construct predictive models, providing data support for urban management decisions.
Clustering algorithms such as K-Means and DBSCAN are commonly used in urban data mining, but their clustering effectiveness for complex data needs improvement. This paper proposes to introduce advanced clustering algorithms such as CLARANS and CLIQUE, and enhance clustering performance through parameter optimization.


 §.§ Research Questions and Innovations

This study aims to address the following questions:
(1) What are the characteristics of public facilities, population structure, crime, and accident distribution in various communities in Calgary?
(2) Which community factors are significantly correlated with safety issues?
(3) Can effective predictive models for crime, security incidents, and traffic accidents be constructed?

The innovations of this study include:
(1) Aggregating cutting-edge urban data such as streetlights, trees, and pets to deconstruct the multidimensional factors influencing community safety;
(2) Introducing advanced clustering algorithms such as CLARANS and CLIQUE, integrating optimized clustering with predictive analysis;
(3) Establishing predictive models based on real data, providing new insights for fine-grained urban governance.


§ LITERATURE REVIEW

The increasing prevalence of community crimes, disorders, and traffic incidents due to rapid urbanization has become a major concern for city dwellers and urban planners alike. These issues not only affect the quality of life of residents but also pose significant challenges to the sustainable development of cities. In recent years, data mining techniques have emerged as a promising approach to tackle these problems by providing new perspectives and insights. The present study focuses on Calgary, Canada, and aims to identify the key factors influencing community safety by analyzing multi-source urban data and developing predictive models that can support decision-making in urban management <cit.>.

K-Means and DBSCAN are among the most widely used clustering algorithms in urban data mining. However, their performance on complex data sets often leaves room for improvement. To address this issue, the current study explores the use of advanced clustering algorithms, such as CLARANS and CLIQUE, and seeks to optimize their performance through parameter tuning. This work builds upon the foundations laid by Ng and Han <cit.> and Hartigan and Wong <cit.> in the field of clustering algorithms. Moreover, the efficiency analysis of the K-Means algorithm conducted by Kanungo et al. <cit.> offers valuable insights into the data-sensitive nature of the algorithm's running time, which is particularly relevant to the data sets used in this study.

When it comes to analyzing crime data, regression analysis is a commonly used approach. The methods employed in this study draw inspiration from the correlated data regression analysis proposed by Liang and Zeger <cit.>, which is particularly useful when dealing with the inherent correlation issues often encountered in biomedical research data. In addition, Schubert et al.'s <cit.> critical review of the DBSCAN algorithm, which highlights the importance of selecting appropriate parameters for optimal performance, is taken into consideration. The density-based clustering structure of data sets, as demonstrated by Ankerst et al. <cit.> using the OPTICS algorithm, also serves as a guiding principle for the approach adopted in this study.

The methodology of this study is further informed by the work of other researchers in the field of spatial data mining. For instance, Bernasco and Elffers <cit.> provide a comprehensive overview of the statistical analysis of spatial crime data, while Anselin et al. <cit.> demonstrate the application of spatial analysis in crime research. These studies not only introduce new methods for analyzing spatial data but also shed light on the intricate relationship between crime and place.

In the era of big data, data integration presents a significant challenge that must be addressed. Dong and Srivastava <cit.> delve into the progress made by the data integration community in tackling issues such as schema mapping, record linkage, and data fusion when working with large, dynamic, heterogeneous, and varying quality data sources. By incorporating cutting-edge urban data, such as information on streetlights, trees, and pets, this study aims to contribute to the understanding of the multifaceted factors that influence community safety in Calgary.

Although the primary focus of this study is on modeling large-scale, global structures, the principles and techniques discussed by Hand <cit.> in the context of detecting adverse drug reactions in the pharmaceutical industry can also be applied to identify anomalies and outliers in urban crime data. This approach contributes to a more comprehensive understanding of community safety.

While this study makes notable contributions to the field, it is not without limitations. Due to constraints in data availability, the study only covers a limited set of community indicators and spans a relatively short time frame. Future research could benefit from collecting more comprehensive data and conducting long-term spatiotemporal analyses. Additionally, the complex and diverse nature of the causes underlying urban safety issues makes it challenging to establish clear causal relationships and mechanisms based solely on data analysis. To address this, subsequent studies could incorporate qualitative methods, such as expert interviews and case studies, to explore the socioeconomic, institutional, and cultural factors that influence community safety.

This study builds upon existing algorithms and methods from the literature, making improvements and extensions to enable more effective processing and analysis of urban safety data. By combining various clustering and regression analysis techniques into predictive models, this research provides valuable scientific evidence and insights that can inform urban governance and contribute to the development of safer cities. The findings underscore the crucial role that data mining techniques can play in addressing the challenges posed by urbanization and promoting the well-being of city residents.




§ DATA SOURCE AND PREPROCESSING

The data used in this study were obtained from the City of Calgary's open data platform, including 2019 data on streetlights, public trees, traffic incidents, crime statistics, licensed pets, census by community, community disorder statistics, and traffic cameras.



 §.§ Data Cleaning and Geocoding

Despite all of the data being acquired from the same source, data cleaning was still necessary in order to extract interesting patterns from the data and to perform further analysis. Starting off, the data was manually inspected, where any columns that were deemed unnecessary or blank were removed. Next, all the datasets were loading into Panda dataframes for easy further cleaning. At this stage all rows that were duplicate entries to another row in any of the seven datasets were removed. For the columns that had either Na values of missing values, these were modified on a case-by-case basis. If the column has located in the crime dataset, the values were replaced with zero as the assumption was made that this meant there was zero crime in this particular neighbourhood. This occurred for the neighbourhoods that were still being developed in Calgary when the 2019 census was being conducted and therefore any crime that may have occurred was deemed irrelevant to the research goal.  

Another issue that arose during the cleaning process was that not all the datasets provided latitude and longitude coordinates for where each of the attributes was located. This was solved by running a python script to join the datasets and assign each a latitude and longitude to allow for clustering analysis. Some datasets contained both latitude and longitude as well as a community name or identifier, some contained just a community name and some only contained latitude and longitude. The decision was made that each dataset should be given both latitude and longitude coordinates and a community name attached to it. To achieve this, three different methodologies were used depending on what each individual dataset already contained. If the dataset already contained latitude, longitude and a community name, no further action was required. For datasets only containing a community name, the community centroid coordinates were assigned to the data to provide it a latitude and longitude. For datasets only containing latitude and longitude, a community boundary shapefile was used, where a script was then ran to see which community the points intersected and then a name was assigned. 

At this point all of the necessary data cleaning had been performed for analysis to be carried out. Geocoding had also been performed on all of the datasets, allowing for summary statistics to then be calculated to perform correlation and regression analysis. 



 §.§ Feature Engineering and Data Integration


With the data cleaning and geocoding complete, the next step was to aggregate the data to the same level in order to perform analysis. Community level was selected as the resolution for all the data to be aggregated to. The reason for this was that it was the finest resolution that was available to be used without having to make assumptions and convert datasets to finer resolutions than they are provided at. As the community census and crime was only at community level, a more fine solution could not be used for aggregation. 

At the community level, we calculated various indicators such as the number of streetlights, average wattage, total wattage, number of trees, number of traffic incidents, total number of crimes and crime categories, number of community disorder events, total number of registered pets, and number of registered cats and dogs. We also extracted demographic features of communities from census data, such as total population, male-female ratio, number of dwellings, and number of apartments.

Finally, all of these statistical indicators were compiled into a new geopanda dataframe to be used for further analysis. The original dataframes were also maintained and used for cluster analysis as many were at a finer resolution and allowed for clustering to be performed on them. The data is now fully ready for data mining processes to be performed with the goal of being able to extract interesting patterns and insight from it. 



§ METHODOLOGY


To best extract patterns from the dataset, several exploratory data analyses and different data mining techniques were used in hopes of extracting interesting patterns. This includes data summarization, visualization, temporal analyses, clustering, correlation, regression, and association rule mining. 



 §.§ Exploratory Data Analysis

Exploratory Data Analysis (EDA) is a crucial step in data analysis, especially when dealing with new datasets. It involves exploring and summarizing the main characteristics, patterns, and relationships present in the data. The primary goal of EDA is to gain insights and understanding of the data to inform further analysis or modeling tasks.


  §.§.§ Data Summarization

Firstly, we started with summarizing the main characteristics of the data, by categories and by community, for the total number of categorical features. For the traffic incident dataset, there were no categorical data but only the descriptions. We mapped some keywords in the description column to a new column 'category'.


  §.§.§ Geospatial Visualization

And visualizations of the data were created and manually analyzed to see if there were any regions of interest within the city that should have further inspection or analysis performed. To visually demonstrate the distribution differences of various indicators among communities, we created interactive maps using the Folium library. Specifically, we used community polygons as basic plotting units, colored the polygons based on the values of each indicator, and added legend explanations. Additionally, we experimented with various color schemes to better highlight spatial patterns.


  §.§.§ Temporal Analysis

Temporal analysis were performed to examine how community crimes, disorders, and traffic incidents change over time. 



 §.§ Correlation Analysis

Then, correlation mining was also performed to determine which attributes were positively or negatively correlated with each other within the city. 


  §.§.§ Correlation Analysis

To explore the correlations between factors and safety issues, we computed the correlation matrix of gdf_comm and plotted heatmaps using the Seaborn library. We focused on the correlations between demographic indicators, indicators related to children and schools, streetlight and tree indicators, pet quantity indicators, and crimes, security incidents, and traffic accidents



 §.§ Cluster Analysis


To further explore the intrinsic structure and similarities among communities, we employed various clustering algorithms onto the cleaned geocoded datasets. In selecting clustering algorithms the goal was to cover as many different approaches as possible and see what would have the best results on the data. Based on our research <cit.>, we found that density-based and grid-based methods are usually more suitable for handling spatial data. For the partitioning approach, K-Means and CLARANS were selected, Agglomerate Clustering was chosen for the hierarchical approach, DBSCAN and OPTICS were selected for the density based approach and CLIQUE was used for the grid-based approach. Overall, the approach for how each method was implemented is quite similar, but nonetheless will be broken down into more detail below. 



  §.§.§ K-Means Clustering

K-Means was the first algorithm implemented. It was implemented on each dataset using the latitude and longitude coordinates to determine clusters. To determine the optimal number of clusters, we used the silhouette score as the evaluation metric, tested a range of 2 to 20 clusters, and plotted the relationship between the silhouette score and the number of clusters. Finally, we selected the optimal value corresponding to the highest silhouette score. The clustering results were visualized using Folium maps.




  §.§.§ CLARANS Clustering

Considering that the K-Means algorithm is susceptible to outliers, we further attempted the CLARANS (Clustering Large Applications based upon RANdomized Search) algorithm. This algorithm minimizes the cost function through random search and can efficiently handle large datasets. We implemented CLARANS clustering using the pyclustering library and conducted grid search on different parameter combinations to achieve the optimal clustering effect. Again, the silhouette score was used to determine what the optimal parameters for the algorithm were. This was repeated for each dataset to determine the optimal parameters. 





  §.§.§ CLIQUE Clustering

To explore hidden patterns in high-dimensional data, we also introduced the grid-based density clustering algorithm CLIQUE (Clustering In QUEst). This algorithm first divides the data space into a series of grid cells, then identifies high-density cells, and finally forms clusters by combining adjacent high-density cells. We implemented CLIQUE clustering using the pyclustering library and conducted sensitivity analysis on parameters such as cell size and density threshold.


  §.§.§ DBSCAN Clustering

To assess a density based clustering method, DBSCAN was implemented on each of the datasets, again with the silhouette score being used to determine the optimal parameters DBSCAN functions by categorizing points as either core, border or outlier/noise points. This is determined based on the density of the points, a point becomes a core point if it has at least the minimum amount of points needed to reach the threshold set by the user within its boundary (also set by the user). Multiple different parameter sets were selected, for both the threshold and the minimum amount of points needed for a point to become a core point.
To explain the algorithm, we first need to define density connected point: Two points are called density connected if there is a core point which is density reachable (i.e. connected through a series of core points) from both the points.





  §.§.§ OPTICS Clustering

Optics clustering is also a density based clustering algorithm, but does not require a minimum amount of points or a distance threshold to function. Instead it requires a reach-ability distance. This creates a reach-ability plot to determine what points can be reached from each other. The results of this are another way to showcase how dense the distribution of points in the dataset are. 






  §.§.§ Hierarchical Clustering

Hierarchical clustering builds a hierarchy of clusters, where the agglomerative clustering method was chosen. This functions by selecting a single data point as a single cluster and then merges pairs of clusters until only one remains. Agglomerative clustering uses a tree-like structure called a dendrogram, which is then cut to determine the optimal number of clusters at different levels. The advantage to hierarchical clustering is that it does not require the number of clusters to be specified in advance, where the data does not need to naturally fall into a specific number of clusters to produce a high quality result. 





 §.§ Predictive Analysis

To predict the safety risks of communities, we constructed multiple linear regression and random forest regression models.


  §.§.§ Data Splitting and Feature Selection

We first used the chi-square test to select variables significantly correlated with the dependent variables (crime, security incidents, traffic accidents). Then, we randomly split the data into training set (80%) and testing set (20%) for model training and evaluation.


  §.§.§ Multiple Linear Regression

We constructed multiple linear regression models using the Sklearn library, with the number of crimes, number of security incidents, and number of traffic accidents as dependent variables and the selected significantly correlated indicators as independent variables. We fitted the models on the training set and evaluated them on the testing set using mean squared error and coefficient of determination.


  §.§.§ Random Forest Regression

Considering the possible nonlinear relationships between variables, we further attempted random forest regression models. Random forest combines multiple decision trees to effectively capture complex interactions between variables. We implemented random forest regression using the Sklearn library and optimized hyperparameters such as the number of trees and maximum depth through grid search. We also calculated the feature importance of each variable to assess its impact on prediction results.


  §.§.§ Model Evaluation and Comparison

We evaluated the predictive performance of multiple linear regression and random forest regression models on the testing set, using mean squared error and coefficient of determination as evaluation metrics. We also plotted scatter plots of actual values versus predicted values to visually demonstrate the model fitting effects. Finally, we conducted comparative analysis of the performance of the two types of models.



§ RESULTS



 §.§ Data Summarization

Fig. 6 reveals a varied landscape of crime in Calgary, with theft from vehicles dominating as the most prevalent offense, followed by theft of vehicles and break and enter incidents involving commercial properties. Assault, both domestic and non-domestic, also remains a concerning issue, with a notable number of reported incidents. Robbery, though relatively lower in count, still poses a risk in public and commercial settings. 


Fig. 7 reveals the top 10 traffic incident categories in Calgary, with "2 vehicle incident" and "Traffic Incident" leading. Issues like road blocking, multi-vehicle incidents, and pedestrian involvement are also significant. 


In Fig 8 and 9, we can easily see that Beltline and Downtown are the two leading communities with highest total number of crimes and disorders in Calgary. And the Forest Lawn community is also notable that comes next.




Fig. 10 highlights the top 10 communities in Calgary with the highest number of traffic incidents. Bridgeland/Riverside leads with 1,244 incidents, followed closely by Beltline and Downtown Core with 1,153 and 1,103 incidents respectively. Burns Industrial, Albert Park, and Manchester Industrial also rank among the communities with notable traffic incident counts.




 §.§ Map Visualization

Fig. 11 shows the map visualization with multi-polygons colored by the number of overall crimes in Calgary in each community. From a geospatial point of view, we can see that large amount of Crimes happened in the centre areas of Calgary, which are the Beltline and Downtown Core. We can also see in the visualization that the communities in the Northeast sector are with slightly more crimes than communities in other sectors.


Fig. 12 shows the map visualization with multi-polygons colored by the number of overall disorders in Calgary in each community. Similar to fig. 11, we can see that large amount of Crimes happened in the centre areas of Calgary, which are the same Beltline and Downtown Core. And for the Northeast sector, the number of disorders are also slights higher than other sectors but not as obvious as previous map visualization for crimes.


Fig. 13 shows the map visualization with multi-polygons colored by the number of overall traffic incidents in Calgary in each community. Comparing to crimes and disorders, the traffic incidents also have a central tendency in Beltline, Downtown Core with the additional Bridgeland-Riverside community. But the distribution of traffic incidents is more dispersed around the centre areas of Calgary. It is interesting that there seems to be more traffic incidents along the Deerfoot Trail and Macleod Trail.




 §.§ Temporal Analysis


Fig. 14 shows the Monthly Crime Trend Over Time by Category in Calgary. We can see a slightly decreasing trend on the overall crimes and theft from vehicle from year 2018 towards 2024.


While in fig. 15, it is showing Disorder Trend Over Time in Calgary. Instead of a decreasing trend, the disorder trend shows a periodic pattern every year from year 2018 to 2024.


And in fig. 16, it shows the Traffic Incident Trend Over Time in Calgary. It is observable that there is a slightly increase trend for the number of traffic incidents from year 2017 to 2024.


For fig 17, 18 and 19, they are showing the average Crimes, disorders or traffic incidents per month in Calgary. We can see in fig. 17 that the average number of crimes increases after April with the warmer weather and reach average maximum in August. After that the average number of crimes decreases with the temperature towards December.


While in fig. 18, we can also observe an similar rising trend for the average number of disorders, reaching the maximum in July.


And for fig. 19, we see a different trend in traffic incidents per month in Calgary. It exhibits an increase trend starting in July towards December which seems to be inversely proportion to the decreasing temperature.


Fig. 20 shows the Number of Crimes Each Year by Category in Calgary. We can see in this figure that, despite the slight differences in number each year, the pattern on the distribution of crime categories are the same.


Fig. 21 illustrates the total number of disorders in Calgary from 2018 to 2024. Apart from year 2024 with incomplete data, disorder counts remained relatively stable from 2018 to 2023, ranging from 83,701 to 94,514.


Fig 22 shows the top 5 Traffic Incident Categories along the years in Calgary. We can see in the figure that the number of 2 vehicle incidents are decreasing significantly while the blocking and general traffic incidents are increasing. As the category column is only done by mapping key words or phrases from the description column. The trends maybe a result of changing habits in writing more detailed descriptions.



 §.§ Correlation Analysis Results

For the correlation analysis, as there are too many attributes in the dataframe of community census, we need to use trial and error method and by our knowledge to screen out attributes that could be related to crimes, disorders and traffic incidents. And we have also tried to screen out attributes that varying across different communities.

Fig 23 is the results of first trial in correlation analysis. It is a correlation heatmap visualization showing the correlations between some attributes in the community census. Despite of availability of correlation results between different attributes in communities, they are not the interest of this project. The interest of this project lies in the last 3 rows which are the correlation of crimes, disorders and traffic incidents in related to community attributes. We can see in the results that the number of crimes and disorders are strongly correlated to the count of dwellings in a community, and it is a much stronger correlation comparing to that of resident counts in a community. Additionally, we can also see that crimes and disorders are strongly correlated to the number of apartments and other apartment related attributes. It indicates that we could probably anticipate more crimes and disorders in a community if there are more apartments in a community or more residents living in an apartment. Between crimes and disorders, they are almost perfectly correlated to each other. While for traffic incidents, there are no obvious correlations to any of the community attributes and it is expected. It is interesting that the number of traffic incidents are relatively correlated to the number of crimes and disorders in a community.


For fig. 24, we were trying to look into the how the number of preschool children and how their family support the school systems in Calgary correlated to the number of crimes, disorders and traffic incidents in a community apart from other attributes that we discussed in fig. 18. We see in the results that the crimes and disorders are not correlated to the number of preschool children, but they showed slight correlation to the number of dwellings that support public school system and both the public and separated school system. It is interesting that the crimes and disorders are strongly correlated to the number of dwellings with unknown support of school system or unable to determine. It might hint that the more the number of families that do not care about the school systems or not to answer such question in the census, the more number of crimes and disorders could be anticipated in a community. And for traffic incidents, the results matched the expectation that it would not be correlated to the number of preschool children or what school systems supported by the dwellings.


For fig. 25, it shows the correlation between crimes, disorders and traffic incidents with the number of street light, their total wattage and the number of trees in a community. The number of crimes, disorders and traffic incidents show weak correlation to the total wattage of street lights in a community and no correlation to the number of trees. Such weak correlation may be the results of external correlation to some other attributes, such as the number of apartments or dwellings in a community.


Fig 26 shows the correlation between crimes, disorders and traffic incidents with the number of pets, cats and dogs. The correlations between the target interests and the pet attributes are weak such that we cannot derive any insights from the results. It is expected that the pet attributes in a community should not be correlated to crimes, disorder and traffic incidents, but we just want to see if the results are not as expected.




 §.§ Regression Results

For the regressions, we were not trying to build any accurate models. Instead, we leveraged the linear and random forest regressors to determine the feature importance of crimes, disorders, and traffic incidents. We fed the regressors with strongly correlated attributes from previous correlation results. For crimes and disorders, the regressors considered the number of dwellings with unknown school support dominates the regression, which are 51% and 49% respectively, followed by the number of persons living in an apartment, which are 23% and 28% respectively. And for traffic incidents, the number of crimes (37%) and disorders (25%) in a community dominates the regression than other attributes.



 §.§ Clustering Results

It should be noted that clustering algorithms were only ran on the datasets that contained latitude and longitude coordinates when acquired. For the datasets where coordinates had to be calculated, clustering was not done as no meaningful results could be acquired, and clusters were non-existent. 


  §.§.§ K-Means

We ran the K-Means clustering algorithm on six of the datasets used. The two datasets that did not receive clustering was the census data, and the community disorder data. Clustering was ran on the crime data for K-means, but it was quickly realized that no meaningful clusters would be derived since the longitude and latitude was calculated based on the community centroids.

To determine the optimal number of cluster for each dataset, silhouette scores were calculated and the cluster number with that score was assigned when the algorithm was re-ran with the optimal parameters. 


Due to time constraints, the only hyper-parameter that was adjusted for K-Means was the number of clusters, the other parameters were assumed to be 5 iterations, a random state of 0 and then the algorithm was only ran on one centroid seed. These decisions were made to speed up computation time and because the data showcased poor clustering results during the initial setup phase. 



  §.§.§ CLARANS Clustering 

After the K-Means results were computer, some small changes were made to the way clustering was done in order to reduce computational time. The major change was that the crime dataset and traffic camera dataset would no longer have clustering algorithms ran on them. The reason for this was discussed above, but to re-iterate is that no meaningful clusters would be derived from these datasets due to the nature of the latitude and longitude values within them. 

The results of the CLARANS clustering algorithm are of a higher quality than K-Means, but still not great overall. The only dataset that has what would be considered good clustering results is the dataframe containing all of the public trees in the City of Calgary. This can be seen in the top right map of Figure 30, where there are some distinct clusters to be found in the map. Otherwise, none of the dataframes have a high silhouette score indicating that there is not much clustering to be found in the datasets - alike K-Means.





  §.§.§ DBSCAN Clustering

DBSCAN clustering is by far the worst algorithm that is implemented on the datasets, this is because for all intents and purposes no matter the dataset, there is only one cluster to be found. This indicates that all of the data is in close spatial proximity to each other, which is this true, but it also means that it is impossible to make any insights from clustering through DBSCAN. 


Looking at Table 3, the Pets dataframe has a very high silhouette score, but the EPS is so low it makes the entire dataframe into one cluster, so the high silhouette score is more or less negated by the fact that no clustering analysis can be done. 




  §.§.§ Hierarchical Clustering

Much alike the other algorithms, hierarchical clustering does not produce very high silhouette scores, nor very good results. Granted, Hierarchical clustering at least produces distinct clusters for all the tested dataframes, but not much meaningful information can be extracted from them. Once again further proving that clustering, no matter the method does not work well on the selected datasets. 





  §.§.§ CLIQUE Clustering



Overall, Clique clustering does not perform well in clustering the data. It can be seen in all the images that one of the clusters dominates the majority of the points in the dataframe. This makes deriving any results from the figure extremely challenging as the Clique algorithm is indicating that there is one major cluster in each dataset with a couple smaller peripheral clusters. This is quite similar to DBSCAN, which also produced very poor results.


  §.§.§ OPTICS Clustering




Optics clustering produces extremely confusing results, with the Lumens dataset, it suggests that there may be clusters of streetlights throughout the city. This is not supported by other clustering algorithms but is interesting nonetheless. It also suggests that there are clusters of trees throughout the city, which is again not overly supported by the other algorithms. Then the next two results are not sufficient for any further analysis, where the clusters are either spanning giant portions of the city or are confined to individual points. Two of the silhouette scores are also in the negative, which indicates that hyper-parameters that are being used and tested. Due to time constraints, there was not time to test further hyper-parameters to confirm this theory. 



§ DISCUSSION AND FUTURE WORK



 §.§ Key Findings

This study utilized data analysis and data mining techniques to delve into the crucial factors influencing community safety in the city of Calgary. 

Through the exploratory data analysis, we revealed patterns, insights and trends in geospatial, categorical and temporal perspectives.

While for correlation analysis, we found significant correlations between crime rates, disorder incidents, traffic accidents, and several community attributes, including population, housing types, streetlight quantity, and tree quantity. It revealed the intrinsic connections between different community attributes and the 3 main concerns. For instance, the population related attributes demonstrated positive correlations with crimes and disorders, while the apartment related attributes exhibited strong positive correlations with crimes and disorders. This implies that the housing structure may be primary factors influencing community crime. On the other hand, crimes and disorders are strongly correlated to the number of dwellings with unknown support of the school system or unable to determine. It implies that, in a community, the more families not caring about the education systems, there could be potentially more crimes and disorders.

And by regression analysis, we tried to find out which attribute contribute mostly on the crimes, disorders and traffic incidents. The results revealed that the number of families that don't care about the education system is contributing a large part in regressing the crimes and disorders. It implies that how people care about the education system in Calgary or the education system itself could be a key factor in crimes and disorders of the city. For traffic incidents, it is as expected the same as the results of correlation analysis, which is depending on the crimes and disorders instead of attributes in a community.

We have also tried on clustering data mining technique. Overall, clustering results were very poor across the board. K-Means overall produced the best results out of all the algorithms, but even then the highest silhouette score was only 0.4, which is not an optimal result. As all clustering algorithms do a poor job at producing distinct cluster across all the datasets, the overall indication is that the data is not overtly clustered or the hyper-parameters are not correctly optimized. OPTICS algorithm showed an interesting characteristic where the silhouette score stayed constant over different set of hyperparameters; which can suggest that data is very difficult to cluster, or there is an insufficient variation in hyperparameter, or our algorithm has reached a local optimization plane and huge shift in the hyperparameter is needed to get out of it.
Most of the clustering gave us a negative silhouette score. This happens when the mean distance between a sample and all other points in the same cluster is greater than the mean distance between that sample and all other points in the next nearest cluster, meaning that the clustering algorithm has assigned a data point to the wrong cluster, as it is more similar to points in another cluster than to points in its own cluster. This further indicates that the clusters are overlapping or poorly separated.

The data cleaning process also has a flaw when it comes to producing clustering results. For the datasets, such as crime, that did not have a latitude and longitude originating from the dataset, are unable to produce meaningful clustering. This is because the same community centroid coordinates were provided for every entry that was located in that community. Causing there to be not enough data points for there to be any meaningful clustering results. This essentially rendered three of the seven datasets for Calgary to be unusable for clustering algorithms. Another issue that arose is that from the selected parameters is that a few number of clusters were found to be optimal, making it incredibly challenging to find any significant clusters when each cluster is covering a large portion of the city. 



 §.§ Innovations

The innovations of this study lie in: (1) integrating multiple heterogeneous data sources including streetlights, trees, crime, and traffic to depict community safety from multidimensional perspectives; (2) introducing advanced clustering algorithms and machine learning models to reveal the distribution patterns and influencing mechanisms of safety issues from both spatial and attribute perspectives; (3) applying techniques such as feature importance analysis to interpret the correlation analysis, providing actionable intelligence for community safety governance.



 §.§ Limitations and Future Directions

Due to time and data availability constraints, this study only covered limited community indicators and data within a short-term timeframe. Future research could collect more comprehensive data and conduct long-term spatio-temporal and Trajectory analyses. Additionally, the causes of urban safety issues are complex and diverse, making it difficult to elucidate the causal relationships and mechanisms between various factors solely from a data perspective. Subsequent research could integrate qualitative methods such as expert interviews and case studies to explore the socioeconomic, institutional, and cultural factors influencing community safety.

While our study focused on Calgary, the analysis framework and technical roadmap can be generalized to other cities. Conducting comparative studies across multiple cities would help discover common patterns, facilitate knowledge exchange, and contribute wisdom to building safer and more inclusive urban environments.



§ CONCLUSION 

In conclusion, this study utilized data mining techniques to analyze key factors affecting community safety in Calgary from multiple perspectives, revealing spatial disparities in safety levels and establishing effective predictive models. These findings not only fill the gap in quantitative urban research but also provide new scientific evidence to urban managers and policymakers, supporting fine-grained community safety governance. As we move towards "smart cities," data mining undoubtedly will play a crucial role. We look forward to more researchers engaging in this field, leveraging the power of data to guide the construction of safer cities.
00
a Hartigan J A, Wong M A. Algorithm AS 136: A k-means clustering algorithm[J]. Journal of the royal statistical society. series c (applied statistics), 1979, 28(1): 100-108.

bT. Kanungo, D. M. Mount, N. S. Netanyahu, C. D. Piatko, R. Silverman and A. Y. Wu, "An efficient k-means clustering algorithm: analysis and implementation," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 7, pp. 881-892, July 2002.

c Ng R T, Han J. CLARANS: A method for clustering objects for spatial data mining[J]. IEEE transactions on knowledge and data engineering, 2002, 14(5): 1003-1016.

d Schubert E, Sander J, Ester M, et al. DBSCAN revisited, revisited: why and how you should (still) use DBSCAN[J]. ACM Transactions on Database Systems (TODS), 2017, 42(3): 1-21.

e Ankerst M, Breunig M M, Kriegel H P, et al. OPTICS: Ordering points to identify the clustering structure[J]. ACM Sigmod record, 1999, 28(2): 49-60.

f Liang K Y, Zeger S L. Regression analysis for correlated data[J]. Annual review of public health, 1993, 14(1): 43-68.

g Dong X L, Srivastava D. Big data integration[C]//2013 IEEE 29th international conference on data engineering (ICDE). IEEE, 2013: 1245-1248.

h Ferreira J, João P, Martins J. GIS for crime analysis: Geography for predictive models[J]. Electronic Journal of Information Systems Evaluation, 2012, 15(1): pp36 49-pp36 49.

i Zhao X, Tang J. Crime in urban areas: A data mining perspective[J]. Acm Sigkdd Explorations Newsletter, 2018, 20(1): 1-12.

j Chen H, Chung W, Xu J J, et al. Crime data mining: a general framework and some examples[J]. computer, 2004, 37(4): 50-56.

k Bernasco W, Elffers H. Statistical analysis of spatial crime data[J]. Handbook of quantitative criminology, 2010: 699-724.

l Anselin L, Cohen J, Cook D, et al. Spatial analyses of crime[J]. Criminal justice, 2000, 4(2): 213-262.

m Hand D J. Principles of data mining[J]. Drug safety, 2007, 30: 621-622.

Optics DarkProgrammerPB. (2019, December 26). OPTICS clustering Algorithm (from scratch). Medium. https://medium.com/@darkprogrammerpb/optics-clustering-algorithm-from-scratch-8d66960e238

DBSCAN Hurra, T. (2023, April 11). DBSCAN — Make density-based clusters by hand. Medium. https://towardsdatascience.com/dbscan-make-density-based-clusters-by-hand-2689dc335120

CLARANS Ng, R., & Han, J. (2002). CLARANS: A method for clustering objects for spatial data mining. Knowledge and Data Engineering, IEEE Transactions On, 14, 1003–1016. https://doi.org/10.1109/TKDE.2002.1033770

Kmeans Oyelade, J., Oladipupo, O., & Obagbuwa, I. (2010). Application of k Means Clustering algorithm for prediction of Students Academic Performance. International Journal of Computer Science and Information Security, 7.

Clustering Neethu C V & Subu Surendran. (2013). Review of Spatial Clustering Methods. International Journal of Information Technology Infrastructure, 2(3), 15–24. http://warse.org/pdfs/2013/ijiti01232013.pdf





 Paper Title: ABC Align: Large Language Model Alignment for Safety & Accuracy.





Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning
    Yuanyang Zhu, Zhi Wang, Member, IEEE, Yuanheng Zhu, Senior Member, IEEE, Chunlin Chen, Senior Member, IEEE, Dongbin Zhao, Fellow, IEEE


The work was supported by the National Natural Science Foundation
of China under Grant 62006111 and Grant 62073160.
	
Yuanyang Zhu, Zhi Wang and Chunlin Chen are with the Department of Control Science and Intelligent Engineering, School of Management and Engineering, Nanjing University, Nanjing 210093, China, and also with the Research Center for Novel Technology of Intelligent Equipment, Nanjing University, Nanjing 210093, China (e-mail: yuanyang@smail.nju.edu.cn; zhiwang@nju.edu.cn; clchen@nju.edu.cn).
 
 Yuanheng Zhu and Dongbin Zhao are with the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China (e-mail: yuanheng.zhu@ia.ac.cn; dongbin.zhao@ia.ac.cn)

    
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Alignment of Large Language Models (LLMs) remains an unsolved problem. Human preferences are highly distributed and can be captured at multiple levels of abstraction, from the individual to diverse populations. Organisational preferences, represented by standards and principles, are defined to mitigate reputational risk or meet legislative obligations. In this paper, we present ABC Align, a novel alignment methodology for LLMs that enables integration of the standards and preferences of a large media organisation into the LLM itself. We combine a set of data and methods that build on recent breakthroughs in synthetic data generation, preference optimisation, and post-training model quantisation. Our unified approach mitigates bias and improves accuracy, while preserving reasoning capability, as measured against standard benchmarks.




§ INTRODUCTION




 §.§ Overview



In this paper, we present a novel approach to the alignment of Large Language Models (LLMs) we call ‘ABC Align’. This alignment is conducted in two settings: the fine-tuning of open-source models, and In-Context Learning (ICL) of closed-source ‘frontier’ models <cit.>. While the techniques and literature of In-Context Alignment (ICA) are less mature than fine-tuning based alignment, we achieve good results across both settings. By design, our methodology is applicable across any range of open- and closed-source models, offering provider independence and flexibility as underlying model capability grows with time. 

We use a variety of data that is both domain-specific and general. These data include news article content, organisational ‘AI Principles’ <cit.>, and human-reviewed question/answer pairs sourced from an internal Retrieval-Augmented Generation (RAG) tool. They are used as input to both generate datasets for fine-tuning, and in the case of the ‘AI Principles’, directly in the frontier model context window itself.

In the fine-tuning alignment setting, we use several techniques from the literature on synthetic data generation, knowledge distillation, and preference optimisation to produce several datasets conditioned on these data. We include results controlling for the impact of each of these methods and data across two open-source models, from Mistral AI and Meta respectively.

We see general performance improvements across industry standard benchmarks. The version of our dataset that combines all data and methods achieves a 23.51% relative performance improvement on Meta’s Llama3-8B model on the TruthfulQA benchmark. Furthermore, these results were achieved using a small dataset of 897 samples, showing substantial improvement over Meta’s own fine-tuned version of the base model, which is trained on more than 10M human-annotated examples <cit.>. 

For the ICA setting, conducted on OpenAI’s GPT4-turbo model, we evaluate the performance of a number of custom system prompts, including the prompt used by our internal RAG tool, and our AI Principles. Most notably in evaluation against the Bias Benchmark for Question Answering (BBQ), a system prompt that uses our AI Principles improves relative performance over baselines by 77.54%.

Aligning LLMs and conducting evaluations are inherently challenging due to difficulty in defining ‘who the alignment is for’, and the need to define benchmarks reflecting a measure of alignment specific to that group, in addition to standard benchmarks. 

Our work aims to show an example of how organisations can answer these questions, integrate their standards into a process of provider-agnostic alignment, hence providing the organisations with choice and agency in how they apply LLMs across a range of use-cases.



 §.§ Background 


Authors’ note: no editorial position is offered or claimed in this publication.
We also do not publish our data, and all LLM prompts are provided to support replication of our work and do not reflect optimisation for production use.

For real-world industrial application, in our case a large public-service media (PSM) organisation, alignment of model outputs to policies and preferences is critical. This is due to charter obligations to provide the public with innovative services <cit.> that mitigate reputational risk while being transparent. 

Many PSMs face additional constraints specific to their operating model, most notably in the link between their sources of funding and restrictions around commercial activity <cit.>. This reality can limit their capacity to support internal R&D efforts around technology, while they simultaneously rely on technology to deliver content to audiences in a highly competitive media marketplace.

LLMs present a unique opportunity for PSMs to improve internal processes. For example, through enhancing content metadata or supporting information retrieval across large corpora. LLMs also carry a unique risk profile <cit.> when working with that content itself, whether it be to transform the content for multi-platform use, or use-cases that sit at the edge of emerging definitions around ‘responsible AI’, such as content generation <cit.>.

Recent progress in model capability that enables processing of content across modalities in a single system – text, audio, video – only serves to enhance this risk profile and magnify impact of the constraints PSMs face, given the computational and operational costs associated with utilising such systems at scale.

Scale is a critical consideration for PSMs both in terms of the audience served and content produced and stored. Leveraging the capabilities of LLMs (and their multimodal derivatives) thus involves a considered trade-off between scale, cost, and safety. 

The Australian Broadcasting Corporation (ABC) has defined a set of AI Principles (see Appendix A) that are designed to guide the adoption and application of AI in a responsible, principled manner. We build on the use of ‘safety’ in the literature, particularly work on ‘Constitutional AI’ <cit.>. We make explicit use of the ABC AI Principles in our methodology, and here include the principles of ‘protecting data’ and ‘mitigating bias’ to broaden the definition of ‘safety’. ‘Accuracy’ maps directly to the AI Principle of the same name. This paper outlines methods that securely leverage the ABC’s data and are evaluated against industry-standard benchmarks that act as proxies for these principles.



 §.§ ABC Align


Our work sits at the intersection of machine learning theory and application. The organisational context in which it is conducted, under the various constraints outlined above, guides our choices of models, data and methods. These methods leverage the unique advantage of media organisations, namely their content, which is created in line with clearly defined policies and principles. We consider these as data valuable for the alignment and evaluation of LLMs, when paired with the approach we outline in this paper.

To best utilise this data, we build on recent work on synthetic data generation aimed at distilling frontier models’ capability into smaller open-source models using ICL <cit.>. Both the open-source and frontier models can be substituted, and the impacts of these changes are measurable. In this paper, we include representative results across several open-source models and a specific frontier model to validate and demonstrate our methods. Our aim is to offer a blueprint that is prescriptive on method only, enabling the replication and use of our work by other PSMs with their own unique data.

Our synthetic datasets, also referred to as augmented datasets, are used by a fine-tuning process that conditions each sample on textual article content, organisational ‘AI principles’, and samples gathered from the live deployment of an internally built assistive tool. This tool enables the capturing of editorial preferences and input at scale, hence also exploiting the ICL abilities of frontier models to guide and align the application to user needs. 

Development of this tool involved the crafting of novel, structured prompts and retrieval architectures, evaluated against user-derived acceptance criteria that were reflective of broader organizational preferences. We defer a lengthier discussion of its implementation to a future publication. 

In addition to our alignment methods in a fine-tuning setting, the results presented in this paper reflect nascent developments in ICA of frontier models, and we expect to build on these in future work, with more detail covered in our 'Future Work' section. We aim to provide a general methodology: general to our organisation’s data, the full complement of available AI models, and application to a range of use-cases specific to media organisations. Such applications can include metadata tagging, sourced information retrieval, and LLM-based approaches to content recommendation <cit.>.

We leave the infrastructure enabling the synthetic dataset creation, knowledge distillation, training and evaluation of LLMs out of scope for this paper. We note that we make extensive use of open-source tools, including Hugging Face's Transformers and TRL <cit.> libraries among others.

Evaluation also forms a key part of our methodology. We build on the work of EleutherAI, specifically their ‘Large Language Model Evaluation Harness’ <cit.> and emerging best practices to measure our alignment efforts against industry-standard benchmarks that reflect the ‘AI Principles’ outlined above. The framework is extensible, enabling custom, organisation-specific evaluations defined in collaboration with non-technical stakeholders. We offer further commentary on this in the ‘Future Work’ section at the end of this paper.



§ RELATED WORK


In this section we outline a brief history of LLM development, and each of the methods we use as part of ABC Align.

Early implementations of the ‘Transformer’ and related attention mechanism were limited in both scale of pre-training data and the number of parameters in the neural network <cit.>. However, the new architecture, with query-, key-, and value-cache to optimise both training and inference, went beyond the capacity of earlier architectures like Long Short-Term Memory (LSTM) and Recurrent Neural Networks (RNNs), both being limited in their ability to process long sequences under fixed computational constraints.

Radfort et al. <cit.> moved beyond masked-language modelling approaches and described the Generative pre-trained Transformer (GPT) architecture, which framed the prediction problem in terms of arbitrary sequence-to-sequence modelling <cit.>. 

Ouyang et al. <cit.> directly improved this ability to follow general instructions via their method of ‘instruction tuning’, framing the model interaction as a ‘request/completion’ dynamic, leading to the conversational-style interfaces seen in recent iterations of GPT-class models.

As this approach was scaled up, the diversity and distribution of training data (consisting of trillions of tokens sourced from web-scale datasets), revealed limitations for the application of these models in a practical setting, where user expectations around accuracy and bias were not met <cit.>.



 §.§ Domain-Specific Large Language Models


LLMs have been developed for application to specific fields or areas of industry, such as biology and finance <cit.>. A canonical example is BloombergGPT <cit.>. Bloomberg has a significant store of domain-specific data (363B tokens) enabling it to perform full pre-training of their own model. Bloomberg’s experience informed the methodology outlined in this paper, as the ABC itself has a large store of data across digital-first content and its archives. 

A complete pre-training process of an ABC LLM is likely to yield a capable, general model, given that digital content alone contains over a billion tokens, and 90+ years of archival content is currently being digitised. However, the expense and risk associated with maintaining a single artifact given the rate of progress in the field <cit.>, have led us to favour a lightweight, modular methodology rather than pre-training a base model from scratch. This would allow us to leverage the value of the ABC’s extensive data but in a way that was agnostic to underlying base models, in a cost-effective and future-proof manner. This has in part been validated by subsequent research, which has demonstrated that OpenAI’s GPT-4 has exceeded the performance of BloombergGPT <cit.>.



 §.§ LLM Alignment


LLM alignment is a rapidly evolving area of research, alongside efforts to further scale up pre-training data and neural network parameter counts. Starting in 2023, early evidence <cit.> emerged that while massive amounts of data were required in the pre-training stage, alignment itself could be facilitated through smaller, high-quality curated datasets with samples numbering in the thousands instead of trillions. In parallel, LLM capabilities were being categorised relative to new evaluation frameworks, specifically around ‘reasoning’ (problem-solving) <cit.>, extending standard definitions of accuracy beyond metrics like ROUGE and BLEU <cit.>.

Our work builds on ‘Constitutional AI’ from Anthropic <cit.> and ‘Less is More for Alignment’ (LIMA) by Meta <cit.>. Constitutional AI focuses on the self-alignment of LLMs based on a ‘constitution’ or set of principles that guide the process of ‘AI feedback’. We adapt this approach to use the ABC AI Principles directly in the creation of our fine-tuning datasets and frontier model alignment prompts. 

LIMA defines the ‘Superficial Alignment Hypothesis’, that states a ‘model’s knowledge and capabilities are learnt almost entirely during pre-training, while alignment teaches it which sub-distribution of formats should be used when interacting with users’. The use of ‘superficial’ here indicates that fine-tuning a subset of parameters, rather than complete pre-training, is the appropriate setting for LLM alignment. Their work examines the impact of a small set of 1000 samples on several alignment metrics, building on <cit.>. Our dataset is thus correspondingly small, high-quality, and informed by our AI Principles.

Our own hypothesis is that given a sub-distribution of fine-tuning and preference data specific to our organisation, we can in turn align LLMs in a way that preserves their general utility, capability, and performance on industry-standard benchmarks that directly map to and reflect our AI Principles. While non-domain-specific data like those from ‘Stack Exchange’, ‘Wikihow’ and ‘Reddit’ can be valuable in an academic fine-tuning setting, their use for alignment in a PSM setting is harder to justify. Consider a user feedback scenario, in the context of PSM obligations outlined in the introduction:


‘Why did your model generate this inaccurate/biased output?’

‘It was aligned using reddit threads!’

A key challenge that the LIMA authors observe is that ‘manually creating diverse prompts and authoring rich responses in a uniform style is laborious’. In our own organisational context, asking editorial staff to label data, even if for a small sample, represents a significant and resource-intensive task.



 §.§ Synthetic Data and Knowledge Distillation: ORCA


The utilisation of ‘synthetic data’ <cit.> represents a novel source of data for training LLMs. It is often cited in wider discussions about the finite quantity of human-created data <cit.>. As with human data, not all synthetic data are created equal. Indeed, low-quality synthetic data, when riddled with hallucinations, inaccuracies, and biases, can significantly impact downstream task performance <cit.>.

To improve the quality of our synthetic data and mitigate these issues, we condition each sample on organisational data: news articles. Beyond the content itself, these articles comply with the ABC’s editorial standards and policies <cit.> at the time of their publication, and provide a real-world example of ‘human annotated data’ that goes well beyond crowd-sourced methods such as or ‘Mechanical Turk’ or similar. We also go beyond algorithms like Self-Instruct <cit.> that use ‘seed instructions’ to generate synthetic data, by providing a unique input for each generated sample.

Knowledge distillation techniques have diversified well beyond the original work of <cit.>. In the context of this paper, the teacher/student dynamic remains, but occurs between frontier model/smaller open-source model. The distillation occurs per-input, given a prompt. The structure of this prompt confines the possible sub-distribution of the synthetic data used by the student model in our fine-tuning process. Here we build on the work of <cit.> on ORCA, where they elicit ‘reasoning traces’ from OpenAI’s GPT-4 based on synthetic samples, with the goal of improving specific capabilities of a smaller 13B parameter model.

We note here that our synthetic data samples are created for the specific purpose of improving underlying model capability rather than meeting editorial standards, following the methods outlined in ORCA. We do not intend on making this data publicly available. This is to avoid any confusion with the published content it is derived from. 




 §.§ Preference Optimisation: ORPO


Soliciting human feedback for alignment efforts proved crucial in the success of tools like ChatGPT <cit.>. In a reinforcement learning setting, numerical instability and sensitivity to training dynamics were seen as a necessary trade-off for improved alignment performance. Model-free techniques such as Direct- and Odds-Ratio Preference Optimisation (DPO/ORPO) provide stable alternatives to the Reinforcement Learning with Human Feedback (RLHF) approach <cit.>.

Following a pre-training stage, models undergo further post-training to make them suitable for general use. Multiple techniques exist to facilitate this, with initial work focused on RLHF <cit.>. This method aims to use human preference data to align pre-trained models and has demonstrated improved results beyond safety and reducing harmfulness <cit.>, which are the key areas of focus for our work.

Recent work on preference optimisation <cit.> aims to elicit similar changes in model behaviour via methods that reduce implementation complexity compared to RLHF. Given the constraints on PSMs (and organisations whose operational focus is not technology itself), these methods provide a way to demonstrate the value of incorporating their own ‘human feedback’ in the alignment of models they use.

In this paper, we make use of an implementation of ORPO that is readily available in Hugging Face’s libraries. ORPO offers good performance when compared with both Supervised Fine-Tuning (SFT) and DPO, including improved computational efficiency for a given batch of training data, making it suitable for the aims of our work.



 §.§ In-Context Alignment


There has been significant recent work on exploiting the ICL capabilities of frontier models <cit.> where the context length has increased by up to an order-of-magnitude, as seen for instance in Google’s Gemini and Anthropic’s Claude series. While performance of fine-tuning in an ICL setting doesn’t yet offer consistent improvements when compared with SFT-based approaches across all tasks, there are specific cases where results are promising <cit.>. 

The work of <cit.> further quantifies the ‘Superficial Alignment Hypothesis’ proposed in LIMA. They measure the token distribution shift between pre-trained and aligned models, specifically the difference in probabilities of predicted tokens given a position in some sample input across the two models. They find that alignment fine-tuning affects a small subset of tokens that they describe as ‘stylistic’, specifically ‘discourse markers, transitional words, and safety disclaimers’. 

Further to the original hypothesis of the LIMA authors, this provides evidence and motivation for the effectiveness of alignment happening in a post-training setting via fine-tuning, and that inference-time alignment may offer comparable results for applications where fine-tuning is unsuitable or not possible. This further validates our decision to not pre-train an LLM, but instead focus on post-training alignment across SFT, PO, and ICA settings in support of our overall aim to provide a lightweight, flexible methodology that considers organisational constraints and the rapid growth of underlying model capabilities.



 §.§ Model Quantisation


As we discussed in the introduction of this paper, PSMs are funding-constrained, thus any training of LLMs needs to be done in an efficient way. Early methods of model quantisation involved performance trade-offs in terms of prediction quality and performance when compared to full-precision training and inference. However, recent advances in model quantisation have enabled rapid and low-cost experimentation. QLoRA <cit.> addresses the shortcomings of earlier methods sufficiently and is our choice for quantisation for the experiments outlined in this paper.



 §.§ Retrieval Augmented Generation & Grounding Synthetic Data


Another key aspect of the ABC Align methodology is the refining of the preference dataset, using user-generated outputs from an internally developed RAG tool. This tool is powered by OpenAI’s GPT-4-32k <cit.> and uses context information retrieved from a corpus of ABC documents to answer user questions. 

Within this context, alignment of the frontier GPT-4 model refers to ensuring that the entire RAG tool produced outputs that were relevant, accurate, and valuable to end users. This ultimately served to reflect whether the application (and thereby the frontier model) met users' expectations, and by proxy their broader organisational values. This effort involved a comprehensive system-wide approach, including not only refining the prompting techniques but also ensuring that the retrieval mechanisms, data sources and generated outputs all complied with the overall expected objectives <cit.>. Evaluation of the system includes using user-acceptance derived metrics, conducting user interviews, and manually reviewing generated samples, ensuring an adequate level of alignment of the frontier GPT-4 model to the ABC's AI Principles.

While we generate ORCA-style synthetic data in this work, RAG grounding of synthetic data to produce reasoning traces that are accurate to, rather than derived from, content data will be explored in future work.



 §.§ Information Theoretic Measures in NLP


A primary focus of this paper is dataset quality, and we demonstrate its impact on LLM performance in multiple fine-tuning settings. While measurement against standard benchmarks is critical, we are also interested in measuring the quality of the data itself, as it is initially transformed during our synthetic data and knowledge distillation process. Information theoretic metrics offer a principled approach to measuring quality and changes in the information that the data represents. For this purpose, we use several well-known metrics, including Shannon entropy, mutual information, and Kullback-Leibler divergence <cit.>.



§ METHODOLOGY



 §.§ Synthetic Data Generation


In this section, we outline two versions of our alignment dataset for full fine-tuning of open-source models, together with the synthetic dataset generation prompts. The inputs to these datasets are news articles, the ABC AI Principles, and samples from our internal RAG tool. 

This tool, while an early prototype, is designed to deliver the capabilities of LLMs directly to users while also providing a means for soliciting editorial feedback and preferences, key to scaling our methods, outlined in the ‘Future Work’ section of this paper.

An automated pipeline was developed to enable the creation of synthetic datasets by augmenting existing data, as outlined in the previous section. The step-by-step process of creating an ABC Align dataset suitable for SFT is as follows:


    
  * Collect news articles as the input.
    
  * Create LLM-generated question/answer pairs relating to each input news article using an ORCA-style prompt.
    
        
  * Prompt: Deduce any reasoning or logical problems from this article:  Generate a complex question and a logical answer that requires step-by-step thinking, and elaborate on this thinking process as part of the answer. Both the question and answer must not refer to the original article. 
    


The resulting dataset is then used to create a dataset suitable for preference optimisation (PO). The answer for each sample is rewritten considering the AI Principles, given two canonical examples of human-reviewed Q&A pairs from our internal RAG tool.


    
  * Rewrite the answers from the SFT dataset
        
  * For ‘chosen’ answers:
        
            
  * Prompt: Here are two high-quality, human-reviewed Question/Answer pairs: Question:  Answer:  Question:  Answer:  Now review a new Question & Answer pair:  – Rewrite the answer to better align with these principles:  Do not reference the principles in your response. Include only the text of the answer.’
        
        
  * For ‘rejected’ answers:
        
            
  * Prompt: Here are two high-quality, human-reviewed Question/Answer pairs: Question:  Answer:  Question:  Answer:  Now review a new Question & Answer pair:  – Rewrite the answer to be unaligned with these principles:  Do not reference the principles in your response. Include only the text of the answer.’
        
    

In all cases, we use OpenAI’s GPT-4-turbo frontier model with a temperature of 0.7 to generate synthetic data for use in a fine-tuning setting. The resulting datasets comprise 897 individual samples.



 §.§ Control Datasets


We include three additional datasets in our experiments. These are OpenORCA <cit.>, LIMA in the SFT setting <cit.>, and UltraFeedback <cit.> for ORPO training.

OpenORCA is a replication of the original ORCA data which was not made public. The dataset is sourced from FLAN-T5 and generated using both OpenAI’s GPT-3.5-turbo and GPT-4 models. The authors aim to reflect the distribution of the dataset outlined in the ORCA paper.

UltraFeedback is an updated version of the dataset used in the ORPO paper that addresses a data contamination issue related to the TruthfulQA benchmark.

The LIMA dataset is the original as published by the authors, and in all cases we randomly take 897 samples for direct comparison with our own datasets.



 §.§ Models and Fine-tuning


We fine-tuned the following pre-trained models: Llama-3-8B <cit.> and Mistral-7B-v0.3 <cit.>, including both the base and instruction-tuned models. We used the default tokenizer of each of the models, and applied Hugging Face’s Zephyr-style chat template (with , , and  tokens).

As described above, we employed a two-step fine-tuning process involving SFT for instruction tuning and ORPO for preference optimisation. To facilitate this, we used TRL’s SFTTrainer and ORPOTrainer <cit.> and scripts from Hugging Face’s Alignment Handbook <cit.>.

ORPO includes an additional training objective, relative ratio loss. It also dynamically penalises ‘disfavoured’ responses, instead of constructing sets of rejected tokens to control for degenerative model behaviour in the SFT setting. 

We trained on single Nvidia A10G GPUs with 24GB memory on cloud infrastructure. We trained for 23 epochs on the SFT dataset, taking around 2.4 hours, and we trained for 10 epochs on the ORPO dataset, taking around 3.8 hours, using a standard checkpoint selection process against selected benchmarks for the models used in our final evaluations. 




 §.§ QLoRA


The SFT/ORPO QLoRA hyperparameters used in this study followed those as set in the Hugging Face Alignment Handbook. An overview of our QLoRA hyperparameters during model fine-tuning are shown in Table <ref>.


Hyperparameter selection for quantisation methods has been shown to influence model performance in numerous ways, where the model may have different optimal ‘rank’ hyperparameters for different metrics, as well as the ratio between alpha/rank <cit.>.

The rank hyperparameter is particularly important since it affects the size of the update matrices, and therefore the number of trainable parameters. It should be noted that these recommendations are evolving, and hyperparameter optimisation may be required for individual use-cases <cit.>.



§ IN-CONTEXT ALIGNMENT


 For our experiments on ICA, we use OpenAI’s GPT-4-turbo model with a temperature of 0.7, and the  evaluation framework. Benchmarking is done in a `' setting rather than traditional multiple choice, due to the limitations around the availability of logits from closed-source APIs.

 Our primary intervention is setting the `' parameter, including three specific prompts, taken directly from our internal RAG tool, the ABC’s AI Principles, and a version of the AI Principles augmented with a Q&A sample sourced from the RAG tool, acting as a canonical example.



 §.§ Dataset Analysis 


This section outlines the information-theoretic and perplexity metrics we used to evaluate the datasets discussed in this study.



 §.§ Shannon Entropy, Mutual Information, KL Divergence


 For dataset analysis, we utilise several information-theoretic metrics on our SFT dataset. Our motivation here is to quantify the changes to average entropy, loss of information, and distribution shift induced by our synthetic data generation process. Our aim is to understand how our methods transform the information present in our input data, as we distil ‘reasoning traces’ suitable for downstream SFT and ORPO fine-tuning. All metrics are normalised.

In Table <ref>, we consider the average Shannon entropy of our input news articles, and the resulting synthetic dataset.




In Table <ref>, we calculate both mutual information and KL divergence across both datasets.



These results demonstrate that, through our process of conditioning synthetic data samples on news articles, we have increased sample complexity relative to the input while keeping shared information low (in-line with our attempt to derive specific patterns with high complexity and abstraction, i.e. ‘reasoning traces’). We have been able to do this while preserving the similarity of the overall distributions, as reflected in the low KL-Divergence score.



 §.§ Perplexity

For perplexity (ppl), we measure the mean ppl per dataset per base model. We calculate the perplexity of all datasets for both pre-trained Llama-3-8B and Mistral-7B-v0.3 models. These metrics are shown in Table <ref> for the ABC Align, LIMA, OpenORCA and UltraFeedback datasets. We note that the ORPO datasets include `chosen' and `rejected' formats, reflecting their preference optimisation nature in line with Hugging Face's Alignment Handbook <cit.>. For the ABC Align ORPO dataset, these were generated by OpenAI’s GPT-4-turbo frontier model to be aligned/unaligned to the ABC AI Principles. 



Interestingly, among the ORPO datasets, the ‘chosen’ format shows lower perplexity than the ‘rejected’ format. Rather than exhibiting equal perplexity for the chosen/rejected formats, this implies that the original model training may have achieved similar results as we aimed to achieve by aligning using the ABC AI Principles. 

We limit our use of ppl to the datasets themselves, rather than the fine-tuning phase. As observed by the LIMA authors, there is an anti-correlation between perplexity measured at training time and generational quality; perplexity can increase without generation quality necessarily decreasing <cit.>. 

Finally, we note that while details of the pre-training datasets of either base model have not been released, data quality is known to impact pre-training performance <cit.>, and indeed Meta does emphasise the use of filtering techniques to select only ‘high-quality’ data <cit.>.



§ MODEL EVALUATION


 In this section we discuss our evaluation strategy, including the mapping between selected benchmarks and the ABC AI Principles. We evaluate both our datasets and model outputs, whereby results are presented in the section below.



 §.§ General Reasoning


In the fine-tuning setting, we want to ensure that we are not training the model to be task-specific. Here we use the ARC dataset, specifically the ‘challenge’ subset, constructed by the Allen AI Institute <cit.>. It comprises 7,787 science questions in a multiple-choice setting aimed towards measuring ‘reasoning capability’. We select this benchmark to ensure that we do not sacrifice general model capabilities while improving bias identification and accuracy, i.e., limiting their application across a range of use-cases – also known as the ‘alignment tax’ <cit.>.



 §.§ ABC AI Principle: ‘mitigating bias’


The Bias Benchmark for Question & Answering (BBQ) <cit.> represents a human-constructed and thorough cross-section of socio-cultural biases in a multiple-choice setting, for the evaluation of a model’s ability to detect bias. In this study, we use , a subset version of the original BBQ, containing samples for evaluation against age, nationality, race, gender, sexual orientation, and physical appearance biases. 

This benchmark serves as a proxy for the ABC AI Principle ‘mitigating bias’, which is defined as ‘We will seek to prevent bias in our AI data or algorithms that could perpetuate and amplify existing inequalities and lead to discrimination’. BBQ is designed specifically to measure whether model outputs contribute to ‘harming marginalised individuals and groups’, thus enabling the identification and mitigation of bias. The list of social biases measured are defined by the US Equal Employment Opportunity Commission <cit.>. We acknowledge the US-centricity of this definition, and outline how we intend to extend our evaluation metrics in a general way, to make them ABC- or other-specific in our ‘Future work’ section at the end of this paper.



 §.§ ABC AI Principle: ‘accuracy’


We use performance on TruthfulQA <cit.> as a proxy for the ABC AI Principle ‘accuracy’. TruthfulQA consists of 817 multiple-choice questions across 38 subjects, including law, health, and politics. This benchmark addresses the limitations of LLMs, where the distribution of their pre-training data may induce the generation of ‘imitative falsehoods’, or answers that sound true without being grounded. TruthfulQA thus represents a robust benchmark for accuracy, where the conditions for a claim to be true are ‘if it describes..[a]..literal truth about the real world’, and the question/answer choices are written by humans. We specifically use the  task.



§ RESULTS

In this section we present the results of our fine-tuning alignment using both SFT and PO datasets.



 §.§ Fine-tuning: SFT (ORCA/LIMA)


In an SFT setting, we evaluate the performance of our ABC Align SFT dataset against data generated using the two methods we build on, ORCA and LIMA respectively. We present our evaluation results in Figure 
<ref> below.





  §.§.§ Evaluation on 


For Mistral-7B-v0.3, we see a decrease in performance compared to the pre-trained model of 0.42% and 0.85% for LIMA and OpenORCA. The Align SFT dataset demonstrates a 3.25% improvement compared to the pre-trained model and those fine-tuned on control datasets. 

For Llama-3-8B, we see improvements over the pre-trained model for LIMA and OpenORCA datasets of 2.78% and 0.59% respectively. The ABC Align dataset demonstrates a 7.17% improvement over the pre-trained baseline.



  §.§.§ Evaluation on 


For Mistral-7B-v0.3, we see a decrease in performance compared to the pre-trained model of 2.17% on the LIMA dataset. For OpenORCA and ABC Align SFT, we see improvements of 12.81% and 14.57% respectively.

For Llama-3-8B, we see an equivalent decrease in performance of LIMA against the pre-trained model of 1.75%. For OpenORCA and ABC Align SFT, we see improvements of 16.34% and 7.20% respectively over the pre-trained baseline.



  §.§.§ Evaluation on 


For Mistral-7B-v0.3, OpenORCA and LIMA improve over pre-trained baselines by 11.01% and 13.69% respectively. ABC Align SFT demonstrates an improvement of 22.64%.

For Llama-3-8B, LIMA improves over the baseline model by 4.56%. OpenORCA and ABC Align improve by 22.57% and 18.26% respectively.



  §.§.§ Analysis


In general, LIMA underperforms in this setting. ABC Align SFT offers the best performance on Mistral-7B-v0.3 across all benchmarks, and on Llama-3-8B for . OpenORCA’s performance exceeds ABC Align SFT on BBQ and TruthfulQA on Llama3-8B.

We see our dataset offers consistent improvements on a specific model, Mistral 7B. Where it underperforms on BBQ & TruthfulQA on Llama-3, it has improved general reasoning capability. The SFT dataset is generated with this aim in mind, as reflected in the ORCA-style prompt. Ideally, we would like to see consistent improvements across a range of open-source models and our three evaluations. These results demonstrate that we need to move beyond SFT and explore PO to achieve this. 




 §.§ Fine-tuning: PO (ORPO)


The PO setting represents the ideal setting for the aims of this paper, where for the ABC Align dataset we have used each of the sources of data unique to our organisation. Preference data in particular is scalable beyond the small number of samples used via our internal RAG tool, to include data sourced from additional workflow integrations. 





  §.§.§ Evaluation on 


For Mistral-7B-v0.3, performance on the pre-trained model decreases by 0.85% for the UltraFeedback (UF) dataset, and ABC Align PO offers a 2.68% improvement. 

For Llama-3-8B, UF and ABC Align PO offer improvements of 2.78% and 6.44% respectively compared to the pre-trained model.



  §.§.§ Evaluation on 


For Mistral-7B-v0.3, UF and ABC Align PO both improve over the pre-trained baseline by 5.26%.

For Llama-3-8B, UF and ABC Align PO both improve over the pre-trained baseline by 7.64% and 7.48% respectively.



  §.§.§ Evaluation on 


For Mistral-7B-v0.3,, we see improvements over the pre-trained baseline for both UF and ABC Align PO of 12.25% and 17.97% respectively.

For Llama-3-8B, we see improvements over the pre-trained baseline for both UF and ABC Align PO of 21.75% and 23.51% respectively.



  §.§.§ Analysis


Here we see ABC Align PO outperforms or offers equivalent performance to the control dataset used by ORPO in all cases. These results demonstrate consistent performance improvements across both base models and all benchmarks. They emphasise the importance of including additional data sources in a PO setting, where ORPO’s penalisation of disfavoured responses better aligns model outputs in a measurable way. 



 §.§ Comparison of Instruction Fine-Tuned (IFT) and SFT/PO Align models


In this section we include results comparing our models and the Instruction Fine-Tuned (IFT) variants of both Mistral-7B-v0.3 and Llama3-8B. Our aim in performing this evaluation is to quantify the impact of large datasets in a fine-tuning setting on the models and benchmarks we have selected.

We note that Mistral do not publish the size of their IFT dataset, while Meta indicates that over 10 million human-annotated samples are included in theirs. We thus restrict our analysis to Llama-3, and include Mistral in our results for completeness. 

We can see in Figure <ref> that our models offer near-equivalent performance gains on . Furthermore, our PO model outperforms the IFT version of Llama3 by 5.2% on , despite our datasets being five orders of magnitude smaller than the IFT dataset. 



The IFT model significantly outperforms both our SFT and PO models on . This result indicates that scaling dataset size and diversity of preference data is likely to offer benefits for specific alignment criteria, represented here by . Identifying socio-cultural biases is a nuanced rather than determinative task, where a wider distribution of samples would be valuable. We elaborate further on this analysis in the ‘Future Work’ section of this paper.  



 §.§ In-Context Alignment


Here we present results of an ICA experiment on a subset of evaluations using data from our ABC Align datasets. This data includes both our internal RAG tool’s system prompt, and prompts designed specifically for the ICA setting, constructed using the ABC AI Principles. These prompts were evaluated using OpenAI’s GPT-4-turbo. It is important to note, as discussed in the ‘In-Context Alignment’ section, that these results are not comparable across the fine-tuning and ICA settings.



We do not include the prompt for our internal RAG tool in this paper. We defer it and a discussion on its construction in a future publication.
 
The prompt referred to as  in Figure <ref> includes our AI Principles themselves, whereas  is an augmentation of the former including a Q&A pair sourced from our internal RAG tool. Please refer to Appendix B for the full v1 prompt.



  §.§.§ Evaluation on 


Compared to the baseline, the improvements across the three versions of our ICA prompts of up to 78.58%.



  §.§.§ Evaluation on 


Results on  demonstrate that our custom prompts reduce performance on this benchmark overall by up to 5.55% across BLEU and ROUGE scores.



  §.§.§ Summary


These results demonstrate that much work remains on ICA in general. We do see significant improvements across the three prompts in comparison to the baseline ‘You are a helpful AI Assistant’ on the BBQ benchmark, including a 26% improvement on BBQ for the AI Principle-based prompt that includes a canonical Q&A pair taken from our internal RAG tool (‘v2’).  However, neither the highly-crafted internal RAG prompt nor the AI Principle-based prompt improve performance on TruthfulQA over the baseline, implying that further work exploring automated prompt optimisation and similar techniques is warranted for improved results.



§ DISCUSSION 

In this paper we have defined an organisational context, a PSM subject to operational constraints, that motivates our choice of models and methods. One constraint that PSMs do not face is availability of high-quality data. Indeed, that is their key competitive advantage. 

In the literature, alignment methods leverage a mixture of synthetic and human-annotated data to achieve their results, however PSMs have domain- and subject-matter expertise available at a scale that the providers of frontier models and related services do not.

Building on this insight, we have demonstrated that by applying several techniques across SFT, PO, and ICA to this data, we can outperform existing methods that use datasets constructed without that domain expertise.

Our experiments have been conducted with a small dataset, and a small number of human-reviewed samples. Integration of the underlying platform into a range of systems that support content workflows, with schema appropriate to the PO, and emerging ICA settings, will enable the scaling of our approach beyond the results presented here.

As PSMs and continue to leverage LLMs to address a variety of use-cases, our results demonstrate that they can both align these models using their existing content data, as well as editorial feedback (in the form of preferences) gathered directly from users. These data can be conditioned via natural-language prompts, which provide an additional mechanism for providing a combination of technical and editorial oversight.

Paired with evolving work on principle alignment, a key part of our contribution in this paper, organisations that implement feedback loops at multiple points of intervention in their workflows can expect to exert a measure of control over large models and their behaviours.

Evaluation of these outputs remains a key consideration. In this paper we have demonstrated the effectiveness of consistent evaluation across multiple controls, through employing the Language Model Evaluation Harness, an open, extensible framework that allows for the definition and integration of organisation-specific benchmarks.



§ LIMITATIONS 


The aim of this paper is not to exhaustively evaluate all possible combinations of algorithm hyperparameters, model architectures and quantisation techniques. Nor has our aim been to evaluate an iterative or complete set of synthetic data and ICA prompt modifications. 

Our primary goal has been to demonstrate that a combination of methods present in the literature and our organisation’s data can be used to achieve our goal of sustainably aligning LLMs to organisational preferences and principles. We achieve this while maintaining the LLM's underlying reasoning capability while operating within organisational constrains and requirements specific to PSMs.



§ FUTURE WORK


In this section, we outline how we will build on the results presented here, beyond optimisation of the individual techniques used in our methodology.



 §.§ Custom Evaluations


Defining custom evaluations, going beyond the industry-standard benchmarks we have used in this paper, is critical to ensure we are aligning LLMs towards organisational standards that are explicitly defined for measuring model capabilities and performance. We will define both general and use-case specific custom evaluations. 



 §.§ Datasets


While we achieve results that are significant given the size of our dataset, particularly in the consistency with which our methods improve reasoning capability, comparisons with IFT versions of open-source models indicate that an increase in PO dataset size and diversity will likely yield further improvements. Following work on custom evaluations, we aim to source a variety of preference data, reflecting domain- and subject-matter expertise, to build on the results presented in this paper. The combination of methods we have outlined give us the flexibility to optimise that data for alignment of LLMs in multiple settings. 



 §.§ Prompts


We see natural language prompts as a key point of human intervention and human oversight, both in the example of our internal RAG tool, and in defining common integration patterns and product identities, via system prompts that either guide outputs or define API integration patterns, or across a range of other applications.

The collaborative definition and evaluation of these prompts will form a key part of future work. The authors believe evolving our alignment framework to facilitate this goal is critical, particularly for emerging ‘agent’ frameworks where LLMs are a key component of goal- and task-directed systems.

In the context of a PSM, the human oversight of AI systems and the accountability & transparency it affords will continue to be a matter of policy, compliance, and likely legislative obligation. By establishing a methodology for alignment and evaluation of LLMs, this can be incrementally improved and expanded upon as our organisation’s obligations and commitments change. We aim to enable our organisation to leverage the capabilities of new models without compromising its values or independence.



 §.§ Multimodality and ICA


Multimodal LLMs have wide-ranging application across text, audio, and video content. We expect to evaluate and refine our methodology in this context. We consider new and emerging evidence that the embedding representation across these modalities is converging, particularly as the underlying models themselves grow larger <cit.>.

By implication, the alignment and evaluation of frontier models via ICA will be critical, and we expect to develop a methodology here that is generally applicable to any further development that accords with these early results.




 §.§ Acknowledgements


The authors would like to thank those who have contributed to the larger body of work that this paper implies. Indeed, while this paper outlines a number of specific contributions to the Machine Learning literature, these experimental results were dependent on an infrastructure and product built by a much larger team. The team are responsible for putting in place the foundations that enable our organisation to work with LLMs at scale. Without their efforts and these foundations, our contribution and future work would not be possible.

We would like to thank the ABC’s complete ML/AI team across all domains of practice, including Scott Burns, Michael Collett, Julie Tran, Mahesh Bisl, Hanna Karlsson, Vijayan Rajendran. 

Our long-term collaboration with the ABC’s Innovation Lab has demonstrated what can be done when we work together across organisational lines. Matthew Heffernan and Andrew Hystek-Dunk worked directly on ABC Assist and were integral to the development process.

We also express our gratitude to the ABC’s Engineering, Product, and Technology leadership for the support to do this work, and our Corporate Strategy team for their invaluable feedback on ABC Assist throughout the development process.





§ APPENDIX A




 §.§ ABC AI Principles


The ABC is committed to ongoing innovation to improve the media services it delivers to audiences. AI technologies have the potential to offer new opportunities to empower content creation, strengthen research and reporting, and make ABC content more accessible. These technologies also offer ways to help our teams work more effectively. At the same time, AI technologies are not without risks and will require careful management to ensure we use these technologies responsibly and in ways that maintain or strengthen the trust that audiences place in the ABC.

Our use of AI will adhere to the following principles. We will develop policies and guidance to provide practical detail on the application of these principles.



  §.§.§ Serving Audiences


We will use and experiment with AI tools that can strengthen the services we provide to audiences.



  §.§.§ Accuracy


We will ensure that our staff are accountable for any ABC content created using AI. This is especially important for news, information and factual content.



  §.§.§ Openness and Transparency


We will inform our audiences about how we are using AI technologies. We will be able to explain how the AI works and how it will affect our audiences. We will not use AI in ways that could mislead our audiences.



  §.§.§ Protecting Data


We will work to ensure that our use of AI protects both the ABC’s data and the personal information the public entrusts to us.



  §.§.§ Mitigating Bias


We will seek to prevent bias in our AI data or algorithms that could perpetuate and amplify existing inequalities and lead to discrimination.



  §.§.§ Respect for Creators' Rights


We will always consider the rights of creators and rights-holders when using AI. We will protect our rights in the works that ABC staff create.



  §.§.§ Experimentation and Evaluation


We will continue to assess and experiment with AI technologies as they evolve to identify opportunities for innovation and mitigate potential risks for our audiences and organisation




§ APPENDIX B


You are a helpful assistant working for ABC Australia and are guided by their AI Principles. 

Here is the list of principles and their definitions: 



  §.§.§ Serving Audiences


We will use and experiment with AI tools that can strengthen the services we provide to audiences. 



  §.§.§ Accuracy

We will ensure that our staff are accountable for any ABC content created using AI. This is especially important for news, information and factual content. 



  §.§.§ Openness and Transparency


We will inform our audiences about how we are using AI technologies. We will be able to explain how the AI works and how it will affect our audiences. We will not use AI in ways that could mislead our audiences. 



  §.§.§ Protecting Data


We will work to ensure that our use of AI protects both the ABC’s data and the personal information the public entrusts to us. 



  §.§.§ Mitigating Bias


We will seek to prevent bias in our AI data or algorithms that could perpetuate and amplify existing inequalities and lead to discrimination. 



  §.§.§ Respect for Creators' Rights


We will always consider the rights of creators and rights-holders when using AI. We will protect our rights in the works that ABC staff create. 



  §.§.§ Experimentation and Evaluation


We will continue to assess and experiment with AI technologies as they evolve to identify opportunities for innovation and mitigate potential risks for our audiences and organisation. 


Here is an example of a question/answer aligned with these principles: 

... 

plainnat




 Paper Title: Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning.









IEEE Transactions on Neural Networks and Learning Systems
Shell et al.: Bare Demo of IEEEtran.cls for Computer Society Journals



§ ABSTRACT 
For on-policy reinforcement learning, discretizing action space for continuous control can easily express multiple modes and is straightforward to optimize.
However, without considering the inherent ordering between the discrete atomic actions, the explosion in the number of discrete actions can possess undesired properties and induce a higher variance for the policy gradient estimator. 
In this paper, we introduce a straightforward architecture that addresses this issue by constraining the discrete policy to be unimodal using Poisson probability distributions.
This unimodal architecture can better leverage the continuity in the underlying continuous action space using explicit unimodal probability distributions.
We conduct extensive experiments to show that the discrete policy with the unimodal probability distribution provides significantly faster convergence and higher performance for on-policy reinforcement learning algorithms in challenging control tasks, especially in highly complex tasks such as Humanoid.
We provide theoretical analysis on the variance of the policy gradient estimator, which suggests that our attentively designed unimodal discrete policy can retain a lower variance and yield a stable learning process.




Reinforcement learning, unimodal probability distributions, Poisson distributions, on-policy learning




Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning
    Yuanyang Zhu, Zhi Wang, Member, IEEE, Yuanheng Zhu, Senior Member, IEEE, Chunlin Chen, Senior Member, IEEE, Dongbin Zhao, Fellow, IEEE


The work was supported by the National Natural Science Foundation
of China under Grant 62006111 and Grant 62073160.
	
Yuanyang Zhu, Zhi Wang and Chunlin Chen are with the Department of Control Science and Intelligent Engineering, School of Management and Engineering, Nanjing University, Nanjing 210093, China, and also with the Research Center for Novel Technology of Intelligent Equipment, Nanjing University, Nanjing 210093, China (e-mail: yuanyang@smail.nju.edu.cn; zhiwang@nju.edu.cn; clchen@nju.edu.cn).
 
 Yuanheng Zhu and Dongbin Zhao are with the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China (e-mail: yuanheng.zhu@ia.ac.cn; dongbin.zhao@ia.ac.cn)

    
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================












§ INTRODUCTION


Deep reinforcement learning (DRL) has presented a powerful paradigm for learning new behaviors from scratch both on physical and simulated challenging tasks <cit.>.
The action space of conventional reinforcement learning (RL) tasks can be discrete, continuous, or some combination of both <cit.>.
In simulation <cit.> and real life <cit.>, continuous control usually requires some subtle parametric functions for a compact representation of action distributions, typically Gaussians, due to an infinite number of feasible actions in a continuous action space <cit.>.
This underlying assumption is that it can enable more refined decisions when control policies can cover all feasible control inputs, which brings difficulty in maximizing an arbitrary function for a continuous action space <cit.>.
Furthermore, it hinders applying dynamic programming approaches to back up value function estimates from successor states to parenting states.

Discretizing a continuous control problem is a scalable solution that retains the simplicity of discrete actions and allows for theoretical analysis <cit.>.
With such a setup, impressive results have been obtained in high-dimensional continuous action spaces <cit.>.
However, the naive discretization approach results in exponentially large discrete action space, where ballooning action space can quickly become intractable.
It would be a balancing dilemma that the resulting output may not be smooth enough when the discretization is coarse, and the number of discretized actions may be highly intractable otherwise.
Moreover, more fine discretization of the action space may struggle to capture the information about the class ordering of the continuous action space, making it challenging to generalize across discrete actions.
Furthermore, it also brings a larger variance of the policy gradient estimator, which may hurt the stability and performance, especially for highly complex tasks <cit.>. 
In practice, more fine-tuned hyperparameters are also required to prevent policy collapse.

To tackle these intractabilities, prior works <cit.> assume that action dimensions are independent and developed to factorize distributions across action dimensions using an ordinal parameterization of the joint distribution over discrete actions, which has shown improved performance in finding near-optimal solutions for high-dimensional tasks.
By parameterizing the discrete policy using an ordinal parameterization, the natural ordering between discrete actions can be implicitly encoded, which provides an additional inductive bias that can improve the generalization across actions <cit.>.
In contrast, the conventional approach of parameterizing the discrete policy as a Gibbs distribution fails to consider the order of actions and ignores the discrepancy relationships between them <cit.>.
Existing methods usually use some heuristic network architecture design to implicitly encode the ordinal information into the distribution over discrete actions, which may not guarantee a unimodal probability distribution to efficiently capture the ordinal information within atomic actions <cit.>.
In their work, they alleviate this issue by designing the loss function to impose an ordering on the action, which does not impose unimodal constraints.
Here, we propose a more explicit and efficient ordinal parameterization method that better incorporates the notion of continuity when parameterizing the distribution over discrete actions, while avoiding the need for an exponentially large number of actions. 
Additionally, we impose an ordering constraint on the action space of each dimension through the loss function to further improve the performance of RL algorithms.


This paper mainly addresses how to efficiently leverage the internal ordering information in the continuous action space to enhance generalization across the discretized set of actions for on-policy RL without an explosion in the number of parameters.
Inspired by the deep ordinal classification approach <cit.>, for each independent action dimension, we constrain the distribution over discrete actions to be unimodal to explicitly exploit the inherent ordering between discrete actions.
To implement the unimodal constraint, we employ the Poisson probability distribution, which allows the probability mass to decrease gradually on both sides of the action with the majority of the mass.
Our framework necessitates learning a probability mass function for each action dimension, substantially reducing the number of units in the network output layer and enabling us to circumvent the curse of dimensionality as the discretization action bins increase (see Fig. <ref>).
It facilitates fine-grained discretization of individual domains without incurring a significant increase in the number of parameters.
The experimental results show that our method outperforms the baselines by a wide margin in the suite of MuJoCo continuous control tasks <cit.>.

We highlight the following contributions of this work: 
 
  
  * We propose a novel ordinal framework that enforces the distribution over discrete actions to be unimodal, effectively capitalizing on the continuity presented in the action space.
      We utilize the probability mass function of the Poisson distribution to enforce unimodal probability distributions for each dimension action space, which can place more confidence on the majority probability distribution beneﬁting from its inherent unimodal distribution.
  
  * Theoretically, we present a variance analysis for the policy gradient estimator, where our unimodal policy attains a lower variance for the estimator.
  
  * We show experimentally that an ordinal parameterization of the unimodal policy outperforms other competing approaches for on-policy RL on continuous control tasks.
  The promising results showcase the versatility and effectiveness of our methods compared to the baselines.


The remaining paper is organized as follows.
In Section <ref>, we summarize preliminaries of RL and related literature.
In Section <ref>, we present the unimodal probability distributions for on-policy RL.
In Section <ref>, we give comprehensive experimental results of learning performance and stability analysis.
The paper is concluded by Section <ref>.



§ BACKGROUND AND RELATED WORK



 §.§ Preliminaries

RL is an approach to solving optimal sequential decision-making tasks, built upon the concept of Markov Decision Processes (MDPs). 
In the finite MDP framework, a tuple of ⟨ S,A,T,R,γ⟩ defines the problem, where S is a countable state space, A is a finite action set, T: S × A × S →[0,1] represents the transition kernel, R: S × A→ℝ is the reward function, and γ∈[0,1) is the discount factor. 
A stochastic policy is defined as π: S × A →[0,1], which maps environmental states to distributions over actions with the constraint ∑_a∈Aπ(a|s)=1,∀ s∈S.
The objective of the RL problem is to find an optimal policy π^* that maximizes the expected total discounted return by

    J(π)=𝔼_τ∼π(τ)[r(τ)]=𝔼_τ∼π(τ)[∑_i=0^∞γ^i r_i],


    Q_π(s, a)=𝔼_s_0=s, a_0=a, τ∼π[∑_t=0^T γ^t r_t],

where τ = (s_0, a_0, s_1, a_1, ...) is a learning episode, π(τ)=p(s_0)Π_t=0^∞π(a_t|s_t)p(s_t+1|s_t,a_t), and r_t is the immediate reward. 
The policy function h(·|θ) can be represented by a deep neural network parameterized by θ. 
For a discrete action space, the Gibbs distribution is commonly used and given by

    π_θ(a_j | s)=exp(h_j(s |θ))/∑_j ∈ A(s)exp(h_j(s |θ)),

while the Gaussian distribution is generally used for a continuous action space and is given by

    π_θ(a | s)=1/√(2 π)σexp(-1/σ^2(h(s |θ)-a)^2).



To evaluate the performance of a policy π_θ, the direct objective function is optimized by estimating the gradient of the expected return

    J(θ)=𝔼_τ∼π_θ(τ)[r(τ)]=∫_τπ_θ(τ) r(τ) d τ,

where r(τ)=∑_i=0^∞γ^i r_i is the return of episode τ.
The basic idea of the policy gradient algorithm <cit.> gives the direction of the performance gradient where the parameters θ should be updated:

    ∇_θ J(θ)    =𝔼_τ∼π_θ(τ)[∇_θlogπ_θ(τ) r(τ)] 
       =∫_τ∇_θlogπ_θ(τ) r(τ) π_θ(τ) d τ
       ≈∑_i=1^m∇_θlogπ_θ(τ^i) r(τ^i),

where (τ^1, …, τ^m) is a batch of learning episodes sampled from policy π_θ.
Next, an ascent step is taken in estimating the direction of the gradient as θ←θ+α∇_θ J(θ).
The process continues until θ converges <cit.>.



 §.§ Related Work

On-policy RL. On-policy RL is a family of RL algorithms aiming to directly optimize the parameters of a policy, which optimizes expected returns by estimating policy gradients.
This estimation is often prone to high variance, and several solutions have been proposed to mitigate this significant challenge, particularly in problems characterized by long horizons or high-dimensional action spaces <cit.>.
Classic vanilla policy gradient (PG) updates are generally unstable with high variance due to difficulties in crediting actions that influence future rewards, where the gradients require knowledge of the probability of the performed action via resorting to parametric distributions.
Natural policy <cit.> improves upon vanilla PG by computing an ascent direction with the Fisher information that approximately ensures a slight change in the policy distribution.
To obtain more stable learning, trust region policy optimization (TRPO) <cit.> utilizes a larger Kullback-Leibler (KL) divergence value to enforce constraints, and performs a line search in the natural gradient direction, ensuring improvements in the surrogate loss function.
Proximal policy optimization (PPO) <cit.> replaces the KL divergence constraint with the clipped surrogate objective, which strikes a favorable balance across sample complexity, simplicity, and wall time.
Moreover, actor-critic using Kronecker-factored trust region (ACKTR) <cit.> builds upon the natural policy gradient framework by calculating gradients through Kronecker-factored approximate curvature within the trust region.
Orthogonal to the above algorithms, we demonstrate that ordinal parameterization with a unimodal constraint for on-policy RL achieves consistently improved performance based on the above representative algorithms.

Policy Representation.
In the context of policy gradient techniques, the policy is generally parameterized using neural networks, where the policy is improved by optimizing the parameter of the approximation function.
In general, the policy uses a Gibbs probability distribution over the discrete action space to sample.
For the continuous control problems, the default choice for the policy in the baseline is parameterized by learning a Gaussian distribution with independent components for each dimensional action space <cit.>.
The Gaussian mixture, maximum entropy, or normalizing ﬂows <cit.> can be used for more expressive policy classes, providing a promising avenue for improved robustness and stability.
Since physical constraints in most continuous control tasks, actions can only take on values within some ﬁnite interval, resulting in an unavoidable estimation bias caused by boundary effects.
To address the shortcomings of the Gaussian distribution with a finite support distribution, the Beta distribution <cit.> provides a bias-free and less noise policy with improved performance.
Further, considering the extremes along each action dimension, a Bernoulli distribution is applied to replace the Gaussian parametrization of continuous control methods and show improved performance on several continuous control benchmarks <cit.>.
Here, we provide insight into how to produce an efficient ordinal parameterization policy while retaining the internal order information of continuous action space with a simple explicit constraint, which can yield state-of-the-art performance compared to baseline policy classes.

Continuous action discretization.
To leverage the continuity between discrete and continuous action space, converting continuous control problems into discrete ones has been introduced by <cit.> with the “bang-bang" controller <cit.>.
Similar discretization methods represent each action in binary format and optimize its policy for MDPs with large action sets <cit.>.
However, such discretization methods need to be improved due to the curse of dimensionality.
Surprisingly, recent works have relieved the limitation by assuming that action dimensions are independent <cit.>, which obtains improved performance in complex control tasks.
Inspired by multi-agent reinforcement learning, Decoupled Q-Networks (DecQN) <cit.> factorizes the overall state-action value function into a linear combination of single action utility functions over action dimensions by combining value decomposition with bang-bang action space discretization.
It achieves better performance while reducing agent action space complexity.
However, such a discretization paradigm makes the strong assumption of independence among each action dimension and hardly captures the continuous order information of actions <cit.>.

To mitigate the exponential explosion of discrete actions, <cit.> utilizes sequence-to-sequence models to develop policies for structured prediction problems, but their strategy has only demonstrated effectiveness in high-dimensional tasks like Humanoid and HumanoidStandup tasks.
Recently, <cit.> parameterizes the discrete distributions with an ordinal architecture and achieves improved performance.
Nevertheless, existing methods face a dilemma in that the number of network outputs increases linearly with the number of action space dimensions, which may bring high variance, resulting in a more unstable learning process.
A key distinction between our work and previous works lies in our explicit utilization of unimodal distributions with only one parameter for each action dimension. 
It can help the policy capture the most confident class while ensuring that the probability gradually decreases on both sides of the class. 
Finally, we find that our proposed method outperforms baseline algorithms, especially in high-dimensional tasks, which should benefit from the unimodal distribution architecture.
We summarize the above continuous action discretization RL works in Table <ref>.





§ UNIMODAL PROBABILITY DISTRIBUTIONS FOR ON-POLICY REINFORCEMENT LEARNING

In this section, we will present our proposed unimodal distribution with Poisson probability distribution for on-policy RL algorithms, which explicitly introduce unimodal probability distribution into the ordinal parameterization method to leverage the continuity in the underlying continuous action space.
First, we present the process of discretizing action space for continuous control tasks and apply it to on-policy RL.
Then, we describe the unimodal ordinal architecture that constrains the discrete policy to be unimodal via the Poisson probability distribution in a practical neural network implementation.
Finally, theoretical analysis shows that our method can relieve the high variance trouble of the policy gradient estimator compared to existing ordinal parameterization methods.




 §.§ Discretizing Action Space for Continuous Control

Consider an environment with state s_t and m dimension action spaces a ∈ℝ^m.
To ensure generality, we consider an action space 𝒜=[-1,1]^m and proceed to discretize each dimension of the action space into K evenly spaced atomic action bins.
As a result, we obtain a discrete and finite set of atomic actions 𝒜_i={2 j/K-1-1 }_j=0^K-1 for the i action dimension. 
To facilitate the joint policy's tractability, the policy generates a tuple of discrete actions accompanied by factorized categorical distributions across action dimensions. 
This approach obviates the need to enumerate every possible actions, as required in the discrete case.
Specifically, we denote a categorical distribution π_θ_i (a_j|s) as the atomic action for actions a_i∈𝒜_i at each step t, where θ_i is the factorization parameters for this marginal distribution induced by Poission probability distributions.
Such representations can build complex policy distributions with explicit probability distributions, where any action subset can be formulated as

    π(a|s) = Π_i=1^mπ_θ _i(a_i|s),

where a = [a_0, a_1, ..., a_K-1]^T and π_θ _i(a_i|s) represent the probabilities of selecting the actions a_i for the i-th dimension action space.
Based on this factorization, we can easily employ RL algorithms to maintain a tractable distribution over joint actions. 
Meanwhile, it can vastly reduce the computation of neural networks due to the fewer learning parameters.



 §.§ Unimodal ordinal architecture

In on-policy RL algorithms, the policy gradient involves a stochastic policy that specifies the probability distribution of selecting an action a given a state s.
Upon discretizing the continuous action space, the complexity of directly training such policy networks could lead to K^m combinations of joint atomic actions, which exponentially increase with the growing number of action dimensions m.
Using the categorical distribution (e.g., Gumbel-Softmax distribution) as a stochastic policy for discrete action space has been well-studied in the RL community.
Popular categorical distributions have been used as a stochastic policy for discrete action spaces due to their ease of sampling and computation.
However, the curse of dimensionality can lead to intractable action spaces, putting significant pressure on the distribution parameterization over discrete actions.
In fact, a fine control policy can be captured more as the resolution of the discretized action space increases. 
This quickly plunges into the dilemma that intractable action space puts high pressure on the parameterizing distributions over discrete actions, which hinders capturing the ordering among actions.

To relieve the scalability issue, a few prior works have been exploring more expressive distributions and notably proposed an ordinal parameterization <cit.>.
It introduces an implicit loss function based on factorizing a tractable distribution over joint actions, where the ordinal policy can easily do both sampling and training.
Motivated from the loss function of <cit.>, it implicitly introduces internal ordering information between classes while maintaining the probabilistic properties of discrete distributions instead of simply parameterizing the discrete policy as a categorical distribution, ignoring the order of actions and the discrepancy relationship between the joint actions.
Existing methods usually utilize some heuristic design of the network architecture to learn distributions for each action dimension, which could not guarantee a unimodal probability distribution output followed by the neural network to efficiently capture the ordinal information within atomic actions.
It generally has a strong underlying assumption that the output logits of the neural network are unimodal for each dimension action space, which makes it hard to guide the neural network to optimize weights and generate probability distributions that closely resemble the optimal action.
In other words, it may allocate greater confidence to multi-action classes, as it generally produces multi-modal probability distribution.

To leverage the continuity in the underlying action space, we focus on how the discretized actions are distributed: why restrict ourselves to the categorical distribution?
For continuous control tasks, a Gaussian or Beta policy can explicitly capture the inherent ordering structure within an infinite number of actions due to the privilege of their unimodal distribution property.
Is it possible to explicitly retain the unimodal structure and inherent ordinal information after discretizing the continuous action space for learning a policy over discrete actions?
Inspired by the deep ordinal classification approach <cit.>, we constrain the distribution of the discretized actions to be unimodal explicitly for utilizing the ordinal information about the underlying continuous space.
Specifically, we learn the probability mass function (PMF) of the Poisson distribution to enforce discrete unimodal probability distributions for each action dimension.
This PMF can facilitate the generation of the most coherent unimodal probability distribution, as it diminishes the probability mass on both sides of the action possessing the majority of the mass, allowing us to adeptly capture the intrinsic ordering structure present within the action space.


The Poisson distribution, a widely recognized discrete probability distribution, is typically used to model the likelihood of observing a specific count of events within a designated time interval.
It is characterized by a positive constant mean rate, denoted as λ∈ R^+, and presumes that events transpire independently of the elapsed time since the previous event. 
In other words, the distribution describes the probability of k ∈ℕ events occurring during a given time interval, where k can take on any non-negative integer value.
For i-th dimension of action space, the Poisson distribution function is denoted as p(k; λ_i)=λ^kexp (-λ_i)/k !, and 0 ≤ k ≤K-1 (minus one since we index from zero).
For a purely technical implementation reason, we focus on the log of the PMF, which can be expressed as

    log[λ_i^kexp (-λ_i)/k !]    =log(λ_i^kexp (-λ_i))-log (k !) 
       =log(λ_i^k)+log (exp (-λ_i))-log (k !) 
       =k log (λ_i)-λ_i-log (k !).

We denote the scalar output of our deep network as F(s) = [f_1(s), f_2(s),..., f_i(s)], where f_i(s)>0 is enforced to be positive using the Softplus nonlinearity.
With such a parameterization, our method only needs to learn M probability mass functions with M units in the network output layer, in contrast to existing ordinal parameterization methods that require learning the complete probability distributions over all dimensions using M*K units in the network output layer <cit.>.
Especially in high-dimensional continuous action spaces, our unimodal method can make it more tractable to implement with deep neural network representations.
Meanwhile, our method can eliminate the assumption that the ordinal parameterization method could produce a unimodal probability distribution only when the logits of the network output are unimodal.
By simply replacing the λ_i in equation (<ref>) with f_i(s), we denote h_i(s)_j to be

    j log (f_i(𝐬))-f_i(𝐬)-log (j !).






In the RL community, the categorical distribution architecture is widely used to parameter a discrete distribution over K action classes, which is represented by K logits denoted as h_i(𝐬)_j.
Due to the fact that the Poisson distribution has infinite support, we use a `right-truncated' operation by applying a Softmax operation and obtain the corresponding probability distribution p(a_ij) as

    p(a_ij|s)=exp(-h_i(s)_j/ τ)/∑_j=0^K-1exp(-h_i(s)_j/ τ), 0≤ j ≤ K-1.

Besides, we introduce the hyperparameter τ to the Softmax function to control the variance of the distribution.[Note that as τ→∞, the probability distribution becomes more uniform, leading to more efficient exploration. Conversely, the distribution becomes more `one-hot' like encouraging more deterministic behavior otherwise.]
After these operations, the probability distribution remains unimodal, which is clearly illustrated in Fig. <ref>.


In our unimodal policy architecture, we retain these logits h_i(s)_j with the support of the Poisson distribution, and then transform them via a Softmax operation to relieve the infinite support problem of Poisson distribution.
Inspired by the ordinal architecture introduced in <cit.>, in order to enhance the additional dependencies between logits h_i(s)_j, we use the ordinal architecture to implicitly inject the information about the action ordering while maintaining all the probabilistic properties of Poisson distribution.
The ability to readily distinguish samples drawn from action k from those drawn from all actions j>k is advantageous, as it facilitates the identification of maxima in such cases. 
Therefore, for all logits ∀ 1 ≤ l ≤ K, we compute the final logits as

    h_i^'(s)_j=∑_j ≤ llog p(a_ij|s) 
    
    +∑_j>llog(1- p(a_ij|s) ),

and derive the final output probability using a Softmax p̂(a_ij) = Softmax(h_i^'(s)_j).
The implementation of this approach is illustrated in terms of the layers at the end of the deep network, as depicted in Fig. <ref>.


It is noteworthy that utilizing independent unimodal probability distributions for producing units guarantees the monotonic relationship (p̂_i0<p̂_i1<p̂_i2,...,<p̂_ik and p̂_ik>p̂_ik+1,...,>p̂_iK), which is desirable for imposing inequality constraints on the outputs to maintain monotonicity.
Combining unimodal probability distributions and the ordinal architecture, the unimodal policy makes the most sense as it allows for easy separation of samples drawn from action k and those drawn from all other actions, enhancing the selected probability of the most conﬁdent action.
With such unimodal parameterization, the form of unimodal probability distributions can easily find maximal concerning actions while retaining a smaller space of neural architectures.





 §.§ Variance Analysis

While PG algorithms are well-suited for parametric distribution functions, it is susceptible to converging to a poor policy that is not even a local optimum. 
One of the primary limitations of the plain policy gradient method is the high variance of the policy gradient estimator.
Using the Gaussian distribution as a stochastic policy in continuous control, π_θ(a| s)=1/√(2 π)σexp(-(a-μ)^2/2 σ^2), has been well-studied and commonly used in the RL community since <cit.>, because the Gaussian distribution is easy to sample and has gradients that are easy to compute.
However, one undesirable property of Gaussian policy is that the variance of the policy gradient estimator tends to increase as the standard deviation σ^2 of the Gaussian distribution decreases <cit.>.
Generally, the policy gradient estimator is given as 

    ĝ_θ=r(a_j) ∇_θlogp̂_θ(j), j ∼p̂_θ(j),

and the variance can be represented as

    𝕍[ĝ_θ]=𝔼[ĝ_θ^2]-𝔼^2[ĝ_θ]

For Gaussian distribution, as the policy improves and becomes more deterministic (σ→ 0), the variance of the policy gradient estimator goes to inﬁnity, which makes PG methods sample-inefficient with respect to interactions with the environment and hinders the applications to real physical control systems.

As mentioned earlier, the variance of the Poisson distribution is equivalent to its mean, whose properties can alleviate the above high variance problem.
Before showing the theoretical analysis for the variance of unimodal policy, we first make the following assumption that considering a simplified setting of a one-step bandit problem with one dimension action space 𝒜 = [-1,1] and fixed constant reward r(a) = R, for all actions a.
We denote p(j) to be the probability of taking the jth action 0 ≤ j ≤K-1, and upon initialization, the policy has very high entropy p(j)≈1/K.


Recall that the policy gradient estimator is given by

    ĝ_θ=r(a_ij) ∇_θlog p_θ(j), j ∼ p_θ(j).

With such a setting, we can derive the expectation of the policy gradient estimator 𝔼[ĝ_θ]=∇_θ J(π_θ)=0 and the variance is 

    𝕍[ĝ_θ]    = ∑_j=0^K-1 r^2(a_j) (∇_θlog p(j))^2 p(j) 
       ≈ R^21/K∑_j=0^K-1(∇_θlog p(j))^2
       ≈ R^21/K∑_j=0^K-1(∇_θ L_j-1/K∑_k=0^K-1∇_θ L_k)^2,

where ∇_θ L_j denotes the jth logit independently depending on θ, and the approximations come from p(j)≈1/K.
In the context of standard neural network initialization schemes, wherein all weight and bias matrices (e.g., θ, w_j^T h, and b_j) are initialized independently, the gradients ∇_θ L_j can be regarded as identically and independently distributed (i.i.d.) random variables. 
The stochastic nature of these variables originates from the random initialization of the neural network parameters.
We employ the notation 𝔼_init[·] to represent the expectation concerning neural network initializations. 
With this notation in place, we analyze the expectation of the specified equation of (<ref>) as follows:

    𝔼_init [𝕍[ĝ_θ]] 
       ≈R^2/K𝔼(  ∑_j=0^K-1( ∇_θ L_j - 1/K∑_k=0^K-1∇_θ L_k)^2) 
        = R^2/K𝔼(∑_i=1^N∑_j=0^K-1( ∇_θ L_j - μ + μ - 𝔼(∇_θ L)  )^2) 
        = R^2/K∑_j=1^K𝔼( ∇_θ L_j - μ)^2 - K (𝔼(∇_θ L)-μ)^2
        = R^2/K( K 𝕍_init( ∇_θ L) - K 𝕍_init (𝔼(∇_θ L)) ) 
        = R^2/N( 𝕍_init( ∇_θ L) - 𝕍_init (𝔼(∇_θ L)) ),

where μ is the exception of ∇_θ L.

Let ĝ_θ_1 and ĝ_θ_2 denote the policy gradient estimator using the Poisson distribution policy and the ordinal distribution policy, respectively.
By analyzing the expectation of (<ref>), we can derive the expressions for the variances of the policy gradient estimators of the Poisson distribution

    𝔼_init [𝕍[ĝ_θ_1]] 
        = R^2( 𝕍_init( ∇_θ_1 L) - 𝕍_init (𝔼(∇_θ_1 L)) ) 
       ≈ R^2K-1/Kσ_1^2∼K-1/K,

where σ_1^2=𝕍_init[∇_θ_1 L_j] is the variance of the PMF of Poisson distribution produced by one logit.
Then we analyze the expectation of (<ref>) with ordinal probability distribution:

    𝔼_init [𝕍[ĝ_θ_2]] ≈ R^2K-1/Kσ_2^2∼K-1/K,

where σ_2^2=𝕍_init [∇_θ_2 L_j] is the variance of the K logits gradients.
Considering the dependence of ∇_θ_2 L_j on θ_2, it has been argued by <cit.> that finer discretization (larger K) theoretically leads to increased variance for the L_j of the neural network output, which may consequently exacerbate model variance and degrade performance. 
One key advantage of employing the Poisson distribution in statistical applications is the equivalence of variance and mean, suggesting that the variance is only mildly influenced by widespread probability mass distribution, particularly with a large action space.

In practice, we opt for larger K values for refined action selection, harnessing the properties of the Poisson probability distribution as opposed to the ordinal policy, which exhibits greater variance and renders policy optimization more difficult, especially in the context of finer discretization. 
Generally, a moderate action discretization K (e.g., 9≤ K ≤15) strikes an appropriate balance between performance enhancement and computational expense.
Nevertheless, using the Poisson distribution presents a drawback when truncating infinite probability mass distributions. 
Inspired by <cit.>, the infinite support issue is tackled using a truncated Poisson distribution, normalizing probabilities to ensure that their summation equals one. 
This strategy closely mirrors our approach, which incorporates a Softmax operation. 
To regulate the distribution variance induced by the Softmax operation, a judicious selection of τ can help alleviate this concern. 
Our unimodal policy implementation spans a range of τ=[1.5, 3] (with varying K ∈ [9,15]).
As a validation of our approach, we assess the proposed methods on an array of complex control tasks, yielding substantial performance gains. 
These findings imply that our methodology benefits from the diminished variance of constrained neural network output units, which induce unimodal probability distributions characterized by low variance.



§ EXPERIMENTS

The goal of our experimental evaluation is to investigate the improvement and stability of our method in comparison to baseline algorithms and previous Gibbs policy approaches on continuous control tasks.
To distinguish between the different policies, we will use the terms Gibbs policy for the Gibbs distribution, ordinal policy for the ordinal distribution, and unimodal policy for the unimodal architecture.
First, we evaluate our method against the Gaussian policy, employing baseline algorithms (TRPO <cit.>, PPO <cit.>, and ACKTR <cit.>) on challenging continuous control tasks provided by the OpenAI gym benchmark suite <cit.>.
We consider Gaussian policy as a comparison with special attention since it is the default policy class implemented in on-policy baselines base <cit.>.
Additionally, we explore other architectural alternatives, such as Gaussian with tanh non-linearity in the output layer and Beta distribution, both of which have been proposed in prior works <cit.>.
Since a wide range of existing algorithms can solve the easier tasks, we pay more attention to the high dimension tasks, such as the 17-dimensional Humanoid <cit.>. 
Further, we compare the Gibbs policy and ordinal policy, which are prior representative techniques.
Finally, we study the stability of our algorithm, which plays a large role in performance and practicals.
We implement all the methods with Tersorflow 1.15.0 framework in Python 3.6 running on Ubuntu 18.04 with 2 AMD EPYC 7H12 64-Core CPU Processors and 3 NVIDIA GeForce RTX 3080 GPUs.


Implementation Details.
Our investigation focuses on the impact of the discrete unimodal probability distribution policy in conjunction with on-policy RL algorithms. 
To this end, we introduce minimal alterations to the original TRPO, PPO, and ACKTR algorithms, as implemented in OpenAI baselines <cit.>. The implementations for all algorithms (PPO, TRPO, ACKTR) are grounded in OpenAI baselines. 
We conduct a comparative analysis of our method against previous approaches on a diverse range of challenging continuous control tasks, as provided by the OpenAI gym benchmark suite, with action constraints set within the [-1,1] interval as a normalizing flow. 
Detailed descriptions of each policy class utilized in our experiments are provided below.
The implementation code can be found in <https://anonymous.4open.science/r/udprl-tnnls>.

Gaussian Policy.
The factorized Gaussian policy <cit.> is represented as π_θ(·| s)=𝒩(μ_θ(s), σ^2), where the mean μ_θ(s) is derived from a two-layer neural network consisting of 128 units per layer for PPO and ACKTR, and 128 units per layer for TRPO.
The diagonal standard deviation σ^2 is defined as σ^2_ii= σ^2_i, with each σ_i being a single variable shared across all states.
We adhere to the default hyperparameter settings established in the baselines.

Gibbs Policy.
The Gibbs policy <cit.> is represented as π_θ(·| s)=exp(h_i(s |θ) )/τ/∑_j ∈ A(s)exp(h_j(s |θ))/τ with Gibbs distribution over the discrete actions space, which is parameterized by a neural network that shares the same architecture as the Gaussian policy described above.
The temperature parameter τ is set to 1.5.
We define K evenly spaced atomic actions for every dimension, ranging from -1 to 1.
For each dimension of the action space, the discrete distribution encompasses K logits L_j(s), where L_j(s) denotes the logits distribution of the j-th action in state s.

Ordinal Policy.
The ordinal policy <cit.> differentiates it from discrete policies by incorporating an ordinal parameterization while maintaining the same number of parameters.

Unimodal Policy.
In contrast to ordinal policies, unimodal policies utilize a Poisson probability distribution to enforce unimodality. 
 The scalar output of our deep network for unimodal policies is represented as [f_1(s), f_2(s),..., f_i(s)] rather than the K logits represented in discrete policies, where f_i(s) indicates the average frequency of events for the i-dimension atomic actions. 
 The policy parameters are consistent with ordinal policies, except for the addition of a Softplus nonlinearity transformation in the final layer to ensure f_i(s) >0. 
 The temperature parameter τ is set to 2.5 for varying discretization K=9,11,15.
 

Gaussian+tanh Policy.
The Gaussian-tanh policies <cit.> are the same as the Gaussian policies above, but the final layer is added a tanh transformation to ensure the mean μ_θ∈ [-1, 1].

Beta Policy.
The Beta policy <cit.> are represented as π_θ(·| s)=f(a+h/2 h|α, β), where h is max of the closed interal of the action space A=[-h, h]. 
Parameters α=α_θ(s) and β=β_θ(s) are determined by a two-layer neural network f_θ(s), incorporating a Softplus activation function in its final layer.
It is important to note that this parameterization can cause instability throughout the learning process, particularly when α_θ(s)→∞ or β_θ(s) →∞, leading to unstable dynamics. 
In practice, this instability is apparent when the trust region size is large (e.g., ϵ=0.005 and ϵ=0.01), and can result in training termination due to numerical errors. 
Conversely, a smaller trust region size (e.g., ϵ=0.001) may yield diminished performance. 
In our implementation, we employ the main thread settings for the Beta policy, which are derived with a trust region size of ϵ=0.01 for TRPO.

Others Hyperparameters.
For value functions, two-layer neural networks are utilized, consisting of 128 hidden units per layer in the case of PPO and ACKTR, and 64 hidden units per layer for TRPO. 
We used the Adam optimizer (Kingma and Ba 2015) with a learning rate 2 · 10^-5, β_1=0.9, and β_2=0.999 for both PPO and TRPO.
We trained with a minibatch size of 64 and a discount factor γ=0.98.
The KL constraint parameter, ϵ, is set within the range of 0.01, 0.001 for TRPO and 0.02, 0.002 for ACKTR. 
All remaining hyperparameters align with those found in the original baseline implementations.





















 §.§ Comparison with Benchmark Baselines

First, as a proof of concept, we compare the Gaussian policy with the unimodal policy (with varying K) in various MuJoCo control tasks using PPO and TRPO, as displayed in Fig. <ref>.
The results show that, overall, PPO and TRPO with unimodal policy perform comparably to Gaussian policy on the easier tasks (e.g., Walker2D) and outperform them on the more complex tasks, both in terms of learning speed and the final performance (e.g., Humanoid).
PPO with unimodal policy also learns considerably faster and achieves more excellent performance than PPO with Gaussian policy. 
This implies that the Gaussian policy may suffer from unbounded distributions and require more samples to optimize its policy in more high-dimensional tasks.
For TRPO, the performance enhancements brought by the unimodal policy align consistently with PPO.
Both performance gains are most significant with PPO and TRPO, which should benefit from our more attentively designed distribution architecture.

Further, we compare straightforward architectural alternatives: Gaussian with tanh non-linearity as the output layer, and Beta distribution.
The primary rationale behind these architectures is their innate use of continuous unimodal probability distributions and their ability to confine sampled actions within the feasible range ([-1, 1] for MuJoCo control tasks).
By construction, our proposed unimodal policy also bound the sampled actions within the feasible range using the right-truncated operation.
The mean and standard deviation results gathered across an extensive array of control tasks, are summarized in Table <ref>.
In comparison to Gaussian and Gaussian with Tanh policies, the Beta policy yields higher scores, thereby mitigating the bias issue resulting from the infinite support mismatch of Gaussian distribution commonly employed in on-policy RL algorithms.
Our unimodal policy consistently outperforms the baselines across the majority of tasks. 
This outcome implies that the unimodal policy, with ordinal parameterizing distributions constrained by the Poisson distribution, can achieve notable performance by effectively considering the order of discrete actions.
Intriguingly, the unimodal policy, being a simplistic distribution, can attain such performance improvement in policy gradient methods.

Moreover, we also apply unimodal policy to another representative on-policy algorithm ACKTR with different atomic actions and compare it with the Gaussian policy on a set of MuJoCo tasks.
As shown in Fig. <ref>, we find that ACKTR, with the unimodal policy, still outperforms its Gaussian approach by a large margin. 
It implies the superiority of our attentive unimodal architecture rather than larger networks.

Finally, we compare the performance of PPO with a unimodal policy against off-policy algorithms on highly complex Humanoid tasks (Humanoid-v2, Humanoid-v3, and HumanoidStandup-v2), including DDPG <cit.>, SQL <cit.>, SAC <cit.>, TD3 <cit.>, and DecQN <cit.>[For a fair comparison, we set the granularity of DecQN’s discretization to 11 bins and the other hyperparameters settings are the same with DecQN <cit.>.]. 
Off-policy algorithms can potentially achieve better sample efficiency than on-policy algorithms by reusing samples, but for highly complex tasks such as Humanoid, even off-policy algorithms take many more samples to learn due to instability and less informative off-policy samples.
DecQN obtains higher competitive performance than the other baselines, which should benefit from the versatility of simple decoupled discrete control via leveraging a linear factorization over action dimensions.
In our experiments, PPO with unimodal policy achieves comparable or even better results than off-policy baselines, demonstrating its competitiveness in general complex applications. 
Our results show that PPO with unimodal policy is a promising method for complex control tasks, with performance comparable to state-of-the-art off-policy methods.




 §.§ Comparison with Discrete Policy

In a similar vein to our work, the extension of discretizing continuous action space is applied to solve high-dimensional control tasks.
We evaluate the unimodal policy alongside the discrete and ordinal policy architectures on tasks with K=11.
As shown in Fig. <ref>, the unimodal policy still significantly outperforms the Gibbs policy.
When compared to the ordinal policy, the unimodal policy learns a superior policy and achieves better performance due to the unimodal probability distribution.
Under a unimodal constraint, it is guaranteed that the two atomic actions on either side of the majority-selected action receive the next greatest amount of probability mass, which can result in a near-optimal policy during the early stages.
By this construction, the unimodal policy has fewer parameters than the discrete or ordinal policy when they share the same encoding architecture.
The results are predicted by our analysis showing that unimodal probability distributions of policy can still obtain high performance and better policies.
Although the Gibbs or ordinal policy can achieve enhanced performance, it is generally noisier without effectively utilizing the ordering information of discrete actions, relying more on the probability distribution of network output. 
By incorporating Poisson probability distribution constraints, the unimodal policy can more efficiently capture the continuity in the underlying continuous action space for generalization across discrete actions, especially in the high-dimension complex action space tasks.



 §.§ Stability and HyperParameter Analysis

PG methods, being generally defined over parametric distribution functions, are susceptible to converging to sub-optimal extrema, where policy entropy is usually utilized to enhance exploration by preventing premature convergence to suboptimal policies. 
We assess the sensitivity of policy entropy and performance with varying numbers of atomic actions K per dimension, presenting the results for PPO in Fig. <ref>.
As shown in Fig. <ref>, our proposed unimodal policy performs better with a larger number of atomic actions K during the learning process.
With a larger K, we can execute more precise actions through fine discretization, ultimately reaching high-quality final policies. 
From the perspective of policy entropy, finer discretization enables more efficient exploration with larger entropy.
By examining both the performance and policy entropy in Fig. <ref>, we observe that a higher absolute value of policy entropy yields better performance with more refined discretization, consistent with findings from <cit.>. 
Additionally, we also note that a moderate action bin K (e.g., 9 ≤ K ≤ 15) is sufficient for striking an appropriate balance between performance improvement and computational cost. 
The more minor variance of policy gradients tends to be more stable with finer discretization (larger K), potentially benefiting from the limited network output units inducing lower variance in the unimodal distributions.

Further, we study the performance of unimodal policy with varying temperature τ with 1.8, 2.4, and learnable bias, where the learnable τ is clipped to [1.5, 3] as a bias.
As shown in Fig. <ref>, our method can achieve robust performance under different temperatures τ.
Note that a lower temperature τ could reduce exploration and encourage more deterministic behavior, which may lead to suboptimal policies without exposure to diverse environments.
The learned τ is more stable under different fixed temperatures τ, though it does not show any significant gain over simply learning it as a bias.
One advantage of this technique is that the network can estimate its uncertainty on a per-example basis. 
Compared to their fixed temperature, it can better trade-off exploration versus exploitation under parameterized conditions.





§ CONCLUSION AND LIMITATIONS

In this paper, we introduce a straightforward technique to enforce a unimodal ordinal probabilistic policy distribution using Poisson distributions. 
Given the inherent ordering between discretized actions in continuous control tasks, this property is crucial to consider in discrete stochastic policy gradients. 
Despite its simplicity, our approach significantly enhances the performance and stability of baseline algorithms, particularly in high-dimensional tasks with complex dynamics. 
Furthermore, the unimodal ordinal policy allows probability distributions to behave more sensibly in benchmark continuous control tasks.
For limitations, since we mainly aim to refine off-the-shelf on-policy RL algorithms, it is not sufficient to conclude that our method can transcend off-policy algorithms.
In a practical view, our unimodal policy is more suitable for improving policy for high-dimension control tasks in bounded time, while the current form does not provide an optimality guarantee.
For future work, we will focus on designing more attentive unimodal distributions and extend them to multi-agent reinforcement learning.

IEEEtranN
























































 Paper Title: LoRaWAN Based Dynamic Noise Mapping with Machine Learning for Urban Noise Enforcement.


	
	
	
	
	How to Use the IEEEtran  Templates
	
	Based Dynamic Noise Mapping with Machine Learning for Urban Noise Enforcement
    
		
			H. Emre Erdem
			, Henry Leung
		
		This work was supported in part by the Alberta Jobs, Economy and Innovation Ministry’s Major Innovation Fund to the Centre for Autonomous Systems in Strengthening Future Communities and in part by the Natural Sciences and Engineering Research Council of Canada.
	
    
======================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
		Static noise maps depicting long-term noise levels over wide areas are valuable urban planning assets for municipalities in decreasing noise exposure of residents.
		However, non-traffic noise sources with transient behavior, which people complain frequently, are usually ignored by static maps.
		We propose here a dynamic noise mapping approach using the data collected via low-power wide-area network (LPWAN, specifically LoRaWAN) based internet of things (IoT) infrastructure, which is one of the most common communication backbones for smart cities.
		Noise mapping based on LPWAN is challenging due to the low data rates of these protocols.
		The proposed dynamic noise mapping approach diminishes the negative implications of data rate limitations using machine learning (ML) for event and location prediction of non-traffic sources based on the scarce data.
		The strength of these models lies in their consideration of the spatial variance in acoustic behavior caused by the buildings in urban settings.
		The effectiveness of the proposed method and the accuracy of the resulting dynamic maps are evaluated in field tests.
		The results show that the proposed system can decrease the map error caused by non-traffic sources up to 51% and can stay effective under significant packet losses.
	
	
	
		Acoustic localization, IoT, LoRaWAN, noise mapping, noise monitoring, wireless sensor networks.
	
	
	

§ INTRODUCTION

	
	Noise exposure is long proven to be a primary health problem due to its negative impacts on main body functions such as sleep, hearing, and mental capacity <cit.>.
	Thus, diminishing noise exposure plays a key role in improved quality of life.
	For this purpose, municipalities around the globe utilize noise management practices to constrain the exposure within healthy limits <cit.>.
	These practices include passive measures such as consideration of static traffic noise maps for new development plans as well as active noise enforcement measures such as elimination of noise limit violations.

	

	Static (or strategic) noise maps used by passive measures depict noise levels of desired areas over extended periods (a year) <cit.>.
	These levels are calculated based on the methodological frameworks such as CNOSSOS-EU, CRTN, and sonROAD18 created by various government authorities <cit.>.
	Since static noise maps present noise levels over extended periods, their outputs represent only perpetual noise sources such as road traffic and industrial facilities and not the intermittent sources with random emergence at arbitrary locations such as construction, loud music or car alarm.
	However, these temporary non-traffic sources constitute a significant portion of civil noise complaints, rendering a more capable alternative to static noise maps essential for effective noise enforcement <cit.>.
	Moreover, relying on noise complaints to mitigate noise violations is a slow process.
	For these reasons, internet of things (IoT) based dynamic noise maps incorporating higher frequency noise level changes are sought for elevated noise enforcement capabilities.

	Dynamic noise maps can be considered as collections of noise maps created at consecutive time steps with short intervals.
	At each time step, either a completely new map or a partial update to the previous map can be created based on the environmental parameters collected from a sensor network.
	No matter what method is used for map generation, existing dynamic mapping research focus on decreasing temporal error of traffic noise and omit the spatiotemporal error caused by non-traffic sources.
	As a result, the noise from non-traffic sources is either filtered out or incorrectly attributed to roads, damaging effectiveness of these maps for noise enforcement.
	
	

	To have a dynamic noise map with a short time step for a low-cost city-wide noise monitoring system, low-power wide-area networks (LPWANs) can be used.
	Although LPWANs provide lower deployment and maintenance costs by eliminating tethered infrastructure via battery-powered wireless nodes, they limit communication data rates significantly.
	For example, LoRaWAN, the most frequently used LPWAN protocol, enables extremely long single-link communication ranges using its LoRa modulation technique in return for extremely low data rates <cit.>.
	Combined with hundreds of nodes having to share the same channel using ALOHA-like medium access, the rate and size of messages each node can transmit is significantly constrained to prevent collisions.
	Using only limited data available, estimation of non-traffic source locations on the dynamic map also becomes challenging.
	
	To fill the gap for dynamic maps with non-traffic sources using LPWANs, this study proposes a dynamic mapping system with machine learning (ML) based acoustic localization to overcome the data rate limitation.
	The localization model uses only periodically transmitted 2-byte sound pressure level (SPL) readings and combines them with its prior knowledge of the acoustic behavior of the environment to predict the non-traffic source location.
	The final dynamic map is created as a joint visualization of traffic and non-traffic sources.
	The main contributions of this article can be listed as follows:
	
	
  * A scalable based dynamic noise mapping approach with non-traffic sources is proposed.
	
  * An ML based acoustic localization approach using simple SPL indicators to overcome the data rate limitations of LoRaWAN is provided.
	
  * A acoustic node hardware design targeting low cost noise mapping applications is presented.
	
	
	The remainder of this article is organized as follows.
	Related work is summarized in Section <ref>.
	The details of our proposed system are provided in Section <ref>.
	Performance evaluations based on field-tests are presented in Section <ref>.
	Finally, the paper is concluded in Section <ref>.



§ RELATED WORK


	Existing noise mapping literature can be grouped into simulation and measurement based approaches <cit.>.
	While simulation based approaches use computational methods to estimate noise levels, measurement based approaches use IoT technologies to collect measurements from acoustic sensors to generate a seamless noise level distribution over a wide region.
	Static noise maps are the most prominent examples of simulation based maps and their update interval of several years provide sufficient flexibility to execute time consuming acoustic simulations unsuitable for dynamic maps with frequent updates.
	Using the computational approach, the need to collect scattered SPL measurements are eliminated.
	Studies adopting this approach usually focus on improving the accuracy and computational efficiency of simulations <cit.>.
	On the other hand, dynamic maps are enabled with measurement based approaches using the spatiotemporal information provided from distributed acoustic sensors.

	A unique IoT-based approach for noise data collection is crowdsensing where microphones and network capabilities of smartphones are used to measure and transmit SPL measurements for map generation <cit.>.
	Although crowdsensing approaches eliminate the usage of a dedicated sensor network, calibration of microphones from different manufacturers and varying availability of up-to-date data may pose significant challenges.
	For example, <cit.> reports almost half of the users removing their application after using it only once.
	Moreover, privacy becomes a significant concern when users' accurate location information is attached to noise measurements <cit.>.
	
	On the other hand, dedicated acoustic sensor networks with cloud connectivity are used as a more robust IoT backbone for noise measurements, eliminating the shortcomings of crowdsensing in return for increased costs.
	However, existing noise mapping studies focus mostly on demonstrating their mapping techniques using pre-recorded measurements without considering the networking aspect <cit.>.
	For example, <cit.> demonstrates the integration of SPL readings into static maps using measurements logged at eight locations without any loss of information.
	Similarly, Aumond use SPL values recorded using a mobile measurement device that attaches position information to its logs <cit.>.
	Analyzing network conformity is another important aspect for effective evaluation of IoT-based mapping systems to demonstrate their feasibility in real world.
	While many noise monitoring studies prefer spatially sparse SPL values, they do not generate a map with fine-grained representation of noise levels.

	Several noise monitoring studies adopt commercial off-the-shelf IoT-compatible platforms such as Raspberry Pi and BeagleBone as their edge platform <cit.>.
	Although these single-board computers provide elevated capabilities with low cost, their power consumption figures prevent long-term battery-powered operation as reported in <cit.> with a daily energy consumption of 43 Wh.
	Hence, <cit.> uses power over ethernet technology to jointly meet network and power requirements of the nodes.
	However, tethered infrastructures are usually not preferred due to their higher long-term costs.
	
	For truly untethered IoT-based noise mapping/monitoring applications, low-power microcontroller based architectures with wireless communications are preferred.
	For this goal, various communication technologies such as WiFi, Zigbee, and can be considered <cit.>.
	While WiFi provides the highest data rates, its power consumption is also the highest, preventing long-term battery-powered operation.
	Zigbee on the other hand provides mediocre performance in both data rate and range, while or another low-power wide-area network (LPWAN) protocol combines the longest range and low power consumption in return for low data rates <cit.>.
	For example, <cit.> reports triple the lifetime for their node with compared to a similar node using WiFi.
	Thus, usually becomes the first choice for non-data-hungry applications.
	Moreover, <cit.> shows that large datasets can be sent with LPWANs if divided into smaller pieces and sent over longer periods.
	In a different based study, LoRa modulation is combined with their unique MAC protocol to enable multi-hop communication unlike the star topology of when sending small-sized SPL and class label <cit.>.
	Similarly, <cit.> uses LoRaWAN to collect SPL measurements to redirect road traffic to decrease noise at noise-sensitive locations such as hospitals.

	No matter which technology is used to transmit noise data, the assumptions on the collected data have a strong impact on the accuracy of generated maps.
	Hence, <cit.> uses a classifier to detect non-traffic noise events and discards audio signals containing these events in SPL calculation to leave traffic noise only for their intended map.
	Although they create event labels in situ, the manual process to remove non-traffic sources prior to SPL calculation prevents it from being realized as a real-time application.
	Conversely, other existing dynamic noise mapping approaches mostly assume contribution from non-traffic sources to noise levels to be negligible, limiting the variety of noise violation types against which dynamic noise maps can be used.

	For a more effective noise enforcement, non-traffic noise sources should also be depicted on the map.
	Such a task is not trivial since the locations of non-traffic sources are arbitrary.
	To overcome this challenge, acoustic localization algorithms can be used to estimate the position of non-traffic sources <cit.>.
	These algorithms estimate source locations using indicators such as steered response power (SRP), direction of arrival (DoA), energy level, or time difference of arrival (TDoA).
	Due to data rate and energy budget limitations, these algorithms may not be realistic for microcontroller and LPWAN-based IoT systems.

	Among the existing acoustic localization methods, energy based methods offer the lowest computational complexity at the edge nodes while requiring small amount of information transmission (energy level only).
	Least squares and maximum likelihood are among the commonly used methods for energy based acoustic source localization <cit.>.
	However, studies adopting these approaches usually assume free-field propagation <cit.>.
	In <cit.> and <cit.>, the localization methods use the energy level ratios of sensor pairs in a non-reverberating environment.
	Similarly, the authors in <cit.> use a feed-forward neural network (FNN) for acoustic localization using energy measurements in free-field environments.
	However, it is shown that omittance of acoustic reflections is known to cause poor performance in urban environments <cit.>.

	Another IoT-compatible localization indicator is TDoA where cross-correlations of audio signals are used to estimate the time difference of these signals.
	Using TDoA information from multiple sensor nodes, source localization in urban environments becomes possible <cit.>.
	Despite slightly higher edge computation requirements, TDoA based approach is an important alternative to energy based methods for urban acoustic localization especially when combined with a phase transform (GCC-PHAT) to prioritize phase over magnitude during correlation calculation.
	Instead of using cross-correlation to extract time delay, SRP based localization approaches calculate signal power at candidate source positions and returns the position with the highest SRP as the estimated location <cit.>.
	Since this approach requires cross-correlations to be calculated for all signal pairs, it requires audio signals from all the nodes to be collected at a central location.
	In <cit.> the authors use 2-second audio with 10 kHz sampling rate, thus each node transmits at least 80 KB of data (using 4-byte floating points) for each localization task which would take hours to be transmitted in LPWAN-based systems.

	In <cit.>, Faraji use DoA for localization, but their DoA calculation method require evaluation of beamformed signal energy in each discrete direction for each pair of 8 microphones whose computational load prevents battery-powered operation beyond 26 hours.
	As an alternative to signal processing based DoA estimation methods, end-to-end ML models eliminate feature extraction step and enable low-computation DoA prediction.
	However, the best offerings of the current literature focus on platforms with significantly higher power consumption than battery-powered nodes used in low-cost IoT-based noise monitoring systems <cit.>.
	
	

§ SYSTEM OVERVIEW

	

	In providing a low-cost LPWAN based dynamic noise mapping system, limited data rate of LPWANs poses a big challenge and the system must be designed to overcome this barrier.
	Our proposed system uses ML models to predict the specifications of the non-traffic noise sources on the dynamic map using sparsely transmitted small-sized SPL information from the end nodes.
	Since the acoustic behavior of outdoor environments may vary drastically due to the different sizes and shapes of  physical objects such as buildings, which makes accurate location prediction difficult.
	Therefore, our proposed dynamic noise mapping system divides the application region into 250 m × 250 m areas and uses area-specific ML models to predict non-traffic noise events as well as their locations and SPLs.
	This way, ML models can learn the acoustic behavior of the specific environment during training, and non-traffic noise sources can be more accurately localized given small amount of SPL data, increasing LPWAN compatibility.
	Such bottom-up approach enables elevated scalability for cities at different sizes.

	In the proposed approach, each area is monitored by nine sensor nodes and the data transmitted by the nodes are collected by multiple gateways.
	Each gateway covers a circular area with a radius of the gateway's communication distance d_gw.
	The frequently used terms of region, cell, and area are visualized in Figure <ref>.	
	
	
	
	For LoRaWAN, although battery-powered nodes eliminate power infrastructure requirement, they may significantly increase maintenance costs if frequent battery replacements are needed.
	To keep power consumption low, the sensor nodes in our proposed system calculate only fast time weighted (LAF) and 15-minute long-term equivalent (LAeq,15min) A-weighted SPL values.
	These measurements constitute only 2 bytes in total and are transmitted with only 5 to 15-minute intervals (Δ t).
	As a result, the transmission of these data consumes only between 0.36 to 0.84 bps of physical bitrate for each node when used with Δ t = 15 minutes and DR0 or Δ t = 5 minutes and DR3, respectively.
	
	To extract useful information from the scarce sensor data, our system uses two ML models.
	While the first model predicts non-traffic noise events (p̂), the other predicts the properties (location and SPL, x̂, ŷ, and l̂) if the first model returns positive.
	This way, computation requirements of the server are decreased by running the secondary model only when a non-traffic noise event exists.
	The predicted non-traffic source is integrated into the map after being simulated as a point source enabling the distinctive output of our system.
	An overview of our proposed approach is shown in Figure <ref>.
	
	

	When used against noise violations, the time until the dynamic map includes the new noise sources is an important metric.
	In addition to transmission interval, the lack of time synchronization among sensor nodes may contribute to this delay if packet arrivals from all the nodes are waited.
	Moreover, packet losses may cause SPL information temporarily unavailable.
	To overcome these issues, dynamic map update tasks are executed with a 1-minute interval using the latest measurements available to stay functional despite possibly lower output accuracy.
	This approach not only increases system robustness but also eliminates the need for acknowledgements, helping decrease network congestion by reducing the use of valuable downlink windows of LoRaWAN.
	
	It is noted that as a result, each area can have at most one non-traffic noise source at a time.
	Considering the size of an area, the probability of this assumption being violated should be very low and can be omitted.
	Alternatively, the size of the areas can be shrunk until this assumption is fully satisfied.

	We make the sizes of our ML models small so that running the models with a 1-minute interval for small regions can be carried out by regular computers.
	If city-wide coverage with hundreds of areas and 1-minute update interval is desired, a powerful server can be used.
	It is noted that the power required for the proposed system is much less than those mapping systems with complete simulations.

	

 §.§ Sensor Node

	To process acoustic signals and provide extended battery lifetime, we need to develop our own acoustic sensor.
	These sensor nodes contain dual microcontrollers, stereo microphones, battery and LoRa transceiver as shown in Figure <ref>.
	All the tasks listed in Figure <ref> are performed on the primary microcontroller while the more powerful secondary microcontroller and microphone are reserved for more complicated edge artificial intelligence (AI) functions for our future studies.

	
	
	
	
	The MEMS microphones located under windscreens at the tip of the extended arms have an overload limit of 120 dB and provide digital audio signal using protocol.
	Thanks to this protocol, nodes can easily switch between mono and stereo audio.
	Also, calibration is not mandatory thanks to the ± 1 dB of sensitivity tolerance <cit.>.
	The nodes first sample audio signal s_t at time t with a sampling rate of 31.25 kHz using direct memory access (DMA) for lower power consumption.
	s_t contains 1024 samples which takes 4096 bytes in the memory as floating-point values.
	However, the default frequency response of the microphone does not fully lie within the specified tolerances of ANSI specifications for the targeted Type 2 sound level meters and requires an equalizer <cit.>.
	Hence, an equalizing filter with frequency-based gain is applied to move the response at non-conforming frequencies into the specified limits.
	In addition, an A-weighting filter is consecutively applied to better mimic the human perception of audio levels as required by most noise enforcement bylaws.
	These two filters are implemented in second-order recursive structure with minimal memory footprint.
	Following the filtering, LAF and LAeq are calculated using:
	
	
    m^lf_t = 20log_10(
    		s_RMS,t/
    		10^(C_s/20))
    		+ C_sr + C_c

	
	
    m^leq_t = 10log_10(1/N_h∑_i=t-N_h + 1^t 10^(m^lf_i / 10))

	
	where C_s, C_sr, and C_c represent microphone sensitivity, reference sensitivity, and calibration constants, respectively.
	In (<ref>), T represents the preferred time duration for the long-term equivalence and N_h represents the length of m^lf storing historical LAF values measured within this duration.
	The calculated SPL measurements are within ±1 dB tolerance as tested in LAeq,1h.
	Our system implementation adopts 15-minute LAeq duration as a good trade-off between response latency and accuracy <cit.>.
	After the calculation of SPL indicators audio data is deleted, the indicators are transmitted as 1-byte integers if transmission interval is reached.
	Otherwise, the node goes to sleep for 15 seconds as a good tradeoff between measurement accuracy and power consumption <cit.>.
	The primary microcontroller can provide power consumption as low as 300 nA in low-voltage retention sleep contributing to daily energy consumption of as low as 50 mWh.
	For detailed consumption analysis readers may refer to <cit.>.
	
	Using the LoRa transceiver with 125 kHz bandwidth, our nodes can theoretically reach uplink data rates of 980 (with SF=10) to 5470 (with SF=7) bps on an unoccupied channel.
	Also, our nodes do not utilize automatic data rate (ADR) functionality to preserve valuable downlink channels.

	

 §.§ Machine Learning Models


	

  §.§.§ Binary classifier model for non-traffic event detection

	For event detection p̂, a binary classification model using a single long short-term memory (LSTM) layer with 100 units is used to utilize LSTMs' high capability in learning temporal changes.
	The only input feature is the change in the latest discrete LAeq measurement at each node (Δ m^leq_n).
	These 9 differences are fed into the network as if they are part of a single feature over discrete time steps instead of individual features to eliminate the need to store historical values and simplify the model.
	The model is trained using an additional dropout layer with 0.5 rate and 1e-4 learning rate.
	
	To save computational resources, the secondary model is executed only when this model predicts a non-traffic noise event.
	Hence, the correct representation of non-traffic source is only possible with high true positive rate of this model.
	
	

  §.§.§ Regression model for source property prediction

	To predict properties (easting (x̂), northing (ŷ), and level (l̂)) of the non-traffic source, a regression model using a dual-branch FNN is used.
	Different from the first ML model, the second ML model uses both LAF and LAeq foreground measurements fused via Kalman filter (KF) to combine the low latency of LAF and low variance of LAeq measurements.
	Here, the term foreground represents the values where background values are subtracted from the measured values (m^lf,fg_n = m^lf_n - m^lf,bg_n), with background levels representing the minimum levels observed at each sensor location.

	Since we are using cross-validation with leave-one-out method to evaluate the performance of the proposed system, a secondary set of features is needed to inform the model on the leave-one-out status.
	This way, the ambiguity on whether the measurement being 0 due to the long distance to the source or leave-one-out is eliminated.
	This extra set of features is called mask and is basically an indicator for the presence of the measurement.
	Mask value of zero means the respective measurement is reserved for cross-validation:
	
	
    k_n = 
    		0,    if node n is used for leave-one-out
    
    		1,    otherwise.

	Using two different groups of features, we adopt the dual-branch structure so that the network can learn distinct representations for each group before concatenating them.
	For this goal, each feature is connected to a layer of 320 neurons before concatenation.
	The resulting layer is connected to a hidden layer with 100 neurons before the final output layer.
	Between consecutive layers, a dropout rate of 0.2 is used to prevent overfitting.
	The model is trained using a learning rate of 1e-3.
		
	

 §.§ Noise Map Generation

	Our system utilizes individual noise maps for traffic and non-traffic sources before combining them for dynamic map iterations.
	Both traffic and non-traffic maps are generated using NoiseModelling simulation software <cit.>.
	While a traffic map uses multiple point sources representing the vehicles on roads, a non-traffic map uses single point source at the predicted location.
	Resulting vector maps are converted to 50 px × 50 px raster images using the nearest neighbor grid in QGIS software before they are combined <cit.>.
	Since our system divides an entire region into multiple areas, map generation is scripted to adapt different areas based on the coordinates given as parameters.
	
	To embed area-specific acoustic behavior into ML models, each model is trained using noise maps generated for their target area.
	Since collecting high amount of real-world noise violation data is only theoretically possible due to strict legislation around such practice, simulated synthetic data is used in our training.
	Although this approach requires additional computation, it is a one-time preliminary process, and it decreases edge computing complexity enabling localization using only the SPL measurements.

	Since the LSTM model predicts non-traffic noise events, it is trained using a scenario where 500 non-traffic noise sources randomly emerge and disappear within the target area at different times.
	The output maps without reaching a mean value of 35 dB are discarded since they do not possess useful information for training.
	After the resulting map is combined with the base traffic map, Δ m^leq_n values at node locations are extracted with a 1-minute interval.
	
	The second ML model is trained using single-shot noise levels discarding the time domain.
	Random sources in 10,000 trials are used to simulate SPL at sensor nodes.
	The resulting maps are further augmented by adding Gaussian noise (Z ∼𝒩(0, 1)).
	
	Finally, the directivity of the simulated sources is randomized to prevent possible bias.
	The contribution of each frequency bin is randomized while dB magnitude of the directivity (v) for each frequency bin at a specific direction (θ) is calculated using:
	
	
    v(θ,i_f) = 40 (cos(θ) - 1/2 i_f + 1)

	
	where i_f ∈ {1,2,…,7} represents frequency bin index for frequencies of {8000,4000,2000,1000,500,250,125} Hz, respectively.
	Generated random directivities are fed into NoiseModelling software to be used in simulations.
	Training, validation, and testing splits use 80%, 20%, and 20% of all the simulated map data, respectively.
	
	

 §.§ LoRaWAN Communication

	
	To evaluate system performance, the efficiency of the communication protocol represented as packet success rate (PSR) must be known.
	This study considers two factors affecting PSR: packet collisions due to time overlap and packet losses due to low link quality.
	
	To analyze collision based packet success rate r_s,c, packet airtime is needed to analyze time overlaps during channel utilization.
	Packet size in symbols and LoRa symbol rate can be used.
	While LoRa modulation uses a fixed preamble size of 8 symbols, the packet payload size (n_pl) varies based on the physical layer payload (PL), spreading factor (SF), redundancy check (CRC), data rate optimization (DE), and code rate (CR):
	
	
    n_pl = 8 + max(0, ⌈2 PL - SF + 7 + 4 CRC/SF - 2 DE⌉(CR + 4) )


	Assuming that a node uses fixed parameters for packet transmissions, packet airtime (a_pkt) is calculated based on preamble and payload sizes (n_pa=8 symbols and n_pl) as well as symbol rate (R_s = BW / 2 ^SF):
	
	
    a_pkt = n_pkt,sym / R_s
    		= (4.25 + n_pa + n_pl) / R_s

	
	Having packet airtime values, packets from two independent nodes sharing the same channel are assumed to collide if the midpoints of the packets' airtime (t_x and t_y) are not as apart as half of their total airtime:
	
	
    c(x, y) = 
    		1,    if  |t_x - t_y | < (a_pkt,x + a_pkt,y)/2
    
    		0,    otherwise.

	
	Consequently, the collision based packet success rate in each cell is calculated as the mean collision rate over all the link pairs:
	
	
    r_s,c = 1 - 1/N_cell×(N_cell-1)∑_x ∈ N_cell∑_y ∈ N_cell
     y≠ x c(x, y)

	

	The PSR based on link quality r_s,q is calculated considering the unrecoverable received packets due to insufficient SNR or signal strength below the receiver (gateway) sensitivity.
	For this analysis, path loss of link n at distance d_n is calculated based on the mean loss at the reference distance d_0, path loss exponent (η), and shadowing modeled as X_σ∼𝒩(0, σ^2):
	
	
    L_p,n(d_n) = L_p(d_0) + 10 ηlog_10(d_n / d_0) + X_σ


	Using the transmit power (P_t), transmitter and receiver antenna gains (G_t and G_r), link path loss (L_p), bandwidth (BW), and noise figure of the receiver hardware (NF), received signal power (P_r) and its SNR can be calculated using <cit.>:
	
	
    P_r = P_t + G_t + G_r - L_p

	
	
    SNR = P_r + 174 - 10 log_10(BW) - NF


	The energy per bit to noise power spectral density ratio can be calculated using SNR, spreading factor (SF) and CR using:
	
	
    E_b/N_0 = SNR - 10 log_10(SF × 4 / (4 + CR)/2^SF)

	
	Furthermore, link bit error rate is calculated as in <cit.>:
	
	
    r_e,bit = Q(log_12(SF)/√(2)×E_b/N_0)

	
	It should be noted that bit error rate is set to 1 for cases where SNR falls below LoRa's demodulation limits or received signal strength is below the modem sensitivity.
	
	To move from bit error rate to packet error rate, packet size in bits must be known.
	This value can be calculated using channel bitrate and packet airtime:
	
	
    R_b = (SF × BW) / 2^SF

	
	
    n_pkt,bit = ⌈ a_pkt× R_b⌉

	

	The conversion from bit error rate to packet success rate for individual link quality is performed with:
	
	
    r_s,q,n = (1 - r_e,bit)^ n_pkt,bit

	
	Therefore, the overall link quality based packet success rate is calculated as the average over all the independent links:
	
	
    r_s,q = 1/N_cell∑_n ∈ N_cell r_s,q,n


	Finally, the PSR of a cell (r_s,cell) with N_cell many nodes is calculated as a combination of success rates with time overlap collisions and imperfect link qualities:
	
	
    r_s,cell = r_s,c× r_s,q

	
	Although we have empirical individual link quality data from the field tests, we are using the above analytical model to define the channel behavior beyond the distances observed in our tests.
	The empirical data is used to verify this model.
	
	As a result, the quality of each link mainly depends on the distance and the SF used.
	This study uses pre-defined assignments (SF ∈{7,8,9,10}) instead of adaptive updates since downlinks diminish the number of possible uplinks significantly.
	For this goal, SF values are assigned based on the link SNR without shadow fading and minimum SNR requirements of LoRa modulation <cit.>.
	Since varying link quality with shadow fading may cause disconnection, each link uses the next SF (SF + 1) whenever possible.
	Although increasing SF by 1 causes longer airtime, it improves link quality beyond the losses caused by time overlap collisions.
	The other parameters used in the channel are as follows: BW=125 kHz, CR=1, DE=0, P_t=20 dBm, G_t=2.5 dBi, G_r=6 dBi, and NF=3 dB.
	For the path loss model, we adopt the parameters from <cit.> (η = 2.08, d_0=40 m, L_p(d_0)=127.41 dB, and σ=3.57) since they enable the best fitment to our empirical data.
		
	

§ EXPERIMENTAL RESULTS

	
	
	The performance of our proposed system in reducing the error caused by non-traffic source is evaluated in field tests using a speaker shown in Figure <ref> positioned at 6 random locations within the selected application area.
	In each test, the same music tracks were played for 15 minutes with the same order with the sound level of around 100 dBA.
	The reason for such duration is to focus on the entire transitional period of the selected SPL indicator (LAeq,15m).
	Each test was followed by a 15-minute break so that the LAeq measurements can settle back to background levels and allow the equipment to be moved to the next location.

	

	The nodes in the area were deployed as shown in Figure <ref> at 2 m height and at randomly selected existing pole locations that satisfy deployment constraints.
	These constraints force the nodes to be deployed with at least 50 meters distance from each other but within 25 meters of a building facade facing away from the building.
	Although these constraints were not mandatory for this study, they were implemented for a fair performance comparison with our prospective research.
	
	

	For the field test performance evaluation, the proposed dynamic mapping method was performed in consecutive fashion, where the data for all the field tests were collected and then the mapping was performed on the collected data.
	Although our system can be run in real time, this approach is preferred to augment more field tests with varying transmission intervals and packet success rates to limit the disturbance caused in the tests.
	During field tests, the data was transmitted with a 1-minute interval and then processed for varying transmission intervals and packet losses in 25 trials each.
	In each trial, each node was assigned with a random time offset to better mimic the lack of time synchronization.
	Also, each trial was augmented further by zeroing the measurements from one sensor at a time and setting its mask for cross validation.
	The zeroed values were used as reference values in error calculations.
	Finally, packet loss rates for the respective Δ_t and d_gw values were applied.
	As a result, the data collected from 6 field tests was expanded into 16,200 tests.
	
	Table <ref> shows PSRs calculated using (<ref>), (<ref>), and (<ref>) and the average number of packets reaching the gateway per area per hour.
	Since time-overlap collision probability increases with shorter transmission intervals, r_s,c decreases with shorter .
	Moreover, longer transmission distances require higher SF, hence longer packet airtime, causing more collisions.
	Thus, the values decrease as the gateway distance increases.
	On the other hand, link quality is independent from transmission interval.
	Due to higher path losses with longer transmission ranges, decreases for higher gateway distances despite using higher SF as explained in Section <ref>.
	The best overall cell packet success rate is achieved at 500 m of gateway distance and 15 minutes of transmission interval.
	However, lower transmission intervals enables the highest number of packets reach gateway despite their lower values, making them more useful for system since they provide more information to the ML models.
	It should be noted that the listed successful packets vary slightly from expected values due to random node deployment and shadowing.

	
	
	While the performance of our LSTM classification model is evaluated using prediction accuracy, the regression (FNN) model is evaluated based on the mean absolute error (MAE) of its predicted locations.
	In case of false negatives from the first model, our system misses its chance to correct the map error.
	On the other hand, our system may worsen the map error trying to attribute small variances to a noise event by predicting the location of a non-existent source.
	In such cases, the regression model usually predicts only a low source SPL (l̂) due to small changes in model inputs, hence the increase in the map error is minimal.
	Nevertheless, prevention of false positives is rendered more important than those of false negatives.
	The performance results of 0.75 precision and 0.57 recall reflect this ambition (higher precision than recall).
	The individual performance results of the ML models are listed in Table <ref> using the metrics of event detection (p̂) accuracy, MAE of individual and joint predicted location (e_x̂, e_ŷ, and e_loc = √(e_x̂^2 + e_ŷ^2)), as well as MAE of predicted source level (e_l̂).
	The trends in these metrics are observed to be highly correlated with the number of successful packets presented in Table <ref>.
	More packets reaching the gateway allow more up-to-date information being available for ML model predictions resulting in better location estimation.
	For example, the classifier provides 0.73 event detection accuracy when 105 packets are successfully transmitted, and this value goes down to 0.55 with 35 packets.
	It should be noted that 0.55 accuracy is close to random guess, but such a low accuracy is still useful in decreasing map error thanks to higher precision than recall as well as false positives' destructive impact being negligible due to low source SPL prediction.
	Similarly, location estimation values get better with more packets available.
	The exception of e_l̂ sometimes getting better with less packets is a result of synthetic data having lower SPL values compared to the real-world data.
	This is also the reason why e_l̂ is always around 7 dB.
	More successful packets cause the prediction to be better according to the synthetic training data, but this implies a larger discrepancy compared to field-test values.
	Compared with the best acoustic localization algorithms in the literature, 54 to 78 m of location estimation error in a 250 m × 250 m area may seem large, however the size of the data used to provide these results is extremely small and the data contains information delay up to 15 minutes.
	For the noise mapping application considered here, these localization results are effective in reducing the map error caused by non-traffic sources which is the ultimate goal here.

	
	
	Figure <ref> shows the map error decrease provided by the system as a result of joint utilization of ML models with red background for active test periods in the plot and samples of generated maps below the plot.
	For all the performance evaluations, the first test was clipped to provide equal total duration for different transmission intervals, leaving 5 tests in total.
	Also, cross validation was performed by using the measurements from 1 out of 9 sensors as reference for error calculations and replacing these measurements with zeros for other system tasks in each of 9 cross validation trials.
	For each test, prior error was considered as the maximum minus minimum SPL level throughout the test in dB representing the impact of non-traffic source.
	In addition, post error represents the mean difference in dB between the measured value (1-minute interval data) and the output map value over all the cross validations and random time offset trials at each time step.
	It should be noted that errors can only be measured at node locations since reference measurements are not available outside the node locations.
	As a result, the distance of the source location to the sensor nodes impact how much of the error will be observed.
	In the plot, post error starts with a high value for each test follows a decreasing trend throughout the test thanks to the increasing localization accuracy with more packets arriving and ends with an occasional peak.
	The peaks at the start of each test were caused by the delay until multiple nodes send their measurements with higher SPL values after a non-traffic source emerged.
	During this short transitional period, event detection model predicted false negatives, hence no correction was applied to the map.
	Similarly, the peaks in the error after the source disappears were caused by the delay until nodes' most recently received measurements contained lower values.
	These peaks in prior error last longer when bigger transmission intervals are used.
	These two problematic cases can be observed in sample maps provided at 4 discrete times in Figure <ref>.
	In the first map only road noise is shown, whereas in reality the test with a non-traffic source started at 11:02.
	Similarly, at 11:17, the source had already stopped making noise, but the presence of the non-traffic source still persisted in the map.
	Also, the occasional peaks between the tests are either caused by false positives in the event detection model or non-controlled noise sources such as birds with loud calls observed during the tests.
	Since our acoustic sensors are designed not to store audio recordings for privacy reasons we cannot perform post-analysis to differentiate between these sources and prefer to provide the cumulative error.
	
	
	
	The mean error correction results throughout all the trials are shown in Figure <ref>.
	The proposed system performs the best with the shortest transmission interval and gateway range.
	Its relative performance drops to 29% at the longest transmission interval and range.
	Although shorter transmission intervals cause lower PSR, the total number of packets successfully transmitted is still higher and improved system performance is observed as shown in Table <ref>.
	Similarly, a shorter gateway range gives better system performance.
	It should be noted that all the localization and error reduction results are reached using measurements from 8 sensors as a byproduct of cross-validation.

	The proposed system assumes full utilization of 8 out of 72 LoRaWAN uplink channels of US915 specification for city-wide coverage.
	This requirement can be relaxed without compromising the system performance using denser gateways leading to less channel occupation thanks to shorter packet airtimes.
	Although the system already meets the 1% maximum duty cycle limit asserted in some regions, denser gateways can also be used to adapt the system to the regions with less available channels and/or stricter fair usage limits.

	

	

§ CONCLUSION

	
	In this paper, we propose a LoRaWAN based dynamic map for urban noise monitoring.
	We develop a novel LoRaWAN acoustic sensor with edge AI capability.
	Realization of dynamic noise mapping with non-traffic sources is shown to be possible for when ML models are utilized to extract useful information from sparsely transmitted data.
	Using experimental data, the proposed system is found to stay operational under high packet collisions when sparse gateway deployment and frequent transmissions are preferred.
	The impact of channel model on this LoRaWAN IoT noise monitoring system is also analyzed.
	
	Our future work is planned to provide improvements to the localization and map error reduction performance using edge ML models while preserving the compatibility with LPWANs.
	
	IEEEtran
		

	-2plus -1fil
	


 Paper Title: Lifelong Person Search.








Lifelong Person Search
    Jae-Won Yang, Seungbin Hong, and Jae-Young Sim, Member, IEEE

    August 5, 2024
=================================================================



§ ABSTRACT 
  Person search is the task to localize a query person in gallery datasets of scene images. Existing methods have been mainly developed to handle a single target dataset only, however diverse datasets are continuously given in practical applications of person search. In such cases, they suffer from the catastrophic knowledge forgetting in the old datasets when trained on new datasets. In this paper, we first introduce a novel problem of lifelong person search (LPS) where the model is incrementally trained on the new datasets while preserving the knowledge learned in the old datasets. We propose an end-to-end LPS framework that facilitates the knowledge distillation to enforce the consistency learning between the old and new models by utilizing the prototype features of the foreground persons as well as the hard background proposals in the old domains. Moreover, we also devise the rehearsal-based instance matching to further improve the discrimination ability in the old domains by using the unlabeled person instances additionally. Experimental results demonstrate that the proposed method achieves significantly superior performance of both the detection and re-identification to preserve the knowledge learned in the old domains compared with the existing methods.



Person search, person re-identification, lifelong learning, and continual learning.





§ INTRODUCTION

Person search is the technique to find the query person from the gallery sets of scene images where multiple persons usually appear simultaneously in each image. It has been drawing much attention due to its practical applicability to various real-world scenarios such as large-scale video understanding, surveillance, and augmented reality.
Different from the person re-identification (re-ID) <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.> that finds the query person from the sets of cropped person images, the person search is a more challenging task that first localizes the bounding boxes of person instances in the scene images and then matches the identities of the detected instances to the query person. The person search can be implemented in the two-step manner by using separately trained two sub-networks of the object detection and re-ID. However, the training of the two-step methods is usually inefficient requiring huge computational complexity, since the detection network extracts the bounding boxes for the person instances from the scene images which are then inputted to the re-ID network to retrieve the features again tailored to the re-ID task. 


To overcome this issue, the end-to-end learning was introduced that jointly trains the person detection and re-ID networks. The end-to-end methods have been mainly developed in the supervised learning manner based on the assumption that both of the training and test data come from a same target dataset <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>. However, in many practical real-world applications, multiple datasets are generated in different places and times that exhibit domain gaps from one another. In such cases, the existing supervised methods trained on a certain dataset usually fail to work on other datasets. Furthermore, re-training the network, whenever the target datasets are changed, suffers from the high computational complexity as well as the catastrophic forgetting <cit.> of the knowledge learned from the previously trained datasets.



In this paper, we first introduce a new problem of lifelong person search (LPS) where the new datasets on different domains are assumed to be sequentially given in order, as shown in Fig. <ref>. The model is forced to be generalized to all domains while preserving the previously learned knowledge without entire re-training using all the datasets. 
The end-to-end LPS is more challenging compared to the lifelong object detection <cit.>, <cit.> and lifelong person re-ID <cit.>, <cit.>, <cit.>, <cit.>, since it suffers from the catastrophic forgetting problem in both sub-tasks of the person detection and re-ID.
Whereas the lifelong person detection is a domain-incremental task where only the same class of person is localized across different domains, the lifelong person re-ID is related to both domain-incremental and class-incremental tasks since the new person identities are additionally given from different domains.
Moreover, the end-to-end person search also suffers from the task conflict problem where the person detection focuses on extracting the common representation of persons distinct from the backgrounds, but the person re-ID attempts to extract the unique representations according to the person identities. This task conflict problem becomes more serious in LPS scenario where the model is encouraged to be continuously adapted to different domains. 
In addition, the lifelong re-ID methods <cit.>, <cit.>, <cit.>, <cit.> have been usually developed for full-body pedestrian images on similar domains. 
However, as shown in Fig. <ref>, the LPS considers the scene images with severely different characteristics across multiple domains, for example, diverse backgrounds, different scales and densities of persons, and even local body parts due to partial occlusion. 

To address the LPS problem, we propose a novel end-to-end framework that generalizes the network to be incrementally adapted to the new domains while preserving the previously learned knowledge in the old domains.
Specifically, we perform the knowledge distillation for both sub-tasks of the person detection and re-ID using the old exemplar data. We first use the prototypes, representative features of person identities, associated with the old exemplar data, to design the rehearsal-based re-ID knowledge distillation loss that enforces the consistency on the distributions of the feature similarity between the old and new models. 
Moreover, we also utilize the hard background proposals additionally to refine the re-ID knowledge distillation loss that alleviate the effect of inaccurately detected person proposals and extract discriminative features for re-ID more reliably.   
In addition, we devise the rehearsal-based instance matching loss to further improve the model's discrimination ability. We minimize the feature discrepancy between the labeled proposals in the old exemplar data and its ground truth old prototypes. We also employ the unlabeled person identities in the old domains as negative samples to preserve the knowledge effectively. Experimental results demonstrate that the proposed method preserves the knowledge of old datasets more faithfully compared with the existing methods, and therefore serves as a very promising tool for LPS.   

The main contributions of this paper are summarized as follows.

	
  * To the best of our knowledge, we first introduce a new and challenging problem of person search with lifelong learning scenario where the model is incrementally trained on the new domains while preserving the previously learned knowledge in the old domains. 
	
  * We propose an end-to-end LPS framework that jointly trains the detection and re-ID networks by using the rehearsal-based knowledge distillation loss and the instance matching loss, that alleviate the catastrophic forgetting of the old knowledge during the training on the new domains.
	
  * We demonstrate the efficiency of the proposed method by providing comprehensive experimental results compared with the existing methods based on the lifelong learning scenario.






§ RELATED WORKS



 §.§ Person Search

Person search has been studied mainly in the supervised manner where the bounding boxes of person instances and the person identities are labeled in the training datasets.
The two-step methods train the person detection and re-ID networks separately to prevent the conflict problem between the two tasks. Zheng et al. <cit.> conducted extensive experiments by training the state-of-the-art methods of the pedestrian detection and person re-ID. They also provided a benchmark PRW dataset. Pu et al. <cit.> proposed a segmentation masking scheme to force the re-ID network to focus on the foreground regions of the detected persons. Lan et al. <cit.> extracted multi-scale features to deal with the scale variation problem of person size in the scene images. Wang et al. <cit.> made the re-ID network more adapted to the detection results by composing the training set with the person images cropped by the pre-trained detection network and the person images cropped by using the bounding box labels. Ke et al. <cit.> performed a data augmentation scheme that shifts the locations of the ground truth bounding boxes for the re-ID network training.

The end-to-end methods jointly train the person detection and re-ID networks. Xiao et al. <cit.> firstly proposed an end-to-end person search network and provided a benchmark CUHK-SYSU dataset. Chen et al. <cit.> employed the background features as negative samples to train the re-ID network. Chen et al. <cit.> separated the feature embedding into the norm and angle which are used as a detection confidence score and an identity feature, respectively.
Zhang et al. <cit.> pretrained an external re-ID network which is then used as a strong teacher model to supervise the re-ID network based on the knowledge distillation framework. Li and Miao <cit.> employed an additional Faster R-CNN header sequentially to extract the superior identity features from the high-quality person proposals. Han et al. <cit.> adaptively controlled the gradient backpropagation to train the sub-networks of the re-ID and part classification according to the quality of detection results. Lee et al. <cit.> suggested a feature standardization scheme and a localization aware memory updating scheme to alleviate the effect of class imbalance and inaccurately detected proposals, respectively.
The transformer architectures were also employed to improved the performance of person search <cit.>, <cit.>, <cit.>, <cit.>. 
Recently, Oh et al. <cit.> assumed the training data of real target domains are not available, and proposed a domain generalizable person search method that uses only an unreal dataset for training.

On the other hand, the weakly-supervised person search has been introduced that uses the labeled bounding boxes only without using the identity labels for training <cit.>, <cit.>, <cit.>, <cit.>. Moreover, domain-adaptation methods have been proposed to address the unsupervised person search problem where both of the bounding box and identity labels are not available <cit.>. Note that the existing methods of person search have been usually developed considering a single target dataset only, and hence suffer from the catastrophic forgetting problem where new target datasets are continuously given in the lifelong learning scenario. 






 §.§ Lifelong Object Detection

Lifelong object detection methods are classified into the class-incremental approach and the domain-incremental approach. 
The class-incremental object detection considers a certain target dataset where the new object classes are incrementally added. Shmelkov et al. <cit.> first introduced the problem of catastrophic forgetting in the object detection. Adaptive distillation has been performed between the intermediate features and the output of the region proposal network based on the end-to-end framework <cit.>. Shieh et al. <cit.> stored a subset of old data into the exemplar memory to alleviate the catastrophic forgetting in old classes. Liu et al. <cit.> focused on the most informative old knowledge by sampling the most reliable foreground prediction from the old model, which are then used as pseudo labels in a transformer based detection network, DETR <cit.>. Dong et al. <cit.> performed self-supervised learning with the DETR network where only a few labeled new object classes appear in the new data.
On the other hand, object detection datasets are associated with different domains according to the variations of background, lighting, and camera viewpoint, even though they contain the same object classes. The domain-incremental object detection assumes incrementally added new domains with the same object class. Li et al. <cit.> used a transformer-based feature extractor to adaptively apply the classification head network to each newly added domain. Mirza et al. <cit.> stored the statistical changes across the domains used to perform the task on the corresponding specific domain. 



 §.§ Lifelong Person Re-ID

Wu and Gong <cit.> first introduced the lifelong person re-ID problem and performed coherence learning for classification, distribution, and representation, respectively. Pu et al. <cit.> adaptively accumulated the knowledge of old domains via the instance-based similarity to improve the generalization ability. Ge et al. <cit.> developed a domain adaptation framework that reduces the gap between the old and new domains by using the augmented new data following the distributions of the old domains. Sun and Mu <cit.> selected diverse and important patches from images by using a differentiable patch sampler to preserve both the local and global relational knowledge. Huang et al. <cit.> developed a relation consistency learning to encourage the new model to return the consistent results of similarity ranking to that of the old model. Yu et al. <cit.> proposed a knowledge transfer scheme via the bi-directional learning that dynamically updates the old model while training the new model. Xu et al. <cit.> used a pair-wise relation matrix to filter out the erroneous knowledge of the old model and transfer the refined knowledge to the new model.




§ PROPOSED METHOD


Fig. <ref> shows the overall architecture of the proposed end-to-end LPS framework. We use the SeqNet <cit.> as a baseline network which consists of the Faster R-CNN <cit.> and the NAE (norm-aware embedding) <cit.> header. 
Let us assume that a sequence of person search datasets in different domains are given in order, as D_1 → D_2 →…→ D_N. The model is trained by using the first dataset D_1.
When the new dataset D_2 is given, we regard the model trained on D_1 as the old model, and construct a new model by replicating the old model.
Then the new model is trained by using D_2 and a small subset of D_1, called exemplar data, to avoid the knowledge forgetting of D_1. We also use the representative features of person identities, called prototypes <cit.>, stored in the old look-up table (LUT) .
Whenever a new dataset D_N is available, the old model is replaced with the new model, and the new model is re-trained by using the new data D_N as well as both the old exemplar data and the old prototypes selected from {D_1, ⋯, D_N-1} to mitigate the catastrophic forgetting.
We use a small subset of the old data following the typical rehearsal (replay) based methodology of lifelong learning <cit.>. 
However, it is worth to note that we do not employ multiple old models but always have a single old model which is updated whenever a new dataset is given. The old model conveys the knowledge of the previous domains and thus the parameters of the old model are frozen during the training of the new model.
We preserve the knowledge of the old data while training the model using the new data via knowledge distillation between the old and new models. 






 §.§ Re-ID Knowledge Distillation



  §.§.§ Prototype-Based Distillation

Existing methods of lifelong person re-ID <cit.> perform the knowledge distillation by matching the distributions of the feature similarity between the old and new models within a mini-batch. However, it may not provide faithful results when applied to LPS, since a mini-batch is composed of relatively small numbers of scene images, and small numbers of identities accordingly, due to the memory constraints. Furthermore, multiple person instances with the same identity are rarely included in a single mini-batch according to the uniqueness prior <cit.>. 

To address this issue for LPS, we utilize the stored prototypes of person identities as informative guidance for the re-ID knowledge distillation.
The prototype is computed by aggregating the features of diverse person instances with the same identity <cit.>. 
Let  denotes the set of the foreground proposals in the exemplar data of a single mini-batch, that is detected by the Faster R-CNN of the new model. Let ^old_i and ^new_i be the L2-normalized features of the i-th proposal in , that are extracted through the NAE headers of the old and new models, respectively. We estimate the target distribution of the feature similarity of ^old_i compared to all the prototypes in , such that the probability q_i,k associated with ^old_i and _k, the k-th prototype in , is given by 

    q_i,k = exp(_k^T^old_i  / τ_d)/∑_∈exp(^T^old_i / τ_d).

We also estimate the predicted distribution of the feature similarity of ^new_i compared to all the prototypes, such that the probability p_i,k associated with ^new_i and _k is given by

    p_i,k = exp(_k^T^new_i  / τ_d)/∑_∈exp(^T^new_i / τ_d).


Note that the predicted similarity distribution with respect to the prototypes in the old domains changes when the model is trained on the new domain, which could cause the forgetting of the re-ID knowledge learned on the old domains. Therefore, we train the new model to generate a more consistent distribution to the target distribution by employing a prototype-based re-ID knowledge distillation loss given by

    ℒ_rkd = 1/||1/||∑_i ∈ I()∑_k ∈ I()  q_i,klogq_i,k/p_i,k,

where I(·) means the index set. 
By minimizing ℒ_rkd, the new model is trained to yield a more consistent distribution to the target distribution with respect to the prototypes in the old domains, and eventually extracts unique and representative features of all person identities alleviating the catastrophic forgetting in re-ID.  





  §.§.§ Hard Background Proposal-Based Distillation

Though the prototypes in the old LUT serve as a good prior for the re-ID knowledge distillation, we further improve the performance by using the background proposals in the old domains additionally. At each iteration, the new model detects the background proposals from the scene images in the exemplar data, as depicted in the red boxes in Fig. <ref>. Inaccurate background proposals are often generated that partially overlap with the foreground person instances. We refer them as the hard background proposals. The hard background proposals convey the partial information of the person identities, exploited to improve the discrimination performance of the person identities. 


Specifically, we sample the hard background proposals that have the higher intersection over union (IoU) scores than a certain threshold λ_ b, with respect to the ground truth bounding boxes of the foreground persons, as depicted in the blue boxes in Fig. <ref>. Then we store the re-ID features of the hard background proposals into the feature memory .
We re-compute the distributions of the feature similarity compared to all the prototypes in  as well as all the features of the hard background proposals in , such that the probabilities q_i,k^+ and p_i,k^+ associated with _k, the k-th element in ∪, are given by 

    q_i,k^+ = exp(_k^T^old_i / τ_d)/∑_∈{∪}exp(^T^old_i / τ_d),


    p_i,k^+ = exp(_k^T^new_i / τ_d)/∑_∈{∪}exp(^T^new_i / τ_d).












Accordingly, we have the refined re-ID knowledge distillation loss as

    ℒ_rkd^+ = 1/||1/|∪|∑_i ∈ I()∑_k ∈ I(∪)  q^+_i,klogq^+_i,k/p^+_i,k.

Consequently, we improve the discrimination performance of the person identities by exploiting more rich information carried by the hard background proposals. Furthermore, the features learned by additionally using the hard background proposals are more robust against the inaccurately detected person proposals, which alleviates the task conflict problem between the detection and re-ID even when the detection knowledge in the old domains is forgotten.



 §.§ Rehearsal-Based Instance Matching

The foreground and background proposals extracted from the exemplar data are used for consistent learning between the old and new models via the re-ID knowledge distillation. Note that, as depicted in the green boxes in Fig. <ref>, some foreground person instances have no identity labels. Such unlabeled instances can also serve as the negative samples for all the labeled identities to learn the discriminative feature representations. At the same time, the new model should be guided to minimize the feature discrepancy across the person instances in the exemplar data that have the same identity. Therefore, we also utilize the unlabeled instances in the exemplar data to further capture the re-ID knowledge in the old domains while the model is trained on the new data. 



The features of the unlabeled proposals are stored in the old circular queue . Let _ L denote the set of the labeled proposals in , and let ^new_i be the feature of the i-th proposal in _ L extracted by the new model. We compute the probability that ^new_i is classified into its ground truth label as

    ρ_i = exp((i)^T^new_i  /τ_r )/∑_∈exp(^T^new_i / τ_r)  +  ∑_∈exp(^T^new_i  / τ_r),

where (i) means the prototype of the ground truth identity of the i-th proposal in _ L, and  denotes the feature of the unlabeled proposals stored in . We train the new model to increase the classification score of the extracted features by employing the rehearsal-based instance matching loss given by

    ℒ_rim = -1/|_L|∑_i ∈ I(_L)logρ_i.

By minimizing ℒ_rim, we reduce the feature discrepancy between the labeled proposal and its ground truth identity while preserving the discrimination performance in the old domains with the help of the unlabeled proposals. 
It is worth to note that the conventional OIM <cit.> loss considers the labeled identities and the unlabeled instances in a single target domain only. On the contrary, the proposed rehearsal-based loss ℒ_rim employs the labeled identities and the unlabeled instances across the old data, aiming to preserve the discrimination performance in the old domains for lifelong learning purpose. 




 §.§ Training and Inference

At the training phase, both of the person detection and re-ID networks are trained in the end-to-end manner. Note that the baseline network of SeqNet <cit.> also uses the losses of ℒ_det and ℒ_oim when training the new model by using the new dataset. To preserve the knowledge of the old domains in terms of the person detection, we additionally use the detection knowledge distillation loss ℒ_dkd of the existing lifelong object detection method <cit.>.
Finally, the total loss function is given by 

    ℒ_total = ℒ_dkd + ℒ_rkd^+  + ℒ_rim + ℒ_det + ℒ_oim.


After the new model is trained with the last dataset D_N,  we discard the old model and only utilize the new model to detect the bounding boxes of the person instances and extract the re-ID features for all the datasets {D_1, D_2 … D_N} at the inference phase.




§ EXPERIMENTAL RESULTS




 §.§ Experimental Setup



  §.§.§ Datasets

We used the three datasets to evaluate the performance of the proposed lifelong person search method. CUHK-SYSU <cit.> and PRW <cit.> are widely used for the person search task. The CUHK-SYSU dataset includes the images obtained from the street snapshots and movies, with the annotations of the bounding boxes and person identities. We set the gallery size to 100. The PRW dataset is composed of the video frames capturing a university campus by six different cameras. We also use a recently released large-scale person search dataset of MovieNet-PS <cit.> gathered from the 385 movie sequences. The MovieNet-PS is a challenging dataset since it includes the scene images with diverse backgrounds, illuminations, and poses of persons to reflect more realistic and challenging scenarios of person search. Moreover, there are many persons partially appearing due to the occlusion, and the person instances with the same identity often wear different clothes. The statistics of the three datasets are shown in Table <ref>. 








  §.§.§ Evaluation Metrics

We evaluated the performance of the person detection and re-ID, respectively, based on the LPS framework. We used the recall and the average precision (AP) to measure the detection performance. The recall calculates the percentage of the true positive bounding boxes where the IoU scores with respect to any ground truth bounding box are higher than 0.5.
The AP computes the average precision of the bounding boxes by measuring the area under the Precision-Recall curve using the IoU scores with respect to the ground truth.
We also used the mean Average Precision (mAP) and the Top-k scores for the re-ID. The mAP calculates the averaged precision of searching a query from the gallery images. It measures the area under the Precision-Recall curve using the feature similarities between the query and all the detected gallery persons. The detected bounding boxes that overlap with the ground-truth with the IoU scores higher than 0.5 are set as the true positives. The Top-k score checks whether at least one of the k-most similar candidates is a true positive or not. We adopted the Top-1 score in this work.



  §.§.§ Implementation Details

We uniformly sampled 2% of the training data from the old datasets to construct the exemplar data, as did in many literatures of lifelong person re-ID <cit.>. 
We store the prototypes of the persons in the exemplar data only into the old LUT, which are not updated during the training on the new domain.
For the model training, we set the batch size for the exemplar data to 2 for each old domain and 5 for the new domain, respectively. We resized the input image to 1500 × 900 and applied the random horizontal flipping. We set the size of the old circular queue  as 1000.
We trained the model until it reaches the highest performance on the first domain, and trained the model for 5 epochs each on the other domains. Since our baseline setting achieves the best performance when the model is trained during 5 epochs for each new dataset, we also trained the model for 5 epochs on the new dataset for fair comparison to the baseline setting.
The initial learning rate is set to 0.003, which is warmed up with a learning rate scheduler and further decayed by the value of 0.1 in the 3rd epoch of each new domain. For the stochastic gradient descent, we set the momentum value to 0.9 and the weight decay to 0.0005. λ_ b, τ_d, and τ_r are empirically set to 0.1, 0.3 and 0.1, respectively. All our experiments were implemented using PyTorch and a single NVIDIA TITAN X GPU.






 §.§ Lifelong Learning Performance

We evaluate the lifelong learning performance of the proposed method with the training order of CUHK-SYSU → PRW → MovieNet-PS in Table <ref>. We first compare two different methods: Joint-Train and FineTune, implemented on our baseline network. The Joint-Train method trains the model by using all the available datasets simultaneously. The FineTune method trains the model on each dataset in order, where the model is initialized with the previously trained weights and then re-trained by using the new dataset. At the inference phase, the performance is evaluated on every dataset by using the model trained on the last dataset.

The Joint-Train method achieves the best performance since it uses all the training datasets at once, however, it requires a huge burden of the computation as well as the storage space. We observe that the proposed method provides comparable results to the Joint-Train method and outperforms the FineTune method in terms of the averaged mAP and Top-1 score, respectively.
Note that the FineTune method sequentially re-trains the model on each dataset without using the previous old datasets, and thus its performance on the last dataset, MovieNet-PS, is relatively high. On the contrary, the proposed method uses a small subset of the old data to alleviate the knowledge forgetting and slightly degrades the performance on the last dataset compared to the FineTune method. However, the proposed method significantly outperforms the FineTune method in the old datasets of CUHK-SYSU and PRW. 


Fig. <ref> shows the performance evaluated on the old datasets of CUHK-SYSU (blue) and PRW (red), when the model is sequentially trained on the new datasets in the x-axis, in order.
As shown by the dashed lines, both the AP and mAP scores are decreased in the FineTune method showing the knowledge forgetting effect of both the detection and re-ID tasks in the LPS scenario. However, the proposed method significantly mitigates such performance degradation as shown by the solid lines, and successfully preserves the old knowledge for LPS.

In addition, we show the performance of the proposed LPS method when the model is trained with different orders of the datasets. In Fig. <ref>, Order1 and Order2 represent the training orders of `MovieNet-PS → CUHK-SYSU  → PRW' and  `PRW → CUHK-SYSU → MovieNet-PS,' respectively. 
In both orders, we see that the proposed method still outperforms the FineTune methods in terms of the detection and re-ID performances, which indicates that the proposed method provides reliable performance regardless of the training orders.















 §.§ Comparison with Two-Step Methods

Note that we first introduce the new problem of LPS in this paper, and there is no existing method fairly comparable to the proposed method. We attempted to conduct the additional comparative experiments by using the recent lifelong re-ID methods of AKA <cit.>, PTKP <cit.> and KRKC <cit.>. The lifelong re-ID methods work on the cropped person images only and cannot be directly applied to our LPS framework that considers the scene images. We instead implemented the two-step person search framework by using the existing lifelong person re-ID methods, where the detection and re-ID networks are trained separately. We used the ResNet-50 as the backbone network for the compared methods. We also trained the detection network by using the detection knowledge distillation loss ℒ_dkd for fair comparison. 

Table <ref> compares the quantitative performance of lifelong learning where we see that the proposed method outperforms the two-step methods of `Det + AKA,' `Det + PTKP,' and `Det + KRKC' in terms of both the detection and re-ID performance. 
It means that whereas the two-step implementation of LPS by using the existing lifelong re-ID methods does not effectively reflect huge domain gaps among the person search datasets with severely different characteristics, the proposed end-to-end framework alleviates such domain gaps faithfully and is more generalizable to diverse person search datasets.
We also compared the upper-bound performance of the person re-ID by adopting the ground-truth (GT) bounding boxes for person detection, denoted as `Det* + AKA,' `Det* + PTKP,' `Det* + KRKC,' and Proposed*. Note that all the methods yield the perfect performance of detection in terms of the recall and AP, but the proposed method achieves better performance of re-ID compared with all the two-step methods.

Fig. <ref> visualizes the qualitative results of the proposed method and the two-step methods. We observe that both of PTKP <cit.> and KRKC <cit.> fail to find the query persons correctly in the challenging cases, for example, with the occluded persons and/or relatively small bounding boxes in the scene images. However, the proposed method successfully matches the query persons even in such cases, demonstrating the robustness to diverse LPS scenarios. 

It is worth to note that the two-step person search framework trains the backbone network when training the detection and re-ID networks, respectively, and hence requires high computational complexity and huge memory space.
In contrary, the proposed end-to-end method shares the backbone network between the jointly trained detection and re-ID networks, and is a more promising tool for LPS considering practical real-world applications.









 §.§ Ablation Study




  §.§.§ Losses

In Table <ref>, we first evaluate the effect of the three losses, the detection knowledge distillation loss ℒ_dkd, the re-ID knowledge distillation loss ℒ_rkd^+, and the rehearsal-based instance matching loss ℒ_rim, respectively. 
Note that detaching all the losses is the same as the FineTune method since the proposed losses are designed to work only when the old data are available.
We see that adding each loss improves the performance, respectively. Specifically, when we use ℒ_dkd, the detection performance is largely increased from that of the FineTune method on the old datasets, and the re-ID performance is also increased accordingly. 
On the other hand, ℒ_rkd^+ and ℒ_rim slightly increase the detection performance of using ℒ_dkd, but significantly improve the re-ID performance on the old datasets of CUHK-SYSU and PRW by huge margins, demonstrating the effectiveness to preserve the re-ID knowledge in the old domains.











 



  §.§.§ Baseline Network

It is worth to note that the proposed method can be applied to any baseline network of person search. We conducted the additional experiment by implementing the proposed method on the transformer based architecture of COAT <cit.>. Table <ref> shows the results where we see that the proposed method significantly improves the performance compared to that of the FineTune method.




  §.§.§ Old Prototype-Based Knowledge Distillation

Table <ref> shows the effect of using the old prototype features for re-ID knowledge distillation. 
The conventional method, Intra-batch, estimates the distributions of the feature similarity with respect to all the detected proposals within a mini-batch. On the other hand, the proposed method matches the distributions of the feature similarity by using the prototype features of all identities stored in the old LUT, and thus provides better results than the Intra-batch scheme.



  §.§.§ Exemplar Data Sampling

Table <ref> shows the results of using different sampling schemes to compose the exemplar data from the old datasets.
`Max BBox' samples the images that have the top 2% largest numbers of ground truth bounding boxes. `Max ID' samples the images that have the top 2% largest numbers of person instances with identity labels. `Random' samples 2% images randomly from the old datasets. 
We see that different sampling schemes provide similar performances to one another, and selected the uniform sampling that yields a slightly better performance of the old knowledge preservation compared to the other ones. 




 §.§ Limitation

The proposed method stores 2% of the old data into the exemplar memory the typical rehearsal (replay) based methodology of lifelong learning <cit.>. Therefore, the size of the exemplar memory increases as we have more and more datasets. As a future research topic, we will investigate other methodologies of lifelong learning that address the limitation of using the exemplar memory. 



§ CONCLUSION

In this paper, we proposed a novel LPS framework where the model needs to be incrementally trained on the new datasets while preserving the knowledge of the old datasets. We implemented the knowledge distillation between the old and new models based on the rehearsal methodology by using the representative prototype features of the labeled foreground persons as well as the hard background proposals in the old exemplar data. We also designed the rehearsal-based instance matching loss to improve the discrimination ability by using the unlabeled person instances in addition to the prototype features. Experimental results evaluated on three datasets of person search showed that the proposed method achieves significantly better performance of lifelong learning compared with the existing methods, and successfully prevents the knowledge forgetting in the old domains. We expect this pioneering work would encourage further research for practical LPS applications.


1
IEEEtran


reidfirst
E. Ahmed, M. Jones, and T. K. Marks, “An improved deep learning architecture for person re-identification,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 3908–3916.

li2018harmonious
W. Li, X. Zhu, and S. Gong, “Harmonious attention network for person re-identification,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2018, pp. 2285–2294.

luo2019strong
H. Luo, W. Jiang, Y. Gu, F. Liu, X. Liao, S. Lai, and J. Gu, “A strong baseline and batch normalization neck for deep person re-identification,” IEEE Trans. Multimedia, vol. 22, no. 10, pp. 2597–2609, Dec. 2019.

ning2020feature
X. Ning, K. Gong, W. Li, L. Zhang, X. Bai, and S. Tian, “Feature refinement and filter network for person re-identification,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 9, pp. 3391–3402, Sep. 2020.

liu2022neural
Z. Liu, L. Zhang, and D. Zhang, “Neural image parts group search for person re-identification,” IEEE Trans. Circuits Syst. Video Technol., vol. 33, no. 6, pp. 2724–2737, Jun. 2022.

huang2020multiscale
Y. Huang, S. Lian, H. Hu, D. Chen, and T. Su, “Multiscale omnibearing attention networks for person re-identification,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 5, pp. 1790–1803, May 2020.

chen2022saliency
C. Chen, M. Ye, M. Qi, J. Wu, Y. Liu, and J. Jiang, “Saliency and granularity: Discovering temporal coherence for video-based person re-identification,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 9, pp. 6100–6112, Sep. 2022.

norm_aware
D. Chen, S. Zhang, J. Yang, and B. Schiele, "Norm-Aware embedding for efficient person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2020.

kim2021prototype
H. Kim, S. Joung, I.-J. Kim, and K. Sohn, "Prototype-guided saliency feature learning for person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 4865–4874.


yan2021anchor
Y. Yan, J. Li, J. Qin, S. Bai, S. Liao, L. Liu, F. Zhu, and L. Shao, "Anchor-free person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, pp. 7690–7699.


han2021end
B.-J. Han, K. Ko, and J.-Y. Sim, "End-to-end trainable trident person search network using adaptive gradient propagation," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 925–933.

lee2022oimnet
S. Lee, Y. Oh, D. Baek, J. Lee, and B. Ham, "OIMNet++: Prototypical normalization and localization-aware learning for person search," in Proc. Eur. Conf. Comput. Vis. (ECCV), 2022, pp. 621–637.

yu2022cascade
R. Yu, D. Du, R. LaLonde, D. Davila, C. Funk, A. Hoogs, and B. Clipp, "Cascade transformers for end-to-end person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 7267–7276.

kirkpatrick2017overcoming
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., "Overcoming catastrophic forgetting in neural networks," Proc. Nat. Acad. Sci., vol. 114, no. 13, pp. 3521–3526, 2017, National Acad Sciences.

shmelkov2017incremental
K. Shmelkov, C. Schmid, and K. Alahari, "Incremental learning of object detectors without catastrophic forgetting," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 3400–3409.

peng2020faster
C. Peng, K. Zhao, and B. C. Lovell, "Faster ilod: Incremental learning for object detectors based on faster rcnn," Pattern Recogn. Lett., vol. 140, pp. 109–115, 2020, Elsevier.

wu2021generalising
G. Wu and S. Gong, "Generalising without forgetting for lifelong person re-identification," in Proc. AAAI Conf. Artif. Intell., vol. 35, no. 4, pp. 2889–2897, 2021.

pu2021lifelong
N. Pu, W. Chen, Y. Liu, E. M. Bakker, and M. S. Lew, "Lifelong person re-identification via adaptive knowledge accumulation," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 7901–7910.


ge2022lifelong
W. Ge, J. Du, A. Wu, Y. Xian, K. Yan, F. Huang, and W.-S. Zheng, "Lifelong Person Re-identification by Pseudo Task Knowledge Preservation," in Proc. AAAI Conf. Artif. Intell., vol. 36, no. 1, pp. 688–696, 2022.

huang2022lifelong
Z. Huang, Z. Zhang, C. Lan, W. Zeng, P. Chu, Q. You, J. Wang, Z. Liu, and Z.-j. Zha, "Lifelong unsupervised domain adaptive person re-identification with coordinated anti-forgetting and adaptation," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 14288–14297.


prw
L. Zheng, H. Zhang, S. Sun, M. Chandraker, Y. Yang, and Q. Tian, "Person re-identification in the wild," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 1367–1376.


chen2018person
D. Chen, S. Zhang, W. Ouyang, J. Yang, and Y. Tai, "Person search via a mask-guided two-stream cnn model," in Proc. Eur. Conf. Comput. Vis. (ECCV), Sep. 2018, pp. 734–750.



lan2018eccv
X. Lan, X. Zhu, and S. Gong, "Person search by multi-scale matching," in Proc. Eur. Conf. Comput. Vis. (ECCV), Sep. 2018.


wang2020tcts
C. Wang, B. Ma, H. Chang, S. Shan, and X. Chen, "TCTS: A task-consistent two-stage framework for person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2020, pp. 11952–11961.


ke2022joint
X. Ke, H. Liu, W. Guo, B. Chen, Y. Cai, and W. Chen, "Joint sample enhancement and instance-sensitive feature learning for efficient person search," IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 11, pp. 7924–7937, 2022.


cuhk
T. Xiao, S. Li, B. Wang, L. Lin, and X. Wang, “Joint detection and identification feature learning for person search,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 3415–3424.


chen2020hoim
D. Chen, S. Zhang, W. Ouyang, J. Yang, and B. Schiele, "Hierarchical online instance matching for person search," in Proc. AAAI Conf. Artif. Intell., vol. 34, no. 07, pp. 10518–10525, 2020.

zhang2021diverse
X. Zhang, X. Wang, J.-W. Bian, C. Shen, and M. You, "Diverse knowledge distillation for end-to-end person search," in Proc. AAAI Conf. Artif. Intell., vol. 35, no. 4, pp. 3412–3420, 2021.


seqnet
Z. Li and D. Miao, “Sequential end-to-end network for efficient person search,” in Proc. AAAI Conf. Artif. Intell., Jul. 2021, vol. 35, no. 3, pp. 2011–2019

cao2022pstr
J. Cao, Y. Pang, R. M. Anwer, H. Cholakkal, J. Xie, M. Shah, and F. S. Khan, "Pstr: End-to-end one-step person search with transformers," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 9458–9467.

fiaz2023sat
M. Fiaz, H. Cholakkal, R. M. Anwer, and F. Shahbaz Khan, "SAT: Scale-Augmented Transformer for person search," in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), Jan. 2023, pp. 4820–4829.

yang2024unleashing
X. Yang, M. Tian, N. Wang, and X. Gao, “Unleashing the Feature Hierarchy Potential: An Efficient Tri-Hybrid Person Search Model,” IEEE Trans. Circuits Syst. Video Technol., 2024.


oh2024domain
M. Oh, D. Kim, and J.-Y. Sim, "Domain generalizable person search using unreal dataset," in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 5, pp. 4361–4368, 2024.



han2021weakly
C. Han, K. Su, D. Yu, Z. Yuan, C. Gao, N. Sang, Y. Yang, and C. Wang, "Weakly supervised person search with region Siamese networks," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 12006–12015.

han2021context
B.-J. Han, K. Ko, and J.-Y. Sim, "Context-aware unsupervised clustering for person search," in British Machine Vision Conf., 2021.



yan2022exploring
Y. Yan, J. Li, S. Liao, J. Qin, B. Ni, K. Lu, and X. Yang, "Exploring visual context for weakly supervised person search," in Proc. AAAI Conf. Artif. Intell., vol. 36, no. 3, pp. 3027–3035, 2022.


wang2023self
B. Wang, Y. Yang, J. Wu, G.-j. Qi, and Z. Lei, "Self-similarity driven scale-invariant learning for weakly supervised person search," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2023, pp. 1813–1822.

daps
J. Li, Y. Yan, G. Wang, F. Yu, Q. Jia, and S. Ding, "Domain adaptive person search," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ECCV), Oct. 2022, pp. 302–318.


almansoori2024ddam
M. K. Almansoori, M. Fiaz, and H. Cholakkal, "DDAM-PS: Diligent domain adaptive mixer for person search," in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), Jan. 2024, pp. 6688–6697.

hao2019end
Y. Hao, Y. Fu, Y.-G. Jiang, and Q. Tian, “An end-to-end architecture for class-incremental object detection with knowledge distillation,” in 2019 IEEE Int. Conf. Multimedia Expo (ICME), Jul. 2019, pp. 1–6.

shieh2020continual
J.-L. Shieh, Q. M. ul Haq, M. A. ul Haq, S. Karam, P. Chondro, D.-Q. Gao, and S.-J. Ruan, "Continual learning strategy in one-stage object detection framework based on experience replay for autonomous driving vehicle," Sensors, vol. 20, no. 23, p. 6777, 2020, MDPI.

liu2023continual
Y. Liu, B. Schiele, A. Vedaldi, and C. Rupprecht, "Continual detection transformer for incremental object detection," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2023, pp. 23799–23808.



carion2020end
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," in Proc. Eur. Conf. Comput. Vis. (ECCV), 2020.



dong2023incremental
N. Dong, Y. Zhang, M. Ding, and G. H. Lee, "Incremental-detr: Incremental few-shot object detection via self-supervised learning," in Proc. AAAI Conf. Artif. Intell., vol. 37, no. 1, pp. 543–551, 2023.



li2022technical
D. Li, G. Cao, Y. Xu, Z. Cheng, and Y. Niu, "Technical report for iccv 2021 challenge sslad-track3b: Transformers are better continual learners," 2022, arXiv:2201.04924.


mirza2022efficient
M. J. Mirza, M. Masana, H. Possegger, and H. Bischof, "An efficient domain-incremental learning approach to drive in all weather conditions," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 3001–3011.

sun2022patch
Z. Sun and Y. Mu, "Patch-based knowledge distillation for lifelong person re-identification," in Proc. 30th ACM Int. Conf. Multimedia, 2022, pp. 696–707.

yu2023lifelong
C. Yu, Y. Shi, Z. Liu, S. Gao, and J. Wang, "Lifelong person re-identification via knowledge refreshing and consolidation," in Proc. AAAI Conf. Artif. Intell., vol. 37, no. 3, pp. 3295–3303, 2023.

xu2024lstkc
K. Xu, X. Zou, and J. Zhou, "LSTKC: Long short-term knowledge consolidation for lifelong person re-identification," in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 14, pp. 16202–16210, 2024.


ren2015faster
S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," Adv. Neural Inf. Process. Syst., 2015.


rebuffi2017icarl
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, "icarl: Incremental classifier and representation learning," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 2001–2010.


qin2023movienet
J. Qin, P. Zheng, Y. Yan, R. Quan, X. Cheng, and B. Ni, “MovieNet-PS: a large-scale person search dataset in the wild,” in ICASSP 2023-2023 IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Jun. 2023, pp. 1–5.









 Paper Title: An Effective Dynamic Gradient Calibration Method for Continual Learning.



[
An Effective Dynamic Gradient Calibration Method for Continual Learning














equal*


Weichen Linequal,yyy
Jiaxiang Chenequal,yyy
Ruomin Huangcomp
Hu Dingsch





yyySchool of Data Science, University of Science
 and Technology of China, Anhui, China.
compDuke University 
schSchool of Computer Science and Technology, University of Science and Technology of China, Anhui, China

Hu Dinghuding@ustc.edu.cn





Machine Learning, ICML

0.3in
]










 

§ ABSTRACT 
Continual learning (CL) is a fundamental topic in machine learning, where the goal is to train a model with continuously incoming data and tasks. Due to the memory limit, we cannot store all the historical data, and therefore confront the “catastrophic forgetting” problem, i.e., the performance on the previous tasks can substantially decrease because of the missing information in the latter period. Though a number of elegant methods have been proposed, the catastrophic forgetting phenomenon still cannot be well avoided in practice. In this paper, we study the problem from the gradient perspective, where our aim is to develop an effective algorithm to calibrate the gradient in each updating step of the model; namely, our goal is to guide the model to be updated in the right direction under the situation that a large amount of historical data are unavailable. Our idea is partly inspired by the seminal stochastic variance reduction methods (e.g., SVRG and SAGA)  for reducing the variance of gradient estimation in stochastic gradient descent algorithms. Another benefit is that our approach can be used as a general tool, which is able to be incorporated with several existing popular CL methods to achieve better performance. We also conduct a set of experiments on several benchmark datasets to evaluate the performance in practice. 




§ INTRODUCTION


In the past years, Deep Neural Networks (DNNs) demonstrate remarkable performance for many different tasks in artificial intelligence, such as image generation <cit.>, classification  <cit.>, and pattern recognition <cit.>. Usually we assume that the whole training data is stored in our facility and the DNN models can be trained offline by using Stochastic Gradient Descent (SGD) algorithms <cit.>.  

However, real-world applications often require us to consider training lifelong models, where the tasks and data are accumulated in a streaming fashion <cit.>. For example, with the popularity of smart devices, a large amount of new data is generated every day. A  model needs to make full use of these new data to improve its performance while keeping old knowledge from being forgotten.  Those applications motivate us to  study the problem of continual learning (CL) <cit.>, where its goal is to develop effective method for gleaning insights from current data while retaining information from prior training data.





  A significant challenge that CL encounters is  “catastrophic forgetting” <cit.>, wherein the exclusive focus on the current set of examples could result in a dramatic deterioration in the performance on previously learned data. This phenomenon is primarily attributed to limited storage and computational resources during the training process; otherwise, one could directly train the model from scratch using all the saved data. To address this issue, we need to develop efficient algorithm for training neural networks from a continuous stream of non-i.i.d. samples, with the goal of mitigating catastrophic forgetting while effectively managing computational costs and memory footprint.

  A number of elegant CL methods have been proposed to alleviate the catastrophic forgetting issue <cit.>. One representative CL approach is referred to as “Experience Replay (ER)” <cit.>, which has shown promising performance in several 

continual learning scenarios <cit.>. Roughly speaking, the ER method utilizes reservoir sampling <cit.> to maintain historical data in the buffer, then extract new incoming training data with random samplings for learning the current task. 

Though the intuition is simple, the ER method currently is one of the  most popular CL approaches that incurs 



moderate computational and storage demands.

Moreover, several recently proposed approaches suggest that the ER method can be combined with knowledge distillation to further improve the performance; for example, 
the methods of DER/DER++ <cit.> and X-DER <cit.> preserve previous training samples alongside their logits in the model as the additional prior knowledge.

Besides the ER methods, there are also several other types of CL techniques proposed in recent years, and please 

refer to <ref> for a detailed introduction.



 §.§ Our Main Ideas and Contributions


  Though existing CL methods can alleviate the catastrophic forgetting issue from various aspects, the practical performances in some scenarios are still not quite satisfying <cit.>.
  
  In this paper, 

we study the continual learning problem from the gradient perspective, and the rationality behind is as follows. In essence, an approach for avoiding the catastrophic forgetting issue in CL,  e.g., the replay mechanisms or the regularization strategies, ultimately manifests its influence on the gradient directions during model updating <cit.>. If all the historical data are available, one could compute the gradient by using the stochastic gradient descent method and obviously the catastrophic forgetting phenomenon cannot happen. The previous  methodologies  aim to approximate the gradient by preserving additional information and incorporating it as a constraint to model updates, thereby retaining historical knowledge. However,   the replay-based methods in practice are often limited by storage capacity, which leads to a substantial loss of historical data information and inaccurate estimation of historical gradients <cit.>.

Therefore, our goal is to develop a more accurate gradient calibration algorithm in each step of the continual learning procedure, which can directly enhance the training quality. 

 We are aware of several existing CL methods that also take into account of the gradients <cit.>, but our idea proposed here is fundamentally different. 
 We draw inspiration from the seminal stochastic variance reduction methods (e.g., SVRG from <cit.> and SAGA from <cit.>), which are originally designed to reduce the gradient variance so that the estimated gradient can closely align with the true full gradient over the entire dataset (including the current and historical data). These variance reduction methods have been extensively studied in the line of the research on stochastic gradient descent method <cit.>; their key idea is to leverage the additional saved full gradient information to calibrate the gradient in the current training step, which leads to significantly reduced gradient variance comparing with the standard SGD method.  This intuition also inspires us to handle the CL problem. In a standard SGD method, the variance between the obtained gradient and the full gradient is due to the “batch size limit" (if the batch size has no bound, we can simply compute the full gradient).  Recall that the challenge of CL is due to the “buffer size limit”, which impedes the use of full historical data (this is similar with the dilemma encountered by SGD with the “batch size limit”). So an interesting question is 
 
 

 Can the calibration idea for “batch size limit” be modified to handle “buffer size limit”? Specifically,  is it possible to develop an effecitve method to compute a SVRG (or SAGA)-like calibration for the gradient in  CL scenarios?


  Obviously, it is challenging to directly implement the SVRG or SAGA algorithms in continual learning because of the missing historical data. 

Note that <cit.> proposed a streaming SVRG (SSVRG) method that realizes the SVRG method within a given fixed buffer, but unfortunately it does not perform quite well in the CL scenarios (as shown in our experimental section). One possible reason is that SSVRG can only leverage information within the buffer and fails to utilize all historical information. 



In this paper, we aim to apply the intuition of SVRG to handle the “buffer size limit” in CL, and our contributions can be summarized as follows:




    
  * First, we propose a novel two-level dynamic algorithm, named Dynamic Gradient Calibration (DGC),   to maintain a gradient calibration in continual learning. DGC can effectively tackle the storage limit and leverage historical data to calibrate the gradient of the model at each current stage. 
   
   Moreover, our theoretical analysis shows that our DGC based CL algorithm can achieve a linear convergence rate. 

    
   
   
    
  * Second, our method can be conveniently integrated with most existing reservoir sampling-based continual learning approaches (e.g., ER <cit.>, DER/DER++ <cit.>, XDER <cit.>, and Dynamic ER <cit.>), where this hybrid framework can 
    
    induce a significant enhancement to the overall model performance. 
    
    Note that storing the gradient calibrator can cause an extra memory footprint; so a key question is whether such an extra memory footprint can yield a larger marginal benefit than simply taking more samples to fill the extra memory? 
    In Fig <ref>, we illustrate a brief example to answer this question in the affirmative; more detailed evaluations are shown in the experimental part. 

    
    
  * Finally, we conduct a set of comprehensive experiments on the popular datasets S-CIFAR10, S-CIFAR100 and S-TinyImageNet; the experimental results suggest that our method can improve the final Average Incremental Accuracy (FAIA) in several CL scenarios by more than 6%. Moreover, our improvement for a larger number of tasks is more significant than that for a smaller number. Furthermore, our adoption of the SVRG-inspired DGC calibration method leads to enhanced stability in minimizing the loss function throughout the parameter optimization process.
    
    
    



 §.§ Related Work





We briefly overview existing important continual learning approaches (except for the ones mentioned before).  We also refer the reader to the recent surveys <cit.> for more details.



  
  A large number of CL methods are replay based, where they often keep a part of previous data through approaches like reservoir sampling <cit.>. Several more advanced data selection strategies focus on optimizing the factors like the sample diversity of parameter gradients or the similarity to previous gradients on passed data, e.g.,  GSS <cit.> and   GCR <cit.>.  
  
  Experience replay can be effectively combined with knowledge distillation. For example, <cit.> proposed to distill colliding effects from the features for new coming tasks, and ICARL <cit.> proposed to take account of the data representation trained on old data. MOCA <cit.> improves replay-based methods by diversifying representations  in the  space. Another replay-based approach is based on generative replay, which obtains replay data by generative models <cit.>.
  



  
  Another way for solving continual learning is through some deliberately designed optimization procedures. For example, the methods GEM <cit.>, AGEM <cit.>, and MER <cit.> restrict parameter updates to align with the experience replay direction, and thereby preserve the previous input and gradient space with old training samples. Different from saving old training samples, <cit.> proposed to adapt parameter updates in the orthogonal direction of the previously saved gradient.   The method AOP <cit.> projects the gradient in the direction orthogonal to the subspace spanned by all previous task inputs, therefore it only keeps an orthogonal projector rather than storing previous data.






  
  To mitigate the problem of forgetting, we can also augment the model capacity for learning new tasks. <cit.> tried to enhance model performance by employing meta-learning techniques  when dynamically extending the model.
  

The method ANCL <cit.> proposes to utilize an auxiliary network to achieve a trade-off between 
plasticity 
and 
stability. Dynamic ER <cit.> introduces a novel two-stage learning method that employs a dynamically expandable representation for learning knowledge incrementally. 



§ PRELIMINARIES



We consider the task of training a soft-classification function f(· ;θ):𝒳→𝒴, where 𝒳 and 𝒴 respectively represent the space of data and the set of labels, and θ is the parameter to optimize. Without loss of generality, we assume 𝒴={1, 2, ⋯, K}. So 
 f(·; θ) maps each x∈𝒳 to some f(x; θ)∈ℝ^K. 
 
 To find an appropriate θ, the classification function  f is usually equipped with a loss function ℓ (f(x;θ), y), which is differentiable for the variables x and θ (e.g., cross-entropy loss). To simplify the notation, we use ℓ(x,y,θ) to denote ℓ(f(x;θ), y).  Given a set of data P={(x^i, y^i)| 1≤ i≤ n} = 𝒳_P×𝒴_P ⊂𝒳×𝒴, the training process is to find a θ such that the empirical risk of ℓ(x,y, θ), i.e., ∑^n_i=1ℓ(x^i,y^i, θ), is minimized. We define the full gradient of ℓ(x,y, θ) on P as 

    𝒢(P, θ)    ≜∇_θℓ(𝒳_P, 𝒴_P, θ) = 1/n∑^n_i=1∇_θℓ(x^i, y^i, θ).




 §.§ Continual Learning Models


In this paper, we focus on two popular CL models: Class-Incremental Learning (CIL) <cit.> and Task-Free Continual Learning (TFCL) <cit.>.


  In the setting of CIL, the training tasks come in a sequence {𝒯_1, 𝒯_2, ⋯, 𝒯_T} with disjoint label space; each time spot t ∈{1,2,⋯,T} corresponds to the task 𝒯_t with a training dataset {(x_t^i,y_t^i)| 1≤ i≤ n_t}. With a slight abuse of notations, we also use 𝒯_t to denote its training dataset. Also, we use 𝒴_t to denote the corresponding set of labels {y_t^i| 1≤ i≤ n_t}. 


Although the training data for individual tasks 𝒯_t is independently and identically distributed (i.i.d.), it is worth noting that the overall task stream {𝒯_1, 𝒯_2, ⋯, 𝒯_T} does not adhere to the i.i.d. assumption due to the evolving label space over time.  We define the overall risk under this CL setting 
  at the current time spot t 
 as follows: 

    ℓ_CL^t (θ) ≜1/t∑_c = 1^t(x, y) ∼𝒯_c𝔼[ℓ(x,y,θ)].








  The setting of TFCL is similar to CIL, where the major difference is that the task identities are not provided in neither the training nor testing procedures. So the TFCL setting is more challenging than CIL because the algorithm is unaware of task changes and the current task identity. In the main part of our paper, we present our results in CIL; in our supplement, we  explain how to extend our results to TFCL.







 §.§ Variance Reduction Methods




A comprehensive introduction to variance reduction methods is provided in <cit.>. Here 
we particularly introduce 

one representative variance reduction method SVRG <cit.> 

, which is closely related to our proposed CL approach. 

The high-level idea of SVRG is to construct a calibration term to reduce the variance in the gradient estimate. The complete optimization process can be segmented into a sequence of stages. We denote the training data as P = {(x^i,y^i)}_i=1^n⊂𝒳×𝒴,  and denote the parameter at the beginning of each stage as θ̃.  The key part of SVRG is to  minimize the variance in SGD optimization by computing 

an additional term μ̃:


    μ̃   ≜𝒢(P, θ̃) =  1/n∑_i = 1^n∇_θℓ(x^i,y^i, θ̃).

In each stage, 
SVRG applies the standard SGD with the term μ̃ to update the parameter θ: 



let

    v^k     =  ∇_θℓ(x^i, y^i,θ^k) - (∇_θℓ(x^i, y^i, θ̃) -μ̃), 
    θ^k+1    = θ^k - η v^k,

where (x^i,y^i) ∈ P is the sampled training data, θ^k represents the  parameter at the k-th step of SGD, and η is the learning rate. Since 𝔼[∇_θℓ(x^i, y^i, θ̃^̃k̃)] = μ̃, v^k is an unbiased estimate of the gradient 𝒢(P,θ^k). Subsequently, the term “∇_θℓ(x^i, y^i, θ̃) - μ̃” in Eq (<ref>) can be regarded as a calibrator to reduce the variance of gradient estimation and achieve a linear convergence rate <cit.>, which is faster than directly using ∇_θℓ(x^i, y^i,θ^k).















§ OUR PROPOSED METHOD



In this section, we propose 
the Dynamic Gradient Calibration (DGC) approach which maintains a gradient calibration during the learning process. A highlight of DGC is that it utilizes the whole historical information to obtain a more precise gradient estimation, and consequently relieves the negative impact of catastrophic forgetting. In <ref>, we introduce ER and analyze the obstacle if we directly combine ER with SVRG. 

In <ref>, we present our DGC  method for addressing the issues discussed in <ref>. In <ref>, we explain how to integrate 
  DGC with other  CL  techniques.



 §.§ Experience Replay Revisited and SVRG



 First, we overview the classical ER <cit.> method as a baseline for the CIL setting. ER employs the reservoir sampling algorithm to dynamically manage a buffer (denoted as ℳ_t) at time t, which serves to store historical data. At each time spot t (and assume the current updating step number of the optimization is k), ER updates the model parameter  θ^k_t following the standard gradient descent method: 

    θ_t^k+1 = θ_t^k - η· v_t^k,

where η is the learning rate and v_t^k is the calculated gradient. If not using any replay strategy, v_t^k is usually calculated on a randomly sampled training data (x^k_t, y^k_t) ∈𝒯_t. It is easy to see that this simple strategy can cause the forgetting issue for shifting data stream since it does not contain any information from the previous data. 

Hence the classical ER algorithm takes a random sample 
(x̅^k_t, y̅^k_t) from the aforementioned buffer ℳ_t (who contains a subset of historical data via reservoir sampling), and computes the gradient:


    v_t^k =1/t∇_θℓ(x^k_t, y^k_t, θ_t^k) + t-1/t∇_θℓ(x̅^k_t, y̅^k_t, θ_t^k).



(1) For simplicity, we assume that the data sets of all the tasks have the same size. So the obtained v_t^k in (<ref>) is an unbiased estimation of the full gradient 𝒢(⋃_c=1^t𝒯_c,θ) at the current time spot t. If they have different sizes, we can simply replace the coefficients “1/t” and “t-1/t” by “|𝒯_t|/∑^t_c=1|𝒯_c|” and “∑^t-1_c=1|𝒯_c|/∑^t_c=1|𝒯_c|”, respectively. (2) Also, we assume that the batch sizes of the random samples from 𝒯_t and ℳ_t are both “1”, i.e., we only take single item (x^k_t, y^k_t) and (x̅^k_t, y̅^k_t) from each of them. Actually, we can also take larger batch sizes and then the Eq (<ref>) can be modified correspondingly by taking their average gradients. 













  Now we attempt to apply the SVRG method to Eq (<ref>).  

Our objective is to identify a more accurate unbiased estimate of the gradient at the current time spot t so as to determine the updating direction.


At first glance, one possible solution is to adapt the streaming SVRG method <cit.> to the CL scenario. 
We treat ℳ_t as the static data set in our memory, and apply the 
 SVRG technique to calibrate the gradient “∇_θℓ(x^k_t, y^k_t, θ_t^k)” and “∇_θℓ(x̅^k_t, y̅^k_t, θ_t^k)” in Eq (<ref>). 

 Similar to the procedure introduced in <ref>, we denote the parameter at the beginning of the current stage s as θ̃_t,s. For simplicity, when we consider the update within stage s, we just use θ̃_t to denote θ̃_t,s. Similar with Eq (<ref>), we define the  terms
 





    μ̃≜𝒢(ℳ_t, θ̃_t),   ṽ≜𝒢(𝒯_t, θ̃_t).


Then we can calibrate the gradient by ∇_θℓ(x_t,y_t,θ̃_t) and ∇_θℓ(x̅_t, y̅_t, θ̃_̃t̃) (which serves as the similar role of ∇_θℓ(x^i, y^i, θ̃ in (<ref>)); the new form of v_t^k becomes

    v_t^k     =1/t(∇_θℓ(x_t, y_t, θ_t^k) - ∇_θℓ(x_t,y_t,θ̃_t) + ṽ) + 
       t-1/t( ∇_θℓ(x̅_t, y̅_t, θ_t^k) -  ∇_θℓ(x̅_t, y̅_t, θ̃_̃t̃) + μ̃).


  
Obviously, if the buffer ℳ_t contains the whole historical data (denote by 𝒯_[1:t) = ⋃_c=1^t-1𝒯_c), the above approach is exactly the standard SVRG. However, because ℳ_t only takes a small subset of 𝒯_[1:t), this approach still cannot avoid information loss for the previous tasks. In next section, we propose a novel two-level dynamic algorithm to record more useful information from 𝒯_[1:t), and thereby reduce the information loss induced by ℳ_t. We also take the approach of Eq (<ref>) as a baseline in <ref> to illustrate the advantage of our proposed approach. 



 §.§ Dynamic Gradient Calibration





To tackle the issue discussed in <ref>,  we propose a novel two-level update approach “Dynamic Gradient Calibration (DGC)” to maintain our calibration term. Our focus is designing a method to incrementally update an unbiased estimation for 𝒢(𝒯_[1:t), θ_t^k). To illustrate our idea clearly, we decompose our analysis to two levels: (1) update the parameter during the training within each time spot t; (2) update the parameter at the transition from time spot t to  t+1 (i.e., the moment that the task 𝒯_t has just been completed and the task 𝒯_t+1 is just coming). 











  (1) How to update the parameter during the training within each time spot t. We follow the setting of the streaming SVRG as discussed in <ref>: 
  the training process at the current time spot t is divided into a sequence of stages;  the model parameter at the beginning of each stage is recorded as θ̃_t. 
  To illustrate our idea for calibrating the gradient v^k_t in Eq (<ref>), we begin by considering an “imaginary” approach: 
  
  we let
 

    v_t^k     =1/t( ∇_θℓ(x_t, y_t, θ_t^k) -∇_θℓ( x_t, y_t,θ̃_t) + 𝒢(𝒯_t,θ̃_t)) 
        + t-1/tΓ,

 where 

    Γ= ∇_θℓ(x̅_t, y̅_t, θ_t^k)- (∇_θℓ(x̅_t, y̅_t, θ̃_t)  - 𝒢(𝒯_[1:t), θ̃_t )). _Calibration from the previous parameter θ̃_t

Different from Eq (<ref>), we compute v^k_t based on the full historical data 𝒯_[1:t), which follows the same manner of SVRG. However, a major obstacle here is that we cannot obtain the exact Γ since 𝒯_[1:t) is not available. This motivates us to design a relaxed form of (<ref>).

We define a surrogate function to approximate Γ, which can be computed through recursion. Suppose each training stage has m≥ 1 steps, then we define

    Γ_DGC(t, k)     = ∇_θℓ(x̅_t, y̅_t, θ_t^k) - 
       (∇_θℓ(x̅_t, y̅_t, θ̃_t)  - Γ'_DGC(t) ), 
    Γ'_DGC(t)     = Γ_DGC(t, m+1),

 in the stage. Note that the term “∇_θℓ(x̅_t, y̅_t, θ̃_t)” in (<ref>) can be computed by the previous parameter θ̃_t during training, so we do not need to store it in buffer. For the initial  t=1 case (i.e., when we just encounter the first task),  we can directly set Γ'_DGC(1)=0. We update the function Γ'_DGC(t) at the end of each training stage in (<ref>), and use the function Γ_DGC(t, k) to approximate Γ in (<ref>). Comparing with the original formulation of Γ in (<ref>), we only replace the term “𝒢 (𝒯_[1:t), θ̃_t )” by “Γ'_DGC(t)”. Also, we have the following lemma to support this replacement.  The detailed proof of lemma <ref> is provided in our supplement.




    𝔼[Γ'_DGC(t) ] = 𝒢 (𝒯_[1:t), θ̃_t )



   We utilize the term “∇_θℓ(x̅_t, y̅_t, θ̃_t)  - Γ'_DGC(t)” of (<ref>) as the calibrator for each updating step, thereby preserving the unbiased nature of the gradient estimator and reducing the variance of gradient estimation.

  (2) How to update the parameter at the transition from  time spot t to t+1. 

 At the end of  time spot t, we update the recorded θ̃_t to θ_t^m+1, and the data 𝒯_t from time t should be integrated into the historical data. In this context, it is essential to update the calibrated gradient accordingly:







    Γ'_DGC(t+1) = 1/t((t-1) ·Γ'_DGC(t) +  𝒢(𝒯_t, θ̃_t) ) .





 The complete algorithm is presented in Algorithm <ref>. Compared with the conventional reservoir sampling based approaches, we only require the additional   
 
  storage for keeping Γ'_DGC(t), and so that the gradient Γ_DGC(t, k) in (<ref>) can be effectively updated by the recursion. Moreover, our method can also conveniently adapt to TFCL where the task boundaries are not predetermined. In such a setting, we can simply treat each batch data (during the SGD) as a “micro” task at the time point and then update the gradient estimation via Eq (<ref>).
  
  
  The detailed algorithm for TFCL is placed in our appendix. 

Similar to the theoretical analysis of SVRG <cit.>, under mild assumptions, the optimization procedure of our DGC method at each time spot t can also achieve linear convergence. We denote the optimal parameter at time spot t as θ_*≜min_θℓ_CL^t(θ). Then we have the following theorem. 













  
 Assume that f(x;θ) is  L-smooth and γ-strongly convex; the parameters 

m ≥10L^2/γ^2 and η = γ/10L. Then we have a linear convergence in expectation for the DGC procedure at time t:

    𝔼[θ̃_t,s+1-θ_*_2^2] ≤1/2^s𝔼[θ̃_t,1-θ_*_2^2]

where θ̃_t,s represents the initialization parameter at the beginning of the s-th stage at time spot t.


  The  proof of Theorem <ref> is provided in appendix. This theorem indicates that the gradient calibrated by our DGC method shares the similar advantages with SVRG. For instance, when updating each task 𝒯_t, the loss function has a smoother decrease (we validate this property in <ref>).





 §.§ Combine DGC with Other CL Methods




Our proposed DGC approach can be also efficiently combined with other CL methods. 
As discussed in Section <ref>, a number of popular CL methods rely on the reservoir sampling technique to preserve historical data in   buffer <cit.>. 



For these methods, such as DER and XDER, we can conveniently combine the conventional batch gradient descent with 

the DGC calibrated gradient estimator Γ_DGC(t,k) defined in <ref>, so as to obtain a more precise gradient estimator with reduced variance based on Eq (<ref>):

    v_t^k    =1/t(∇_θℓ(x_t, y_t, θ_t^k) - ∇_θ(x̅_t,y̅_t,θ̃_t) + 𝒢(𝒯_t,θ̃_t)  )+ 
       t-1/t[ αΓ_DGC(t,k) + (1-α)∇_θℓ(x̅_t, y̅_t, θ^k_t)],

where α is a given parameter to control the proportion of the two unbiased estimations Γ_DGC(t,k) and ∇_θℓ(x̅_t, y̅_t, θ^k_t) of the gradient 𝒢(𝒯_[1:t),θ_t^k). 
According to the theoretical analysis in SSVRG <cit.>, 

the selection of α should be related to 1/L, where L is the smoothness coefficient of the model f(x;θ), i.e., 

    L = max_θ_1,θ_2 ∈Θ|∇_θ f(x;θ_1) - ∇_θ f(x;θ_2)|/|θ_1-θ_2|,


where Θ represents the parameter space. The experimental study on the impact of α is placed in our supplement. In <ref>, we show that the amalgamation of the CL method and our DGC calibration procedure can yield a more precise update direction, and consequently enhance the ultimate model performance. 

























§ EXPERIMENTS



We conduct the experiments to compare with

various baseline methods across different datasets. We consider both the CIL and TFCL models. 










  Datasets We carry out the experiments on three widely employed datasets S(Split)-CIFAR10, S-CIFAR100 <cit.>, and S-TinyImageNet <cit.>. S-CIFAR10 is the split dataset by 
partitioning CIFAR10 into 5 tasks, each containing two categories; similarly, S-CIFAR100 and S-TinyImageNet are the datasets by respectively partitioning 
 CIFAR100 and TinyImageNet  into 10 tasks, each containing 10 (S-CIFAR100) and   20 (S-TinyImageNet) categories. 
 

  Baseline methods We consider the following baselines. (1) Replay-based methods: ER <cit.>, DER++
  <cit.>, XDER <cit.>, MOCA <cit.>, GSS <cit.>, GCR <cit.>, HAL <cit.>, and ICARL <cit.>. (2) Optimization-based methods: AGEM <cit.>, AOP <cit.>, and SSVRG <cit.>.
 
  (3) Dynamic architecture method: Dynamic ER <cit.>.  We integrate DGC with ER, DER++, XDER, and Dynamic ER, and assess their performances in CIL. For  TFCL, we consider its combination with ER and DER++.
For convenience, we use “DGC-Y” to denote the combination of  

 DGC  with a CL method “Y”. For example, DGC-ER denotes the method combining ER and DGC methods.
 

  
  
  

  Evaluation metrics

We employ the Average Accuracy (AA) <cit.> and the final Average Incremental Accuracy (FAIA) <cit.> to assess the performance. These two metrics are both widely used for continual learning. Let a_k,j∈ [0,1] (k≥ j) denote the classification accuracy evaluated on the testing set of the task 𝒯_j after learning  𝒯_k. The value AA at time spot i is defined as

    AA_i ≜1/i∑_j=1^i a_i,j. 

In particular, we  name the value AA_T as the final Average Accuracy (FAA).
The final AIA is defined as 

    FAIA≜1/T∑_i=1^TAA_i.




We also use Final Forgetting (FF) from <cit.> to measure the forgetting of the model throughout the learning process
(the formal definition of FF and its numerical results are placed in the supplement).
Each instance of our experiments is repeated by 10 times.









 §.§ Results in CIL



  Hyper-parameters selection In our implementation, we fixed the values of epoch and batch size, which implies that the total number of optimization steps (i.e., the value s× m) is also fixed. In our experiments, we set the value m=200, so the value of s in Algorithm <ref> is also determined.



 Except for the Dynamic ER (which is dynamically expanded), all other testing methods have constant storage limits. The results shown in Table <ref> reveal that our DGC method can bring improvements to the combined methods on the testing benchmarks.
 For the sake of clarity, we also underline the best-performing method except Dynamic ER and DGC-Dynamic ER in Table <ref>.
Among the methods with constant storage limits, DGC-ER achieves the best results on S-CIFAR10 when the buffer size is 2000, and DGC-XDER achieves the best results on S-CIFAR100 and S-TinyImageNet.
It is worth noting that our DGC method can also be conveniently integrated with existing advanced dynamic expansion representation techniques, such as Dynamic ER,  
which demonstrates the improvements  to certain extent, e.g.,  it achieves an improvement more than 6% on S-CIFAR10/100 with buffer size 2000.   We also record the training time of these baseline methods in our supplement.









  In the subsequent experiment, we investigate the performance of DGC compared to other baseline methods with varying the number of tasks. Throughout the experiment, we maintain a constant buffer size of 2000. As outlined in Table <ref>, our results demonstrate that DGC can bring certain improvements to ER, DER++, and XDER with setting the number of tasks to be 5 and 20.
  
Moreover, similar to the previous research in <cit.>,   the results shown in Table <ref> also imply that increasing the number of tasks could exacerbate the catastrophic forgetting issue. This phenomenon occurs because the model faces a reduced volume of data on each specific task, and thereby necessitates the capability of retaining the information of historical data to guide the model updates. 
As can be seen, the improvement obtained by DGC for the case of 20 tasks usually is more significant compared with the case of 5 tasks. For example, DGC-DER++ achieves a 3.94% improvement with 20 tasks versus a 1.18% improvement with 5 tasks, while DGC-XDER exhibits a 3.08% improvement with 20 tasks and only 0.42% with 5 tasks. These results highlight the advantage of DGC for mitigating catastrophic forgetting by effectively utilizing historical data through gradient-based calibration. 





 §.§ Results in TFCL



 
We then conduct the experiments in TFCL. The curves shown in Figure <ref> depict the average AA_t evolutions with varying the “implicit” task number t on S-CIFAR100. We call it “implicit” since the value t is not given during the training, which means that the design of the algorithm cannot rely on task boundaries or task identities. Therefore, we only compare those baseline methods that are applicable to TFCL model. The results suggest that DGC can bring improvements to ER and DER++ on AA_t for almost all the ts; in particular, DGC-DER++ achieves the best performance and also with small variances. 
Through approximating the full gradient, the GCR and SSVRG methods can relieve the catastrophic forgetting issue to a certain extent, but they still suffer from the issue of storage limit, which affects their effectiveness for estimating the full gradient. Consequently, these methods exhibit performance downgrade and larger variance in Figure <ref>. Comparing with them, the approaches integrated with DGC illustrate more consistent and stable improvements.
More detailed results for TFCL are available in our supplement.



 §.§ Smoothness of Training with DGC





  Since our proposed DGC approach stems from the variance reduction method of SGD, a natural question is whether it also improves the smoothness during the training process.
  We  compare the loss trajectories on the entire training dataset before and after implementing DGC.  From Figure <ref> we can see that the classical ER method has erratic fluctuations in loss during the training process; this could cause some practical problems in a real-world CL scenario, e.g., 
  we may need to pay more effort to carefully adjust the learning rate and determine the stopping condition.
 
  In contrast, the  DGC method has a smoother reduction in loss, and ultimately yields lower loss values.
  



§ CONCLUSION


In this paper, we revisit the experience replay method and aim to utilize historical information to derive a more accurate gradient for alleviating catastrophic forgetting. Inspired by the variance reduction methods for SGD, we introduce a new approach “DGC” to dynamically manage a gradient calibration term in CL training. Our approach can be conveniently integrated with several existing continual learning methods, contributing to a substantial improvement in both CIL and TFCL. Moreover, the improved stability of training loss reduction can also ease our practical implementation. 


 

§ ACKNOWLEDGEMENTS

We want to thank the anonymous reviewers and Xianglu Wang for their helpful comments. The research of this work was supported in part by the National Key Research and Development Program of China 2021YFA1000900, the National Natural Science Foundation of China 62272432, and the Natural Science Foundation of Anhui Province 2208085MF163. 
 





§ IMPACT STATEMENT

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.


icml2024











§ OMITTED PROOFS






    𝔼[Γ'_DGC(t) ] = 𝒢 (𝒯_[1:t), θ̃_t ).




We prove it by induction. When t = 1,  both the two-side terms are 0, so the equation holds. Assume the equation holds when t = t_1, and we consider the change of  Γ'_DGC(t) at the time spot t_1.

Firstly, Γ'_DGC(t_1) is updated at the end of every stage.  After the update, Γ'_DGC(t_1)=Γ_DGC(t_1,m+1). Meanwhile, we have

    𝔼[Γ_DGC(t_1,m+1)] 
        = 𝔼[∇_θℓ(x̅_t_1, y̅_t_1, θ_t_1^m+1) - (∇_θℓ(x̅_t_1, y̅_t_1, θ̃_t_1)  - Γ'_DGC(t_1))] 
       = 𝒢(𝒯_[1:t_1),θ_t_1^m+1) - 𝒢(𝒯_[1:t_1),θ̃_t_1) + 𝒢(𝒯_[1:t_1),θ̃_t_1) 
       = 𝒢(𝒯_[1:t_1),θ_t_1^m+1) 
       = 𝒢(𝒯_[1:t_1),θ̃_t_1)  ,

where the first equation comes from Eq (<ref>) and the second equation is based on the inductive hypothesis.

  Secondly, in the update from  time spot t to t+1,  we have

    𝔼[Γ'_DGC   (t_1+1)] 
       = 𝔼[1/t_1((t_1-1) ·Γ_DGC(t_1,m+1) +  𝒢(𝒯_t_1, θ̃_t_1) )] 
       = 1/t_1((t_1-1) ·𝒢(𝒯_[1:t_1),θ̃_t_1) +  𝒢(𝒯_t_1, θ̃_t_1) ) 
       = 𝒢(𝒯_[1:t_1),θ̃_t_1+1)

 based on Eq (<ref>).
This implies that our conclusion also holds when t = t_1+1.



























































































































































Assume that f(x;θ) is  L-smooth and γ-strongly convex; the parameters 

m ≥10L^2/γ^2 and η = γ/10L. Then we have a linear convergence in expectation for the DGC procedure at time t:

    𝔼[θ̃_t,s+1-θ_*_2^2] ≤1/2^s𝔼[θ̃_t,1-θ_*_2^2]

where θ̃_t,s represents the initialization parameter at the beginning of the s-th stage at time spot t.





Our proof  is inspired by the idea of <cit.>. To simplify the notations, we define 

    v_t^k = ∇ψ_x_t(θ_t^k)-∇ψ_x_t(θ̃_t)+μ̃
 through combining Eq (<ref>) and (<ref>) (we replace “Γ” in Eq (<ref>) by (<ref>)), where

    ∇ψ_x_t(θ_t^k)    ≜1/t∇_θℓ(x_t,y_t,θ_t^k) + t-1/t∇_θℓ(x̅,y̅,θ_t^k) , 
    ∇ψ_x_t(θ̃_t)    ≜1/t∇_θℓ( x_t, y_t, θ̃_t) + t-1/t∇_θℓ(x̅,y̅,θ̃_t) , 
    μ̃   ≜1/t𝒢(𝒯_t, θ̃_t) + t-1/tΓ^'_DGC(t) .


Before proving Theorem 3.3, we provide the following key lemmas first.

 

    𝔼[v_t^k_2^2] ≤ 2 L[𝔼[||θ_t^k-θ_*||_2^2]+𝔼[||θ̃_t-θ_*||_2^2]].




    𝔼[v_t^k_2^2] 
    ≤    2 𝔼[∇ψ_x_t(θ_t^k)-∇ψ_x_t(θ_*) _2^2]
        +2 𝔼[∇ψ_x_t(θ̃_̃t̃)-∇ψ_x_t(θ_*)  -∇ℓ_CL^t(θ̃_t)_2^2] 
    
    =     2 𝔼[ ∇ψ_x_t(θ_t^k)-∇ψ_x_t(θ_*)_2^2]
       +2 𝔼 [∇ψ_x_t(θ̃_t)-∇ψ_x_t(θ_*) 
       -𝔼[∇ψ_x_t(θ̃_t)-∇ψ_x_t(θ_*)] _2^2] 
    ≤    2 𝔼[∇ψ_x_t(θ_t^k)-∇ψ_x_t(θ_*)_2^2]
       +2 𝔼[∇ψ_x_t(θ̃_t)-∇ψ_x_t(θ_*)_2^2] 
    ≤    2 L[𝔼[||θ_t^k-θ_*||_2^2]+𝔼[||θ̃_t-θ_*||_2^2]],

where the first inequality comes from the fact  a+b_2^2≤ 2a_2^2+2b_2^2  for any vectors a,b  and  Lemma <ref>,  the second inequality is based on the fact  𝔼[ξ-𝔼ξ_2^2]=𝔼[ξ_2^2]-𝔼 [ξ]_2^2≤𝔼[ξ_2^2]  for any random vector  ξ, and the third inequality is based on L-smooth.



 

    𝔼[θ_t^k-θ_*_2^2] ≤ (1-2ηγ+2Lη^2)𝔼[||θ_t^k-1-θ_*||_2^2] 
       + 2Lη^2𝔼[||θ̃_t-θ_*||_2^2].



Note that 𝔼 [v_t^k]= 𝔼 [∇ψ_x_t(θ_t^k)] =∇ℓ_CL^t(θ_t^k-1), and so we have

    𝔼[θ_t^k-θ_*_2^2] 
    
    =    θ_t^k-1-θ_*_2^2-2 η(θ_t^k-1-θ_*)^⊤𝔼 [v_t^k-1]+η^2𝔼[v_t^k-1_2^2] 
    ≤   θ_t^k-1-θ_*_2^2-2 η(θ_t^k-1-θ_*)^⊤∇ℓ_CL^t(θ_t^k-1)
        +2 L η^2[𝔼[||θ_t^k-1-θ_*||_2^2]+𝔼[||θ̃_t-θ_*||_2^2]] 
    ≤   θ_t^k-1-θ_*_2^2-2 ηγ ||θ_t^k-1-θ_*||_2^2
        +2 L η^2[𝔼[||θ_t^k-1-θ_*||_2^2]+𝔼[||θ̃_t-θ_*||_2^2]],

where the first inequality comes from  Lemma <ref>, and the second inequality is based on the γ-strong convexity of  ℓ_CL^t(θ). We take the expectation on θ_t^k-1 from the above inequality and then have

    𝔼[θ_t^k-θ_*_2^2] 
       ≤𝔼 [θ_t^k-1-θ_*_2^2]-2 ηγ𝔼[||θ_t^k-1-θ_*||_2^2] 
        +2 L η^2[𝔼[||θ_t^k-1-θ_*||_2^2]+𝔼[||θ̃_t-θ_*||_2^2]] 
        = (1-2ηγ+2Lη^2)𝔼[||θ_t^k-1-θ_*||_2^2] + 2Lη^2𝔼[||θ̃_t-θ_*||_2^2].






We now proceed to prove the theorem. To simplify notation, we denote 𝔼θ_t^k-θ_*_2^2 as p_k.

Based on lemma <ref>, we have

    p_k≤(1-2 ηγ+2 η^2 L) p_k-1+4 η^2 L  p_k-1.

By setting  η = γ/10L and m ≥10L^2/γ^2, we have

    p_m+1   ≤(1-γ^2/10 L^2) p_m≤(1-γ^2/10 L^2)^m p_1
       ≤exp(-γ^2/10 L^2 m) p_1≤1/2 p_1,

i.e

    𝔼[θ_t^m+1-θ_*_2^2] ≤1/2𝔼[θ_t^1-θ_*_2^2].

Subsequently, we consider the update during different stage s, so that the initial parameter θ̃_t,s is θ_t^1  and updated parameter θ̃_t,s+1 = θ_t^m+1  is selected after all of the updates have been completed. Then Eq (<ref>) becomes

    𝔼[θ̃_t,s+1-θ_*_2^2] ≤1/2𝔼[θ̃_t,s-θ_*_2^2].

Through recursively applying  the above equation from stage 1 to stage s+1, we have 

    𝔼[θ̃_t,s+1-θ_*_2^2] ≤1/2^s𝔼[θ̃_t,1-θ_*_2^2].





§ DETAILED DGC PROCEDURE IN TFCL

We consider the data streams {𝒯_1, 𝒯_2, ⋯ 𝒯_T} with each 𝒯_i being the batch training data in TFCL. Algorithm <ref> shows how to implement the DGC procedure in this model. 












§ DETAILS FOR EXPERIMENTAL SETUP


  Architecture and hyperparameter Most of the comparison methods in our experiments use the implementation of Mammoth [https://github.com/aimagelab/mammoth]. For the Dynamic ER <cit.> method we use the implementation of PyCIL [https://github.com/G-U-N/PyCIL], and for the GCR method we use the implementation of Google-research [https://github.com/google-research/google-research/tree/master/gradient_coresets_replay]. To be consistent with the selection of these open source frameworks, all the methods use ResNet18 <cit.> as the base network. All the networks are trained from scratch. 
For the hyperparameter selection of different methods, we directly use the original hyperparameters used in these open-source frameworks, which are obtained by using grid search on 10% of the training set.


  Training details 
  
  To maintain consistency, we utilize the default parameter settings and network architectures provided by the framework. As in previous studies <cit.>, random crops and horizontal flips are used as data augmentation in all experiments. All methods that do not incorporate DGC are optimized using standard SGD. In our DGC calibrated method, we empirically set m = 200. 
 
  To fairly compare the methods with constant storage limits, we uniformly train for 50 epochs in each task on S-CIFAR10 and S-CIFAR100, and 100 epochs in each task on S-TinyImageNet. The batch size is all set to  be 32. According to the experimental description in the Dynamic ER method, 
  
  we train the first task for 200 epochs on all datasets and train all subsequent tasks for 170 epochs;    the batch size is set to be 128.



§ OTHER EXPERIMENTAL RESULTS



  Selection of α We explore the impact of different α values to the performance of our DGC method. In the experiment, we fix the buffer size to 2000. Table <ref> shows that DGC can improve ER under different α values. 
  
  In all the experiments of <ref>, we fix the value of α to be 1e^-3.

  Selection of m As stated in Theorem <ref>, when m is sufficiently large, our DGC method has a linear convergence in expectation. The results in Table <ref>  show that when m is greater than 100, DGC-ER can significantly improve the performance of ER. Table <ref> also suggests that our method is relatively robust to the selection of m. 





  FF and FAA results in CIL 
  we use Final Forgetting (FF) <cit.> to measure the forgetting of the model throughout the learning process.
Following the previous work, it is defined as FF≜1/T-1∑_j=1^T-1(max _k ∈{1, …, T-1} a_k,j-a_T,j).
  Table <ref> and Table <ref> show that DGC based methods can reduce forgetting while improving the performance in all the cases.  For most instances, our DGC based methods achieve the best performance.






Detailed results in TFCL Table <ref> shows that our DGC method can improve the performance in TFCL. The DGC-DER++ method achieves the best performance in all the cases.

  Training Time We set b=500 and conduct experiments on S-CIFAR100 to measure the training time of several algorithms on the first three tasks in the CIL scenario. Table <ref> shows that our method and GCR have roughly the same training time, which is higher than AGEM but lower than other baselines. The results on the remaining tasks and other datasets are similar. Therefore, the increase caused by our method on training time is not significant, compared with most of the baselines.  






 Paper Title: Conformal Trajectory Prediction with Multi-View Data Integration in Cooperative Driving.


Beyond full statevector simulation with 
    Andrea Pasquale^1,2, Andrea Papaluca^3, Renato M. S. Farias^1,4, Matteo Robbiati^2,5, Edoardo Pedicillo^1,2,
and Stefano Carrazza^1,2,5
    August 5, 2024
===========================================================================================================================================



§ ABSTRACT 
Current research on trajectory prediction primarily relies on data collected by onboard sensors of an ego vehicle. With the rapid advancement in connected technologies, such as vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication, valuable information from alternate views becomes accessible via wireless networks. The integration of information from alternative views has the potential to overcome the inherent limitations associated with a single viewpoint, such as occlusions and limited field of view. In this work, we introduce V2INet, a novel trajectory prediction framework designed to model multi-view data by extending existing single-view models. Unlike previous approaches where the multi-view data is manually fused or formulated as a separate training stage, our model supports end-to-end training, enhancing both flexibility and performance. Moreover, the predicted multimodal trajectories are calibrated by a post-hoc conformal prediction module to get valid and efficient confidence regions. We evaluated the entire framework using the real-world V2I dataset V2X-Seq. Our results demonstrate superior performance in terms of Final Displacement Error (FDE) and Miss Rate (MR) using a single GPU. The code is publicly available at: <https://github.com/xichennn/V2I_trajectory_prediction>.









§ INTRODUCTION



Trajectory prediction plays a critical role in autonomous driving. Typically, we rely on the on-board sensors of an ego vehicle to gather environmental information necessary for performing various autonomous driving tasks. However, with the rapid advancement in connected technologies, such as vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication, valuable information from alternate views becomes accessible via wireless networks. The integration of information from alternative views has the potential to overcome the inherent limitations associated with a single viewpoint, such as occlusions and limited field of view. We categorize the data obtained solely from ego vehicle on-board sensors or infrastructure as single-view information, whereas the accessibility to multiple viewpoints is referred to as multi-view data. Figure <ref> depicts an intersection scenario where the AV faces a potential left-turn collision due to its field of view being obstructed by a large truck. Roadside cameras, strategically positioned to have an unobstructed view of the entire intersection, provide a comprehensive overview of the traffic situation. This enhanced perspective allows the AV to receive critical information and effectively avoid the impending collision.

From a trajectory predictive modeling perspective, significant research efforts have been dedicated to leveraging single-view datasets collected by the ego vehicle <cit.>. These efforts typically involve modeling the temporal dependencies, agent-agent interactions and agent-lane relations. However, the challenge arises when dealing with multi-view data, particularly in how to effectively fuse the information due to overlapping field of views.

Existing work on multi-view data fusion in the context of cooperative driving predominantly focuses on collaborative perception tasks, with limited research addressing trajectory prediction. <cit.> pioneered the creation of the first V2I real-world dataset for trajectory prediction studies. They manually fused multi-view data using trajectory association and stitching techniques at each frame, followed by the application of single-view trajectory prediction models. Although this approach is intuitive, it fails to fully exploit the motion behavior captured by data from each view, resulting in suboptimal outcomes. To address this limitation, <cit.> proposed a novel approach that encodes trajectory information from each view as independent graph nodes, thereby minimizing information loss. They formulated the node association problem as a graph link prediction task and introduced a cross-attention module to fuse node embeddings from associated nodes across different views. While their model can be trained end-to-end, the node association process necessitates pretraining, leading to the formulation of two optimization objectives.

We introduce a trajectory prediction framework that utilizes multi-view data without the need for explicit association between different perspectives. Rather than developing a specialized multi-view model, our approach seamlessly integrates with state-of-the-art single-view trajectory models, maximizing the utility of existing research efforts. No special training strategies are required. We can easily leverage the pretrained single-view trajectory models to expediate the training. Specifically, for trajectory data from each single-view, we employ established graph neural network (GNN) based models, such as LaneGCN <cit.>, HiVT <cit.>, to capture temporal dependencies, agent-agent interactions, and agent-lane relations. Then we utilize a cross-graph attention module to fuse the node embeddings from different views. The fused final embeddings will then go through a multimodal decoder to get future trajectory predictions. 

Existing works have modeled the multi-modality explicitly by introducing anchors <cit.>, mixture models <cit.>, or implicitly through latent variables such as Conditional Variational Auto-encoder (CVAE) or generative models <cit.>. The implicit models often face the issue of mode collapse, therefore we will model the multi-modality by a mixture model built upon MLP. It is a common strategy to assign a higher score to the modality closer to the ground truth during training. This strategy, however, may encounter robustness issues during inference. Figure <ref> presents two examples where the top-scored prediction is not the closest to the ground truth trajectory.



 This arises from the inherent randomness in the ground truth data, posing challenges for the model to learn the priorities of multiple predictions. Relying on the scores to rank the predictions can lead to hazardous results in safety-critical scenarios. Additionally, while most mixture models like GMM provide both point estimates and variance estimation, the reliability of these uncertainty quantification (UQ) remains uncertain. Unfortunately, this aspect is often overlooked in the model evaluation process. However, UQ can offer valuable insights for downstream decision-making processes. To address these challenges, we introduce conformal prediction (CP) <cit.>, as a post-hoc module to our framework. CP creates statistically rigorous uncertainty intervals for the predictions that are guaranteed to contain the ground truth with a user-specified probability. 

To summarize, our contributions in this work lie in twofolds:
1) We introduce V2INet, a novel trajectory prediction framework designed to model multi-view data by extending existing single-view models. Unlike previous work, our model supports end-to-end training, enhancing both flexibility and performance.
2) We introduce conformal prediction as a post-hoc module to calibrate the prediction results. This results in statistically rigorous uncertainty intervals, significantly enhancing the reliability of the predictions. These calibrated predictions are particularly beneficial for downstream decision-making tasks, such as motion planning.



§ RELATED WORK



 §.§ Cooperative Driving Dataset

Collaborative perception has been the most extensively studied area in collaborative driving <cit.>. It leverages the wireless communication technologies, such as V2V and V2I, to share information, which enables perception beyond line-of-sight and field-of-view hence overcoming common perceptual shortcomings with individual perception, such as blind spot, occlusions and long-range issues <cit.>. Many datasets have been collected in both simulation environment <cit.> and real-world <cit.> to facilitate the study. A comprehensive comparison can be found in Table <ref>. While majority of them focused on the upstream tasks such as detection, tracking and segmentation, to enable well-informed decision-making for autonomous vehicles, however, it's critical to also incorporate V2X data for predicting the behavior of surrounding traffic participants. V2X-Seq/Forecasting <cit.> and V2X-Traj <cit.> present two real-world datasets specifically designed for cooperative trajectory prediction tasks. The experiments in V2X-Seq/Forecasting demonstrate that leveraging the infrastructure-side trajectories can enhance the trajectory prediction performance. V2X-Traj aims for more general V2X scenarios, extending to V2V cooperation.




 §.§ Cooperative Information Association and Fusion







The main challenge in cooperative trajectory prediction lies in the multi-view data source fusion. In cooperative scenarios, vehicles gather safety-related data using sensors like radar, lidar, cameras, and GPS. This data is standardized into Basic Safety Message (BSM) format, ensuring compatibility across vehicles and infrastructure <cit.>. BSM messages, containing crucial information such as position and speed, are broadcasted periodically. Upon receiving BSM messages, vehicles combine this data with their own sensor data to enhance accuracy. Advanced algorithms have been developed to associate and fuse the multi-source data. Given the temporal and spatial dimensions of the collected trajectories, there has been research focusing on the communication delays alignment <cit.> and pose errors alignment <cit.>. Based on the aligned data, <cit.> utilized CBMOT <cit.>, a multi-object tracking method, to fuse the infrastructure and ego-vehicle trajectories at each single frame and then trained the network taking in the fused dataset. While straightforward, this method fails to capture the motion behavior provided by the infrastructure across all time steps, leading to suboptimal results. In contrast, <cit.> encoded trajectory information from each view as independent graph nodes, formulating the association process as a graph linking problem. While effective, this approach necessitates separate training procedures. Building upon their work, we propose the utilization of a cross-graph attention mechanism to fuse multi-view information, eliminating the need for additional training processes.


 §.§ Uncertainty Quantification

In the trajectory prediction task, numerous sources of uncertainty exist, including inherent multi-modality, partial observability, short time scales, data limitations, intention type imbalances, and domain gaps. Most existing trajectory prediction models address uncertainty by maximizing the likelihood of an assumed distribution, such as Gaussian or Laplace <cit.>. However, trajectories with the largest likelihood are often nonsensical <cit.>. Alternatively, some research focuses on approximating Bayesian inference for deep learning models using techniques like Monte Carlo dropout <cit.>, which involves performing stochastic forward passes through the network and averaging the results. Among these approaches, <cit.> stands out as the only work that incorporates collaborative uncertainty among agents into the modeling process to guide the ranking of multimodal trajectories by uncertainty, albeit requiring special training strategies. However, none of the predicted uncertainties from these methods offer finite sample coverage guarantees, which is suboptimal for safety-critical applications such as vehicle trajectory prediction.

Conformal prediction (CP) <cit.> has emerged as a widely adopted uncertainty quantification method, owing to its simplicity, generality, theoretical rigor, and low computational overhead. Notably, CP is agnostic to the underlying model and data distribution, making it highly versatile. It seamlessly integrates with any pre-trained model to deliver statistically valid prediction regions. Of particular relevance to our multimodal trajectory prediction task is recent progress in generalizing CP to time-series forecasting. For instance, <cit.> introduced the Copula conformal prediction algorithm for multivariate, multi-step time series forecasting, applicable to any multivariate multi-step forecaster. Additionally, <cit.> focused on generating non-conformity score functions that yield multimodal prediction regions with minimal volume. Moreover, <cit.> employed CP to generate statistical uncertainty intervals from Gaussian mixture model outputs, obtaining separate prediction intervals corresponding to each GMM component prediction. Furthermore, <cit.> proved the validity of CP on graph data. Inspired by their work, we explore the potential of applying CP to the multimodal trajectory prediction comparing different CP methods.



§ PROBLEM FORMULATION


At the scenario level, We have trajectory data 𝒯 from both the vehicle and infrastructure viewpoints, denoted as 𝒯^𝒱 and 𝒯^ℐ, respectively. While 𝒯^𝒱 and 𝒯^ℐ share overlapping information where their fields of view intersect, 𝒯^ℐ also provides complementary information, being free from occlusions. Our modeling objective is to utilize the information from 𝒯^ℐ to improve the accuracy of trajectory prediction based solely on 𝒯^𝒱. 

The trajectory prediction task involves leveraging historical trajectories 𝒯^𝒱_h ∈ℝ^N^𝒱× T_h × a_h and 𝒯^ℐ_h ∈ℝ^N^ℐ× T_h × a_h, alongside contextual information, typically HD maps denoted as ℳ, to forecast future trajectories 𝒯^𝒱_f ∈ℝ^N^𝒱× T_f × a_f. Here, N^𝒱 and N^ℐ represent the number of observed actors from the vehicle and infrastructure perspectives, respectively. T_h denotes the historical time horizon, and T_f is the prediction horizon. a_h and a_f represent the number of node features which we consider the vehicle center location defined by its x- and y-coordinates. Notably, the trajectory data from both views are defined within the same coordinate system. For the HD map, we opt for a vectorized representation due to its lightweight nature and efficiency <cit.>. This vector map is depicted by lane centerlines, which are composed of lane segments. We denote it as ℳ∈ℝ^N^l × a_l, where N^l is the number of lane segments and a_l is the number of lane attribute.

We formulate the overall probabilistic distribution as P(𝒯^𝒱_f | 𝒯^𝒱_h, 𝒯^ℐ_h, ℳ^𝒱, ℳ^ℐ). Driven by the critical safety demands inherent in trajectory prediction, we aim to incorporate uncertainty quantification to preempt any potentially consequential model failures. Let 𝒟 be the set of scenarios of the form (𝒯^𝒱_h, 𝒯^ℐ_h, ℳ^𝒱, ℳ^ℐ, 𝒯^𝒱_f). We split the dataset into training 𝒟_train, validation 𝒟_val, calibration 𝒟_cal and test 𝒟_test. One black-box deep learning model is trained on 𝒟_train and evaluated on 𝒟_val. We achieve the uncertainty quantification by conformal prediction in a post-hoc way on 𝒟_cal.  

At the agent level, we denote the features and labels of agent i from the vehicle view as X^𝒱 = 𝒯^𝒱_h,i and Y^𝒱 = 𝒯^𝒱_f,i for brevity. In real-world scenarios, future trajectories may exhibit multimodal behavior, often approximated by mixture models, resulting in Ŷ^𝒱∈ℝ^K × T_f × a_f, where K represents the number of mixtures or modes. Given a new agent sample X^𝒱_test from 𝒟_test, we seek to construct the prediction intervals 𝒞(X^𝒱_test) ∈ℝ^K × T_f × a_f × 2 such that it covers the ground truth label Y^𝒱_test under a predefined coverage rate leveraging conformal prediction.

CP proceeds in three steps. First, we define a nonconformity score A: 𝒳×𝒴∈ℝ^K × T_f × a_f to quantify how well Y conforms to the prediction at X. Typically, we choose a metric of disagreement between the prediction and the ground truth as the non-conformity score, such as the Euclidean distance. Second, given the predefined miscoverage rate α, we compute the 1-α quantile of the non-conformity scores on the calibration set {A(X_1, Y_1),⋯,A(X_n, Y_n)}, where n is the number of agents. The resulting quantile is denoted as Ĥ∈ℝ^T_f × a_f. Last, when presented with a new test agent X^𝒱_test, CP constructs the prediction interval 𝒞(X^𝒱_test)={Y^𝒱∈𝒴: A(X^𝒱_test, Y^𝒱) ≤Ĥ}.




§ METHODOLOGY



 §.§ Overview

Our method V2INet consists of two key components, predictive modeling and post-hoc conformal prediction. An overview of our proposed model is illustrated in Figure <ref>. 
We first represent the scenario data collected from both views as graphs. A single view encoder is then applied separately to each graph, encoding various information such as agent-agent interactions, temporal dependencies, and agent-lane information. Subsequently, The vehicle-view embedding is fused with the infrastructure-view embedding through a cross-graph attention module. Finally, the updated vehicle-view embedding pass through a multi-modal decoder, providing multimodal predictions for all the agents of interest. The post-hoc conformal prediction is then applied at the agent level to construct valid prediction intervals given a predefined coverage rate.




 §.§ Scene Representation

We adopt an ego-centric coordinate system utilizing vectorized representation as first introduced in <cit.>. First, data from both views are transformed such that the ego vehicle is centered at the origin, with its heading aligned along the positive x-axis. Each trajectory is then characterized as a sequence of displacements  {Δ𝐱^t}^0_t=-(T_h-1), where Δ𝐱^t∈ℝ^2 is the 2D displacement from time step t-1 to t. Similarly, each lane segment is represented as Δ𝐱^l ∈ℝ^2, which captures the 2D displacement from the starting coordinate to the end coordinate of lane segment l. We then construct scenario graphs consisting of actor nodes and lane nodes for both views separately. The edge attribute is the absolute relative position between two nodes.



 §.§ Single View Encoder

To encode the spatiotemporal information, including agent-agent interactions, temporal dependencies, and context information captured by agent-lane relations for data from each view, existing graph-based models offer effective solutions. Models like LaneGCN <cit.>, HiVT <cit.>, and VectorNet <cit.> incorporate these components and can be readily employed. This encoding step can be performed efficiently at the edge device, such as the vehicle's on-board computers and the roadside unit (RSU) for infrastructure data, before broadcasting, thereby minimizing computational overhead.

We exemplify here with HiVT <cit.> where only attention mechanism is employed, with the adoption of a rotation-invariant representation. To attend to the local information, at each timestamp, the surrounding actors' information is aggregated and the embeddings at each time stamp subsequently go through a transformer to capture the temporal dependencies. After obtaining the spatio-temporal embeddings, the surrounding lane information is aggregated for each actor. The update actor node embeddings is then passed to a global interaction module which aggregates the local context of different actors and updates each actor’s representation to capture long-range dependencies and scene-level dynamics. Here's the proofread version:

We denote the final embedding for actor node i from the vehicle view as 𝐡_i, and actor node j from the infrastructure view as 𝐡_j. Both 𝐡_i and 𝐡_j are in ℝ^d_h, where d_h is the embedding dimension.



 §.§ Cross-graph Attention Module

To effectively utilize information from the infrastructure side, we employ a cross-graph attention module that aggregates information captured by the infrastructure view encoder. Specifically, the embedding from the vehicle view 𝐡_i is transformed into the query vector, while the embedding from the infrastructure view 𝐡_j alongside the relative position at the last observed time step 𝐱_i^t=0 - 𝐱_j^t=0, are utilized to compute the key and value vectors. We denote 𝐡_ij=(𝐡_j, 𝐱_i^t=0 - 𝐱_j^t=0) representing the concatenation of node and edge attribute from infrastructure node j to vehicle node i:

    𝐪_i=𝐖^Q_f𝐡_i,    𝐤_ij=𝐖^K_f𝐡_ij,   𝐯_ij=𝐖^V_f𝐡_ij

where 𝐖^Q_f, 𝐖^K_f, 𝐖^V_f∈ R^d_k × d_h are learnable matrices for linear projection and d_k is the transformed dimension. The resulting query, key and value vectors are then taken as input to the scaled dot-product attention block:

     
           α_ij = softmax(𝐪_i^T/√(d_k) ·[{𝐤_ij}_j ∈𝒩_i]),

           𝐦_i = ∑_j ∈𝒩_iα_ij𝐯_ij,

           𝐠_i = sigmoid(𝐖^gate[𝐡_i,𝐦_i]),

           𝐡̃_i = 𝐠_i ⊙𝐖^self𝐡_i+(1-𝐠_i)⊙𝐦_i
    

where 𝒩_i is the set of agent i's neighbors, 𝐖^gate and 𝐖^self are learnable matrices and ⊙ denotes element-wise product. We followed the structure in HiVT to fuse the infrastructure information with a gating function. The attention block supports multiple heads. Finally, we apply another MLP block to obtain the final fused embedding 𝐡̃_i ∈ℝ^d_h for agent i from the vehicle view. 


 §.§ Mixture Model Based Decoder and Learning

There are two widely used mixture models for describing the multimodal trajectories, Gaussian and Laplacian. Previous methods <cit.> have found that the ℓ_1-based loss function derived from the Laplace distribution usually leads to superior prediction performances, as it is more robust to outliers. Hence, we will parameterize the future trajectories following Laplace distribution. For each agent, the decoder receives the final embedding as inputs and outputs K possible future trajectories and the mixing coefficient of the mixture model for each agent. The decoder are consisted of three MLPs, one for predicting the future locations μ^t_i,k∈ℝ^2 for agent i and its mode k at each time step t, one for predicting the associated uncertainty 𝐛^t_i,k∈ℝ^2 assuming independence of the x- and y-coordinates, the last one followed by a softmax is for producing the scores for each mode. 

To ensure the prediction diversity <cit.>, instead of optimizing all the predicted trajectories, only the mode k̃ closest to the ground truth is optimized. The closeness here is defined as the average Euclidean distance between ground-truth locations and predicted locations across all future time steps. The Loss includes both regression loss and classification loss

    
    J = J_reg + ε J_cls


Here, ε is the weight of the classification loss. We employ the negative log-likelihood as the regression loss:

    
    J_reg = -1/nT_f∑^n_i=1∑^T_f_t=1log𝐏(𝐲_i^t - 𝐲̂_i^t | μ̂^t,k̃_i,𝐛̂^t,k̃_i)



where 𝐏(|) is the probability density function of Laplace distribution and μ̂_i^t,k̃, 𝐛̂_i^t,k̃ are the mean and uncertainty estimates of the best mode k̃.

For J_cls, cross-entropy loss is applied:

    
    J_cls = -1/n∑^n_i=1∑^K_k=11_k logπ_k


where 1_k is a binary indicator if k is the best mode, π_k is the mixing coefficient of mode k. The model is trained on 𝒟_train and evaluated on 𝒟_val.



 §.§ Post-hoc uncertainty quantification module

Uncertainty quantification methods are evaluated on two key properties: validity and efficiency. Validity is established when the predicted confidence level exceeds or equals the probability of events falling within the predicted range, while efficiency refers to minimizing the size of the confidence region. We utilize conformal prediction to obtain both valid and efficient prediction intervals. The standard conformal prediction methods typically operate with scalar point estimates for regression problems. However, since our output consists of a multimodal multivariate time-series, we need to make certain adaptations to accommodate this complexity. Following the steps outlined in Section <ref>, we proceed with non-conformity score function definition, quantile computation and prediction interval construction.


  §.§.§ Non-conformity Score Functions

As emphasized in <cit.>, the usefulness of the prediction sets is primarily determined by the score function, we adopt three score functions A and compare their performance.

Z-score. As the decoder returns both mean and variance predictions, we define the Z-score function as:

    Z = |Y -Ŷ|/B̂


where Z ∈ℝ^K × T_f × 2 and B̂ = {{b̂^̂t̂,̂k̂}_t=0^T_f}_k=0^K. Here, we compute scores separately for x- and y-coordinates, reflecting the observation that motion uncertainty varies significantly across different dimensions.

L2-norm. Next, we consider the Euclidean distance, the most commonly used metric in regression problems:

    L_2 = ||Y -Ŷ||_2

where L_2 ∈ℝ^K × T_f.

L1-norm. Recognizing that the L2-norm disregards dimension differences, we consider L1-norm:

    L_1 = ||Y -Ŷ||_1

where L_1 ∈ℝ^K × T_f × 2.



  §.§.§ Quantile Computation

Given a predefined miscoverage rate α∈ [0,1], the 1-α quantile of the non-conformity scores is calculated on the calibration set 𝒟_cal. Conventionally, the quantile is determined as follows: Ĥ=quantile({A(X_1, Y_1),⋯,A(X_n, Y_n)}, (1-α)(1+1/n)) where A(X_i, Y_i) is a scalar, and n denotes the total number of agents in 𝒟_cal. However, in our case, we deal with multimodal time-series scores, which are multivariate when using Z-score and L1-norm, hence necessary adaptations are needed. For brevity, let Γ_i = A(X_i, Y_i), with Γ_i ∈ℝ^K × T_f × 2 for Z-score and L1-norm, and ℝ^K × T_f for L2-norm.

To enhance the efficiency of our prediction intervals, we focus on computing the quantile using only the mode k̃ that exhibits the smallest average Euclidean distance to the ground truth trajectory. This reduces the computation to multivariate time-series. Within this framework, we investigate two established methods: CF-RNN <cit.> and CopulaCPTS <cit.>.

CF-RNN. Since the time-series predictions are obtained from the same embedding, this work proposes the application of Bonferroni correction to the calibration scores to maintain the desired miscoverage rate α. Specifically, the original α is divided by T_f, yielding Ĥ=quantile({Γ_i}_i=0^n, (1-α/T_f)(1+1/n)) for single-variate case.

CopulaCPTS. As implied by its name, this method models the joint probability of uncertainty for multiple predicted time steps using a copula. The calibration set 𝒟_cal is split into two subsets: 𝒟_cal-1, which estimates a Cumulative Distribution Function (CDF) for the nonconformity score of each time step, and 𝒟_cal-2, utilized to calibrate the copula. The copula function captures the dependency between time steps and can enhance the efficiency of prediction intervals.

For multivariate scores generated by the L1-norm and Z-score functions, we adopt the Bonferroni correction for each dimension, as inspired by CF-RNN. Specifically, we use α/2 as the miscoverage rate for both x- and y-coordinates.



  §.§.§ Prediction Interval Construction

We utilize the obtained quantile Ĥ to form the prediction intervals for new examples in 𝒟_test:

    𝒞(X^𝒱_test)={Y^𝒱∈𝒴: A(X^𝒱_test, Y^𝒱) ≤Ĥ}

Specifically, for Ĥ obtained from the Z-score, we have:

    𝒞(X^𝒱_test)=[Ŷ^𝒱_test - B̂Ĥ, Ŷ^𝒱_test + B̂Ĥ]

and for Ĥ obtained from the L2-norm and L1-norm, we have 

    𝒞(X^𝒱_test)=[Ŷ^𝒱_test - Ĥ, Ŷ^𝒱_test + Ĥ]



§ EXPERIMENTS



 §.§ Experimental Setup

In this section, We introduce the specifics of the dataset, the evaluation metrics and the implementation details including hardware, hyperparameters, etc.


  §.§.§ Dataset

We evaluate the proposed model on the publicly available large-scale and real-world V2I dataset V2X-Seq <cit.>, which provides the trajectories of agents from both vehicle and infrastructure sides, along with vector map data. V2X-Seq consists of 51,146 V2I scenarios, where each trajectory is 10 seconds long with a sampling rate of 10 Hz. The task involves predicting the motion of agents in the next 5 seconds, given initial 5-second observations from both infrastructure and vehicle sides. The dataset has been split into train and validation. Our trained model is evaluated on the validation set, allowing for comparison with existing models. For post-hoc conformal prediction, we divide the validation set into calibration set and test set at a 4:1 ratio. For a discussion on the calibration data size, please refer to <cit.>.



  §.§.§ Evaluation Metrics
 
Model metrics. For model evaluation, the standard metrics in motion predictions are adopted, including minimum Average Displacement Error (minADE), minimum Final Displacement Error (minFDE), and Miss Rate (MR), where errors between the best predicted trajectory among the K=6 modes and the ground truth trajectory are calculated. The best here refers to the trajectory that has the minimum endpoint error. The ADE metric calculates the L2 distance across all future time steps and averages over all scored vehicles within a scenario, while FDE measures the L2 distance only at the final future time step and summarizes across all scored vehicles. MR refers to the ratio of actors in a scenario where FDE are above 2 meters. 

Conformal prediction metrics. We assess validity and efficiency for each method. Validity is evaluated by reporting independent and joint coverage on the test set, aiming for coverage levels close to the desired confidence level 1-α. The independent coverage for each agent is calculated as:

    ind. coverage_1-α = 1/T_f∑_X,Y ∈𝒟_test1(Y ∈𝒞(X))

We identify the maximum independent coverage among the K modes for each agent. For all metrics, the final reported value is the average among all agents in 𝒟_test.

For efficiency, we calculate the average size of the predicted 2D area across all time steps for the mode k̃ with the maximum independent coverage:

    size = 1/T_f||𝒞(X)_k̃||

The 2D area is defined as ellipsis for Z-score and L1-norm, and circle for L2-norm.

The joint coverage is defined as if there exists a mode k such that the truth values across all time steps fall into the confidence region:

    joint coverage_1-α = 1(∃k∈ K: ∀t∈ T_f: Y_k,t∈𝒞(X))




  §.§.§ Implementation Details

Model. The model was trained for 64 epochs on an Nvidia V100S GPU with 32GB memory using AdamW optimizer <cit.>. Hyperparameters including batch size, initial learning rate, weight decay and dropout rate are 32, 1e-3, 1e-4 and 0.1, respectively. The learning rate is decayed using the cosine annealing scheduler <cit.>. Our model employs the original setting of HiVT with 64 hidden units for the single view encoder and 1 layer of cross-graph attention module with 8 heads. The radius of all local regions is 50 meters. The number of prediction modes is set to 6.

Conformal prediction. With the pretrained model and datasets 𝒟_cal and 𝒟_test, we execute all conformal prediction methods on the CPU. We evaluate the methods on the three defined score functions at three different α levels: 0.2, 0.1, 0.05. The optimization step in CopulaCPTS remains consistent with the original work.



 §.§ Results Analysis

In this section, we first examine the model's prediction performance, assessing it from both quantitative and qualitative perspectives. Next, we showcase the effectiveness of the post-hoc uncertainty quantification method.


  §.§.§ Model Performance

Comparison with state-of-the-art. We benchmark our proposed model against state-of-the-art models using the V2X-Seq dataset, as detailed in <cit.>. Results are summarized in Table <ref>. Both TNT <cit.> and HiVT <cit.> are single-view models. <cit.> evaluated them under two settings: Ego, where only the vehicle-view data is utilized, and PP-VIC, which employs a two-stage method with both vehicle view and infrastructure data. Specifically, in PP-VIC, data from both views are fused offline with some tracking method and then the stitched trajectories were fed into a single-view model. We retrained the HiVT model under both Ego and the PP-VIC setting with 64 epochs and present our results in Table <ref>. Comparing the Ego and PP-VIC results, it's uncovered by <cit.> that integrating information from the infrastructure side can enhance prediction accuracy. V2X-Graph represents the current state-of-the-art model, employing a graph link prediction module to associate two-view data, followed by fusing the embeddings of the associated nodes using attention mechanism. However, this node association module requires pre-training, leading to a two-stage training process.  Our method demonstrates the best performance in terms of minFDE and MR. Without the explicit node association from both views, our attention based fusion module can attend to the most relevant nodes from both views through learning, which significantly simplifies the modeling framework and facilitates ease of training, all while achieving better results.  


Qualitative results. We visualize our prediction results and six representative scenarios are shown in Figure <ref>. To maintain clarity and simplicity in the visualization, each scenario displays the ground truth and multimodal prediction results only for the target agent, although predictions for all agents are accessible. In the first column, scenarios S1 and S4 depict the target agent turning right. In the second column, the target agent is shown going straight, leaving (S2) and approaching (S5) an intersection. The last column presents scenarios where the target agent is merging (S3) and turning left (S6). It's shown that our model predictions capture the multimodal behavior. In scenarios S2 and S5, it's exhibited in the form of different velocity profiles. Since lane information is integrated via an attention mechanism rather than as rigid constraints, off-road and road rule-violating predictions and may occur, as shown in S4 and S6. Notably, there are more uncertainties when agent making turns, as evidenced by the dispersion among the predictions. Furthermore, it's important to highlight that the prediction with the highest probability does not always align with the ground truth, as demonstrated in S1 and S3, resulting from the inherent model and data uncertainties. 



We further compare the prediction results from model HiVT Ego, HiVT PP-VIC and ours on selected scenarios, as shown in Figure <ref>. It's observed that HiVT Ego, relying solely on vehicle-view data, consistently predicted incorrect modes. In contrast, both HiVT PP-VIC and our model, incorporating additional data from the infrastructure view, demonstrate improved accuracy in capturing the ground truth.





  §.§.§ Post-hoc Uncertainty Quantification Performance

Quantitative comparison. We show in this section that conformal prediction method produces more valid and efficient confidence regions than the model predictions. The evaluation results are shown in Table <ref>.


The Mixture column presents UQ results derived directly from the multimodal predictions. We first examine the metrics of independent and joint coverage. It's evident that the model predicted intervals fail to meet the validity requirements across all specified miscoverage rates. Interestingly, CF-RNN yields overly conservative results, maintaining coverage levels around 0.99 regardless of the miscoverage rate. This could indicate that Bonferroni correction, often employed with shorter horizons, might not be suitable for our predicted horizon of 50. CopulaCPTS achieves coverage levels very close to the desired values across all specified miscoverage rates α.

Next, we inspect the metric "size", which serves as an indicator of UQ efficiency. Benchmark the size from model predictions in the Mixture column, CF-RNN produces extremely large confidence region which is expected considering the overly conservative coverage observed earlier. CopulaCPTS yields comparative and even smaller size compared to the benchmark. 

Finally, we examine the impact of different score functions. In both CF-RNN and CopulaCPTS, Z-score appears as the least efficient among all three functions. Since Z-score is computed using the model-predicted uncertainties B̂, it will always be more inefficient than the benchmark size in the "Mixture" column. One plausible explanation provided in <cit.>, is that although widely used, B̂ in the Z-score function is not directly related to the quantiles of the label distribution. Therefore, it may not be the most suitable score function option in our case. On the other hand, both L2-norm and L1-norm demonstrate significantly higher efficiency, particularly within the CopulaCPTS method. L1-norm, in particular, considers variations among different dimensions, leading to enhanced efficiency. Taking into account both validity and efficiency requirements, we have highlighted the best results among all methods in red.

Qualitative analysis. We visualize the uncertainty prediction from the mixture model and the CopulaCPTS, as shown in Figure <ref>. Three scenarios are selected in each column. We can see that the mixture model predictions fail to cover the ground truth while CopulaCPTS achieves the coverage at the specified α. 
 





§ CONCLUSION

We have presented a novel model framework with multi-view data integration in the cooperative driving setting. Our proposed model is straight forward and can be built upon any existing graph-based single-view models. It has demonstrated its effectiveness and advantages over existing benchmarks. Moreover, we have incorporated a post-hoc uncertainty quantification module, providing valid and efficient confidence regions, which is crucial in safety-critical tasks such as trajectory prediction. 

The proposed framework has certain limitations. From the model's perspective, we currently treat all road agents as the same type. However, in the public V2X-Seq dataset, there are different vehicle types, such as trucks, vans, buses, motorcycles, etc. Future work should address the different characteristics of these vehicle types to enhance model performance. Moreover, better methods for encoding the lane information to eliminate off-road and road-rule-violating predictions should be investigated. From the uncertainty quantification perspective, we simplify the quantile computation on the multimodal prediction into computation on the single best mode. This approach loses valuable distributional information. Therefore, exploring score functions that consider the distribution could lead to more accurate uncertainty quantification for multimodal results. Furthermore, we evaluate the uncertainty assuming independence among the agents. Future work can incorporate agent correlations based on the graph structure to better reflect the underlying uncertainty relations and provide more insightful results.

unsrt  
  




 Paper Title: A new approach for encoding code and assisting code understanding.












A new approach for encoding code  and assisting code understanding
    
Mengdan Fan
Peking University 

Bejing, China 

fanmengdan@pku.edu.cn
Wei Zhang*
Peking University 

Bejing, China 

zhangw@sei.pku.edu.cn
Haiyan Zhao
Peking University 

Bejing, China 

zhhy.sei@pku.edu.cn
Zhi Jin*
Peking University 

Beijing, China 

zhijin@pku.edu.cn

    August 5, 2024
====================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Some companies (e.g., Microsoft Research and Google DeepMind) have discovered some of the limitations of GPTs' autoregressive paradigm next-word prediction, manifested in the model's lack of planning, working memory, backtracking, and reasoning skills. 
GPTs rely on a local and greedy process of generating the next word, without a global understanding of the task or the output.
We have confirmed the above limitations through specialised empirical studies of code comprehension. 
Although GPT-4 is good at producing fluent and coherent text, it cannot handle complex logic and generate new code that haven’t been seen, and it relies too much on the formatting of the prompt to generate the correct code.
We propose a new paradigm for code understanding that goes beyond the next-word prediction paradigm, inspired by the successful application of diffusion techniques to image generation (Dalle2, Sora) and protein structure generation (AlphaFold3), which have no autoregressive constraints.
Instead of encoding the code in a form that mimics natural language, we encode the code as a heterogeneous image paradigm with a memory of global information that mimics both images and protein structures.
We then refer to Sora's CLIP upstream text-to-image encoder model to design a text-to-code encoder model that can be applied to various downstream code understanding tasks.
The model learns the global understanding of code under the new paradigm heterogeneous image, connects the encoding space of text and code, and encodes the input of text into the vector of code most similar to it.
Using self-supervised comparative learning on 456,360 text-code pairs, the model achieved a zero-shot prediction of new data. 
This work is the basis for future work on code generation using diffusion techniques under a new paradigm to avoid autoregressive limitations.



next-word prediction, CLIP, global information





§ INTRODUCTION


Some studies have found that there are several limitations of the GPT's autoregressive next-word prediction paradigm.
Microsoft Research notes that GPT-4 lacks planning, working memory, ability to backtrack, and reasoning abilities <cit.>.
For example, although the model has enough knowledge to answer the question, the architecture of the GPT-4 cannot give the whole correct answer at once, but needs to be guided step by step to give the correct answer.  
The autoregressive nature of GPT-4 forces it to solve problems sequentially.
Google Deepmind notes that GPTs are not applicable to most areas of mathematics due to the high cost of converting human proofs into a machine-verifiable autoregressive format <cit.>.
And permuting the premise order can cause a performance drop of over 30 % <cit.>.
When producing full natural-language proofs on a set of geometry problems from the IMO competitions, GPT-4 has a success rate of only 0 %, often making syntactic and semantic bugs throughout its output, and showing little understanding of the geometry knowledge and the problem statements themselves.
In summary, GPTs rely on a local and greedy process to generate the next word, without a global understanding of the task or the output <cit.>.


We further do empirical studies in the field of code understanding and confirm the limitations of the autoregressive paradigm.
For example, in terms of code understanding and generation, GPT-4 cannot understand complex logic with multi-step operations and generates incomplete code.
Moreover, GPT-4 relies excessively on the form of the prompt. Even if the input text has the same meaning, but just different syntax, the generated code will be much different and often contain bugs.
GPT-4 is also not good at matrix operation. 
It often introduces new bugs while fixing matrix operation's bugs.
But after giving the prompt step by step, GPT-4 can generate a correct answer, which shows that the model is actually trained with enough knowledge, but the autoregressive paradigm makes it not answer the question well.
All of these issues confirm the limitations of GPT-4 noted by Microsoft and Google. 
The limitations makes it overly dependent on the given prompt, unable to understand the meaning of the code globally, and also unable to generate complete and correct code including multi-step operation/matrix operation at one time.


Diffusion technology has made significant progress in the generation of images <cit.> and life molecules <cit.> recently. It is a wonder that the model does not have limitations like that of GPTs. The model can learn the global information of the picture/life molecule, and generate the picture/life molecule at one time instead of step by step.
Diffusion is a class of latent variable models inspired by considerations from non-equilibrium thermodynamics, which was originally used to generate high-quality images <cit.>.
In particular, the CLIP+diffusion model (Dalle2) can create new pictures that have never been seen in the data set according to the input text, such as panda mad scientist mixing sparkling chemicals, artstation <cit.>.
The model seems to really understand the image.
Where CLIP is a transferable visual models from natural language supervision, which provides text embedding vector for diffusion model to generate high-quality images <cit.>.
The core idea of CLIP is to embed images and texts into a shared semantic space so that the embeddings of similar images and texts are closer in this space.

We expect that the CLIP+Diffusion technique can be applied to code generation to avoid the limitations of the autoregressive paradigm and find that code has similar properties with both natural language and image.
Both natural language and code use tokens to construct sentences or programs, and both need to follow certain grammatical rules to be correctly understood; existing techniques for code generation have drawn on deep learning techniques (e.g., GPTs) for natural language. 
There are a lot of ambiguity in natural language, such as polysemy, context dependence, and so on. 
Out-of-vocabulary (OOV) problems often occur when the model is dealing with natural language, i.e. the vocabulary encountered is not in its predefined vocabulary.
However, the code has almost no ambiguity and can avoid the problem of OOV by dividing the namespace. So we think the code does not need to use the autoregressive paradigm next-word prediction.
In addition, we find similarities between codes and images: color images consist of components such as yellow, blue, black, etc., and each component (e.g., yellow) contains different entities (e.g., light yellow, dark yellow, etc.).
The code consists of components such as classes, methods, variables, etc., and each component (e.g., method) contains different entities (e.g., method 1, method 2, etc.).
We call the property that an object is composed of different types of components as heterogeneity.

We propose a transferable pre-training CLIP model for code understanding, which is the basis of generating high-quality code using diffusion technology and other code understanding tasks.
For code encoding, we first broke the autoregressive encoding paradigm of GPTs imitating natural language, and propose a single-channel, one-dimensional, heterogeneous image encoding paradigm imitating images. 
Because according to the human programming mind, code is written in a sequence, so we encode the code as a single-channel, one-dimensional, heterogeneous image.
Built-in or common classes, methods, attributes, operators, and other components of the code are encoded as fixed numerical IDs.
The different entities of each component (for example, the built-in methods print() and len() in Python) are encoded as similar IDs, in order to imitate similar pixels in the image.
User defined components only have unique IDs in the same namespace, and different entities of each component are also encoded as similar IDs.
This encoding scheme avoids the OOV problem, so we do not need to embed the tokens, ensuring that the new paradigm imitate images, not natural languages.
For pretraining, we propose a code encoder consisting of one-dimensional convolution and local pooling, which is suitable for the above code encoding paradigm.
The encoder not only learns the preceding part of the code of the next token like GPTs, but iteratively learns the global information of code with the memory of global information.

















§ MOTIVATION

GPTs' autoregressive next-word prediction paradigm has some limitations, such as lack of planning, working memory, ability to backtrack, and reasoning abilities, and do not apply to most areas of mathematics <cit.>.
Researchers believe that GPTs lack a global and deep understanding of the task or the output because they
rely on a local and greedy process of generating the next word <cit.>.
In addition, during the long-term use of GPT-4 in the field of code understanding, we further confirmed the limitations through following cases studies.



 §.§ Cases Studies




  §.§.§ Lack of ability to handle complex logic

We asked GPT-4 some code problems with complex logic and
multi-step operations, such as "My training data is ndarray with shape (359, 4), which means 359 arrays, and each array contains 4 lists (length is 1149, 3000, 3000, 18 respectively.).The label of the training data is ndarray with shape (359,).
Now, I want to train an embedding network whose input is the training data and whose output is an ndarray with shape (359, 4), representing 359 arrays, each containing 4 lists of length n. The output is an ndarray with shape (359, 4). How should it be implemented?
"
The statement may seem convoluted, so we have added a summarising "In a nutshell, we want the training data to be input into this embedding network, and each array contains 4 lists that are embedded to the same length n."
Although human beings can understand the meaning of the problem. GPT-4 gives us incomplete code full of logical bugs as shown in Fig. <ref>. E.g., GPT-4 does not understand the shape of the output we want (batch size, 4), and wrongly concatenates 4 lists of each item. The training process is also not complete, e.g., there is no for loop of epochs. The variables indicated by red wavy line are also not explained. 





  §.§.§ Lack of working memory

If we ask the same question again after some rounds of dialogue because we forgot the early answer, GPT-4 often give a completely different answer. This will result in almost all dialogue rounds being invalid.
For example, we repeated the GPT-4 question "How do I use a terminal command to output a commit that changes more than n files in a repository?" twice, with several rounds of dialogue between them. GPT 4 answered two different wrong answers as shown in Fig. <ref> (a) and (b) respectively.




  §.§.§ Overly dependent on prompt format


GPT-4 will provide different answers to questions with different formats but the same meaning. For example, when we ask "How do I use a terminal command to find the commits that change more than n files in a repository?" and "How can I use a terminal command to find the commits with a number of changed files greater than n in a repository?" GPT-4 give very different answers as shown in Fig. <ref>



  §.§.§ Not good at matrix operation

GPT-4 often fails to handle matrix operations correctly or introduces new bugs when handling matrix operations. For example, as shown in Fig. <ref>(a), when we asked GPT-4 why the figure (a) code reported an error: "ValueError: could not broadcast input array from shape (359,1149) into shape (359,)". GPT-4 will allow us to modify according to Fig. <ref>(b).
But it introduces new bugs into the code: ValueError: only one element tensors can be converted to Python scalars.
GPT-4 ignores global correctness and instead gets stuck in solving local problems.





  §.§.§ Lack of creativity

GPT-4 is not creative. For example, if we ask GPT-4 to invent an algorithm for predicting earthquakes, GPT-4 cannot provide an answer, even if it is to provide some ideas for this scientific question.



 §.§ Discussion




  §.§.§ Using CLIP+Diffusion <cit.> in image understanding/generation can avoid some limitations similar to using GPTs in code understanding/generation

As shown Fig. <ref>(a) and (b), even if a piece of text with complex logic is input,  CLIP+Diffusion can globally understand the text and correctly generate images with complex structures. 
The lines and colors in (a) are complex, precise, and smooth. The complex fur and texture in (b) are generated clearly.
Even if the form of the input text is changed, as long as the meaning is the same, the generated image is still correct.
In addition, CLIP+Diffusion also has creativity. 
As shown Fig. <ref>(c) and (d), panda mad scientist and a cat dressed as french emperor
napoleon holding a piece of cheese hardly ever appear in real life and the training data. But CLIP+Diffusion technology can create unprecedented images based on human descriptions.
Therefore, we believe that if this technology is applied to code understanding/generation, it will greatly improve performance in this area.





  §.§.§ In contrast to CLIP+Diffusion for image understanding/generation, why do GPTs for code understanding/generation have the above limitations



We find that the CLIP model would memorize the entire image information globally while training, rather than only memorizing the previous code to infer the next token as GPTs do.
For example, CLIP learned about the global distribution of fur color and texture, as well as the structure and position information of facial features of many pictures of shiba inu. 
Thus, the image representation corresponding to the textual description given by the human output by CLIP already has global information about the image <cit.>. Then Diffusion further restores a high-quality image on this image representation.

However, due to GPTs adhering to the paradigm of predicting the next token, the model is unable to globally remember fully functional code information.
As shown in Fig. <ref>, if we ask GPT-4 to generate a logically complex code, even if GPT-4 has already learned the complex code, because it can only predict the next token based on the previous code, GPT-4 cannot generate a structurally complete code.
Other limitations of GPTs should also be caused by this paradigm.
Specifically, after a few rounds of dialogue, when we ask the same question again, due to changes in the previous text, the answers given by GPT-4 are completely different.
And, questions with the same meaning but different forms are different previous texts, so GPT-4 will generate different answers.
In addition, the reason why GPTs are not good at handling matrix operations and has creativity is because it does not truly understand the code. Understanding code requires learning the entire code information globally during the training process, rather than local context. Overall, we believe that GPTs' autoregressive next-word prediction paradigm is the fundamental reason for the limitations of GPTs.



  §.§.§ Why is the autoregressive paradigm adopted by GPTs to understand/generate code

Autoregressive paradigm is used for understanding and generating code by GPTs because it is mainly applicable to understanding and generating natural language. 
Both code and natural language are tools that people use to communicate. Natural language is used for communication between humans, while code is used for communication between humans and computers.
Both natural language and code are token sequences with specific syntax rules.
However, natural language is more flexible than code. Natural language can change according to different contexts, e.g. the meaning of words can change according to the context.
That is, the same words have ambiguity in different contexts.
Therefore, the understanding and generation of natural language require a strict reliance on contextual analysis. 
This is also the main reason why GPTs use autoregressive paradigm.
Although Transformer encoders can globally learn the information of the entire text and classify the predicted words, Transformer decoders still use the autoregressive paradigm in the generation process <cit.>.
This paradigm generates tokens step by step to ensure semantic correctness, relying on the previous text for each step, rather than outputting complete sentences.
And GPTs only adopt the decoder of the transformer.
We summarize the above analysis in columns 1 and 3 of table <ref>.



  §.§.§ What are the challenges and solution in proposing a new paradigm that can avoid the limitations of GPTs

To avoid the limitations of GPTs, we attempt to mimic images and propose a new structured coding paradigm for code, making it easier to apply CLIP+Diffusion technology.
But we found that the code not only has features of images but also features of natural language.
This has led to the challenge of proposing a new paradigm. 
As shown in columns 2 and 4 of table <ref>, in terms of imitating images, we find there are similarities between codes and images.
Images consist of components/colors (e.g., yellow, blue), and each component contains different entities (e.g., light yellow, dark yellow for yellow).
Code consists of components/types of entities (e.g., classes, methods, variables), and each component contains different entities (e.g., method 1, method 2 for method).
We call the property an object is composed of different types of components is heterogeneity.

However, we found that different tokens in the code cannot be represented with finite numbers, leading to Out-of-vocabulary (OOV) issues.
OOV problem means that the vocabulary encountered is not in the predefined vocabulary, which is often the case when the model is dealing with natural language.
If the embedding technique in natural language is used to solve this problem, it will result in the inability of the CLIP+Diffusion technique to be applied. This is the challenge of proposing a new paradigm.
However, unlike natural language, the namespaces of entities in code could much smaller than those of words in natural language.
The namespace of entities in the code is related to the functional scope of the code, which can be packages, folders, files, or even code snippets.
However, the namespace of most words in natural language is often the entire vocabulary. 
Therefore, if the namespace is strictly partitioned and numbers are used repeatedly to encode these cross-namespace tokens, this may solve the OOV problem.






§ APPROACH




 §.§ Heterogeneous image paradigm

The code has a highly standardized structure and is not as flexible as the structure of natural language, so we first break the commonly used autoregressive next-word prediction paradigm for code understanding and generation, and propose a novel single-channel, one-dimensional, heterogeneous image paradigm.
Both code and image have heterogeneity, meaning that both image and code are composed of different components.
Therefore, we refer to images and consider different components in the code (e.g., classes, methods, variables, operators, numbers, symbols) as different components in the image (e.g., red, yellow, blue, green, etc.).
Different entities within the same component, e.g., class 1 and class 2, etc. in class) are represented by similar numerical values (IDs) to simulate the pixel values of different entities within the same component in the image (e.g., dark red, light red, pink, etc. in red).
To prevent significant differences in the values assigned to tokens due to too many different tokens (OOV problem), we take a series of measures:
1) Clean the code. Replace the messy data that will affect the accuracy of code tokenization with placeholders of different categories. For example, the messy string output by the print function is occupied with 'STR'. 
This step is optional and needs to be ensured that removing such messy data will not reduce the accuracy of the text-to-code matching results.
2) Divide tokens of entities such as the definition of classes, methods, and variables into built-in tokens in the code library and user-defined tokens,
3) Divide User-defined tokens into namespaces,
4) The built-in tokens are represented by fixed numerical values (IDs),
5) User-defined tokens are only represented as temporary values (can be reused within other namespaces) in the namespace.
As shown in Fig. <ref>(a), a function is first represented as a 2-dimensional heterogeneous image, where keywords are represented by different similar red pixel values (e.g. bright red, pink, light red, etc.), variables are represented by similar green pixel values, and operators/symbols (e.g. , :, () are represented by similar yellow pixel values.



In addition, we also follow the same process 1)-5) for tokens that represent entity calls (such as calls to methods or class attributes).
In order to further reduce the number of such tokens and avoid OOV issues, we abstracted these types of token. 
That is, we use the same numerical value to indicate the tokens that contain the same entity call, e.g., a_param.strip and address.strip use the same numerical value to indicate.
We then maintain a list for each such numerical value for users to manually find which specific token it is.
Using the encoding method described above, we created a Python vocabulary on the 456,360 code snippets in the training set in the CodeSearchNet data set[https://opendatalab.org.cn/CodeSearchNet].
CodeSearchNet corpus contains code snippets and associated document comments from various programming languages from open-source repositories on GitHub. This corpus has been initiated and maintained by GitHub to encourage the development of code search and code understanding technologies.
The number of tokens and the corresponding IDs are different in different programming language. 
For example, the specific components and their numerical coding ranges in Python are shown in the Tab. <ref>.



For color images, there are three primary colors (red, blue, green), and other colors can be composed of these three primary colors. So, a color image consists of three channels.
The heterogeneous image proposed in this paper has only a single channel. Different components are equally important, rather than the fact that most components/colors in the image are composed of three basic components/colors.
According to human programming thinking, code is written sequentially.
This means that the dependence of each token on the tokens above and below it is not as high as the dependence on the tokens before and after it. 
Thus, the two-dimensional heterogeneous image is finally flattened into a one-dimensional heterogeneous image, and line breaks and spaces are retained and treated as symbols, as shown in Fig. <ref>(b).
This structure also looks like a living molecule chain (proteins, DNA, RNA, etc.). 
For example, they are all composed of basic elements and have heterogeneity, and they also look like a chain.
DeepMind has applied diffusion technology to predict the structure of living molecules. 
This further inspires us to explore and promote the application of diffusion technology in code understanding and generation. Designing a robust pre-trained model for text-to-code is the most fundamental task.
Overall, code has similarities with both images and natural language. 
The new paradigm simulates the numerical encoding of pixels in images on the token's numerical encoding in code. 
But in terms of shape, the new paradigm still looks like a sequence of natural language.



 §.§ Contrastive Language-Code Pre-training
 



  §.§.§ Architecture

We adopt a similar architecture of the Contrastive Language Image Pre-training (CLIP) model <cit.>, and propose the Contrastive Language Code Pre-training (CLCP) model. 
During the training phase, we jointly trains an code encoder and a text encoder to predict the correct (code, text) pairs of a batch.
As shown in Fig. <ref>,
given a batch of N (code, text) pairs, CLCP is trained to predict which of the N× N possible (code, text) pairings in the batch are the correct pairs.
CLCP learns a multi-modal embedding space by jointly training an code encoder and a text encoder.
In view of the strong advantages of the pre-trained transformer model in understanding natural language <cit.>, we continue to train directly on the model with our data. However, because we have adopted a new code coding paradigm, we have designed a new code encoder, which will be introduced later.

The model parameters are updated by the loss function, which maximizes the cosine similarity C_i· T_i(1≤ i≤ N) of the code embeddings C_i and the text embeddings T_i of the correct pairs N in the batch while minimizing the cosine similarity C_i· T_j(1≤ i, j≤ N, i≠ j) of the code embeddings C_i and text embeddings T_j of the incorrect pairings N^2-N.
Specifically, we optimize a symmetric cross entropy loss over these similarity scores <cit.>.
We expect these two encoders to be used for many downstream tasks for code understanding in the future, such as using a text encoder to encode input text into a vector in the multi-modal embedding space and then inputting that vector into the diffusion model to restore high-quality code.
Alternatively, use code encoders in different languages to translate code written in one programming language into code written in another programming language, etc.





  §.§.§ Code encoder based on one-dimensional convolution


For the code encoder, we refer to two-dimensional convolution for learning on images <cit.>, and design a one-dimensional convolutional and pooling neural network suitable for one-dimensional heterogeneous images. 
As shown in Fig. <ref>, the source code is first encoded in one-dimensional heterogeneous images according to the new encoding paradigm.
Secondly, we use one-dimensional convolution to convolve one-dimensional heterogeneous images with a kernel size of k and a step size of s to learn the semantic and structural information of the code.
That is, the convolution kernel slides on the input one-dimensional image with a step size of s, and at each step the IDs in the one-dimensional image are multiplied by the corresponding weights in the convolution kernel and summed to obtain the IDs of the dimensions of the output new vector/feature map.
Each value of the new vector will be activated by the Relu function.
Third, a one-dimensional (1D) max or average pooling is applied to the activated new vector, with window size k' and step size s'.
The basic idea of pooling operation is to divide the input 1D image (also known as a vector) into several sub-vectors and calculate the maximum or average value of each sub-vector as the value of each dimension of the output vector.
Overall, convolution operations are mainly used to learn semantic and structural information between code tokens, while pooling is mainly used to aggregate information from sub-code fragments.
Finally, the code encoder consists of M blocks, each containing a convolution layer and a pooling layer.

Fig. <ref> shows the overview of a code encoder containing two blocks.
Each convolution layer outputs
multiple feature maps, each of which is calculated by sliding a 1D convolution kernel on the input numerical encoding (i.e., the process in Fig. <ref>(b)).
Each pooling layer receives the output of the convolution layer activated by Relu, performs pooling operations on them respectively, outputs the same number of feature maps, and then passes them to the next block.
The multiple feature maps output by the last block are flattened and input into a fully connected layer.



  §.§.§ Strategies for optimizing the training process

We use Relu function <cit.> to activate the output of the convolutional layer and use the He initialization <cit.> method to initialize the parameters of the convolutional layer.
Compared with traditional Sigmoid and Tanh activation functions, Relu has a linear relationship when the input is greater than 0, which means it will not saturate (i.e. the gradient will not approach 0). This helps to solve the problem of gradient vanishing, especially in deep networks <cit.>.
However, in deep networks, if the weight values are large, the accumulation of gradients during backpropagation can lead to gradient explosion problems <cit.>. 
So we use He Initialization <cit.> to avoid the above problem.
The core idea of Kaiming initialisation is to adjust the initial values of the weights based on the number of input nodes, in order to maintain consistency in the variance of the activation values across all layers.
Specifically, if a neuron has n input nodes, its weights are initialised to a Gaussian or uniform distribution with a mean of 0 and a variance of 2/n.
The advantage of this is that no matter how deep the network is, the distribution of activation values will remain within a reasonable range, avoiding the problem of vanishing or exploding gradients and making the network easier to train.

In addition, Batch Normalization (BN) is a commonly used technique that normalizes each batch of data.
The BN layer can reduce the data distribution changes caused by parameter variations in the network, thereby avoiding gradient vanishing or exploding problems during training and improving the convergence performance of the model.
However, by normalizing the input of each layer, BN may suppress the diversity of some representations <cit.> because BN forces each layer's input to have similar mean and variance during training, which may result in the model losing sensitivity to some important changes and diversity in the input distribution. 
This means that BN may not be applicable to certain types of neural networks, especially those that require preserving certain characteristics of the input distribution.
For example, CNN models used for image classification typically have the first few layers responsible for extracting low-level features of the image, such as edges, corners, textures, etc. In these layers, each feature map may focus on a specific pattern within the image.
However, BN independently normalizes each batch of feature maps, which may disrupt the relative differences between feature maps.
Therefore, when learning the code Heterogeneous image in this paper, we do not use BN to avoid blurring the distribution characteristics of the input image.
But in RQ2 of section 4 below, we will compare the effects of using BN and not using BN.







§ EXPERIMENTS




 §.§ Research Questions




The goal of this paper is to propose a new code encoding paradigm and design an applicable text-code encoder training model that can be migrated to different downstream tasks of code understanding so that the model may avoid the limitations of the autoregressive model on code understanding.
Following questions are set on 4×NVIDIA Tesla V100 with 32GB RAM:
RQ1. How effective is CLCP on Zero-Shot Transfer?
It is difficult for the proposed CLCP to achieve high accuracy on the zero-shot task with a limited amount of data (456,360 (code, text) pairs) compared to the dataset (400 million (image, text) pairs) used for the CLIP model. 
For example, some previous work also used natural language supervision for image representation learning like CLIP <cit.>.
However, due to the limited amount of training data, their effects are not as amazing as those of the CLIP (e.g., Li et al <cit.> reach only 11.5% accuracy on ImageNet in a zero-shot setting), so they have not attracted enough attention.
Collecting a large amount of high-quality data is still an arduous task in the future.
In this paper, to evaluate whether the proposed method has application value, we change the size of training and testing sets to magnify the effect of the model on the zero-shot task and evaluate whether the model can effectively learn the global information of the code.



RQ2. Ablation experiment.
In this RQ, we conducted some ablation experiments to explore training strategies for learning code heterogeneous images, rather than training strategies that are applicable to all models.
Specifically, pooling operations reduce the dimensionality of feature maps by aggregating information from adjacent regions, which inevitably results in the loss of some detailed information. For example, in max/average pooling, only the max/average value within each region is retained, while all other values are discarded.
This type of information loss can gradually accumulate in multi-layer networks, leading to poor model performance when fine features are required.
Therefore, in this RQ, we removed the pooling layer of CLCP model to verify its effectiveness.

In addition, we use He initialization <cit.> to avoid gradient explosion problems in backpropagation, and do not use BN to avoid blurring the distribution characteristics of the input image.
In this RQ, we remove He initialization and add BN respectively to verify the effectiveness of the optimization strategy we adopted during the training process.


RQ3. Will the prompt engineering improve the results?
Prompt engineering is a common method to improve the effect of model based on natural language supervision <cit.>.
We observed that the description of the code in the dataset is complex and redundant.
The common categories of redundancy is summarized in table <ref>.
So we try to clean up and reconstruct the text to improve the experimental results.




 §.§ Dataset

Different program languages need different code parsers to make vocabularies. 
Therefore, the number of tokens and the corresponding IDs are different in different programming languages. 
To reduce the workload of program language preprocessing, this paper makes an exemplary evaluation in Python. 
We use the Python data set of the CodeSearchNet corpus[https://opendatalab.org.cn/CodeSearchNet], including 456,360 (code, text) pairs in the training set and 22176 pairs in the testing set.
The CodeSearchNet corpus contains multiple public code libraries from GitHub, covering a variety of programming languages (such as Java, python, JavaScript, etc.).
The public code libraries contain a large number of code snippets from a variety of open-source software repositories, and the document comments corresponding to the code snippets help to understand the function and purpose of the code. 
This corpus has been initiated and maintained by GitHub to encourage the development of code-understanding technologies.
We selected 13,760 samples in the testing set that were completely different from the training set category by identifying different text descriptions and manually verifying them. These samples were used as an initial data set to evaluate the accuracy of the model in zero-shot tasks.



 §.§ Experimental Setup


Because the amount of data is limited, we imitate the works <cit.> referred to in the clip paper to do some preliminary exploration to verify the proposed approach is effective (e.g. the accuracy in <cit.> is only 11.5%), instead of pursuing the effect of training under 400 million data as in the clip model.



  §.§.§ Settings for RQ1


Dataset size:
We observe whether the proposed approach is effective on the zero-shot transfer task by gradually increasing the size of the dataset and the depth of the model.
As for data set size, each time we randomly sample datasets of different sizes to evaluate the improvement of the model in zero-shot transfer compared to the results of random prediction. 
Specifically, the size of the training set is 30000, 60000, 120000, 240000, and 456,360 respectively. 456,360 is the size of the entire training set in the dataset.
For the above training set, we conducted two experiments. The testing set for the first experiment consists of 50 randomly selected samples. 
The testing set for the second experiment consists of 50, 100, 300, 600 and 1000 randomly selected samples. 

Baselines: 
CLCP_lp: CLCP using local pooling layers. 
For the different sizes of training/testing sets, this type of CLCP is divided into five subtypes: CLCP_lp(3), CLCP_lp(4), CLCP_lp(5),  CLCP_lp(6),  CLCP_lp(7) which represent CLCP_lp with 3, 4, 5, 6 and 7 blocks. 
A block indicates a functional unit that includes a convolution layer and a pooling layer.
CLCP_gp: Similarly, CLCP uses global pooling layers. This type of CLCP is divided into five subtypes: CLCP_gp(3), CLCP_gp(4), CLCP_gp(5), CLCP_gp(6), and CLCP_gp(7).


CLCP_rn: 
Because ResNet <cit.> has achieved advanced performance in image coding and has been applied to the CLIP model, we use the same architecture as ResNet but replace the 2D convolution with 1D convolution. 
That is, the model consists of an input layer, N residual blocks and a global pooling layer. Each block contains three convolution layers. 
Among them, the first two convolution layers process the input data in series, while the third convolution layer is used to convolute the input by 1×1 convolution directly. 
Similarly, for different sizes of training/testing sets, N is taken as 3, 4, 5, 6 and 7.
Finally, the output of the third convolution layer is added to the output of the second convolution layer as the final output of a residual block.

Evaluation Metrics:
Accuracy (Acc.): The proportion of the correctly matched (code, text) pairs in the L testing pairs. 
If all L codes are randomly matched with the texts, the estimated acc (EA) is 1/L.















  §.§.§ Settings for RQ2


Dataset size:
We use the same dataset sizes in RQ1.

Baselines: 
Two most effective baselines in RQ1 are used, CLCP_lp and CLCP_rn with different blocks.
In addition, their variants are also used as baselines:
CLCP_lp+BN: A variant of CLCP_lp where the input of each convolutional layer is normalised by BN.
CLCP_lp-Pool: A variant of CLCP_lp where the pooling layer is removed.
CLCP_lp-Init: A variant of CLCP_lp where the He initialization is removed.
CLCP_rn+BN: A variant of CLCP_rn where the input of each convolutional layer is normalised by BN.
CLCP_rn-Pool: A variant of CLCP_rn where the pooling layer is removed.
CLCP_rn-Init: A variant of CLCP_rn where the He initialization is removed.

Evaluation Metrics:
In this RQ, we calculate the difference between the average accuracy of CLCP_lp/CLCP_rn with different blocks and the average accuracy of their variants across all sizes of datasets to determine the variation in the effect of the variants relative to the original model.




  §.§.§ Settings for RQ3


Data quality:
The text that describes the code in CodeSearchNet contains a lot of redundant information. 
For example, it contains user-defined paths, folder structures, and some demonstration information containing code results with special symbols. 
The redundant information can reduce the effectiveness of the model. 
Therefore, we perform prompt engineering on the dataset, i.e. we filter the redundant information in the text. The specific categories of redundant information are listed in Table <ref>.
We call the dataset filtered by prompt engineering is called CodeSearchNet(p).
Both CodeSearchNet(p) and CodeSearchNet are used in RQ2.



Dataset size:
We use the same dataset sizes in RQ1 for CodeSearchNet and CodeSearchNet(p) respectively. 

Baselines: 
We use the two most effective baselines in RQ1, which are CLCP_lp and CLCP_rn.
Similarly, for different sizes of training/testing sets of CodeSearchNet and CodeSearchNet(p), CLCP_lp is divided into CLCP_lp and CLCP_lp(p), each of which is further divided into 5 subtypes that represent CLCP_lp with 3, 4, 5, 6 and 7 blocks. 

Evaluation Metrics:
Use the same metrics of RQ1.



 §.§ Experiment Results


Answer to RQ1:
As shown in Fig. <ref> (stacked area chart), the legend represents the size of the model/the size of the training set. The different colors represent the growth in the model accuracy in different model sizes and dataset sizes.
We can see that as the sizes of the training set and model increase, the performance of all models improves on the testing sets of different size.
And all models perform better than the estimated Acc(EA) of random prediction.
In particular, the overall effect of variants using global pooling is not as good as that of variants using local pooling.
This may be because global pooling compresses the entire feature map into a single value by averaging or maximizing operations, which to some extent loses some important local information. If this information is critical to the task, the effectiveness of global pooling will be reduced.
Local pooling is a pooling operation performed on a local region (usually a small sliding window) of a feature map, taking the maximum or average value within that region as the output, in order to preserve the salient information in the feature map.
Compared to global pooling, local pooling can retain more local information, which is more advantageous for learning code images that require fine feature representation.

For the results in Fig <ref>, when the size of the test set increases synchronously with the size of the training set and model, the performance of all variants of the model decreases.
That is to say, the model is underfitting.
This may be because the training data are too limited compared to CLIP, which was trained on 400 million data points. 
In particular, because we have not used embedding techniques in NLP but have redefined the paradigm of code representation, the model needs to learn the dependency relationships between tokens on its own. 
Therefore, this type of code image data has a high degree of diversity and complexity. 
So, the model may not be able to learn the general patterns of the data from a limited data.






















Answer to RQ2:
Fig. <ref> shows the decrease in accuracy of models CLCP_lp and CLCP_rn compared to the original model when adding or removing different components.
EA represents the estimated Acc of the baseline random prediction.
The vertical axis represents adding (+) or deleting (-) components, and the horizontal axis represents decreased average ACC. of CLCP_lp and CLCP_rn with different sizes across all different training/testing dataset sizes. 
The model size and dataset size are the same as that of RQ1.
Legends of different colors represent different original models.

As shown in fig. <ref>, removing the pooling layer did not alleviate underfitting as expected, but resulted in a significant decrease in accuracy.
The accuracy of CLCP_lp-Pool is even lower than that of random prediction.
We observed that after 2-3 epochs of training, the validation loss of CLCP_lp-Pool and CLCP_rn-Pool no longer decreased, which means that the training process of the models fell into local optima.
Therefore, after removing the pooling layer, the model experienced overfitting.
In addition, as expected, after removing He initialization, the performance of both CLCP_lp-Init and CLCP_rn-Init decreased compared to the original model.
After adding the BN layer, the performance of CLCP_lp and CLCP_rn also decreased.
This may because BN independently normalizes each batch of code feature maps, which may disrupt the relative differences between feature maps.



Answer to RQ3:
As the sizes of the training set and model increase, the performance of CLCP with and without prompt engineering on the testing sets of same and different size are show in Fig. <ref> and <ref>.
The overall performance of CLCP without prompt engineering (CLCP_lp and CLCP_rn) is worse than that of CLCP with prompt engineering (CLCP_lp(p) and CLCP_rn(p)).
With an increase in data volume and models, the effectiveness of all models on the same testing dataset is steadily improving. In contrast, the effectiveness of all models decreases when the sizes of the testing dataset increase, regardless of whether prompt engineering is used. 
Therefore, using prompt engineering on the dataset can improve the performance of the model to some extent.








§ THREAT TO VALIDITY

Internal validity.
Due to hardware and data limitations, we did not use as many datasets as OpenAI used to train CLIP models to train CLCP models. We simply followed CLIP's previous work of proposing text-to-image (which was only initially trained and tested on limited data) and proposed an initial text-to-code work. This results in the trained CLCP being not robust and cannot be directly used for downstream tasks of code understanding.
The main purpose of this paper is to provide a new perspective for researchers engaged in code understanding. In future work, we will collect more datasets and apply for more funding to evaluate our model on better hardware environments. And because we did not use pre-trained embeddings for code encoding, but instead proposed a new encoding paradigm and retrained it, we did not compare with other works that used ready-made embeddings.
External validity.
The source code of clip model is not open, and many training details have not been published.
Therefore, we replicated and improved its framework ourselves, which to some extent reduced the performance of proposed model.





























§ CONCLUSION

Microsoft Research and Google DeepMind have found many limitations in the GPTs' autoregressive paradigm, manifested in the model's lack of planning, working memory, backtracking, and reasoning skills.
Microsoft Research found that GPT relies on a local greedy process of generating the next word, without a global understanding of the task or output. We further summarized the limitations of the autoregressive paradigm on code understanding through empirical research. That is to say, GPTs cannot handle complex logic and generate new code that has not been seen before, and they rely too much on the format of prompts to generate correct code. To address this issue, we propose a new code encoding paradigm inspired by the successful application of diffusion techniques in image generation, such as the Dalle2 and Sora models. We first discovered that the structure of code has both natural language characteristics and image/protein molecular chain characteristics, and encoded the code into a heterogeneous image paradigm with global information memory that mimics the structure of images and proteins. Then, we refer to Sora's CLIP upstream text-to-image encoder model and design a text-to-code encoder model that can be applied to various downstream code understanding tasks. This model learns the global understanding of code under the new paradigm, connects the encoding space of text and code, and encodes text input into the most similar code vector.
By conducting self-supervised comparative learning on 456360 text code pairs, the model achieved zero-shot prediction for new data.
This work is the foundation for using diffusion techniques to generate code in a new paradigm to avoid autoregressive limitations in the future.










ieeetr




 Paper Title: WebApp1K: A Practical Code-Generation Benchmark for Web App Development.

	
Candidate Distant Trans-Neptunian Objects Detected by the New Horizons Subaru TNO Survey
    [
    
========================================================================================



§ ABSTRACT 
We introduce WebApp1K, a practical code-generation benchmark to measure LLM ability to develop web apps. This benchmark aims to calibrate LLM output and aid the models to progressively improve code correctness and functionality. The benchmark is lightweight and easy to run.

We present the initial version of WebApp1K, and share our findings of running the benchmark against the latest frontier LLMs. First, open source LLMs deliver impressive performance, closely trailing behind GPT-4o and Claude 3.5. Second, model size has strong correlation with code correctness. Third, no prompting techniques have been found to lift performance either universally to all models, or significantly to a single model.

	


§ INTRODUCTION

Large language models (LLM) have been widely adopted in the software industry to enhance project velocity and code quality. Many coding benchmarks have been developed focusing on different aspects such as algorithms<cit.>, data science<cit.>, class-level code<cit.>, execution<cit.>, resolving software bugs<cit.>, etc., more efforts are needed to calibrate LLMs' ability to assist software development for consumer-grade applications. While the current norm of LLM application is in the form of copilot or code assistant to existing software developers<cit.>, we argue that LLMs already, and will continue to, empower more and more non-developers (by traditional standards) to write code. Below are some scenarios.



  * A handful of first-time startup builders brainstorm a new social network helping a group of underprivileged individuals, and bootstrap themselves to launch a prototype to the public. LLMs play pivotal role during each phase of the project, e.g. planning, tech stack choice, task assignment, code generation, and deployment.


  * A small group of employees volunteer to a greenfield project funded by a mid-sized software company. While enjoying the freedom of scope and high-level blessing, the team finds itself understaffed and under-resourced. The founding members spares no time to reach an all-hands-on-deck decision, and turns everyone into a developer with the aid of LLMs, very similar to the above startup scenario.


  * A college student in their senior year, determined to be a top competitor on the job market, decided to boost their GitHub profile by some impressive demos. They first consulted LLMs on popular frameworks and best practices, then delved into a project and continue to utilize LLMs to help maximize velocity. In particular, LLMs are used to generate, from scratch, functional code expected to be bug-free and ready-to-build.


Arguably, the ensemble of users exemplified above is a much larger group than the professional developer community. Undoubtedly LLMs will deliver tremendous and life-altering values to this group of users. However, as will be revealed by this study, the improvement space for LLMs is huge. To this end, we present a preliminary work to benchmark LLM performance on practical application development.

We aim the consumers of this benchmark to be practitioners working closely with LLMs, e.g. those who ship models (pre-training and post-training), or those who build LLM applications and decide how to choose/use their models. Lots of considerations are taken into the benchmark usability.


	
  * Marketability Since the benchmark aims to evaluate usefulness of LLMs to practical software development, the model artifact, i.e. the generated code, must meet the needs of customers or employers. As such, LeetCode-type algorithmic problems, while extremely valuable in other contexts, are not applicable here. Moreover, the evaluation would be a lot easier if the knowledge and capabilities to create marketable code is inherently possessed by LLMs, i.e. present in their training dataset.

	
  * Metric In order to present a clear direction to model hill climbing, the metric should be simple with unambiguous answer to each problem of the benchmark. Fortunately, low-hanging fruits are available because the code correctness as the most basic criteria is still an unsolved problem.

	
  * Overhead Training LLMs is a high-budget and high-risk process introducing lots of stress. To exacerbate the problem, training environments and tools are primitive, also vulnerable to deprecation given the continuously escalating model scale and the training infra it demands. Therefore, one should assume that bare-bone Linux is all we can count on. Anything more sophisticated, e.g. sandbox or virtual environment, will decrease the benchmark adopt-ability, also taking more time to run, potentially causing training delays.


We are blessed by achievements of open source contributors to address all these challenges. By building our benchmark around the most successful open-source frameworks, the generated code is widely considered marketable by the industry. Also these frameworks have abundant quality code and best practices shared on the Internet, already handily included in the training dataset of modern LLMs. Finally, the rich ecosystem of these frameworks equips us with stable and well-distributed testing tools to address the metric and overhead challenges.

We introduce WebApp1K, a benchmark to evaluate web app code generation performance of LLMs. We start with React, a JavaScript framework widely recognized as the top choice for web app development. Below are our main findings:


	
  * Open source models deliver strong results. The top performing model is DeepSeek Coder V2 which closely trails behind GPT-4o and Claude 3.5 Sonnet, the top two LLMs.

	
  * There is significant performance gap between the big model and the small model of the same family, proving that the scaling laws still apply in our evaluation context of code generation.
	
	
  * No prompting techniques have been found to universally improve performance for all models or significantly lift performance of a single model. On the other hand, the strong performance of top performers (GPT and Claude) under a very simple prompt suggests that LLM should be able to code correctly as long as the expectations (i.e. test cases) are clear.


We share three artifacts on GitHub and Huggingface: the dataset containing all 1000 problems of WebApp1K<cit.>, the script<cit.> to run WebApp1K, and the leaderboard<cit.>.

The rest of this report is organized as follows. Section <ref> details how WebApp1K was built. Section <ref> presents our evaluation methodology, various experiments, and performance results of the latest LLMs. Finally, Sec <ref>, conclude contributions and discusses future works. 


§ BENCHMARK

We choose React<cit.> as the first evaluation subject of our benchmark. Introduced in 2013, it has remained a top choice for web app builders for the quality and stability of the framework itself, productivity gains to developers, and the rich ecosystem. In addition to these marketabiliy benefits, it also constructs a higher level abstraction above HTML syntax and DOM manipulation, allowing developers to encapsulate their business logic to succinct and reusable code. These technical advantages also benefit the code generation evaluation on LLMs.

Consider a blogging website, in which a user adds comment to an existing blog post. This user journey is simulated by the unit test in Table <ref>. Here,  is a lightweight setup to mock a successful API response without running any additional software components. The following  lines simulate user actions (text input, mouse click etc.). Finally,  lines examine the expected outcome, i.e. the mocked API should be invoked exactly once and the system response of success should appear on the updated webpage. Similarly, the pairing failure case is shown in Table <ref>, where a mocked API failure is expected to lead to error message on the updated webpage.




Table <ref> shows the verbatim code generated by GPT-4o as an attempt to pass these two tests, which it does. Besides the toyish API and purposely simplified application semantics, the code is functional and conceptually reusable to a web application which needs social commenting functionality.


The WebApp1K benchmark consists of 1000 problems, each depicting an unique user journey illustrated above. The construction of this benchmark is a combination of human curating and data synthesis by LLMs.

First, human proposes 20 web apps and their feature sets. They are listed in Table <ref> of the Appendix section. Then, within the scope of each app, we ask GPT-4o to describe 50 different user journeys, then review and iterate with the LLM until quality and relevancy are satisfactory. Finally, we feed each user journey to GPT-4o, prompting it to codify the journey into a success case and a failure case. Each of them strictly follows the  sequence illustrated in Table <ref>. All test cases can be found in our GitHub repo<cit.>.


§ EVALUATION RESULTS



 §.§ Experiment Setup

The most straightforward way for us to access LLMs are public token-based APIs. For top close-sourced models, our only option is via the owners' APIs. The top open-sourced models are hosted by a few platforms, among which we choose Fireworks<cit.>.

We evaluate three closed-source, and three open-source model families listed in Table <ref>.


Below is the prompt we use. The majority of it is the test code exemplified in Tables <ref> and <ref>.

    Generate {file_name} to pass the tests below:    
       {success_test_code}{failure_test_code}. RETURN CODE ONLY.


We use pass@k, a metric defined in <cit.> and commonly accepted by subsequent coding task evaluation works. Due to budget and rate limit constraints, we evaluate each LLM-problem pair 10 times, i.e. n=10. Since k must be smaller or equal to n, we measure pass@1, pass@5, and pass@10. 


 §.§ Parameter Tuning

Although each API bears its minor difference, all APIs are heavily influenced by the design of OpenAI API. Table <ref> lists the tunable parameters exposed by each API. Since we do not know the default parameter value set by each API provider, we explicitly set the same parameter values to all LLMs under evaluation, whenever applicable. To limit the search space, we only tune temperature and top_p, the two most popular parameters available on all platforms. For other parameters, we assign fixed value to them across all LLMs.


We conducted a grid search to locate a sweet spot at which all LLMs deliver near-best results. We chose 100 random problems from the benchmark, 5 out of each application. We then choose the large model out of the five leading model families, and measured their pass@1 (n=1) on the discrete 2D space of temperature and top_n, where temperature = 0, 0.1, 0.2, ..., 1, and top_p = 0, 0.1, 0.2, ..., 1.


Table <ref> presents the lowest and highest Pass@1 value by each LLM in this grid search. Based on the results, we finalize our parameters as follows.

    temperature = 0.2 
    
    	top_p = 0.8 
    
    	top_k = 40 
    
    	presence_penalty = 0 
    
    	frequency_penalty = 0


As will be shown later, results of our full-scale evaluations also align with this small-scale experiment, except for the Deepseek coder model<cit.> whose performance exceeds expectation. Also worth noting is that open-source models exhibit larger performance variation than closed-source models.


 §.§ Prompt Experiments

We also study whether more sophisticated can lift the model performance.

The first experiment is system prompt, which assigns an explicit role to the LLM and raises its awareness. Available in all APIs we run, it complements the user prompt (Equation (<ref>)) which gives detailed instructions to LLM. Equation (<ref>) shows our system prompt.

    You are a code generator.


The second experiment is verbose comment, which aims to help LLMs better understand the semantics of  tests it tries to pass. For each of the 1000 problems, we feed its test code to GPT-4o and ask for English summary of the expectation in multiple sentences. The summary is then inserted into the test code. Table <ref> shows the verbose comment variant of the test code in Table <ref>.


 
The third experiment is error debugging. If the generated code fails the test, we add the failed code and the error log to the prompt, hoping the LLM will generate the correct code by learning from its own mistakes. Below is the prompt.

    {failed_implementation}
       The above code is the implementation of {file_name}. It failed the tests below
       {success_test_code}{failure_test_code}
       Below is the test log
       {error_log}
       Try to generate {file_name} again to pass the tests. RETURN CODE ONLY.


For all three prompt variants, we measure pass@1 (n=1) against all 1000 problems of the benchmark. Also in each experiment, we apply one prompt variant only, and compare it against the control test using the original prompt (Equation (<ref>)). Table <ref> summarizes the relative performance gains/loss of each variant.


To our surprise, we are unable to find a prompt variant delivering universally positive (or negative) impacts to all LLMs. Also we observe the huge swing in the error debugging column. The situation is unique here because this technique is not needed if the model output is correct on the first try. Strong models like GPT-4o can produce high Pass@1 (n=1) closed to 0.9, which significantly shrinks the sample size.

As such, we can not recommend LLM users to adopt or avoid any prompting technique we have experimented. More investigations are definitely needed here, especially why a seemingly harmless prompt variant should cause regression. 

Also for the sake of fair evaluation, we decide to use the original prompt (Equation (<ref>)) in our full-scale evaluation.


 §.§ Results

Table <ref> shows results of full-scale evaluation in the form of leaderboard. Models are ranked by Pass@10. where gpt-4o is the winner. Results of Pass@5 completely agrees with this ranking. Results of Pass@1 also largely agrees, except claude-3.5-sonnet which surpasses gpt-4o as the winner.


We believe the ranking of this leaderboard meets expectations of the LLM community. It is worth calling out the truly impressive performance of open source LLMs, especially Deepseek.

Also when comparing the large and small models of the same family, it is evident that scaling laws still holds. In fact, of the six open source models whose parameter counts are available, their performance ranking aligns with their size ranking, except deepseek-coder-v2-lite-instruct which itself is a coder model.



§ CONCLUSION AND FUTURE WORKS

This report presents our preliminary contribution to LLM code generation. We proposed WebApp1K, a practical and comprehensive benchmark focusing on web app development. To recap findings via this benchmark:

	
  * Open-source LLMs deliver impressive results
	
  * Model size plays pivotal role to code generation performance
	
  * No prompting techniques have been found to either universally help all LLMs or significantly help a single LLM


We plan for the following future works.

First, more frontier LLMs are released as of the writing of this report, namely Llama 3.1<cit.> and Mistral Large 2<cit.>. We will report our evaluation results as soon as possible.

Second, we need to make the benchmark more difficult, as it is already saturated by top performers, i.e. GPT-4o and Claude 3.5. While this benchmark, as it stands now, can help other models to improve themselves, we need to make improvement space for the best models, as bigger and stronger models will be released sooner rather than later.

Finally, this work produce lots of error logs which we will dive deeper. We expect to uncover useful insights, which could help models of all rankings to continue the hill climbing.


§ APPENDIX: APPLICATIONS OF WEBAPP1K


plain



 Paper Title: Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network.



Article Title]Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network










1]Bin Chengchengbin2422@mails.jlu.edu.cn

[2]Jiaxuan Lulujiaxuan@pjlab.org.cn






[1]College of Software, Jilin University, Changchun, 130012, Jilin, China

[2]Shanghai AI Lab, Xuhui District, 200232, Shanghai, China





With the advent of the era of foundation models, pre-training and fine-tuning have become common paradigms. Recently, parameter-efficient fine-tuning has garnered widespread attention due to its better balance between the number of learnable parameters and performance. However, some current parameter-efficient fine-tuning methods only model a single modality and lack the utilization of structural knowledge in downstream tasks. To address this issue, this paper proposes a multi-modal parameter-efficient fine-tuning method based on graph networks. Each image is fed into a multi-modal large language model (MLLM) to generate a text description. The image and its corresponding text description are then processed by a frozen image encoder and text encoder to generate image features and text features, respectively. A graph is constructed based on the similarity of the multi-modal feature nodes, and knowledge and relationships relevant to these features are extracted from each node. Additionally, Elastic Weight Consolidation (EWC) regularization is incorporated into the loss function to mitigate the problem of forgetting during task learning. The proposed model achieves test accuracies on the OxfordPets, Flowers102, and Food101 datasets that improve by 4.45%, 2.92%, and 0.23%, respectively. The code is available at https://github.com/yunche0/GA-Net/tree/master.








[
    *
    
=====





§ INTRODUCTION



With the onset of the era of foundational models, we have gradually entered a paradigm of pre-training and fine-tuning. In terms of downstream task adaptation, full fine-tuning requires adjusting all the parameters of a model to adapt to downstream tasks. However, as the scale of models and the number of tasks increase, such a method becomes inefficient. Consequently, numerous studies have focused on parameter-efficient fine-tuning, exploring strategies to efficiently adapt existing foundation models to downstream tasks. Previous parameter-efficient fine-tuning methods can be mainly categorized into three approaches. The first is prompt tuning <cit.><cit.>, which aims to achieve fine-tuning by modifying the model input rather than the model structure. The second is prefix tuning <cit.>, which involves updating only task-specific trainable parameters within each layer. The third is adapter tuning <cit.><cit.>, which achieves parameter-efficient fine-tuning by inserting adapter modules with a bottleneck architecture between layers.

In the realm of multi-modal learning, parameter-efficient fine-tuning methods have gained attention in recent research. Prompt vectors are used to align multi-modal data, achieving efficient multi-modal fusion in low-resource settings <cit.>. The main idea of CoOp <cit.> is to automatically design prompt texts. It keeps the pre-training parameters unchanged and uses a small amount of data to learn the appropriate cues. The TaskRes <cit.> method directly adjusts the weights of text-based classifiers without requiring extensive prompt updates to text encoders or elaborate adapters. π-Tuning optimizes parameters by predicting and interpolating task similarity across visual, language, and visual-language tasks, achieving efficient cross-task transfer learning <cit.>. However, these works do not consider modeling the complex associations between modalities.

With respect to modeling complex associations for parameter-efficient fine-tuning, methods related to graph neural networks (GNNs) have been explored. The concept of timely tuning has been introduced <cit.>. In the molecular domain, MolCPT <cit.> enhances graph embeddings by encoding additional molecular motif information. However, these works primarily focus on how to perform parameter-efficient fine-tuning for purely graph structures, without applying them to language-image multi-modal modeling.

Therefore, we propose a framework that combines graph structures with multi-modal parameter-efficient fine-tuning methods, enabling the learning of multi-modal information while considering the complex associations between modalities. The proposed model comprises four main modules: Multi-Modal Feature Extraction, Multi-Modal Graph Construction, Graph Adapter Net (GA-Net), and Prediction. In the Multi-Modal Feature Extraction module, each image is processed by a pre-trained MLLM model to obtain a text description for each image. The image and its corresponding text description are then processed by frozen image and text encoders to generate image features and text features, respectively. These features are combined into multi-modal features through feature concatenation. In the Multi-Modal Graph Construction module, a graph is constructed based on the similarity of multi-modal feature nodes. The GA-Net module then mines suitable knowledge from the graph nodes, resulting in features that have fully learned both textual and image information while considering their adjacency relationships. In the Prediction module, the loss function incorporates cross-entropy and EWC regularization <cit.>, which can mitigate the forgetting problem in task learning.

Compared with the current SOTA method, the proposed method improves by 4.45% in the OxfordPets dataset, 2.92% in the Flowers102 dataset and 0.23% in the Food101 dataset.

The contributions of this paper are summarized as follows:

    
  * A parameter-efficient fine-tuning method based on graph networks, GA-Net, is proposed. The proposed method combines graph structures with multi-modal parameter-efficient fine-tuning, learning both textual and image information while considering the adjacency relationships between different tokens.
    
  * EWC regularization is introduced into the loss function. By incorporating the correlation between parameter importance and the loss function, the EWC loss effectively retains knowledge from previous tasks and reduces interference with previous tasks when learning new tasks, thereby alleviating the problem of forgetting in task learning.
    
  * Compared to the SOTA model, the proposed model improves the test accuracy by 4.45% on the OxfordPets dataset, 2.92% on the Flowers102 dataset and 0.23% on the Food101 dataset.




§ RELATED WORK


Parameter-Efficient Fine-Tuning Methods (PEFT). Full fine-tuning involves modifying all the model's parameters to suit downstream tasks. Yet, as models grow in scale and the number of tasks expands, this approach becomes increasingly inefficient. To address this issue, in recent years, the natural language processing (NLP) community has explored parameter-efficient fine-tuning techniques (PEFT) <cit.><cit.><cit.><cit.><cit.>. These techniques only require adjusting a small subset of parameters, thereby improving efficiency <cit.>. For example, prompt tuning methods <cit.> attempt to achieve fine-tuning by modifying the model input rather than the model structure. Prefix tuning <cit.> updates only task-specific trainable parameters in each layer. Adapter tuning <cit.><cit.> inserts adapter modules with bottleneck architectures between layers to achieve parameter-efficient fine-tuning. Additionally, methods like BitFit <cit.> update only the bias terms and freeze the remaining parameters, while LoRA <cit.> reduces the number of trainable parameters by decomposing the weight matrix into low-rank matrices.

In multi-modal learning, parameter-efficient fine-tuning methods have gained widespread attention in recent research <cit.><cit.><cit.><cit.><cit.>. Using prompt vectors to align multi-modal data achieves efficient multi-modal fusion in low-resource environments, excelling in tasks involving two or more data modalities <cit.>. Research on scaling large multi-modal models (such as LLaVA and MiniGPT-4) has shown that parameter-efficient training methods like LoRA/QLoRA perform well in both multi-modal and language tasks, with performance comparable to full-model fine-tuning<cit.>. The main idea of CoOp <cit.> is to automatically design prompt texts. It keeps the pre-trained parameters unchanged and learns suitable prompts using a small amount of data. CLIP-adapter <cit.> inserts a random learnable module into the middle of the model. By updating this module, it better adapts to downstream tasks. Unlike CLIP-Adapter, Tip-Adapter <cit.> does not require SGD to train the adapter. Instead, it constructs a query-key cache model from few-shot supervision to obtain the adapter's weights. TaskRes <cit.> directly adjusts the weights of the text-based classifier without needing extensive prompt updates to the text encoder or carefully designed adapters. π-Tuning optimizes parameters by predicting and interpolating task similarity across visual, language, and visual-language tasks, achieving efficient cross-task transfer learning, especially effective in data-scarce situations <cit.>. PMF significantly reduces training memory usage by adding prompt vectors only in the deeper layers of a single-modal Transformer <cit.>. However, these methods do not model the complex associations between modalities.

Graph Neural Networks (GNN). Early work by Scarselli et al. <cit.> laid the foundation by proposing a framework for learning node representations in graphs, capable of capturing dependencies between nodes through iterative information passing. The pioneering work by Bruna et al. <cit.> applied convolutional neural networks to graph data in the spectral domain. However, this approach faced challenges in computational efficiency and generalization across different graph structures. The graph convolutional networks (GCNs) proposed by Kipf and Welling <cit.> are among the most influential models, performing convolution by aggregating feature information from the local neighborhood of nodes, thus achieving efficient and scalable learning. The graph attention networks (GATs) introduced by Veličković et al. <cit.> employ an attention mechanism to weight the importance of neighboring nodes, allowing for more flexible and dynamic information aggregation. Additionally, GraphSAGE by Hamilton et al. <cit.> introduced a sampling-based approach for large-scale graph representation learning. Numerous prior works have applied GNNs to association modeling tasks across various domains, including vision-based<cit.><cit.>, text-based<cit.><cit.>, and graph-based<cit.><cit.><cit.><cit.> applications.

Parameter-efficient tuning methods have also seen some exploration in the GNN domain. The concept of timely tuning has been applied <cit.>. Although methods like GPF <cit.> and GraphPrompt <cit.> are also parameter-efficient, they struggle to match the benchmarks set by full fine-tuning in non-few-shot settings. GPPT <cit.> designed a specific framework for GNNs, but its application is limited to node-level tasks. In the molecular domain, MolCPT <cit.> enhances graph embeddings by encoding additional molecular motif information. However, the aforementioned works design parameter-efficient fine-tuning methods specifically for pure graph structures. To the best of our knowledge, graph networks have not been applied in the field of multi-modal parameter-efficient fine-tuning.





§ METHOD



As shown in Figure <ref>, the model presented in this paper is composed of four main modules: Multi-Modal Feature Extraction, Multi-Modal Graph Construction, GA-Net, and Prediction. The primary function of the Multi-Modal Feature Extraction is to use pre-trained models to extract features from images and text. Then, through Multi-Modal Graph Construction, a multi-modal graph is built to relate the connections between different modalities. The proposed GA-Net is the only trainable part of the network, where a down projection is first introduced, followed by a GCN, and finally an up projection to propagate and update the vertex features in the graph. Lastly, in the Prediction module, we propose a combined loss function of EWC loss and cross-entropy loss to further enhance the network's performance. The details of each module are as follows:












 §.§ Multi-Modal Feature Extraction


In this module, we first use a pre-trained MLLM model to generate general text descriptions corresponding to the images. These descriptions do not involve the category names used in the final classification. Next, the images and their corresponding text descriptions are fed into frozen image encoders (such as ViT <cit.> or ResNet <cit.>) and text encoders (such as BERT <cit.>), respectively, to generate image features and text features. Finally, the image features and text features are combined into multi-modal features through feature concatenation. Mathematically, suppose X_i is the input image, and MLLM(∗) represents the MLLM model, the text description of image X_i obtained through the MLLM model can be expressed as

    X_i^t = MLLM(X_i)


Let I(X_i) and T(X_i^t) represent the series of tokens obtained from the frozen image encoder (ViT/ResNet) and text encoder (BERT), respectively. These can be expressed as

    V_i^I = {X_i^I^1, X_i^I^2, …, X_i^I^N} = I(X_i)



    V_i^T = {X_i^T^1, X_i^T^2, …, X_i^T^N} = T(X_i^t)


where each token is of dimension E. The image tokens encode information from each patch location, while the text tokens encode information from each word location. V_i^I and V_i^T are sets of tokens. Each element in V_i^I represents the i-th token in the image, and each element in V_i^T represents the i-th token in the text. N denotes the number of tokens. All tokens form a series of text and image vertices:

    V_i = {V_i^I, V_i^T} = Concat(V_i^I, V_i^T)

where Concat(∗) represents feature concatenation, and V_i represents a node after concatenating text features and image features.






 §.§ Multi-Modal Graph Construction

To uncover the structural knowledge in the text embedding space for downstream tasks, i.e., the relationships between different semantics, and given the diverse visual features of different samples, we can measure finer-grained relationships between different semantics in the visual and textual space. Thus, we can construct a multi-modal graph structure G = {V_i, E} = {V_i^I, V_i^T; E}, where V_i^I and V_i^T can be seen as the sets of image vertices and text vertices, respectively. E represents the set of edges.

We build the graph using the similarity of multi-modal features via a predefined threshold γ. When the similarity between two multi-modal features is greater than γ, an undirected edge is created between these two vertices, representing the adjacency relationships between all multi-modal features:

    E_ij =
        
          1,    if  i ≠ j  and Sim(V_i, V_j) > γ
    
          0,    otherwise

where

    Sim(V_i, V_j) = V_i · V_j/V_iV_j

represents the similarity between multi-modal nodes V_i and V_j. γ is the similarity threshold, and an edge is constructed when the similarity between two vertices in the graph exceeds this threshold.



 §.§ Graph Adapter Net

We propose a parameter-efficient fine-tuning method based on graph networks called Graph Adapter Net (GA-Net), where the rest of the network is frozen and only the GCN <cit.> is fine-tuned during downstream tasks. The advantage of this method is that it can adapt to downstream tasks and improve model performance without significantly increasing the number of model parameters. Furthermore, since adapter fine-tuning only requires training a small number of parameters, it can significantly reduce computational resources and time costs for fine-tuning. The unique aspect of GA-Net is its ability to update features based on our constructed multi-modal graph structure, allowing fine-tuning while preserving adjacency relationships. The process of GA-Net can be represented by the following formula:

    C_X_i = W_up( GCN( W_down(X_i) ) )






where C_x represents the vertex feature matrix. The GA-Net utilizes a bottleneck architecture, including a down-projection W_down: ℝ^n_in→ℝ^n_mid, a Graph Convolutional Network (GCN) layer, and an up-projection W_up: ℝ^n_mid→ℝ^n_out. For vertex feature aggregation, we utilize the GCN, and the update formula for the vertex features in each layer can be formulated as:

    C^(l+1) = σ( D̂^-1/2ÂD̂^-1/2 C^(l) W^(l))

where Â is the normalized adjacency matrix, defined as Â = A + I, where A is the original adjacency matrix and I is the identity matrix. D̂ is the degree matrix of Â, with diagonal elements D̂_ii = ∑_j Â_ij. W^(l) is the trainable weight matrix of the l-th layer, with dimensions F_l × F_l+1. σ is the activation function, applied element-wise to the matrix. C^(l) represents the node feature matrix of the l-th layer of the GCN, with dimensions N × F_l, where N is the number of nodes and F_l is the feature dimension of the l-th layer. C^(l+1) represents the node feature matrix of the (l+1)-th layer of the GCN, with dimensions N × F_l+1, where F_l+1 is the feature dimension of the (l+1)-th layer.





 §.§ Prediction

In the prediction stage, we introduce Elastic Weight Consolidation (EWC) regularization <cit.> into the generic cross-entropy loss function. By incorporating the importance of parameters and their association with the loss function, the EWC algorithm can effectively retain knowledge from previous tasks and reduce interference when learning new tasks, thereby addressing the problem of catastrophic forgetting in task learning.

First, we calculate the importance of parameters based on the training results of previous tasks, using the Fisher Information Matrix I_A. The Fisher Information Matrix is calculated as follows:

    I_A = 𝔼[∂^2 ℒ(θ)/∂θ^2|_θ^*] = 𝔼[(∂ℒ(θ)/∂θ∂ℒ(θ)/∂θ^⊤) |_θ^*]

where I_A is the Fisher Information Matrix representing the importance of parameters; 𝔼 is the expectation operator; ℒ(θ) is the cross-entropy loss function, representing the model's CE loss under parameters θ, with L_CE = ℒ(θ); θ are the model parameters; θ^* are the optimal parameters obtained from previous task training; ∂^2 ℒ(θ)/∂θ^2 is the second derivative of the loss function with respect to the parameters, indicating curvature information; ∂ℒ(θ)/∂θ is the first derivative of the loss function with respect to the parameters, indicating gradient information.

The EWC loss is defined as:

    L_ewc = ℒ_ℬ(θ) + λ/2∑_i I_A(i) (θ_i - θ_i^*)^2

where ℒ_ℬ(θ) is the loss function for the current task B; λ is a hyperparameter that balances the learning of new tasks and the retention of old tasks; I_A(i) are the diagonal elements of the Fisher Information Matrix, representing the importance of parameter θ_i in task A; θ_i^* are the parameters learned in task A. The model, regularized by EWC, is used for the final prediction, improving performance on new tasks while mitigating the forgetting problem.

The total loss is the sum of the ordinary cross-entropy loss L_CE and the EWC loss L_ewc:

    L_total = L_CE + L_ewc






§ EXPERIMENT



 §.§ Experiment Settings

We validated our model on three downstream classification tasks: Oxford Pets <cit.>, Flowers102 <cit.>, and Food101 <cit.>. All these datasets belong to the fine-grained classification category. The Oxford Pets dataset contains 37 categories (25 dog breeds and 12 cat breeds) with a total of 7,349 images. The Flowers102 dataset includes 102 categories with a total of 8,189 images. The Food101 dataset consists of 101 food categories with a total of 101,000 images. These datasets are not only rich in categories but also possess high fine-grained characteristics, making them ideal for evaluating the model's performance in distinguishing similar categories.



  §.§.§ Implementation Details

We use the LlaVA <cit.> model to generate general text descriptions corresponding to the images, ensuring that these descriptions do not mention the category names for final classification. Unless otherwise stated, we use the pre-trained backbone ViT-B/16 <cit.> as the visual encoder to produce visual features. We optimized the model for 100 epochs. During training, we used a batch size of 16 and an Adam optimizer with an initial learning rate of 1 × 10^-3, which decays following a cosine learning rate schedule.



 §.§ Comparisons with State-of-the-Arts




We compared the proposed model with several state-of-the-art parameter-efficient fine-tuning methods, including TaskRes <cit.>, CoOp <cit.>, CLIP-adapter <cit.>, and Tip-Adapter <cit.>, on the Oxford Pets, Flowers102, and Food101 datasets. As shown in Table <ref>, the experimental results demonstrate that our model consistently outperforms previous parameter-efficient fine-tuning models on the average performance of the benchmark datasets. Our model achieved an average performance of 90.03%, outperforming Tip-Adapter by 3.19% and TaskRes by 2.82%. The model's test accuracy improved by 4.45% and 2.92% on the Oxford Pets and Flowers102 datasets, respectively, compared to the state-of-the-art methods. Our model still performed the best on the Food101 dataset. The more significant improvement on the Oxford Pets and Food101 datasets is due to the higher need for multi-modal associations, which are better modeled through GNN. Similarly, Tip-Adapter performs better than other SOTA methods as it combines the strengths of prompts and adapters, introducing task-related prompts into the model to provide more multi-modal associations.







 §.§ Model Efficiency



As shown in Table <ref>, we conducted experiments on the parameter quantities of different methods. The parameter consumption of Tip-Adapter is exceptionally high and is not in the same range as other methods. Among the remaining three models, TaskRes has a higher parameter count than CLIP-Adapter, with CoOp having the least parameter count. Our model has a parameter count of 0.056M, which is less than most models and only slightly higher than CoOp. However, the accuracy of our model significantly surpasses that of CoOp, indicating that our model achieves a good balance between accuracy and parameter count.




When the image encoder and text encoder are not frozen, the parameter size is 195,337,765. When the image encoder and text encoder are frozen, the parameter size is 56,869. Therefore, with only 0.029% of the parameters being trainable, our model can surpass the state-of-the-art (SOTA) models on multiple datasets. Our model ranks second in terms of model efficiency among SOTA models. This is partly because the text descriptions in the CoOp model are generated using fixed handcrafted sentences, which are relatively simple text descriptions. Compared to our model, even though the CoOp model has fewer training parameters, the quality of the textual information is lower, resulting in significantly lower accuracy on two datasets.




 §.§ Ablation Study




As shown in Table <ref>, the baseline is a simple linear layer trained on single-modal features, which are also extracted using a pre-trained model. Without using our foundational modules, the accuracy is only 86.20%. After applying EWC regularization, the accuracy improves by 1.65% to 87.85%. When using multi-modal learning, the performance further increases by 2.23% to 90.08%. The improvement with multi-modal learning is because leveraging two modalities simultaneously provides greater capability compared to using a single modality. Finally, by integrating the complete graph method, the accuracy improves by 2.55% to 92.23%. The performance boost from incorporating the graph is due to its ability to better model the relationships between tokens, thereby demonstrating the effectiveness of our proposed method.




The text descriptions for each image are generated by MLLM. In the baseline experiments, replacing all text descriptions uniformly with "A photo of a pet/flower" eliminates text information during training. Ablation experiments show that different combinations of methods and features enhance the model's performance to varying degrees. The GA-Net model achieves the best performance by combining EWC<cit.>, multi-modal features, and GNN. Introducing EWC regularization improves model performance by 1.65%; introducing GNN enhances performance by 2.55%; and incorporating multi-modal learning boosts performance by 2.23%. The significant performance improvements from multi-modal learning and GNN indicate that our model can effectively capture complex associations between modalities. EWC regularization helps mitigating the forgetting problem in task learning, making its effect more pronounced with larger datasets.

In terms of memory cost, Tip-Adapter has the lowest memory consumption, while CoOp has the highest. This is contrary to the parameter usage, indicating that, to some extent, a greater number of parameters can lead to lower memory consumption. Although our model also incurs a significant memory cost, it is still less than CoOp and performs better. While Tip-Adapter consumes less memory than our model, our model requires far fewer parameters and delivers significantly better performance. Furthermore, our model needs to store adjacency relations between different tokens, which inevitably consumes some memory space. This aspect contributes to the performance of our model. This demonstrates that our model achieves a good balance between accuracy and parameter consumption.





 §.§ Hyperparameter Study




To investigate the impact of hyperparameters, specifically similarity thresholds, we analyzed different similarity thresholds on the OxfordPets and Flowers102 datasets, as shown in Figure <ref>. The results indicate that different datasets are affected differently by varying similarity thresholds. For both datasets, the accuracy increased most significantly within the threshold range of 0.3 to 0.5. Additionally, the accuracy improvement for the Flowers102 dataset was more pronounced within this range compared to the OxfordPets dataset, suggesting that the Flowers102 dataset is more sensitive to adjacency relations in the graph. When the similarity threshold reaches 0.7, the model achieves peak accuracy on both datasets. For thresholds less than 0.7, accuracy shows an increasing trend with rising similarity thresholds. However, once the threshold exceeds 0.7, accuracy changes minimally, but the training burden increases, indicating that a similarity threshold of 0.7 is optimal.






§ CONCLUSIONS


This paper comprehensively reviews the limitations of previous parameter-efficient fine-tuning methods in low-data environments. These methods only model a single modality and lack the utilization of structural knowledge in downstream tasks. Therefore, we propose a novel parameter-efficient fine-tuning model, GA-Net, which extracts knowledge suitable for features from each multi-modal feature node in the graph, resulting in features that have fully learned both textual and image information while considering their adjacency relationships. Experiments on three fine-grained classification tasks, Oxford Pets, Flowers102, and Food101, demonstrate that the GA-Net model is effective in parameter-efficient fine-tuning.

The limitations of the model stem from the generation of text descriptions. In this paper, we use MLLM to generate text descriptions for each image. However, these prompts are simple and lack sufficient diversity. We believe that providing more diverse and accurate prompts for downstream tasks, such as using refined image caption methods, would better model the textual structural knowledge and further improve the performance of GA-Net.




§ DECLARATIONS




    
  * Funding
    
    This research received no external funding.

    
  * Conflict of Interest/Competing Interests
    
    We declare that we have no competing interests.

    
  * Ethics Approval and Consent to Participate
    
    This study did not involve any human or animal subjects, hence ethical approval and consent to participate are not applicable.

    
  * Consent for Publication
    
    We have reviewed the manuscript and consent to its publication.

    
  * Data Availability
    
    The datasets used in this study are all publicly available:
    
        
  * Oxford Pets dataset: <https://www.robots.ox.ac.uk/ vgg/data/pets/>
        
  * Flowers-102 dataset: <https://www.robots.ox.ac.uk/ vgg/data/flowers/102/102flowers.tgz>
        
  * Food101 dataset: <https://www.kaggle.com/datasets/dansbecker/food-101>
    

    
  * Materials Availability
    
    Not applicable.

    
  * Code Availability
    
    The code used in this study is available at: <https://github.com/yunche0/GA-Net/tree/master>

    
  * Author Contributions
    
    We all contributed significantly to the research and manuscript preparation, including the conception and design of the study, data collection and analysis, and manuscript writing. We have read and approved the final version of the manuscript.




























 Paper Title: OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attack.














OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attack
    Kuo Gai, Sicong Wang, Shihua Zhang
        

The first two authors contributed equally. The authors are with the Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China, and also with the School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China (Corresponding author: Shihua Zhang, E-mail: zsh@amss.ac.cn).

    August 5, 2024
=======================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Deep neural networks (DNNs) are vulnerable to small adversarial perturbations of the inputs, posing a significant challenge to their reliability and robustness. Empirical methods such as adversarial training can defend against particular attacks but remain vulnerable to more powerful attacks. Alternatively, Lipschitz networks provide certified robustness to unseen perturbations but lack sufficient expressive power. To harness the advantages of both approaches, we design a novel two-step Optimal Transport induced Adversarial Defense (OTAD) model that can fit the training data accurately while preserving the local Lipschitz continuity. First, we train a DNN with a regularizer derived from optimal transport theory, yielding a discrete optimal transport map linking data to its features. By leveraging the map's inherent regularity, we interpolate the map by solving the convex integration problem (CIP) to guarantee the local Lipschitz property. OTAD is extensible to diverse architectures of ResNet and Transformer, making it suitable for complex data. For efficient computation, the CIP can be solved through training neural networks. OTAD opens a novel avenue for developing reliable and secure deep learning systems through the regularity of optimal transport maps. Empirical results demonstrate that OTAD can outperform other robust models on diverse datasets. 



Adversarial defense, Lipschitz network, optimal transport, convex integration problem




§ INTRODUCTION

Deep neural networks (DNNs) are the most crucial component of the artificial intelligence (AI) field. DNNs are rapidly becoming the state-of-the-art approaches in many tasks, i.e., computer vision, speech recognition, and natural language processing. Theoretical explorations of DNNs inspire the understanding of deep learning and development of new algorithms <cit.>. However, DNNs are vulnerable to adversarial attacks, i.e., a well-chosen small perturbation of the input data can lead a neural network to predict incorrect classes.

Various strategies have been proposed to enhance existing models <cit.>. These strategies include adversarial training <cit.>, where adversarial examples are generated during training and added to the training set. However, these modified models often exhibit vulnerabilities against strong adversaries <cit.>, primarily because DNNs require large gradients to represent their target functions and attacks can always take advantage of the gradients to construct adversarial examples. To stop playing this cat-and-mouse game, a growing body of studies have focused on certified robustness. 

One straightforward approach to certified robustness involves constraining their Lipschitz constant. Existing approaches to enforce Lipschitz constraints can be categorized into three groups: soft regularization <cit.>, hard constraints on weights <cit.> and specifically designed activations <cit.>. However, compared to standard networks, these approaches show suboptimal performance even on simple datasets like CIFAR10. The strict Lipschitz constraints during training may hinder the model's ability to find a more effective Lipschitz function. Additionally, the target function is not Lipschitz everywhere, especially in classification problems on continuous data distributions where the function cannot be Lipschitz at the boundary between two classes.

In this paper, we propose a novel two-step model named OTAD to combine the strengths of the mentioned approaches (Fig. 1). The objective is to achieve a robust and accurate learned function at the terminal stage of training without enforcing Lipschitz constraints throughout the entire training process. Inspired by optimal transport theory, we leverage the theory that the optimal transport map is the derivative of a convex function ϕ and possesses regularity properties, implying the map ∇ϕ is locally Lipschitz under moderate conditions. Based on this, we can learn the discrete optimal transport map through neural networks during training and compute the robust output of the model that satisfies the local Lipschitz property. 

In detail, we first employ a DNN to acquire the optimal transport map from data to the feature for classification (Fig. 1). Gai and Zhang <cit.> have demonstrated that ResNet with weight decay tends to approximate the Wasserstein geodesics during training. Therefore, we first utilize ResNet to obtain a discrete optimal transport map T from data points to their features. T can accurately classify the training data due to the approximation power of ResNet. Subsequently, we employ a robust model based on the discrete optimal transport map T instead of the learned ResNet. For arbitrary given input x in the inference process, our objective is to find an appropriate feature y such that a Lipschitz function f exists, satisfying f being consistent with the discrete optimal transport map on the training set and f(x)=y. Given a set {(x_i, T(x_i))}_i∈ I, the goal is to find a convex and smooth function g such that ∇ g(x_i)=T(x_i). This problem can be formalized into a convex integration problem (CIP). We demonstrate that solving a quadratically constrained program (QCP) based on recent advances in first-order methods <cit.> can find a solution to the CIP and yield a feasible value of y. 

However, the QCP is much slower than the inference of a DNN. To address this issue, we train a Transformer named CIP-net as an alternative to the optimization algorithm for efficient computation. Theoretically, we derive an upper bound of the Lipschitz constant of the Transformer block, demonstrating the strong performance of CIP-net.

To further improve the performance of OTAD, we extend it with various architectures and metric learning. First, we adapt OTAD to Transformer-based architecture such as ViT <cit.>, as Transformers employ residue connections in forward propagation, and the invariant dimension of the feature aligns well with the optimal transport setting. Second, since finding neighbors is a critical step in OTAD, the l_2 distance may not effectively characterize the similarity of data closing to a manifold in high dimensional space. Consequently, we explore metric learning to find more suitable neighbors and explore the trade-off between accuracy and vulnerability in such cases. We implement various experiments to test the proposed OTAD model and its variants under different settings. Empirical results demonstrate superior performance to adversarial training methods and Lipschitz networks across diverse datasets.

The rest of this paper is organized as follows. In section 2, we present the background of our method: the optimal transport theory and regularity of the optimal transport map. In section 3, we develop the Optimal Transport-based Adversarial Defense model (OTAD) and its implementation details. In section 4, we perform extensive experiments to demonstrate the defense ability of OTAD. 






§ RELATED WORKS



 §.§ Adversarial training

Adversarial training aims at resisting adversarial attacks by worst-case risk minimization. Consider a classifier neural network g with trainable parameters θ, let ℒ(g(x),y) denote the  classification loss on data x and its label y, then the objective of g is

    min_θ𝔼_p(x,y)[max_x^'∈ℬ(x)ℒ(g(x^'),y)]

where p(x,y) is the joint distribution of data and labels, ℬ(x) is a neighborhood of x. However, it is impossible to search the whole region since the loss surface of networks is complex. To approximate it, adversarial training methods <cit.> add adversarial examples (often found by gradient descent) to the training set during training. After training, the model is robust to the type of attack chosen in the training process. The defense can still be `broken' by stronger adversaries <cit.>. To escape the cat-and-mouse game, training neural networks with bounded Lipschitz continuity has been considered a promising way to defend against attacks.  



 §.§ Adversarial purification

Adversarial purification aims at reconstructing the clean data point by conventional transform <cit.> or generative model before classification  <cit.>. Let P be the distribution of clean data. For adversarial noise corrupted data point x+n_ad, generative model-based adversarial purification trains a network G to project the noisy data back to the manifold of clean data distribution, i.e., G(x+n_ad)=x. The classifier C is trained on P and will classify G(x+n_ad) with better accuracy. 

Adversarial purification can defend unseen attacks because the generative model G is trained independently from adversarial attacks n_ad and classifiers C. However, we can still construct adversarial examples by taking the gradients of the generative process. Such adversarial noise will not be purified <cit.>. The effectiveness of adversarial purification methods is highly related to the performance of generative models, such as energy-based models (EBM)<cit.>, generative adversarial networks (GAN) <cit.> and auto-regressive generative models<cit.>. The diffusion model is a rising effective generative model. Diffusion-based adversarial methods have achieved competitive performance on large-scale image datasets <cit.>. For small datasets (e.g., single-cell gene expression data) with specific noise or missing values, it can be hard to train an effective generative model, indicating that adversarial purification may not be a general method for various datasets.
 


 §.§ Lipschitz networks and random smoothing

Training neural networks under a Lipschitz constraint puts a bound on how much its input can change in proportion to a change in its input. Existing methods fall into three categories: soft regularization, hard constraints for weights, and specifically designed activations. 

Regularizations such as penalizing the Jacobian of the network <cit.> can constrain the Lipschitz constant locally but don't provably enforce a global Lipschitz constraint. Thus, adding such regularizations cannot solve the fragility of adversarial attacks on neural networks. 

On the other hand, as 1-Lipschitz functions are closed under composition, it suffices to constrain 1-Lipschitz affine transformations and activations. Several prior works <cit.> enforce the Lipschitz property by constraining the spectral norm of each weight matrix to be less than one. Another method <cit.> projects the weights closer to the manifold of orthogonal matrices after each update. These models provably satisfy the Lipschitz constraint but lack expressivity to some simple Lipschitz functions. 

To enhance the expressivity of the Lipschitz network, Anil et al. <cit.> proposes a gradient norm-preserving activation function named GroupSort and proves that the networks with this activation and matrix-norm constraints are universal approximators of Lipschitz function. Singla et al. <cit.> further provides some tricks such as certificate regularization to boost the robustness. Zhang et al. <cit.> propose to use l_∞-distance function which is also proved to have universal Lipschitz function approximation property. Though many improvements have been made in this direction, the performance of the above Lipschitz networks is still not satisfactory even on simple datasets like CIFAR10, partially because the strict Lipschitz constraint in the training process impedes the model from finding a better Lipschitz function. Fundamentally different from pursuing Lipschitz constraints in the training process, we propose a novel model that provides robustness by regularity property of optimal transport map while employing powerful architectures like ResNets and Transformers.

Randomized smoothing is another way to obtain a (probabilistic) certified robustness guarantee. This technique uses a Gaussian smoothed classifier, which predicts the most likely label when Gaussian noise is added to the input of the original classifier. Conversely, OTAD integrates neighborhood information but finds robust output satisfying local Lipschitz properties instead of using the original classifier, which may be vulnerable to attacks.



§ METHOD BACKGROUND



 §.§ Optimal transport

Optimal transport theory provides valuable tools for quantifying the closeness between probability measures, even when their supports do not overlap. 𝒫_2(ℝ^d) denotes the set of Borel probability measures with finite second-order moment. For two probability measures μ,ν∈𝒫_2(ℝ^d), let Π(μ, ν) denote the set of all joint distributions π(x,y) whose marginals are μ and ν respectively. The Wasserstein distance is defined as the solution of the Kantorovich problem <cit.>:

    W_2(μ,ν):=(inf_π∈Π(μ,ν)∫x-y_2^2dπ(x,y))^1/2


In the Monge formulation, maps are considered instead of joint distributions. The Borel map T pushes forward μ to ν, i.e., T_#μ=ν. For any set A⊂ℝ^d, T_#μ(A)=μ(T^-1(A)). If the feasible map T exists, the Monge formulation is equivalent to the Kantorovich formulation:

    W_2(μ,ν)=(inf_T:T_#μ=ν∫x-T(x)_2^2dμ(x))^1/2


The Brenier theorem <cit.> asserts that if μ is absolutely continuous, there always exists a convex function ϕ such that ∇ϕ_#μ =ν and ∇ϕ is the optimal transport map sending μ to ν. This convex function ϕ is called a Brenier potential between μ and ν. 

Consider a constant speed geodesic (μ_t) on 𝒫_2(ℝ^d) induced by an optimal transport map T connecting μ and ν:

    μ_t=((1-t)I+tT)_#μ,   μ_1=ν

The continuity equation for μ_t is given by:

    d/dtμ_t+∇· (v_tμ_t)=0

where v_t is a vector field on ℝ^d. By Benamou-Brenier formula in <cit.>, the constant speed geodesic can be recovered by minimizing the following energy function leading to the third definition of the Wasserstein distance

    W_2(μ,ν)=(inf_μ_t,v_t∫_0^1v_t_L^2(μ_t)^2dt)^1/2

Here, the infimum is taken among all solutions (μ_t,v_t) satisfying continuity with μ_0=μ and μ_1=ν.



 §.§ Regularity of OT maps

Regularity in optimal transport is usually understood as the property that the map ∇ϕ is L-Lipschitz, equivalent to ϕ being L-smooth. Assume μ and ν are supported on a bounded open set with their density function on the support set bounded away from zero and infinity. When the target domain is convex, Caffarelli <cit.> proved that the Brenier map can be guaranteed locally Lipschitz. The optimal transport map exhibits discontinuities at singularities when the target domain is non-convex. Nevertheless, the map remains locally Lipschitz on the support set excluding a Lebesgue negligible set <cit.>.

In this paper, we consider regularity (smoothness) and curvature (strong convexity) as the conditions that must be enforced when computing the optimal transport map. Our objective is to find a potential function ϕ that is l-strongly convex and L-smooth, i.e.,

    lx-y≤∇ϕ(x)-∇ϕ(y)≤ Lx-y





§ OTAD

Let f denote the learned function at the terminal stage of the training process, f^* denote the target function that maps the training data points to their labels and {x_1,⋯,x_n} denote the training data points i.i.d. sampled from distribution P_x. The primary objective of robust learning is to ensure that the learned function f simultaneously achieves the following two goals:


  * Well-approximation of the target function on the training data, i.e., the loss function 

    ∑_i=1^nℒ(f(x_i),f^*(x_i))

is minimized.

  * Control over the change in output under input perturbations. That requires the learned function is Lipschitz continuous with a small constant L. For any pairs of inputs x_1, x_2 ∈ℝ^d, the Lipschitz continuity condition is given by:

    f(x_1)-f(x_2)≤ Lx_1-x_2


OTAD utilizes DNNs with residual connections to approximate the target function on the training data and enforces the local Lipschitz property by solving a CIP problem. 



 §.§ ResNet-based OTAD

Consider the classification problem, we first train a m-block ResNet R(·) and a classifier H(·) to classify the dataset {x_i}_i=1^n with labels {y_i}_i=1^n. Given input x, the forward propagation of the k-th block of R is defined as

    x^k=x^k-1+R_k(x^k-1),    k∈{1,2,⋯,m}, x^0=x

where R_k denotes the shallow network inside the k-th block of R. Let R(x) be the last-layer features, i.e., R(x)=x^m. Here {x^k_1,x^k_2,⋯,x^k_n} can be viewed as particles of the distribution P_k, then the sequential of distribution {P_k}_k=1^m is a discretization of a continuity equation

    ∂ p_t/∂ t+ ∇· (v_t p_t)=0

According to the Benamou-Brenier formula in (<ref>), when p_0 and p_1 are fixed, the Wasserstein geodesic curve induced by optimal transport map can be recovered by minimizing the energy 

    ∫_0^1v_t_L^2(p_t)dt

A discretization of this energy concerning the ResNet R(·) is 

    ∑_i=1^n∑_k=1^mR_k(x_i^k-1)_2^2

Using the energy as a regularizer, the objective to train the ResNet R(·) and the linear classifier H(·) is 

    min_θ∑_i=1^nℒ(H(R(x_i)),y_i)+ α∑_i=1^n∑_k=1^mR_k(x_i^k-1)_2^2
          x_i^k=x_i^k-1+R_k(x_i^k-1),    k∈{1,2,⋯,m}, x_i^0=x_i

where ℒ is the loss function for classification, θ denotes all the trainable parameters in H(R(·)), and α is the hyperparameter to balance the two terms. Gai and Zhang <cit.> demonstrate that the weight decay operation plays a similar role with the regularizer in (<ref>) for a Wasserstein geodesic. Let z_i=R(x_i). If the problem (<ref>) is well solved, then we obtain a network-based classifier H(R(·)) with high training accuracy and a discrete optimal transport map from the data x_i to feature z_i. 

Though the network R(·) learns the discrete optimal transport map T, it can still be vulnerable to small perturbations. Therefore, we need to find a more robust function f̃ with f̃(x_i)=z_i. Let h denote the potential function of f̃, i.e., f̃(·)=∇ h(·). Since the optimal transport map is locally Lipschitz and has singularities when the target domain is non-convex, then for a given test data point x^', we can only trust training data near x^' to constrain f̃(x^'). Let N_K(x^') be the set of K nearest neighbors of x^' from the training dataset. Assume h is l-strongly convex L-smooth on the neighborhood of x^', then finding appropriate value ∇ h(x^') can be formulated to a convex integration problem:

Let I be a finite index set and ℱ_l,L denote the class of l-strongly convex and L-smooth function on ℝ^d. Given a set S={(x_i,z_i)}_i∈ I, the ℱ_l,L convex integration problem is finding a function f∈ℱ_l,L, such that z_i=∇ f(x_i) for all i∈ I. 


First, we need to determine the existence of such a function with respect to N_K(x^'). We introduce the definition of ℱ_l,L-integrable in <cit.>.

Consider the set S={(x_i,z_i)}_i∈ I. The set S is ℱ_l,L-integrable if and only if there exists a function f∈ℱ_l,L such that z_i=∇ f(x_i) for all i∈ I. 

According to Theorem 3.8 in Taylor <cit.>, if the set {(x,z)| x∈ N_K(x^'), z=T(x)} is ℱ_l,L-integrable, testing equals to find feasible values of u_i:=h(x_i) satisfying

    ∀ x_i,x_j∈ N_K(x^'),   u_i ≥ u_j+⟨ z_j, x_i-x_j⟩
    +1/2(1-ℓ / L)·
       (1/Lz_i-z_j^2
    +ℓx_i-x_j^2-2 ℓ/L⟨ z_j-z_i, x_j-x_i⟩)
 
If {(x,z)| x∈ N_K(x^'), z=T(x)} is ℱ_l,L-integrable, let h be the desire l-strongly convex L-smooth function. For test data x^', we can find a feasible value of v=h(x^') and z^'=∇ h(x^') by solving the following QCP problem (Theorem 3.14 in <cit.>)

    min _v ∈ℝ, z^'∈ℝ^d v
     s.t.    ∀ x_i∈ N_K(x^'), v ≥ u_i+⟨ z_i, x^'-x_i⟩
       +1/2(1-ℓ / L)(1/Lz^'-z_i^2
    +ℓx^'-x_i^2.
       . -2 ℓ/L⟨ z_i-z^', x_i-x^'⟩)


Then we obtain the feature z^' used to classify x^'. If {(x,z)| x∈ N_K(x^'), z=T(x)} is not ℱ_l,L-integrable, we repeat the procedures above with smaller l and larger L until we find feasible values. The solution always exists when l is sufficiently small and L sufficiently large. The training and testing scheme of ResNet-based OTAD is summarized in Algorithms 1 and 2. 






 §.§ Transformer-based OTAD

ResNet-based OTAD requires training an m-block dimension-invariant ResNet. Although dimension-invariant ResNets can be easily implemented for various data and tasks, their fixed dimensionality throughout the forward propagation limits their expressive power. To address this, we extend OTAD to the popular Transformer architecture <cit.>, named OTAD-T. Unlike ResNets, Transformers embed the input into a high-dimensional space, and subsequent Transformer blocks maintain this dimensionality, resulting in a model with good expressive power while keeping the dimensionality unchanged. 

Due to the residual connections in the Transformer, the forward propagation of the Transformer can also be viewed as a discretization of a continuity equation. Thus, it approximates the Wasserstein geodesic curve induced by the optimal transport map under the regularizer of (<ref>) at the terminal phase of training. One can use the discrete optimal transport map learned by the Transformer and do the same test procedure as ResNet-based OTAD.

In this paper, we focus on the Vision Transformer (ViT) <cit.>. Given an input image x_i, ViT first divides the input image into N non-overlapping patches {x_i,p}_p∈ N, then each patch undergoes linear embedding E(·), where it is projected into a higher-dimensional space. The sequence of patch embeddings {E(x_i,p)}_p∈ N is added positional information by an position operator P(·), resulting {P(E(x_i,p))}_p∈ N. The embeddings of patches {P(E(x_i,p))}_p∈ N are fed into the standard Transformer blocks and obtain features {z_i,p}_p∈ N. Let 

    x̃_i = 
    [ P(E(x_i,1)); P(E(x_i,2));           ⋮; P(E(x_i,N)) ],
      z̃_i = 
    [ z_i,1; z_i,2;     ⋮; z_i,N ].

The forward propagation of the k-th block of ViT is 

    x̃_i^k_Attn   = x̃_i^k-1+Attn_k(x̃_i^k-1)
    x̃_i^k   = x̃_i^k_Attn+MLP_k(x̃_i^k_Attn)

where Attn_k and MLP_k denote the attention and MLP block in the k-th block of ViT. The forward propagation of the Transformer can also be viewed as a discretization of a geodesic curve in Wasserstein space.
The corresponding discrete optimal transport map transforms {x̃_i}_i=1^n into the feature {z̃_i}_i=1^n. Then for given test data x^', we can embed it and search a neighborhood set in {x̃_i}_i=1^n. Finally, we estimate an output feature through solving a problem analogous to (<ref>).



 §.§ Neural network for CIP

In the procedure of OTAD, the most time-consuming step is solving the CIP, particularly the QCP problem for the feature of test data. Traditional QCP solvers like MOSEK <cit.> can be time-consuming when solving large-scale QCPs.

Recently, neural networks have been applied to various optimization problems <cit.>, achieving faster solving speeds and even higher accuracy than traditional solvers. Neural networks can efficiently capture complex patterns and dependencies in data, making them well-suited for high-dimensional and large-scale optimization tasks. To improve the inference speed of OTAD-T, we designed an end-to-end neural network to replace traditional solvers for solving the entire convex integration problem. This method is called OTAD-T-NN. 

Specifically, the inputs of CIP are the embeddings of test data x̃, its neighbors in training set N_K(x̃) and the features corresponding to the neighbors Z(x̃)={z̃_i|x̃_i ∈ N_K(x̃)}. The output is the solution of the QCP solver, i.e., the estimated feature z̃ of test data. Since the attention block in the Transformer can learn the complex relation between tokens, we train a Transformer to align the inputs and outputs of the CIP, using the solutions from the QCP solver as the training data. We use the MSE loss to fit the QCP solver's solutions. The resulting Transformer is called CIP-net. Assume we have a training set S, here N_K(x̃) is the neighborhoods set in S excluded x̃, then we can train the CIP-net by

    min∑_x̃∈ SCIP-net(x̃,N_K(x̃),Z(x̃))-QCP(x̃)_2^2


When the process of solving the CIP is replaced by CIP-net, the resulting OTAD-T-NN becomes differentiable. Then the gradient with respect to inputs can be used to construct adversarial examples. However, experiments show that OTAD-T-NN remains robust, indicating that the robustness of OTAD is not only due to gradient obfuscation. As a result of that, we need to understand the robustness of a Transformer block by bounding its Lipschitz constant.

However, dot product self-attention has been proven to be not globally Lipschitz <cit.>, which results in OTAD-T-NN not being globally Lipschitz. Nonetheless, adversarial robustness is more concerned with locally Lipschitz, meaning the Lipschitz constant within the range of adversarial examples. We provide the upper bound for the local Lipschitz constant of dot-product multihead self-attention when the input is bounded.


Given a sequence x_1,x_2,...,x_N ∈ℝ^D, the input X = [x_1,x_2,⋯,x_N]^T ∈ℝ^N × D is bounded by X_F ≤ M. For 1 ≤ r ≤ R, R is the number of head, let Q^(r), K^(r), V^(r)∈ℝ^D × D/R, and W ∈ℝ^D × D, assume all parameters are bounded by 
    max_r=1, …, R{Q^(r)_F, K^(r)_F, V^(r)_F, W_F }≤ M_θ.
 
The multi-head self-attention F is defined by

    F(X) = [f^1(X),⋯, f^R(X)]W,

where f^(r) is single-head dot-product self-attention defined by

    f^(r)(X):=softmax(X Q^(r)(X K^(r))^⊤/√(D / R)) X V^(r).

Then F with bounded input is Lipschitz with the following bound on Lip_2 (F):

    Lip_2(F) ≤√(R) M_θ^2 (  M_θ^2√(D/R) M^2 (√(N)+1) + √(N))




Note that the local Lipschitz constant Lip_2(f)=sup _X_F ≤ MJ_f(X)_2, we can bound Lip_2(f) by the Jacobian J_f_2 with bounded input.

Let A^(r)=K^(r)Q^(r)⊤/√(D / R)∈ℝ^D × D, consider the map f̃ from ℝ^N × D to ℝ^N × D, where f^(r)=f̃^(r) V^(r),

    f̃:=softmax(XA^⊤X^⊤) X

The Jacobian of f̃ is 

    J_f̃=[[  J_11     … J_1 N;     ⋮     ⋱     ⋮; J_N 1     … J_N N ]] ∈ℝ^N D × N D

where J_ij = X^⊤ P^(i)[E_ji XA^⊤ + XA δ_ij] + P_ij I. Here, E_ij∈ℝ^N × N is a binary matrix with zeros everywhere except at the (i, j)-th entry, δ_ij is the Kronecker delta, and P^(i) := diag(P_i:) - P_i:^⊤ P_i:. The vector P_i: is defined as softmax(XA^⊤x_i).

Consider the i-th row J_i in J_f̃, note that AB≤AB,A+B≤A+B and [A_1, ⋯, A_N]≤∑_iA_i,

    J_i _2 = 
    [J_i 1, ⋯, J_i N]_2 
       ≤∑_j  J_ij_2 
       ≤∑_j X^⊤ P^(i)E_j i XA^T_2+ X^⊤ P^(i) XA^T_2
    + ∑_j P_ij I_2 
       ≤A_2 ( ∑_j X^⊤ P^(i)E_j i X_2+ X^⊤ P^(i) X_2) + 1


Note that X^⊤ P^(i) X is a covariance matrix of discrete distribution 𝕏, where ℙ(𝕏=x_j)=P_ij, j = 1,...,N, ∑_i P_ij =1.

    X^⊤ P^(i) X_2 = Cov𝕏_2 ≤Tr( Cov𝕏) 
       = ∑_j P_i jx_j-∑_k P_i k x_k_2^2  = ∑_j P_i jx_j_2^2 - ∑_k P_i k x_k_2^2 
       ≤∑_j P_i jx_j_2^2 ≤∑_j x_j_2^2 = X_F^2


By Cauchy-Schwarz inequality,

    ∑_j X^⊤ P^(i) E_j i X_2 =
    ∑_j P_i j(x_j-∑_k P_i k x_k) x_i^⊤_2 
        = ∑_j ( √(P_i j)x_j-∑_k P_i k x_k_2√(P_i j)x_i_2) 
       ≤( ∑_j P_i jx_j-∑_k P_i k x_k_2^2) ^1/2( ∑_j P_i jx_i_2^2) ^1/2
        = √(Tr( Cov𝕏) )x_i_2 ≤X_Fx_i_2

Thus, 

    J_i _2 ≤∑_j  J_ij_2 
       ≤A_2 ( ∑_j X^⊤ P^(i)E_j i X_2+ X^⊤ P^(i) X_2) + 1 
       ≤A_2 ( X_Fx_i_2 +X_F^2 )  + 1



Let A be a block matrix with block columns or rows A_1, …, A_N. Then 
A_2 ≤√(∑_iA_i_2^2).


Given the bound of input and Cauchy-Schwarz inequality, we have ∑_i x_i_2 ≤√(N)X_F ≤√(N)M, then by lemma <ref>, 

    J_f̃_2 ≤√(∑_i  J_i_2^2 )
       ≤√(∑_i (A_2 ( X_Fx_i_2 +X_F^2 )  + 1 ) ^2 )
       ≤(
    		A_2^2 X_F^2 ∑_i ( x_i_2^2 + 2x_i_2X_F+ X_F^2 ) 
         + 2A_2 X_F ∑_i
    		( x_i_2 + X_F)  + N
    )^1/2
        =  (
    		A_2^2 X_F^4 (N+1) + 2A_2^2 X_F^3 ∑_i x_i_2 
       	+ 2  A_2 X_F^2 N 
    		+ 2 A_2 X_F ∑_i x_i_2 + N
    )^1/2
       ≤√(A_2^2 X_F^4 (√(N)+1)^2 + 2A_2 X_F^2(N+√(N)) + N)
        = A_2 X_F^2(√(N)+1) + √(N)
       ≤A_2 M^2 (√(N)+1) + √(N)


Hence, Lip_2(f̃) ≤K Q^T_2√(D/R) M^2 (√(N)+1) + √(N). For single-head dot-product self-attention f^(r)=f̃^(r)V^(r), any parameter matrix A_2 ≤A_F ≤ M_θ, then 

    Lip_2(f^(r))    ≤V^(r)_2 ( K^(r) Q^(r)T_2√(D/R) M^2 (√(N)+1) + √(N)) 
       ≤ M_θ(  M_θ^2√(D/R) M^2 (√(N)+1) + √(N))

Finally, for multi-head F, F(X) = [f^1(X),⋯, f^R(X)]W, by lemma <ref>,

    Lip_2(F)    ≤(√(∑_rJ_f^(r)_2^2))W_2 
       ≤√(R) M_θ^2 (  M_θ^2√(D/R) M^2 (√(N)+1) + √(N))



The theorem shows that the local Lipschitz constant of self-attention is bounded by the norm of the parameter matrix. Consequently, training with weight decay can help reduce the Lipschitz constant of self-attention, resulting in a more robust model.



 §.§ Finding better neighbors through metric learning

In the inference time of OTAD, the feature of test data x^' is related to its K nearest neighbors. For classification tasks, incorrect class neighbors contain a lot of obfuscated information. Therefore, finding the correct neighbors is crucial for the effectiveness of OTAD. 
Usually, we can use l_2 distance to find neighbors. However, l_2 distance is not a good metric in high dimensional space, i.e., two samples with a small l_2 distance may not necessarily be semantically similar. To this end, we adopt a metric learning method to discover a more suitable metric for neighbor searching.

Metric learning aims at capturing the semantic relationships between data. Classical metric learning methods learn an optimal metric from the specific data and task, such as Mahalanobis distance and its kernelizations <cit.>. Subsequently, the methods evolved to focus on learning a feature map ψ(·) so that the semantically similar samples are closer in feature space, as measured by a given specific metric like l_2 distance. To efficiently learn a suitable metric in a high-dimensional complex space, we introduce deep metric learning, i.e., ψ(·) is implemented as a DNN. For classification tasks, we optimize the deep metric learning (DML) network ψ_θ(·) by a chosen loss function such as contrastive loss <cit.> or triplet loss<cit.>. The optimized DML-net ψ_θ(·) ensures the features from the same class are closer together while features from different classes are pushed apart, enhancing the accuracy and effectiveness of the learned metric for tasks like neighbor searching and classification.

In this paper, we choose ResNet and one attention block in deep metric learning for neighbor searching. Our results demonstrate that deep metric learning can significantly enhance the performance of OTAD in handling complex data. However, DML-net itself remains vulnerable to adversarial attacks. For instance, an untargeted Projected Gradient Descent (PGD) attack can disrupt the learned features, making OTAD worse. In a word, we can always search for a robust DML-net ψ_θ(·) <cit.> for neighbor searching to improve the performance of OTAD while defending against adversarial attacks. If ψ_θ(·) is not a robust model, we face a trade-off: better neighborhoods but increased vulnerability.




§ EXPERIMENTAL RESULTS

In this section, we evaluate OTAD and its extensions. For OTAD, we train it on datasets of diverse data types, e.g., image, single-cell transcriptomics, and industrial tabular data. We compare the performance of OTAD against adversarial attacks on these datasets. We change the hyperparameter in OTAD to show its effect on the Lipschitz constant.
For OTAD-T, we test it with DML-net on more complex datasets against adversarial attacks and evaluate its robustness.
For OTAD-T-NN, we show its inference time and robustness against gradient-free and gradient attacks. We also compare our method with the commonly used k nearest neighborhood-based method (KNN). Since our model is constructed based on the theoretical understanding of residual networks <cit.>, we also test similar models based on plain networks without residual connections for comparison. Finally, we discuss the limitation of our model on synthetic data.



 §.§ Experimental setup

For OTAD, we set the weight decay hyper-parameter to 5× 10^-4 since we empirically found that weight decay can help ResNet learn the geodesic curve. We used an SGD optimizer with 0.9 momentum on the image, single-cell transcriptomics, and industrial data. We used the fully connected (FC) ResNet with 5, 4, and 10 ResNet blocks on the three datasets, respectively. The intermediate dimension in each block is set to be the same as the dimension of inputs.

For OTAD-T on CIFAR10 and TinyImageNet, we train Vision Transformers (ViT) with 6 blocks and 8 heads. The input images are divided into 64 patches mapped to a 256-dimensional embedding. The models are optimized using the AdamW optimizer with a cosine annealing learning rate scheduler.

For the metric in neighborhood searching, we primarily use the l_2 distance. We choose shallow networks as DML-nets to obtain more similar neighbors while maintaining robustness, including 3-block ResNet, 2-layer CNN, 1-block ViT, and an attention block. 
Specifically, we choose the ResNet-18 pre-trained by ImageNet-1K as DML-net for TinyImageNet. DML-nets are trained/finetuned using the triplet loss, focusing on the difference between the anchor-positive and anchor-negative distances of the learned features.



 §.§ Evaluation of attacks

We mainly consider the l_2 attack in this paper as the l_2 distance is adopted in the construction of OTAD. Let ϵ be the l_2 norm of the adversarial perturbations. Since the output of OTAD is obtained by solving the QCP, we cannot take the derivative of the output with respect to the input directly as neural networks. We choose gradient-free attacks to test the models. We introduce three attacks to evaluate our OTAD model on classification tasks:

Adaptive CW attack <cit.> with gradient-free solver Adaptive CW attack generates adversarial examples by solving CW loss contained defense module, and we solve this optimization problem by NGOpt in Nevergrad <cit.>, which is a gradient-free solver. We use NGOpt with budget = 850 and different mutation σ = 0.1, 0.15,0.2.

Backward Pass Differentiable Approximation (BPDA) <cit.> composited with PGD attack <cit.> BPDA is typically used when there exists an optimization loop or non-differentiable operations in defense modules. OTAD involves an optimization loop for solving the convex integration problem. Thus, the back-propagation of this part is approximated by the original network. We run 100 iterations of the PGD attack.

Square Attack <cit.> A score-based black-box attack via random search, which only accesses the inputs and outputs of models.


For the regression problem, we use a non-adaptive white-box attack that finds a perturbation to maximize the l_2 distance between the model's output and the label <cit.>. 

For classification problems, we use standard accuracy (the performance of the defense method on clean data) and robust accuracy (the performance of the defense method on adversarial examples generated by adaptive attacks) to measure the performance. The robust accuracy is estimated on 1000 samples randomly sampled from the test set. For the regression problem, we adopt the metrics MSE, E, and SMAPE from Gupta et al. <cit.> to measure the regression performance where the lower, the better.



 §.§ Model comparison and settings

We empirically compare OTAD with three types of defense methods, i.e., adversarial training, adversarial purification, and Lipschitz networks. For the adversarial training methods, we choose the commonly used PGD <cit.> and FGSM <cit.> adversarial training. We evaluate the methods by unseen threats and thus use l_∞ adversarial training except for industrial data. 
The noise level of the l_∞ adversarial training ϵ is set to ϵ=0.1 on MNIST, ϵ=8/255 on CIFAR10, ϵ = 4/255 on TinyImageNet.
For the adversarial purification methods, we choose median filter <cit.>, STL <cit.>, GAN <cit.> and DiffPure <cit.> to represent image pre-processing, optimization-based reconstruction, generative model-based purification models, respectively. When using the BPDA + PGD attack on these purification methods, we approximate the purification process using the identity map during back-propagation. 
For the non-purification methods, BPDA + PGD attack reduces to standard PGD. For the Lipschitz networks, we compare with l_∞-distance net <cit.> and a 1-Lipschitz l_2 network SOC+, which incorporates skew orthogonal convolutions with certificate regularization and Householder activation <cit.>. The hyperparameters of these models are the same as the original papers.




 §.§ OTAD is robust on diverse scenarios



  §.§.§ Performance on MNIST

On the MNIST dataset, we use OTAD with K =10, L =2, l=0, δ ^1 =δ ^2 = 0.2 to defend three adaptive attacks of different noise levels as mentioned above. 

OTAD achieves better robust accuracy in most settings (Tables <ref>, <ref>, <ref>), only slightly worse than SOC+ against BPDA + PGD. OTAD guarantees a small Lipschitz constant, which makes gradient-free solvers fail to find a reliable adversarial example.




To further investigate the ability of OTAD, we vary the hyper-parameters L-l and record the standard accuracy, robust accuracy, local Lipschitz constant of OTAD, and relative error (RE). The local Lipschitz constant is computed at a noise level ϵ around 0.3 that equals the adaptive attack. RE measures the gap between ResNet R(·) and OTAD feature f̃(·),

    RE(x) = R(x)-f̃(x)_2^2R(x)_2^2


We can see that L-l controls the local Lipschitz constant of OTAD (Fig. <ref>b). Thus, we may set smaller L-l to make OTAD more robust. However, smaller L-l is not always better. The local Lipschitz constant of the original ResNet is approximately 5.6. Smaller OTAD Lip (< 5.6) results in better robustness (smaller disparity between standard and robust accuracy) but worse standard accuracy (Fig. <ref>a). That is because OTAD deviates from the ground truth. A larger OTAD Lip results in worse robustness, larger RE, and worse standard accuracy (Fig. <ref>c). That is because OTAD deviates from the ResNet.
The best result can be achieved with L-l slightly smaller than the empirical Lipschitz constant of the original ResNet.





















  §.§.§ Performance on the single-cell transcriptomics data

OTAD is suitable for other data types. We use the MERFISH retina single-cell transcriptomics data by Chen et al. <cit.> and preprocess it with Scanpy. The dataset is a 7-class cell-type classification problem containing 82122 single cells (65697 for training and 16425 for testing), and each cell contains expression profiles for 500 genes. Since it is hard to use generative models (adversarial purification) for single-cell transcriptomics data, we only compared OTAD with the PGD adversarial training (Table <ref>). OTAD still achieves better robust accuracy than the PGD adversarial training.




  §.§.§ Performance on the industrial data

In addition to classification, OTAD is also suitable for the regression task on the tabular data. We use the red wine quality dataset <cit.>, containing 1,599 samples (1279 for training and 320 for testing) with 11 features. The features are physicochemical and sensory measurements for wine. The output is a quality score ranging from 0 to 10. The testing results under different noise levels are shown in Table <ref>. Due to DNN's robustness to random noise, the metrics computed by adding random noise can be regarded as standard. The attack makes the metrics worse, and OTAD makes the gap closer. The adversarial training is trained by adversarial examples (ϵ = 0.2). OTAD achieves better results in most cases. Thus, OTAD is suitable for both classification and regression.





 §.§ OTAD-T can deal with more complex data

For more complex datasets such as CIFAR10 and TinyImageNet, we test OTAD-T with DML-net for K neighbor searching with parameters K=10, L=2, l=0, and δ^1 = δ^2 = 1. As with OTAD, we first choose a gradient-free attack, BPDA + PGD, with a noise level of ϵ=0.5 on both datasets. For CIFAR10, we focus on two types of DML-nets, i.e., a 3-block ResNet and an attention block.
The results show that OTAD-T with a 3-block ResNet achieves the best accuracy, and OTAD-T with an attention block demonstrates robustness but has lower standard accuracy (Table <ref>).



Introducing DML-net brings additional robustness challenges. We use an untargeted PGD attack to disrupt the learned features and test the performance of OTAD-T under this DML attack. Table <ref> shows the robustness of OTAD-T with different DML-nets under the DML attack on CIFAR10. Despite OTAD-T with ResNet's strong defense against BPDA attacks, it can still be vulnerable to DML attacks. For a non-robust DML-net, we face a trade-off, i.e., better neighborhoods but increased vulnerability.



We also show the effectiveness of OTAD-T on TinyImageNet. The ResNet-18 is chosen for DML-net. Table <ref> shows the results compared with other defense methods, OTAD-T with ResNet-18 achieves the best accuracy. Moreover, a precise DML-net can even help improve the standard accuracy of OTAD-T, outperforming the original network. The cooperation of the two networks achieves better results and it is difficult to attack when DML-net is unknown.






 §.§ OTAD-T-NN achieves fast inference

We design an end-to-end 6-block Transformer encoder (CIP-net) for fast inference (OTAD-T-NN). The number of tokens in CIP-net equals 2K+1, where K (K=10) is the number of neighbors. The training set of CIP-net is given by the solutions of the QCP solver of OTAD-T with attention. Thus, CIP-net is a neural network designed for this specific optimization problem. OTAD-T-NN achieves a much faster inference speed. We compare the average inference time of OTAD-T and OTAD-T-NN per sample on CIFAR10 (Table <ref>), and a 3-block ResNet is chosen for DML-net.



Besides efficient inference, Table <ref> shows the performance of OTAD-T-NN against BPDA + PGD remains robust. Additionally, though the CIP-net is trained on solutions from OTAD-T with attention, OTAD-T-NN with ResNet remains highly efficient.

OTAD-T-NN is now differentiable. We can use gradient-based adversarial attacks, such as PGD, to test the robustness of OTAD-T-NN. Table <ref> shows the performance of defense methods against PGD attack. We do not test the STL and median filter due to the non-differentiable nature of their purification processes. Experiments show that OTAD-T-NN remains robust against PGD, indicating that gradient obfuscation is not the main reason for the robustness of OTAD.



The robustness of OTAD is due to solving the convex integration problem, resulting in a Lipschitz optimal transport map. CIP-net attempts to approximate this map and is trained with QCP solutions. If we train CIP-net with the original network's features {z̃_i}_i=1^n instead of the QCP solver's solutions, OTAD-T-NN becomes vulnerable to attacks again. Now we introduce a loss function that allows training on both the QCP solver's solutions and the original network's features, controlled by the parameter α,

    min∑_x̃∈ S   ( αCIP-net(x̃) - QCP(x̃)_2^2  
        + (1-α) CIP-net(x̃) - z̃_2^2 )


We compared the robustness against the PGD attack of OTAD-T-NN trained by different values of α in Table <ref>. As α decreases, the robustness of the model also decreases, indicating that OTAD-T-NN's robustness is primarily derived from the convex integration problem.






 §.§ Compared to KNN

The KNN classifier predicts the class of a sample based on the majority class of its K nearest training samples, which is robust. OTAD integrates information from neighboring points to compute a robust interpolation for the test data, functioning as a form of weighted KNN. Here, we compare OTAD and OTAD-T with KNN on MNIST and CIFAR10, respectively, while L = 2 in OTAD and L = 12 in OTAD-T. All methods search neighbors by the l_2 distance. For test data with K neighbors, the KNN feature of the test point is derived from the mean of the features of these K neighbors, while the OTAD feature is obtained by solving the convex integration problem. Table <ref> shows the standard accuracy of these methods with different Ks. Although OTAD outperforms KNN by only an average of 0.83% on MNIST, OTAD-T outperforms KNN by an average of 7.7% on CIFAR10 across K= 5, 10, 15. That demonstrates that OTAD shows distinctly higher performance.



 §.§ Plain network-based OTAD exhibits reduced robustness
 
OTAD is designed based on DNNs with residual connections, as the forward propagation of a residual network approximates an optimal transport map. By leveraging the regularity of the optimal transport map, the DNN-induced discrete optimal transport map can be made continuous by solving the convex integration problem, resulting in a mapping with local Lipschitz continuity. What happens if we remove the residual connections? We can conjecture that the OTAD based on a plain network might exhibit reduced robustness. 

To investigate the necessity of residual connections for OTAD, we test the performance of ResNet and plain networks on relatively simple datasets, such as MNIST and Fashion MNIST. We compare the test accuracy, OTAD's accuracy on clean samples, and robust accuracy under Square Attack (Table <ref>).




We train 5-block fully-connected ResNets on both MNIST and Fashion MNIST, the plain networks have the same architecture but without residual connections. These plain networks achieve similar test accuracy to ResNets on both datasets in these experiments. On MNIST, the plain networks exhibit slightly better standard and robust accuracy than ResNets. However, on Fashion MNIST, the ResNets significantly outperform the plain networks in terms of robustness, demonstrating the advantage of residual connections in handling more complex datasets. Here we test models on simple datasets MNIST and Fashion MNIST because the ViT architecture without residual connection is hard to train and we cannot compare the robustness of it to that with the residual connection fairly.





 §.§ The limitations of OTAD for complex data

OTAD shows effectiveness on diverse types of data. However, OTAD depends on a DNN with an invariant dimension in each residual block and the neighborhoods of input. Thus, 

when this invariant-dimension DNN is ineffective and the neighborhoods of test data contain obfuscated information, OTAD will become less effective. We show this limitation with the synthetic data for classification.

We generate 50000 training and 10000 test data of dimension 128. The data for each class k are sampling from a Gaussian random vector 𝒩(μ_k,Σ), where μ_k∈ℝ^d is uniformly sampling from the unit sphere. We change the variance from 0.1 to 0.6 to control the difficulty of the classification task. We train a 3-block ResNet on each synthetic dataset and record test accuracy. The performance of OTAD is shown in Table <ref>. 

As the task becomes difficult, the gap (RE) becomes wider. OTAD deviates from the ResNet more, resulting in worse standard accuracy, especially in the case of std = 0.3. That is because OTAD depends on the neighborhood of input. For a difficult task, the neighborhood contains lots of obfuscated information, hindering the performance of OTAD.







§ CONCLUSION AND DISCUSSION
 
DNNs are fragile to adversarial attacks. To address this issue, we developed a novel two-step model OTAD to achieve accurate training data fitting while preserving the local Lipschitz property. First, we train a DNN to obtain a discrete optimal transport map from the data to its features. Taking advantage of the regularity property inherent in the optimal transport map, we employ convex integration to interpolate the map while ensuring the local Lipschitz property. The CIP can be solved through QCP or neural networks for fast computation. Our model suits popular architectures such as ResNets and Transformers. Experimental results demonstrate the superior performance of OTAD compared to other robust models on diverse types of datasets well-trained by a DNN with unchanged latent dimensionality. There are some directions to explore in the future:

Defense through the cooperation of two networks: In the section on finding better neighborhoods, we use another network to find similar neighbors as inputs to the classification network. This strategy is analogous to adversarial purification, where a generative network is adopted to purify the adversarial noise and provide clean data to the classification network. In both cases, it is hard to attack the system when the designed network is unknown. Are there other cooperation strategies for two or more networks?

Defense using the inherent property of DNNs: DNNs exhibit various types of implicit regularization or bias. Here, we make the most of DNNs with residual connections approximating geodesic curves in the Wasserstein space and strengthen their robustness by interpolating the discrete optimal transport map. We may find more inspiration for building robust networks from the explorations on implicit bias and make the foundation studies useful.  

Defense based on distance between data points: The intuition to use Euclidean distance or other distance is that the norm of adversarial noise is small compared with the norm of clean data points and the distance between data points. In this paper, we propose to find neighborhoods of test data to assist classification. But the l_2 distance may not find semantic similar data because high dimensional data points tend to be close to a manifold. Data points closing in l_2 distance may be far from each other on the manifold. How to construct a robust and accurate distance to measure the similarity between data while distinguishing data and noise is still an open question.




§ ACKNOWLEDGMENTS

This work has been supported by the National Key Research and Development Program of China [No. 2019YFA0709501 to S.Z.], the National Natural Science Foundation of China [Nos. 12326614, 12126605], the R&D project of Pazhou Lab (Huangpu) [No. 2023K0602 to S.Z.], and the CAS Project for Young Scientists in Basic Research [No. YSBR-034 to S.Z.].



IEEEtran






