Paper Title: Privacy-preserving datasets by capturing feature distributions with Conditional VAEs.



: Augmented Masked Autoencoder Pretraining on Public Brain MRI Data for 3D-Native Segmentation
    Asbjørn Munk1,2* Jakob Ambsdorf1,2* Sebastian Llambias1,2 Mads Nielsen1,2

    August 5, 2024
==============================================================================================



§ ABSTRACT 


Large and well-annotated datasets are essential for advancing deep learning applications, however often costly or impossible to obtain by a single entity. In many areas, including the medical domain, approaches relying on data sharing have become critical to address those challenges. While effective in increasing dataset size and diversity, data sharing raises significant privacy concerns. Commonly employed anonymization methods based on the k-anonymity paradigm often fail to preserve data diversity, affecting model robustness. This work introduces a novel approach using Conditional Variational Autoencoders (s) trained on feature vectors extracted from large pre-trained vision foundation models. Foundation models effectively detect and represent complex patterns across diverse domains, allowing the  to faithfully capture the embedding space of a given data distribution to generate (sample) a diverse, privacy-respecting, and potentially unbounded set of synthetic feature vectors. Our method notably outperforms traditional approaches in both medical and natural image domains, exhibiting greater dataset diversity and higher robustness against perturbations while preserving sample privacy. These results underscore the potential of generative models to significantly impact deep learning applications in data-scarce and privacy-sensitive environments. The source code is available at https://github.com/francescodisalvo05/cvae-anonymizationgithub.com/francescodisalvo05/cvae-anonymization.



 



§ INTRODUCTION


Over the past decade, Deep Neural Networks have made significant progress, impacting a wide range of industries. A key driver of this progress has been the increasing availability of large, well-annotated datasets. However, such datasets are rare in several domains, such as medical image analysis <cit.>. While data-sharing between institutions offers a potential solution, it also raises privacy concerns over personal data. To this extent, data privacy has been a focal point for decades. 

For instance, the k-anonymity paradigm <cit.> was introduced in 2002 to obscure user data in relational databases, ensuring that no individual could be distinguished from at least k-1 others. Building on this idea, the k-Same <cit.> method emerged to address privacy concerns in surveillance technologies. It aggregates disjoint clusters of k data points (i.e., faces) with their centroid, either in the pixel space or in the eigenspace.

With the advent of generative adversarial networks (GANs) <cit.>, new possibilities for generating synthetic images have emerged. However, there is a risk of embedding personal information in the latent space. To address this, methods such as k-Same-Net <cit.> enforced k-anonymity within GANs' latent space during training. Furthermore, GANs have struggled with issues such as mode collapse, where models fail to produce diverse outputs, thereby limiting their generalizability <cit.>. Orthogonally, recent advancements have shifted focus to vision foundation models <cit.>, typically trained in a self-supervised fashion on large unlabeled datasets. These large models using vision transformers <cit.> excel at capturing high-level complex patterns and contextual information across different domains. Moreover, their robustness against image corruptions and distribution shifts <cit.> is an established benefit. Researchers are now proposing foundation models for specific domains including healthcare <cit.>, climate <cit.>, geospatial data <cit.>, and more. The rich and low-dimensional feature representation derived from these models makes them a suitable alternative (to the higher dimensional raw images) for pursuing downstream tasks, such as classification.  

Building on the potential of those highly complex and informative representations, as illustrated in Figure <ref>, we train a Conditional Variational Autoencoder () on the feature space of a pre-trained foundation model to accurately capture the training distribution. The generative process, conditioned on class labels, effectively mimics the original distribution while enhancing privacy and expanding the diversity of the generated feature vectors. After capturing the training data distribution with our , we demonstrate a paradigm that no longer relies on disclosing the original images (or their embeddings) but simply the frozen  decoder allowing the sampling of any required dataset size. Despite tremendous benefits with respect to data and patient privacy, this also substantially reduces the required data exchange from an order of gigabytes (images) to a few megabytes (decoder weights). 

In summary, our contributions are:


	
  * We propose a novel anonymization paradigm using Conditional Variational Autoencoders (s) trained on the feature space of foundation models, which provides a more informative and compact representation than the traditional image space.
	
  * We demonstrate superior feature diversity and classification performance with our  compared to reference approaches across both medical and natural datasets. This also highlights the effectiveness of foundation model's feature representation, even in areas beyond its initial training focus. 
	
  * We show that the diverse feature representation obtained via dynamic sampling and decoding through 's decoder enhances model robustness against perturbations.





§ RELATED WORKS


As digital data usage expands, the need for robust privacy measures becomes critical. The k-anonymity framework, proposed by Sweeney <cit.>, addresses this by ensuring that an individual cannot be distinguished from at least k-1 others within database entries. This method involves generalizing or suppressing specific identifying attributes such as zip code, gender, and other sensitive information. While effective at masking identities, this approach often leads to significant data loss, reducing the utility of the data. With the rise of video surveillance systems in New York, a seminal work <cit.> introduced k-Same for face de-identification, translating these principles to more complex scenarios, including the pixel space and the eigenspace. This is achieved by collapsing groups of k elements into a single representative point, namely the centroid. Despite its application in real-world medical systems <cit.>, the inherited data loss and lower data diversity can substantially affect the generalization <cit.> and robustness <cit.> of machine learning based solutions, particularly in domains where data is already scarce.

Moreover, building on the capabilities of GANs to generate high-quality synthetic samples, k-Same-Net <cit.> applies the k-anonymity paradigm within GANs' latent space for face de-identification. Enhancing this further, k-SALSA <cit.> recently introduced local style alignment to preserve granular details in retinal images, while PLAN <cit.> introduces an auxiliary identity classifier to prevent sample collisions. However, GANs still face significant challenges due to their high computational demands, training instability, and mode collapse <cit.>, which can affect their efficiency and reliability.

Before the advent of GANs, Variational Autoencoders (VAEs) have already demonstrated their generative capabilities. For face de-identification, Yang et al. <cit.> generate realistic yet untraceable images by clustering and re-synthesizing facial features. Taehoon et al. <cit.> enforce privacy by introducing noise into the latent vectors, obscuring identities before reconstruction. Other than the image domain, VAEs have been employed for anonymizing speaker data <cit.> and tabular data <cit.>, as well. Conditional Variational Autoencoders (CVAEs) enhance the versatility of VAEs by allowing the generative process to be conditioned on additional labels or attributes, effectively tailoring the output to specific data characteristics. Hajihassnai et al. <cit.> introduce ObscureNet <cit.>, a CVAE-like architecture to anonymize sensor data, that conditions the latent space on private attributes (, age, gender). The architecture consists of one input encoder, multiple external discriminators (one per attribute), and a conditional decoder. Apart from anonymization, the conditional generative process of CVAEs has been used to augment training data <cit.>, addressing issues such as data scarcity and imbalance. 

In contrast, our approach first enhances the conditional generative process by shifting the generation from the pixel space to the embedding space, utilizing large pre-trained foundation models. This transition enables a more efficient feature generation, thanks to the more compact and informative feature representation. In contrast to ObscureNet <cit.>, we employ one conditional encoder rather than multiple discriminators, which allows our architecture to scale more efficiently with the number of classes. Furthermore, instead of simply augmenting our training dataset, we focus on mimicking its embedding-based training distribution, aiming to replicate a diverse and privacy-aware distribution. We further integrate this generative process during training of downstream tasks, which allows us to dynamically generate new samples until convergence. 



§ METHOD


The proposed approach to capture and anonymize a given training dataset is illustrated in Figure <ref>. Initially, we describe the process of feature extraction through vision foundation models. This is followed by a description of 's training procedure. We then introduce two anonymization strategies: the first generates a persistent synthetic replica of the training dataset, while the second continuously generates data on the fly for downstream tasks. 



 §.§ Feature extraction
 

We employ a large pre-trained model to extract and subsequently store feature embeddings and their associated labels (f_i,y_i) ∈ℱ for each labeled input image (x_i,y_i) ∈𝒳, with an associated categorical class distribution 𝒞. The extracted feature representations f_i represent the positional information of x_i within the feature space of the selected feature extractor. Here, close points are more likely to share similar image patterns while distant points indicate semantic differences, potentially across different classes. This descriptive property of the feature space is beneficial for effectively capturing the training distribution. It is much less pronounced in the original pixel space, which suffers from inherent information redundancy and the challenge of calculating meaningful distances in high-dimensional spaces. Additionally, from a storage perspective, while a conventional 2D RGB image of dimensions 224×224 occupies approximately 150,000 pixels, the corresponding image embeddings typically have around 1,000 components, reducing data storage by several orders of magnitude. For our experiments, we rely on DINOv2 ViT-B/14 <cit.>, one of the most recent open-source foundation models that achieved state-of-the-art performance in a variety of tasks, demonstrating the high quality of their latent feature representations.



 §.§  training
 

Conditional Variational Autoencoders extend traditional Variational Autoencoders by integrating specific attributes, such as class labels, into the data generation process. This extension allows the  to not only produce data that closely resembles the original samples in terms of content but also aligns with the conditioned attributes, thereby improving both the relevance and specificity of the generated data. In our setup, the  is trained to accurately learn and replicate the distribution of a feature dataset ℱ, conditioned on the corresponding labels, i.e., (f_i,y_i=c). This training allows us to subsequently generate new anonymous feature vectors a_j for a chosen class ỹ_j that reflect the original training distribution. We can thus utilize only the pre-trained decoder to sample arbitrarily many samples per class c, adhering to the original categorical class distribution 𝒞. This approach ensures that the generated data maintains consistency with the original dataset while enhancing privacy and utility, likewise.



 §.§ Offline anonymization
 

The first way to anonymize our dataset is via storing a persistent anonymized replica of the data. To this end, once the  is appropriately trained, we generate synthetic feature vectors that reflect the initial data size and class distribution. This anonymization process is described in Algorithm <ref> and requires a pre-trained  model, a number of synthetic samples N and a categorical distribution 𝒞 representing the class probabilities of the original dataset. This ensures the generation of similar class proportions in the synthetic dataset. For each of the desired N data points (step 2), a class label ỹ_j is sampled from 𝒞 (step 3). Next, a latent variable z_j is drawn from a standard normal distribution (step 4) to serve as a stochastic input for data generation. These sampled variables (z_j,ỹ_j) are fed into 's decoder, which generates a new data point a_j conditioned on the class label (step 5). Finally, both the generated data point and its label are then stored (step 6). Upon completion, the algorithm outputs the anonymized datasets, consisting of these synthetically generated feature vectors and their labels, effectively balancing privacy preservation with the utility of the dataset for downstream tasks.





 §.§ Online anonymization



Another way to ensure data anonymization is to eliminate the use of persistent datasets entirely. Our proposed  model exemplifies this approach by replacing the traditional dataset with the 's pre-trained decoder. Specifically, we iteratively generate new data on the fly during the training of our task-specific head, meaning no persistent data is required anymore. This method not only overcomes the need to store or send large volumes of sensitive data but also enables the sharing of a simple model ('s decoder) capable of replicating the training data distribution at an arbitrary incidence rate without actual data exchange. Moreover, in the context of model sharing, such as in federated learning <cit.>, our approach offers additional security benefits. While federated learning allows for task-specific models trained on private data to be shared without exchanging the data itself, it is not without risks; model weights can potentially reveal training data characteristics <cit.>. 

Our approach iteratively generates anonymized and stochastic data. This can introduce an additional layer of security, further mitigating the risks associated with traditional data- and model-sharing approaches.




§ EXPERIMENTAL RESULTS


Initially, we outline the process of feature extraction and the training of our Conditional Variational Autoencoder (). Subsequently, we evaluate the classification performance and feature diversity using our anonymized method compared to k-Same <cit.>, the foundational anonymization method that influenced machine-learning-related applications, such as GANs <cit.>. Additionally, we evaluate the classification performance under inference perturbations, showcasing the robustness of the generated feature representation. Lastly, we perform a qualitative analysis of the initial feature space and the anonymized one, in order to visually comprehend the representativeness of the proposed method. 


 §.§ Feature extraction and  training


For feature extraction, we used the DINOv2 ViT-B/14 foundation model <cit.>, which has been trained on 142 million natural images and outputs image embeddings of size 768. To optimize the training process of our , these embeddings were pre-generated and stored on disk <cit.>. For datasets lacking official validation splits, we extracted a stratified sample (10%) from the training data to ensure representativeness. Note that both validation and test sets were kept non-anonymized, preserving their semantic integrity.  

The  architecture comprises a conditional encoder and a conditional decoder. The encoder first concatenates input features with one-hot class labels and then processes it through two linear layers having 256 and 100 dimensions, respectively. The latent variable z is again concatenated with the one-hot class labels and passed to a symmetric decoder that mirrors the encoder's architecture. We train the  with the Adam optimizer, learning rate of 0.001, and early stopping. In addition to the MSE loss, a KL-divergence loss is employed to enforce a standard normal prior in the latent space weighted with β = 0.1, prioritizing the reconstruction. These hyperparameters are standardized across all experiments to ensure consistent comparisons.



 §.§ Feature diversity and downstream performance



Evaluation metrics   The k-Same method simplifies anonymization by collapsing groups of k data points into their centroid. While effective for anonymization, this approach often results in significant information loss and increased data sparsity. Conversely,  effectively maintains the initial distribution's representation, enabling the generation of more samples than originally used, without the compromise on diversity. Yu et al. <cit.> estimate data diversity through the convex hull and the maximum dispersion, defined as the sum of all pairwise distances within the (anonymized) dataset. Although calculating the convex hull is impractical for datasets with fewer samples than dimensions, it is trivial to demonstrate that  achieves higher maximum dispersion compared to k-Same, as the latter's regression to the centroid inherently reduces pairwise distances. 

To provide a more nuanced comparison, we analyze the average nearest neighbor distance 𝒟 between the original feature set ℱ and the anonymized feature set 𝒜 (both of size N). This is formally defined in Equation <ref>:


    𝒟(𝒜,ℱ) = 1/N∑_j = 1^N d_min(a_j,ℱ)


where d_min(a_i,ℱ) is the minimum Euclidean distance between a_j ∈𝒜 and all f_i ∈ℱ. Intuitively, this is closely related to the mean of the minimum LPIPS <cit.> used in PLAN <cit.>. In fact, the latter measures the average minimum distance between generated samples and their closest real ones, using the activations of a pre-trained network. Moreover, an equally important goal of any anonymization technique is to preserve downstream performance while ensuring data privacy. In our evaluation, considering image classification as the downstream task, we use the area under the receiver operating curve (AUC) as the evaluation metric. This allows us to take into account different class imbalance ratios across the evaluated datasets. Following prior works in feature space evaluation <cit.>, we train a linear layer on top of the anonymized embeddings using the Adam optimizer with a learning rate of 0.001 and early stopping. To fairly compare our method against k-Same, we replicate the same exact class proportions of the original dataset. 

Reference method and datasets   Consistent with prior works <cit.>, we evaluate the performance of our  against k-Same using k ∈{2,5,10,15}. We exclude the previously mentioned GAN-based methods such as k-SALSA <cit.> and PLAN <cit.>, since they applied k-Same within GAN's latent space to generate anonymous synthetic images. Our analysis begins with a wide range of real-world, small-sized, medical datasets to underscore the clinical relevance of our method. From the MedMNIST+ collection <cit.>, we use BreastMNIST with resolution 224×224 (breast ultrasound, binary, 780 samples). Additionally, we include datasets from the MedIMeta collection <cit.>: Skin Lesion (dermatoscopy, multi-class, 1011 samples) and Axial Organ slices (CT, multi-class, 1645 samples). Another challenging dataset is OCTDL <cit.> (OCT imaging, multi-class, 2064 samples), which exhibits a severe class imbalance. To extend our evaluation to the natural image domain, we selected multi-class datasets from the  module [https://pytorch.org/vision/stable/datasets.htmlhttps://pytorch.org/vision/stable/datasets.html], focusing on those with moderate sample sizes. This includes STL-10 <cit.>, DTD <cit.>, Oxford-Pets <cit.>, and FGVC-Aircraft <cit.>, allowing us to cover a wide spectrum of general imaging tasks. 


Results   Figure <ref> presents a scatterplot with AUC on the x-axis and average nearest neighbor distance 𝒟 on the y-axis, positioning the most effective methods towards the top-right corner. To ensure the reliability of our findings, all results are averaged across three different seed runs.   consistently exhibits both high diversity and high downstream performance (𝒟↑ , AUC↑). In contrast, as expected, lower k-Same configurations (k=2,5) achieve comparable downstream performance but lack diversity (𝒟↓ , AUC↑), whereas higher k-Same settings (k=10,15) increase diversity at the expense of performance (𝒟↑ , AUC↓). Interestingly, consistent results are observed with the OCTDL dataset, which is particularly challenging. Specifically, it presents a severe class imbalance, with a majority class of 752 samples and a minority of just 13. Furthermore, the medical results hold true on natural datasets as well, although the differences in classification performance are less pronounced, partly because these datasets were included in the DINOv2 training set <cit.>. 
 




 §.§ Robustness of adaptive data sampling
 

This experiment evaluates the robustness of our proposed  method, which dynamically generates new samples a_j of class ỹ_j ∼𝒞 at every batch. Here, the inherent data diversity provided by our method is expected to contribute positively to model robustness. To evaluate this claim, a common approach involves testing against image corruptions that simulate real-world distortions <cit.>. However, these corruptions in the pixel space might already be mitigated by the inherent robustness of foundation models <cit.>. Therefore, we propose a perturbation test by injecting Gaussian noise into the test feature embeddings. We systematically apply this noise with zero mean and a gradually increasing standard deviation σ = {1, 2, 3}. Here, we evaluate the performance of  against k-Same. Therefore, we first train our  and then use its decoder to continuously generate new samples during training. Instead of only sampling from the latent space with a standard Gaussian distribution (μ=0, σ^2=1) as reported in step 4 of Algorithm <ref>, we sample with variance σ^2 = {0.5,1.0,1.5}. This allows us to vary the prototypicality of the generated vectors. Specifically, since we enforce a standard normal distribution (σ^2=1) during training, a lower sampling variance during data generation yields samples closer to the class prototypes, and vice versa. The classification pipeline follows the previous settings, reporting the average AUC over three seed runs. 


Results   Table <ref> shows the AUC observed on the clean test feature vectors (σ=0) and on those subjected to varying levels of Gaussian noise.  consistently ranks among the top performers. Although  may initially show slightly lower performance in a noise-free environment, it substantially outperforms k-Same as the noise level increases. This is particularly pronounced in the Skin Lesion and OCTDL, where  (σ^2=0.5) outperforms 2-Same at σ=3 by 4.2% and 3.2%, respectively. 
Notably, this lower sampling variance remarkably improves performance and robustness across all datasets, suggesting that a higher prototypicality might be beneficial in settings with limited and imbalanced data. These results underscore the ability of our approach to enhance model robustness across diverse domains, each presenting unique challenges related to data size and data imbalance.




 §.§ Qualitative results



In addition to quantitative metrics, we conduct a qualitative analysis of the feature space, illustrated in Figure <ref>, using t-distributed stochastic neighbor embedding (t-SNE) <cit.>. Notably, our feature extractor was not originally trained on medical data, therefore it may encounter difficulties with tight class separation due to the subtle nature of diagnostic features. For instance, in the BreastMNIST dataset, which consists of breast ultrasound images, the fine-grained details necessary for accurate diagnosis are not always clearly represented. Nevertheless, subtle differences within the class still seem to be detectable, as indicated by the shape of small clusters scattered throughout the 2-dimensional feature space. 
With respect to the evaluated anonymization techniques, the limitations of the k-Same method, such as information loss and data sparsity, become particularly evident. Although DINOv2 ViT-B/14 may not well separate classes in such datasets, our  approach manages to maintain the original distribution. Clearly, unlike k-Same, our method avoids information loss, ensuring our privacy-aware feature distribution closely mirrors the initial one.




§ DISCUSSION AND CONCLUSION
 

Limitations   While our method does not directly expose training data, we did not rigorously investigate formal privacy guarantees. Furthermore, the effectiveness of s heavily depends on the capabilities of the chosen feature extractor. In addition, this approach could further benefit from domain-specific foundation models, as they may be able to capture more nuanced patterns. Lastly, our data-sharing framework assumes that all parties involved use the same feature extractor. However, with the increasing interest over general foundation models <cit.>, this assumption becomes more reasonable and can be justified by the considerable benefits and enhanced downstream performance. 

Conclusion   We demonstrate that conditional generative models, such as s, effectively address challenges related to privacy and data utility across a diverse range of datasets. This approach further shows high robustness even with small sample sizes and severe class imbalances. Future research can build upon our insights, investigating the sensitive role of the sampling variance, exploring further conditional generative models, or extending our approach to be trainable in an end-to-end manner for downstream tasks.




§ ACKNOWLEDGMENTS


This study was funded through the Hightech Agenda Bayern (HTA) of the Free State of Bavaria, Germany.








 Paper Title: Towards Flexible Evaluation for Generative Visual Question Answering.















jihuishan, siqingyi, linzheng@iie.ac.cn
0009-0002-7815-9508, 0000-0001-8433-0215, 0000-0002-8432-1658

  Institute of Information Engineering, 
 Chinese Academy of Sciences
  School of Cyber Security, 
 University of Chinese Academy of Sciences
  Beijing
  China



wangweiping@iie.ac.cn
0000-0002-8618-4992

  Institute of Information Engineering, 
 Chinese Academy of Science
  Beijing
  China


⁎Corresponding author: Zheng Lin, linzheng@iie.ac.cn















§ ABSTRACT 
Throughout rapid development of multimodal large language models, a crucial ingredient is a fair and accurate evaluation of their multimodal comprehension abilities. Although Visual Question Answering (VQA) could serve as a developed test field, limitations of VQA evaluation, like the inflexible pattern of Exact Match, have hindered MLLMs from demonstrating their real capability and discourage rich responses. 
Therefore, this paper proposes the use of semantics-based evaluators for assessing unconstrained open-ended responses on VQA datasets. As characteristics of VQA have made such evaluation significantly different than the traditional Semantic Textual Similarity (STS) task, to systematically analyze the behaviour and compare the performance of various evaluators including LLM-based ones, we proposes three key properties, i.e., Alignment, Consistency and Generalization, and a corresponding dataset Assessing VQA Evaluators (AVE) to facilitate analysis. 
In addition, this paper proposes a Semantically Flexible VQA Evaluator (SFVE) with meticulous design based on the unique features of VQA evaluation.
Experimental results verify the feasibility of model-based VQA evaluation and effectiveness of the proposed evaluator that surpasses existing semantic evaluators by a large margin. The proposed training scheme generalizes to both the BERT-like encoders and decoder-only LLM. Relaed codes and data available at https://github.com/jihuishan/flexible_evaluation_for_vqa_mm24Our Repository.








<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224.10010225.10010227</concept_id>
       <concept_desc>Computing methodologies Scene understanding</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010224.10010225.10010228</concept_id>
       <concept_desc>Computing methodologies Activity recognition and understanding</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010179.10003352</concept_id>
       <concept_desc>Computing methodologies Information extraction</concept_desc>
       <concept_significance>100</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Scene understanding
[300]Computing methodologies Activity recognition and understanding
[100]Computing methodologies Information extraction









13 April 2024
[accepted]21 July 2024




Towards Flexible Evaluation for Generative 
 Visual Question Answering
    Weiping Wang
    August 5, 2024
======================================================================






§ INTRODUCTION
 
Visual Question Answering (VQA) evaluates the multimodal comprehension abilities 
by posing questions about given images and comparing the model's responses with annotated answers<cit.>. 
However, current VQA evaluation metrics have made it tough for evaluating the rich responses of Multimodal Large Language Models (MLLMs).


Most VQA datasets comply with a triplet format and each sample consists of a question, image and annotation. Annotations are often a single word or phrase <cit.> or a set of ten candidate answers <cit.>. Current evaluation metrics, Exact Match <cit.> (for samples without candidate answers) and VQA Score <cit.> (for samples with ten candidate answers), both require the responses to be identical in morphology with the annotation to be considered correct. Variations in tense, singular or plural forms and synonyms are not allowed, let alone sentence-style responses from MLLMs. 

Traditional vison-language models treat VQA as a classification problem <cit.>, where answers collected from the training set are used to establish pre-defined classes, and the possible responses are constrained to these classes. Thus the problem of evaluating multifarious responses does not exist. However, MLLMs treat VQA as a generative problem <cit.> and generates assorted responses. Meanwhile, the growing trend that the MLLM community prefers zero-shot test, has made it even tougher for models to generate responses that are identical to the ground-truth answers. As shown in Figure <ref>, semantically equivalent but morphologically distinct responses are not accepted.









Although it is possible to force the model to output a single word with a harsh prompt, such remediation may potentially damage the performance and make it unfair for different MLLMs, especially for those with poor instruction-following ability and those that tend to response with long sentences. 
As MLLMs inherit the in-context learning capability of LLMs, it is feasible to introduce in-context examples to force short responses. However, since different MLLMs contain different in-context learning capability, such practice interferes a fair evaluation of MLLMs' multimodal comprehension performance.

There thus is an urgent need for a metric that aligns well with human judgment and accommodates various response types while ensuring consistent evaluation despite variations in response morphology.





To compare different evaluators, traditional Semantic Textual Similarity (STS) task 
measures the difference between predicted scores and human annotation results from a single aspect of semantic relevance. However, both intuition and our experiments (refer to Section <ref> and <ref>) suggest that there is a significant difference between the evaluation of correctness in VQA responses and the traditional assessment of media-text relevance in STS. 

Therefore, to systematically evaluate the performance of an evaluator, with the unique characteristics in the task of VQA response evaluation taken into consideration, we propose three quantitative key properties, i.e., Alignment, Consistency and Generalization. Alignment stands for the overall correspondence of predicted scores with human annotation. Consistency measures how well an evaluator accommodates semantically equivalent responses of different morphology and length. Generalization indicates the variance of performance on different sources of data. Further, to facilitate comprehensive analysis of the performance and behaviour of evaluators, we provide a human-annotated dataset Assessing VQA Evaluators (AVE) that grades the correctness of model responses towards ground-truth labels on VQA datasets. AVE is further augmented by ChatGPT and WordNet <cit.> to increase the diversity.

As pilot experiment shows (refer to Section <ref>), formulaic metrics (BLEU <cit.>, ROUGH <cit.>, METEOR <cit.>) and model-based metrics <cit.> perform poorly on the VQA response evaluation task. Therefore, we propose a novel evaluator that is trained with meticulously designed pretraining tasks. The tasks are designed for improving the embedding representation of VQA text, which utilizes contrastive learning to guide the evaluator to capture the fine-grained difference within a text pair and ignore the noise in morphology and length.
Experiments demonstrate that the proposed pretraining tasks significantly improve the performance of our evaluator on the AVE dataset, making the evaluator's prediction aligns much better with human judgement.








The contribution of this paper can be concluded as follows:


    
    
    

    
  * This paper addresses the dilemma, where rich responses of MLLMs hinder fair evaluation under current metric, by proposing semantic-similarity-based evaluation that applies to various VQA responses.
    
  * This paper proposes three quantitative key properties in VQA response evaluation based on its characteristics, and a high-quality human-annotated dataset, AVE, for assessing different evaluators comprehensively. 
    In addition, we evaluate the performance of various types of existing semantic similarity evaluators on the proposed AVE dataset.
    
  * Experimental results demonstrate the feasibility of applying model-based methods to the flexible evaluation of VQA responses as well as the effectiveness of our proposed evaluator. Our evaluator significantly surpasses existing methods, including ChatGPT and the SOTA embedding model Voyage-lite-02-Instruct [As of the time of submission, Voyage-lite-02-Instruct achieves the best performance on the task of Semantic Textual Similarity (STS).] by a large margin. Our training scheme generalizes to both encoder-only and decoder-only models.

    





§ RELATED WORK




 §.§ Visual Question Answering
 

As the answer space of most open-ended VQA datasets is limited and the same answer applies for multiple questions (the most common 2,000 answers in the training set of VQA v2<cit.> is able to cover about 94% questions in its validation set), early methods <cit.> treat VQA as a classification task, which adopt answers in the training set as class labels, unable to predict unseen classes. Generative methods on VQA <cit.> treat VQA as a generation task and facilitates much more variant responses. 



 §.§ Semantic Textual Similarity
 

Current semantic evaluation tasks include Semantic Textual Similarity (STS) <cit.> that assesses to what extent the two sentences are related, Paraphrase Identification <cit.> that decides whether two texts express the same meaning, and Natural Language Inference <cit.> that determines the logical relationship between texts. The essence of these tasks lies in quantifying the degree of semantic equivalence between sentences, which is a fundamental challenge due to the complexity and variability of natural language. Methods in STS include formulaic methods like BLEU <cit.> and model-based ones <cit.>. The former mainly relies on n-gram or other statistic features between the candidate and reference to calculate the overlap and import penalty for noise. The latter utilizes models as encoders to extract the information and compare between the candidate and reference. Early model-based evaluator <cit.> compares the similarity of each pair each time, which is computation-consuming. Later works <cit.> first generate embeddings separately for the candidate and reference, then simply calculate the cosine similarity between them as the similarity score. Due to the style of STS, either formulaic or model-based methods pay more attention to the overall similarity and are less capable of detecting fine-grained semantic difference, as shown in our experiments. 



 §.§ Multimodal Comprehension Evaluation of MLLMs

As a developed realm and valuable resource of high-quality data, VQA has been applied in the evaluation of MLLMs. Several works <cit.> propose multiple-choice datasets to make evaluation simple and straight. For example, MME <cit.> proposes a smart quantitative analysis of MLLMs with manually designed instruction-answer pairs that strictly limit responses to be yes or no. Therefore, all MLLMs are evaluated relatively fairly. However, although MME is insightful and effective, such detour avoids the problem of evaluating open-ended response directly. It ignores previous huge amount of VQA data and costs additional human annotation, limiting the scale of the dataset and making it tough for expanding. MM-vet <cit.>, LVLM-eHub <cit.> and ConvBench <cit.> classify VQA into the integration of multiple key abilities, and manually annotates corresponding VQA samples of their required abilities. Then, they use ChatGPT for evaluation (which we show to be less capable in evaluating the correctness of open-ended responses, refer to Section <ref>). The classification of VQA abilities is insightful and aids to the probing of specific abilities of MLLMs. Yet MM-vet requires high-quality annotation to identify the VQA abilities each question requires and are thus limited to a small amount too.



§ SEMANTIC EVALUATION OF VQA

With the rapid development of MLLMs, current metrics in VQA response evaluation are too stubborn to assess the rich generation and hinder evaluating MLLMs' performance with existing VQA datasets. Meanwhile, as mentioned in Section <ref>, current semantic evaluation models and tasks are inconsistent with the goal of flexible VQA response evaluation.






Therefore, this paper proposes the task of semantic evaluation of VQA, aiming at introducing flexible similarity-based soft evaluation with continuous scores into the assessment of VQA responses, contrary to inflexible metrics like Exact Match or VQA Score that require identical morphology of responses towards ground-truth labels. Such flexible evaluation enables to assess the rich generation from MLLMs and thus enables to use existing VQA datasets for probing MLLMs' multimodal comprehension ability.







 §.§ Characteristics of VQA Evaluation
 

The proposed task of semantic VQA response evaluation shares significant difference with existing semantic evaluation tasks like STS and contains its own characteristics.



  
Discrimination Granularity As mentioned in Section <ref>, traditional semantic evaluation tasks typically focus on the overall meaning in texts, rather than capturing the fine-grained detailed difference. However, the core of semantic VQA evaluation is comparing the response with annotated answer under the same question[Considering the polysemy and ambiguity of words and phrases, the question text is indispensable for evaluating the semantic correctness.], where both texts share large overlap in meaning as the questions are same, demanding fine-grained semantic discrimination. 



  
Text Length As VQA answers are generally much shorter than texts in STS [About 97.9% answers in VQA v2 <cit.>, 97.7% in OKVQA <cit.>, 99.9% in GQA <cit.> are shorter than three words. The average length of text in STS-12<cit.> is 12.5, which is much longer.], n-gram based formulaic metrics like BLEU <cit.> will be more easily affected by the context in response. Model-based metrics are also vulnerable to such length shift, as their training data barely cover similar pattern.



  
Distribution Shift The texts in STS datasets <cit.> come from general domains, like news and social media, while different VQA datasets comply to different sub-tasks, like knowledge <cit.> or reasoning <cit.>. Such distribution shift causes inconsistent evaluation on responses from different VQA datasets. 



 §.§ Three Key Properties in VQA Evaluation
 
To systematically evaluate the performance of a VQA evaluator, we propose three quantitative key properties, i.e., Alignment, Consistency and Generalization.



  
Alignment Alignment assesses the overall performance of similarity scores predicted by evaluators with that of human annotation, in the metric of Spearman's Rank Correlation following similar setting in previous works<cit.>.


  
Consistency A smart evaluator shall catch the key information in responses and ignore the noise text, e.g., the response of elephants shall be scored equally with Theses are elephants under the question of What are the animals?. Therefore, Consistency measures how close the different responses sharing the same meaning are scored.


  
Generalization Considering various VQA datasets focus on various sub-tasks and come from various sources, Generalization depicts how well an evaluator is able to handle text from different domains. Refer to Section <ref> for quantitative definitions.



 §.§ A Dataset Assessing VQA Evaluators
 



To comprehensively compare and analyze the behaviour of different evaluators on VQA responses, taking the proposed three key properties into consideration, we propose a dataset Assessing VQA Evaluators (AVE). 
By collecting multiple MLLMs' responses on multiple datasets, the proposed dataset simulates a real scene of applying evaluators to evaluate the quality of various VQA responses. In order to compare the evaluators' scoring results with human judgement, we provide human annotation of the semantic correctness of responses towards ground-truth answers. The construction process of AVE is shown as follows:





  
Response Collection First, we collect responses of five models, LLaVA <cit.>, BLIP2 <cit.>, mPLUG-Owl <cit.>, OFA-large <cit.>, Qwen-VL <cit.> on the validation set of four datasets, OKVQA<cit.>, A-OKVQA<cit.>, VQA v2<cit.> and GQA<cit.> (balanced testdev set). 



  
Sampling Results Second, we sample in the responses while controlling the sampling amount of each dataset to be the same. In addition, samples that are answered correctly, i.e., the response is identical with the ground-truth answer, are excluded.



  
Human Annotation Third, three annotators are asked to measure the semantic similarity[We considered multiple aspects of measuring the correctness of a response towards the ground-truth answer, yet at last we come to the single aspect of semantic similarity for annotation. Refer to Appendix for more explanation.] of each sampled response towards the ground-truth label and annotate an integral similarity score from 0 to 10, under certain rules (refer to Appendix for more details). Then the scores are averaged over the three annotators.



  
Description Generation Fourth, in order to simulate MLLM responses with sentences instead of words or phrases, we select responses that are shorter than three words for augmentation [For longer responses, we manually condense it.]. The augmentation is conducted in two ways. The first way comes from using ChatGPT (refer to Appendix for prompts) to convert each pair of question and response into three descriptions and asking ChatGPT to select two descriptions that are closest to the original question-answer pair as augmented responses. For example, the question of What are the animals?  and the response of elephants are fed into ChatGPT, and it generates descriptions like The animals are elephants. The second way comes from applying manually designed answer templates (refer to Appendix) to increase the diversity of descriptions rather than fully relying on ChatGPT. Now each sample contains three descriptions, i.e., the two ChatGPT-augmented descriptions and a manual-template-fitted one.



  
Synonym Generation Fifth, we use WordNet <cit.> to locate a synonym for each answer. For cases where multiple synonyms exist, we choose the most common synonym by countering frequencies of words in Brown Corpus <cit.>. Meanwhile, we ask ChatGPT to introduce a shift in morphology to simulate cases that the outputs are merely different in tenses or singular or plural forms. The augmented answer is deemed to contain the same meaning with small disturbance on the style. 



  
Manual Filter At last, to ensure high-quality of the dataset, the three annotators also conduct manual filter (refer to Appendix for rules) to eliminate ambiguous samples, especially those generated from the fifth stage.

Generally speaking, the whole AVE dataset consists of three parts generated from above: Part 1 contains original answers and original responses (without augmentation). Part 2 contains original answers and the generated three descriptions of responses. Part 3 contains tense-shifted answers and original responses. The whole procedure is depicted in Figure <ref>. 

Meanwhile, the AVE dataset can also be clustered by the involved four datasets that each sample belongs to, OKVQA, A-OKVQA, VQA v2 and GQA, merging the Part 1 to Part 3 together and classifies by the sources of data only. Refer to Section <ref> for how they are used. 

The total sample amount of the final AVE dataset is 3,592, with each sample containing four types of augmentation results, as described above. The dataset is then split into a validation set and a test set with the ratio of 3:7. 
The distribution of annotated scores is shown in Appendix, which is relatively smooth. 
To evaluate the inter-annotator agreement, following previous works <cit.>, we apply Krippendorff's alpha <cit.> and obtain a result of 0.713.






 §.§ The Proposed Evaluation Indicators
 
In AVE dataset, a sample consists of a question q_i, a ground-truth label a_i, a source dataset label d_i, a response r_i and a human-annotated score s_i. The task of VQA response evaluation can be defined as: given a question-answer pair, an evaluator f(q_i, a_i, r_i) is expected to predict the annotated similarity score s_i with the output o_i.


    o_i = cos(f(q_i, r_i), f(q_i, a_i))



    score_f = Spearman(O, S)


where cos stands for cosine similarity, score_f is the performance score of the evaluator f, with O and S indicating the lists of all predicted scores and annotated scores respectively. Note that the metrics used for evaluating evaluators is Spearman's rank coefficient of correlation (Spearman).


The key properties of Alignment, Consistency and Generalization introduced in Section <ref> are computed as follows:


  
Alignment We use the average result of an evaluator on all parts of the proposed AVE dataset as alignment:


    Alignment = 1/N_Parts∑^N_Parts_i=1 score_f_i


where N_Parts is the number of Parts in the AVE dataset and N_sets is the number of involved VQA datasets in AVE, according to different type of division. The score_f_i is the spearman score of evaluator f on the i th part of AVE. 



  
Consistency Consistency measures how close the responses of the same meaning with different morphology are evaluated. We regard the variance of the same sample among the three parts of AVE as consistency:


    Consistency = log(1/( 1/N_samples∑^N_samples_j=1var(o_j_1, o_j_2, o_j_3)))


where N_samples is the amount of samples and var denotes calculating the variance. Then, o_j_1, o_j_2, o_j_3 are the predicted scores of the evaluator on Part 1, 2, 3 for the same sample j, respectively.



  
Generalization Generalization measures the difference of performance on various datasets, and we define it as the variance of the performance on each involved VQA dataset:


    Generalization = log 1/ var( align_OKVQA, 
    
    align_A-OKVQA, align_VQAv2, align_GQA )


where align_dataset is the mean Alignment score on the AVE data belonging to the corresponding VQA dataset. 




§ SEMANTICALLY FLEXIBLE VQA EVALUATOR



With the three key properties of an ideal evaluator taken into consideration, we propose a novel evaluator based on meticulously designed pretraining tasks.






 §.§ Pretraining Tasks

To guide the model to be sensitive to the key information between answer and response, this paper introduces several pretraining tasks to enhance the embedding. Data for augmentation come from a random sampling of VQA data in the training sets of OKVQA<cit.>, A-OKVQA<cit.>, TDIUC<cit.>, VG-QA<cit.>, GQA<cit.> and VQA v2 <cit.>, which end to a total amount of 105, 311 samples. All augmented samples are mixed for training.



  
NLI data In previous works <cit.>, models perform well with the natural language inference datasets SNLI <cit.> and MNLI <cit.>, where each sample includes a premise, an entailment and a contradiction.

In addition, these NLI datasets are all manually constructed, ensuring the high quality of their data, and the premise shares limited overlap with entailment compared with the sentence pairs in back-translation datasets. 
Therefore, to ensure the fundamental discriminating ability of models to capture overall meaning of sentences, we adopt NLI data and regard the premise-entailment pairs as postive pairs and premise-contradiction pairs as negative pairs.



  
Candidate answers To make the best of available VQA datasets, for datasets with ten candidate answers, OKVQA, A-OKVQA and VQA v2, we consider candidate answers as correct answers as well. Then, for each sample, the most common candidate answer and a less common one are used to form a positive pair, with a random answer sampled from the answer space as negative. 



  
Synonym and Antonym In VQA response evaluation, semantically similar answers shall receive similar scores. We replace the answer with a synonym by WordNet <cit.>, and if the antonym of an answer exists, we then pair up the answer and antonym as a negative pair, else we pair up the answer and a randomly sampled answer from the answer space as a negative pair. In addition, 

we use ChatGPT to produce synonyms as well, as ChatGPT is able to capture contextual information in the question and thus generates more accurate synonyms.




  
Generated descriptions To simulate the output of MLLMs, for each sample, we provide ChatGPT with its question and answer to generate three descriptions with small disturbance of the same meaning. Then, we construct positive samples by pairing up the original answer and each generated description. For negative samples, we replace the answer in the generated description with a randomly sampled answer. The goal is to pull the embedding representation of a natural language description close to its simple form of a single answer, so that responses with different length but carrying similar meanings will receive similar scores. In addition, the negative pair is constructed by replacing the key answer word in the description, therefore guiding the model to be sensitive to the key words and to ignore the noise.



 §.§ Model Framework


Following previous works <cit.> on the STS task <cit.>, we use cosine similarity for distance calculation between embeddings<cit.>.
As shown in Figure <ref>, we adopt the simple contrastive learning framework <cit.> and contrastive learning with in-batch hard negatives <cit.>. 


The backbone encoder in this paper is RoBERTa <cit.>. In order to gain better generalization and comprehension ability, we apply the decoder-only LLM LLAMA2 <cit.> with the prompt<cit.> of Summarize the text {text} in a single word:. Then, the hidden states of the first generated new token is considered as the embedding vector.



The contrastive learning with in-batch hard negatives loss <cit.> is defined as follows:


    loss_ibn = -loge^sim(h_i,h_i^+)/τ/∑^N_j=1(e^sim(h_i,h_j^+)/τ+e^sim(h_i,h_i^-)/τ)


where h_i is the embedding representation of sample i, h_i^+ and h_i^- respectively denote the representation of the positive sample and in-batch hard negative sample of sample i.





§ EXPERIMENTS




 §.§ Implementation Details

Experiments in this paper is based on transformer package<cit.> on Pytorch. We use AdamW <cit.> optimizer, and the hyper-parameters of AdamW, betas, eps and weight-decay are set to 0.9, 0.999, 1e-8 and 0.01. We use a cosine scheduler and the batch size and peak learning rate for encoders are 128, 1e-5 for RoBERTa-base <cit.>, VisualBERT <cit.> and LXMERT <cit.>, 32, 6e-6 for RoBERTa-large and 8, 4e-6 for LLAMA2 <cit.>. 



 §.§ Baselines
 
To assess to what extent existing models are competent for the VQA response evaluation, this paper collects four types of common methods for semantic similarity evaluation and refer to them as: formulaic, PLM, LLM, and API.

    
  * Formulaic methods contain BLEU <cit.>, ROUGE <cit.> and METEOR <cit.>. These methods base on n-grams for assessing the overlap. As VQA answers are usually short, we also report 2-gram results for BLEU and ROUGE. 
    
  * PLM refers to the Pretrained Language Models, which are generally small in sizes and typically in BERT-like encoder-only structures. 
    SBERT (Sentence BERT) <cit.> embeds texts into vectors with BERT and apply cosine similarity to measure the distance as textual similarity. 
    SIMCSE <cit.> provides both unsupervised and supervised methods, and this paper selects the supervised and better-performing one trained on NLI datasets for comparison. 
    BGE <cit.> follows a multi-task learning scheme that collects and pretrains on multifarious datasets for better generalization.
    AnglE <cit.> aims to mitigate the gradient saturation issue encountered when using cosine distance by projecting vectors onto the complex plane and introducing an angular loss. 
    
  * LLM refers to large language models. This paper selects four of the well-performing LLMs, Baichuan2 <cit.>, Qwen <cit.>, LLAMA-2 <cit.> and Mistral <cit.>.
    
  * API refers to the remote usage of models by API online, including ChatGPT and Text-embedding-v3-large from OpenAI, and Voyage-lite-02-instruct from Voyage AI. Refer to Appendix for prompts. The latter two are embedding models that produce text embedding of given text, which are then used to calculate similarity score by cosine distance.







 §.§ Main Experiments
 







Table <ref> exhibits the performance comparison of various evaluators. The results are on the test set of AVE, with specific scores on each part of the dataset, as described in Section <ref>. To promote a comprehensive assessment of existing methods, this paper compares the performance with four common types of methods for semantic evaluation, as introduced in Section <ref>. The last row of Types contains our results from training with the proposed pretraining tasks on the corresponding model. Then, the column of Alignment contains the separate results on each of the three parts in AVE datasets and their average.



  §.§.§ Performance of Formulaic Methods

Formulaic methods, i.e., BLEU, ROUGE and METEOR perform poorly in Alignment scores, and some of them drop below 0, indicating 

adverse scores to the human annotation. Such phenomenon is expected, as the n-gram matching strategy of BLEU and ROUGE is unable to handle the synonyms or variations in tenses and singular or plural forms. For METEOR, however, it applies port stem <cit.> and synonym matching to preprocess the 1-gram in both the candidate and reference, restoring words to stems and thus performs better. 

In addition, it is interesting to notice that although the Alignment of BLEU and ROUGE are much lower that that of METEOR, their Generalization scores are much higher. There are two reasons to this anomaly. First, BLEU and ROUGE fail to handle the task well and their prediction can be considered random, thus the sources of data do not affect the results, just like RoBERTa-large (w/o CL). Second, these n-gram evaluators do not involve semantics, therefore the sources of data that causes word distribution shift matter less. 



  §.§.§ Performance of PLMs



The BERT-like models pretrained for textual similarity prediction, i.e., SBERT, SIMCSE, BGE, AnglE (the latter four models), show much better performance than RoBERTa-large (w/o CL) and formulaic methods, indicating the basic textual similarity tasks are helpful to the VQA response evaluation task, but they fail to align well with human judgement, compared to SFVE results under the same structure of BERT-large. In addition, the performance on Part 1 and 3 of PLM models are similar, and the major gap lies in the capability of processing long responses. 




  §.§.§ Performance of LLMs

For LLMs (refer to Appendix for the detailed prompt) including ChatGPT, they fail to gain satisfactory results on AVE. Naturally, LLM performs better than RoBERTa w/o CL, and Generalization scores are slightly higher than PLMs, which we attribute to the better generalization ability of LLMs. Although LLMs obtain acceptable results on STS tasks, just like the formulaic methods, they encounter significant performance drop on the VQA response evaluation. Such phenomenon verifies the significant difference between the evaluation of STS and VQA responses and the necessity in the task of VQA evaluation.

The performance of embedding models (the latter two models) on STS is higher than LLMs but the VQA response evaluation performance is still low. The reason to their incompetence on AVE, as we speculate, is that these models focus more on retrieving and capturing the general meaning of given texts than discovering the fine-grained difference between given pair of texts. In addition, such focus of capturing the general meaning has also empowered them with the ability to ignore noise in morphology and text length, thus gaining high scores of Consistency despite the low Alignment scores.



  §.§.§ Performance of SFVE

The section of SFVE (ours) in the table presents our results on AVE. The pretraining tasks effectively improve the Alignment scores of all three models and bring moderately better Consistency and Generalization performance. From the prospective of model sizes, the 125M Roberta-base demonstrates similar capability with the 355M Roberta-large with merely a gap of 0.8%. The same applies for the 7b LLAMA2, which surpasses RoBERTa-large by 0.9%. Giant increase in model sizes brings minor improvement in scores. 
We believe the reason is that the similarity measure, either in STS or AVE, is relatively simple for models to comprehend and implement, where a simple structure with limited parameters achieves excellent performance with proper training. 

Therefore, for practical usage of evaluators during the training of generative VQA models, considering the significantly larger computation cost in LLAMA2-7b than RoBERTa, we recommend utilizing SFVE-RoBERTa base or large for a rough validation of model performance each certain steps or epoch, and use SFVE-LLAMA2 for more accurate evaluation near the best steps or epochs.





 §.§ Ablation Experiments


To analyze the influence of each pretraining task, Table <ref> provides ablation results by removing a pretraining task each time and by training on a single task alone. From the table it is clear that all pretraining tasks contribute to the final performance more or less. 

The most important task is Synonym and Antonym, which causes a drop of 16.0% in Alignment scores on average and damages Consistency as well. In addition, when trained only on such data, the model performs the best. We believe the importance of training on Synonym and Antonym task lies in aligning the representation of synonyms and increasing the difference towards antonyms and other answers.

The second influencing pretraining task is Generated descriptions, without which the model can not directly learn to align the representation between semantically similar texts with different length. Yet the removal of it does not substantially damage the results on other parts than Part 2, which consists of long responses.

Meanwhile, the removal of NLI data matters almost the same as Generated descriptions. As mentioned before, NLI data focuses more on the coarse-grained meaning between text pairs while AVE requires a finer semantic discrimination. However, for a model that barely handles the task (shown in the row of w/o All tasks), we believe the easier data in NLI aid to fertilizing the basic capability in semantic evaluation. Yet the NLI data alone is insufficient, as shown in the row of NLI data only.





 §.§ Practical Application

    
To demonstrate the practical values of our proposed evaluator in flexible VQA response evaluation, we collect responses of multiple MLLMs and compare the results with different evaluators by overall scores and case study.



As shown in Table <ref>, VQA score is clearly incompetent for assessing assorted responses from MLLMs. Since all responses from mPLUG-Owl are sentences, VQA Score even comes to 0. In the comparison of LXMERT and mPLUG-Owl, both BGE and SBERT indicate LXMERT generates better responses than mPLUG-Owl. However, taking the case study in Figure <ref> into consideration, we verify that existing well-performing methods, BGE and SBERT, fail to perform consistent evaluation and bias towards short responses while penalizing longer ones. For example, in (a) of Figure <ref>, LXMERT response pan receives a much higher score than the mPLUG-Owl response which is a descriptive sentence containing the correct answer. In (d), the descriptive text and the single word response receive similar scores under our SFVE, but SBERT considers the short answer of LXMERT is much better than the descriptive sentence of mPLUG-Owl. Similar phenomena exist in BGE as well. As in (e), mPLUG-Owl response describing garbage collector is a better response than LXMERT output police, yet the latter receives even higher scores. 

In addition, not only does the length impede a fair evaluation, but the incompetence in fine-grained semantics discrimination also causes absurd results. Like in (c) and (g), where LXMERT answers are less correct but they receive competitive or even higher scores than reasonable responses from BGE and SBERT. Such error, we speculate, is caused by focusing more on the overall meaning of text, as the questions are the same within a pair, while neglecting fine-grained difference.

Due to the phenomena above, it is clear the superficial superiority of LXMERT over mPLUG-Owl is merely a mistake by incompetent VQA evaluators, which also demonstrates the importance of fairness and consistency in VQA evaluation. Therefore, we consider the proposed pretraining tasks and SFVE effective, not only on our proposed AVE dataset, but also in practical application where previous methods fail to perform fair and insightful evaluation. 



§ CONCLUSION

This paper proposes a practical task of utilizing semantic correctness to evaluate unconstrained open-ended VQA responses, facilitating the assessment of MLLMs' multimodal comprehension abilities by VQA data.
We propose three key properties for assessing VQA evaluators, i.e., Alignment, Consistency and Generalization.
In addition, this paper proposes a new dataset assessing VQA evaluators (AVE) to comprehensively analyze multiple aspects of evaluators. Based on contrastive learning with meticulously designed pretraining tasks, this paper provides a Semantically Flexible VQA Evaluator (SFVE) that performs significantly better than existing evaluators on VQA evaluation and the training scheme generalizes to both the encoder-only and decoder-only models.









ACM-Reference-Format









§ TRADITIONAL VQA EVALUATION METRICS

Traditional VQA evaluation metrics contain Exact Match <cit.> and VQA Score <cit.>. They apply for different settings in VQA datasets. For datasets where each sample contains only one correct answer, like DAQUAR <cit.>, TDIUC <cit.>, GQA <cit.>, the Exact Match is used. If each sample contains ten candidate answers, like VQA v2 <cit.>, OKVQA <cit.>, VizWiz <cit.>, VQA Score is commonly used.



  
Exact Match Exact Match calculates by judging whether the response is identical to the annotated ground-truth answer, and if matches, the score is 1, otherwise 0.



  
VQA Score VQA Score evaluates how many times the response appear in the ten candidate answers, and is computed as follows:

    accuracy = min(# correct hits/3,1)

As there are ten candidate answers, # correct hits represents numbers of matched answers, which means as long as there are three or more candidates are the same with the predicted answer, the answer will be considered fully correct, and gets a score of 1. 



§ PROMPT FOR DECODER MODELS

The following is the similarity calculation prompt provided to the LLMs and ChatGPT in the experiments:

Sentence similarity evaluation here refers to the task of measuring the semantic similarity score between two sentences. For example, "what a good day" and "how nice the weather is" are almost the same, your output shall be {"score":0.91}. Now please evaluate the similarity score between the following two sentences: sentence1: sample["sentence1"]. sentence2:  sample["sentence2"]. The score shall range continuously from 0-1. DO NOT output anything else but the .json-style dictionary, like {"score":x}, where x is your predicted score.

The sample["sentence1"] and sample["sentence2"] indicate a pair of texts for similarity calculation. The question and answer are concatenated with the prompt "Question: {question} Answer: {answer}" before similarity calculation, importing contextual information, just the same as other models in experiments.

ChatGPT prompt for converting each question-answer pair into a description:

Concatenate the question with the answer and form assertions. For example, Question:What kind of dog is in the photo?  Answer:golden retriever.  Assertion: The dog in the photo is a golden retriever. Infer for the following: Question: {question} Answer: {answer}. Please think of three different forms of naturally-sounded assertions for this question-answer pair with small disturbance but do not output them. Choose the two assertions that are closest in meaning to the original question-answer for output. Output shall be in .json style so that I can directly save them in a .txt and open by json. Do not output anything else including explanation, reasoning or instructions.







§ ANNOTATION SCORE DISTRIBUTION




The score distribution of the dataset Assessing VQA Evaluators (AVE) is shown in Figure <ref>. The annotation covers all scores.




§ SELECTION OF ANNOTATING ASPECT


For the evaluation of the quality of responses, we considered adopting multiple aspects for analyzing. The text summary task <cit.> adopts four aspects, i.e., relevance, consistency, fluency and coherence for analyzing the generated summary based on the reference text. As VQA responses are generally short, the fluency and coherence that measure the fluency of the text is less necessary. The aspect of consistency measures whether the generated summary contains hallucination that generates untrue information. In the scene of VQA response evaluation, such measure corresponds to the overlap in semantics from the response towards the ground-truth answer, which is similar to relevance that measures how well the generated text captures the key points. Therefore, we decide to use semantic similarity as the only score of annotation.



§ ANNOTATION RULES




 §.§ Scores Annotation

All of the following in this section is the annotation rules provided to the three annotators during the annotation of scores. 

Scoring Format: Discrete integer scoring from 0 to 10 (e.g., 0, 1, 2, 3).

Scoring Rules:

Note that it is not about judging whether the response to the question is correct, but whether the response and the standard answer are semantically the same under the question (For example, even if the standard answer is obviously unreasonable, as long as the answer and response are semantically similar, a high score shall be given).

Semantically similar (the answer is fairly correct in meaning) but different in specific form (for example, the meaning expressed is similar but different in word choice, tense, number), score 6-10 based on the degree of semantic similarity.
Examples:

(1) Question: What is the last letter on the license plate?
Standard Answer: letter j
Response: j
Scoring (this is a reasonable range, just mark a specific score when actually annotating): 9-10
Reason (no need to mark, this is to help understand the rules): Under this question, the response and the standard answer are semantically the same, only the form is different. Similarly, synonyms should also be scored highly.

(2) Question: The young man above the swimming pool is wearing what?
Standard Answer: swimsuit
Response: trunks
Scoring: 7-8
Reason: The question asks what is being worn, and swimsuit (swimwear) includes trunks (swim shorts), so the answer is quite correct. This kind of inclusive or included relationship should be scored based on the semantic similarity of the two words. In addition, trunks as swim shorts is a less common meaning and requires more attention to the different meanings of words, not entirely based on experience.

(3) Question: How many trees are there?
Standard Answer: 3
Response: three
Scoring: 10
Reason: The meanings are exactly the same.

Semantically dissimilar, the answer is incorrect, but the answer is a possible answer for that type of question, score 1-5 based on the degree of semantic similarity.
Examples:

(1) Question: What color do you think the trousers the boy is wearing have?
Standard Answer: white
Response: blue
Scoring: 2-3
Reason: The question asks about color, and although the answer blue is different from the standard answer white, both are common answers under the category of color questions. Furthermore, if it's white and black, the difference between the two is greater than between white and blue, so the scoring range should be further reduced to 1-2.

Semantically dissimilar, but the answer and the standard answer mean the same under the question, then score 4-7 based on the correctness.
Examples:

(1) Question: What is lit?
Standard Answer: cake
Response: candle
Scoring: 5-7
Reason: Although cake and candle are very different semantically, in this question, they actually mean the same thing. Therefore, one should not only look at the answer but also focus on the question.

For numerical type answers, score 1-8 based on how much the number in the standard answer and the number in the response differ.
Examples:

(1) Question: How many trees are there?
Standard Answer: 4
Response: 5
Scoring: 3-5
Reason: Although 4 and 5 are different, the difference between them is not particularly large. If the standard answer is still 4, but the response becomes 1, then the scoring should be appropriately lowered to 1-3. If the standard answer is 70, and the response is 75, then it can be considered quite correct, scoring 5-7. If the standard answer is 1 and the response is 0, then score 1-2. Judge the score based on whether the numbers are relatively close to each other.

(2) Question: When did this accident happen?
Standard Answer: 1945
Response: 1940
Scoring: 5-7
Reason: The two years are quite close. But if the answer becomes the 1940s (i.e., 1941-1949), it includes 1945, and the range is not particularly large, so it is quite correct, scoring 6-8.

For answers with significantly different meanings, score based on semantic similarity without range restrictions.
Examples:

(1) Question: What color bathing suit is the woman wearing?
Standard Answer: no woman
Response: red
Scoring: 0-1
Reason: The meanings are very different.




 §.§ Manual Filter


The following is the annotation rules provided to the three annotators during the last stage, manual filter, in the construction of our AVE dataset.

Read the question, answer, response and all augmentation results carefully, and decide whether the augmentation has changed the original meaning. Labels shall be in yes, no, unsure.



§ ANSWER TEMPLATES

In the fourth step of constructing the proposed dataset, i.e., description generation, beside collecting ChatGPT-transformed results, we augment each short response with manual written templates: (1) Answer: {response}., (2) The answer to this question is {response}., (3) As shown in the image and question, the answer is {response}., (4) The answer you are asking for is {response}., (5) As can be deduced from the image, the answer to this question is {response}., (6) The answer to your question appears to be {response}, as shown in the image..



 Paper Title: Lifelong Person Search.








Lifelong Person Search
    Jae-Won Yang, Seungbin Hong, and Jae-Young Sim, Member, IEEE

    August 5, 2024
=================================================================



§ ABSTRACT 
  Person search is the task to localize a query person in gallery datasets of scene images. Existing methods have been mainly developed to handle a single target dataset only, however diverse datasets are continuously given in practical applications of person search. In such cases, they suffer from the catastrophic knowledge forgetting in the old datasets when trained on new datasets. In this paper, we first introduce a novel problem of lifelong person search (LPS) where the model is incrementally trained on the new datasets while preserving the knowledge learned in the old datasets. We propose an end-to-end LPS framework that facilitates the knowledge distillation to enforce the consistency learning between the old and new models by utilizing the prototype features of the foreground persons as well as the hard background proposals in the old domains. Moreover, we also devise the rehearsal-based instance matching to further improve the discrimination ability in the old domains by using the unlabeled person instances additionally. Experimental results demonstrate that the proposed method achieves significantly superior performance of both the detection and re-identification to preserve the knowledge learned in the old domains compared with the existing methods.



Person search, person re-identification, lifelong learning, and continual learning.





§ INTRODUCTION

Person search is the technique to find the query person from the gallery sets of scene images where multiple persons usually appear simultaneously in each image. It has been drawing much attention due to its practical applicability to various real-world scenarios such as large-scale video understanding, surveillance, and augmented reality.
Different from the person re-identification (re-ID) <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.> that finds the query person from the sets of cropped person images, the person search is a more challenging task that first localizes the bounding boxes of person instances in the scene images and then matches the identities of the detected instances to the query person. The person search can be implemented in the two-step manner by using separately trained two sub-networks of the object detection and re-ID. However, the training of the two-step methods is usually inefficient requiring huge computational complexity, since the detection network extracts the bounding boxes for the person instances from the scene images which are then inputted to the re-ID network to retrieve the features again tailored to the re-ID task. 


To overcome this issue, the end-to-end learning was introduced that jointly trains the person detection and re-ID networks. The end-to-end methods have been mainly developed in the supervised learning manner based on the assumption that both of the training and test data come from a same target dataset <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>. However, in many practical real-world applications, multiple datasets are generated in different places and times that exhibit domain gaps from one another. In such cases, the existing supervised methods trained on a certain dataset usually fail to work on other datasets. Furthermore, re-training the network, whenever the target datasets are changed, suffers from the high computational complexity as well as the catastrophic forgetting <cit.> of the knowledge learned from the previously trained datasets.



In this paper, we first introduce a new problem of lifelong person search (LPS) where the new datasets on different domains are assumed to be sequentially given in order, as shown in Fig. <ref>. The model is forced to be generalized to all domains while preserving the previously learned knowledge without entire re-training using all the datasets. 
The end-to-end LPS is more challenging compared to the lifelong object detection <cit.>, <cit.> and lifelong person re-ID <cit.>, <cit.>, <cit.>, <cit.>, since it suffers from the catastrophic forgetting problem in both sub-tasks of the person detection and re-ID.
Whereas the lifelong person detection is a domain-incremental task where only the same class of person is localized across different domains, the lifelong person re-ID is related to both domain-incremental and class-incremental tasks since the new person identities are additionally given from different domains.
Moreover, the end-to-end person search also suffers from the task conflict problem where the person detection focuses on extracting the common representation of persons distinct from the backgrounds, but the person re-ID attempts to extract the unique representations according to the person identities. This task conflict problem becomes more serious in LPS scenario where the model is encouraged to be continuously adapted to different domains. 
In addition, the lifelong re-ID methods <cit.>, <cit.>, <cit.>, <cit.> have been usually developed for full-body pedestrian images on similar domains. 
However, as shown in Fig. <ref>, the LPS considers the scene images with severely different characteristics across multiple domains, for example, diverse backgrounds, different scales and densities of persons, and even local body parts due to partial occlusion. 

To address the LPS problem, we propose a novel end-to-end framework that generalizes the network to be incrementally adapted to the new domains while preserving the previously learned knowledge in the old domains.
Specifically, we perform the knowledge distillation for both sub-tasks of the person detection and re-ID using the old exemplar data. We first use the prototypes, representative features of person identities, associated with the old exemplar data, to design the rehearsal-based re-ID knowledge distillation loss that enforces the consistency on the distributions of the feature similarity between the old and new models. 
Moreover, we also utilize the hard background proposals additionally to refine the re-ID knowledge distillation loss that alleviate the effect of inaccurately detected person proposals and extract discriminative features for re-ID more reliably.   
In addition, we devise the rehearsal-based instance matching loss to further improve the model's discrimination ability. We minimize the feature discrepancy between the labeled proposals in the old exemplar data and its ground truth old prototypes. We also employ the unlabeled person identities in the old domains as negative samples to preserve the knowledge effectively. Experimental results demonstrate that the proposed method preserves the knowledge of old datasets more faithfully compared with the existing methods, and therefore serves as a very promising tool for LPS.   

The main contributions of this paper are summarized as follows.

	
  * To the best of our knowledge, we first introduce a new and challenging problem of person search with lifelong learning scenario where the model is incrementally trained on the new domains while preserving the previously learned knowledge in the old domains. 
	
  * We propose an end-to-end LPS framework that jointly trains the detection and re-ID networks by using the rehearsal-based knowledge distillation loss and the instance matching loss, that alleviate the catastrophic forgetting of the old knowledge during the training on the new domains.
	
  * We demonstrate the efficiency of the proposed method by providing comprehensive experimental results compared with the existing methods based on the lifelong learning scenario.






§ RELATED WORKS



 §.§ Person Search

Person search has been studied mainly in the supervised manner where the bounding boxes of person instances and the person identities are labeled in the training datasets.
The two-step methods train the person detection and re-ID networks separately to prevent the conflict problem between the two tasks. Zheng et al. <cit.> conducted extensive experiments by training the state-of-the-art methods of the pedestrian detection and person re-ID. They also provided a benchmark PRW dataset. Pu et al. <cit.> proposed a segmentation masking scheme to force the re-ID network to focus on the foreground regions of the detected persons. Lan et al. <cit.> extracted multi-scale features to deal with the scale variation problem of person size in the scene images. Wang et al. <cit.> made the re-ID network more adapted to the detection results by composing the training set with the person images cropped by the pre-trained detection network and the person images cropped by using the bounding box labels. Ke et al. <cit.> performed a data augmentation scheme that shifts the locations of the ground truth bounding boxes for the re-ID network training.

The end-to-end methods jointly train the person detection and re-ID networks. Xiao et al. <cit.> firstly proposed an end-to-end person search network and provided a benchmark CUHK-SYSU dataset. Chen et al. <cit.> employed the background features as negative samples to train the re-ID network. Chen et al. <cit.> separated the feature embedding into the norm and angle which are used as a detection confidence score and an identity feature, respectively.
Zhang et al. <cit.> pretrained an external re-ID network which is then used as a strong teacher model to supervise the re-ID network based on the knowledge distillation framework. Li and Miao <cit.> employed an additional Faster R-CNN header sequentially to extract the superior identity features from the high-quality person proposals. Han et al. <cit.> adaptively controlled the gradient backpropagation to train the sub-networks of the re-ID and part classification according to the quality of detection results. Lee et al. <cit.> suggested a feature standardization scheme and a localization aware memory updating scheme to alleviate the effect of class imbalance and inaccurately detected proposals, respectively.
The transformer architectures were also employed to improved the performance of person search <cit.>, <cit.>, <cit.>, <cit.>. 
Recently, Oh et al. <cit.> assumed the training data of real target domains are not available, and proposed a domain generalizable person search method that uses only an unreal dataset for training.

On the other hand, the weakly-supervised person search has been introduced that uses the labeled bounding boxes only without using the identity labels for training <cit.>, <cit.>, <cit.>, <cit.>. Moreover, domain-adaptation methods have been proposed to address the unsupervised person search problem where both of the bounding box and identity labels are not available <cit.>. Note that the existing methods of person search have been usually developed considering a single target dataset only, and hence suffer from the catastrophic forgetting problem where new target datasets are continuously given in the lifelong learning scenario. 






 §.§ Lifelong Object Detection

Lifelong object detection methods are classified into the class-incremental approach and the domain-incremental approach. 
The class-incremental object detection considers a certain target dataset where the new object classes are incrementally added. Shmelkov et al. <cit.> first introduced the problem of catastrophic forgetting in the object detection. Adaptive distillation has been performed between the intermediate features and the output of the region proposal network based on the end-to-end framework <cit.>. Shieh et al. <cit.> stored a subset of old data into the exemplar memory to alleviate the catastrophic forgetting in old classes. Liu et al. <cit.> focused on the most informative old knowledge by sampling the most reliable foreground prediction from the old model, which are then used as pseudo labels in a transformer based detection network, DETR <cit.>. Dong et al. <cit.> performed self-supervised learning with the DETR network where only a few labeled new object classes appear in the new data.
On the other hand, object detection datasets are associated with different domains according to the variations of background, lighting, and camera viewpoint, even though they contain the same object classes. The domain-incremental object detection assumes incrementally added new domains with the same object class. Li et al. <cit.> used a transformer-based feature extractor to adaptively apply the classification head network to each newly added domain. Mirza et al. <cit.> stored the statistical changes across the domains used to perform the task on the corresponding specific domain. 



 §.§ Lifelong Person Re-ID

Wu and Gong <cit.> first introduced the lifelong person re-ID problem and performed coherence learning for classification, distribution, and representation, respectively. Pu et al. <cit.> adaptively accumulated the knowledge of old domains via the instance-based similarity to improve the generalization ability. Ge et al. <cit.> developed a domain adaptation framework that reduces the gap between the old and new domains by using the augmented new data following the distributions of the old domains. Sun and Mu <cit.> selected diverse and important patches from images by using a differentiable patch sampler to preserve both the local and global relational knowledge. Huang et al. <cit.> developed a relation consistency learning to encourage the new model to return the consistent results of similarity ranking to that of the old model. Yu et al. <cit.> proposed a knowledge transfer scheme via the bi-directional learning that dynamically updates the old model while training the new model. Xu et al. <cit.> used a pair-wise relation matrix to filter out the erroneous knowledge of the old model and transfer the refined knowledge to the new model.




§ PROPOSED METHOD


Fig. <ref> shows the overall architecture of the proposed end-to-end LPS framework. We use the SeqNet <cit.> as a baseline network which consists of the Faster R-CNN <cit.> and the NAE (norm-aware embedding) <cit.> header. 
Let us assume that a sequence of person search datasets in different domains are given in order, as D_1 → D_2 →…→ D_N. The model is trained by using the first dataset D_1.
When the new dataset D_2 is given, we regard the model trained on D_1 as the old model, and construct a new model by replicating the old model.
Then the new model is trained by using D_2 and a small subset of D_1, called exemplar data, to avoid the knowledge forgetting of D_1. We also use the representative features of person identities, called prototypes <cit.>, stored in the old look-up table (LUT) .
Whenever a new dataset D_N is available, the old model is replaced with the new model, and the new model is re-trained by using the new data D_N as well as both the old exemplar data and the old prototypes selected from {D_1, ⋯, D_N-1} to mitigate the catastrophic forgetting.
We use a small subset of the old data following the typical rehearsal (replay) based methodology of lifelong learning <cit.>. 
However, it is worth to note that we do not employ multiple old models but always have a single old model which is updated whenever a new dataset is given. The old model conveys the knowledge of the previous domains and thus the parameters of the old model are frozen during the training of the new model.
We preserve the knowledge of the old data while training the model using the new data via knowledge distillation between the old and new models. 






 §.§ Re-ID Knowledge Distillation



  §.§.§ Prototype-Based Distillation

Existing methods of lifelong person re-ID <cit.> perform the knowledge distillation by matching the distributions of the feature similarity between the old and new models within a mini-batch. However, it may not provide faithful results when applied to LPS, since a mini-batch is composed of relatively small numbers of scene images, and small numbers of identities accordingly, due to the memory constraints. Furthermore, multiple person instances with the same identity are rarely included in a single mini-batch according to the uniqueness prior <cit.>. 

To address this issue for LPS, we utilize the stored prototypes of person identities as informative guidance for the re-ID knowledge distillation.
The prototype is computed by aggregating the features of diverse person instances with the same identity <cit.>. 
Let  denotes the set of the foreground proposals in the exemplar data of a single mini-batch, that is detected by the Faster R-CNN of the new model. Let ^old_i and ^new_i be the L2-normalized features of the i-th proposal in , that are extracted through the NAE headers of the old and new models, respectively. We estimate the target distribution of the feature similarity of ^old_i compared to all the prototypes in , such that the probability q_i,k associated with ^old_i and _k, the k-th prototype in , is given by 

    q_i,k = exp(_k^T^old_i  / τ_d)/∑_∈exp(^T^old_i / τ_d).

We also estimate the predicted distribution of the feature similarity of ^new_i compared to all the prototypes, such that the probability p_i,k associated with ^new_i and _k is given by

    p_i,k = exp(_k^T^new_i  / τ_d)/∑_∈exp(^T^new_i / τ_d).


Note that the predicted similarity distribution with respect to the prototypes in the old domains changes when the model is trained on the new domain, which could cause the forgetting of the re-ID knowledge learned on the old domains. Therefore, we train the new model to generate a more consistent distribution to the target distribution by employing a prototype-based re-ID knowledge distillation loss given by

    ℒ_rkd = 1/||1/||∑_i ∈ I()∑_k ∈ I()  q_i,klogq_i,k/p_i,k,

where I(·) means the index set. 
By minimizing ℒ_rkd, the new model is trained to yield a more consistent distribution to the target distribution with respect to the prototypes in the old domains, and eventually extracts unique and representative features of all person identities alleviating the catastrophic forgetting in re-ID.  





  §.§.§ Hard Background Proposal-Based Distillation

Though the prototypes in the old LUT serve as a good prior for the re-ID knowledge distillation, we further improve the performance by using the background proposals in the old domains additionally. At each iteration, the new model detects the background proposals from the scene images in the exemplar data, as depicted in the red boxes in Fig. <ref>. Inaccurate background proposals are often generated that partially overlap with the foreground person instances. We refer them as the hard background proposals. The hard background proposals convey the partial information of the person identities, exploited to improve the discrimination performance of the person identities. 


Specifically, we sample the hard background proposals that have the higher intersection over union (IoU) scores than a certain threshold λ_ b, with respect to the ground truth bounding boxes of the foreground persons, as depicted in the blue boxes in Fig. <ref>. Then we store the re-ID features of the hard background proposals into the feature memory .
We re-compute the distributions of the feature similarity compared to all the prototypes in  as well as all the features of the hard background proposals in , such that the probabilities q_i,k^+ and p_i,k^+ associated with _k, the k-th element in ∪, are given by 

    q_i,k^+ = exp(_k^T^old_i / τ_d)/∑_∈{∪}exp(^T^old_i / τ_d),


    p_i,k^+ = exp(_k^T^new_i / τ_d)/∑_∈{∪}exp(^T^new_i / τ_d).












Accordingly, we have the refined re-ID knowledge distillation loss as

    ℒ_rkd^+ = 1/||1/|∪|∑_i ∈ I()∑_k ∈ I(∪)  q^+_i,klogq^+_i,k/p^+_i,k.

Consequently, we improve the discrimination performance of the person identities by exploiting more rich information carried by the hard background proposals. Furthermore, the features learned by additionally using the hard background proposals are more robust against the inaccurately detected person proposals, which alleviates the task conflict problem between the detection and re-ID even when the detection knowledge in the old domains is forgotten.



 §.§ Rehearsal-Based Instance Matching

The foreground and background proposals extracted from the exemplar data are used for consistent learning between the old and new models via the re-ID knowledge distillation. Note that, as depicted in the green boxes in Fig. <ref>, some foreground person instances have no identity labels. Such unlabeled instances can also serve as the negative samples for all the labeled identities to learn the discriminative feature representations. At the same time, the new model should be guided to minimize the feature discrepancy across the person instances in the exemplar data that have the same identity. Therefore, we also utilize the unlabeled instances in the exemplar data to further capture the re-ID knowledge in the old domains while the model is trained on the new data. 



The features of the unlabeled proposals are stored in the old circular queue . Let _ L denote the set of the labeled proposals in , and let ^new_i be the feature of the i-th proposal in _ L extracted by the new model. We compute the probability that ^new_i is classified into its ground truth label as

    ρ_i = exp((i)^T^new_i  /τ_r )/∑_∈exp(^T^new_i / τ_r)  +  ∑_∈exp(^T^new_i  / τ_r),

where (i) means the prototype of the ground truth identity of the i-th proposal in _ L, and  denotes the feature of the unlabeled proposals stored in . We train the new model to increase the classification score of the extracted features by employing the rehearsal-based instance matching loss given by

    ℒ_rim = -1/|_L|∑_i ∈ I(_L)logρ_i.

By minimizing ℒ_rim, we reduce the feature discrepancy between the labeled proposal and its ground truth identity while preserving the discrimination performance in the old domains with the help of the unlabeled proposals. 
It is worth to note that the conventional OIM <cit.> loss considers the labeled identities and the unlabeled instances in a single target domain only. On the contrary, the proposed rehearsal-based loss ℒ_rim employs the labeled identities and the unlabeled instances across the old data, aiming to preserve the discrimination performance in the old domains for lifelong learning purpose. 




 §.§ Training and Inference

At the training phase, both of the person detection and re-ID networks are trained in the end-to-end manner. Note that the baseline network of SeqNet <cit.> also uses the losses of ℒ_det and ℒ_oim when training the new model by using the new dataset. To preserve the knowledge of the old domains in terms of the person detection, we additionally use the detection knowledge distillation loss ℒ_dkd of the existing lifelong object detection method <cit.>.
Finally, the total loss function is given by 

    ℒ_total = ℒ_dkd + ℒ_rkd^+  + ℒ_rim + ℒ_det + ℒ_oim.


After the new model is trained with the last dataset D_N,  we discard the old model and only utilize the new model to detect the bounding boxes of the person instances and extract the re-ID features for all the datasets {D_1, D_2 … D_N} at the inference phase.




§ EXPERIMENTAL RESULTS




 §.§ Experimental Setup



  §.§.§ Datasets

We used the three datasets to evaluate the performance of the proposed lifelong person search method. CUHK-SYSU <cit.> and PRW <cit.> are widely used for the person search task. The CUHK-SYSU dataset includes the images obtained from the street snapshots and movies, with the annotations of the bounding boxes and person identities. We set the gallery size to 100. The PRW dataset is composed of the video frames capturing a university campus by six different cameras. We also use a recently released large-scale person search dataset of MovieNet-PS <cit.> gathered from the 385 movie sequences. The MovieNet-PS is a challenging dataset since it includes the scene images with diverse backgrounds, illuminations, and poses of persons to reflect more realistic and challenging scenarios of person search. Moreover, there are many persons partially appearing due to the occlusion, and the person instances with the same identity often wear different clothes. The statistics of the three datasets are shown in Table <ref>. 








  §.§.§ Evaluation Metrics

We evaluated the performance of the person detection and re-ID, respectively, based on the LPS framework. We used the recall and the average precision (AP) to measure the detection performance. The recall calculates the percentage of the true positive bounding boxes where the IoU scores with respect to any ground truth bounding box are higher than 0.5.
The AP computes the average precision of the bounding boxes by measuring the area under the Precision-Recall curve using the IoU scores with respect to the ground truth.
We also used the mean Average Precision (mAP) and the Top-k scores for the re-ID. The mAP calculates the averaged precision of searching a query from the gallery images. It measures the area under the Precision-Recall curve using the feature similarities between the query and all the detected gallery persons. The detected bounding boxes that overlap with the ground-truth with the IoU scores higher than 0.5 are set as the true positives. The Top-k score checks whether at least one of the k-most similar candidates is a true positive or not. We adopted the Top-1 score in this work.



  §.§.§ Implementation Details

We uniformly sampled 2% of the training data from the old datasets to construct the exemplar data, as did in many literatures of lifelong person re-ID <cit.>. 
We store the prototypes of the persons in the exemplar data only into the old LUT, which are not updated during the training on the new domain.
For the model training, we set the batch size for the exemplar data to 2 for each old domain and 5 for the new domain, respectively. We resized the input image to 1500 × 900 and applied the random horizontal flipping. We set the size of the old circular queue  as 1000.
We trained the model until it reaches the highest performance on the first domain, and trained the model for 5 epochs each on the other domains. Since our baseline setting achieves the best performance when the model is trained during 5 epochs for each new dataset, we also trained the model for 5 epochs on the new dataset for fair comparison to the baseline setting.
The initial learning rate is set to 0.003, which is warmed up with a learning rate scheduler and further decayed by the value of 0.1 in the 3rd epoch of each new domain. For the stochastic gradient descent, we set the momentum value to 0.9 and the weight decay to 0.0005. λ_ b, τ_d, and τ_r are empirically set to 0.1, 0.3 and 0.1, respectively. All our experiments were implemented using PyTorch and a single NVIDIA TITAN X GPU.






 §.§ Lifelong Learning Performance

We evaluate the lifelong learning performance of the proposed method with the training order of CUHK-SYSU → PRW → MovieNet-PS in Table <ref>. We first compare two different methods: Joint-Train and FineTune, implemented on our baseline network. The Joint-Train method trains the model by using all the available datasets simultaneously. The FineTune method trains the model on each dataset in order, where the model is initialized with the previously trained weights and then re-trained by using the new dataset. At the inference phase, the performance is evaluated on every dataset by using the model trained on the last dataset.

The Joint-Train method achieves the best performance since it uses all the training datasets at once, however, it requires a huge burden of the computation as well as the storage space. We observe that the proposed method provides comparable results to the Joint-Train method and outperforms the FineTune method in terms of the averaged mAP and Top-1 score, respectively.
Note that the FineTune method sequentially re-trains the model on each dataset without using the previous old datasets, and thus its performance on the last dataset, MovieNet-PS, is relatively high. On the contrary, the proposed method uses a small subset of the old data to alleviate the knowledge forgetting and slightly degrades the performance on the last dataset compared to the FineTune method. However, the proposed method significantly outperforms the FineTune method in the old datasets of CUHK-SYSU and PRW. 


Fig. <ref> shows the performance evaluated on the old datasets of CUHK-SYSU (blue) and PRW (red), when the model is sequentially trained on the new datasets in the x-axis, in order.
As shown by the dashed lines, both the AP and mAP scores are decreased in the FineTune method showing the knowledge forgetting effect of both the detection and re-ID tasks in the LPS scenario. However, the proposed method significantly mitigates such performance degradation as shown by the solid lines, and successfully preserves the old knowledge for LPS.

In addition, we show the performance of the proposed LPS method when the model is trained with different orders of the datasets. In Fig. <ref>, Order1 and Order2 represent the training orders of `MovieNet-PS → CUHK-SYSU  → PRW' and  `PRW → CUHK-SYSU → MovieNet-PS,' respectively. 
In both orders, we see that the proposed method still outperforms the FineTune methods in terms of the detection and re-ID performances, which indicates that the proposed method provides reliable performance regardless of the training orders.















 §.§ Comparison with Two-Step Methods

Note that we first introduce the new problem of LPS in this paper, and there is no existing method fairly comparable to the proposed method. We attempted to conduct the additional comparative experiments by using the recent lifelong re-ID methods of AKA <cit.>, PTKP <cit.> and KRKC <cit.>. The lifelong re-ID methods work on the cropped person images only and cannot be directly applied to our LPS framework that considers the scene images. We instead implemented the two-step person search framework by using the existing lifelong person re-ID methods, where the detection and re-ID networks are trained separately. We used the ResNet-50 as the backbone network for the compared methods. We also trained the detection network by using the detection knowledge distillation loss ℒ_dkd for fair comparison. 

Table <ref> compares the quantitative performance of lifelong learning where we see that the proposed method outperforms the two-step methods of `Det + AKA,' `Det + PTKP,' and `Det + KRKC' in terms of both the detection and re-ID performance. 
It means that whereas the two-step implementation of LPS by using the existing lifelong re-ID methods does not effectively reflect huge domain gaps among the person search datasets with severely different characteristics, the proposed end-to-end framework alleviates such domain gaps faithfully and is more generalizable to diverse person search datasets.
We also compared the upper-bound performance of the person re-ID by adopting the ground-truth (GT) bounding boxes for person detection, denoted as `Det* + AKA,' `Det* + PTKP,' `Det* + KRKC,' and Proposed*. Note that all the methods yield the perfect performance of detection in terms of the recall and AP, but the proposed method achieves better performance of re-ID compared with all the two-step methods.

Fig. <ref> visualizes the qualitative results of the proposed method and the two-step methods. We observe that both of PTKP <cit.> and KRKC <cit.> fail to find the query persons correctly in the challenging cases, for example, with the occluded persons and/or relatively small bounding boxes in the scene images. However, the proposed method successfully matches the query persons even in such cases, demonstrating the robustness to diverse LPS scenarios. 

It is worth to note that the two-step person search framework trains the backbone network when training the detection and re-ID networks, respectively, and hence requires high computational complexity and huge memory space.
In contrary, the proposed end-to-end method shares the backbone network between the jointly trained detection and re-ID networks, and is a more promising tool for LPS considering practical real-world applications.









 §.§ Ablation Study




  §.§.§ Losses

In Table <ref>, we first evaluate the effect of the three losses, the detection knowledge distillation loss ℒ_dkd, the re-ID knowledge distillation loss ℒ_rkd^+, and the rehearsal-based instance matching loss ℒ_rim, respectively. 
Note that detaching all the losses is the same as the FineTune method since the proposed losses are designed to work only when the old data are available.
We see that adding each loss improves the performance, respectively. Specifically, when we use ℒ_dkd, the detection performance is largely increased from that of the FineTune method on the old datasets, and the re-ID performance is also increased accordingly. 
On the other hand, ℒ_rkd^+ and ℒ_rim slightly increase the detection performance of using ℒ_dkd, but significantly improve the re-ID performance on the old datasets of CUHK-SYSU and PRW by huge margins, demonstrating the effectiveness to preserve the re-ID knowledge in the old domains.











 



  §.§.§ Baseline Network

It is worth to note that the proposed method can be applied to any baseline network of person search. We conducted the additional experiment by implementing the proposed method on the transformer based architecture of COAT <cit.>. Table <ref> shows the results where we see that the proposed method significantly improves the performance compared to that of the FineTune method.




  §.§.§ Old Prototype-Based Knowledge Distillation

Table <ref> shows the effect of using the old prototype features for re-ID knowledge distillation. 
The conventional method, Intra-batch, estimates the distributions of the feature similarity with respect to all the detected proposals within a mini-batch. On the other hand, the proposed method matches the distributions of the feature similarity by using the prototype features of all identities stored in the old LUT, and thus provides better results than the Intra-batch scheme.



  §.§.§ Exemplar Data Sampling

Table <ref> shows the results of using different sampling schemes to compose the exemplar data from the old datasets.
`Max BBox' samples the images that have the top 2% largest numbers of ground truth bounding boxes. `Max ID' samples the images that have the top 2% largest numbers of person instances with identity labels. `Random' samples 2% images randomly from the old datasets. 
We see that different sampling schemes provide similar performances to one another, and selected the uniform sampling that yields a slightly better performance of the old knowledge preservation compared to the other ones. 




 §.§ Limitation

The proposed method stores 2% of the old data into the exemplar memory the typical rehearsal (replay) based methodology of lifelong learning <cit.>. Therefore, the size of the exemplar memory increases as we have more and more datasets. As a future research topic, we will investigate other methodologies of lifelong learning that address the limitation of using the exemplar memory. 



§ CONCLUSION

In this paper, we proposed a novel LPS framework where the model needs to be incrementally trained on the new datasets while preserving the knowledge of the old datasets. We implemented the knowledge distillation between the old and new models based on the rehearsal methodology by using the representative prototype features of the labeled foreground persons as well as the hard background proposals in the old exemplar data. We also designed the rehearsal-based instance matching loss to improve the discrimination ability by using the unlabeled person instances in addition to the prototype features. Experimental results evaluated on three datasets of person search showed that the proposed method achieves significantly better performance of lifelong learning compared with the existing methods, and successfully prevents the knowledge forgetting in the old domains. We expect this pioneering work would encourage further research for practical LPS applications.


1
IEEEtran


reidfirst
E. Ahmed, M. Jones, and T. K. Marks, “An improved deep learning architecture for person re-identification,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 3908–3916.

li2018harmonious
W. Li, X. Zhu, and S. Gong, “Harmonious attention network for person re-identification,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2018, pp. 2285–2294.

luo2019strong
H. Luo, W. Jiang, Y. Gu, F. Liu, X. Liao, S. Lai, and J. Gu, “A strong baseline and batch normalization neck for deep person re-identification,” IEEE Trans. Multimedia, vol. 22, no. 10, pp. 2597–2609, Dec. 2019.

ning2020feature
X. Ning, K. Gong, W. Li, L. Zhang, X. Bai, and S. Tian, “Feature refinement and filter network for person re-identification,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 9, pp. 3391–3402, Sep. 2020.

liu2022neural
Z. Liu, L. Zhang, and D. Zhang, “Neural image parts group search for person re-identification,” IEEE Trans. Circuits Syst. Video Technol., vol. 33, no. 6, pp. 2724–2737, Jun. 2022.

huang2020multiscale
Y. Huang, S. Lian, H. Hu, D. Chen, and T. Su, “Multiscale omnibearing attention networks for person re-identification,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 5, pp. 1790–1803, May 2020.

chen2022saliency
C. Chen, M. Ye, M. Qi, J. Wu, Y. Liu, and J. Jiang, “Saliency and granularity: Discovering temporal coherence for video-based person re-identification,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 9, pp. 6100–6112, Sep. 2022.

norm_aware
D. Chen, S. Zhang, J. Yang, and B. Schiele, "Norm-Aware embedding for efficient person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2020.

kim2021prototype
H. Kim, S. Joung, I.-J. Kim, and K. Sohn, "Prototype-guided saliency feature learning for person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 4865–4874.


yan2021anchor
Y. Yan, J. Li, J. Qin, S. Bai, S. Liao, L. Liu, F. Zhu, and L. Shao, "Anchor-free person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, pp. 7690–7699.


han2021end
B.-J. Han, K. Ko, and J.-Y. Sim, "End-to-end trainable trident person search network using adaptive gradient propagation," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 925–933.

lee2022oimnet
S. Lee, Y. Oh, D. Baek, J. Lee, and B. Ham, "OIMNet++: Prototypical normalization and localization-aware learning for person search," in Proc. Eur. Conf. Comput. Vis. (ECCV), 2022, pp. 621–637.

yu2022cascade
R. Yu, D. Du, R. LaLonde, D. Davila, C. Funk, A. Hoogs, and B. Clipp, "Cascade transformers for end-to-end person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 7267–7276.

kirkpatrick2017overcoming
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., "Overcoming catastrophic forgetting in neural networks," Proc. Nat. Acad. Sci., vol. 114, no. 13, pp. 3521–3526, 2017, National Acad Sciences.

shmelkov2017incremental
K. Shmelkov, C. Schmid, and K. Alahari, "Incremental learning of object detectors without catastrophic forgetting," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 3400–3409.

peng2020faster
C. Peng, K. Zhao, and B. C. Lovell, "Faster ilod: Incremental learning for object detectors based on faster rcnn," Pattern Recogn. Lett., vol. 140, pp. 109–115, 2020, Elsevier.

wu2021generalising
G. Wu and S. Gong, "Generalising without forgetting for lifelong person re-identification," in Proc. AAAI Conf. Artif. Intell., vol. 35, no. 4, pp. 2889–2897, 2021.

pu2021lifelong
N. Pu, W. Chen, Y. Liu, E. M. Bakker, and M. S. Lew, "Lifelong person re-identification via adaptive knowledge accumulation," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 7901–7910.


ge2022lifelong
W. Ge, J. Du, A. Wu, Y. Xian, K. Yan, F. Huang, and W.-S. Zheng, "Lifelong Person Re-identification by Pseudo Task Knowledge Preservation," in Proc. AAAI Conf. Artif. Intell., vol. 36, no. 1, pp. 688–696, 2022.

huang2022lifelong
Z. Huang, Z. Zhang, C. Lan, W. Zeng, P. Chu, Q. You, J. Wang, Z. Liu, and Z.-j. Zha, "Lifelong unsupervised domain adaptive person re-identification with coordinated anti-forgetting and adaptation," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 14288–14297.


prw
L. Zheng, H. Zhang, S. Sun, M. Chandraker, Y. Yang, and Q. Tian, "Person re-identification in the wild," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 1367–1376.


chen2018person
D. Chen, S. Zhang, W. Ouyang, J. Yang, and Y. Tai, "Person search via a mask-guided two-stream cnn model," in Proc. Eur. Conf. Comput. Vis. (ECCV), Sep. 2018, pp. 734–750.



lan2018eccv
X. Lan, X. Zhu, and S. Gong, "Person search by multi-scale matching," in Proc. Eur. Conf. Comput. Vis. (ECCV), Sep. 2018.


wang2020tcts
C. Wang, B. Ma, H. Chang, S. Shan, and X. Chen, "TCTS: A task-consistent two-stage framework for person search," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2020, pp. 11952–11961.


ke2022joint
X. Ke, H. Liu, W. Guo, B. Chen, Y. Cai, and W. Chen, "Joint sample enhancement and instance-sensitive feature learning for efficient person search," IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 11, pp. 7924–7937, 2022.


cuhk
T. Xiao, S. Li, B. Wang, L. Lin, and X. Wang, “Joint detection and identification feature learning for person search,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 3415–3424.


chen2020hoim
D. Chen, S. Zhang, W. Ouyang, J. Yang, and B. Schiele, "Hierarchical online instance matching for person search," in Proc. AAAI Conf. Artif. Intell., vol. 34, no. 07, pp. 10518–10525, 2020.

zhang2021diverse
X. Zhang, X. Wang, J.-W. Bian, C. Shen, and M. You, "Diverse knowledge distillation for end-to-end person search," in Proc. AAAI Conf. Artif. Intell., vol. 35, no. 4, pp. 3412–3420, 2021.


seqnet
Z. Li and D. Miao, “Sequential end-to-end network for efficient person search,” in Proc. AAAI Conf. Artif. Intell., Jul. 2021, vol. 35, no. 3, pp. 2011–2019

cao2022pstr
J. Cao, Y. Pang, R. M. Anwer, H. Cholakkal, J. Xie, M. Shah, and F. S. Khan, "Pstr: End-to-end one-step person search with transformers," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 9458–9467.

fiaz2023sat
M. Fiaz, H. Cholakkal, R. M. Anwer, and F. Shahbaz Khan, "SAT: Scale-Augmented Transformer for person search," in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), Jan. 2023, pp. 4820–4829.

yang2024unleashing
X. Yang, M. Tian, N. Wang, and X. Gao, “Unleashing the Feature Hierarchy Potential: An Efficient Tri-Hybrid Person Search Model,” IEEE Trans. Circuits Syst. Video Technol., 2024.


oh2024domain
M. Oh, D. Kim, and J.-Y. Sim, "Domain generalizable person search using unreal dataset," in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 5, pp. 4361–4368, 2024.



han2021weakly
C. Han, K. Su, D. Yu, Z. Yuan, C. Gao, N. Sang, Y. Yang, and C. Wang, "Weakly supervised person search with region Siamese networks," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 12006–12015.

han2021context
B.-J. Han, K. Ko, and J.-Y. Sim, "Context-aware unsupervised clustering for person search," in British Machine Vision Conf., 2021.



yan2022exploring
Y. Yan, J. Li, S. Liao, J. Qin, B. Ni, K. Lu, and X. Yang, "Exploring visual context for weakly supervised person search," in Proc. AAAI Conf. Artif. Intell., vol. 36, no. 3, pp. 3027–3035, 2022.


wang2023self
B. Wang, Y. Yang, J. Wu, G.-j. Qi, and Z. Lei, "Self-similarity driven scale-invariant learning for weakly supervised person search," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2023, pp. 1813–1822.

daps
J. Li, Y. Yan, G. Wang, F. Yu, Q. Jia, and S. Ding, "Domain adaptive person search," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ECCV), Oct. 2022, pp. 302–318.


almansoori2024ddam
M. K. Almansoori, M. Fiaz, and H. Cholakkal, "DDAM-PS: Diligent domain adaptive mixer for person search," in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), Jan. 2024, pp. 6688–6697.

hao2019end
Y. Hao, Y. Fu, Y.-G. Jiang, and Q. Tian, “An end-to-end architecture for class-incremental object detection with knowledge distillation,” in 2019 IEEE Int. Conf. Multimedia Expo (ICME), Jul. 2019, pp. 1–6.

shieh2020continual
J.-L. Shieh, Q. M. ul Haq, M. A. ul Haq, S. Karam, P. Chondro, D.-Q. Gao, and S.-J. Ruan, "Continual learning strategy in one-stage object detection framework based on experience replay for autonomous driving vehicle," Sensors, vol. 20, no. 23, p. 6777, 2020, MDPI.

liu2023continual
Y. Liu, B. Schiele, A. Vedaldi, and C. Rupprecht, "Continual detection transformer for incremental object detection," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2023, pp. 23799–23808.



carion2020end
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," in Proc. Eur. Conf. Comput. Vis. (ECCV), 2020.



dong2023incremental
N. Dong, Y. Zhang, M. Ding, and G. H. Lee, "Incremental-detr: Incremental few-shot object detection via self-supervised learning," in Proc. AAAI Conf. Artif. Intell., vol. 37, no. 1, pp. 543–551, 2023.



li2022technical
D. Li, G. Cao, Y. Xu, Z. Cheng, and Y. Niu, "Technical report for iccv 2021 challenge sslad-track3b: Transformers are better continual learners," 2022, arXiv:2201.04924.


mirza2022efficient
M. J. Mirza, M. Masana, H. Possegger, and H. Bischof, "An efficient domain-incremental learning approach to drive in all weather conditions," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 3001–3011.

sun2022patch
Z. Sun and Y. Mu, "Patch-based knowledge distillation for lifelong person re-identification," in Proc. 30th ACM Int. Conf. Multimedia, 2022, pp. 696–707.

yu2023lifelong
C. Yu, Y. Shi, Z. Liu, S. Gao, and J. Wang, "Lifelong person re-identification via knowledge refreshing and consolidation," in Proc. AAAI Conf. Artif. Intell., vol. 37, no. 3, pp. 3295–3303, 2023.

xu2024lstkc
K. Xu, X. Zou, and J. Zhou, "LSTKC: Long short-term knowledge consolidation for lifelong person re-identification," in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 14, pp. 16202–16210, 2024.


ren2015faster
S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," Adv. Neural Inf. Process. Syst., 2015.


rebuffi2017icarl
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, "icarl: Incremental classifier and representation learning," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 2001–2010.


qin2023movienet
J. Qin, P. Zheng, Y. Yan, R. Quan, X. Cheng, and B. Ni, “MovieNet-PS: a large-scale person search dataset in the wild,” in ICASSP 2023-2023 IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Jun. 2023, pp. 1–5.









 Paper Title: Certifying Robustness of Learning-Based Keypoint Detection and Pose Estimation Methods.















xushengl@andrew.cmu.edu
0000-0003-4342-7234


twei2@andrew.cmu.edu
0000-0003-2505-4585


siminliu@andrew.cmu.edu
0009-0009-1495-6920


ziweiwa2@andrew.cmu.edu
0000-0001-9225-8495


  Carnegie Mellon University
  5000 Forbes Avenue
  Pittsburgh
  Pennsylvania
  USA
  15213



Luis.E.Mattei-Mendez@boeing.com

taylor.s.loper@boeing.com

joshua.neighbor@boeing.com

  The Boeing Company
  PO Box 3707
  Seattle
  Washington
  USA
  98214-2207



fhutchin@nrec.ri.cmu.edu

cliu6@andrew.cmu.edu
0000-0002-3767-5517


  Carnegie Mellon University
  5000 Forbes Avenue
  Pittsburgh
  Pennsylvania
  USA
  15213













§ ABSTRACT 
  This work addresses the certification of the local robustness of vision-based two-stage 6D object pose estimation. The two-stage method for object pose estimation achieves superior accuracy over the single-stage approach by first employing deep neural network-driven keypoint regression and then applying a Perspective-n-Point (PnP) technique.
    Despite advancements, the certification of these methods' robustness, especially in safety-critical scenarios, remains scarce. This research aims to fill this gap with a focus on their local robustness on the system level—the capacity to maintain robust estimations amidst semantic input perturbations. The core idea is to transform the certification of local robustness into a process of neural network verification for classification tasks. The challenge is to develop model, input, and output specifications that align with off-the-shelf  verification tools. To facilitate verification, we modify the keypoint detection model by substituting nonlinear operations with those more amenable to the verification processes. Instead of merely injecting random noise into images, as is common, we employ a convex hull representation of images as input specifications to more accurately depict semantic perturbations. Furthermore, by conducting a sensitivity analysis, we propagate the robustness criteria from pose estimation to keypoint accuracy, and then formulating an optimal error threshold allocation problem that allows for the setting of a maximally permissible keypoint deviation thresholds. Viewing each pixel as an individual class, these thresholds result in linear, classification-akin output specifications. Under certain conditions, we demonstrate that the main components of our certification framework are both sound and complete, and validate its effects through extensive evaluations on realistic perturbations. To our knowledge, this is the first study to certify the robustness of large-scale, keypoint-based pose estimation given images in real-world scenarios. 







<ccs2012>
   <concept>
       <concept_id>10010520.10010553</concept_id>
       <concept_desc>Computer systems organization Embedded and cyber-physical systems</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010224.10010245</concept_id>
       <concept_desc>Computing methodologies Computer vision problems</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
 </ccs2012>


[500]Computer systems organization Embedded and cyber-physical systems
[300]Computing methodologies Computer vision problems





Certifying Robustness of Learning-Based Keypoint Detection and Pose Estimation Methods
    Changliu Liu
    
======================================================================================





§ INTRODUCTION

In the realm of computer vision, vision-based 6D object pose estimation, i.e., 3D rotation and 3D translation of an object with respect to the camera,  serves as a pivotal method for identifying, monitoring, and interpreting the posture and movements of objects through images <cit.>

This technology is fundamental in granting machines the ability to comprehend the physical environment, finding its utility in diverse domains such as robotics <cit.>, augmented reality <cit.>, and human-computer interaction <cit.>. The evolution of deep learning and the adoption of neural networks, particularly convolutional neural networks (CNNs), have markedly surpassed traditional techniques that depend on manually engineered features. Within the spectrum of learning-based approaches, a distinct classification exists: single-stage methods directly estimate the 6D pose from an image <cit.>. Conversely, a more widespread and accurate category of methods employs a two-stage strategy, initially regressing sparse keypoints <cit.> or dense pixels <cit.> from the image, followed by the utilization of a Perspective-n-Point (PnP)-based strategy for pose estimation through established 3D-2D point correspondences.

Despite the increasing efforts to boost the empirical robustness of these methods against challenges like occlusions, fluctuating lighting, and varied backgrounds, the focus on validating or certifying the reliability of vision-based pose estimation systems remains minimal. The absence of performance assurances for these frameworks raises concerns about their integration into safety-critical applications. In this work, our objective is to certify the robustness of learning-based keypoint detection and pose estimation approaches given input images. We focus on the aspect of local robustness, which refers to the ability to maintain consistent performance or predictions when the input data is perturbed around a given input point. The core question is determining whether pose estimation stays within an acceptable range when the input image undergoes perturbations. To the best of our knowledge, this study is the first one to certify the robustness of large-scale, keypoint-based pose estimation problem encountered in the real world.

Given the crucial role of neural networks in learning-based visual pose estimation, certifying their robustness is inherently linked to neural network verification <cit.>. This area of verification has attracted significant attention in recent years, driven by the paradox of widespread adoption of neural network solutions without adequate assurance of their reliability, primarily due to their opaque nature <cit.>. What distinguishes our problem from existing neural network (NN) verification efforts is our focus on system-level properties rather than on verifying attributes of isolated neural networks, which typically relate directly to the NN outputs <cit.>. For example, in robustness verification of classification models, the objective is to ensure the predicted class remains unchanged despite variations in input. Our research, however, targets the pose estimation framework, wherein the NN constitutes only one component of the entire system. The verification encompasses system-wide requirements, necessitating not just the evaluation of the keypoint detection model's robustness but also that of the PnP-based method.





 §.§ Overview of the Approach

To certify the robustness of the two-stage keypoint-based pose estimation framework, our main idea is to convert the local robustness certification of pose estimation into a standard neural network verification problem for classification networks. The primary challenge involves crafting three verification components—model, input, and output specifications—in a manner that is compatible with existing verification tools. A graphical overview is provided in Fig. <ref>. 


  
Model modification for verification. Considering the keypoint detection model involves complex nonlinear operations such as the softmax function, a variant of the model that is more amenable to verification is created, and an analysis is conducted to understand the properties that are maintained between the original and this modified model. 


  
Input specification through convex hulls. To account for realistic semantic perturbations in input images, we define the input space as a convex hull of possible perturbed images, which captures variations in a mathematically rigorous manner, allowing for a linear representation of input perturbations. Such a specification outperforms existing methods that simply introduce random independent noise into images.


  
Output specification via sensitivity analysis. The core of connecting system requirements with the neural network's output lies in conducting a sensitivity analysis of the downstream PnP method. By understanding how variations in detected keypoints affect the estimated pose, it's possible to translate system-level pose accuracy requirements into error thresholds for keypoint detection. By treating each pixel as a separate class, these thresholds are then used to define classification-like linear output specifications.



 §.§ Contributions

Our contributions can be summarized as follows:

    
  * We propose a local robustness certification framework for the learning-based keypoint detection and pose estimation pipeline;
    
  * We analyse the soundness and completeness properties of this certification framework;
    
  * We demonstrate the method's efficacy through validation on a real-world scale keypoint-based pose estimation problem.























§ RELATED WORK




 §.§ Formal Verification of Neural Networks

The objective of verifying neural networks involves ensuring they meet certain standards of safety, security, accuracy, or robustness. This essentially means determining the truth of a specific claim about the outputs of a network based on its inputs. In recent years, there has been a significant influx of research in this area. For comprehensive insights into neural network verification, one can refer to <cit.>. Verification techniques are generally divided into three main groups: reachability-based approaches, which perform a layer-by-layer analysis to assess network output range <cit.>; optimization methods, which seek to disprove the assertion <cit.>; and search-based strategies which combine with reachability analysis or optimization to identify instances that contradict the assertion <cit.>. In 2020, VNN-COMP <cit.> launched as a competition to evaluate the capabilities of advanced verification tools spanning a variety of tasks, including collision detection, image classification, dataset indexing, and image generation. However, these methods treat deep neural networks in isolation, concentrating on analyzing the input-output relationship.

Concurrently, there is research focused on the system-level safety of cyber-physical systems (CPS) incorporating neural network components, particularly within the system and controls domain. They broadly fall into two categories. The first category <cit.> focuses on ensuring the correctness of neural network-based controllers, taking their input from the structured outcomes of the state estimation module, regardless of the state estimation module is based on perception or not. Neural network controllers of this type generally consist of several fully connected layers, making them relatively straightforward to verify. The second category focuses on validating the closed-loop performance of vision-based dynamic systems that incorporate learning-based components. Among these, studies such as <cit.> examine LiDARs as the perception module, processed by multi-linear perceptrons (MLPs) with a few hidden layers. Other approaches, primarily applied to runway landing and lane tracking, deal with high-dimensional inputs from camera images, employing methods like approximate abstraction of the perception model <cit.>, contract synthesis <cit.>, simplified networks within the perception model <cit.>, or a domain-specific model of the image formation process <cit.>. 
Nevertheless, studies directly dealing with high-dimensional inputs from camera images are still limited due to the images' high dimensionality and unstructured data nature, in contrast to structured robot states such as position and velocity.





 §.§ Certification of Keypoint Detection and Pose Estimation Methods

The investigation of certification methods for pose estimation is relatively limited. <cit.> introduced a certifiable approach to keypoint-based pose estimation from point clouds by correcting keypoints identified by the model, ensuring the correctness guarantee of the pose estimation. <cit.> expanded on this by integrating the correction concept with ensemble self-training. In a similar vein, By propagating the uncertainty in the keypoints to the object pose, <cit.> created a keypoint-based pose estimator for point clouds that is provably correct and is characterized by definitive worst-case error bounds. The above work focuses on point clouds as opposed to images. <cit.> applied an advanced deep learning technique for uncertainty quantification to assess the uncertainty (i.e., the predicted distribution of a pose) in multi-stage 6D object pose estimation methods. In contrast, <cit.> aimed to design a neural network that processes camera images to directly predict the aircraft's position relative to the runway with certifiable error bounds. Of all these studies, <cit.> is the most similar to our work, which focuses on the verification of keypoint detection, excluding the examination of the PnP method for system-wide assurances. Their approach verifies the neural network in isolation and is limited to very slight perturbations, failing to encompass realistic semantic variations.




§ BACKGROUND

In this work, we represent scalars and scalar functions by italicized lowercase letters (x), vectors and vector functions by upright bold lowercase letters (), matrices and matrix functions by upright bold uppercase letters (), and sets and set functions with calligraphic uppercase letters ().



 §.§ Keypoint-based Pose Estimation

The keypoint-based approach consists of two steps to estimate the 6D pose from a 2D model image. First, a neural network is tasked with predicting the 2D locations of keypoints, whose 3D locations are predefined within the object model. A common strategy involves the use of heatmap regression, wherein ground-truth heatmaps are created by placing 2D Gaussian kernels atop each keypoint. The heatmap pixel values are interpreted as the likelihood of each pixel being a keypoint. These heatmaps are then used to guide the training through an ℓ_2 loss. The detection network can be divided into two parts. A backbone network, denoted by b, inputs a 2D image to produce unnormalized heatmaps, which is first followed by a softmax layer that transforms unnormalized heatmaps into normalized ones, and then by argmax operations, or another layer of differentiable spatial to numerical transformation (DSNT) <cit.>  for keypoint extraction. We refer to the part after the softmax (including) as the head network. The entire network is represented by

=  () = h∘b(), 

where ∈^H× W× C represents a 2D RGB image with dimensions being H× W× C, and ∈^K× 2 denotes the 2D coordinates of K keypoints. Here, ∘ denotes function composition. To enhance accuracy and robustness, it's often essential to preprocess the input image  before it is passed to the network, such as resizing and color normalization. Denote this preprocessing step by _0, leading to the equation =  () = h∘b∘_0().  In what follows, we omit the preprocessing step unless it is critical to consider it.



The second step employs the Perspective-n-Point (PnP) algorithm, which executes a nonlinear least squares (NLS) optimization to estimate the 6D pose from established 3D-to-2D correspondences. An illustration of the pose estimation  for an airplane is presented in Fig. <ref>. Let ∈^3×3 denote the camera intrinsic parameter matrix, _k ∈^3 denote the 3D coordinate of keypoint k, where k = 1, …, K, and _k ∈^2 denote the corresponding 2D coordinate. These 3D-2D correspondences are formed through the following perspective projection model:

    λ_k [ _k;  1 ]
       =  (_k + ),

where ∈^3×3 and ∈^3 represent the rotation matrix and translation vector, respectively, that establish the transformation between the object and camera coordinate systems, with λ_k representing the scaling factor. The objective of the PnP method is to approximate this transformation, represented by  and , by minimizing the ℓ_2 norm of the reconstruction error:

    ⟨, ⟩  =   ,       ∑_k=1^K (_k + ) -   λ_k [ _k;  1 ]^2_2.


Let the notation ⟨, ⟩ = (, ) represent the PnP procedure, where ∈^K× 3 denotes the 3D coordinates of keypoints. Note that the keypoint-based pose estimation framework described above is the basic version. Numerous adaptations have been developed to increase accuracy and robustness <cit.>, particularly in challenging conditions such as occlusions, varying viewpoints, and different lighting scenarios. This paper concentrates on validating the robustness of this foundational pipeline, marking a crucial step towards certifying the effectiveness of more intricate keypoint-based pose estimation techniques. Hereafter, we use Ψ to represent this pipeline, which includes keypoint detection followed by the application of a PnP method, that is, ⟨, ⟩ = Ψ() = (, ()).








 §.§ Verification of Neural Networks

Consider a multi-layer neural network representing a function , which takes an input ∈_⊆^d_0 and produces an output ∈_⊆^d_n, where d_0 is the input dimension, and d_n is the output dimension. Any non-vector inputs or outputs are restructured into vector form. The verification process entails assessing the validity of the following input-output relationships defined by the function :
∈⇒ = () ∈,
where sets, ⊆_ and ⊆_, are referred to as input and output constraints, respectively. 

In the context of confirming the robustness of a classification network, the goal is to ascertain that all samples within a proximal vicinity of a specified input x_0 receive an identical classification label. Assuming the target label is i^∗∈{1, …, d_n}, the specification for verification is that y_i^∗ > y_j for every j not equal to i^∗. The constraints on inputs and outputs are established accordingly:
= {| - _0_p ≤ϵ}, 
 = {| y_i^∗ > y_j,  ∀ j ≠ i^∗},
where ϵ represents the maximum permissible deviation in the input space. The metric used to quantify disturbance can be any ℓ_p norm.

Neural network verification algorithms can generally be categorized into three main types: reachability analysis, optimization, and search. NN verification essentially seeks to transform the nonlinear model checking problem into piece-wise linear satisfiability problems, and it can be applied to various nonlinearities, including ReLU and, more recently, softmax <cit.>. Two pivotal attributes—soundness and completeness—are of critical importance. A verification algorithm is sound if it only confirms the validity of a property when the property is indeed valid. It is complete if it consistently recognizes and asserts the existence of a property whenever it is actually present. There is a trade-off between computational complexity and conservativeness (or in-completeness).






§ PROBLEM FORMULATION


The most common type of input specification involves limiting the ℓ_p-norm of the variation to a threshold, that is, - _0 _p≤ϵ. However, ℓ_p perturbation,  often characterized by a small value ϵ,  is not a correct mathematical description of more realistic perturbations, as the independent nature of pixel-wise perturbations falls short in creating perturbations that reflect semantic correlations between pixels, such as large variations in lighting, weather conditions, and the effects of camera motion blur. Another approach for input perturbations is to directly add a generative model that perturbs the input image before the original neural network, such as <cit.>, and then verify them together. However, the verification result can be biased by the generative model used and it is not user friendly due to the difficulty in controlling the changes made to the image. To address these shortcomings, we adopt a strategy based on the convex hull, which involves combining a seed image with a collection of perturbed images. These perturbed images can be created through different methods including simulators and learning-based generative models. Also, the convex hull specification can directly enable users to specify the perturbed images, which makes the perturbation specifications more user friendly.


Given a seed image _0 and a set of n perturbed images {_1, …, _n}, the convex hull constituted by these images is defined by the set of all their possible convex combinations. Mathematically,

    = { |  = ∑_i = 0^n α_i _i,   s.t. α_i ≥ 0, ∑_i=0^n α_i  = 1.}.


Images within the convex hull ∈ result from varying degrees of continuous blending among the provided images.  Through the convex combination of perturbation instances, we can model environmental and sensor-related perturbations, including changes in brightness, contrast, weather conditions, motion blur, and dust on lens. Convex hull perfectly captures the entire range of brightness or contrast shifts, and does a reasonable approximation of color shifts, provided they are small. However, this approach does not capture translational perturbations of the object or perturbations related to camera movements, such as changes in the viewpoint. Preliminary methods exist that characterize pixel-wise perturbations caused by camera motion <cit.>, but these methods cover only a very limited range. Investigating robustness against such perturbations will be addressed in future research.  A collection of perturbed images and sampled images from the convex hull can be found in Fig. <ref> in Section <ref>. 

Given two rotation matrices  and , we define the function _r (, ) = |() - () |, where (·) represents the function that converts rotations to Euler angles, and |·| denotes the element-wise absolute value. Thus, _r measures the rotational difference along each axis. Similarly, define _t (, ) =  | - |, which calculates the translational difference per axis.





 Given a convex hull representation  consisting of a seed image _0 and n perturbed images, and a keypoint-based pose estimation framework Ψ.
 
 
 
 
 The problem is to certify whether the pose estimation framework Ψ is robust to any image within the set . Mathematically,

    [ _r (,  );  _t (, ) ]≤[ ϵ_r; ϵ_t ]  s.t. ⟨, ⟩ = Ψ(),  ∀ ∈,

where ⟨, ⟩ =  (, ) represents the pose estimated given the ground-truth 3D keypoint coordinates  and ground-truth 2D keypoint coordinates   for _0, and ϵ_r ∈^3 and ϵ_t ∈^3 denote the error thresholds specified by the user for rotation and translation. We refer to  and  as the nominal transform, and  and  as the perturbed transform. In simpler terms to describe Eq. (<ref>), the variation between the nominal transform and the perturbed transform is kept within these predefined thresholds.



If a pre-processing step, such as resizing or color normalization, is applied to an image before it is input into the keypoint detection model, then the convex hull should be constructed using images that have also undergone these pre-processing steps. Consequently, any image within the convex hull is represented as = ∑_i=0^n α_i _0(_i), where _0 denotes the pre-processing function. 




 The vector of error thresholds ϵ_r and ϵ_t can specify requirements from system designers in an interpretable manner. For example, such requirement can be setting the error thresholds for translation to be within 1 meter. The difficulty presented by Problem <ref> is that, the error thresholds ϵ are defined with respect to pose estimation rather than the direct outputs of neural networks—keypoints—and there are nonlinear mappings between keypoints and pose estimation. 
 
To address Problem <ref>, the core of our approach is to  transform the local robustness certification problem into a standard neural network verification problem for classification networks. This involves adapting three verification components—model, input, and output specifications—to be compatible with existing verification tools. As detailed in Section <ref>, we have modified the keypoint detection model to handle complex operations like the softmax function, simplifying the verification process. The input space is the a convex hull of possible perturbed images, which captures semantic variations  compared to traditional methods that introduce random noise. Section <ref> outlines how the output specification is established through a sensitivity analysis of the PnP method, detailed further in Section <ref>. This analysis helps translate system-level pose accuracy requirements into error thresholds for keypoint detection. Finally, Section <ref> is dedicated to analyzing the soundness and completeness of our proposed verification framework.





§ FORMULATION OF VERIFICATION OF THE NEURAL NETWORK


To convert Problem <ref> into an equivalent problem that can be verified using existing NN verification tools, we introduce approaches to modify the NN model and specify the output constraints.



 §.§ Verification-Friendly Keypoint Detection Model


As introduced in Section <ref>, a softmax layer follows the backbone network to generate probabilistic heatmaps. This step introduces verification challenges due to the highly nonlinear exponential function in softmax. To address this, we propose to maintain the backbone network but replace the head network with an alternative one. A critical insight is that during the inference phase, the unnormalized heatmaps output by the backbone network b already provide significant clues about the locations of keypoints. The subsequent operations within the head network h, such as softmax and DSNT, essentially serve to aggregate this information across the entire heatmap. This aggregation allows for the generation of predictions in an average manner. In essence, the peaks within the heatmaps play a pivotal role in these predictions, and their locations remain unchanged between unnormalized and normalized heatmaps. Consequently, by focusing on the characteristics of these peaks, we can bypass some of the complexities introduced by the head network's nonlinear operations while still capturing the essential information required for accurate keypoint detection. 


By appending an average pooling layer followed by an argmax layer as the new head network, referred to as h', to the backbone network, we create a proxy model, denoted by proxy = h' ∘b. We refer to the original model as the target model. The introduction of this proxy model simplifies the transformation of the verification problem into one akin to classification, where each pixel is treated as a unique category, as illustrated in Fig. <ref>. The average pooling layer offers two advantages. Firstly, its downsampling effect significantly reduces the number of categories (pixels), thereby simplifying the complexity of verification. Secondly, by aggregating local features within each pooling region, it ensures a more accurate representation than applying argmax directly. 







The selection of pooling parameters should be guided by the error threshold on pose estimation ϵ_r and ϵ_t, as elaborated in Section <ref>. This leads to a scenario where the pooling parameters assigned to each heatmap, corresponding to individual keypoints, vary. This variation arises because keypoints have differing levels of influence on the accuracy of pose estimation. Consequently, keypoints with lesser influence on pose estimation are allocated a larger permissible error range, whereas those that play a critical role in pose estimation are subjected to stricter error ranges. It is worth noting that, with certain assumptions, the target model and proxy model are equivalent in terms of verification. The analysis on how assurances regarding the proxy model's performance can be extended to the target model is conducted in Section <ref>.


































 
 


















 








































 
 















 



 §.§ Output Specification: Polytope Representation

For the coordinate _k of the k-th keypoint, where _k represents its 2D coordinate, let _k denote its averaged ground-truth coordinate following the application of the average pooling and argmax layers. To ensure the classification result is consistent across perturbations, the output specification necessitates that the value at v̅_k exceeds the values at all other entries. This requirement can be encapsulated by a half-space-based polytope, represented as _k _k ≤_k with



    _k = 
    [  1  0  … -1  …  0;  0  1  … -1  …  0;  ⋮  ⋮  ⋱  ⋮  ⋮  ⋮;  0  0  … -1  …  0;  ⋮  ⋮  ⋮  ⋮  ⋱  ⋮;  0  0  … -1  …  1;    ]
    ,   _k = 
    [    y_1;    y_2;      ⋮; y_v̅_k;      ⋮;    y_n ]
    ,   _k = 
    [ 0; 0; ⋮; M; ⋮; 0 ],

where the v̅_k-th column of _k is populated with (-1)'s, and all diagonal elements set to 1 with the exception of the element at (v̅_k, v̅_k), the components of _k are set to zero except for the v̅_k-th element, which is assigned a significantly large integer M. Consequently, the specification for all keypoints is denoted by ≤ with

    𝐀 = 
    [ 𝐀_1   0   0   …   0;   0 𝐀_2   0   …   0;   0   0 𝐀_3   …   0;   ⋮   ⋮   ⋮   ⋱   ⋮;   0   0   0   … 𝐀_K ]
    ,    = 
    [ _1; _2; _3;  ⋮; _K ]
    ,    = 
    [ _1; _2; _3;  ⋮; _K ].

The construction of Eq. (<ref>) presupposes verifying all keypoints. However, should there be a keypoint that does not require verification, potentially due to being an outlier, the matrices associated with it are then adjusted accordingly: _k = 𝐈 and _k = [M, …, M]^T.





























§ ERROR PROPAGATION AND DETERMINATION OF POOLING PARAMETERS

To calculate the parameters for average pooling, the initial step involves mapping the error thresholds ϵ_r and ϵ_t from the pose estimation onto the keypoints' error thresholds that may be correlated to each other, which is accomplished by employing sensitivity analysis methods.

Subsequently, we formulate an optimal error threshold allocation problem, aiming to assign independent error thresholds to each keypoint. This step is crucial for determining the allowable errors for each keypoint individually, ensuring that the overall pose estimation adheres to predefined error limits. The idea is graphically depicted in Fig. <ref>.










 §.§ Error Propagation via Sensitivity Analysis

Sensitivity analysis for nonlinear optimization explores the impact of first-order changes in the optimization parameters on the locally optimal solution <cit.>. It involves analyzing an unconstrained parameterized nonlinear optimization problem expressed as min_ F(; ), with  representing the parameter vector. Here, let z represent the optimal objective value given the parameter vector , i.e.,
z () = min_ F(; ).
Sensitivity analysis <cit.> connects the first-order derivatives ∂ and δ z with ∂:

    [ -F_; -F_ ]∂  
        = [ F_ -1; F_  0 ][   ∂; ∂ z ].

In the context of pose estimation, the keypoint coordinates  are considered as parameters and the pose to be estimated represents the decision variables, the notation for partial derivatives ∂ is substituted with the notation for deivation δ. Consequently, Eq. (<ref>) is transformed into

    [ -G_; -G_ ]δ
        = 
        [ G_ -1; G_  0 ][   δ; δ z ],

where ∈^6 = [, ] represents the 6D pose vector, with the 3D rotation matrix  expressed in the axis-angle or Euler angles representations form , and G corresponds to the NLS optimization objective outlined in Eq. (<ref>). Note that δ∈^2K. The following notation is introduced

    _ :=  [ -G_; -G_ ]∈^7× 2K,  _ :=  [ G_ -1; G_  0 ]∈^7×7.


If _ is invertible, then it follows that
_^-1_δ =  [   δ, δ z]^T.
    
    
    
By extracting the rows in the matrix _^-1_ that correspond to δ and naming the resulting matrix _, we obtain the linear mapping 

_δ = δ.

This leads to the establishment of a linear relationship between  2D keypoints errors and 6D pose errors. Upon specification of the keypoints  and the transform ξ, the matrix _ becomes determined. 

Let Ξ represent the set of tolerable errors, indicating the range within which deviations of the perturbed transform from the nominal transform are acceptable, meaning δ∈Ξ. Let δ denote the derived tolerable keypoint  errors that has the following property:

 δ =  {δ | _δ   =  δ∈Ξ}.

Note that δ is an approximation of actual tolerable keypoint errors that lead to tolerable pose errors δξ. As the pose error bounds decrease, the accuracy of the PnP linearization improves, narrowing this disparity. In what follows, we assume that Ξ is defined as a polytope represented by linear inequalities, specifically Ξ= {δ | _ξδ≤_}. This inequality can be established based on a user-defined thresholds ϵ_r and ϵ_t as outlined in Problem <ref>. By the linear mapping, the set δ can also be characterized as a polytope:
    δ = {δ | _ξ_δ≤_} = {δ | _δ≤_},
where _ =  __ and _ = _.  While larger set  δ is desirable, it is an approximation of actual tolerable errors due to the linearization of sensitivity analysis. To mitigate the risk of over-approximation, we  introduce a scaling factor to reduce the size of  set δ in the next section.



 §.§ Optimal Error Threshold Allocation

The tolerable errors of keypoints are interrelated within the derived polytope set δ. In this section, we allocate the tolerable errors across each keypoint to ensure their tolerances are independent, as current verification tools lack the capability to verify dependencies between  keypoints. The allocated error threshold  aims to be maximized, provided that the pose deviation remains tolerable.

For a given error vector δ, we can uniquely define an axis-aligned, axis-symmetric hyper-rectangle, denoted as (δ) ⊂^2K, centered at the origin with each element δ_i representing the half-length of its sides. Axis symmetry indicates that the error thresholds are identical in opposite directions. We establish the following problem for optimal error threshold allocation:

    max_δ∈_+^2K      w_1 ∏_k=1^2Kδ_k + w_2 Δ
    s.t.     (δ)  ⊆δ, 
         δ_k≥Δ,     k = 1, …, 2K.


Constraint (<ref>) ensures that the hyper-rectangle is encompassed within the set of tolerable errors δ, while constraint (<ref>) sets a minimum threshold for the side lengths. The goal defined in objective (<ref>) is to optimize a weighted sum of two objectives: the first seeks to identify the largest possible axis-aligned, axis-symmetric hyper-rectangle within a convex polytope, and the second strives to extend the minimal side length as much as feasible.

To address the optimal error threshold allocation problem, we translate constraint (<ref>) into a series of linear inequalities, drawing from the method in <cit.> for identifying the largest axis-aligned hyper-rectangle inscribed within a convex polytope. Considering a hyper-rectangle = {∈^d |≤≤} and a convex polytope = {|≤}, <cit.> demonstrates that ⊆ if and only if

    ∑_j=1^d (_ij^+ _j - _ij^- _j) ≤_i,    i = 1, …, n.

where _ij^+ = max{_ij,0} and _ij^- = max{-_ij,0}. Given the axis symmetry in our scenario, it follows that _j = -_j. Substituting _j with δ_j,  with _, and  with _, we can reformulate Ineq. (<ref>) as

    ∑_j=1^2Kδ_j (_ ij^+  + _ ij^-) ≤_ i,    i = 1, …, 6.

Given that |_ ij| = _ ij^+ + _ ij^- where |·| returns element-wise absolute values, it follows that

    ∑_j=1^2Kδ_j |_ ij|≤_ i,    i = 1, …, 6.

In matrix notation, Ineq. (<ref>) is expressed as
|_| δ≤_.
Ultimately, the problem of optimal error threshold allocation, encompassing objectives (<ref>) through constraints (<ref>), is equivalent to

    max_δ∈_+^2K      w_1 ∏_k=1^2Kδ_k + w_2 Δ
    s.t.     κ  |_| δ≤_, 
         δ_k≥Δ,    k = 1, …, 2K,


where _+ represents the set of non-negative integers, and κ∈_+ is the scaling factor that controls the actual size of polytope. The effect of the scaling factor κ on the allocation is discussed in Section <ref>. The solution, denoted by δ^*, returns the maximum tolerable errors for all keypoints while ensuring that the pose errors remains within the error thresholds. This solution will determine the parameters for average pooling in Section <ref>.






































































































































 §.§ Determination of Average Pooling Parameters



A widely accepted practice to train the keypoint detection model is to regress the heatmaps centered on the ground truth keypoint. Building on this approach, we propose the following assumption and support it with evaluation results detailed in Section <ref>.


   The softmax layer in the target model _target produces a distribution with a peak at its center and exhibits symmetry along axes that intersect this peak.


Upon determining the independent error threshold for each keypoint, we proceed to define the parameters for average pooling in the proxy model _proxy. Focusing on a particular keypoint with coordinates = (_h, _v), and considering the error thresholds δ^*_h and δ^*_v allocated for horizontal and vertical directions respectively, we define the set 

    _δ = { (h, v) | | h - _h| ≤δ^*_h  | v - _v| ≤δ^*_v }.

Only pixels within _δ are considered acceptable potential predictions for the keypoint. The average pooling parameters for this keypoint are selected to ensure every pixel in _δ is contained by the same pooling patch. This guarantees that all these pixels are pooled into the same after-pooling pixel as the keypoint, thereby belonging to the same class.

We begin by examining how predictions are generated in the proxy model proxy. Based on Assumption <ref>, the unnormalized heatmap returned by the backbone network _b exhibits symmetry along axes passing through this peak, as softmax operation will not change the overall shape of the unnormalized heatmap. To infer the keypoint prediction, we concentrate on the average pooling patch that predominantly overlaps with the vicinity of the peak. Since the average pooling patches also exhibit axial symmetry, patches closer to the peak yield larger averaged pixel values. Consequently, the pooling patch with its center closest to the peak results in the predicted keypoint. This insight enables us to transform the determination of the location of the maximal pixel value after pooling into identifying the closest pooling patch to the peak of the unnormalized heatmap under Assumption <ref>. Next, we compute the average pooling parameters for each keypoint to ensure that: (i) the ground-truth keypoint aligns with the center of an average pooling patch, and (ii) all pixels in proximity to this pooling patch fall within the allocated error thresholds from the ground truth keypoint, making them suitable to act as the peak of the unnormalized heatmap. Let = (s_h, s_v) denote the stride, = (k_h, k_v) the kernel size, and = (p_h, p_v) the patch size.





  
Determination of stride and kernel parameters. 
The initial step involves determining the stride and kernel parameters, denoted as s_h and k_h, respectively, as in Fig. <ref>. We focus on the horizontal axis (similar logic applies to the independent vertical axis). Let  represent the pooling patch centered at a keypoint, with _-1 and _+1 denoting the adjacent patches to the left and right, respectively. Additionally, let b_-1 and b_+1 denote the perpendicular bisectors between these patches. The distance between these two perpendicular bisectors corresponds to the horizontal stride. To position the ground-truth keypoint at the center, an odd stride is required. As long as the peak of the unnormalized heatmap lies between b_-1 and b_+1, the pooling patch  is the closest to the peak, resulting in the predicted keypoint. Let δ^*_h denote the allocated error threshold. To ensure that peaks falling between b_-1 and b_+1 remain permissible with respect to the error threshold, the threshold should be no less than half the stride distance from the ground-truth keypoint on either side, i.e.,
δ^*_h ≥1/2 (s_h - 1),
which leads to the inequality s_h ≤ 2δ^*_h + 1. Choosing s_h =  2 δ^*_h + 1 minimizes the post-pooling size. It's worth noting that there are no constraints on the kernel parameter. For simplicity, we set the kernel k_h to be equal to the stride s_h.



















  
Determination of the padding parameter.
The next step involves determining the padding parameter to center a pooling patch at the ground-truth keypoint. Padding can be added from the left (horizontal) or from the top (vertical) to achieve this. Considering the horizontal axis where the kernel is equal to the stride, the idea is to identify the closest pooling patch to the keypoint from its right side and then shift it left by the distance between the keypoint and its center, as illustrated in Fig. <ref> and <ref>. Assuming k_h = s_h, let r_h be the number of pixels the keypoint is from the left side of the closest pooling patch, calculated as r_h ≡_h k_h. (i) If r_h ≤k_h + 1/2, the closest patch center to the right of the keypoint, belongs to the pooling patch containing the keypoint. (ii) Otherwise, the closest patch center to the right comes from the pooling patch immediately to the right of the one containing the keypoint. The horizontal padding parameter p_h is determined as k_h+1/2 - r_h if r_h ≤k_h+1/2, otherwise, as 3k_h+1/2 - r_h.









Note that the shift is not unique because multiples of the stride can be added to p_h.

 
 
 
 
 
 
 
 
 
 
 
 




§ THEORETICAL ANALYSIS

With the method we introduced (to reformulate the problem into a verifiable form using off-the-shelf verification tools), it is important to understand whether false analysis results will be introduced. To answer that question, we perform the following analysis.



 §.§ Properties of the Proxy Model

We examine two target models. The models ending with a DSNT layer and an argmax layer are represented by dsnt and argm, respectively. The term target is used to represent either target model. The notation ^k is employed to identify the k-th predicted keypoint. Recall that _k denote the averaged ground-truth coordinate of the k-th keypoint after the application of average pooing and argmax layers. Before providing theoretical results, we establish formal definitions for both the soundness and completeness related to the proxy model, which are visually represented in Fig. <ref>.



    A proxy model is deemed sound if, for cases where the proxy model's predicted keypoint aligns with the averaged ground-truth keypoint _k, the deviation of the target model's prediction from the ground truth _k does not exceed the error threshold δ_k^*. Specifically, 
    
    proxy^k() = _k ⇒|target^k() - _k |≤δ_k^*,   for  k = 1, …, K.

     Here, |·| indicates the computation of element-wise absolute differences.


    A proxy model is deemed complete if the difference between the target model's prediction and the ground truth  _k is confined within the error threshold δ^*_k, then the predicted keypoint by the proxy model matches the averaged ground truth _k. That is, 
    
    |target^k() - _k |≤δ_k^* ⇒proxy^k() = _k,   for  k = 1,…, K.




If Assumption <ref> holds, the proxy model is sound, that is,

    proxy^k() = _k    ⇒|dsnt^k() - |≤δ_k^*,     for   k = 1, …, K. 
    proxy^k() = _k    ⇒|argm^k() - |≤δ_k^*,     for   k = 1, …, K.



To prove Theorem <ref>, we first prove the equivalence of two target models.
    
      If Assumption <ref> holds, the two target models produce the same predictions, i.e.,

    dsnt^k() =  argm^k(),      for    k = 1, …, K.

    
    




 Before delving into the proof, we first introduce how DSNT extracts a keypoint from a normalized heatmap. Let  represent a normalized heatmap of dimensions m × n, which is generated by a softmax layer. Define  as an m × n matrix where each entry _ij is calculated by _ij = 2j-(n+1)/n. Consequently, each column of  contains identical values. These values fall within the range of (-1, 1), with the center value at 0. To determine the column coordinate _h of the keypoint, DSNT computes the sum of the element-wise product between  and , expressed as _h = ∑_i, j_ij_ij. The row coordinate is obtained in a similar way.

Without loss of generality, due to independence of horizontal and vertical axes, we will show for the horizontal direction. Considering a symmetric heatmap with dimensions m × n, we initially position the center of the distribution at the m+1/2-th row and the n+1/2-th column, effectively placing the distribution's center at the heatmap's midpoint. Upon applying the DSNT process, the resulting keypoint coordinate is (0, 0). This outcome is attributed to the distribution's symmetry along both the horizontal and vertical axes that intersect at its center, with the DSNT operation cancelling out symmetric values relative to the origin. Subsequently, we consider a horizontal displacement of the distribution by c columns, where a positive c indicates a shift to the right. The column coordinate _h, as determined by DSNT following the horizontal shift, is then













    ∑_i, j_ij_ij 1= ∑_i, j'_ij'(_ij' +  2c/n) 2= ∑_i, j'_ij'_ij' + ∑_i, j'_ij'2c/n 3=   0 + 2c/n∑_i, j'_ij' 4= 2c/n


In 1, the pixel located at (i, j') is the corresponding pixel at (i, j) prior to the horizontal shift, with both pixels having identical values, denoted as _ij = _ij'. The value 2c/n represents the  difference across every c columns within the  matrix .
For 3, the expression ∑_i, j'_ij'_ij' equals zero, reflecting the distribution's initial centering at the heatmap's midpoint. In 4, the probability over distribution sums up to 1. The calculation of the DSNT column coordinate as 2c/n∈ (-1, 1) aligns it with the heatmap's (n+1/2+c)-th column. On the other hand, the argmax operation in argm^k returns the  distribution's center, located at the (n+1/2+c)-th column following the shift. Consequently, we establish that dsnt^k() = argm^k(), thereby completing the proof.


According to Proposition <ref>, it is sufficient to validate Eq. (<ref>) that connects proxy with argm to prove Theorem <ref>. The distinction between the proxy and argm models lies solely in their method of handling the unnormalized heatmap, specifically whether they use a softmax or an average pooling layer as depicted in Fig. <ref>. With this, we are poised to validate Theorem <ref>, a conclusion that naturally follows from the way we identify average pooling parameters.

Considering the k-th keypoint _k, if proxy^k() = _k, it implies that average pooling parameters ensure that the closest pooling patch __k to the ground-truth keypoint _k is exactly centered at _k.

 The selection of stride parameters ensures that the heatmap's peak is within a δ^*_k distance from the center of __k, which is also the ground-truth coordinate _k. Given that the softmax layer preserves the peak's position, the prediction argm^k precisely returns the peak's location, leading to the conclusion that argm^k() is within a δ^*_k distance from _k, i.e.,
 |argm^k() - |≤δ_k^*.






If Assumption <ref> holds, the proxy model is complete, that is,

    |dsnt^k() - |≤δ_k^*    ⇒proxy^k() = _k,    for   k = 1, …, K. 
    |argm^k() - |≤δ_k^*     ⇒proxy^k() = _k,    for   k = 1, …, K.
 


Based on Proposition <ref>, we focus exclusively on the target model argm. The selection of pooling parameters guarantees that every pixel within the δ^*_k proximity of the ground-truth keypoint—eligible to be recognized as the predicted keypoint—shares the same closest pooling patch. This closest pooling patch is identical to that of the ground-truth keypoint. As a result, these pixels yield an averaged pixel that is the same as that of the ground-truth keypoints.



 


 §.§ Properties of the Optimal Error Threshold Allocation

In the previous section, we examined the properties of the proxy model after determining the error threshold for keypoints. This section focuses on the properties of the optimal error threshold allocation. We introduce the concepts of probabilistic soundness and probabilistic completeness to measure the relationship between tolerable errors in pose and keypoints, as depicted in Fig. <ref>. 

Considering the set of acceptable pose errors Ξ, let δ_Ξ represent the corresponding set of acceptable keypoint errors. Note that δ_Ξ might not be polytopic, due to the non-linear nature of the PnP method, even though Ξ is polytopic. Let μ(·) denote the measure of a set.
    
    Probabilistic soundness is defined by the ratio μ(δ_Ξ∩)/μ(), which is the fraction of keypoint errors in  that result in tolerable pose errors within Ξ.
    
    Likewise, considering the set of tolerable errors on keypoints , let Ξ_ represent the corresponding set of tolerable errors on poses.
     
     Probabilistic completeness is defined as the ratio μ(Ξ_∩Ξ)/μ(Ξ), which represents the fraction of permissible pose errors in Ξ that result in errors on keypoints within .
    
Deriving an exact or lower bound for these two metrics is difficult, due to the challenges in making definitive conclusions using linearization-based sensitivity analysis, as the accuracy of local linearizations varies with instances, such as varying poses. In Section <ref>, we perform statistical tests to approximate these two values, offering insights into the practicality of our framework.
 
    













































































































































 







§ EVALUATION RESULTS





 §.§ Verified Keypoint Detection Model and Perturbations



  §.§.§ Verified keypoint detection model

  The verified keypoint detection model, illustrated in Fig. <ref>, is modified from the proxy model proxy. This verified model includes three components:


  
1. Backbone model  The dataset contains 7,320 images, each with dimensions of 1920 × 1200. 


    
  * CNN. The architecture includes 5 convolutional layers and an equal number of deconvolutional layers, designed to handle inputs of size 64×64×3. The model comprises 39 layers and contains around 6.57× 10^5 trainable parameters.
    
  * ResNet-18. This model includes 8 residual blocks  and  processes inputs of size 256×256×3. It is composed of 84 layers and possesses approximately 1.2× 10^7 trainable parameters, which is approximately an order of magnitude larger than existing works <cit.>.








 
 





  
2. Average pooling Consider that the parameters for average pooling are tailored based on optimally allocated error thresholds that may vary across different keypoints, thereby necessitating distinct pooling parameters for each. To address this issue, the second component includes the division of a multi-channeled unnormalized heatmap layer into individual channels. Each of these channels is then processed through its own average pooling layer. This design allows for simultaneous verification of all keypoints, eliminating the need to verify each keypoint individually.



  
3. Flattening and concatenation. The third component involves the flattening of the previously splitted channels, which are then concatenated into a single-dimensional format for verification.



  §.§.§ Perturbations

 Out of 7000 images, we randomly sample 200 images as seed images, adding local or global perturbations to them.


  
Local object occlusions
We created a set of 40 realistic semantic disturbances featuring personnel and vehicles typically encountered at an airport. These disturbances are depicted in Fig. <ref>. To create perturbed images, we randomly selected 20 objects as patches, each up to 150 pixels in size, positioned randomly on the seed images, with each perturbed image having one patch. The perturbed images are classified into two groups: overlapping and non-overlapping, based on whether the patches overlap with the airplane in the image. There are 4000 perturbed images in total, including 893 overlapping and 3107 non-overlapping images. The convex hull's complexity is adjusted by changing the number of perturbed images m, where m ranges from 2 to 4. These perturbed images are randomly selected, allowing us to systematically assess how the addition of semantic disturbances affects the robustness and performance of the system. Note that the convex hull is comprised solely of either overlapping or non-overlapping images. A collection of perturbed images and sampled images from the convex hull are presented in Fig. <ref>.



  
Local block occlusions Given a seed image, we generate 4 perturbed images. In each perturbed image, 
a 3 × 3 square is placed either away from the airplane or centered over a randomly selected keypoint to create non-overlapping or overlapping images, respectively. All pixel values within this square are randomly selected from the range [0, 255]. A convex hull is formed using these 5 images per seed image, of which all perturbed images are non-overlapping or overlapping ones.



  
Global perturbations
We change each pixel's value through two types of global perturbations: brightness and contrast. For brightness, a variation value b ∈ℤ is applied such that each pixel's value increases by b, that is, I' = (I + b), where I represents the original pixel values, I' the new pixel values, and  ensures the values remain within the range [0, 255]. For contrast perturbation, a variation value c ∈ℝ adjusts each pixel's value by a percentage c, formulated as I' = (I × (1 + c)).  The convex hull is constructed as follows. For a positive value b ∈ℤ_+, two perturbed images are created for b and -b, respectively. These images act as vertices of the convex hull. Along with the seed image, this approach facilitates verification of the model's robustness to any brightness variation within the range [-b, b]. The same methodology applies to contrast variations c ∈ℝ_+. We examine the effects for b values of {1, 2} and c values of 5× 10^-4, 5× 10^-3, 1× 10^-2}.










 

 
 
 
 
 
 




 §.§ Validity of Assumption <ref>


To assess the degree of symmetry and unimodality exhibited by the normalized heatmaps, we use four metrics: skewness, the disparity between mean and median, the disparity between mean and mode (peak), and the Interquartile Range (IQR), which is the difference between the third quartile (Q3) and the first quartile (Q1). A skewness near zero and a minimal difference between the mean and median indicate a symmetrical distribution. Combined with small skewness and close mean and median, a small IQR and close mean and mode suggest unimodality. These metrics are calculated individually for each image dimension. The evaluation uses the ResNet-18-based backbone model  across 3000 seed images and 4000 perturbed images with object occlusions and is presented in Fig. <ref>. The findings reveal skewness values approximately zero and mean and median values that are closely aligned, especially when considering the heatmap dimensions of 256 × 256. The difference between mean and mode and IQR values are also minor relative to the total range of 256. Thus, these statistical outcomes demonstrate that the normalized heatmaps predominantly feature unique peaks and exhibit axis symmetry, supporting Assumption <ref>.







 


 §.§ Probabilistic Soundness and Completeness of Optimal Error Threshold Allocation



  §.§.§ Optimal tolerance allocation

Note that the outcomes of allocating optimal tolerance are influenced by images and defined thresholds for acceptable pose errors. In our experiments, we manipulate these thresholds by incorporating a scaling factor α∈_+, resulting in adjusted error thresholds αϵ_r and αϵ_t.  The pose error thresholds are set as ϵ_r = α· [10^∘, 10^∘, 10^∘] and ϵ_t = α· [4, 4, 20]. The threshold factor α is varied across the values {0.5, 1.0, 1.5, 2.0}. We maintain the scaling factor κ at 1.0 as specified in Eq. (<ref>), and assign the weights w_1 and w_2 values of 1 and 5, respectively. Our analysis includes 3000 images, each contributing 5000 sets of perturbed keypoints, accumulating a total of 1.5 × 10^7 samples. The mean and standard deviation of the tolerance allocated in the horizontal and vertical directions are presented in Fig. <ref>. As the allowable pose errors grows, the tolerance allocated per keypoint increases, predominantly uniform in both the horizontal and vertical axes, with the exception of the 10-th and 11-th keypoints, which are symmetrical with respect to the body axis of the plane (see Fig. <ref>), exhibit larger tolerances horizontally, while the 5-th and 23-th keypoints, aligned along the body axis, have larger tolerances vertically. 

 


  §.§.§ Probabilistic soundness
 In this part, we intend to evaluate the probabilistic soundness of optimal error threshold allocation by verifying if the pose estimation errors resulting from perturbed keypoints fall within the predefined error thresholds. One approach is to perform random sampling within the hyper-rectangle (δ^*). However, given the high dimensionality of , which is 2^2K with K exceeding 20 in our scenario, this method demands an extraordinarily large sample size to achieve sufficient coverage. To address this issue, we only evaluate on vertices. For a seed image we randomly choose vertices from the hyper-rectangle (δ^*), which we then add to the coordinates of the ground-truth keypoints to create perturbed keypoints. Mathematically, in matrix form,
= ⊙δ^* + ,
where the elements of matrix  are randomly set to either -1 or 1, the symbol ⊙ represents element-wise multiplication, and  denotes the perturbed keypoints. This allows the perturbation of keypoints to reach the maximum tolerable errors. 

The simulation setup is largely the same as the previous section. We vary both the scaling factor κ in Eq. (<ref>) and the threshold factor α, and compute the ratio of samples where the pose estimation errors remain within thresholds over the total number of samples.  We report the ratio of samples resulting in tolerable pose errors. The results are summarized in Table <ref>. As we can see, only a small number of samples result in poses that surpass the error threshold. There is a noticeable trend where, vertically, as κ increases, the ratio of acceptable samples rises due to the shrinking of the keypoint error threshold polytope. Horizontally, increasing α results in a lower ratio, as linearization becomes less accurate when the pose moves away from the reference point.
 





















 























  §.§.§ Probabilistic completeness
 
To compute completeness, we randomly sample from the set of tolerable pose errors Ξ, use the perspective projection model (<ref>) to determine keypoints, and then compute the reprojection error relative to the ground truth. We verify whether these reprojection errors remain within the permissible error bounds on keypoints, represented by the hyper-rectangle . This simulation follows the same setup as used for probabilistic soundness. The results are presented in Table <ref>. Vertically, as κ increases, completeness diminishes due to the reduction in the size of , leading to more samples exceeding these bounds. On the other hand, there is no discernible trend when viewed horizontally.

When examining Table <ref>, a trade-off between soundness and completeness is evident. Our framework exhibits better soundness, implying that the set of allocated keypoint errors, , is relatively small compared to the ground-truth set δ_Ξ. This smaller size of  accounts for the reduced completeness observed in the results. To support this, Fig. <ref> illustrates the reprojection errors relative to the optimally allocated bounds (rectangle) for keypoints 1, 3 and 5 from a specific image under the conditions α = 0.5 and κ = 1.0. These reprojection errors are calculated by randomly generating pose errors within tolerable ranges and calculating the difference between ground-truth keypoints and those obtained from perspective projection model (<ref>). The illustration shows that a significant number of samples exceed the computed bounds, contributing to the overall low completeness observed across all keypoints. Therefore, if the verification tool returns a hold, it implies that the pose estimation is robust with high probability. On the other hand, if the tool returns a violation, we cannot draw any definitive conclusions. The conservativeness in error allocation mainly stems from the need to decouple the dependencies among keypoints.





















 §.§ Verification of Local and Global Perturbations

In this section, we assess the robustness of the pose estimation method when subjected to various levels of perturbations. We aim to answer three key questions:

    
    
    
  * How computationally efficient is the resulting neural network verification problem?
    
  * How accurate is the proposed certification method for robust pose estimation?




  §.§.§ Metric


We measure the performance using verification times and verified rates. Verified rate is defined as the proportion of cases where the verification algorithm confirms robustness against those where seed images produce acceptable pose estimation errors.   Another critical measure is verification accuracy which is defined as the proportion of cases where the verification algorithm confirms robustness against those that are indeed robust. However, determining the exact number of truly robust instances is impractical. Consequently, the verified rate serves as a lower bound of verification accuracy as the instances with acceptable pose estimation errors from seed images exceed those that are truly robust.













  §.§.§ Verification results


 We employ the verification toolbox  () <cit.>, which accepts convex hulls as input specifications. is the state-of-the-art verifier that supports a wide range of verification algorithms and is the most user-friendly to extend. It follows a branch-and-bound strategy to divide and conquer the problem efficiently. Two parameters guide this process:  determines the division of an unknown branch into smaller branches for further refinement, and  dictates the approach to navigating through the branch. We set  to use breadth-first search and   to bisect the branch. The computing platform is a Linux server equipped with an Intel CPU with 48 cores running at 2.20GHz and 376GB of total memory, approximately 150GB of which is available owing to multiple users. Additionally, the server includes 4 NVIDIA RTX A4000 GPUs, each with 16GB of memory.

 
 


 
 



  
Local object occlusions for the CNN-based model The results presented in Tab. <ref> indicate that for non-overlapping images, verification time increases as the number m of perturbed images forming the convex hull rises. As the error bounds for pose estimation expand, with threshold factor α increasing from 1.0 to 1.5, the time decreases. A similar effect is observed when the scaling factor κ decreases from 1.5 to 1.0. This reduction occurs because the optimally allocated error thresholds on keypoints expand with increasing α and decreasing κ, as illustrated in Fig. <ref>, which results in fewer nodes. A similar trend is observed for overlapping images. We emphasize that the verification process for overlapping images requires more time than for non-overlapping ones, given the same number of perturbed images and identical pose estimation error bounds, indicating that the keypoint detection model is effective in suppressing disturbances external to the airplane. 

In reference to the verified rates displayed in Tab. <ref>, for non-overlapping images, the rate remains stable regardless of the number of perturbed images, given the same κ and α. Conversely, there is an increase in the verified rate with a decrease in κ and an increase in α, as larger allocated keypoint error thresholds or larger allowable pose error thresholds  result in more images being verified as robust. In the case of overlapping images, a notable trend is the decline in the verified rate when the number of perturbed images increases. This is because a rise in the number of perturbed images means more objects are overlaid on the airplane, which compromises the accuracy of predictions and, in turn, decreases the number of images verified as robust.






  
Global perturbations for the CNN-based model A similar trend to that seen with local perturbations emerges, as indicated in Tables <ref> and <ref>. The verification can handle contrast variations c of 1% and brightness variations b of 2/255. Greater variations in contrast and brightness lead to longer verification times, whereas a larger α reduces verification time and increases the verified rate. 



  
Local block occlusions for the ResNet-18-based model We set κ = 1.0 and α = 1.5. For the convex hull comprised of non-overlapping images, the verification time is 314.4±227.0 in seconds, with a verification rate of 93.5%. Conversely, for the convex hull consisting of overlapping images, the verification time significantly escalates to 1571.7±1213.5 seconds, and the verification rate is 94.0%. 


 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 


 






























 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 









 







§ CONCLUSIONS

In this study, we introduce a framework designed to certify the robustness of learning-based keypoint detection and pose estimation methods. Given system-level requirements, our approach transforms the certification of PnP-based pose estimation into the standard verification for classification, allowing us to leverage off-the-shelf tools. The evaluation results demonstrated that our framework can handle realistic semantic perturbations compared to existing methods. We emphasize that our certification framework is general for safety-critical applications that depend on accurate keypoint detection. These include airport runway detection for automatic landing, pedestrain pose estimation for autonomous driving, and anatomical landmark identification for robot-assisted surgery. Future directions of this framework could include: 1)	Expanding the input specifications to represent more perturbations, such as the translational movement of objects. 2) Reducing the conservativeness caused by independent error allocation among keypoints.



§ ACKNOWLEDGEMENT

This material is based upon work supported by The Boeing Company. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of The Boeing Company.


ACM-Reference-Format




 Paper Title: Scaling Backwards: Minimal Synthetic Pre-training?.









Scaling Backwards







R. Nakamura and R. Tadokoro et al.




National Institute of Advanced Industrial Science and Technology (AIST)

Tohoku University University of Amsterdam, now at University of Technology Nuremberg 
University of Oxford
Tokyo Institute of Technology








Scaling Backwards:
Minimal Synthetic Pre-training?

    Ryo Nakamura1,*
0009-0009-0858-1229 
Ryu Tadokoro2,*
0009-0001-9473-3832
Ryosuke Yamada1
0000-0002-2154-8230


Yuki M. Asano 30000-0002-8533-4020
Iro Laina40000-0001-8857-7709
Christian Rupprecht40000-0003-3994-8045
Nakamasa Inoue50000-0002-9761-4142
Rio Yokota50000-0001-7573-7873
Hirokatsu Kataoka10000-0001-8844-165X

    August 5, 2024
====================================================================================================================================================================================================================================================================================================================================






§ ABSTRACT 
Pre-training and transfer learning are an important building block of current computer vision systems. While pre-training is usually performed on large real-world image datasets, in this paper we ask whether this is truly necessary. To this end, we search for a minimal, purely synthetic pre-training dataset that allows us to achieve performance similar to the 1 million images of ImageNet-1k. We construct such a dataset from a single fractal with perturbations. With this, we contribute three main findings. (i) We show that pre-training is effective even with minimal synthetic images, with performance on par with large-scale pre-training datasets like ImageNet-1k for full fine-tuning. (ii) We investigate the single parameter with which we construct artificial categories for our dataset. We find that while the shape differences can be indistinguishable to humans, they are crucial for obtaining strong performances. (iii) Finally, we investigate the minimal requirements for successful pre-training. Surprisingly, we find that a substantial reduction of synthetic images from 1k to 1 can even lead to an increase in pre-training performance, a motivation to further investigate “scaling backwards”. Finally, we extend our method from synthetic images to real images to see if a single real image can show similar pre-training effect through shape augmentation. We find that the use of grayscale images and affine transformations allows even real images to “scale backwards”. The code is available at <https://github.com/SUPER-TADORY/1p-frac>.







§ INTRODUCTION











^* These authors contributed equally



In image recognition, pre-training allows discovering fundamental visual representations for downstream task applications. Pre-training enhances the performance of visual tasks and enables the use of small-scale task-specific datasets. Recently, pre-training has been used as a key technology to construct foundation models trained on massive datasets with over hundreds of millions of images. 
In some cases, a foundation model enables adaptation of zero-shot recognition without the need for additional data. 


Pre-training is often interpreted as discovering universal structures in large-scale datasets that later facilitate adaptation to down-stream tasks.
In this paper, we challenge this interpretation by providing a minimal pre-training dataset, generated from a single fractal, that achieves similar downstream performance. 
At the heart of this investigation is the question of whether pre-training might simply be a better weight initialization rather than the discovery of useful visual concepts.
If true, performing expensive pre-training with hundreds of millions of images might not be necessary. 


This additionally frees pre-training from licensing or ethical issues.


Since the rise of deep neural networks, the ImageNet dataset <cit.> has been one of the most commonly used pre-training datasets. Originally, pre-training has been conducted through supervised learning (SL) with human-provided teacher labels. However, it has become clear that pre-training can also be achieved without human-provided labels, through self-supervised learning (SSL) <cit.>. 




In this context, Asano et al. <cit.> successfully acquired visual representations while dramatically reducing the number of required images.

They concluded that SSL can yield sufficient image representations even with a single training example, but only for earlier layers of a recognition model.
However, it is unclear how these findings translate to modern architectures and representation learning methods. Based on this, vision transformers (ViT) <cit.> have been pre-trained with only 2,040 real images <cit.> through an instance discrimination learning signal.

More recently, it has been shown that basic visual representations can be acquired without using real images and human-provided labels. The trend of artificially generated labeled images is on the rise for synthetic pre-training <cit.>. Formula-driven supervised learning (FDSL) generates images from generative formulas, and labels from their parameters <cit.>. Under the FDSL framework, one can adapt the synthetic pre-training image dataset by altering formulas <cit.>. 


While a million-order image dataset was constructed in FractalDB <cit.>, our findings suggest that synthetic pre-training can be reduced to significantly fewer fractal images.


Motivated by these findings, we believe it is possible to find the key essence of pre-training for image recognition. 
ViT training can be done with as few as 1,000 artificially generated images <cit.>. Here, we believe that equivalent performance can be achieved with even fewer images. 
This consideration is undoubtedly important as we approach a minimal synthetic pre-training dataset in image recognition, which goes against the trend of foundation models toward increasing the dataset scale.


In the present paper, we thus introduce a minimal synthetic dataset,
namely 1-parameter Fractal as Data (), which consists of a single fractal as shown in Figure <ref>, as well as a loss function for pre-training with it.
Our contributions regarding minimal synthetic pre-training are as follows:

Ordinal minimalism: We introduce the locally perturbed cross entropy (LPCE) loss for pre-training with a single fractal. It utilizes perturbed fractal images for training, where neural networks learn to classify small perturbations.
In experiments, we demonstrate that pre-training can be performed even with a single fractal. The pre-training effect of  is comparable to that of a million-scale labeled image dataset.

Distributional minimalism:
We introduce the locally integrated empirical (LIEP) distribution p_Δ that has a controllable perturbation degree Δ to investigate the minimal support of the probability density distribution of synthesized images.
We observed positive pre-training effects even with a small Δ producing shape differences that cannot be distinguished by humans.
We also show that if Δ is too small
the visual pre-training collapses. From these observations, we establish the general bounds

for generating good pre-training images from a mathematical formula.

Instance minimalism: 
Based on the experimental results, the synthetic images should not simply contain complex shapes. The use of recursive image patterns, similar to objects in nature, should be applied in visual pre-training.
Experiments with augmented categories from a real image have shown that good pre-training effects can be achieved by performing “affine transformations on edge-emphasized objects in grayscale”. These operations are found to be almost synonymous with the configuration of the proposed .



[6]r50mm
   
   
 
   Scaling backwards in synthetic pre-training (Accuracies on CIFAR-100, Real: ImageNet, Synth: Fractal images).
   
    
        [0.8pt]
        
        Type\#Img     1     1k     1M 

        [0.5pt]
        Real     N/A     76.9     85.5  

        [gray]0.8 Synth     84.2     84.0     81.6 

        
    
    



In summary, we significantly reduce the size of the pre-training dataset from 1M images (fractal database (FractalDB) <cit.>) or 1k images (one-instance fractal database (OFDB) <cit.>) down to 1 and show that this even improves the pre-training effect (Table <ref>), which motivates “scaling backwards”.  






§ RELATED WORK



The present study focuses primarily on finding the minimal pre-training dataset with artificially generated images. 




  
Pre-training in image recognition. Pre-training in image recognition has demonstrated significant success with vast amounts of labeled images. Pre-training helps neural networks to acquire fundamental visual representations which are essential for improving the performance of diverse downstream tasks in image recognition. In particular, researchers have used larger-scale image datasets for supervised learning (SL), starting from million-scale datasets (e.g., ImageNet <cit.>, Places <cit.>). More recently, 
the largest datasets used are  
reaching billion-scale (e.g., IG-3.5B <cit.>, JFT-300M/3B/4B <cit.>). 

Pre-training can be also achieved without any human intervention.
One of the most effective approaches is self-supervised learning (SSL <cit.>), which has been developed in order to alleviate the manual annotations on large-amount of images. SSL has been studied as an expected alternative to supervised pre-training and has sometimes been shown to surpass supervised pre-training. 

Despite the development of supervised and self-supervised pre-training, the number of real images used in visual pre-training continues to increase.

Exploratory research questions such as “what is a minimal synthetic pre-training dataset?” have received less attention.
However, they could give us more insights on what is learned during pre-training and how it could be improved.




  
Pre-training with limited data. In order to address ethical issues such as privacy and fairness associated with real images, a pre-training with a very limited number of real images and synthetic pre-training without any real images have been on the rise. In this context, synthetic pre-training with copy-and-paste learning <cit.>, domain randomization from 3D graphics <cit.>, learning from primitive patterns <cit.>, and dataset distillation <cit.> have been recognized as promising approaches to acquiring visual representations. More recently, a sophisticated approach, formula-driven supervised learning (FDSL) <cit.>, enabled automatic construction of a labeled image dataset without any real images or human annotations. FDSL can generate images and their corresponding labels using a mathematical formula. Along these lines, One-instance FractalDB (OFDB) has successfully pre-trained a ViT using only 1,000 synthetic images generated from a mathematical formula, without using real images <cit.>. Moreover, several studies have been reported in the context of extremely limited data <cit.>. In particular, Asano et al. have shown that learning visual representations is possible using a single image for self-supervised learning. Venkataramanan et al. achieved a comparable method to ImageNet pre-training with only 10 walking videos and their cropped images. 


By analyzing the acquisition of visual representations through the use of minimal synthetic pre-training datasets, we believe that the FDSL framework will be key to discovering the essence of pre-training mechanisms in image recognition.




§ SCALING BACKWARDS WITH A SINGLE FRACTAL


This section presents the proposed methodology to explore the minimal requirements for successful pre-training by utilizing purely synthetic images of fractals.
Specifically, we introduce the 1 parameter Fractal as Data (), which consists of only a single fractal, with a method to pre-train neural networks on it.
Our key idea is to introduce the locally integrated empirical (LIEP) distribution p_Δ over perturbed fractal images, enabling pre-training even with a single fractal image. Since the LIEP distribution is designed so that it converges to the empirical distribution p_data(x) = δ(x-I) of a single image I when the perturbation degree Δ∈ℝ_≥ 0 goes to zero, we can narrow down the support of the distribution by decreasing Δ as shown in Figure <ref>a.
Below, we begin with a preliminary for defining the empirical distributions of two fractal databases, namely FractalDB <cit.> and OFDB <cit.>.
We then introduce  with the LIEP distribution.



 §.§ Preliminary


FractalDB <cit.>. 
Kataoka et al. have introduced a method for effectively pre-training neural networks with FractalDB, a set of fractal images generated by the iterated function systems (IFSs).
Specifically, FractalDB ℱ consists of one million synthesized images: ℱ =
{(Ω_c, {I_i^c}_i=0^M-1)}_c=0^C-1, where Ω_c is an IFS, I^c_i is a fractal image generated by Ω_c, C = 1,000 is the number of fractal categories, and M = 1,000 is the number of images per category.
Each IFS characterizes each fractal category c and is defined as follows:

    Ω_c = {ℝ^2; w_1, w_2, …, w_N_c; p_1, p_2, …, p_N_c},

where w_j : ℝ^2→ℝ^2 is a 2D affine transformation given by

    w_j(v)
    =
    [ a_j b_j; c_j d_j ]v +
    [ e_j; f_j ] (v∈ℝ^2),

and p_j is a probability mass distribution.
Each fractal image I^c_i renders a fractal F = {v_t}_t=0^T⊂ℝ^2 into a 2D image, where points v_t are determined by a recurrence relation that applies affine transformations as v_t+1 = w_σ_t (v_t) for t = 0, 1, 2, ⋯, T-1. Here, the initial point is set as v_0 = (0, 0)^⊤, and the index σ_t is sampled at each t independently from the probability mass distribution p(σ_t = j) = p_j.
Pre-training with FractalDB utilizes the cross-entropy loss function:

    ℒ = -𝔼_x,y ∼ p_data[log p_θ(y|x)],

where p_θ is the category distribution predicted by a neural network and θ is a set of learnable parameters.
The joint empirical distribution p_data is defined over the dataset as follows:

    p_data(x,y; ℱ) = 1/MC∑_i=0^M-1∑_c=0^C-1δ(x-I^c_i) δ(y - c)

where δ is the Dirac's delta function.
Models pre-trained on this dataset perform comparable to those pre-trained on real-world image datasets, such as ImageNet-1k and Places365, in some downstream tasks.




OFDB <cit.>.
This dataset consists of 1,000 fractal images.
Specifically, OFDB ℱ_OF involves only one representative image per category, i.e., ℱ_OF =
{Ω_c, I_c}_c=0^C-1.
Therefore, the joint empirical distribution reduces to 

    p_data(x,y; ℱ_OF) = 1/C∑_c=0^C-1δ(x-I_c) δ(y - c).

Models pre-trained on this dataset perform comparable to or even better than those pre-trained on FractalDB. This work has shown that there exists a small but essential set of images for visual pre-training.
However, reducing the number of fractals C to less than 1,000 degrades the performance.



 §.§ Pre-training with a Single Fractal


Scaling backwards.
To further facilitate the analysis of what images are minimally required for successful visual pre-training, we introduce , which ultimately reduces the number of IFSs and images to one each as ℱ_OP = (Ω, I).
With this dataset, the empirical distribution is given by

    p_data(x,y; ℱ_OP) = δ(x - I) δ(y).

However, we notice that with this distribution, training neural networks using cross-entropy loss is not straightforward because p_θ(y=0|x) ≡ 1 (∀ x) gives a trivial solution to the loss minimization problem.
To address this, we introduce
locally perturbed cross entropy (LPCE) loss ℒ_Δ, a variant of the cross entropy loss defined with the LIEP distribution.


Definition 1.

Let I_ϵ∈𝒳 be a perturbed image, where 𝒳 is a set of images, ϵ∈ℝ^d is a small perturbation with d ∈ℕ_>0 and I_0 = I is the original image.
We define the LIEP distribution by

    p_Δ(x, y)
       =
    1/|ℛ_Δ|∫_ℛ_Δδ(x - I_ϵ) δ(y - ϵ) d ϵ

where ℛ_Δ⊂ℝ^d is a compact set containing the origin and |ℛ_Δ| is its volume with order O(|Δ|^d).


Definition 2.

We define the LPCE loss by

    ℒ_Δ   = -𝔼_x, y ∼ p_Δ[ log p_θ(y | x) ],

where p_Δ is the LIEP distribution.


If ℛ_Δ is a small hypercube or hypersphere, it is straightforward to see that p_Δ approaches to the empirical distribution of Eq. (<ref>) when Δ goes to zero.
Therefore, this loss allows us to analyze visual pre-training effects by narrowing the support of the distribution around a single image.

With , we apply perturbation to the affine transformations. As such, the perturbation ϵ is in ℝ^6*j. A perturbed image I_ϵ is obtained by the noisy affine transformations:

    w_j(v; ϵ_j
    )
    =
    ( [ a_j b_j e_j; c_j d_j f_j ]
    +
    ϵ_j)
    [ v; 1 ]

where ϵ_j ∈ℛ_Δ = [-Δ/2, Δ/2]^6*j⊂ℝ^6*j is a hypercube with a side length Δ, and
|ℛ_Δ| = Δ^6*j.
Note that the numerical integration of Eq. (<ref>) is used in practice, which is an approximation obtained by uniformly sampling L points in ℛ_Δ, where we set L = 1,000 by default.

Visualization.
Figure <ref>b shows examples of perturbed images to compute the LPCE loss. While most of shape differences are indistinguishable to humans, a neural network learns to distinguish the perturbations applied to the single image by minimizing the LPCE loss.

Complexity of Ω.
We use the σ-factor proposed by Anderson et al. <cit.> to evaluate the complexity of IFSs.
As shown in Figure <ref>c,
small values of σ produce complex fractal shapes.







§ EXPERIMENTS

Experimental setup. What is minimal visual pre-training, and what properties of images lend themselves to reducing their count? In order to clarify these questions, we conduct experiments following these steps:


    
  * Exploration study (Tables <ref>-<ref>): We verify whether a single fractal is sufficient for pre-training. Moreover, we compare the extent to which the shapes used in pre-training can be simplified, using object contours, Gaussian, and uniform distributions.
    
  * Hyperparameter study (Tables <ref>-<ref>): We investigate the effects of three hyperparameters Δ, σ and L.
    
  * Scaling study (Table <ref>): We compare  with other large-scale datasets in terms of fine-tuning accuracy on ImageNet-1k.
    
  * Analysis and discussion (Tables <ref>-<ref>): We discuss data augmentation, computational cost for synthesizing images, and pre-training using a single real-world image.
    
  * Applications (Tables <ref> and <ref>): We conduct comparisons in various fine-tuning datasets.


Implementation details. In order to verify the effects of pre-training, we measure the accuracy of downstream tasks through fine-tuning. We use ViT <cit.> for all experiments. Specifically, we employ ViT-Tiny (ViT-T) in the the exploration and hyperparameter studies, and employ ViT-Base (ViT-B) for the scaling study. The pre-training parameters and data augmentation methods for pre-training and fine-tuning follow conventional methods in OFDB. In OFDB, data augmentation is based on DeiT. For exploration studies, CIFAR-100 (C100) <cit.> and/or ImageNet-100 (IN100) <cit.> are used as fine-tuning datasets. In the comparisons, we assign seven representative datasets used in OFDB paper, namely C10/C100, Cars <cit.>, Flowers <cit.>, Pascal VOC 2012 (VOC12) <cit.>, Places-30 (P30) <cit.>, IN100. See supplementary material for more detailed experimental settings.



 §.§ Exploration Study


Pre-training with .
Table <ref> compares  with FracdalDB (1M images, 1k categories) and OFDB (1k images, 1k categories).

As can be seen, the pre-training effect of  is comparable to that of FractalDB and OFDB. The number of fractal categories is significantly reduced from C=1,000 to only 1.
This shows the effectiveness and efficiency of  and the LPCE loss.


Comparison with SOTA FDSL datasets.
Table <ref> compares
 with two SOTA FDSL datasets:
(1) Radial Contour Database (RCDB) <cit.>, which consists of 1 million images of synthetic polygons, and (2) Visual Atoms (VA) <cit.>, which consists of 1 million images of parameterized wave functions.
The “1K” counterparts in the table reduce the number of images per category to one.
We also applied the LPCE loss to these two datasets by applying perturbations to the radios of a polygon or wave.
As can be seen,  performs the best when a single image is used for pre-training and outperforms the 1K FDSL datasets.


















Pre-training with a noisy image.

To validate the necessity of a complex shape when pre-training with a single image, we applied the LPCE loss to two noisy images: an image of Gaussian noise and an image of uniform noise. 


Since these two noise images are generated by determining the parameters of the parametric distributions, Gaussian and uniform, the LPCE loss can be computed by giving small perturbations to the parameters.
Table <ref> shows these images and fine-tuning results.
For example, here the ϵ is added to the mean vector and covariance matrix of the Gaussian distribution. 
The results revealed that noise images fail in visual pre-training, performing worse than training from scratch.
This suggests that acquiring fundamental visual representations through pre-training requires an image of a certain structured object, such as a fractal.






 §.§ Hyperparameter Study

Perturbation degree. Table <ref> shows the results obtained by different perturbation degrees Δ for the LIEP distribution.
The results indicate that setting the value to 0.1 yields the highest performance.
Interestingly, the performance significantly improved when Δ was increased from 0.01 to 0.05.
This suggests that the support of the empirical distribution must have a certain size to obtain positive pre-training effects.


σ-factor.
Table <ref> investigates the effects of the σ-factor, which measures the complexity of fractal shapes.
As can be seen, the most complex shape yields the highest performance.
It is worth noting that, even with a fractal of σ = 6.0, which looks like Gaussian noise, we observed positive pre-training effects.
This suggests that the shapes obtained from IFS are crucial for pre-training.










Numerical integration. 
Tables <ref> reduces the number of sampling points L of the numerical integration when approximately computing loss with Eq. (<ref>).
We see that larger numbers result in high performance.
Interestingly, even with L=16, the pre-training effect was positive (better than training from scratch).





 §.§ Scaling Study


ImageNet-1k fine-tuning (Table <ref>). 
While this work discusses minimal requirements for visual pre-training, increasing the approximation precision of the numerical integration could further improve the performance and benefit training of large models.
In Table <ref>, we applied the LPCE loss with L=21,000 to the ViT-B model and compare fine-tuning accuracy on ImageNet-1k.

Surprisingly, and despite using a single fractal as data, our ViT-B pre-trained on  outperforms pre-training on ImageNet-21k.
This shows the strength of our approach in particular because we simply keep the same fine-tuning protocols as used for the original Imagenet-21k to ImageNet-1k transfer proposed in <cit.>, putting us at a potential disadvantage.






 §.§ Analysis and Discussion

Relationship with data augmentation (Table <ref>). We consider that the variation in image patterns also depends on data augmentation. While traditional methods primarily rely on the data augmentation techniques described in the DeiT paper <cit.>, we investigated which data augmentation methods contribute significantly by excluding one or two techniques from the base data augmentation methods and observing the changes in fine-tuning accuracy on C100 dataset. The experimental results in Table <ref> reveal that excluding Random Cropping (RandomCrop) significantly deteriorates accuracy, 

suggesting that omitting parts of the image region and causing partial loss facilitates the acquisition of the ability to focus on any part without relying on grasping the overall shape,
greatly reducing the pre-training effect. Furthermore, when both Mixup and Cutmix are excluded, a degradation in accuracy is observed, indicating that mixing images and categories enables better feature recognition.





Dataset construction times (Table <ref>). 
We calculated the processing times in fractal category search and image rendering. By comparing to the conventional approaches, our  recorded much faster category search (0.0022 hours nearly equals to 8 seconds) and image rendering (0.036 hours nearly equals to 129 seconds).  requires only one category with a parameter set. This very simple procedure leads to efficiently construct a minimum requirement for pre-training in image recognition.



Linear probing (Figure <ref>). 
To compare supervised ImageNet-1k pre-training with our  pre-training in the context of early layer visual representations, we executed a linear probing experiment on the C100 dataset. In the ViT-T model with {1, 2, 3} layers, the  pre-trained model outperformed the model pre-trained on ImageNet-1k. This demonstrates the superior quality of early, low-level representations learned from our dataset compared to ImageNet.
In effect, layers up to layer 3 in ViT-T can be sufficiently pre-trained and frozen using our .



Pre-training with a single real image and shape augmentation.
[8]r0.6 
  

  
    < g r a p h i c s >

  Shape augmentation with three geometry transformations from a single real image.
  
  



It was found that pre-training effects comparable to a dataset of one million real-world images can be achieved with , but is it possible to perform pre-training similarly with a real image? 
In this section, we verify whether visual features can be acquired through pre-training using a single real image (each image in Figure <ref>) with the LPCE loss using image representations similar to FDSL (Canny <cit.> images) and geometric transformations for perturbation ({affine, elastic, polynomial} transformations in Figure <ref>). 

The results of pre-training effects with L=1,000

are shown in Tables <ref> and <ref>. Note that the results in these tables represent the average score on C100 over the five real images in Figure <ref>. See the supplementary material for results for each of image.
According to the results, we find that contour-emphasized Canny images omitting color information and affine transformation perturbation yield better pre-training, as measured by the fine-tuning accuracies. The operations are similar to the classification of labeled fractal images via IFS.










































































































 §.§ Applications

Fine-tuning on commonly used datasets (Table <ref>). We compare fine-tuning accuracies in three types of dataset configurations. For supervised learning (SL) we compare pretraining on ImageNet-1k and Places-365, for self-supervised learning (SSL) we compare  DINO <cit.> and masked auto-encoder (MAE) <cit.> pre-trained on ImageNet and PASS, and models trained on the fractal datasets FDSL on FractalDB-1k and RCDB-1k. Following the setting of one-instance per category <cit.>, we also evaluate using random patch augmentation (“w/ Aug.”). 

We compare our  with 1-image SSL <cit.> with DINO and MAE (we use Real Img #1 of Figure <ref>). 

The results show that our  pre-training with one single image demonstrated high fine-tuning accuracies (87.5% and 88.2% with and without random patch augmentation) which are close to the performance of self-supervised ImageNet-1k pre-training with MAE that utilizes 1.28M images and yields 88.5%. Surprisingly, despite using only one fractal image with perturbations, our  without random patch augmentation has demonstrated fine-tuning accuracies that are equal to or even higher than that of conventional OFDB-1k/FractalDB-1k, which uses 1,000 parameter sets with fractal categories and 1,000/1,000,000 images:  obtains mean performances of 87.5%  vs OFDB-1k 87.2% vs FractalDB-1k 87.1%.


Fine-tuning on specialized and structured datasets in VTAB <cit.> (Table <ref>). We also verify performance on a couple of specialized and structured datasets in VTAB (visual task adaptation benchmark; Retinopathy <cit.>, Resisc45 <cit.>, Camelyon <cit.>, CLEVR-Count <cit.>, and sNORB-Azim <cit.>). The experiment is more suitable to evaluate and compare the pre-training methods since the image domains and their labels are distinct from both real and fractal images on ImageNet and . According to Table <ref>, our  pre-trained model performed relatively similar fine-tuning accuracies on all listed datasets in the table. By comparing to self-supervised ImageNet-1k with MAE, our  surpassed the fine-tuning accuracies on three (Retinopathy, Resisc45, and sNORB-Azim) out of the five datasets. 





§ CONCLUSION AND DISCUSSION

This paper examined what a minimal dataset for synthetic pre-training might look like. We proposed the 1-parameter Fractal as Data () dataset that succeeds in pre-training even with a single fractal by utilizing the locally perturbed cross entropy (LPCE) loss.
The following findings were presented:

What is a minimal synthetic pre-training dataset? In this paper, we applied IFS to generate labeled images. In this context, preparing a single set of parameters a-f in IFS is the least informative. It recorded fine-tuning accuracy nearly equivalent to conventional methods such as FractalDB and OFDB (Tables <ref> and <ref>), succeeding in pre-training with minimal image pattern utilization. Moreover, it is not that any perturbation in image space is acceptable, as Gaussian/uniform distributions failed in pre-training (Table <ref>). It clarified the importance of geometric transformations with certain regularities like fractal geometry or contour shapes (Table <ref>).

What properties of images lead to image reduction? In , we investigated shape changes and pre-training effects by adopting the LPCE loss and the σ-factor that contribute to fractal shape variation. It was revealed that pre-training efficacy varies with shape variation (Tables <ref>, <ref>). For real images, it is not just about including more edge components, but about including affine transformations, 

which closely resembles the classification of labeled fractal images through IFS (Tables <ref> and <ref>).

By extensively ablating data augmentation showed, we demonstrated the essential role of random cropping and MixUp.  


Pre-training effects with the proposed method. Supervised pre-training on ImageNet was more accurate for datasets where object labels are assigned to real images (Table <ref>), but in VTAB's Specialized/Structured datasets (Table <ref>), performances were relatively close. On the other hand, our  pre-trained ViT performed a similar mean accuracy by comparing to MAE self-supervised ImageNet pre-training (MAE 88.5 vs. our  w/ Aug. 88.2).
Furthermore,
increasing the number of sampling points L showed performance improvement even with 21k categories and demonstrated good performance in ImageNet-1k fine-tuning (Table <ref>), indicating the potential for building large models with limited data resources.


Broader impact.
Our proposed dataset  does not suffer from licensing or ethical issues such as biases and has a large potential to serve as a clean pretraining dataset. 
While not investigated in this paper, this approach also opens the possibility for  significant gains in training speed by keeping the data on the GPU and applying transformations there, removing the GPU-CPU transfer bottleneck.

Limitation. In this paper, we have explored achieving a minimal synthetic pre-training in image recognition based on fractal geometry with IFS (a single fractal, and their perturbations). 
Despite this, we believe it is also necessary to find a similarly minimal pre-training dataset containing real images – for some applications and increased interpretability. This might allow for quick and clear calibration of pre-training, similar to how calibration images are used in photography. We did not investigate this direction in this paper but leave this for future work.





§ ACKNOWLEDGEMENTS



Computational resource of AI Bridging Cloud Infrastructure (ABCI) provided by National Institute of Advanced Industrial Science and Technology (AIST) was used.









splncs04






















 Paper Title: Virchow 2: Scaling Self-Supervised Mixed Magnification Models in Pathology.



Matched Guiding and Controlled Injection in Dark-Current-Free, 10-GeV-Class, Channel-Guided Laser Plasma Accelerators 


    A. J. Gonsalves
    August 5, 2024
========================================================================================================================

‡ Corresponding author. siqi.liu AT paige DOT aifootnote

‡ Corresponding author. kseverson AT microsoft DOT com
footnote


§ ABSTRACT 
    Foundation models are rapidly being developed for computational pathology applications. However, it remains an open question which factors are most important for downstream performance with data scale and diversity, model size, and training algorithm all playing a role. 
    In this work, we present the result of scaling both data and model size, surpassing previous studies in both dimensions, and introduce two new models: Virchow 2, a 632M parameter vision transformer, and Virchow 2G, a 1.85B parameter vision transformer, each trained with 3.1M histopathology whole slide images. 
    To support this scale, we propose domain-inspired adaptations to the DINOv2 training algorithm, which is quickly becoming the default method in self-supervised learning for computational pathology. We achieve state of the art performance on twelve tile-level tasks, as compared to the top performing competing models. Our results suggest that data diversity and domain-specific training can outperform models that only scale in the number of parameters, but, on average, performance benefits from domain-tailoring, data scale, and model scale. 




§ INTRODUCTION


Stained tissue microscopy slides, also known as whole slide images (WSIs), are routinely collected as part of the standard of care in cancer. The digitization of these slides, at increasingly large scales, has enabled a revolution in computational pathology (CPath) towards the use of foundation models. The interest in and success of foundation models are primarily driven by the ability to learn general representations from vast amounts data that include diverse stains, tissue types, and disease without task-specific labels via self-supervised learning approaches. The use of these models represents a paradigm shift in CPath, where previously, expert systems were trained on a single tissue type with expensive, curated labels for a single task. The generalizability and robustness of a foundation model is therefore desirable for the pathology domain as there are many tasks such as diagnosis, disease subtyping, biomarker quantification, estimation of treatment response, and survival prediction. Motivated by these factors, there have been recent efforts to collect large pathology image datasets and subsequently several foundation models have been proposed  <cit.>. 

Scaling up the dataset and model size has been observed to significantly boost the performance and transferability of foundation models in other domains <cit.>. In the natural image domain, foundation models use millions of images (e.g. ImageNet <cit.>, JFT-300M <cit.> and LVD-142M <cit.>) to train models with hundreds of millions to billions of parameters (e.g. ViT <cit.>). Early works <cit.> have suggested these trends exist in CPath as well, both in terms of dataset and model size, however the topic remains underexplored. Furthermore, the extent to which scaling trends observed in the natural image domain hold without adaptation to the unique aspects of CPath data is unknown.





In this work, we explore scaling along both the data and model size axes, while adjusting the training method with domain-inspired modifications. We present Virchow 2 and Virchow 2G, mixed magnification foundation models for computational pathology. Virchow 2, a 632M parameter vision transformer (ViT-H) expands on our work developing Virchow <cit.>, scaling the dataset size from 1.5M WSIs from a single institution with routine hemotoxylin and eosin (H&E) staining, to 3.1M WSIs from globally diverse institutions with diverse staining such as immunohistochemistry (IHC). Virchow 2G builds on Virchow 2 by increasing the model size to 1.8B parameters (ViT-G).

We evaluate performance on various in-domain and out-of domain benchmarks at multiple magnifications and demonstrate the benefits of our domain-inspired adaptations at scale.


§ BACKGROUND & RELATED WORK





 §.§ Self-Supervised Learning

Self-supervision is a learning paradigm  where generalizable features are learned from data using a pretext task, an objective that is constructed from implicit information contained in an input. Joint-embedding self-supervised learning (JE-SSL) methods are a subclass of SSL approaches that pose the learning objective in terms of alignment and diversity <cit.>. Alignment is accomplished by encouraging features, otherwise known as embeddings, of pairs or sets of samples generated from the same source image via the application of an augmentation policy to be close to one another <cit.>. Diversity provides the necessary support to learn representations that avoid collapse or trivial solutions across the entire set of observations  <cit.>. These methods, in particular self-distillation variants, have emerged as powerful techniques in no-tuning transfer, both in natural images and in CPath <cit.>. 




 §.§ CPath Foundation Models


In the past two years, at least ten CPath foundation models have been proposed in the literature (see Tab. <ref> for an overview). The CTransPath model <cit.> was the first and trained a 28M parameter Swin transformer model using MoCoV3 <cit.> using 15M tiles from 32K WSIs. Since that time, two works <cit.> have proposed models using SimCLR <cit.>, but the vast majority <cit.> have used DINOv2 <cit.> or the related iBOT <cit.>. The prevalence of DINOv2 appears to be largely motivated by its success with natural images as few studies have released comparative analyses for the choice of algorithm. Kang et al. <cit.> compared four algorithms,  DINO <cit.>, MoCo v2 <cit.>, SwAV <cit.> and Barlow Twins <cit.> and found no clear best method. Conversely, Chen et al. <cit.> found that the choice of algorithm has a large effect on performance, observing that MoCoV3 significantly underperformed DINOv2 in CPath, despite the two approaches having comparable performance on natural image tasks.

Although CPath foundation models have largely aligned on their use of DINOv2, they are quite diverse in terms of the training dataset. Earlier works <cit.> relied primarily on open access data, specifically The Cancer Genome Atlas (TCGA) <cit.>, a repository of approximately 30K WSIs. Models were trained using this data with varying numbers of tile samples ranging from 4-50M. More recently <cit.>, proprietary datasets ranging from 100K-1M WSIs have emerged, allowing for training datasest sizes to reach billions of tiles. It is worth noting that characterizing data scale in CPath is non-obvious as various works have quantified it in terms of WSIs, tiles (crops of WSIs used for training), and patients. There are further data dimensions which are also of interest such as source institution, disease status, tissue of origin, stain, and scanner.




§ LEARNING IN THE PATHOLOGY DOMAIN



Unlike natural images, which are a 2D projection of a 3D scene, CPath images are acquired from essentially 2D slices of tissue that are stained and scanned at various resolutions or magnifications (e.g. 5×, 10×, 20×, 40×) through a digitization process. The resulting image features vastly differ from those found in the natural image domain, as tissue patterns are repetitive, non-unique, pose-invariant, and contain meaningful but minimal color variation due to routine staining procedures, such as H&E. CPath foundation models are trained with self-supervised learning on tissue tiles, i.e. image crops, as WSIs are gigapixel images that are costly to train with directly. Tile size is selected based on compute cost, and the quality of tile content can be highly variable. Despite these differences from the natural image domain, most CPath foundation models have directly used training algorithms developed for natural images and instead focus on the dataset as the primary differentiator. Given the crucial impact of augmentation, algorithm, and model size on foundation model performance, we present modifications to the DINOv2 recipe at scale. In the following sections, we address domain differences by adopting more domain-relevant augmentations and regularization techniques through the lens of feature alignment and diversification. 



 §.§ Domain-specific augmentations


In the context of joint-embedding self-supervised learning, feature invariance and equivariance are learned through selective alignment of image pairs, also known as views, that are generated by perturbing an input with a set of random augmentations. The quality of features is dependent on the complexity of the pretext task as well as its relationship to the downstream objective. 



Augmentations are typically categorized as either photometric or geometric, and most augmentation strategies have been designed for object-centric natural images. In the majority of instances, these augmentations are directly applied in pathology applications, but there are a small number of exceptions. Examples of pathology-specific photometric augmentations include stain jitter and transfer <cit.>, where colors are augmented within and across tiles, aiming to account for differences in staining protocols and facilitate the learning of color invariances. In benchmarking studies, Tellez et al. <cit.>, Ciga et al. <cit.>, and Gullapally et al. <cit.> investigated the impact of color augmentation and demonstrated improvements in performance using domain-specific approaches. Since then, several models have employed domain-orientated approaches <cit.>. However, the benefit of these augmentations when applied to mixed stain datasets, e.g. datasets that contain IHC, is unclear. Other photometric augmentation methods such as image solarization have been omitted in some CPath works  <cit.>, as this augmentation is hypothesized to generate color profiles that are not useful for learning relevant invariances. 

The random crop operation is arguably the most important geometric augmentation <cit.>. This augmentation introduces feature co-occurrences within and across samples. These co-occurrences may be explicitly generated based on the augmentation parameters which in turn govern the amount of expected overlap  <cit.> or implicitly generated based on the data distribution. The crop augmentation is paired with a resize operation due to a fixed input size and a desire for scale invariance (the resulting combination is referred to as crop-and-resize). As cell morphology plays an important role in understanding tissue structure and disease, depending on the aspect ratio of the crop, a resize may introduce unwanted distortions, affecting tissue and cell shapes.
Indeed, Ciga et al. <cit.> investigated the impact of random cropping on model performance and found less random cropping generally improved performance although the largest observed improvement in any setting was 5%. Chen et al. <cit.> deviates from DINOv2's default parameters and explores less aggressive crop-and-resize scale parameters without changing aspect ratio.
As noted above, one unique aspect of pathology data is that tiles are sourced from larger WSIs. As a result of this workflow, image size is a design variable and boundaries can be extended without the need for padding or resizing. Because of this, we propose a translation-like augmentation that uses a larger source field of view to create crops with minimal resizing thereby minimizing morphological distortions while maintaining the same output size and expected overlap between image views. This operation differs from typical translations as it does not introduce imaging artifacts along boundaries, i.e. padded values. It also benefits from extending the relative field-of-view of the WSI without computational overhead. To demonstrate how this is possible, consider replacing the L × L-sized training tile with an N × N-sized source region and a target L × L tile to be sub-sampled within the larger window, where N > L (see Fig. <ref>). The particular value for N can be selected based on the desired intersection over union of views. This alternative augmentation, which we refer to as extended-context translation (ECT), can serve as a drop-in replacement for traditional crop-and-resize approaches and can be combined with other augmentations such as photometric augmentations. 





 §.§ Accounting for Tissue Redundancy


Self-supervised alignment tasks are inherently unstable and require additional regularization to encourage diversity and avoid dimension collapse. DINOv2 encourages diversity through contrastive and non-contrastive objectives. Non-contrastive diversity is achieved using asymmetric centered and sharpened distillation between the student and teacher models, while contrastive diversity aims to maximize differential entropy using KoLeo. Different diversity regularization objectives often share similar optima such as uniformity of the hypersphere; however, the dynamics of each method throughout training are considerably different and introduce practical challenges that must be addressed.

Feature diversity is dependent on the training data distribution and training methodology. In CPath, the likelihood of contrasting two similar tissue tiles may be high, unlike when contrasting images from large uncurated natural image datasets. Therefore, any contrastive diversity estimators should be constructed under the assumption that features are not independent, may be very close together, and cannot be arbitrarily separated. DINOv2 approximates differential entropy using KoLeo defined as 

    H_KoLeo(f) = 1/n∑_i=1^n min_j ≠ ilog d(𝐳_i, 𝐳_j),

where i indexes the samples in a batch of size n, d is a distance measure and 𝐳 is an embedding normalized to the hypersphere <cit.>. The inclusion of an entropy maximization objective using KoLeo aims to spread out learned embeddings via a convex regularizer. While this method has demonstrated value in natural image pretraining settings for linear and nearest neighbour classification objectives where sample diversity is high  <cit.>, it is clear that when two samples are very similar, and distance approaches zero, the loss approaches infinity. In practice, KoLeo is stabilized with the addition of an ϵ > 0 and is implemented such that samples are not compared across devices. These factors introduce numeric and hardware dependent hyperparameter that are unexplored.

Any entropy estimator can be used as an effective replacement in order to mitigate the issues associated with KoLeo in a setting where features may be clustered, if the estimator is bounded for nearby points. We select the kernel density estimator (KDE), defined as

    H_KDE(f) = - 1/n∑_i=1^n  log∑_j = 1^n k(𝐳_i, 𝐳_j),

where k is an appropriate kernel function and all other terms are as defined above. Self-comparisons ensure the the gradients are bounded. Similar objectives have been used in various other methods <cit.>. The kernel density estimator has different regularization effects for local behaviors, i.e. for samples with large and small distances, but maintains the population effect of promoting diversity. Examples of kernels include Gaussian, von Mises-Fisher, inverse multiquadratic, or Laplacian kernel each of which allows for different repulsion characteristics that have bounded gradients which can also vanish for a given pair. In our work, density estimation for entropy regularization is computed using an unnormalized von Mises-Fisher (vMF) kernel, 

    k_vMF(𝐱, 𝐲) = exp(κ𝐱^⊤𝐲),

where κ is a scaling constant also known as the concentration. The vMF kernel is selected because of its favorable computational qualities and its demonstrated success in encouraging diverse embeddings <cit.>. 




§ STABILIZING SELF-SUPERVISED VISION TRANSFORMERS AT SCALE


Vision transformers are notoriously challenging to train. Architecture based instability in self-supervised learning protocols have been empirically correlated with high gradient norms occurring in the patch embedding layer at the stem of the network for large batch sizes <cit.>. As model size is increased, large gradient norms become increasingly problematic across layers of the ViT and partial failures may not be recoverable without the manual intervention of a complete rollback prior to the event. This is costly, time consuming, and may not guarantee success. To address the architectural challenges associated with training large models, dual patch normalization (DPN) <cit.> and query-key normalization (QKN) <cit.> have been introduced in the literature and are adopted in our work. Both normalization techniques aim to limit problematic gradients, aside from gradient clipping techniques, by standardizing activations throughout the network. The inclusion of DPN circumvents the need to freeze the randomly initialized weights of the linear projection in the patch embedding layer.

Optimizing ViTs requires adaptive methods like AdamW and its variations <cit.>. The stability of AdamW depends on the efficacy of the warm-up phase, on the learning rate, and on the selection of momentum parameters. Gradient spikes have been correlated with the ratio of first and second moments for an update step which provide insights on when a collapse event is likely to occur <cit.>. For example, instability has been observed in the weights of the patch embedding layer <cit.>. It is possible to reduce these issues by lowering the second moment hyperparameter but this may incur reduced downstream task performance. StableAdamW is explored as a means of stabilizing weight updates when the ratio between the first and second moments are too large (such as when nearly converged weights suddenly need a correction during a long training run) by mitigating large update steps with root-mean-square clipping <cit.>.

Aside from architectural and optimization instability, self-supervised methods commonly suffer from partial or complete dimension collapse as a result of the imbalance between alignment and diversity objectives. Like DINO, DINOv2 primarily avoids collapse over class and patch tokens using an asymmetric centered and sharpened cross-entropy loss between the student and teacher networks. Stability over the distillation objective is highly dependent on the sharpness of the distributions which is controlled by temperature parameters. DINO studied the effects of varying teacher temperature for a fixed student temperature. Results demonstrated that higher teacher temperatures yield better transfer performance but require a warm-up period to avoid collapse, whereas a lower choice of temperature produced marginal performance drops with increased stability <cit.>. Moreover, registers <cit.> may be added mitigate the possibility of encountering high norm tokens as model size increases. High norm tokens are explored in object-centric settings where background information is redundant and of lesser importance. In contrast, pathology image tiles filtered to the tissue foreground have no such bias, thus the addition of registers is precautionary. 




§ ABLATIONS INVESTIGATING LEARNING IN THE PATHOLOGY DOMAIN


We present an ablation study to evaluate modeling choices for the pathology domain. In order to understand the performance impacts of various augmentations and regularization methods prior to model and data scaling, a sweep is performed at a smaller scale on limited data to assess the viability of each technique. The goal of the ablation is to gain better insights into how different elements of a complex training procedure are coupled, despite the possibility that any performance gains or losses attributed to these changes may not be a factor at scale for long training horizons. The axes of exploration are in terms of geometric and photometric augmentations through the use of crop-and-resize and ECT, as well as the presence of solarization, while regularization is explored through the use of KoLeo and KDE.



 §.§ Training data


The ablation training dataset is comprised of 1.5M H&E stained WSI from 120K MSKCC patients scanned at 5×, 10×, 20×, 40× resolutions across 17 tissue types (the original Virchow training set <cit.>). WSI are subdivided into 224×224 or 392×392 non-overlapping tiles and filtered to include a minimum tissue coverage of 45% as determined by a hue, saturation, and value (HSV) filter with an acceptance criterion in ranges [90, 180], [8, 255], [103, 255], respectively.



 §.§ Pretraining metholodology


A ViT-B/16 is trained using variations of DINOv2 <cit.> on 224×224 global views and 96×96 local views sampled from image tiles. The method uses a batch size of 1024, a stochastic depth drop rate of 0.4, and the AdamW optimizer with a learning rate equal to 2 × 10^-4 on a cosine schedule for approximately 112K iterations, or equivalently 115M tiles sampled from the dataset detailed in section <ref>, using 16 Nvidia V100 GPUs. The student DINO and iBOT temperature is set to 0.1 while the teacher temperature is warmed up from 0.04 to 0.07 over a period of 12K iterations.

The augmentation policy is composed of horizontal and vertical flips, grayscale, and color jitter, while the crop-and-resize is interchanged with ECT. The crop-and-resize operation is applied to 224×224 tiles and follows the default parameters used in DINOv2, which include scale ranges of (0.32, 1.0) and (0.05, 0.32) for global and local views with an aspect ratio range of (0.75, 1.33). ECT is applied to regions of size 392×392 pixels and uses a base scale range of (0.9, 1.1) and aspect ratio range of (0.95, 1.05). The scale range is then adjusted using the relative ratio of the source and target size for both global and local views.

Entropy regularization is performed using KoLeo, a differential method, or KDE, a resubstitution method. Density estimation is performed using the vMF kernel and the concentration is set to 5 for all experiments based on empirical results in the literature.



 §.§ Tile benchmarks

A mix of seven in-distribution (ID) and out-of-distribution (OOD) tile level classification tasks across various magnifications are used to evaluate the quality and separability of the learned features of the teacher network via a linear probe protocol.

PanMSK XX× is an in-distribution cancer detection task performed at 5×, 10×, and 20× magnifications. 1,196,171 224 × 224 pixel tile samples, sourced from 3,999 WSI from MSKCC representing 17 tissue types, are labeled either cancer or benign <cit.>.

PCam is a public (PatchCamelyon) dataset of 327,680 lymph node images labeled as cancer or benign <cit.>. Images are upsampled from 96 × 96 at 10× magnification to 224 × 224 for analysis.

MHIST is a public dataset of 3152 colorectal polyp images (5× magnification 224 × 224 pixels) labeled benign or precursor <cit.>, used as a “minimalist histopathology image analysis dataset” (MHIST). Since a validation data split is not provided, we randomly split 10% of the training set into a validation set.

CRC is a public dataset with colorectal cancer (CRC), composed of 100,000 colorectal images (20× magnification 224 × 224 pixels) classified into 9 morphologies <cit.>.

MIDOG is a binary mitotic cell detection task that we created from the public MItosis DOmain Generalization (MIDOG++) dataset, containing 21,806 mitotic and non-mitotic events labeled on 503 7K × 5K pixel WSI regions from several tumor, species, and scanner types at 40× magnification <cit.>. This data was converted into a binary classification task by expanding each 50 × 50 pixel annotation to 224 × 224 regions and then randomly shifting in the horizontal and vertical regions such that the event is not centered in the tile. All negative instances that overlapped with positive instances were removed from the dataset. 



 §.§ Linear evaluation protocol


Performance is assessed by training a linear classifier on z-score normalized frozen embeddings generated from non-augmented samples. The classifier is trained with a batch size of 4096 using stochastic gradient descent with a cosine learning rate schedule from 1×10^-2 to 0 for a total of 12.5K iterations, to maximize convergence. The classifier checkpoint achieving the lowest validation loss was selected for evaluation on a holdout test set. All benchmark datasets are partitioned to include a validation set if not provided by the public resource. If no validation set exists, a random split of 10% is created from the training set.



 §.§ Ablation results

Tab. <ref> presents the results of the ablation analysis. In a majority of tasks (5/7), using all three domain-inspired changes, results in the best performance as well as the best average performance on in-distribution tasks. Using only ECT and KDE regularization has the best performance of 3/7 tasks and the highest average performance on out-of-distribution tasks, although the gap with all three is only 0.002. Interestingly, we observe that using only ECT leads to a small drop in performance while using only KDE regularization leads to a small improvement in performance on out-of-distribution tasks. Combining the two leads to a pronounced improvement, as noted above. All changes improved in-distribution tasks. 





§ RESULTS


We improve on the training recipe of Virchow with pathology specific tuning and scale up the training first along the data axis (Virchow 2) and then along the model axis (Virchow 2G). We evaluate these models against other foundation models in pathology on tile-level benchmark tasks.




 §.§ Training data


Virchow 2 is trained on a dataset of 3.1M biopsy and resection WSIs processed from 225,401 patients across 5×, 10×, 20×, 40× magnifications, with both hematxylin and eosin (H&E) and immunohistochemical (IHC) staining. This dataset extends the 1.5M WSI dataset that Virchow was trained on and is no longer restricted to 17 tissue types (indeed, it contains nearly 200 recorded tissue types). In addition to samples collected by the Memorial Sloan Kettering Cancer Center (MSKCC) 15% of the WSIs and 57% of the patients are sourced from diverse institutions worldwide that submitted challenging cases to MSKCC for consultation. 



Non-overlapping region tiles of size 392×392 pixels with at least 65% tissue by area are used for model training. Note that Virchow 2 and Virchow 2G extract 224×224 crops from these tiles. Tissue segmentation is performed with a trained fully-convolutional network, with post processing via Otsu thresholding with  thresholds of (0.4, 0.5).

Due to the long-tail nature of the data, balancing is explored in terms of tissue type, diagnosis, stain, and magnification. Metadata is not complete for all samples and the presence of cancer, precursor, and neoplastic tissue does not imply the lack of benign tiles within these large samples. Given these sources of noise, we aim to balance with the best available data. Cancer is sampled at 40% and precursor is sampled at 15%, which closely matches the observed frequency of precursor samples in the dataset (17.9%, where 12.3% contain both cancer and precursor and 5.6% contain only precursor). Benign and benign non-neoplasms are sampled at approximately half their observed frequencies for a total of 8%. Benign neoplasms are sampled at 2%. Samples with unknown diagnosis are sampled at 35% (see Fig. <ref>c for the observed distribution). Tissue type balancing is performed to flatten out the distribution while maintaining the proportion of IHC in an H&E dominant dataset. This approach primarily decreases the prevalence of breast and increases the prevalence of tissues with small sample size and unknown tissue (see Fig. <ref>d). Magnification balancing is performed to increase uniformity in batches during training under the assumption that each resolution may contain unique feature sets that should be contrasted against one another. 40% of WSI do not have 40× magnifications, therefore these slides are sample 1.5× more often than those which do not. Sampling of the magnifications is also balanced to yield a final approximate distribution of 20%, 40%, 20%, and 20% for 40×, 20×, 10×, 5×, respectively.



 §.§ Pretraining methodology


Two models are trained for a full schedule using insights from the ablations detailed in section <ref>. Both models include ECT instead of crop-and-resize augmentations as well as KDE rather than KoLeo for diversity regularization. Global tiles of size 224×224 pixels and local tiles of size 98×98 are extracted from the 392×392 extended context source regions, following the DINOv2 multi-view recipe. Each model is trained on 512 Nvidia V100 GPUs. Further configuration details are found in appendix <ref>.

Virchow 2, a ViT-H/14 with 4 registers, explores the effects of increased data scale and diversity across a mix of magnifications paired with the algorithmic changes proposed. Virchow 2 is trained with solarization on a cosine schedule and is optimized with AdamW for a base learning rate equal to 2×10^-4 with a batch size of 4096 over a total of 2B unbalanced tiles. 

Following the modifications made to the ViT-g architecture to maximize compute efficiency <cit.>, we introduce Virchow 2G, a ViT-G/14 that adopts similar architectural changes. Virchow 2G increases the embedding dimension from 1,664 to 1,792 to be a multiple of 256 and increases the number of attention heads from 16 to 28 in order to have 64 dimensions per head. Additionally, Virchow 2G adopts additional normalization via DPN and QKN and has 8 registers. Virchow 2G is trained without solarization on a cosine schedule and is optimized with StableAdamW for a base learning rate equal to 1×10^-4 and a batch size of 3072 over a total of 2B balanced tiles. For stability, the second moment optimizer hyperparameter was reduced from 0.999 to 0.95. Additionally, both iBOT and DINO head teacher temperatures are reduced from 0.07 to 0.04. Further configuration details are found in appendix <ref>.



 §.§ Embedding generation

For a 224 × 224 input tile image, a Virchow embedding is defined as the concatenation of the class token and the
mean across all 256 of the patch tokens. This produces an embedding size of 2,560 (1,280 × 2) for Virchow, or 3,584 (1,792 × 2) for Virchow 2 / 2G. The class token is used for H-optimus-0 <cit.>, Prov-GigaPath <cit.>, UNI <cit.>, and Phikon <cit.>. For CTransPath <cit.>, the mean of all tokens is used as there is no class token.



 §.§ Tile benchmarks

The benchmarking protocol is the same as described in Sec. <ref>. In addition to the aforementioned tasks, we add an additional four out-of-distribution tasks.

CRC No-Norm is a variation of the CRC task where the testing set does not use Macenko stain normalization. This simulates a distribution shift between the training or validation sets and the testing set.

WILDS is a public dataset (Camelyon17-WILDS) of 455,954 images, each at a resolution of 96 × 96 pixels at 10× magnification upsampled to 224 × 224 derived from the larger Camelyon17 dataset <cit.>. The images are labeled to indicate the presence or absence of tumor. The validation and testing data are from institutions not represented in the training data.

DLBCL is a binary 5-year survival prediction task from the public DLBCL-Morph dataset containing 42 tissue microarrays from 209 Diffuse Large B-Cell Lymphoma (DLBCL) cases/patients, with H&E staining and five IHC stains (CD10, BCL6, MUM1, BCL2, and MYC.) at 40× magnification <cit.>. Due to damaged or missing cores, some stains are missing for some patients. Training, validation, and testing sets are created on the patient level.

TCGA-TILS is a public dataset of 304,097 images, each at a resolution of 100 × 100 pixels at 20× magnification. The images are labeled to indicate the presence of tumor-infiltrating lymphocytes (TILs) if at least two TILs are present <cit.>.

HEST-Benchmark is a public dataset of 74 spatial transcriptomics profiles from 48 patients grouped into 10 tasks based on organ <cit.>. The aim of the task is to predict the expression levels of the 50 most variable genes from 112 × 112 μm H&E stained image patches, each centered on a spatial transcriptomics spot. Here we present the results of a random forest classifier following the protocol described in Jaume et al. <cit.>, as opposed to the linear protocol described in Sec. <ref>.



 §.§ Scaling results

Virchow 2 and 2G, along with Virchow and other baseline foundation models, were evaluated on in-distribution (ID) tile-level benchmarks (Tab. <ref>) and out-of-distribution (OOD) tile-level benchmarks (Tab. <ref>). The pathology-specific modifications to the training recipe of Virchow 2, along with increased training data scale and diversity, yielded a substantial boost in ID performance, raising the average weighted F1 score from 0.944 (Virchow) to 0.966 (Virchow 2). Gains in ID performance from further scaling the model size with Virchow 2G were more modest, further raising the score to 0.971. OOD average weighted F1 score increased from 0.877 with Virchow to 0.885 with Virchow 2 and 0.894 with Virchow 2G, demonstrating the benefits of model and data scale, as well as algorithmic improvements. Indeed, Fig. <ref> demonstrates a log-linear trend in average OOD performance with model and data scale, across all models.

Although H-optimus-0 and Prov-GigaPath both use a ViT-g with 1.1B parameters, Virchow with ViT-H and 632M parameters matches or exceeds their performance on average, being surpassed only on PanMSK 20×, MHIST, PCam, and WILDS by H-optimus-0 and on PanMSK 5×, PCam, and WILDS by Prov-GigaPath. Virchow 2, also with 632M parameters but with a larger training set and improved training recipe, outperforms H-optimus-0 and Prov-Gigapath on all ID (PanMSK) tasks and MHIST, and bridges the gap on WILDS. Scaling up the model size to ViT-G (1.8B parameters) with Virchow 2G, results in top performance on all benchmark tasks. We note that while Virchow 2G achieves a weighted F1 score of 0.948 on TILS, as compared to the top score of 0.949 (Virchow, H-optimus-0), these scores are similar and performance on this task appears to have saturated. A major source of improvement lies in the addition of mixed magnifications. Virchow 2 and Virchow 2G both significantly improve on ID PanMSK tasks at 10× and 5× magnification. High resolution 40× OOD tasks such as MIDOG and DLBCL further illustrate the benefits of training on magnifications other than 20×.

While we use the concatenation of the class [CLS] token with the mean of the patch tokens for all Virchow models, experiments are also performed using only the class token, with models denoted as “Virchow [CLS]”, “Virchow 2 [CLS]”, and “Virchow 2G [CLS]”. While Virchow 2 [CLS] performs similarly to Virchow 2, Virchow and Virchow 2G benefit from the concatenated embedding. Using Virchow 2 [CLS] may be favourable to using Virchow 2 in practice due to the reduced storage cost for smaller embeddings; however, further task-specific validation is necessary to confirm that performance is retained in this configuration.

Tab. <ref> presents the Pearson correlation coefficient results for the HEST-benchmark. Here we observe less consistency in the performance of the models, possibly because the training protocol is fixed and the hyperparameters are not tuned. Virchow 2G has the best performance on 1/10 tasks, second best on 5/10 tasks, and the best average performance. H-optimus-0 has the best performance on 3/10 tasks, second best on 1/10 tasks, and the second best average performance. 












§ DISCUSSION


Our results suggest that foundation model performance in computational pathology continues to benefit from model and data scale. Nevertheless, substantial gains can still be achieved by methodological improvements even at small scales. We therefore built Virchow 2 and Virchow 2G, which scale the dataset size and then the model size, respectively, on top of modifications of the DINOv2 training setup, tailored to the pathology domain.

Our ablation results demonstrate that such modifications can significantly improve benchmark performance. Changing the KoLeo regularizer to a kernel density estimator (KDE) also boosted performance. This is likely because the conditions that produce instability with KoLeo are particularly common in pathology images: that is, tiles within a minibatch are much more likely to be similar to each other compared to ImageNet <cit.> images within a minibatch. Furthermore, the proposed extended-context translation (ECT) is more fitting for pathology data than crop-and-resize because it avoids distorting the cell morphology which is crucial for interpreting pathology images. Although decreasing random resizing during training may reduce the model's scale invariance, we can retain it for pathology by training across multiple magnifications. Similarly, removing solarization boosted performance (especially on PCam), as also previously observed <cit.>. Indeed, it has been shown that the effects of augmentation are often data dependent and vary across classes <cit.>, however, with a sufficiently large and diverse training dataset, the importance of photometric data augmentations diminishes <cit.>. Nevertheless, suitable image transforms, like cropping, are still necessary between the two views of an input in joint embedding models like DINOv2.

Further adjustments to the training recipe were required to allow scaling Virchow both along the data size and the model size axes. Without KDE, DINOv2 training collapsed before enough of the 3.1M slide dataset could be processed. Additional stabilizations were required to then scale up the model from ViT-H (632M parameters) to ViT-G (1.8B parameters). We note that largest models previously trained with DINOv2 had 1.1B parameters (ViT-g) <cit.>. The types of challenges encountered are classified into numerical issues and collapse events. Numerical issues are determined when a training run fails due to NaNs that could not be caught and mitigated and may be due to the use of FP16 rather than BF16. These are addressed by DPN, QKN, and gradient and update clipping in StableAdamW. Collapse events are categorized by sudden spikes in a loss component when features collapse to a lower dimensional subspace that is correlated with the degradation in downstream performance. Collapse primarily occurs in the iBOT head and is mitigated by reducing the learning rate, reducing the β2 momentum parameter in StableAdamW from 0.999 to 0.95, and reducing the teacher temperature from 0.07 to 0.04 based on the guidance in DINO <cit.>. No collapse was observed in the DINO head. It is possible that diversity regularization provided sufficient support to avoid catastrophic events where asymmetric centering and sharpening was insufficient. These modifications reduce the convergence rate but allow training longer. 

Based on the scaling plots (Fig. <ref>) and performance of Virchow 2 and Virchow 2G, benchmark performance continues to improve with model and data scale; however, additional benchmarks on which performance has not yet saturated may be necessary to gauge further gains (like MIDOG, DLBCL, or PanMSK, all of which contain magnifications other than 20×). No substantial improvements with model scale were observed on WILDS, CRC, CRC-No-Norm, or TILS beyond ViT-H (632M parameters). Indeed, F1 scores for these tasks, as well as PCam and PanMSK are between 0.95 and 0.99 (where 1.00 is the maximum), suggesting that further performance gains may be insignificant or unmeasurable due to label noise. MHIST results are difficult to interpret due to their high variance. Although PanMSK <cit.> covers 17 tissue types (all the tissue types that Virchow was trained on), it does not evaluate the long tail of tissue types learned by Virchow 2 and Virchow 2G and it could be made more challenging by enriching it for rare cancers. Indeed, in order to quantify the benefits and impact of model and data scale, it is imperative to prepare diverse benchmarks across different tasks, as some benchmarks may saturate more quickly than others. Without a reliable suite of downstream tasks, confident claims cannot be made about the quality of a model.

Ultimately, there is still room to improve benchmark performance along the axes of model scale, data scale, and algorithmic improvements. We demonstrate the value of pathology-specific improvements, as well as training on mixed magnifications, allowing improved performance on diverse benchmark tasks towards a unified image-based foundation model.



§ ACKNOWLEDGEMENTS

We gratefully thank Philip Rosenfield from Microsoft and Djamilia Dierov from Paige for their contributions in making this collaboration possible, Philippe Mathieu for distributed inference support, Mark Fleishman for data support, and Wayne Hendricks and Alexander van Eck at Paige and Jim Jernigan, Lifeng Li, Ben Huntley, Wyman Chong, Oleg Losinets, and the rest of the team at MSR for infrastructure support.

plainnat
 





§ APPENDIX



 §.§ Foundation models in CPath








 §.§ Hyperparameters









 Paper Title: SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition.



 



SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition
    
	Hao Tan, 
	Zichang Tan, Member, IEEE, 
        Jun Li,
        Jun Wan, Senior Member, IEEE, 

        Zhen Lei, Fellow, IEEE,
        Stan Z. Li, Fellow, IEEE
Hao Tan, Jun Li, Jun Wan and Zhen Lei are with the State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation Chinese Academy of Sciences (CASIA), Beijing 100190, China, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China.
Jun Wan is also with the School of Computer Science and Engineering, Macau University of Science and Technology, Macau, China.
Zhen Lei is also with the Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China (e-mail: {tanhao2023, lijun2021, jun.wan, zhen.lei}@ia.ac.cn).
Zichang Tan is with the Sangfor Technologies Inc., and also with the Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences (e-mail: tanzichang@foxmail.com).
Stan Z. Li is with the Westlake University (e-mail: Stan.ZQ.Li@westlake.edu.cn).

    August 5, 2024
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
  Multi-label image recognition is a fundamental task in computer vision. 
  Recently, Vision-Language Models (VLMs) have made notable advancements in this area. 
  However, previous methods fail to effectively leverage the rich knowledge in language models and often incorporate label semantics into visual features unidirectionally. 
  To overcome these problems, we propose a Split-and-Synthesize Prompting with Gated Alignments (SSPA) framework to amplify the potential of VLMs. 
  Specifically, we develop an in-context learning approach to associate the inherent knowledge from LLMs.
  Then we propose a novel Split-and-Synthesize Prompting (SSP) strategy to first model the generic knowledge and downstream label semantics individually and then aggregate them carefully through the quaternion network.
  Moreover, we present Gated Dual-Modal Alignments (GDMA) to bidirectionally interact visual and linguistic modalities while eliminating redundant cross-modal information, enabling more efficient region-level alignments.
  Rather than making the final prediction by a sharp manner in previous works, we propose a soft aggregator to jointly consider results from all image regions.
  With the help of flexible prompting and gated alignments, SSPA is generalizable to specific domains.
  Extensive experiments on nine datasets from three domains (i.e., natural, pedestrian attributes and remote sensing) demonstrate the state-of-the-art performance of SSPA.
  Further analyses verify the effectiveness of SSP and the interpretability of GDMA.
  The code will be made public.



Multi-label image recognition, vision-language models, quaternion network, gate mechanism





§ INTRODUCTION


Multi-label recognition (MLR) <cit.>
is a fundamental task in the field of computer vision,
where multiple labels are supposed to be recognized in a single image.
This ability to capture the diversity of visual content is paramount in applications like image tagging <cit.>, human attribute recognition <cit.>, and recommendation systems <cit.>.


With the rise of large-scale vision-language pre-training (VLP) <cit.>, 
many approaches <cit.> began to leverage linguistic modality to mitigate the lack of semantic information from a single visual input.
Typically, these approaches involve extracting semantic knowledge from the language model and employing it as supplementary information to assist the visual model in learning better label representations.
Due to the extensive semantics embedded in the language model and the well-aligned cross-modal features learned by VLP, 
these methods have made remarkable progress in multi-label recognition. 



However, the application of vision-language modeling in the domain of multi-label image recognition is still in its infancy. 
As summarized in Fig. <ref>, there are mainly two limitations in existing works <cit.>: 
1) Existing methods take pure category names <cit.> or plain templates <cit.> (e.g., “”) as the inputs of the text encoder, 
which lack sufficient knowledge acquisition for these powerful models and limit the fine-grained alignments between label semantics and spatial regions,

and thus often require explicit efforts to model the label relationships.
2) The linguistic modality is merely adopted as a supplement of semantic information for visual features<cit.>, where C additional classifiers (C is the number of candidate labels) are required to learn the category centers, which remain static during inference, hindering the generalization abilities among different domains.
Besides, within the cross-modal interaction, different image patches (including background and negative areas) interact equivalently with label representations <cit.>, introducing redundant information for multiple labels.
Such unshielded interaction significantly undermines the efficiency of cross-modal interaction.
For example, ADDS <cit.> requires stacking six layers of cross-modal interaction to yield a satisfying performance, which is resource-consuming and inefficient.

To tackle the first problem, we propose a Split-and-Synthesize Prompting (SSP) strategy to first capture generic knowledge and downstream semantics individually and then aggregate them carefully to yield unified label representations. 
In the split stage, we first propose a Knowledge-Aware Prompting (KAP), where we query the inherent knowledge in Large Language Models (LLMs) <cit.> through in-context learning <cit.> to generate informative descriptions. 
The descriptions not only contain details that distinguish an object from other categories, but also provide underlying relationship information.
For instance, our generated sentence “” 
describes the label relationship of “”, “” and “”.
After fed into the text encoder, our method embeds the label relationships in the latent space rather than exhausting to model them explicitly.
However, the KAP remains static and is not informative enough for the vastly changing visual context.
Therefore, inspired by recent success in prompt learning <cit.>, we introduce Context-Aware Prompting (CAP), where we adopt learnable prompts to facilitate the learning of domain-relevant semantics.
Moreover, to capture visual-related label semantics, we incorporate a Dynamic Semantic Filtering (DSF) module in CAP to adaptively excavate the visual-related semantic cues.
Compared to KAP, CAP is capable of incorporating input-relevant label semantics, greatly facilitating the adaptation to specific domains. 

In the synthesis stage, 

we develop a simple yet general approach called Quaternion Semantic Modeling (QSM), which leverages quaternion network <cit.> to aggregate generic knowledge and downstream semantics under the guidance of cross-modal orthogonal cues.
Since quaternion networks are proficient at relation modeling and exploration of inter-modal and intra-modal correlations <cit.>, the generic knowledge, downstream semantics and visual cues can be effectively synthesized within the quaternion latent space, generating unified representations for candidate labels.

For the second problem, we argue that simply enriching visual representations is unilateral and incomplete.
Since the label semantics might change as the visual scenes vary <cit.>, we consider to jointly rectify visual representations and adjust category centers.
To this end, we propose a Gated Dual-Modal Alignments (GDMA) module.
Specifically, we integrate visual information into the label embeddings, yielding context-aware label representations, and symmetrically integrate the label semantics into visual features, resulting in semantic-related visual representations. 
Different from common cross-modal interaction, we suggest that for multi-label recognition, some image regions (e.g., negative areas) might introduce redundant information for specific labels.
Directly aligning all image regions with label semantics might lead to noisy and subpar results.
Therefore, we propose a gated cross-modal attention mechanism to overcome the problem, where a gate vector is learned to suppress the redundant information and activate the useful cross-modal signals.
To facilitate the alignments between label semantics and regional features, we propose a soft aggregator to perform fine-grained alignments based on the importance score of each image patch.
In the prediction stage, unlike previous method <cit.> that directly takes global visual feature to determine the existence of all labels, or CDUL <cit.> that employs a hard aggregator, our soft aggregator synthesize the information from different image patches in a smooth manner, which brings significant improvements.

To sum up the work, we propose a Split-and-Synthesize Prompting with Gated Alignments (SSPA) framework for multi-label image recognition. 
Our main contributions are:


  * We propose a novel Split-and-Synthesize Prompting strategy, where generic knowledge and downstream semantics are modeled individually and then aggregated carefully in quaternion space. Leveraging quaternion network to aggregate distinct knowledge has not been explored before.

  * We present a Gated Dual-Modal Alignments module, where we first interact visual and label representations bidirectionally while eliminating redundant cross-modal signals, then we develop a soft aggregator to combine information from image regions for multi-label recognition. 

  * To the best of our knowledge, this is the first attempt to leverage inherent knowledge from LLMs to benefit multi-label image recognition through in-context learning, which further embeds the label relationships in the latent space rather than explicitly modeling them. The proposed prompting approach for LLMs is versatile and easy to transfer to other domains. 

  * With the help of flexible prompting and gated alignments, the proposed method is generalizable to specific domains. 
Extensive experiments on various datasets (including natural, pedestrian attributes and remote sensing) demonstrate the state-of-the-art performance of our method.




§ RELATED WORK




 §.§ General Multi-Label Classification

Multi-label recognition serves as a fundamental task in the field of computer vision.
Most methods focus on the natural image domain, where the candidate labels are often common objects such as “bicycle” and “dining table”.
Early approaches <cit.> consider a single visual modality as input and focus on the modeling of label co-occurrence.
Some of them rely on the Recurrent Neural Network (RNN) <cit.> and graph-based models <cit.>.
For example, Wang et al. <cit.> explore the semantic correlation among labels by cascading the RNN to the feature extractor.
Wang et al. <cit.> utilize the LSTM to capture the dependencies among semantic regions.
Ye et al. <cit.> propose a dynamic graph convolutional network (GCN) to model the content-aware relations for co-occurred categories.
Further works turn to visual attention for implicit relation mining.
For instance, Lanchantin et al. <cit.> utilize a transformer encoder to explore the correlations among visual features and labels.
Zhu et al. <cit.> introduce self-attention to capture spatial relationships and then regularize the predictions.
With the help of attention mechanism, ML-Decoder <cit.> proposes a versatile classification head consisting of cross-attention layers, while Q2L <cit.> shares similar ideas by stacking multiple transformer decoders to query the output logits.
Most recent works further enhance the performance from the perspective of knowledge distillation <cit.> and loss improvements <cit.>.

However, uni-modal methods lack label-related semantic information, which limits their generalization ability.
With the rise of language models such as BERT <cit.>, many approaches <cit.> turn to leveraging linguistic modality to complement the semantic information.
Based on the extracted representations for candidate labels, these methods further focus on the interactions of different modalities.
You et al. <cit.> propose a cross-modality attention module to aggregate visual features and label embeddings, while additional classifiers are learned for predictions.
Similarly, Zhu et al. <cit.> construct a two-stream transformer network to explore the textual-visual interactions and an MLP is trained for recognition.
Wang et al. <cit.> superimpose multiple layers of graph to incrementally inject label semantics to feature learning.
Chen et al. <cit.> and Zhu et al. <cit.> both insert semantic information to visual features unidirectionally through a low-rank bilinear pooling, while <cit.> further consider the scene-conditioned label co-occurrence.
Besides, some methods take advantage of VLMs to tackle open-vocabulary multi-label recognition, which is a more challenging task since the method needs to generalize to unseen targets.
DualCoOp <cit.> and DualCoOp++ <cit.> introduce minimal parameters to CLIP by learning coupled prompts. 
ADDS <cit.> proposes a dual-modal decoder, while the interactions are unshielded and heavy, which requires stacking multiple layers to achieve better open-vocabulary performance.
MKT <cit.> achieves this through a two-stream framework and feature distillation.
TAI <cit.> proposes to learn prompts from text-only data, which could generalize to image recognition.

Besides natural image domain, multi-label recognition is also common in pedestrian and remote sensing images.
Pedestrian attribute recognition (PAR) aims to detect attributes (e.g., age and clothing) for images of pedestrians.
The prevalent approach is to explore attribute relationships.
For example, Tan et al. <cit.> propose to learn attribute and contextual relations, respectively.
Recent methods also begin to utilize VLMs.
For example, Wang et al. <cit.> designs a region-aware prompting approach to transfer CLIP model, while the partition of regions is a strong prior for the pedestrian attributes domain and is not versatile to other tasks.
Multi-label remote sensing image classification is to recognize the aerial scenes (e.g., “parking lot“ and “forest”) in an aerial image.
GeRSP <cit.> utilizes extra knowledge to boost the recognition in remote sensing domain.
RemoteCLIP <cit.> proposes a foundation vision-language model that shows exceptional generalization abilities for various remote sensing applications.

Different from previous multi-modal approaches, we explore the mutual interactions between visual and linguistic modalities, and further map the context-aware label representations into dynamic category centers, which greatly enhances the generalization.
Compared to the methods designed for specific MLR, our method is versatile to different domains.
A main challenge of MLR in specific domains is the semantic gap between target labels (e.g., human attributes or visual scenes) and natural objects,
while we mitigate this by resorting to LLMs for pertinent semantic extraction and applying gated mechanisms for robust alignments.



 §.§ Vision-Language Models

Large-scale vision-language pre-training (VLP) has emerged as a powerful paradigm for a wide range of visual tasks <cit.>.
With a contrastive-based pre-training approach, vision-language models (VLMs) such as CLIP <cit.> and ALIGN <cit.> learn a joint representation for visual and linguistic modalities, showing an encouraging ability for efficient transfer learning and zero-shot predictions.
More recently, there has been a growing interest in bridging pre-trained LLMs and vision foundation models to build VLM.
BLIP-2 <cit.> achieves this by training an additional Q-former.
While MiniGPT-4 <cit.> and LLaVA <cit.> attain impressive multi-modal abilities by an extra linear projection.
The emergence of VLMs has benefited plenty of downstream tasks, e.g., 
person re-identification <cit.>, action recognition <cit.> and face anti-spoofing <cit.>.
Our method is built on CLIP, constructing an effective way to unleash the potential of VLMs in MLR.




 §.§ Prompt Learning

Transferring the knowledge from the powerful VLMs to downstream tasks can be computationally expensive in the era of foundation models.
However, prompt learning which is initially introduced in the field of Natural Language Processing (NLP) <cit.>, provides an efficient way for adapting VLMs.
CoOp <cit.> introduces a set of learnable prompts for textual input and yields remarkable few-shot performance.
CoCoOp <cit.> instead adopts visual features to generate input-adaptive prompts.
The follow-up works <cit.> further improve the generalization abilities.
For example, ProGrad <cit.> achieves this by gradient correction,
and MaPLe <cit.> achieves it by introducing coupled prompts in both visual and text modalities.
Different from previous works that only use fixed prompts <cit.> or learnable prompts <cit.> to adapt VLMs,
we for the first time propose a novel split-and-synthesize prompting strategy to facilitate the extraction of generic and adaptive label semantics concurrently.








 §.§ Tool-Use of Large Language Models

Large Language Models (LLMs) have demonstrated their comprehensive knowledge that can be beneficial in various NLP tasks, while this trend has extended to computer vision research.
Several studies have been proposed to investigate how LLMs can assist in downstream computer vision tasks.
CuPL <cit.> leverages LLM-generated descriptors to enhance zero-shot classification.
Toubal et al. <cit.> use LLM to enable subjective visual classification.
DVDet <cit.> introduces interactive LLM to improve open-vocabulary object detection.
OPT2I <cit.> applies LLM to revise the user prompts, improving the prompt-image consistency in text-to-image generation task.
SLD <cit.> utilizes LLM to post-rectify the generation results from diffusion models, reducing the error-prone generations when addressing complex user prompts.
In this work, we develop an efficient prompt template to leverage LLMs to generate pertinent descriptions for candidate labels, facilitating the generalization on different MLR domains.





§ PROPOSED METHOD





 §.§ Preliminary


Notations. 
For multi-label image recognition,
assume the input image I∈ℝ^H× W × 3 is labeled with C candidate categories, where y∈ℝ^C represents the multi-hot label vector and y_j=1 means the input image contains the j^th label and vice versa. 
For the input image I,
we employ the pre-trained image encoder (e.g., ResNet <cit.> or ViT <cit.>) from CLIP <cit.> to extract visual features, denoted as [x_0, X]=(I), where X=[x_1, ..., x_M] ∈ℝ^M × d and x_0 denotes the class token in ViT or pooled features in ResNet, which stands for the global visual feature.
M indicates the number of patches, and d is the feature dimension.


Quaternion networks.
A quaternion Q is expressed in a hyper-complex space, which extends the real values into a four-dimensional space, known as the quaternion algebra ℍ:

    Q = r1 + xi + yj + zk,   r, x, y, z ∈ℝ,

where r is the real part of Q and (i, j, k) denote the orthogonal imaginary axes with i^2 = j^2 = k^2 = ijk=-1.
Typically, quaternion networks are useful for capturing complex correlations and interdependencies among diverse features and they contain embedded information that can be expressed by a real matrix <cit.> as follows:

    Q^mat = [ [   r - x - y - z;   x   r - z   y;   y   z   r - x;   z - y   x   r ]].


Different from common operations in real-valued space, quaternion networks calculate on quaternions for all parameters, including inputs, outputs, weights and bias.

For the real-valued feature vectors, the corresponding quaternion can also be expressed in matrix form Q = r + ix + jy + kz, where r, x, y and z are real-valued vectors.
We suggest using quaternion networks to establish multi-perspective relationships among diverse features <cit.>.
In this paper, we adopt quaternion networks to effectively aggregate generic knowledge, downstream semantics and visual context.


Cross-Modal Attention.
Cross-modal attention (CMA) takes different sources as input and is good at capturing cross-modal interactions.
Different from standard cross-attention<cit.>, we only introduce linear projection W_E for the query.
Besides reducing complexity, removing linear mappings of key and value preserves the original semantics, while projecting query controls the transmission of cross-modal signals.
Suppose the inputs are denoted as E and Z, the process is formulated as:

    CMA(E, Z) = softmax( (EW^E)Z^𝖳/√(d))Z.





 §.§ Framework Overview

As shown in Fig. <ref>, the framework of SSPA is conceptually simple: the decision is based on both global and regional perspectives.
For global branch, we simply align the label representations (denoted as T^g) with the global image feature x_0, which largely preserves the knowledge of the original CLIP.
For regional alignment branch, since there are richer visual context, more fine-grained label semantics are supposed to improve the cross-modal alignments.
Therefore, we first use SSP to extract knowledge-aware and context-aware label semantics, which are then aggregated in the proposed QSM.
Then, the GDMA is applied to perform gated alignments between visual context and label semantics.
In the following sections, we focus on the details of the regional branch.



 §.§ Split-and-Synthesize Prompting


Knowledge-Aware Prompting (KAP).
For multi-label recognition, the essential factors are to distinguish objects from other similar categories and discover the label relationships.
We aim to leverage inherent knowledge in LLMs to assist the process.
However, naive prompts to LLMs result in descriptions of poor quality, for instance, “”, which is not useful and informative for recognition.
Instead, as shown in Fig. <ref>, we develop an efficient prompt template to encourage the LLM to generate detailed descriptions (e.g., shapes, colors and sizes) with underlying label relationships, such as “”,
which depicts notable features of airplane and relationships with “” and “”.
By shifting the “domain description” and “in-context examples” parts in the prompts, we can activate the corresponding knowledge of LLMs and seamlessly transfer the model to specific domains.
After fed into the pre-trained text encoder, the generic knowledge and label relationships are embedded into the latent space.

Suppose the prompts are denoted as 𝒫, we take the LLM to generate descriptions 𝒟^LLM = LLM(𝒫).
Note that our method is flexible to the choice of LLM, and we use LLaMA3 in our experiments.
To encourage attention to the target object, we further concatenate a sentence to the 𝒟^LLM.
The final description for the j^th label is 𝒟_j=“{𝒟^LLM_j}. ”, which is then fed into text encoder:

    t_j^ka = text-encoder(𝒟_j),   j=1,2,...,C,

where T^ka=[t_1^ka, t_2^ka,..., t_C^ka] ∈ℝ^C× d is the knowledge-aware text embeddings.




Context-Aware Prompting (CAP).
The knowledge from KAP is static and agnostic to the input context.
To better capture the downstream knowledge, we introduce learnable prompts in CAP to facilitate the learning of task-relevant label semantics.
Inspired by CoOp <cit.>, we prepend L prompt tokens to each label.
To reduce complexity, the prompt tokens are shared among all candidate labels.
The resulted input is denoted as S_j=[p^1][p^2]...[p^L][c_j], 
where p^l∈ℝ^d is the l^th learnable token, and c_j is the word embedding of the j^th category name.
Then S is fed into the text encoder to get the learnable text features T^ln.

To further collect context-aware semantics, we propose a Dynamic Semantic Filtering (DSF) module.
Specifically, we compare each text feature in T^ln with visual region features X to determine the importance of visual context.
Then, the features for each label are extracted based on the importance weights.
Therefore, the resulted text features are filtered by the input visual context and are adaptive to varied visual scenes.
In practice, to preserve the CLIP extracted visual context, we only perform linear transformation to the text features, which results in a cross-modal attention layer.
The process of CAP can be summarized as follows:

    t_j^ln   = text-encoder(S_j),   j=1,2,...,C, 
    T^ln   = CMA(T^ln, X) + T^ln, 
    T^ca   = MLP(T^ln) + T^ln,

where T^ln=[t_1^ln, t_2^ln,..., t_C^ln] ∈ℝ^C× d denotes the learnable text embeddings and MLP is a multi-layer perceptron.
T^ca=[t_1^ca, t_2^ca,..., t_C^ca] ∈ℝ^C× d represents the context-aware text embeddings.
For simplicity, we omit all the symbols for layernorm operations.

Synthesis.
Different from the conventional feature fusion methods that typically employ concatenation or summation, 
we propose a Quaternion Semantic Modeling (QSM) module to aggregate the knowledge-aware text embeddings T^ka and context-aware text embeddings T^ca.
Furthermore, we take global visual feature x_0 to guide the synthesis of generic knowledge and downstream semantics.
The simple combination of {T^ka, T^ca, x_0} yields a rudimentary multi-perspective feature.
Then we take a linear transformation W^Q to encode it into quaternion latent space:

    F^mp = (T^ca + T^ka + x_0)W^Q,  F^mp∈ℝ^C× d.

Then, we map F^mp into the quaternion by slicing up its real-valued features into four equidimensional vectors:

    Q^mp = F^mp_s1 + iF^mp_s2 + jF^mp_s3 + kF^mp_s4,
       where  F^mp_si∈ℝ^C×d/4,   i=1,2,3,4.

Such mapping does not introduce extra trainable layers, making our method simple and universal.
Suppose the quaternion weight matrix is W = W^R + iW^I + jW^J + kW^K, where W^R, W^I, W^J and W^K are real-valued trainable matrix.
Performing the linear operation between W and Q^mp by using distributive property we get:


    [ℛ(W⊗Q^mp) 
    ℐ(W⊗Q^mp) 
    𝒥(W⊗Q^mp) 
    𝒦(W⊗Q^mp) 
    ]
        =[W^R   -W^I   -W^J   -W^K 
    W^I   W^R   -W^K   W^J 
    W^J   W^K   W^R   -W^I 
    W^K   -W^J   W^I   W^R 
    ][F^mp_s1
    F^mp_s2
    F^mp_s3
    F^mp_s4
    ],


where ⊗ is known as the Hamilton product <cit.>. ℛ(·), ℐ(·), 𝒥(·) and 𝒦(·) are to obtain the features on the real or imaginary components. 
The quaternion linear operation is similar to a mixture of standard linear and channel-wise separable linear transformations.
Suppose the quaternion linear layer is denoted as Q^ψ:

    Q^ψ(h) = W⊗h,

where h is the input vector.
The quaternion layers used in this work are formulated as follows:

    F^mp   =ϕ(Q^ψ_1(F^mp)),
    T^uf   =ϕ(Q^ψ_2(F^mp)),

where ϕ is the ReLU activation.
By stacking two quaternion linear layers Q^ψ_1 and Q^ψ_2,
the interdependencies between generic knowledge and downstream semantics, and cross-modal signals are well modeled in the quaternion latent space, which generates holistic and unified label representations T^uf.




 §.§ Gated Dual-Modal Alignments

While previous methods perform unidirectional interaction between visual and linguistic modalities, we consider this as a mutually promoting process. 
However, the dense interaction across modalities would contain noisy and redundant signals, which are misleading for the recognition of specific categories.
Therefore, we introduce a gate mechanism into our cross-modal alignments, which performs bidirectional interactions while eliminating redundant signals.

Gate mechanism.
Suppose the cross-modal inputs are denoted as P and U, respectively. Inspired by <cit.>, the gate operation is employed as follows:

    f(P, U)   =tanh([P, U, P-U, P⊙U]W^f + b^f), 
    
        v(P, U)   =σ([P, U, P-U, P⊙U]W^g + b^g), 
    
        g(P, U)   =v(P, U)⊙ f(P, U) + [1-v(P, U)]⊙U,

where {W^f, W^g, b^f, b^g} are trainable parameters, ⊙ denotes the Hadamard product, and σ is sigmoid function. f(P, U) denotes the modulated input, and v(P, U) is a learned gate vector with each element controlling the influence of a corresponding pair of interactions.
g(P, U) denotes gated output.




Gated dual-modal alignments.
GDMA consists of two symmetric attention modules, i.e., gated visual-to-semantic attention and gated semantic-to-visual attention.
As shown in Fig. <ref>, the interactions are first fully explored by cross-modal attention, and then gated by Eq. <ref>.
For gated visual-to-semantic attention, the process can be summarized as:

    T^vs   = CMA(T^uf, X), 
    T^g   = g(T^vs, T^uf), 
    T^fn   = MLP(T^g) + T^g,

where T^fn∈ℝ^C× d denotes the final label representations.
Similarly, for gated semantic-to-visual attention, the process is summarized as follows:

    X^sv   = CMA(X, T^uf), 
    X^g   = g(X^sv, X), 
    X^fn   = MLP(X^g) + X^g,

where X^fn∈ℝ^M× d is the final visual representations.
Since T^fn encapsulates both generic semantic knowledge and downstream visual context, it can seamlessly serve as the category centers for candidate labels, without additional learning of classifiers.
During inference, our method can simultaneously rectify the visual representations and adjust the category centers according to the specific inputs, yielding much better generalization performance.
   
Soft aggregator. To jointly consider information from different image regions and facilitate fine-grained alignments, we propose a soft aggregator. 
Specifically, the presence probability p_ij^R of the j^th label in the i^th local patch is computed as:

    p_ij^R = X_i^fn (T_j^fn)^𝖳,   j=1,2,...,C.

While background and negative areas might have low probabilities, we take the softmax to determine the importance of the i^th local patch for the j^th label:

    γ_ij = exp(p^R_ij/τ)/∑_k=1^M exp(p^R_kj/τ),   i=1,2,...,M,

where τ is a learnable temperature term.
The aggregated score of the j^th label is calculated based on the weighted average of regional predictions, which is written as follows: 

    p^R_j = σ(∑_i=1^Mγ_ij p^R_ij),   j=1,2,...,C.




 §.§ Training Objective

In this work, we employ the Asymmetric Loss <cit.> for multi-label classification.
For the regional branch, the loss is calculated based on p^R_j, which is formulated as:

    .ℒ^R=1/C∑_j=1^C{(   1-p^R_j)^γ^+logp^R_j,  y_j=1,
     
        (   p^R_j)^γ^-log(1-p^R_j),  y_j=0,
    ..

where γ^+ and γ^- are asymmetric focusing parameters for positive and negative samples, respectively.

For the global branch, the predictions are generated by comparing the similarities between global visual feature x_0 and text features T^g, which is denoted as p^G = σ(x_0 (T^g)^𝖳).
The global loss ℒ^G is calculated by asymmetric loss similar to Eq. <ref>.
The final predicted score is the average from global and regional perspectives:

    p_j = p^G_j + p^R_j/2,   j=1,2,...,C.

The final training objective is formulated as:

    ℒ=ℒ^G + λℒ^R,

where λ is to control the importance of regional decisions.




§ EXPERIMENTS




 §.§ Datasets and Metrics

Datasets.
We conduct experiments on three popular benchmarks in MLR, i.e., MS-COCO <cit.>, VOC 2007 <cit.> and NUS-WIDE <cit.>.
MS-COCO <cit.> contains 80 common categories and we use official train2014 (82K images) and val2014 (40K images) splits for training and testing.
VOC 2007 <cit.> contains 20 object categories and we use the official trainval (5K images) and test (5K images) splits for training and testing.
NUS-WIDE <cit.> is more noisy, which contains 81 common categories and we use the train (12K images) and test (8K images) splits for training and testing.

Our method can be applied to other downstream domains, and we also evaluate on additional datasets from pedestrian attribute recognition (including PA100K <cit.>, RAPv1 <cit.> and PETA <cit.>) and remote sensing image classification (including MultiScene <cit.>, MLRSNet <cit.> and AID <cit.>).
PA100K <cit.> contains 26 pedestrian attributes and we take the official train (80K images) and test (10K images) splits for training and test.
RAPv1 <cit.> contains 51 pedestrian attributes and we take the official train (30K images) and test (8K images) splits for training and test.
PETA <cit.> contains 35 pedestrian attributes and we take the official train (9K images) and test (7K images) splits for training and test.
MultiScene <cit.> contains 36 aerial scenes and we take the “clean” set which contains 7K images for training and 7K images for testing.
MLRSNet <cit.> is a large-scale remote sensing dataset which contains 60 aerial scenes and we take the official train (80K images) and test (20K images) splits for training and test.
AID <cit.> contains 17 aerial scenes and we take the official train (2K images) and test (0.6K images) splits for training and test.


Evaluation Metrics.
For natural and remote sensing datasets, the mean average precision (mAP) is reported to evaluate the overall performance.
Following <cit.>, we also report Class-wise Precision (CP), Recall (CR), F1 (CF1), and the average Overall Precision (OP), Recall (OR), F1 (OF1).
Note that “CF1” and “OF1” are more informative since Precision and Recall vary with the threshold.
To fairly compare with state-of-the-art, we further report top-3 results.
For pedestrian attribute recognition datasets, following previous work <cit.>, we adopt mean Accuracy (mA) as label-based metrics, Accuracy, Precision, Recall and F1-score as instance-based metrics.








 §.§ Implementation Details

Our method is built on CLIP <cit.>.
The text encoder remains frozen.
The number of learnable prompt tokens L is set to 4.
γ^+ and γ^- are set as 0 and 2, respectively.
λ is set as 1.
For fair comparisons, the input images are resized to 448× 448 for natural image datasets, and 224× 224 for pedestrian attributes and remote sensing datasets.
The network is trained for 30 epochs using AdamW <cit.> optimizer with a batch size of 32.
The learning rate is set as 0.0001 and decays with cosine policy.
Following previous works <cit.>, we apply exponential moving average with a decay of 0.9997.
We also implement CLIP-FT as a strong baseline, which fine-tunes all parameters including both image and text encoders.




 §.§ Comparison with State-of-the-art

Natural image MLR.
The comparisons on MS-COCO, PASCAL VOC 2007, and NUS-WIDE are shown in Table <ref>, Table <ref> and Table <ref>, respectively.
SSPA achieves state-of-the-art performance across various backbones and resolutions on all datasets, surpassing other methods with a decent margin.
On the MS-COCO, compared with multi-modal method SALGL <cit.>, our method exhibits considerable performance gains, exceeding them by 2.9% mAP, which suggests the superiority of fully exploiting linguistic modality.
Compared with ML-GCN <cit.> that also maps label representations into category centers while neglecting the visual context, SSPA achieves 5.7% gains in mAP, demonstrating the effectiveness of learning input-adaptive category centers.
Moreover, our method outperforms all other methods on the resolution of 576× 576 and ViT-B/16 backbone, surpassing previous SOTA by 2.0% and 1.8% mAP respectively.
On the NUS-WIDE, our method surpasses all other methods on ResNet101 and ViT-B/16 backbones, achieving 67.7% and 69.9% mAP, respectively, which demonstrates the robustness of SSPA when addressing the noisy real-world images.
On the PASCAL VOC 2007, SSPA also outperforms all other methods.
With ViT-B/16 backbone, the AP on all 20 categories exceeds 90.6%, achieving 97.5% mAP,
which demonstrates the effectiveness of our method in handling objects of distinct sizes and semantics.
The experimental results clearly confirm the superiority of our proposed SSPA, 
and also show good generalizability to different network architectures.

Pedestrian attribute recognition.
As shown in Table <ref>, SSPA demonstrates superior performance compared to those specialized PAR approaches.
As for F1, our method achieves the best results on all datasets and all backbone settings, e.g., surpassing SOTA method by 0.9%/2.2% on RAPv1 using ResNet50/ViT-B.
As for mA, our method also achieves considerable improvements, e.g., outperforming SOTA method by 1.1% on PA100K using ViT-B.
It is worth noting that compared to CLIP-FT that fine-tunes all parameters (including text and image encoders) and PromptPAR <cit.> that has further specialized designing, our method exhibits notable advantages on RAPv1 and PETA, which demonstrates the effectiveness and versatility of our usage of CLIP.


Remote sensing image MLR.
We fine-tune the most recent method <cit.> in the field of remote sensing recognition.
RemoteCLIP <cit.> pre-trains a CLIP-style model specialized in remote sensing tasks, which we further fine-tune on the MLR datasets.
As presented in Table <ref>, 
SSPA achieves superior performance on all datasets, surpassing RemoteCLIP-FT by 1.2%/1.3% mAP on MultiScene using ResNet50/ViT-B.










 §.§ Ablation Studies

Effect of global-regional framework and soft aggregator.
As shown in Fig. <ref>, incorporating regional cues largely enhances the performance while aggregating global and regional decisions brings further improvements (3.3%/3.4% mAP on COCO/NUS).
For regional branch, our proposed soft aggregator outperforms the simple averaging approach.
The performance gap of hard aggregator <cit.> is due to that rigid rules are error-prone, and a softer version mitigates the problem. 


Effect of proposed modules.
Fig. <ref> shows the effectiveness of the proposed SSP and GDMA.
We employ CLIP with the fixed template “A photo of a {category}” as baseline.
The limited performance indicates that directly transferring CLIP is not effective for MLR.
In contrast, SSP enhances mAP by a significant margin (5.0%/7.0% mAP on COCO/NUS), showing the effectiveness of our prompting pipeline. 
While GDMA further improves the performance, which demonstrates that both SSP and GDMA play pivotal roles in SSPA.


Ablation on the split stage of SSP.
As shown in Table <ref>, the performance degrades without KAP or CAP, indicating the importance of split prompting.
Removing LLM descriptions also results in a performance drop, which indicates that our method benefits from the inherent knowledge within LLMs.
Besides, DSF brings further promotion, which is due to that incorporating visual context facilitates the learning of downstream semantics.
Notably, on PETA and MultiScene, CAP (+1.9%/+1.7%) is more effective than KAP (+1.0%/+0.9%), which indicates that learning of downstream semantics is more important when transferring to those domains of a large gap.

















Ablation on the synthesis stage of SSP.
As shown in Table <ref>, 
direct summation or concatenation yields degraded performance, implying that the generic knowledge and downstream semantics require more fine-grained synthesis.
Compared to MLP synthesis, which shares similar number of parameters and computation overhead with our proposed QSM, the QSM achieves better performance, revealing the superior capabilities of quaternion latent space to capture comprehensive relationships and inter-modal dependencies.

Ablation on GDMA.
In Table <ref>, we show that employing visual-to-semantic attention (V-to-S) or semantic-to-visual attention (S-to-V) individually leads to degraded performance, while the latter yields worse results.
Overall, using both concurrently (i.e., GDMA) brings significant improvements. 
This confirms the superiority of bidirectional interactions and further indicates that incorporating visual context into label representations is of more importance. 
One reason is that V-to-S attention enables input-adaptive category centers during inference.
As for the gate mechanism in GDMA, 
Table <ref> further shows that the proposed gated alignment brings notable improvements compared to direct interaction.










 §.§ Further Analyses


Visualization of attention maps.
As shown in Fig. <ref>, SSPA can accurately perceive 
common objects of distinct size (e.g., “backpack” and “car”)and similar appearance (e.g., “chair” and “couch”) in natural images.
For pedestrian images, SSPA can focus on specific body parts, e.g., emphasizing the arms area when detecting “short sleeves”, and concentrating on the legs when determining the lower body clothing.
For remote sensing images, SSPA can perceive large land covers (e.g., “trees” and “water”) and fine-grained objects (e.g., “cars” and “airplane”).
These observations demonstrate the excellent discriminative capability of the proposed SSPA.


Interpretability of gate vectors.
To validate the effectiveness of the proposed gated alignments, we visualize the gate vectors in Fig. <ref>.
For remote sensing images, the gate vectors activate most areas since the land covers often dominate the whole image.
For natural and pedestrian images, the gate mechanism successfully suppresses most background information, which can enhance the efficiency of cross-modal interactions, yielding more pertinent visual and label representations.


Is using label representations as category centers better?
In Table <ref>, “Uni-D” denotes a baseline similar to previous approaches, where C additional classifiers are learned.
Our method improves the mAP by 4.4% and 3.6% on COCO and NUS, respectively, suggesting the superiority of constructing dynamic category centers through label representations.
Similar conclusions can be drawn on BERT <cit.>, showing the robustness of SSPA to the text encoder.











Data bias of LLM.
Different pre-training corpora may lead to biased output from LLM.
In Table <ref>, we reveal that appropriate prompts are sufficient for LLMs to generate accurate information with less noise, regardless of the pertaining corpora (or data bias).
When replacing LLaMA3-8B with Vicuna or LLaMA (using the same prompt template), similar performance is achieved, exhibiting stableness and robustness.

ImageNet pretraining.
As shown in Table <ref>, with ImageNet pre-trained backbone, our method still outperforms previous methods (e.g., ML-GCN and Q2L) significantly, which shows the alignment abilities of SSPA.
Besides, we suggest using pre-trained VLMs (e.g., CLIP) for better performance.

Computation overhead.
During deployment, the text features from the text encoder can be extracted in advance and 
preserved offline on testing devices, and the corresponding computation could be ignored.
As shown in Table <ref>,
compared to the baseline, our method achieves considerable performance gains with limited computational increase.
Moreover, SSPA achieves much better performance with comparable FLOPs and parameters than previous SOTAs, e.g., Q2L. 
In fact, the parameters and FLOPs of our QSM are similar to that of MLP, and we only require one QSM and one GDMA layer, which only brings a few extra computations. 










§ CONCLUSION


In this work, we proposed SSPA, a novel framework for multi-label image recognition.
We developed an SSP strategy to aggregate generic knowledge from LLMs and label semantics from target tasks, which makes great use of linguistic knowledge to enhance the recognition of multiple labels.
We further presented GDMA to bidirectionally interact text embeddings and visual features while filtering out redundant signals, largely enhancing the efficiency of cross-modal alignments.
Extensive experiments on different domains show the state-of-the-art performance and generalization abilities of SSPA.

Limitations and broader impacts.
The major limitation is that there are unannotated objects in our training images, which could impact the performance in real-world applications. 
One of our future works will focus on overcoming this problem by expanding the method to open-vocabulary scenarios.
As for the social impacts, since this work aims to develop a general method without targeting specific applications, which does not directly involve societal issues.
However, when applied to the pedestrian domain, it can also have social and cultural implications by influencing perceptions of privacy, surveillance, and individual autonomy. 
The ethical and legal implications on the collection, storage, and use of pedestrian attribute data should be carefully considered.

IEEEtran





 Paper Title: Conformal Trajectory Prediction with Multi-View Data Integration in Cooperative Driving.


Beyond full statevector simulation with 
    Andrea Pasquale^1,2, Andrea Papaluca^3, Renato M. S. Farias^1,4, Matteo Robbiati^2,5, Edoardo Pedicillo^1,2,
and Stefano Carrazza^1,2,5
    August 5, 2024
===========================================================================================================================================



§ ABSTRACT 
Current research on trajectory prediction primarily relies on data collected by onboard sensors of an ego vehicle. With the rapid advancement in connected technologies, such as vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication, valuable information from alternate views becomes accessible via wireless networks. The integration of information from alternative views has the potential to overcome the inherent limitations associated with a single viewpoint, such as occlusions and limited field of view. In this work, we introduce V2INet, a novel trajectory prediction framework designed to model multi-view data by extending existing single-view models. Unlike previous approaches where the multi-view data is manually fused or formulated as a separate training stage, our model supports end-to-end training, enhancing both flexibility and performance. Moreover, the predicted multimodal trajectories are calibrated by a post-hoc conformal prediction module to get valid and efficient confidence regions. We evaluated the entire framework using the real-world V2I dataset V2X-Seq. Our results demonstrate superior performance in terms of Final Displacement Error (FDE) and Miss Rate (MR) using a single GPU. The code is publicly available at: <https://github.com/xichennn/V2I_trajectory_prediction>.









§ INTRODUCTION



Trajectory prediction plays a critical role in autonomous driving. Typically, we rely on the on-board sensors of an ego vehicle to gather environmental information necessary for performing various autonomous driving tasks. However, with the rapid advancement in connected technologies, such as vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication, valuable information from alternate views becomes accessible via wireless networks. The integration of information from alternative views has the potential to overcome the inherent limitations associated with a single viewpoint, such as occlusions and limited field of view. We categorize the data obtained solely from ego vehicle on-board sensors or infrastructure as single-view information, whereas the accessibility to multiple viewpoints is referred to as multi-view data. Figure <ref> depicts an intersection scenario where the AV faces a potential left-turn collision due to its field of view being obstructed by a large truck. Roadside cameras, strategically positioned to have an unobstructed view of the entire intersection, provide a comprehensive overview of the traffic situation. This enhanced perspective allows the AV to receive critical information and effectively avoid the impending collision.

From a trajectory predictive modeling perspective, significant research efforts have been dedicated to leveraging single-view datasets collected by the ego vehicle <cit.>. These efforts typically involve modeling the temporal dependencies, agent-agent interactions and agent-lane relations. However, the challenge arises when dealing with multi-view data, particularly in how to effectively fuse the information due to overlapping field of views.

Existing work on multi-view data fusion in the context of cooperative driving predominantly focuses on collaborative perception tasks, with limited research addressing trajectory prediction. <cit.> pioneered the creation of the first V2I real-world dataset for trajectory prediction studies. They manually fused multi-view data using trajectory association and stitching techniques at each frame, followed by the application of single-view trajectory prediction models. Although this approach is intuitive, it fails to fully exploit the motion behavior captured by data from each view, resulting in suboptimal outcomes. To address this limitation, <cit.> proposed a novel approach that encodes trajectory information from each view as independent graph nodes, thereby minimizing information loss. They formulated the node association problem as a graph link prediction task and introduced a cross-attention module to fuse node embeddings from associated nodes across different views. While their model can be trained end-to-end, the node association process necessitates pretraining, leading to the formulation of two optimization objectives.

We introduce a trajectory prediction framework that utilizes multi-view data without the need for explicit association between different perspectives. Rather than developing a specialized multi-view model, our approach seamlessly integrates with state-of-the-art single-view trajectory models, maximizing the utility of existing research efforts. No special training strategies are required. We can easily leverage the pretrained single-view trajectory models to expediate the training. Specifically, for trajectory data from each single-view, we employ established graph neural network (GNN) based models, such as LaneGCN <cit.>, HiVT <cit.>, to capture temporal dependencies, agent-agent interactions, and agent-lane relations. Then we utilize a cross-graph attention module to fuse the node embeddings from different views. The fused final embeddings will then go through a multimodal decoder to get future trajectory predictions. 

Existing works have modeled the multi-modality explicitly by introducing anchors <cit.>, mixture models <cit.>, or implicitly through latent variables such as Conditional Variational Auto-encoder (CVAE) or generative models <cit.>. The implicit models often face the issue of mode collapse, therefore we will model the multi-modality by a mixture model built upon MLP. It is a common strategy to assign a higher score to the modality closer to the ground truth during training. This strategy, however, may encounter robustness issues during inference. Figure <ref> presents two examples where the top-scored prediction is not the closest to the ground truth trajectory.



 This arises from the inherent randomness in the ground truth data, posing challenges for the model to learn the priorities of multiple predictions. Relying on the scores to rank the predictions can lead to hazardous results in safety-critical scenarios. Additionally, while most mixture models like GMM provide both point estimates and variance estimation, the reliability of these uncertainty quantification (UQ) remains uncertain. Unfortunately, this aspect is often overlooked in the model evaluation process. However, UQ can offer valuable insights for downstream decision-making processes. To address these challenges, we introduce conformal prediction (CP) <cit.>, as a post-hoc module to our framework. CP creates statistically rigorous uncertainty intervals for the predictions that are guaranteed to contain the ground truth with a user-specified probability. 

To summarize, our contributions in this work lie in twofolds:
1) We introduce V2INet, a novel trajectory prediction framework designed to model multi-view data by extending existing single-view models. Unlike previous work, our model supports end-to-end training, enhancing both flexibility and performance.
2) We introduce conformal prediction as a post-hoc module to calibrate the prediction results. This results in statistically rigorous uncertainty intervals, significantly enhancing the reliability of the predictions. These calibrated predictions are particularly beneficial for downstream decision-making tasks, such as motion planning.



§ RELATED WORK



 §.§ Cooperative Driving Dataset

Collaborative perception has been the most extensively studied area in collaborative driving <cit.>. It leverages the wireless communication technologies, such as V2V and V2I, to share information, which enables perception beyond line-of-sight and field-of-view hence overcoming common perceptual shortcomings with individual perception, such as blind spot, occlusions and long-range issues <cit.>. Many datasets have been collected in both simulation environment <cit.> and real-world <cit.> to facilitate the study. A comprehensive comparison can be found in Table <ref>. While majority of them focused on the upstream tasks such as detection, tracking and segmentation, to enable well-informed decision-making for autonomous vehicles, however, it's critical to also incorporate V2X data for predicting the behavior of surrounding traffic participants. V2X-Seq/Forecasting <cit.> and V2X-Traj <cit.> present two real-world datasets specifically designed for cooperative trajectory prediction tasks. The experiments in V2X-Seq/Forecasting demonstrate that leveraging the infrastructure-side trajectories can enhance the trajectory prediction performance. V2X-Traj aims for more general V2X scenarios, extending to V2V cooperation.




 §.§ Cooperative Information Association and Fusion







The main challenge in cooperative trajectory prediction lies in the multi-view data source fusion. In cooperative scenarios, vehicles gather safety-related data using sensors like radar, lidar, cameras, and GPS. This data is standardized into Basic Safety Message (BSM) format, ensuring compatibility across vehicles and infrastructure <cit.>. BSM messages, containing crucial information such as position and speed, are broadcasted periodically. Upon receiving BSM messages, vehicles combine this data with their own sensor data to enhance accuracy. Advanced algorithms have been developed to associate and fuse the multi-source data. Given the temporal and spatial dimensions of the collected trajectories, there has been research focusing on the communication delays alignment <cit.> and pose errors alignment <cit.>. Based on the aligned data, <cit.> utilized CBMOT <cit.>, a multi-object tracking method, to fuse the infrastructure and ego-vehicle trajectories at each single frame and then trained the network taking in the fused dataset. While straightforward, this method fails to capture the motion behavior provided by the infrastructure across all time steps, leading to suboptimal results. In contrast, <cit.> encoded trajectory information from each view as independent graph nodes, formulating the association process as a graph linking problem. While effective, this approach necessitates separate training procedures. Building upon their work, we propose the utilization of a cross-graph attention mechanism to fuse multi-view information, eliminating the need for additional training processes.


 §.§ Uncertainty Quantification

In the trajectory prediction task, numerous sources of uncertainty exist, including inherent multi-modality, partial observability, short time scales, data limitations, intention type imbalances, and domain gaps. Most existing trajectory prediction models address uncertainty by maximizing the likelihood of an assumed distribution, such as Gaussian or Laplace <cit.>. However, trajectories with the largest likelihood are often nonsensical <cit.>. Alternatively, some research focuses on approximating Bayesian inference for deep learning models using techniques like Monte Carlo dropout <cit.>, which involves performing stochastic forward passes through the network and averaging the results. Among these approaches, <cit.> stands out as the only work that incorporates collaborative uncertainty among agents into the modeling process to guide the ranking of multimodal trajectories by uncertainty, albeit requiring special training strategies. However, none of the predicted uncertainties from these methods offer finite sample coverage guarantees, which is suboptimal for safety-critical applications such as vehicle trajectory prediction.

Conformal prediction (CP) <cit.> has emerged as a widely adopted uncertainty quantification method, owing to its simplicity, generality, theoretical rigor, and low computational overhead. Notably, CP is agnostic to the underlying model and data distribution, making it highly versatile. It seamlessly integrates with any pre-trained model to deliver statistically valid prediction regions. Of particular relevance to our multimodal trajectory prediction task is recent progress in generalizing CP to time-series forecasting. For instance, <cit.> introduced the Copula conformal prediction algorithm for multivariate, multi-step time series forecasting, applicable to any multivariate multi-step forecaster. Additionally, <cit.> focused on generating non-conformity score functions that yield multimodal prediction regions with minimal volume. Moreover, <cit.> employed CP to generate statistical uncertainty intervals from Gaussian mixture model outputs, obtaining separate prediction intervals corresponding to each GMM component prediction. Furthermore, <cit.> proved the validity of CP on graph data. Inspired by their work, we explore the potential of applying CP to the multimodal trajectory prediction comparing different CP methods.



§ PROBLEM FORMULATION


At the scenario level, We have trajectory data 𝒯 from both the vehicle and infrastructure viewpoints, denoted as 𝒯^𝒱 and 𝒯^ℐ, respectively. While 𝒯^𝒱 and 𝒯^ℐ share overlapping information where their fields of view intersect, 𝒯^ℐ also provides complementary information, being free from occlusions. Our modeling objective is to utilize the information from 𝒯^ℐ to improve the accuracy of trajectory prediction based solely on 𝒯^𝒱. 

The trajectory prediction task involves leveraging historical trajectories 𝒯^𝒱_h ∈ℝ^N^𝒱× T_h × a_h and 𝒯^ℐ_h ∈ℝ^N^ℐ× T_h × a_h, alongside contextual information, typically HD maps denoted as ℳ, to forecast future trajectories 𝒯^𝒱_f ∈ℝ^N^𝒱× T_f × a_f. Here, N^𝒱 and N^ℐ represent the number of observed actors from the vehicle and infrastructure perspectives, respectively. T_h denotes the historical time horizon, and T_f is the prediction horizon. a_h and a_f represent the number of node features which we consider the vehicle center location defined by its x- and y-coordinates. Notably, the trajectory data from both views are defined within the same coordinate system. For the HD map, we opt for a vectorized representation due to its lightweight nature and efficiency <cit.>. This vector map is depicted by lane centerlines, which are composed of lane segments. We denote it as ℳ∈ℝ^N^l × a_l, where N^l is the number of lane segments and a_l is the number of lane attribute.

We formulate the overall probabilistic distribution as P(𝒯^𝒱_f | 𝒯^𝒱_h, 𝒯^ℐ_h, ℳ^𝒱, ℳ^ℐ). Driven by the critical safety demands inherent in trajectory prediction, we aim to incorporate uncertainty quantification to preempt any potentially consequential model failures. Let 𝒟 be the set of scenarios of the form (𝒯^𝒱_h, 𝒯^ℐ_h, ℳ^𝒱, ℳ^ℐ, 𝒯^𝒱_f). We split the dataset into training 𝒟_train, validation 𝒟_val, calibration 𝒟_cal and test 𝒟_test. One black-box deep learning model is trained on 𝒟_train and evaluated on 𝒟_val. We achieve the uncertainty quantification by conformal prediction in a post-hoc way on 𝒟_cal.  

At the agent level, we denote the features and labels of agent i from the vehicle view as X^𝒱 = 𝒯^𝒱_h,i and Y^𝒱 = 𝒯^𝒱_f,i for brevity. In real-world scenarios, future trajectories may exhibit multimodal behavior, often approximated by mixture models, resulting in Ŷ^𝒱∈ℝ^K × T_f × a_f, where K represents the number of mixtures or modes. Given a new agent sample X^𝒱_test from 𝒟_test, we seek to construct the prediction intervals 𝒞(X^𝒱_test) ∈ℝ^K × T_f × a_f × 2 such that it covers the ground truth label Y^𝒱_test under a predefined coverage rate leveraging conformal prediction.

CP proceeds in three steps. First, we define a nonconformity score A: 𝒳×𝒴∈ℝ^K × T_f × a_f to quantify how well Y conforms to the prediction at X. Typically, we choose a metric of disagreement between the prediction and the ground truth as the non-conformity score, such as the Euclidean distance. Second, given the predefined miscoverage rate α, we compute the 1-α quantile of the non-conformity scores on the calibration set {A(X_1, Y_1),⋯,A(X_n, Y_n)}, where n is the number of agents. The resulting quantile is denoted as Ĥ∈ℝ^T_f × a_f. Last, when presented with a new test agent X^𝒱_test, CP constructs the prediction interval 𝒞(X^𝒱_test)={Y^𝒱∈𝒴: A(X^𝒱_test, Y^𝒱) ≤Ĥ}.




§ METHODOLOGY



 §.§ Overview

Our method V2INet consists of two key components, predictive modeling and post-hoc conformal prediction. An overview of our proposed model is illustrated in Figure <ref>. 
We first represent the scenario data collected from both views as graphs. A single view encoder is then applied separately to each graph, encoding various information such as agent-agent interactions, temporal dependencies, and agent-lane information. Subsequently, The vehicle-view embedding is fused with the infrastructure-view embedding through a cross-graph attention module. Finally, the updated vehicle-view embedding pass through a multi-modal decoder, providing multimodal predictions for all the agents of interest. The post-hoc conformal prediction is then applied at the agent level to construct valid prediction intervals given a predefined coverage rate.




 §.§ Scene Representation

We adopt an ego-centric coordinate system utilizing vectorized representation as first introduced in <cit.>. First, data from both views are transformed such that the ego vehicle is centered at the origin, with its heading aligned along the positive x-axis. Each trajectory is then characterized as a sequence of displacements  {Δ𝐱^t}^0_t=-(T_h-1), where Δ𝐱^t∈ℝ^2 is the 2D displacement from time step t-1 to t. Similarly, each lane segment is represented as Δ𝐱^l ∈ℝ^2, which captures the 2D displacement from the starting coordinate to the end coordinate of lane segment l. We then construct scenario graphs consisting of actor nodes and lane nodes for both views separately. The edge attribute is the absolute relative position between two nodes.



 §.§ Single View Encoder

To encode the spatiotemporal information, including agent-agent interactions, temporal dependencies, and context information captured by agent-lane relations for data from each view, existing graph-based models offer effective solutions. Models like LaneGCN <cit.>, HiVT <cit.>, and VectorNet <cit.> incorporate these components and can be readily employed. This encoding step can be performed efficiently at the edge device, such as the vehicle's on-board computers and the roadside unit (RSU) for infrastructure data, before broadcasting, thereby minimizing computational overhead.

We exemplify here with HiVT <cit.> where only attention mechanism is employed, with the adoption of a rotation-invariant representation. To attend to the local information, at each timestamp, the surrounding actors' information is aggregated and the embeddings at each time stamp subsequently go through a transformer to capture the temporal dependencies. After obtaining the spatio-temporal embeddings, the surrounding lane information is aggregated for each actor. The update actor node embeddings is then passed to a global interaction module which aggregates the local context of different actors and updates each actor’s representation to capture long-range dependencies and scene-level dynamics. Here's the proofread version:

We denote the final embedding for actor node i from the vehicle view as 𝐡_i, and actor node j from the infrastructure view as 𝐡_j. Both 𝐡_i and 𝐡_j are in ℝ^d_h, where d_h is the embedding dimension.



 §.§ Cross-graph Attention Module

To effectively utilize information from the infrastructure side, we employ a cross-graph attention module that aggregates information captured by the infrastructure view encoder. Specifically, the embedding from the vehicle view 𝐡_i is transformed into the query vector, while the embedding from the infrastructure view 𝐡_j alongside the relative position at the last observed time step 𝐱_i^t=0 - 𝐱_j^t=0, are utilized to compute the key and value vectors. We denote 𝐡_ij=(𝐡_j, 𝐱_i^t=0 - 𝐱_j^t=0) representing the concatenation of node and edge attribute from infrastructure node j to vehicle node i:

    𝐪_i=𝐖^Q_f𝐡_i,    𝐤_ij=𝐖^K_f𝐡_ij,   𝐯_ij=𝐖^V_f𝐡_ij

where 𝐖^Q_f, 𝐖^K_f, 𝐖^V_f∈ R^d_k × d_h are learnable matrices for linear projection and d_k is the transformed dimension. The resulting query, key and value vectors are then taken as input to the scaled dot-product attention block:

     
           α_ij = softmax(𝐪_i^T/√(d_k) ·[{𝐤_ij}_j ∈𝒩_i]),

           𝐦_i = ∑_j ∈𝒩_iα_ij𝐯_ij,

           𝐠_i = sigmoid(𝐖^gate[𝐡_i,𝐦_i]),

           𝐡̃_i = 𝐠_i ⊙𝐖^self𝐡_i+(1-𝐠_i)⊙𝐦_i
    

where 𝒩_i is the set of agent i's neighbors, 𝐖^gate and 𝐖^self are learnable matrices and ⊙ denotes element-wise product. We followed the structure in HiVT to fuse the infrastructure information with a gating function. The attention block supports multiple heads. Finally, we apply another MLP block to obtain the final fused embedding 𝐡̃_i ∈ℝ^d_h for agent i from the vehicle view. 


 §.§ Mixture Model Based Decoder and Learning

There are two widely used mixture models for describing the multimodal trajectories, Gaussian and Laplacian. Previous methods <cit.> have found that the ℓ_1-based loss function derived from the Laplace distribution usually leads to superior prediction performances, as it is more robust to outliers. Hence, we will parameterize the future trajectories following Laplace distribution. For each agent, the decoder receives the final embedding as inputs and outputs K possible future trajectories and the mixing coefficient of the mixture model for each agent. The decoder are consisted of three MLPs, one for predicting the future locations μ^t_i,k∈ℝ^2 for agent i and its mode k at each time step t, one for predicting the associated uncertainty 𝐛^t_i,k∈ℝ^2 assuming independence of the x- and y-coordinates, the last one followed by a softmax is for producing the scores for each mode. 

To ensure the prediction diversity <cit.>, instead of optimizing all the predicted trajectories, only the mode k̃ closest to the ground truth is optimized. The closeness here is defined as the average Euclidean distance between ground-truth locations and predicted locations across all future time steps. The Loss includes both regression loss and classification loss

    
    J = J_reg + ε J_cls


Here, ε is the weight of the classification loss. We employ the negative log-likelihood as the regression loss:

    
    J_reg = -1/nT_f∑^n_i=1∑^T_f_t=1log𝐏(𝐲_i^t - 𝐲̂_i^t | μ̂^t,k̃_i,𝐛̂^t,k̃_i)



where 𝐏(|) is the probability density function of Laplace distribution and μ̂_i^t,k̃, 𝐛̂_i^t,k̃ are the mean and uncertainty estimates of the best mode k̃.

For J_cls, cross-entropy loss is applied:

    
    J_cls = -1/n∑^n_i=1∑^K_k=11_k logπ_k


where 1_k is a binary indicator if k is the best mode, π_k is the mixing coefficient of mode k. The model is trained on 𝒟_train and evaluated on 𝒟_val.



 §.§ Post-hoc uncertainty quantification module

Uncertainty quantification methods are evaluated on two key properties: validity and efficiency. Validity is established when the predicted confidence level exceeds or equals the probability of events falling within the predicted range, while efficiency refers to minimizing the size of the confidence region. We utilize conformal prediction to obtain both valid and efficient prediction intervals. The standard conformal prediction methods typically operate with scalar point estimates for regression problems. However, since our output consists of a multimodal multivariate time-series, we need to make certain adaptations to accommodate this complexity. Following the steps outlined in Section <ref>, we proceed with non-conformity score function definition, quantile computation and prediction interval construction.


  §.§.§ Non-conformity Score Functions

As emphasized in <cit.>, the usefulness of the prediction sets is primarily determined by the score function, we adopt three score functions A and compare their performance.

Z-score. As the decoder returns both mean and variance predictions, we define the Z-score function as:

    Z = |Y -Ŷ|/B̂


where Z ∈ℝ^K × T_f × 2 and B̂ = {{b̂^̂t̂,̂k̂}_t=0^T_f}_k=0^K. Here, we compute scores separately for x- and y-coordinates, reflecting the observation that motion uncertainty varies significantly across different dimensions.

L2-norm. Next, we consider the Euclidean distance, the most commonly used metric in regression problems:

    L_2 = ||Y -Ŷ||_2

where L_2 ∈ℝ^K × T_f.

L1-norm. Recognizing that the L2-norm disregards dimension differences, we consider L1-norm:

    L_1 = ||Y -Ŷ||_1

where L_1 ∈ℝ^K × T_f × 2.



  §.§.§ Quantile Computation

Given a predefined miscoverage rate α∈ [0,1], the 1-α quantile of the non-conformity scores is calculated on the calibration set 𝒟_cal. Conventionally, the quantile is determined as follows: Ĥ=quantile({A(X_1, Y_1),⋯,A(X_n, Y_n)}, (1-α)(1+1/n)) where A(X_i, Y_i) is a scalar, and n denotes the total number of agents in 𝒟_cal. However, in our case, we deal with multimodal time-series scores, which are multivariate when using Z-score and L1-norm, hence necessary adaptations are needed. For brevity, let Γ_i = A(X_i, Y_i), with Γ_i ∈ℝ^K × T_f × 2 for Z-score and L1-norm, and ℝ^K × T_f for L2-norm.

To enhance the efficiency of our prediction intervals, we focus on computing the quantile using only the mode k̃ that exhibits the smallest average Euclidean distance to the ground truth trajectory. This reduces the computation to multivariate time-series. Within this framework, we investigate two established methods: CF-RNN <cit.> and CopulaCPTS <cit.>.

CF-RNN. Since the time-series predictions are obtained from the same embedding, this work proposes the application of Bonferroni correction to the calibration scores to maintain the desired miscoverage rate α. Specifically, the original α is divided by T_f, yielding Ĥ=quantile({Γ_i}_i=0^n, (1-α/T_f)(1+1/n)) for single-variate case.

CopulaCPTS. As implied by its name, this method models the joint probability of uncertainty for multiple predicted time steps using a copula. The calibration set 𝒟_cal is split into two subsets: 𝒟_cal-1, which estimates a Cumulative Distribution Function (CDF) for the nonconformity score of each time step, and 𝒟_cal-2, utilized to calibrate the copula. The copula function captures the dependency between time steps and can enhance the efficiency of prediction intervals.

For multivariate scores generated by the L1-norm and Z-score functions, we adopt the Bonferroni correction for each dimension, as inspired by CF-RNN. Specifically, we use α/2 as the miscoverage rate for both x- and y-coordinates.



  §.§.§ Prediction Interval Construction

We utilize the obtained quantile Ĥ to form the prediction intervals for new examples in 𝒟_test:

    𝒞(X^𝒱_test)={Y^𝒱∈𝒴: A(X^𝒱_test, Y^𝒱) ≤Ĥ}

Specifically, for Ĥ obtained from the Z-score, we have:

    𝒞(X^𝒱_test)=[Ŷ^𝒱_test - B̂Ĥ, Ŷ^𝒱_test + B̂Ĥ]

and for Ĥ obtained from the L2-norm and L1-norm, we have 

    𝒞(X^𝒱_test)=[Ŷ^𝒱_test - Ĥ, Ŷ^𝒱_test + Ĥ]



§ EXPERIMENTS



 §.§ Experimental Setup

In this section, We introduce the specifics of the dataset, the evaluation metrics and the implementation details including hardware, hyperparameters, etc.


  §.§.§ Dataset

We evaluate the proposed model on the publicly available large-scale and real-world V2I dataset V2X-Seq <cit.>, which provides the trajectories of agents from both vehicle and infrastructure sides, along with vector map data. V2X-Seq consists of 51,146 V2I scenarios, where each trajectory is 10 seconds long with a sampling rate of 10 Hz. The task involves predicting the motion of agents in the next 5 seconds, given initial 5-second observations from both infrastructure and vehicle sides. The dataset has been split into train and validation. Our trained model is evaluated on the validation set, allowing for comparison with existing models. For post-hoc conformal prediction, we divide the validation set into calibration set and test set at a 4:1 ratio. For a discussion on the calibration data size, please refer to <cit.>.



  §.§.§ Evaluation Metrics
 
Model metrics. For model evaluation, the standard metrics in motion predictions are adopted, including minimum Average Displacement Error (minADE), minimum Final Displacement Error (minFDE), and Miss Rate (MR), where errors between the best predicted trajectory among the K=6 modes and the ground truth trajectory are calculated. The best here refers to the trajectory that has the minimum endpoint error. The ADE metric calculates the L2 distance across all future time steps and averages over all scored vehicles within a scenario, while FDE measures the L2 distance only at the final future time step and summarizes across all scored vehicles. MR refers to the ratio of actors in a scenario where FDE are above 2 meters. 

Conformal prediction metrics. We assess validity and efficiency for each method. Validity is evaluated by reporting independent and joint coverage on the test set, aiming for coverage levels close to the desired confidence level 1-α. The independent coverage for each agent is calculated as:

    ind. coverage_1-α = 1/T_f∑_X,Y ∈𝒟_test1(Y ∈𝒞(X))

We identify the maximum independent coverage among the K modes for each agent. For all metrics, the final reported value is the average among all agents in 𝒟_test.

For efficiency, we calculate the average size of the predicted 2D area across all time steps for the mode k̃ with the maximum independent coverage:

    size = 1/T_f||𝒞(X)_k̃||

The 2D area is defined as ellipsis for Z-score and L1-norm, and circle for L2-norm.

The joint coverage is defined as if there exists a mode k such that the truth values across all time steps fall into the confidence region:

    joint coverage_1-α = 1(∃k∈ K: ∀t∈ T_f: Y_k,t∈𝒞(X))




  §.§.§ Implementation Details

Model. The model was trained for 64 epochs on an Nvidia V100S GPU with 32GB memory using AdamW optimizer <cit.>. Hyperparameters including batch size, initial learning rate, weight decay and dropout rate are 32, 1e-3, 1e-4 and 0.1, respectively. The learning rate is decayed using the cosine annealing scheduler <cit.>. Our model employs the original setting of HiVT with 64 hidden units for the single view encoder and 1 layer of cross-graph attention module with 8 heads. The radius of all local regions is 50 meters. The number of prediction modes is set to 6.

Conformal prediction. With the pretrained model and datasets 𝒟_cal and 𝒟_test, we execute all conformal prediction methods on the CPU. We evaluate the methods on the three defined score functions at three different α levels: 0.2, 0.1, 0.05. The optimization step in CopulaCPTS remains consistent with the original work.



 §.§ Results Analysis

In this section, we first examine the model's prediction performance, assessing it from both quantitative and qualitative perspectives. Next, we showcase the effectiveness of the post-hoc uncertainty quantification method.


  §.§.§ Model Performance

Comparison with state-of-the-art. We benchmark our proposed model against state-of-the-art models using the V2X-Seq dataset, as detailed in <cit.>. Results are summarized in Table <ref>. Both TNT <cit.> and HiVT <cit.> are single-view models. <cit.> evaluated them under two settings: Ego, where only the vehicle-view data is utilized, and PP-VIC, which employs a two-stage method with both vehicle view and infrastructure data. Specifically, in PP-VIC, data from both views are fused offline with some tracking method and then the stitched trajectories were fed into a single-view model. We retrained the HiVT model under both Ego and the PP-VIC setting with 64 epochs and present our results in Table <ref>. Comparing the Ego and PP-VIC results, it's uncovered by <cit.> that integrating information from the infrastructure side can enhance prediction accuracy. V2X-Graph represents the current state-of-the-art model, employing a graph link prediction module to associate two-view data, followed by fusing the embeddings of the associated nodes using attention mechanism. However, this node association module requires pre-training, leading to a two-stage training process.  Our method demonstrates the best performance in terms of minFDE and MR. Without the explicit node association from both views, our attention based fusion module can attend to the most relevant nodes from both views through learning, which significantly simplifies the modeling framework and facilitates ease of training, all while achieving better results.  


Qualitative results. We visualize our prediction results and six representative scenarios are shown in Figure <ref>. To maintain clarity and simplicity in the visualization, each scenario displays the ground truth and multimodal prediction results only for the target agent, although predictions for all agents are accessible. In the first column, scenarios S1 and S4 depict the target agent turning right. In the second column, the target agent is shown going straight, leaving (S2) and approaching (S5) an intersection. The last column presents scenarios where the target agent is merging (S3) and turning left (S6). It's shown that our model predictions capture the multimodal behavior. In scenarios S2 and S5, it's exhibited in the form of different velocity profiles. Since lane information is integrated via an attention mechanism rather than as rigid constraints, off-road and road rule-violating predictions and may occur, as shown in S4 and S6. Notably, there are more uncertainties when agent making turns, as evidenced by the dispersion among the predictions. Furthermore, it's important to highlight that the prediction with the highest probability does not always align with the ground truth, as demonstrated in S1 and S3, resulting from the inherent model and data uncertainties. 



We further compare the prediction results from model HiVT Ego, HiVT PP-VIC and ours on selected scenarios, as shown in Figure <ref>. It's observed that HiVT Ego, relying solely on vehicle-view data, consistently predicted incorrect modes. In contrast, both HiVT PP-VIC and our model, incorporating additional data from the infrastructure view, demonstrate improved accuracy in capturing the ground truth.





  §.§.§ Post-hoc Uncertainty Quantification Performance

Quantitative comparison. We show in this section that conformal prediction method produces more valid and efficient confidence regions than the model predictions. The evaluation results are shown in Table <ref>.


The Mixture column presents UQ results derived directly from the multimodal predictions. We first examine the metrics of independent and joint coverage. It's evident that the model predicted intervals fail to meet the validity requirements across all specified miscoverage rates. Interestingly, CF-RNN yields overly conservative results, maintaining coverage levels around 0.99 regardless of the miscoverage rate. This could indicate that Bonferroni correction, often employed with shorter horizons, might not be suitable for our predicted horizon of 50. CopulaCPTS achieves coverage levels very close to the desired values across all specified miscoverage rates α.

Next, we inspect the metric "size", which serves as an indicator of UQ efficiency. Benchmark the size from model predictions in the Mixture column, CF-RNN produces extremely large confidence region which is expected considering the overly conservative coverage observed earlier. CopulaCPTS yields comparative and even smaller size compared to the benchmark. 

Finally, we examine the impact of different score functions. In both CF-RNN and CopulaCPTS, Z-score appears as the least efficient among all three functions. Since Z-score is computed using the model-predicted uncertainties B̂, it will always be more inefficient than the benchmark size in the "Mixture" column. One plausible explanation provided in <cit.>, is that although widely used, B̂ in the Z-score function is not directly related to the quantiles of the label distribution. Therefore, it may not be the most suitable score function option in our case. On the other hand, both L2-norm and L1-norm demonstrate significantly higher efficiency, particularly within the CopulaCPTS method. L1-norm, in particular, considers variations among different dimensions, leading to enhanced efficiency. Taking into account both validity and efficiency requirements, we have highlighted the best results among all methods in red.

Qualitative analysis. We visualize the uncertainty prediction from the mixture model and the CopulaCPTS, as shown in Figure <ref>. Three scenarios are selected in each column. We can see that the mixture model predictions fail to cover the ground truth while CopulaCPTS achieves the coverage at the specified α. 
 





§ CONCLUSION

We have presented a novel model framework with multi-view data integration in the cooperative driving setting. Our proposed model is straight forward and can be built upon any existing graph-based single-view models. It has demonstrated its effectiveness and advantages over existing benchmarks. Moreover, we have incorporated a post-hoc uncertainty quantification module, providing valid and efficient confidence regions, which is crucial in safety-critical tasks such as trajectory prediction. 

The proposed framework has certain limitations. From the model's perspective, we currently treat all road agents as the same type. However, in the public V2X-Seq dataset, there are different vehicle types, such as trucks, vans, buses, motorcycles, etc. Future work should address the different characteristics of these vehicle types to enhance model performance. Moreover, better methods for encoding the lane information to eliminate off-road and road-rule-violating predictions should be investigated. From the uncertainty quantification perspective, we simplify the quantile computation on the multimodal prediction into computation on the single best mode. This approach loses valuable distributional information. Therefore, exploring score functions that consider the distribution could lead to more accurate uncertainty quantification for multimodal results. Furthermore, we evaluate the uncertainty assuming independence among the agents. Future work can incorporate agent correlations based on the graph structure to better reflect the underlying uncertainty relations and provide more insightful results.

unsrt  
  




 Paper Title: Force Sensing Guided Artery-Vein Segmentation via Sequential Ultrasound Images.



 

Y. Geng et al.



State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China School of Artificial Intelligence, University of Chinese Academy of Sciences Centre for Artificial Intelligence and Robotics, HK Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong SAR City University of Hong Kong, Hong Kong SAR School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK

liuhongbin@ia.ac.cn

Force Sensing Guided Artery-Vein Segmentation via Sequential Ultrasound Images
    Yimeng Geng1,2 Gaofeng Meng1,2,3 Mingcong Chen3,4 Guanglin Cao1,2,3 Mingyang Zhao3Jianbo Zhao1,2 Hongbin Liu1,3,5^()

    August 5, 2024
=========================================================================================================================



§ ABSTRACT 
Accurate identification of arteries and veins in ultrasound images is crucial for  vascular  examinations and interventions in  robotics-assisted surgeries. However, current methods for ultrasound vessel segmentation  face challenges in distinguishing between arteries and veins due to their morphological similarities. 
To address this challenge, this study introduces a novel force sensing guided segmentation approach to   enhance artery-vein segmentation accuracy by leveraging their distinct deformability. Our proposed method utilizes force magnitude to identify key frames with the most significant vascular deformation in a sequence of ultrasound images. These key frames are then integrated with the current frame through attention mechanisms, with weights assigned in accordance with force magnitude. Our proposed force sensing guided framework can be seamlessly integrated into various segmentation networks and achieves significant performance improvements in multiple U-shaped networks such as U-Net, Swin-unet and Transunet. Furthermore, we contribute the first multimodal ultrasound artery-vein segmentation dataset, Mus-V, which encompasses both force and image data simultaneously. The dataset comprises 3114 ultrasound images of carotid and femoral vessels extracted from 105 videos, with corresponding force data recorded by the force sensor mounted on the US probe. Our code and dataset will be publicly available.









§ INTRODUCTION

Vascular intervention surgery (VIS) is  a minimally invasive surgical technique involving catheter or device insertion into blood vessels <cit.>.  This real-time and low-risk surgery plays a crucial role in treating various  diseases such as coronary artery diseases <cit.> and 
peripheral vascular diseases <cit.>. Accurate segmentation of arteries and veins is vital in VIS, enabling clinicians to precisely identify target vessels for intervention and minimizing the risk of inadvertent injury to adjacent structures <cit.>. As a non-invasive, radiation-free and real-time modality, ultrasound imaging is widely used in VIS <cit.>.  Therefore, artery-vein segmentation in ultrasound images is crucial for enhancing the safety and efficacy of VIS. 

Ultrasound vessel segmentation, a subset of medical image segmentation, is developing rapidly, primarily because of the rapid evolution of deep learning techniques <cit.>.
In particular, a series of U-shaped networks have demonstrated outstanding performance <cit.> and been widely applied in ultrasound vessel segmentation. Xie et al. <cit.> directly utilizes UNet <cit.> to segment carotid artery lumen.  Blanco et al. <cit.> presents a modified UNet to exploit adjacency information in spatially neighboring ultrasound images for  intravascular ultrasound images segmentation.  Groves et al. <cit.> 
 employs both Mask-RCNN <cit.> and UNet to segment carotid artery (CA) and internal jugular vein (IJV).




However, current ultrasound vessel segmentation networks either solely focus on morphological features within a single image <cit.>, making it challenging to differentiate arteries and veins that are morphologically similar, or only segment vessels without distinguishing between arteries and veins <cit.>. In clinical practice, doctors typically differentiate arteries from veins by palpating the patient's skin and observing the degree of vascular deformation in ultrasound images <cit.>. During this process, arteries exhibit minimal deformation under compression, while veins undergo significant deformation and can even be completely occluded <cit.>. Inspired by this method, we propose a novel force sensing guided approach for ultrasound artery-vein segmentation.This method aims to precisely segment arteries and veins using tactile force data, especially in scenarios involving robotic ultrasound systems.

 Robotic Ultrasound Systems (RUS)  can improve the accuracy and safety of VIS through accurate image guidance and robotic arm operation, and provide better operating experience for doctors <cit.>. We set up a simple RUS in our experiments as shown in Fig. <ref>(a). The sensor mounted on the US probe can collected real-time force data, which is employed as a prior to identify key frames that represent maximum deformation.  These key frames effectively capture and store vascular deformation information, aiding in distinguishing the distinct deformability of arteries and veins as shown in Fig. <ref>(b). Then we can retrieve the stored information by computing the correlation between  key frames and the current frame to get enhanced features.  To validate the effectiveness of the proposed method, we also contribute a benchmark dataset containing both ultrasound images and force data called Mus-V. 

The main contributions are:
(1) We present a novel force sensing guided segmentation approach that utilizes force data to assist in artery-vein segmentation, leading to  enhanced segmentation accuracy. 
(2) Our method is versatile and demonstrates substantial performance improvement across multiple segmentation networks. 
(3) We contribute the first publicly available ultrasound artery-vein segmentation dataset with force data, providing a baseline for future researches.




§ ULTRASOUND VASCULAR DATASET WITH FORCE DATA
 
We introduce Mus-V: the first ultrasound artery-vein segmentation dataset containing both force and image data. The dataset collects ultrasound videos of carotid and femoral blood vessels, with the arteries and veins labelled separately for vascular analysis and identification. In addition, the dataset also records the force data collected by the force sensor mounted on the ultrasound probe during the collection of ultrasound videos. The structure of the robotic ultrasound system consists of an ultrasound (US) probe, a robotic arm carrying the US probe and a force/torque sensor mounted on the US probe (see Fig. <ref>(a)).  In each data collection session, an ultrasound video is recorded in conjunction with corresponding force data.  The contact force between the US probe and the skin is systematically adjusted to vary gradually. Meanwhile, the deformation of blood vessels is also strengthened or reduced as Fig. <ref>(b) shows. Subsequent to acquisition, the ultrasound videos and force data undergo aligning, cleaning, downsampling, and annotation.


A total of 3114 ultrasound images were sampled from 105 videos. The training and validation datasets comprise 2203 and 911 images, respectively. The force data is organized in a matrix format, with each row corresponds to an ultrasound image. Each column, from the first to the last, represents the force components in the X, Y, and Z directions, as well as the moment components in the X, Y, and Z directions, respectively.



§ METHOD



 §.§ Overview

The most obvious difference between arteries and veins in US images lies in the extent of deformation when contact force varies. Therefore, capturing the deformation of vessels over time is crucial for accurate segmentation, which requires to compare different frames within the same video. Deformation magnitude correlates directly with force magnitude, thereby we utilize contact force data to identify key frames for reference. The selected key frames exhibit the most substantial degree of vascular deformation. Specific method for selecting key frames is described in Sec. <ref>. These key frames aid in  differentiation between arteries during the subsequent segmentation process.



Image segmentation networks based on the UNet architecture typically consist of an encoder that downsamples features and a decoder that upsamples features. Skip connections are utilized between different stages of the encoder and the decoder to fully leverage information at various resolutions. In the proposed fusion approach, key frames and the current segmentation frame are firstly fed into a shared encoder for feature extraction, with the final stage of the encoder outputting features with low resolution and high channel numbers. Subsequently, these features are processed by our proposed force-guided attention module. The fused features then enter the decoder to restore the original resolution and obtain the mask. This design allows for the encoder and decoder to be replaced with any U-shaped network, thereby enhancing the versatility of the method.


 §.§ Force-Driven Keyframe Selection

In the Robotic Ultrasound System, magnitude of contact force varies continuously, and changes between two consecutive frames may not be particularly evident. The magnitude of deformation is directly positively correlated with  magnitude of force along the z-axis. Therefore, we utilize force along the z-axis to select key frames that contain the most significant deformation information.  The target ultrasound image for segmentation is denoted as K_cur.  K_cur is one frame from an ultrasound video with a duration of T and 𝒦={ K_0,…,K_T } indicates  the set of all the frames in the same video. Correspondingly,  ℱ={ F_0,…,F_T } represents the absolute value of force along the z-axis collected with 𝒦.

As illustrated in Eq. <ref>, we select the image frame corresponding to the maximum force, denoted as K_max, and the frame corresponding to the minimum compression force, denoted as K_min. These frames, K_min and K_max are referred to as key frames.  


    {[ K_min = K_min_t  F_t; K_max = K_max_t  F_t ].

Therefore, K_min corresponds to the most dilated form of the blood vessel, while K_max corresponds to the most compressed form.


 §.§ Force-Guided Attention Module

The output of the encoder has high-dimensional channels which are computationally expensive. Inspired by <cit.>, we firstly apply an encoding layer to the output for channel reduction. As illustrated in Fig. <ref>, the encoding layer including  a 1x1 convolution and a 3x3 convolution will generate a key feature and a value feature. We respectively stack the key features and value features related to key frames K_min and K_max, and obtain a 4-dimension key feature D_V∈ R^N× C_K× H × W and a 4-dimension value feature D_K ∈ R^N× C_V× H × W, where N=2. E_K and E_V are the corresponding  key and value feature related to K_cur.

For frames in  proximity to a key frame, the reference value of the key frame is negligible. For example, if K_cur = 
K_max, then K_max will fail to provide any meaningful information. Therefore, we propose a force-based dynamic weighting strategy, assigning a higher weight to the key frame with lower similarity to the current frame. Detailed calculation of weights is illustrated in  Sec. <ref>.


After obtaining force-based weights w_min and w_max, we perform element-wise multiplication between the weights and D_V to get weighted values D̂_V. Then, E_K and E_V  are transposed and reshaped to E_K∈ R^C_K × M and E_V∈ R^C_V × M, M=H× W. Meanwhile, D_K and D̂_V are  permuted and reshaped to D̂_K∈ R^C_K × NM and D̂_V∈ R^C_V × NM.

Subsequently, we conduct matrix multiplication between D_K and E_K, and utilize a softmax layer to compute the attention map S∈ R^N× M,

    S_ij=exp(D^i_K· E^j_K)/∑_j=1^Eexp(D^i_K· E^j_K)

A larger value of S_ij indicates that the i^th position in D_K has a greater influence on the j^th position in E_K
Memory. After obtaining the attention map S, we multiply S and D̂_V to get the force-guided enhanced feature D̃_V.
Finally, we combine D̃_V with the value feature of the current frame. For simplicity, we utilize feature concatenation to get the aggregated multimodal feature f as follows:

    f = concat(D̃_V,E_V)





 §.§ Force-Based Dynamic Weights

Adhering the principle that a higher weight is assigned to the key frame with lower similarity to the current frame, we design a weight calculation formula  based on  force magnitudes. The dynamic weights are directly proportional to the difference in force magnitudes and lies within the range of [ 0,1  ],

    w_min=F_cur-F_min/F_max-F_min


    w_max=F_max-F_cur/F_max-F_min

where  w_min and w_max are  force-based dynamic weights for K_min and K_max. For each input, a new pair of force-based weights is computed and applied. F_cur, F_min, F_max are force respectively corresponding to K_cur, K_min and K_max.


§ EXPERIMENTS



 §.§ Implementation and Setup  

Our experiments are implemented based on PyTorch on a RTX 4090 GPU.  We employ the ReduceLROnPlateau <cit.> 
policy to automatically adjust the learning rate and utilize RMSprop <cit.> as the optimizer. Momentum, weight decay and batch size are set to 0.9, 1e-8 and 4 for all experiments, respectively. 
We set the maximum number of epochs as 120 for experiments. For data augmentation, we apply random horizontal and vertical flip, random intensity transformation, random translation, random rotation  for input images and key frames for all experiments. All images are resized to 256x256.
We also utilize Torchio <cit.> to add random bias fields and random noise to input images and key frames. To address the issue of class imbalance, we employed weighted cross-entropy loss with weights of 1, 30.7, and 23.1 for background, arteries and veins, respectively. 



 §.§ Comparison with State-of-the-Arts

We conduct comprehensive experiments by applying force sensing guidance to three different U-shaped networks, namely UNet, Swin-unet and Transunet. The force-guided attention module is inserted as a “neck” between the encoder and the decoder of three basic networks. For Transunet, we choose to use the R50-Vit-B16 version. Table <ref> displays the performance metrics and FLOPs of all the methods. The network applied with force sensing guidance is named with the prefix “FG” followed by the network name.  In conclusion, Force Sensing Guided TranUnet (FG-Transunet) achieves the highest performance, attaining  75.63% mIoU and 0.8547 dice coefficient. Additionally, the application of force sensing guidance results in performance enhancements of 1.2%, 2.79% and 1.25% mIoU in Swin-UNet, UNet and Transunet, respectively . Qualitative results are presented in Fig. <ref>
  


 §.§ Ablation Studies

We conduct several ablation experiments to validate the effectiveness of all the components of the proposed approach, utilizing UNet <cit.> as our baseline. To save computational resources and training time, we also exploit Force Sensing Guided UNet (FG-UNet) as the base model for all ablation experiments.



To validate the effectiveness of the  force-based dynamic weights, we conducted experiments by removing the weights multiplication. The experimental results are shown in Table <ref>. After removing the force-based weights, the mIoU decreases by 0.66%.


To investigate the effect of the proposed force-driven key frame selection strategy, we conduct an experiment that simply selecting two frames preceding the current frame in time as key frames. The results are displayed in Table <ref>. The performance has decreased significantly in the aspect of both mIoU and dice coefficient  after removing the key frame selection. The mIoU decreases by 1.14% and  the dice coefficient  decreases by 0.039.



§ CONCLUSION

In this study, we introduced a pioneering force sensing guided approach for ultrasound artery-vein segmentation. Our method includes innovative components such as force-driven key frames selection, force-guided attention module and force-based dynamic weights. Extensive experiments demonstrated that our proposed approach led to   notable performance enhancements across multiple U-shaped networks. Furthermore, we presented Mus-V, the first publicly available ultrasound artery-vein  segmentation dataset including images and force. Future directions include further optimizing the performance by exploring additional methodologies, as well as expanding the capacity of Mus-V.









splncs04




 Paper Title: Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network.



Article Title]Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network










1]Bin Chengchengbin2422@mails.jlu.edu.cn

[2]Jiaxuan Lulujiaxuan@pjlab.org.cn






[1]College of Software, Jilin University, Changchun, 130012, Jilin, China

[2]Shanghai AI Lab, Xuhui District, 200232, Shanghai, China





With the advent of the era of foundation models, pre-training and fine-tuning have become common paradigms. Recently, parameter-efficient fine-tuning has garnered widespread attention due to its better balance between the number of learnable parameters and performance. However, some current parameter-efficient fine-tuning methods only model a single modality and lack the utilization of structural knowledge in downstream tasks. To address this issue, this paper proposes a multi-modal parameter-efficient fine-tuning method based on graph networks. Each image is fed into a multi-modal large language model (MLLM) to generate a text description. The image and its corresponding text description are then processed by a frozen image encoder and text encoder to generate image features and text features, respectively. A graph is constructed based on the similarity of the multi-modal feature nodes, and knowledge and relationships relevant to these features are extracted from each node. Additionally, Elastic Weight Consolidation (EWC) regularization is incorporated into the loss function to mitigate the problem of forgetting during task learning. The proposed model achieves test accuracies on the OxfordPets, Flowers102, and Food101 datasets that improve by 4.45%, 2.92%, and 0.23%, respectively. The code is available at https://github.com/yunche0/GA-Net/tree/master.








[
    *
    
=====





§ INTRODUCTION



With the onset of the era of foundational models, we have gradually entered a paradigm of pre-training and fine-tuning. In terms of downstream task adaptation, full fine-tuning requires adjusting all the parameters of a model to adapt to downstream tasks. However, as the scale of models and the number of tasks increase, such a method becomes inefficient. Consequently, numerous studies have focused on parameter-efficient fine-tuning, exploring strategies to efficiently adapt existing foundation models to downstream tasks. Previous parameter-efficient fine-tuning methods can be mainly categorized into three approaches. The first is prompt tuning <cit.><cit.>, which aims to achieve fine-tuning by modifying the model input rather than the model structure. The second is prefix tuning <cit.>, which involves updating only task-specific trainable parameters within each layer. The third is adapter tuning <cit.><cit.>, which achieves parameter-efficient fine-tuning by inserting adapter modules with a bottleneck architecture between layers.

In the realm of multi-modal learning, parameter-efficient fine-tuning methods have gained attention in recent research. Prompt vectors are used to align multi-modal data, achieving efficient multi-modal fusion in low-resource settings <cit.>. The main idea of CoOp <cit.> is to automatically design prompt texts. It keeps the pre-training parameters unchanged and uses a small amount of data to learn the appropriate cues. The TaskRes <cit.> method directly adjusts the weights of text-based classifiers without requiring extensive prompt updates to text encoders or elaborate adapters. π-Tuning optimizes parameters by predicting and interpolating task similarity across visual, language, and visual-language tasks, achieving efficient cross-task transfer learning <cit.>. However, these works do not consider modeling the complex associations between modalities.

With respect to modeling complex associations for parameter-efficient fine-tuning, methods related to graph neural networks (GNNs) have been explored. The concept of timely tuning has been introduced <cit.>. In the molecular domain, MolCPT <cit.> enhances graph embeddings by encoding additional molecular motif information. However, these works primarily focus on how to perform parameter-efficient fine-tuning for purely graph structures, without applying them to language-image multi-modal modeling.

Therefore, we propose a framework that combines graph structures with multi-modal parameter-efficient fine-tuning methods, enabling the learning of multi-modal information while considering the complex associations between modalities. The proposed model comprises four main modules: Multi-Modal Feature Extraction, Multi-Modal Graph Construction, Graph Adapter Net (GA-Net), and Prediction. In the Multi-Modal Feature Extraction module, each image is processed by a pre-trained MLLM model to obtain a text description for each image. The image and its corresponding text description are then processed by frozen image and text encoders to generate image features and text features, respectively. These features are combined into multi-modal features through feature concatenation. In the Multi-Modal Graph Construction module, a graph is constructed based on the similarity of multi-modal feature nodes. The GA-Net module then mines suitable knowledge from the graph nodes, resulting in features that have fully learned both textual and image information while considering their adjacency relationships. In the Prediction module, the loss function incorporates cross-entropy and EWC regularization <cit.>, which can mitigate the forgetting problem in task learning.

Compared with the current SOTA method, the proposed method improves by 4.45% in the OxfordPets dataset, 2.92% in the Flowers102 dataset and 0.23% in the Food101 dataset.

The contributions of this paper are summarized as follows:

    
  * A parameter-efficient fine-tuning method based on graph networks, GA-Net, is proposed. The proposed method combines graph structures with multi-modal parameter-efficient fine-tuning, learning both textual and image information while considering the adjacency relationships between different tokens.
    
  * EWC regularization is introduced into the loss function. By incorporating the correlation between parameter importance and the loss function, the EWC loss effectively retains knowledge from previous tasks and reduces interference with previous tasks when learning new tasks, thereby alleviating the problem of forgetting in task learning.
    
  * Compared to the SOTA model, the proposed model improves the test accuracy by 4.45% on the OxfordPets dataset, 2.92% on the Flowers102 dataset and 0.23% on the Food101 dataset.




§ RELATED WORK


Parameter-Efficient Fine-Tuning Methods (PEFT). Full fine-tuning involves modifying all the model's parameters to suit downstream tasks. Yet, as models grow in scale and the number of tasks expands, this approach becomes increasingly inefficient. To address this issue, in recent years, the natural language processing (NLP) community has explored parameter-efficient fine-tuning techniques (PEFT) <cit.><cit.><cit.><cit.><cit.>. These techniques only require adjusting a small subset of parameters, thereby improving efficiency <cit.>. For example, prompt tuning methods <cit.> attempt to achieve fine-tuning by modifying the model input rather than the model structure. Prefix tuning <cit.> updates only task-specific trainable parameters in each layer. Adapter tuning <cit.><cit.> inserts adapter modules with bottleneck architectures between layers to achieve parameter-efficient fine-tuning. Additionally, methods like BitFit <cit.> update only the bias terms and freeze the remaining parameters, while LoRA <cit.> reduces the number of trainable parameters by decomposing the weight matrix into low-rank matrices.

In multi-modal learning, parameter-efficient fine-tuning methods have gained widespread attention in recent research <cit.><cit.><cit.><cit.><cit.>. Using prompt vectors to align multi-modal data achieves efficient multi-modal fusion in low-resource environments, excelling in tasks involving two or more data modalities <cit.>. Research on scaling large multi-modal models (such as LLaVA and MiniGPT-4) has shown that parameter-efficient training methods like LoRA/QLoRA perform well in both multi-modal and language tasks, with performance comparable to full-model fine-tuning<cit.>. The main idea of CoOp <cit.> is to automatically design prompt texts. It keeps the pre-trained parameters unchanged and learns suitable prompts using a small amount of data. CLIP-adapter <cit.> inserts a random learnable module into the middle of the model. By updating this module, it better adapts to downstream tasks. Unlike CLIP-Adapter, Tip-Adapter <cit.> does not require SGD to train the adapter. Instead, it constructs a query-key cache model from few-shot supervision to obtain the adapter's weights. TaskRes <cit.> directly adjusts the weights of the text-based classifier without needing extensive prompt updates to the text encoder or carefully designed adapters. π-Tuning optimizes parameters by predicting and interpolating task similarity across visual, language, and visual-language tasks, achieving efficient cross-task transfer learning, especially effective in data-scarce situations <cit.>. PMF significantly reduces training memory usage by adding prompt vectors only in the deeper layers of a single-modal Transformer <cit.>. However, these methods do not model the complex associations between modalities.

Graph Neural Networks (GNN). Early work by Scarselli et al. <cit.> laid the foundation by proposing a framework for learning node representations in graphs, capable of capturing dependencies between nodes through iterative information passing. The pioneering work by Bruna et al. <cit.> applied convolutional neural networks to graph data in the spectral domain. However, this approach faced challenges in computational efficiency and generalization across different graph structures. The graph convolutional networks (GCNs) proposed by Kipf and Welling <cit.> are among the most influential models, performing convolution by aggregating feature information from the local neighborhood of nodes, thus achieving efficient and scalable learning. The graph attention networks (GATs) introduced by Veličković et al. <cit.> employ an attention mechanism to weight the importance of neighboring nodes, allowing for more flexible and dynamic information aggregation. Additionally, GraphSAGE by Hamilton et al. <cit.> introduced a sampling-based approach for large-scale graph representation learning. Numerous prior works have applied GNNs to association modeling tasks across various domains, including vision-based<cit.><cit.>, text-based<cit.><cit.>, and graph-based<cit.><cit.><cit.><cit.> applications.

Parameter-efficient tuning methods have also seen some exploration in the GNN domain. The concept of timely tuning has been applied <cit.>. Although methods like GPF <cit.> and GraphPrompt <cit.> are also parameter-efficient, they struggle to match the benchmarks set by full fine-tuning in non-few-shot settings. GPPT <cit.> designed a specific framework for GNNs, but its application is limited to node-level tasks. In the molecular domain, MolCPT <cit.> enhances graph embeddings by encoding additional molecular motif information. However, the aforementioned works design parameter-efficient fine-tuning methods specifically for pure graph structures. To the best of our knowledge, graph networks have not been applied in the field of multi-modal parameter-efficient fine-tuning.





§ METHOD



As shown in Figure <ref>, the model presented in this paper is composed of four main modules: Multi-Modal Feature Extraction, Multi-Modal Graph Construction, GA-Net, and Prediction. The primary function of the Multi-Modal Feature Extraction is to use pre-trained models to extract features from images and text. Then, through Multi-Modal Graph Construction, a multi-modal graph is built to relate the connections between different modalities. The proposed GA-Net is the only trainable part of the network, where a down projection is first introduced, followed by a GCN, and finally an up projection to propagate and update the vertex features in the graph. Lastly, in the Prediction module, we propose a combined loss function of EWC loss and cross-entropy loss to further enhance the network's performance. The details of each module are as follows:












 §.§ Multi-Modal Feature Extraction


In this module, we first use a pre-trained MLLM model to generate general text descriptions corresponding to the images. These descriptions do not involve the category names used in the final classification. Next, the images and their corresponding text descriptions are fed into frozen image encoders (such as ViT <cit.> or ResNet <cit.>) and text encoders (such as BERT <cit.>), respectively, to generate image features and text features. Finally, the image features and text features are combined into multi-modal features through feature concatenation. Mathematically, suppose X_i is the input image, and MLLM(∗) represents the MLLM model, the text description of image X_i obtained through the MLLM model can be expressed as

    X_i^t = MLLM(X_i)


Let I(X_i) and T(X_i^t) represent the series of tokens obtained from the frozen image encoder (ViT/ResNet) and text encoder (BERT), respectively. These can be expressed as

    V_i^I = {X_i^I^1, X_i^I^2, …, X_i^I^N} = I(X_i)



    V_i^T = {X_i^T^1, X_i^T^2, …, X_i^T^N} = T(X_i^t)


where each token is of dimension E. The image tokens encode information from each patch location, while the text tokens encode information from each word location. V_i^I and V_i^T are sets of tokens. Each element in V_i^I represents the i-th token in the image, and each element in V_i^T represents the i-th token in the text. N denotes the number of tokens. All tokens form a series of text and image vertices:

    V_i = {V_i^I, V_i^T} = Concat(V_i^I, V_i^T)

where Concat(∗) represents feature concatenation, and V_i represents a node after concatenating text features and image features.






 §.§ Multi-Modal Graph Construction

To uncover the structural knowledge in the text embedding space for downstream tasks, i.e., the relationships between different semantics, and given the diverse visual features of different samples, we can measure finer-grained relationships between different semantics in the visual and textual space. Thus, we can construct a multi-modal graph structure G = {V_i, E} = {V_i^I, V_i^T; E}, where V_i^I and V_i^T can be seen as the sets of image vertices and text vertices, respectively. E represents the set of edges.

We build the graph using the similarity of multi-modal features via a predefined threshold γ. When the similarity between two multi-modal features is greater than γ, an undirected edge is created between these two vertices, representing the adjacency relationships between all multi-modal features:

    E_ij =
        
          1,    if  i ≠ j  and Sim(V_i, V_j) > γ
    
          0,    otherwise

where

    Sim(V_i, V_j) = V_i · V_j/V_iV_j

represents the similarity between multi-modal nodes V_i and V_j. γ is the similarity threshold, and an edge is constructed when the similarity between two vertices in the graph exceeds this threshold.



 §.§ Graph Adapter Net

We propose a parameter-efficient fine-tuning method based on graph networks called Graph Adapter Net (GA-Net), where the rest of the network is frozen and only the GCN <cit.> is fine-tuned during downstream tasks. The advantage of this method is that it can adapt to downstream tasks and improve model performance without significantly increasing the number of model parameters. Furthermore, since adapter fine-tuning only requires training a small number of parameters, it can significantly reduce computational resources and time costs for fine-tuning. The unique aspect of GA-Net is its ability to update features based on our constructed multi-modal graph structure, allowing fine-tuning while preserving adjacency relationships. The process of GA-Net can be represented by the following formula:

    C_X_i = W_up( GCN( W_down(X_i) ) )






where C_x represents the vertex feature matrix. The GA-Net utilizes a bottleneck architecture, including a down-projection W_down: ℝ^n_in→ℝ^n_mid, a Graph Convolutional Network (GCN) layer, and an up-projection W_up: ℝ^n_mid→ℝ^n_out. For vertex feature aggregation, we utilize the GCN, and the update formula for the vertex features in each layer can be formulated as:

    C^(l+1) = σ( D̂^-1/2ÂD̂^-1/2 C^(l) W^(l))

where Â is the normalized adjacency matrix, defined as Â = A + I, where A is the original adjacency matrix and I is the identity matrix. D̂ is the degree matrix of Â, with diagonal elements D̂_ii = ∑_j Â_ij. W^(l) is the trainable weight matrix of the l-th layer, with dimensions F_l × F_l+1. σ is the activation function, applied element-wise to the matrix. C^(l) represents the node feature matrix of the l-th layer of the GCN, with dimensions N × F_l, where N is the number of nodes and F_l is the feature dimension of the l-th layer. C^(l+1) represents the node feature matrix of the (l+1)-th layer of the GCN, with dimensions N × F_l+1, where F_l+1 is the feature dimension of the (l+1)-th layer.





 §.§ Prediction

In the prediction stage, we introduce Elastic Weight Consolidation (EWC) regularization <cit.> into the generic cross-entropy loss function. By incorporating the importance of parameters and their association with the loss function, the EWC algorithm can effectively retain knowledge from previous tasks and reduce interference when learning new tasks, thereby addressing the problem of catastrophic forgetting in task learning.

First, we calculate the importance of parameters based on the training results of previous tasks, using the Fisher Information Matrix I_A. The Fisher Information Matrix is calculated as follows:

    I_A = 𝔼[∂^2 ℒ(θ)/∂θ^2|_θ^*] = 𝔼[(∂ℒ(θ)/∂θ∂ℒ(θ)/∂θ^⊤) |_θ^*]

where I_A is the Fisher Information Matrix representing the importance of parameters; 𝔼 is the expectation operator; ℒ(θ) is the cross-entropy loss function, representing the model's CE loss under parameters θ, with L_CE = ℒ(θ); θ are the model parameters; θ^* are the optimal parameters obtained from previous task training; ∂^2 ℒ(θ)/∂θ^2 is the second derivative of the loss function with respect to the parameters, indicating curvature information; ∂ℒ(θ)/∂θ is the first derivative of the loss function with respect to the parameters, indicating gradient information.

The EWC loss is defined as:

    L_ewc = ℒ_ℬ(θ) + λ/2∑_i I_A(i) (θ_i - θ_i^*)^2

where ℒ_ℬ(θ) is the loss function for the current task B; λ is a hyperparameter that balances the learning of new tasks and the retention of old tasks; I_A(i) are the diagonal elements of the Fisher Information Matrix, representing the importance of parameter θ_i in task A; θ_i^* are the parameters learned in task A. The model, regularized by EWC, is used for the final prediction, improving performance on new tasks while mitigating the forgetting problem.

The total loss is the sum of the ordinary cross-entropy loss L_CE and the EWC loss L_ewc:

    L_total = L_CE + L_ewc






§ EXPERIMENT



 §.§ Experiment Settings

We validated our model on three downstream classification tasks: Oxford Pets <cit.>, Flowers102 <cit.>, and Food101 <cit.>. All these datasets belong to the fine-grained classification category. The Oxford Pets dataset contains 37 categories (25 dog breeds and 12 cat breeds) with a total of 7,349 images. The Flowers102 dataset includes 102 categories with a total of 8,189 images. The Food101 dataset consists of 101 food categories with a total of 101,000 images. These datasets are not only rich in categories but also possess high fine-grained characteristics, making them ideal for evaluating the model's performance in distinguishing similar categories.



  §.§.§ Implementation Details

We use the LlaVA <cit.> model to generate general text descriptions corresponding to the images, ensuring that these descriptions do not mention the category names for final classification. Unless otherwise stated, we use the pre-trained backbone ViT-B/16 <cit.> as the visual encoder to produce visual features. We optimized the model for 100 epochs. During training, we used a batch size of 16 and an Adam optimizer with an initial learning rate of 1 × 10^-3, which decays following a cosine learning rate schedule.



 §.§ Comparisons with State-of-the-Arts




We compared the proposed model with several state-of-the-art parameter-efficient fine-tuning methods, including TaskRes <cit.>, CoOp <cit.>, CLIP-adapter <cit.>, and Tip-Adapter <cit.>, on the Oxford Pets, Flowers102, and Food101 datasets. As shown in Table <ref>, the experimental results demonstrate that our model consistently outperforms previous parameter-efficient fine-tuning models on the average performance of the benchmark datasets. Our model achieved an average performance of 90.03%, outperforming Tip-Adapter by 3.19% and TaskRes by 2.82%. The model's test accuracy improved by 4.45% and 2.92% on the Oxford Pets and Flowers102 datasets, respectively, compared to the state-of-the-art methods. Our model still performed the best on the Food101 dataset. The more significant improvement on the Oxford Pets and Food101 datasets is due to the higher need for multi-modal associations, which are better modeled through GNN. Similarly, Tip-Adapter performs better than other SOTA methods as it combines the strengths of prompts and adapters, introducing task-related prompts into the model to provide more multi-modal associations.







 §.§ Model Efficiency



As shown in Table <ref>, we conducted experiments on the parameter quantities of different methods. The parameter consumption of Tip-Adapter is exceptionally high and is not in the same range as other methods. Among the remaining three models, TaskRes has a higher parameter count than CLIP-Adapter, with CoOp having the least parameter count. Our model has a parameter count of 0.056M, which is less than most models and only slightly higher than CoOp. However, the accuracy of our model significantly surpasses that of CoOp, indicating that our model achieves a good balance between accuracy and parameter count.




When the image encoder and text encoder are not frozen, the parameter size is 195,337,765. When the image encoder and text encoder are frozen, the parameter size is 56,869. Therefore, with only 0.029% of the parameters being trainable, our model can surpass the state-of-the-art (SOTA) models on multiple datasets. Our model ranks second in terms of model efficiency among SOTA models. This is partly because the text descriptions in the CoOp model are generated using fixed handcrafted sentences, which are relatively simple text descriptions. Compared to our model, even though the CoOp model has fewer training parameters, the quality of the textual information is lower, resulting in significantly lower accuracy on two datasets.




 §.§ Ablation Study




As shown in Table <ref>, the baseline is a simple linear layer trained on single-modal features, which are also extracted using a pre-trained model. Without using our foundational modules, the accuracy is only 86.20%. After applying EWC regularization, the accuracy improves by 1.65% to 87.85%. When using multi-modal learning, the performance further increases by 2.23% to 90.08%. The improvement with multi-modal learning is because leveraging two modalities simultaneously provides greater capability compared to using a single modality. Finally, by integrating the complete graph method, the accuracy improves by 2.55% to 92.23%. The performance boost from incorporating the graph is due to its ability to better model the relationships between tokens, thereby demonstrating the effectiveness of our proposed method.




The text descriptions for each image are generated by MLLM. In the baseline experiments, replacing all text descriptions uniformly with "A photo of a pet/flower" eliminates text information during training. Ablation experiments show that different combinations of methods and features enhance the model's performance to varying degrees. The GA-Net model achieves the best performance by combining EWC<cit.>, multi-modal features, and GNN. Introducing EWC regularization improves model performance by 1.65%; introducing GNN enhances performance by 2.55%; and incorporating multi-modal learning boosts performance by 2.23%. The significant performance improvements from multi-modal learning and GNN indicate that our model can effectively capture complex associations between modalities. EWC regularization helps mitigating the forgetting problem in task learning, making its effect more pronounced with larger datasets.

In terms of memory cost, Tip-Adapter has the lowest memory consumption, while CoOp has the highest. This is contrary to the parameter usage, indicating that, to some extent, a greater number of parameters can lead to lower memory consumption. Although our model also incurs a significant memory cost, it is still less than CoOp and performs better. While Tip-Adapter consumes less memory than our model, our model requires far fewer parameters and delivers significantly better performance. Furthermore, our model needs to store adjacency relations between different tokens, which inevitably consumes some memory space. This aspect contributes to the performance of our model. This demonstrates that our model achieves a good balance between accuracy and parameter consumption.





 §.§ Hyperparameter Study




To investigate the impact of hyperparameters, specifically similarity thresholds, we analyzed different similarity thresholds on the OxfordPets and Flowers102 datasets, as shown in Figure <ref>. The results indicate that different datasets are affected differently by varying similarity thresholds. For both datasets, the accuracy increased most significantly within the threshold range of 0.3 to 0.5. Additionally, the accuracy improvement for the Flowers102 dataset was more pronounced within this range compared to the OxfordPets dataset, suggesting that the Flowers102 dataset is more sensitive to adjacency relations in the graph. When the similarity threshold reaches 0.7, the model achieves peak accuracy on both datasets. For thresholds less than 0.7, accuracy shows an increasing trend with rising similarity thresholds. However, once the threshold exceeds 0.7, accuracy changes minimally, but the training burden increases, indicating that a similarity threshold of 0.7 is optimal.






§ CONCLUSIONS


This paper comprehensively reviews the limitations of previous parameter-efficient fine-tuning methods in low-data environments. These methods only model a single modality and lack the utilization of structural knowledge in downstream tasks. Therefore, we propose a novel parameter-efficient fine-tuning model, GA-Net, which extracts knowledge suitable for features from each multi-modal feature node in the graph, resulting in features that have fully learned both textual and image information while considering their adjacency relationships. Experiments on three fine-grained classification tasks, Oxford Pets, Flowers102, and Food101, demonstrate that the GA-Net model is effective in parameter-efficient fine-tuning.

The limitations of the model stem from the generation of text descriptions. In this paper, we use MLLM to generate text descriptions for each image. However, these prompts are simple and lack sufficient diversity. We believe that providing more diverse and accurate prompts for downstream tasks, such as using refined image caption methods, would better model the textual structural knowledge and further improve the performance of GA-Net.




§ DECLARATIONS




    
  * Funding
    
    This research received no external funding.

    
  * Conflict of Interest/Competing Interests
    
    We declare that we have no competing interests.

    
  * Ethics Approval and Consent to Participate
    
    This study did not involve any human or animal subjects, hence ethical approval and consent to participate are not applicable.

    
  * Consent for Publication
    
    We have reviewed the manuscript and consent to its publication.

    
  * Data Availability
    
    The datasets used in this study are all publicly available:
    
        
  * Oxford Pets dataset: <https://www.robots.ox.ac.uk/ vgg/data/pets/>
        
  * Flowers-102 dataset: <https://www.robots.ox.ac.uk/ vgg/data/flowers/102/102flowers.tgz>
        
  * Food101 dataset: <https://www.kaggle.com/datasets/dansbecker/food-101>
    

    
  * Materials Availability
    
    Not applicable.

    
  * Code Availability
    
    The code used in this study is available at: <https://github.com/yunche0/GA-Net/tree/master>

    
  * Author Contributions
    
    We all contributed significantly to the research and manuscript preparation, including the conception and design of the study, data collection and analysis, and manuscript writing. We have read and approved the final version of the manuscript.




























 Paper Title: Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLM.



On-the-fly Point Feature Representation for Point Clouds Analysis
    Yang Xulei
    August 5, 2024
=================================================================




§ ABSTRACT 
Automatic furniture layout is long desired for convenient interior design.
Leveraging the remarkable visual reasoning capabilities of multimodal large language models (MLLMs), recent methods address layout generation in a static manner, lacking the feedback-driven refinement essential for interactive user engagement.
We introduce Chat2Layout, a novel interactive furniture layout generation system that extends the functionality of MLLMs into the realm of interactive layout design.
To achieve this, we establish a unified vision-question paradigm for in-context learning, enabling seamless communication with MLLMs to steer their behavior without altering model weights. Within this framework, we present a novel training-free visual prompting mechanism. This involves a visual-text prompting technique that assist MLLMs in reasoning about plausible layout plans, followed by an Offline-to-Online search (O2O-Search) method, which automatically identifies the minimal set of informative references to provide exemplars for visual-text prompting. By employing an agent system with MLLMs as the core controller, we enable bidirectional interaction. The agent not only comprehends the 3D environment and user requirements through linguistic and visual perception but also plans tasks and reasons about actions to generate and arrange furniture within the virtual space. Furthermore, the agent iteratively updates based on visual feedback from execution results. Experimental results demonstrate that our approach facilitates language-interactive generation and arrangement for diverse and complex 3D furniture.






§ INTRODUCTION




3D furniture layout generation plays a crucial role in various applications such as interior design, game development, and virtual reality. Traditional methods for furniture layout planning often formulate the task as a constrained optimization problem <cit.>. However, these approaches typically require rules for scene formation and graph annotations, necessitating the expertise of professional artists. This makes these methods less accessible to non-professionals and inflexible to dynamic environments where user preferences and spatial constraints may vary. 

Recent learning-based approaches address these limitations by employing
neural networks <cit.>, 
to automate the object selection and placement.
Learning from large datasets, these generative methods can produce diverse plausible layouts <cit.>. 
However, despite the promise, they face difficulties in accommodating to objects that are not presented in the training set, limiting their adaptability and versatility.

The rapid advancement of large language models (LLMs) has opened up new possibilities for enhancing user interactions in furniture layout generation. Researchers leverage LLMs to reason about layout plans based on text descriptions of furniture items, including their positions and dimensions <cit.>. These approaches enable users to express their desired living spaces in natural language, eliminating the need for extensive training on large datasets.
However, these methods suffer from notable limitations. While some of these methods <cit.> utilize visual information to refine placement, they still primarily rely on textual input without fully incorporating visual perception. As a result, they often produce plausible yet impractical layouts that fail to align with user expectations or spatial constraints.
Furthermore, the absence of agent memory and feedback mechanisms prevents multi-turn conversations, hindering users' ability to interact with the process to iteratively refine the generated layout plan.


Recent developments have highlighted two significant trends in the field: 
First, Multimodal LLMs (MLLMs) <cit.> have attracted huge attention for their impressive capabilities in various vision-language tasks by introducing visual perceptions into LLMs. 
Second, LLM agents <cit.> emerge as powerful problem-solvers for handling complex tasks through feedback mechanisms and iterative refinement.
Inspired by these advancements, we aim to develop a MLLM agent specifically designed for generating furniture layouts.


In this paper, we introduce Chat2Layout, a language-interactive method for generating furniture layouts that employs a MLLM as its agent. To begin, we establish a unified
vision-question paradigm for in-context learning, which standardizes communication with the MLLM to guide its reasoning utilizing both textual and visual information without the need to update model weights.
Within this framework, we present a novel training-free visual prompting mechanism composed of two key components:
1) We develop a visual-text prompting technique that assists the MLLM in tackling specific layout tasks;
2) We propose an Offline-to-Online search (O2O-Search) method to automatically identify the minimal support set from an example database, facilitating efficient in-context learning from a limited number of references.


Building upon these techniques, we develop a MLLM agent system that automatically perceives 3D indoor environments through both linguistic and visual modalities. The agent is capable of understanding user requirements, planning tasks, reasoning about the necessary actions to generate and arrange furniture within a virtual environment, and learning from the visual feedback of its executed results.
By leveraging our sophisticated unified vision-question paradigm, which incorporates both a visual-text prompting technique and an O2O-Search method, in conjunction with the MLLM agent system, our Chat2Layout enables the execution of diverse, complex, and language-interactive 3D furniture generation and placement. This enables new user experiences that are previously unsupported, setting a new milestone in the field of furniture layout generation.

Our main contributions can be summarized as follows:


  * A MLLM agent system that enables language-interactive 3D furniture generation and layout. This system supports a multi-turn conversations, allowing users to interact dynamically with the 3D environment and iteratively refine the layouts.

  * A unified vision-question paradigm for MLLM to effectively respond to a wide range of tasks. This paradigm incorporates a visual-text prompting technique that facilitates grid-based placement for layout generation, and a O2O-Search method that significantly boosts MLLM's reasoning capabilities in generating furniture plans.

  * Comprehensive support for various furniture layout applications, including layout completion, rearrangement, open-set placement, and multi-conventional interaction, as shown in Figure <ref>.





§ RELATED WORK



Furniture Layout Generation.
Traditional optimization-based layout generation <cit.> relies on prior knowledge of reasonable configurations, such as procedural modeling or pre-defined scene graphs. Such priors require professional expertise, and are less flexible in dynamic environments. 





Generative methods address these challenges with CNN <cit.>, 
VAE <cit.>,
GCN <cit.>,
transformer <cit.>, or diffusion <cit.> architectures trained on large-scale datasets <cit.>. 
InstructScene <cit.> handles concrete layout instructions by training a generative model on a scene-instruction dataset, incorporating a semantic graph prior and a diffusion decoder.
However, these methods struggle with placing objects not included in the training data. In contrast, supports open-set furniture placement with required furniture pieces automatically generated.

LLMs offer a new sight of text-based layout reasoning, bypassing the dataset limitations.
LayoutGPT <cit.> enables LLMs to deliberate over layout plans with bounding boxes and orientation of furniture items.
AnyHome <cit.> enhances LayoutGPT with manually defined placement rules via text instructions.
While AnyHome utilizes visual information, it primarily relies on textual input for placement and only uses visual cues for minor adjustments. Without proper visual understanding, these methods can produce plausible but impractical results. Additionally, the absence of agent memory and feedback mechanisms prevents multi-turn conversations, hindering users from interacting with the process to iteratively refine the generated layout plan.




LLMs as Agents.
With their potential to be powerful general problem solvers, LLMs have been used as the core controller for agents <cit.>. 
Recently, MLLMs <cit.> demonstrate even more impressive performance 
by incorporating visual perception.
Yet, MLLMs as agents has not been fully explored. We are the first to design such an agent system for furniture layout generation, facilitating continuous layout arrangements.


In-context Learning.
ICL integrates task demonstrations into prompts to enhance the reasoning ability of LLMs,
such as Chain-of-Thought (CoT) <cit.> and automatic prompt design <cit.>. Recently, visual prompting has been used in vision tasks, such as overlaying masks with a red circle <cit.>, highlighting regions <cit.>, using multiple circles with arrows <cit.>, and labeling objects with alphabet numbers <cit.>. 
Other techniques select representative exemplars from a candidate set for prompting. These reference exemplars are defined as support set.
Offline methods calculate pairwise similarities among the dataset for selecting the top-k exemplars <cit.>,
while online methods calculate the similarity between the test prompt and candidates to select exemplars <cit.>.  
These methods have not been fully explored for visual tasks with MLLMs like ours.

Text-to-3D Generation.
For layout visualization, 
current methods often pre-select or retrieve furniture objects from a 3D dataset,
limiting the ability to meet diverse user expectations.
Instead, our method uses text-to-3D generation to create furniture items.
Text-to-3D generations <cit.> create 3D content like mesh <cit.>, neural radiance field <cit.>, or gaussian splatting <cit.> from text prompts in an optimization <cit.> or generative manner <cit.>.
Compared to generative methods, recent optimization-based approaches offer broader diversity with fast-optimization processes <cit.>. 
In this work, we adopt Tripo3D <cit.>, which is capable of producing high-quality 3D furniture swiftly.




§ OVERVIEW OF OUR AGENT






 §.§ Classical Agent System


An agent system typically comprises four main components <cit.>:
Environment: The context within which the agent operates, such as web engines or mobile apps, defining the state space of the agent. The agent's actions can modify the environment, thereby influencing its own decision-making processes.
Perception: The agent's sensory component, collecting information from the environment through various modalities like vision and audio. The perceived input is transformed into neural signals and transmitted to the brain for further processing.
Brain: The central processing unit of the agent, responsible for storing knowledge and memories, as well as performing essential functions such as information processing and decision-making. It enables the agent to reason and plan, handle unforeseen tasks, and exhibit intelligent behaviors.
Action: The component that receives and executes action sequences from the brain, once the decisions are made, allowing the agent to interact with the environment.







 §.§ MLLM-Based Layout Agent


Chat2Layout draws inspiration from existing research on LLMs as agents, but it distinguishes itself by uniquely integrating an additional visual modality.
To create such an agent system specialized for interactive layout generation, we implement specific modifications to the agent framework (Figure <ref>):
Environment: Our environment includes a user and an observer. The user provides text-based requirements, while the observer monitors the 3D user interface to summarize visual elements and furniture attributes like positions and dimensions. The observer also engages in self-reflection, prompting the agent to review initial layouts and request corrections.
Perception: The agent perceives both textual and visual information from the environment, encompassing user requirements, furniture attributes, and visual room scenes.
Brain: Based on the perceived information, the agent reasons about the necessary 3D actions for layout plan generation.
Action: Once the 3D action list is determined, these actions are executed within the 3D visualization engine, bringing the layout plan to life.





 §.§ Brain


Our core techniques are centered around the Brain module, the central decision-making component of our agent. Its input includes a specific user task and scene information perceived by the Perception module, while its output is a sequence of 3D actions that can be executed by the Action module. Every interaction with the MLLM adheres to our vision-question paradigm.
The Brain first decomposes each task into a series of atomic tasks. Then, for each atomic task, it performs visual-text prompting to formulate prompts incorporating references generated by O2O Search for engaging with the MLLM. Subsequently, it receives 3D actions, which are then passed on to the Action module for execution.

We give definitions of the most important terms of Brain below, and will describe its key technical components in following sections.

Task. 
A task is defined by user requirements described in text. supports open-vocabulary and unrestricted user descriptions, whether abstract or concrete, complex or simple, mixed or focused. This flexibility leverages the commonsense knowledge inherent in LLMs to guide and control the design generation process.

Atomic Task. 
An atomic task is the most basic unit of a task that cannot be further decomposed. We identify three specific types of atomic tasks: Addtext, Removetext, and Placement(translate, scale, rotate)text.


3D Action. 
We define five fundamental actions within the 3D engine to manipulate objects:

    
  * `Addobjkey, text': Invokes the text-to-3D API `Text-2-3Dtext' to generate a textured mesh from a text description, then assigns an object key `objkey' to this mesh.
    
  * Removeobjkey: Removes the object identified by `objkey' from the scene.
    
  * Translateobjkey, (x,y,z): Moves an object to a new position specified by the coordinates (x,y,z).
    
  * Scaleobjkey, (x,y,z) - Adjusts the size of an object according to the scale factors.
    
  * Rotateobjkey, angle - Rotates an object around its center by a specified angle.







§ VISION-QUESTION PARADIGM



We observe a similar prompt pattern in each interaction with the MLLM. Therefore, we adopt a unified vision-question paradigm as the template for every interaction, as depicted in Figure <ref>.

In this paradigm, we first assign the MLLM a Role such as “You are an expert in 3D interior design”, which has proven crucial in enhancing the LLMs's performance. For specific tasks involving mathematical reasoning, such as object scale prediction and 3D placement, we add addition constraints as “You are an expert in 3D interior design with a strong math background”.


We incorporate Environment information, includes textual descriptions like object names (represented by `objkey'), positions, and dimensions of visualized objects, along with multi-view visual captures of the user interface.


We introduce a visual-text prompting method that formulating each specific Question with visual and textual information to aid the MLLM in reasoning about plausible layout plans.


Additionally, to improve MLLM's accuracy through in-context learning (ICL) from a few contextual examples, we propose an O2O-Search method for selecting the support set as Reference. This approach automatically identifies the minimal informative support set required for effective in-context learning.


By combining visual and textual information, our unified vision-question paradigm enables more accurate interpretations and responses for each Task, as visual cues help disambiguate textual input, leading to improved decision-making.

Exemplar prompts following our vision-question paradigm can be found in our supplementary material.




  
Task Decomposition
Given a user task, before applying visual-text prompting, we need to first decompose it into a sequence of atomic tasks. Figure <ref> illustrates the vision-question paradigm for task decomposition, including the assigned role and the support set provided by O2O-Search. The support set is presented in JSON format, suitable for representing structured data. Note that, as a special case of vision-question paradigm, environment and visual-text prompting are not included in task decomposition as it is a text-only problem. For other interactions discussed in Section <ref>, we adhere to the complete vision-question paradigm.






§ VISUAL-TEXT PROMPTING





 §.§ Object Initialization


We generate furniture items using `Text-2-3Dtext', which come with arbitrary poses and sizes, which are not ideal for placement. Therefore, before incorporating them into a layout, we initialize them with properly aligned poses and scales using visual-text prompting.
In these paradigms, we provide environment information, including visual captures of the scene and object bounding boxes, along with references from our O2O-Search.

Pose Alignment.
First, we calculate the Axis-Aligned Bounding Box (AABB) and the Optimal/Oriented Bounding Box (OBB) <cit.>. 
Then, we align each object's pose by rotating it from its OBB to match its AABB.


Scale Prediction. 
We formulate a prompt to predict the appropriate scaling factor for each object. 
After adjusting the furniture items to their correct sizes, we compile the set of furniture items as F = {f_1, …, f_n}.



 §.§ Object Placement


The generation of a layout can be defined as follows: given a set of furniture items F = {f_1, …, f_n}, where each item f_i is characterized by its identifier Objkey(f_i) and bounding box Rect(f_i), the goal is to determine a placement 
P = {(x_1, y_1, z_1, θ_1), …, (x_n, y_n, z_n, θ_n)} for all items. Here, (x_i, y_i, z_i) and θ_i denote the position and orientation of the item f_i, respectively.
Unlike previous works that primarily focus on placing objects on the floor, 
our approach allows for arbitrary placement on or under other objects, or on the wall. We categorize F into three placement types, F_floor, F_wall, and F_other, through interaction with the MLLM agent, following the vision-question paradigm.
Instead of directly predicting the layout plan in a single interaction like LayoutGPT <cit.> and AnyHome <cit.>, we employ Chain-of-Thought (CoT) prompting to systematically reasons through the plan step by step, which leads to a more precise process.


Floor Placement. 
We first arrange items in F_floor using the Grid Coverage Algorithm in Figure <ref>. We overlay a grid on the floor, where each cell is identified by a unique alphanumeric ID. Assuming that walls are along the top and left edges of the floor, we label wall-adjacent cells as `wall'. We then ask the agent to specify the grid cells that each furniture item in F_floor should cover.
Since directly assigning cells for a large set of items can be challenging, we propose a local by local placement method (Figure <ref>-(a)), which stages the larger items before smaller ones, leading to simplified tasks and improved performance.
For very large objects that can cause inaccurate coverage predictions, we propose a scale-adaptive grid placement method, which initially place them on a sparse grid to minimize the number of covered cells (Figure <ref>-(b)-left) and then project them onto a denser grid for further placement of additional items (Figure <ref>-(b)-right).

3D Placement.
After placing F_floor, we apply the same grid coverage strategy to F_wall by similarly overlaying a grid on the wall. In addition, for all items in F_floor that are adjacent to the wall, we project them to the wall grid before placing F_wall to avoid collision between these two types of items.
We then apply the same strategy to place F_other, such as putting a laptop on top of a table. In this way, our layout generation method is highly versatile and adaptable to irregular floor plans, with or without walls.






Orientation Prediction.
After placing all furniture items, we further correct their orientations. The aforementioned pose alignment technique ensures that the frontal face of each item aligns with one of four views: left, right, front, or back. The agent only needs to identify the frontal face from the four candidate views (Figure <ref>).
After that, we mark the orientation of the frontal face on the grid with a red arrow (Figure <ref>). Finally, another chat session is then used to determine the rotation angle for the final face orientation.






 §.§ Self-Reflection


Self-reflection empowers agents to iteratively improve by refining decisions and correcting past errors <cit.>. To this end, we incorporate a visual self-reflection mechanism into our agent system. After all furniture items are placed, we start a continuous chat by asking whether the result meets user requirements, and is reasonable and realistic, allowing the agent to visually inspect the arrangement within the 3D interface. If the placement does not satisfy user requirements or violates realistic and rational constraints, the agent is prompted to adjust the layout accordingly. Furthermore, we integrate surface collision detection during object placement. This is particularly important for irregularly shaped objects, as it helps prevent collisions or floating objects. 




§ OFFLINE-TO-ONLINE REFERENCE SEARCH



To provide references for the visual-question paradigm, we propose an Offline-to-Online Search method (O2O-Search) to identify the minimal informative support set.
 
Given a database containing N examples D = { (X_i, Y_i) }_i=1^N, where each input set X_i = (u_i, t_i, v_i) consists of a user text prompt u_i, an indoor scene text description t_i, and visual captures of the scene v_i, and Y_i represents the corresponding output result, all in JSON format. Our O2O-Search method aims to automatically identify the minimal informative support set within this dataset.

Offline Grouping. 
We observe that each example D_i=(X_i, Y_i) can be categorized based on its characteristics. To achieve this, we assign a category identifier to each example in D, effectively dividing it into distinct groups. Within each group, we measure the similarity between any two examples (D_i, D_j) using the following criteria:

    f(D_i,D_j)=D_CLIP(u_i,u_j)+α D_CLIP(t_i,t_j)+β D_DINO(v_i,v_j),

where D_CLIP is the CLIP cosine similarity <cit.> and D_DINO is the DINOv2 cosine similarity <cit.>.
The information entropy of an example D_i is defined as:

    E(D_i)=∑_j=0,j≠ i^Nf(D_i,D_j).

We calculate the information entropy for each group and select the top-K demonstrations with the highest information entropy values within each group.

Online Retrieval. 
Given a test input X_t during user interaction, we ask the agent to assign a category identifier. We then select the group corresponding to this identifier as the final supporting set, providing the most relevant reference examples for the current task.




§ EXPERIMENTS



In this section, we detail the evaluation setup, present ablation studies and comparisons to baselines, and showcase diverse results generated by out method. For further implementation details and additional evaluations, please refer to our supplementary materials.




 §.§ Evaluation Setup



To evaluate the quality of our layout generation, we first collected a set of various real scenes online and wrote corresponding text descriptions for each. We then generated the dataset by producing 10 layout plans for each scene, similar to cases 1, 2, and 7 in Figure <ref>. We then conduct a comparative analysis using the following metrics:


Out of Bound Rate (OOB). This measures the percentage of layout plans where objects extend beyond the room's boundaries or intersect with other objects, indicating the quality of spatial arrangement.
We report OOB at the layout plan level, 
checking for the presence of these issues with each plan, following LayoutGPT and AnyHome.

Orientation Correctness (ORI). This evaluates the correctness of object orientations within the layout context.
Correct orientation is defined as the object being easily accessible and usable by users in the room.
For example, a wardrobe adjacent to but facing a wall is considered incorrectly oriented.
We manually check each layout plan and report the percentage of layout plans where furniture has no unreasonable face orientations.

CLIP Similarity (CLIP-Sim). This assesses the alignment between the user's text description and the scene content by calculating the text-to-image CLIP similarity, using multi-view renderings of the 3D layout scene.
Random viewpoint perturbations are incorporated to enhance the robustness of our evaluations.




 §.§ Ablation Studies


























Visual-Text Prompting.
In Table <ref>, we evaluate the effectiveness of visual-text prompting. Results demonstrate that visual-text prompting significantly enhances layout generation compared to the baseline of Ours-w/o-VTP, which relies solely on text descriptions. This is evidenced by a lower OBB, better ORI, and higher CLIP-Sim scores.

We also observe a marginal improvement in the baseline of Ours-Visual, which directly integrates visual captures of the environment without visual-text prompting, compared to Ours-w/o-VTP. This suggests that while visual information is beneficial, its inclusion alone without structural integration does not lead to substantial performance gains. In contrast, our visual-text prompting technique enables more effective utilization of visual information, contributing to better overall performance.

Furthermore, our full method incorporating the self-reflection mechanism further enhances layout generation performance compared to the baseline of Ours-w/o-SR by providing feedback to verify and refine results. 


O2O-Search.
We also conduct an ablation study to assess the effectiveness of our O2O-Search method. We randomly select 30% of the questions from our example dataset D to form the test set. Our method is compared against two baselines: 1) An offline-only method that calculates pairwise similarity across the entire dataset and selects the top-k exemplars without grouping <cit.>; and 2) An online-only method that computes similarity between the test prompt and all candidates in the dataset to select supporting references <cit.>.

We present the accuracy across various sizes of the support set k in Figure <ref>. When k=0, meaning no support set, the accuracy remains significantly low, underscoring the important role that support references play. As we increase the value of k, there is a general trend of improved accuracy for all baselines;. However, this improvement begin to consistently plateau around k = 8, suggesting diminishing returns on accuracy gains for larger k. This indicates that there is an optimal range of k that maximizes accuracy without incurring too much computational cost due to excessive size of references. Thus, we use k=8 in our method.
Importantly, our O2O-Search method consistently outperforms both the offline and online baselines in accuracy across all values of k, demonstrating its superior scalability and effectiveness in utilizing larger support sets to boost performance.

We also observe a substantial degradation in OBB, ORI, and CLIP-Sim metrics when O2O-Search is removed, as shown in Table <ref>. This further highlights the curcial role that our O2O-Search plays in enhancing the overall performance of our system.



 §.§ Comparisons



We compare with three existing methods: InstructScene <cit.>, LayoutGPT <cit.>, and AnyHome <cit.>, with AnyHome being a concurrent work. 
For a fair comparison, we use the same furniture set generated by our model, with identical scale values and initial orientations, for layoutGPT and AnyHome. 
Moreover, for InstructScene, we provide a detailed description of our generated layout plan to obtain their results.
It should be noted that in our method, 
the furniture is generated using a text-to-3D API, while layoutGPT and AnyHome require searching a large dataset. But here We use the same set generated by ours for a fair layout quality comparison.


Quantitative Comparison.
Table <ref> provides the quantitative comparison. 
InstructScene demonstrates weaker performance in terms of ORI and CLIP-Sim compared to all LLM-based methods. 
LayoutGPT, while solely leveraging an online-only support set without visual input, struggles to produce satisfactory results.
AnyHome attempts to address the issues of LayoutGPT by incorporating a set of manually defined placement rules in texts.
However, it only uses visual information for furniture position and pose refinement through SDS optimization, limiting its overall effectiveness.
In contrast, achieve the best results on all metrics, thanks to the integration of visual input and a more advanced support set search method.

Qualitative Comparison.
Figure <ref> presents the qualitative comparison.
Compared to our method, other approaches are more susceptible to issues like out-of-bound placements, incorrect orientations, and poor content alignments.
Specifically, InstructScene is limited to concrete text instructions detailing spatial relations and may generate non-existent objects due to text-to-graph conversion errors. 
Both InstructScene and LayoutGPT lack support for wall placements such as paintings.
AnyHome, while outperforming the other two, relies on pre-defined human-knowledge rules, leading to illogical placements, such as neglecting common-sense spacing between objects.
In addition, all three methods are prone to boundary and orientation issues, impacting usability and aesthetics.
In contrast, demonstrates superior performance by fully exploiting visual information. 
Moreover, supports multi-turn conventional interactions, as demonstrated in Figure <ref>, a feature absent in other methods.



 §.§ More Visual Results



We present our interactive layout generation results in Figure <ref>, a feature unsupported by other methods. empowers users to create layouts from scratch, transforming an empty space into diverse designs such as an ancient Chinese-style bedroom or a modern minimalist living room. Users can also perform partial rearrangements, as illustrated in the washing room scenario. supports precise 3D placements, such as positioning a vase on a table in case one or hanging a painting on a wall in case two. Additionally, users can modify layouts through instructions, including inserting new items, removing objects, and adjusting placements in terms of scale, rotation, and translation. Notably, excels at handling open-vocabulary instructions, accommodating both abstract and concrete descriptions, whether mixed or focused.


 
supports diverse layout generation by prompting the agent to provide multiple layouts for the same set of furniture items. As illustrated in Figure <ref>, we show three cases, each with four distinct layout plans. enables the creation of varied layout plans, accommodating objects on both floors and walls, while ensuring that the orientations of objects are not only diverse but also logical within the context of the room and its functionality.


Figure <ref> illustrates the versatility and adaptability of our method in handling a variety of layout configurations, including irregular floor shapes and multi-room setups. For instance, we employ an irregular grid to generate a layout for an L-shaped game room. In multi-room scenarios, we sequentially generate each individual room, demonstrating the system's capability in managing complex spatial arrangements across multiple interconnected spaces.






§ CONCLUSION


In this paper, we introduce Chat2Layout, a novel method to indoor layout generation that expands the capabilities of MLLMs beyond vision interpretation and text generation. Our method establishes a unified vision-question paradigm for in-context learning, enabling communication with the MLLM to guide its reasoning with both textual and visual information. Within this framework, we introduce a training-free visual prompting technique that facilitates the creation of realistic and contextually appropriate layouts. In addition, we employ an O2O-Search method to identify the minimal support set as references for effective prompting. Overall, our method interprets and handles user requirements and 3D environments both linguistically and visually and execute challenging furniture arrangement tasks. This enables diverse and complex 3D layout generation, providing interactive user experiences that surpass previous methods.


















We provide supplementary material for Chat2Layout, covering a summary of related furniture layout generation works,
implementation details, prompts for floor placement, and pose alignment. 
Additionally, we provide a user study and discuss our user interface.
Lastly, we discuss the limitations of our method.




§ SUMMARY OF LAYOUT GENERATION WORKS










Table <ref> provides a summary of optimized-based methods <cit.>, learning-based methods <cit.>, and LLM-based methods  <cit.> on furniture layout generation.
In comparison, supports a wider range of applications than these baselines.
Specifically, supports layout completion, layout rearrangement, text instruction, 3D placement (onto or under others, or on walls), open-set placement, and multi-conventional interaction.
Notably, supporting open-set placement means that is not restricted to predefined furniture objects nor reliant on retrieving items from a 3D furniture dataset. Moreover, our agent system facilitates continuous user interaction through multi-run conversations, allowing users to iteratively refine the layout plan until desired outcome is reached. 
It should be noted that AnyHome supports only single-turn conversations for layout adjustments without memorization, whereas our method supports multi-turn interactions through the agent, favoring long-term and complex interactions.




§ IMPLEMENTATION DETAILS



We employ GPT-4 vision <cit.> as the MLLM to implement our agent. We develop the pipeline in Python and implement user-interface/visualization in Open3D. The system is deployed on Windows/Linux workstations without GPUs.
To ensure the creation of satisfactory and style-consistent 3D meshes from a text prompts, we generate 16 candidates using `Text-2-3Dtext' and have the agent select the most suitable one.
We also incorporate style descriptions (if applicable) into `Text-2-3Dtext' to maintain stylistic coherence. For instance, in a Gothic-style living room, if a user requests adding a chair, `text' in `Text-2-3Dtext' would specify `a Gothic-style chair' rather than simply `a chair'.
For support set searching, we compile databases containing 50 to 100 candidate examples per question. In Equation 1, we set α=0.3 and β=0.2 to prioritize textual similarity while integrating textual and visual environmental information. In addition, we utilize the `clip-vit-base-patch16' model to calculate CLIP similarity. 
In our O2O-Search algorithm, the example is categorized by characteristics. For instance, `0’ indicates a removal task and `1’ an abstract addition task, and we have 8 categories in total for task decomposition.



§ POSE ALIGNMENT








Before arranging the layout, we generate a set of required furniture items using `Text-2-3Dtext'. However, the initial poses of these objects may be arbitrary. To address this, we 
perform pose alignment, as illustrated in Figure <ref>.
We begin by calculating both the Axis-Aligned Bounding Box (AABB) and the Optimal/Oriented Bounding Box (OBB) <cit.> for each object. By comparing the AABB and OBB, we determine the pose difference between them. Then, we rotate the object based on the calculated pose difference, effectively aligning the AABB and OBB to achieve a more suitable initial pose for the object within the scene.




§ PROMPTS FOR FLOOR PLACEMENT







Figure <ref> illustrates the exemplar prompts used for object placement on the floor. This vision-text prompting approach, employed by our floor placement algorithm, adheres to the vision-question paradigm, consisting of a role, environment information, visual-text prompts, and a reference set provided by our O2O-Search. All other prompts in our system follow a similar structure, but with variations in text descriptions and visual captures specific to their respective tasks.
Leveraging the memory capabilities of LLMs, our agent enables users to ask follow-up questions such as “Provide more results" or request layout completion, as demonstrated in the bottom part of Figure <ref>.







§ USER INTERFACE





We design a user-friendly interface using Open3D, as shown in Figure <ref>. Users can easily input text instructions and initiate an interaction with our agent system by clicking the button. The generated visual results are displayed in the left widget, providing a feedback on the layout design. Our interface also offers controls for adjusting the viewpoint and lighting, allowing users to optimize the visualization for better understanding of the 3D layout. For a more comprehensive demonstration of our interface, please refer to our accompanying video demo.




§ USER STUDY









We conduct a user study, illustrated in Figure <ref>, comparing AnyHome, LayoutGPT, and InstructScene with Chat2Layout. Participants are presented with 20 questions, each accompanied by text instructions and four layout options. They are asked to select the option that best: 1) matches the given instructions; and 2) presents a reasonable and realistic layout plan, free from unreasonable collisions or boundary exceedances. After collecting 41 valid responses, we analyze the user preference rates for both instruction matching and layout quality. According to the results, our approach demonstrate superior performance, surpassing the other methods in user preference across both criteria.

We have conducted another user study to evaluate the interaction in Table <ref>. We invited six non-professional participants to create scenes using our system. Initially, we provided a 10-minute tutorial. Participants were then asked to create 5 predefined scenes and 5 scenes of their choice. Each participant then completed a questionnaire using the five-point NASA Task Load Index (NASA-TLX, 1=very low to 5=very high) <cit.> to evaluate usability and perceived workload. 
In Table 2, the mental demand was low, indicating that users need little effort when using our system. While our system is not perfect in terms of efficiency due to the unstable GPT-4 API, it performs satisfactorily with a high performance value and demonstrates satisfying user satisfaction with a low frustration value.







§ LIMITATIONS






While our agent system facilitates a wider range of user interactions and generates more reasonable layout plans than existing methods, it still faces several challenges, as shown in Figure <ref>.


Firstly, when tasked with arranging a large number of furniture pieces within a severely limited space, collision issue becomes evident. This red rectangle in the left figure highlights how spatial constraints can lead to suboptimal furniture placement and overlapping elements. This occurs because when the available space is insufficient for the size and quantity of objects, it poses a challenge for our agent system to devise a reasonable layout plan. In future iterations, implementing a checking mechanism and user reminders will be crucial to effectively address this issue.





Secondly, our system's reliance on existing text-to-3D generation methods presents another challenge. Currently, these methods struggle to produce a harmonized set of furniture items that adhere to a consistent style when given a series of text instructions. While we have partially mitigated this issue by generating multiple candidate items and allowing the agent to select the most suitable one, the problem is not entirely resolved, as occasionally one or more items may not fit seamlessly within the scene.
For example, as shown in Figure <ref>-right, the generated wardrobe lacks stylistic consistency with the other items.
It would be a valuable advancement to develop a text-to-3D method capable of producing a cohesive set of style-conscious assets, which we intend to explore in the future.


Finally, efficiency remains a limitation of our system. Generating a scene from scratch currently takes around one minute due to the interaction time with the GPT-4 vision API and the generation of a set of candidate 3D furniture—the exact time depends on the complexity of the generated scene. The instability of the GPT-4 vision API sometimes induces errors due to unstable network and negatively impacts user experience. We believe that efficiency can be significantly improved in the future with the development of more efficient large language models (LLMs).








 Paper Title: Embedding Space Selection for Detecting Memorization and Fingerprinting in Generative Models.




Different behaviour of the gas-phase and stellar metallicity in the central part of MaNGA galaxies
    
        I. A. Zinchenko<ref>,<ref> J. M. Vílchez<ref>
       
    
==================================================================================================



§ ABSTRACT 
In the rapidly evolving landscape of artificial intelligence, generative models such as Generative Adversarial Networks (GANs) and Diffusion Models have become cornerstone technologies, driving innovation in diverse fields from art creation to healthcare. 
Despite their potential, these models face the significant challenge of data memorization, which poses risks to privacy and the integrity of generated content. 
Among various metrics of memorization detection, our study delves into the memorization scores calculated from encoder layer embeddings, which involves measuring distances between samples in the embedding spaces. 
Particularly, we find that the memorization scores calculated from layer embeddings of Vision Transformers (ViTs) show an notable trend - the latter (deeper) the layer, the less the memorization measured. It has been found that the memorization scores from the early layers' embeddings are more sensitive to low-level memorization (e.g. colors and simple patterns for an image), while those from the latter layers are more sensitive to high-level memorization (e.g. semantic meaning of an image). We also observe that, for a specific model architecture, its degree of memorization on different levels of information is unique. It can be viewed as an inherent property of the architecture.
Building upon this insight, we introduce a unique fingerprinting methodology. This method capitalizes on the unique distributions of the memorization score across different layers of ViTs, providing a novel approach to identifying models involved in generating deepfakes and malicious content. 
Our approach demonstrates a marked 30% enhancement in identification accuracy over existing baseline methods, offering a more effective tool for combating digital misinformation.





§ INTRODUCTION


In the rapidly evolving field of artificial intelligence, generative models such as Generative Adversarial Networks (GANs) <cit.> and Diffusion Models <cit.> have emerged as pivotal tools, driving innovation across a myriad of domains including art <cit.>, media synthesis <cit.>, healthcare <cit.>, and autonomous systems <cit.>.
These models possess the remarkable capability to generate new, synthetic data instances that are often indistinguishable from real data, thereby unlocking new frontiers in data augmentation <cit.>, realistic content creation <cit.>, etc.
However, alongside their remarkable capabilities, these models harbor the intrinsic challenge of data memorization.

Data memorization occurs when models inadvertently replicate exact or near-exact pieces of their training data <cit.>. It presents significant concerns relating to privacy breaches, the integrity of generated content, and the perpetuation of biases ingrained in the training datasets.
The nuanced issue of memorization within generative models necessitates robust mechanisms for detecting and quantifying memorization. This mechanism can not only aid in understanding how models learn and replicate data patterns <cit.> but also in implementing safeguards against misuse, such as unauthorized replication of sensitive information <cit.>. 
Currently, the practice of measuring memorization primarily involves analyzing the embeddings generated from various layers of neural networks, such as Convolutional Neural Networks (CNNs) <cit.> and Transformers <cit.>. 
However, the decision regarding which layer's embeddings should be employed for memorization measurement has not been formally standardized or thoroughly explored. 
Common practice often gravitates towards utilizing the embeddings from the penultimate layer of these networks. The lack of justification to the layer choice underscores a pressing gap in the existing methodologies, highlighting the need for a systematic exploration to identify the most effective layers for capturing memorization phenomena accurately. This paper aims to bridge this gap by conducting a comprehensive investigation into the efficacy of memorization measurement using embeddings from different layers of CNNs and transformers. We hope to establish informed guidelines for embedding layer choice, thereby enhancing the ability to correctly quantify the extent of data memorization using the existing methods of measurement.

In this research we study C_T-score, an embedding-based data-copying metric proposed in <cit.>, to measure model memorization with ViT layer embeddings. Beyond merely selecting the best layers for memorization detection, we introduce and validate a novel approach for model fingerprinting that leverages the unique distribution of the C_T-scores layer-wise. The idea comes from the experiment observation, that each model shows a distinctly unique trend in C_T-scores calculated by embeddings from different layers.
This approach is grounded in the understanding that a model's memorization characteristics are uniquely shaped by its architecture, the data it is trained on, and the optimization techniques used during training. It demonstrates superior performance relative to existing baseline methods in the task of model identification by generated images. This again highlights the vast unexplored possibility of layer-wise study of memorization scores.



§ RELATED WORKS

Research in data memorization has shown that deep neural networks can memorize random labels due to having significantly more learnable parameters than training examples, indicating a potential to prioritize memorization over generalization <cit.>. 
This observation is complemented by findings from <cit.>, who demonstrated that standard regularization strategies are often inadequate in preventing such memorization. 
Furthermore, <cit.> applied a geometric analysis method to quantify the extent of memorization, and discovered the preferentially learning for different layers.

In the realm of generative models, particular attention has been directed towards Generative Adversarial Networks (GANs) and Diffusion Models. <cit.> explored methods to reduce memorization in GANs without compromising output quality. 
Concurrently, <cit.> investigated strategies to mitigate data copying effects in Diffusion Models. 
Assessment techniques for generative models have also evolved, incorporating metrics like the popular Inception Score (IS), Frechét Inception Distance (FID) <cit.>, Precision and Recall test <cit.>, and Self Supervised Copy Detection (SSCD) <cit.>.


Model fingerprinting aims to create a unique identifier for a machine learning model by analyzing its performance, structure, and behavior to distinguish it from other models. Studies involving model fingerprinting to trace the origins of digital artifacts have grown <cit.>. Such efforts are crucial for understanding the source of generative outputs and are instrumental in the battle against the misuse of AI technologies, such as the production of deepfakes. These studies lay the groundwork for exploring more refined mechanisms to detect and mitigate memorization and to ensure the ethical use of generative models in producing digital content.




§ OBSERVATION

Our research uses C_T-score as the metric of model memorization. To calculate C_T-scores, generated samples are projected into an embedding space, which is often obtained from neural network encoders. Both CNN-based and transformer-based encoders are used in our initial experiment. The variance of layer-wise C_T-score trends resulting from the selection of encoder types is noted. More importantly, an interesting trend is consistently observed when the encoder is transformer-based.









 §.§ What is C_T Score


The definition of C_T score depends on the concept of data-copying. 
<cit.> defines that a generative model is data-copying the training set T when in some regions, the model's output distribution Q is systematically closer to T than the true underlying distribution P, which Q aims to learn. C_T-score is designed as a metric to assess the degree of data-copying by sampling and examining the generated data in relation to the training dataset (T) and test datasets (which is seen as samples from P). A higher C_T score indicates a lower degree of data-copying and shows the model Q's differentiation from T; whereas a lower score indicates a higher degree of data-copying and Q's lack of originality. 


Initially, embeddings for the training, test, and generated datasets are computed using pre-trained models, providing high-dimensional representations that capture the essence of the data's features. 

The embeddings are clustered using the KMeans algorithm, dividing the data embeddings into distinct cells that form a partition Π of the instance space 𝒳. This step is crucial for assessing data-copying behavior not only globally on the entire 𝒳 but also locally within each cell.

Within each cell π, samples are drawn from the test and generated data embeddings. A distance function d: 𝒳→ℝ is needed here to measure how close a sample is to the training set. For all the experiments in this paper, the nearest neighbor distance d(x) = min_t ∈ T ||x-t||^2 is adopted. The samples P_n (drawn from the test dataset) and Q_m (drawn from the generated dataset) are passed into function d and result in L_π(Q_m) and L_π(P_n), which are considered as samples from the distance distribution. Notice that the distance function d serves as indicators of similarity, with shorter distances suggesting higher levels of similarity and potential copying.
Next, a Z_U score is determined using the Mann-Whitney U test by comparing L_π(Q_m) and L_π(P_n) and quantifying their statistical difference. A higher Z_u score indicates a greater disparity, implying less similarity and, consequently, less copying. Finally, the C_T score is calculated as a weighted average of the Z_U scores across all cells. The weighting is based on the proportion of the samples in each cell, and cells with an insufficient number of samples are excluded based on a predetermined threshold, ensuring the score only reflects cells with a meaningful representation of generated data. In short, the C_t score can be formulated as


    C_T(P_n, Q_m) = ∑_π∈Π_τ P_n(π) Z_U(L_π(P_n), L_π(Q_m); T)/∑_π∈Π_τ P_n(π)


where P_n(π) denotes the proportion and Π_τ denotes the set of cells with the number of samples above the threshold τ. 




 §.§ Notable trend for ViT models


Using the C_T-score formula provided, we conduct initial experiments to calculate layer-wise C_T-scores for encoders based on both transformer and CNN architectures.  Our findings reveal a unique trend observed in transformer encoders. C_T-scores are consistently increasing for latter layers, indicating lower memorization. 
This phenomenon is especially pronounced in the ViT architecture (as shown in Figure <ref>, <ref>, <ref> with different ViT implementations). It appears to be a characteristic feature of transformer-based encoders, suggesting a distinct pattern of information memorization associated with the layer hierarchy.

In contrast,  CNN-based encoders demonstrate a markedly different behavior. 
Our observations in Figure <ref> indicate that CNNs exhibit a relatively consistent or flat C_T-score trend across their layers. 
This uniformity in C_T-score distribution suggests that CNN encoders process and represent data in a more homogeneous manner, distinguishing them as stable representers of information when compared to their transformer counterparts.
This stability suggests that, in the context of CNN encoders, the choice of layer for embedding extraction might not significantly impact memorization detection, echoing practices seen in other methodologies like Fréchet Inception Distance (FID) <cit.> where models like InceptionV3 <cit.> are preferred for image embeddings extraction due to their consistency.

However, the intriguing results from the ViT models offer a contrasting perspective. 
Our observation of increasing C_T-scores across the layers of transformer encoders prompts a compelling hypothesis that different layers of the ViT model specialize in distinct levels of feature learning. In the context of images, the early layers of ViT concentrate on encoding low-level features, such as simple patterns, colors, and textures. In contrast, the latter layers focus on encoding high-level, more abstract features, such as object categories and complex scene relationships. This hypothesis necessitates a layer-specific analysis on using ViT encoders for embedding extractions, which is key to understanding how the choice of layers influences the memorization detection task.




 §.§ Low level and high level memorization



We hypothesize that generative models exhibit distinct patterns of memorization at both high and low levels of information, which are crucial for their functioning and output generation.
At the low level, these models fall into memorizing basic patterns, features, and textures. 
This form of memorization represents a limited understanding to elemental aspects such as edges, colors, and simple shapes. 
On the other hand, high-level memorization involves the integration and synthesis of these basic elements into more complex and abstract concepts. 
In visual models, this could mean recognizing and generating faces or landscapes. 
This high-level memorization is not merely a sum of low-level features but represents a more sophisticated semantic understanding and recombination.

In light of the intriguing trend of increasing C_T-scores observed in Vision Transformer (ViT) models, our study sought to establish a connection between this trend and the concepts of high-level and low-level memorization. We hypothesize that different layers within the ViT model capture distinct types of memorization representation.
Specifically, the embeddings from the initial (front/early) layers would primarily capture low-level memorization, focusing on fundamental features and patterns. The embeddings from the deeper (latter) layers would lean towards high-level memorization, characterized by the integration and synthesis of these basic elements into more complex and abstract representations.



§ EXPERIMENT


We design an experiment to measure the C_T-score trends using a Vision Transformer (ViT) encoder. Five datasets are created from the original CIFAR-10 using various augmentation techniques, which specifically perturb or preserve high-level or low-level information in the images. We expect to see a perturbation reflected in the early or latter parts of the C_T-Layer curve when the augmentation targets the low-level or high-level information, respectively. We use a BigGAN-deep model as the baseline model and the C_T-Layer curve calculated from its generation as the baseline curve to be compared against.




 §.§ Dataset preparation






We curate five distinct datasets (examples: Figure <ref>), which are augmented versions of CIFAR-10 to delve into the intricacies of low-level and high-level data processing mechanisms. Each modified version of the CIFAR-10 dataset (Figs. <ref>-<ref>) is treated as an output from “pathological generative models,” memorizing either low-level or high-level information disproportionately. Our objective is to ascertain whether modifications to either low-level or high-level features precipitate discernible alterations in the C_T-Layer curve, indicating the effect of such modifications. 
The deliberate modification of the CIFAR-10 dataset results in a high average dissimilarity between the augmented and original datasets. 
Consequently, the C_T-scores predominantly indicate underfitting (high positive values). Nevertheless, the relative trend observed in the C_T-Layer curve remains insightful as we are only interested in seeing shifts from the baseline consistently-increasing trend.

For the analysis of low-level features, we construct two datasets (Figs.<ref>-<ref>). We aim for these two datasets to imitate the generation of models that primarily alter low-level details while preserving high-level semantics. The first dataset (<ref>) is generated through the application of rotation-based modifications to CIFAR-10 images, incorporating a spectrum of angular orientations. 
The second dataset (<ref>) is constructed from the process of downsampling the original images followed by upsampling them back to their original dimensions.

Although the first two datasets intend to only perturb the low-level semantics, the processes of rotation and down-upsampling still corrupt the objects in the images to a degree, affecting the high-level information. To address this issue, the subsequent two datasets are crafted to imitate the generation of models that extract and preserve the training image's high-level semantics. In our exploration of such data processing techniques, we employ a fine-tuned ResNet-18 model (details of the fine-tuning process are provided in Appendix <ref>) to conduct image segmentation on images from the CIFAR-10 dataset.
This segmentation isolates the foreground elements of the images, which are then seamlessly integrated onto alternative backgrounds.
Specifically, the modification entails replacing the original image backgrounds with Gaussian noise <ref> and real-world landscapes <ref>, sourced from the BG-20k <cit.> dataset. 
This strategic modification is designed to ensure that the fundamental semantics of the image—primarily its main object—remain largely unaffected.

Following the four datasets that preserve high-level information while altering low-level details, we construct an additional dataset (<ref>) that largely preserves the low-level details while significantly alters the high-level information. This dataset is generated by fragmenting each original CIFAR-10 image into 16 equal-sized square patches and shuffling them. Each image in this dataset has the exact collection of pixels, though arranged differently, as its original counterpart. This can be viewed as an intentional preservation of the low-level image features since color and pattern details within each piece are exactly the same as in the original.





 §.§ C_T-Scores Analysis


We employ a non-pathological generative model (specifically,  BigGAN-deep model in Fig.<ref> ) as a baseline model, and use a ViT-base model as the encoder to derive the C_T-layer curves for all six models (Fig.<ref>). The observed shifts of the augmented dataset C_T-layer trends from the baseline curve closely align with our hypothesis in <ref> that embeddings from initial layers capture low-level memorization, while those the from deeper layers exhibit high-level memorization.


For datasets augmented with low-level detail modifications – rotation and resolution changes (<ref>, <ref>) – we observe a significant shift in their C_T-layer trends at the initial layers, compared with the baseline increasing trend (<ref>). The C_T-layer curves for both rotation and down-upsampling display an early peak at the initial layers. Since these two datasets are more dissimilar to the original CIFAR-10 in terms of low-level details,
this observation underscores the sensitivity of the network's earlier layers to alterations in low-level image details.

On the other hand, the analysis of high-level detail preservation, achieved through the alteration of image backgrounds while preserving the foreground elements, reveals the latter network layers' focus on high-level semantic information. There are discernible drops of the relative C_T-scores on the last layer of the network for the two segmentation methods (<ref>, <ref>), which both intend to cause overfitting on high-level information by preserving the semantics of original CIFAR-10.

In the analysis of the C_T-scores curve for the shuffled dataset (<ref>), the observed deviation is relatively subtle, attributable to the inherently increasing trend of the baseline reference. Consequently, even an enhancement in the scores of the deeper layers aligns with this prevailing trend. Notably, the stability in the score of the front layers, despite significant alterations to high-level details, substantiates the hypothesis that the deeper layers are chiefly tasked with the extraction of high-level semantic information from the input images.



 §.§ Further Experiment

Our first experiment analyzes layer-wise C_T-scores trends on reflecting the memorization of low-level and high-level information. It supports our hypothesis in <ref> on the difference in memorization detection between different layers of encoder embeddings. It heavily relies on the assumption that all the specially augmented images are samples from some imaginary “pathological generative models.” In contrast, the next experiment uses samples from real trained generative models of two specific architectures - denoising diffusion probabilistic models (DDPM) <cit.> and denoising diffusion implicit models (DDIM) <cit.>. DDPM and and DDIM are both Diffusion Models that generate data distributions from noises by adding noise to data over time steps and then learning to reverse the process. DDPM follows strict Markovian assumption at inference time, while DDIM follows a deterministic non-Markovian formulation. We're interested in seeing if layer-wise C_T-scores can capture the memorization characteristics of the two models. 

For robustness purpose, multiple checkpoints are taken at several epochs for both DDPM and DDIM (details of the training are provided in Appendix <ref>). Only checkpoints at epochs after the loss function largely converges are recorded to ensure the image generation is meaningful. In this case, epoch number 120, 150, 180, 210, 240, and 270 are used. The layer-wise C_T-Layer scores curves <ref> show a similar increasing trends for both architectures on all checkpoints. This meets our expectation because it's unlikely that the added number of training epochs will change the model memorization natures drastically. The distinction between DDPM and DDIM architectures might not be prominent due to the fact that they adopt very similar diffusion methods. 

However, the subtle divergence between the overall DDPM trend (orange curves) and DDIM trend (blue curves) is still noticeable. This makes it intriguing to compare the intra-architecture (e.g. DDPM180 and DDPM210) and inter-architecture (e.g. DDPM180 and DDIM180) C_T-Layer curve similarities. By treating each C_T-Layer curve as a vector in ℝ^11, where 11 is the number of encoder layers, the cosine similarity between each pair of C_T-Layer curves is computed and recorded in a heat map <ref>. The overall intra-architecture similarity is higher than the inter-architecture similarity, even if the raw curves in <ref> look fairly similar. This experiment result suggests that the memorization characteristics are unique to model architectures, not to specific training checkpoints.






§ FINGERPRINTING

Our findings indicate that the C_T-Layer curve serves as a meaningful indicator of the model's handling of different levels of details, suggesting that this curve could act as a distinctive feature reflective of the model. This insight paves the way for utilizing the C_T-Layer curve as a technique for fingerprinting generative models, an application growing in importance within the field as people are more and more worried about deepfake and malicious use of generated media.



 §.§ Introduction to Model Fingerprinting Using C_T-Layer Curves


Traditional methods of fingerprinting generated images often rely on access to the processed training images <cit.> utilized by the model during its training phase, a requirement that is not always practical or feasible. In contrast, our proposed methodology offers a novel approach to extract characteristic features of a given model architecture by analyzing its corresponding C_T-Layer scores <ref>. For the purpose of this study, we employ the ViT-base model as our encoder; However, the methodology is adaptable and could potentially be extended to other variations of ViT encoders.



In more detail, our method involves calculating the C_T score for each layer of the target image set. Subsequently, we employ these scores in conjunction with L2-norm and cosine-similarity nearest neighbor analysis to identify the closest match within our dataset. This dataset comprises C_T-Layer scores for a diverse array of known models, enabling the identification and matching process. Through this approach, we introduce a robust mechanism for model fingerprinting that bypasses the need for direct access to a model's training images, thereby offering a more versatile and accessible method for characterizing and identifying generative models.



 §.§ Experimental Setup for Evaluating Fingerprinting Accuracy


To empirically validate our proposed fingerprinting methodology, we conduct a preliminary experiment encompassing a diverse set of 18 model samples derived from various generative model architectures. The architectures are ContraGAN, BigGAN, and SNGAN for GANs and DDPM, DDIM, and PNDM for Diffusion Models. For each architecture, we have three model samples, each generating 500 images, resulting in a total of 1,500 generated images per architecture.

To establish a comparative framework, we implement a baseline method that utilizes the mean, standard deviation (std), and Fréchet Inception Distance (FID) score to extract features from the generated images and perform nearest-neighbor matching. Additionally, we compare our approach with several neural network classification methods, including a vanilla CNN model trained from scratch, a fine-tuned ViT-base model, and a fine-tuned ResNet50 model (details of the training and fine-tuning process are provided in Appendix <ref>).

Our experimental setup is meticulously crafted to evaluate the accuracy of our method. From the 1,500 generated images for each model architecture, we first separate 500 images as the testing set. The remaining 1,000 images are then used to form two training datasets (Datasets_small with 500 images and Datasets_full with the entire 1,000 images) to create ablation studies. For our C_T method, we formulate the training-set C_T-scores for each model using its corresponding training images. At testing time, for each target, we calculate its C_T-score using the testing set and identify the best match in the training-set C_T-scores with different matching methods. The overall accuracy is determined by the proportion of correctly matched target models. For the baseline method, the overall process mirrors that of the C_T method, but we use baseline statistical scores instead. 

We train different neural networks on training sets and evaluate their accuracy on the testing set. We use majority votes for final classification since each model takes a single image as the input, while our C_T and baseline methods are applied to the entire testing dataset. By employing this systematic approach, we aim to provide a thorough and robust assessment of our proposed method's performance relative to both statistics-based and neural network-based techniques.





 §.§ Results and Performance Analysis


The results outlined in Table <ref> underscore the effectiveness of our proposed C_T-layer methodology. Compared to the baseline methods, our approach offers a significant improvement in performance, demonstrating its capability to capture and leverage architectural nuances for model fingerprinting. The C_T-Layer Method records a remarkable increase in accuracy for both the full and small sets, with an outstanding 100% accuracy in the full set and an 83.3% accuracy in the small set using the combination of L2 Norm and Cosine Similarity. This performance significantly outstrips the baseline methods, which peak at 66.6% and 83.3% accuracy for similar conditions. 

Additionally, our methodology outperforms conventional models such as Vanilla CNN and ViT-base, and it achieves comparable results to ResNet50, while without the need for training. Although our method employs a ViT-base model as the encoder, direct finetuning a ViT-base model to learn fingerprinting classification yields suboptimal results due to limited data availability — a common challenge in fingerprinting scenarios.

Moreover, we evaluate the training duration of our method against other models. The baseline method is exceptionally quick due to its simplistic statistical nature. However, our approach demonstrates comparable processing times to ResNet50 and is faster than the ViT-base model, underscoring its efficiency.

These results suggest that the C_T-layer scores, by leveraging layer-specific responses to various image details, provide an effective method for model identification and characterization. This improvement is critical in scenarios where precision and reliability in model fingerprinting are paramount. That the C_T-Layer Method outperforms the established models confirms its efficacy and potential for generative model fingerprinting.



§ CONCLUSION


This study pioneers an approach to understanding data memorization in generative models, providing a detailed analysis of the impact of layer selection in Convolutional Neural Networks (CNNs) and Transformers on memorization detection. It reveals distinct memorization patterns between CNNs and Vision Transformers (ViTs), underscoring the importance of layer-specific analysis for accurate memorization quantification. Furthermore, the introduction of a novel model fingerprinting technique leveraging C_T-score distributions marks a significant advancement in identifying generative models, offering potential tools for the ethical use of generative models, and addressing critical concerns around privacy, content integrity, and the proliferation of deepfakes.



  
Limitations and future directions

While our proposed technique offers significant advancements, several practical limitations and challenges need to be considered for real-world applications.

The choice of layer for memorization detection varies depending on the type of memorization (low-level vs. high-level), which can be challenging to distinguish without extensive analysis. Future work should develop hybrid detection methods that analyze both low-level and high-level features and automate layer selection algorithms for better accuracy. Although our fingerprinting method does not require direct access to the model's training dataset, it still necessitates a general, compact, and sufficiently generic dataset to serve as a practical baseline. Future directions should include curating such datasets and implementing dynamic updates to include new and emerging generative model characteristics.

Our method requires pre-computed C_T-Layer score data for various generative model architectures. Future efforts should focus on expanding this database to include new models and establishing collaborations with developers for early access. The computational complexity of computing C_T-Layer scores for each layer can be intensive. Future research should aim to optimize computation processes and leverage high-performance computing resources to improve efficiency and scalability.

Implementing our fingerprinting technique requires careful consideration of ethical and privacy implications. Establishing clear guidelines and ethical standards for deployment, ensuring data privacy, and developing protocols for transparency and accountability will be crucial. Collaborations with ethicists, policymakers, and industry stakeholders will help align technological advancements with societal values and norms.


tmlr



§ APPENDIX






 §.§ Fine-Tuning the ResNet-18 Model for Image Segmentation



This section provides details on the fine-tuning process of the ResNet-18 model for image segmentation tasks on the CIFAR-10 dataset.

We start with a pre-trained ResNet-18 model from PyTorch <cit.> and modify its architecture to suit the image segmentation task based on <cit.>'s implementation. The first convolutional layer is replaced to adapt to CIFAR-10 image dimensions. The final fully connected layers are removed, and a bilinear upsampling layer is added to increase the resolution of feature maps. Additionally, a convolutional layer for classification with 10 output channels (corresponding to the CIFAR-10 classes) is appended, followed by another bilinear upsampling layer to match the output size to the input size. The CIFAR-10 dataset is used for both training and testing. For training, the images are transformed using random cropping, horizontal flipping, normalization, and conversion to tensors. For testing, the images are normalized and converted to tensors without any augmentation.

The training procedure involves setting up the loss function, optimizer, and learning rate scheduler. The model is trained for 30 epochs with the following details:


    
  * Loss Function: Binary Cross-Entropy with Logits Loss (BCEWithLogitsLoss)
    
  * Optimizer: Stochastic Gradient Descent (SGD) with a learning rate of 0.05, momentum of 0.9, and weight decay of 1e-4
    
  * Learning Rate Scheduler: Cosine Annealing with a minimum learning rate of 0.001
    
  * Batch Size: 128 for training, 100 for testing
    
  * Number of Epochs: 30
    
  * Number of Workers: 16 for data loading


The training and evaluation of the model are conducted on a machine equipped with an NVIDIA GeForce GTX 1080 Ti GPU. This computational environment facilitates the efficient processing of the CIFAR-10 dataset and the fine-tuning of the ResNet-18 model.

During training, both training and validation loss and accuracy are monitored to evaluate the model's performance. These metrics are plotted to visualize the model's learning progress over the epochs. The segmentation results are also visualized by converting the segmented outputs to images, highlighting the model's ability to segment different classes within the CIFAR-10 dataset.





 §.§ Training DDIM and DDPM Models



This section details training DDIM and DDPM models and collecting their checkpoints at multiple epochs.

The DDPM model implementation follows the original DDPM paper <cit.>. We use the official training and inference code posted by the authors on the Hugging Face website. The DDIM implementation follows a GitHub Repository: <cit.>. Table <ref> summarizes the details of training and inference configurations of the two methods.



 §.§ Baseline Neural Network Models for Fingerprinting Performance Comparison



This section provides details on the baseline neural network models used for performance comparison in the context of fingerprinting tasks. Specifically, we describe the training of a vanilla Convolutional Neural Network (CNN) model, a fine-tuned ResNet-50 model, and a fine-tuned Vision Transformer (ViT) model from PyTorch <cit.>.

The table below summarizes the architecture, loss function, optimizer, and other training settings for the three models:



The performance of the models is evaluated using accuracy metrics. During training, the average loss and accuracy per epoch are monitored. After training, a majority vote mechanism is used for evaluation on the test dataset. Predictions for each label are collected, and the most common prediction is considered as the final prediction for that label. The accuracy is calculated based on the correct majority votes, providing insight into the models' classification capabilities for the fingerprinting task.



