Paper Title: Knowledge-Guided Prompt Learning for Lifespan Brain MR Image Segmentation.



 
Brain MRI Segmentation with Knowledge-Guided Prompt Learning 



L. Teng et al. 


School of Biomedical Engineering & State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai 201210, China 
Shanghai United Imaging Intelligence Co., Ltd. Shanghai 200230, China 
feng.shi@uii-ai.com 
 
Shanghai Clinical Research and Trial Center, Shanghai 201210, China 
 
dgshen@shanghaitech.edu.cn 

Knowledge-Guided Prompt Learning for Lifespan Brain MR Image Segmentation
    Lin Teng 1 Zihao Zhao 1 Jiawei Huang 1 Zehong Cao 2 Runqi Meng 1 
Feng Shi 2^() 
Dinggang Shen 1,2,3^()
    August 5, 2024
===========================================================================================================



§ ABSTRACT  
Automatic and accurate segmentation of brain MR images throughout the human lifespan into tissue and structure is crucial for understanding brain development and diagnosing diseases. However, challenges arise from the intricate variations in brain appearance due to rapid early brain development, aging, and disorders, compounded by the limited availability of manually-labeled datasets. 
In response, we present a two-step segmentation framework employing Knowledge-Guided Prompt Learning (KGPL) for brain MRI.  
Specifically, we first pre-train segmentation models on large-scale datasets with sub-optimal labels, followed by the incorporation of knowledge-driven embeddings learned from image-text alignment into the models. The introduction of knowledge-wise prompts captures semantic relationships between anatomical variability and biological processes, enabling models to learn structural feature embeddings across diverse age groups. 
Experimental findings demonstrate the superiority and robustness of our proposed method, particularly noticeable when employing Swin UNETR as the backbone. Our approach achieves average DSC values of 95.17% and 94.19% for brain tissue and structure segmentation, respectively. 
Our code is available at https://github.com/TL9792/KGPLhttps://github.com/TL9792/KGPL.

 
 




§ INTRODUCTION
  
Dense segmentation of brain MR images involves voxel-wise labeling from different brain tissues, i.e., white matter (WM), gray matter (GM), cerebrospinal fluid (CSF), into fine-grained cortical and subcortical subregions, such as 106 structures according to Desikan-Killiany (DK) atlas <cit.> and Schaefer 400-parcels as described in <cit.>. 
Lifespan[Lifespan refers to the period between birth and death, emphasizing the wide age range covered in our study.] brain segmentation is essential for quantitative brain development analysis and neurodegenerative disorders diagnosis <cit.>. Manual annotation is time-consuming and labor-intensive. Hence, an automated brain image segmentation method is urgently needed. 

Several software packages, such as FMRIB Software Library (FSL) <cit.> and FreeSurfer <cit.>, have been developed to automatically segment brain images. However, processing each subject using these tools costs several hours, and the obtained results often require further manual corrections by experienced experts. Recent advancements in deep learning, such as convolutional neural network (CNN) based U-Net <cit.>, Vision Transformer (ViT) based methods <cit.> like UNETR <cit.>, and Swin Transformer based <cit.> Swin UNETR <cit.>, have shown promise in medical image segmentation. 

Brain morphology is dynamic in the human lifespan, leading to considerable appearance variations in brain images <cit.>. For instance, CSF exhibits gradual expansion, while GM experiences progressive reduction with aging, particularly among elderly individuals with Alzheimer's disease (AD).
However, most previous efforts are established on a single time point, and primarily focus on image-based information while overlooking the complementary textual information that provides essential contextual details and insights into the pathology or anatomy, which often results in lacking generalizability over a range of ages <cit.>. 
Additionally, the performance of learning-based methods is often restricted by the scarcity of annotated datasets, especially when labeled training samples for specific time points are limited in practice. To solve this problem, transfer learning <cit.> and fine-tuning <cit.> strategies have been widely adopted. 
Jia et al. <cit.> proposed an efficient fine-tuning method called Visual Prompt Tuning (VPT), which incorporates a small set of random learnable parameters into the input space. 
Directly applying random parameters representing unknown information may hinder model training <cit.>. 

To tackle the challenges mentioned above, we present a two-step framework with knowledge-guided prompt learning (KGPL) tailored for the dense segmentation of brain MR images across diverse age groups, which are the first to incorporate knowledge-wise prompts into brain segmentation models. Specifically, drawing inspiration from VPT, we replace random embeddings with knowledge-driven embeddings, learned from the pre-trained text encoder in BiomedCLIP[BiomedCLIP indicates Biomedical Contrastive Language-Image Pre-training (CLIP) <cit.>, which is pre-trained on large-scale figure-caption pairs by jointly training an image encoder and a text encoder, being able to well establish the semantic connection between images and language <cit.>.], into the input space of the encoder. The knowledge-wise learnable embeddings exploit the relationship between images and brain structural and developmental-specific biomedical characteristics, aiming to empower models to learn distinguishing representations of brain data acquired from different age phases. 
Experimental results demonstrate that our proposed method outperforms state-of-the-art (SOTA) baselines, and performs robustly in different backbone networks. 




§ METHODOLOGY
  
We aim to develop an automatic and robust approach to segment brain structure, based on brain MRI and specific biomedical attribute information. 
Fig. <ref> illustrates the overall framework. Specifically, we first pre-train segmentation models on large-scale datasets with sub-optimal annotation (Sec. <ref>).  
Subsequently, we refine the pre-trained models on relatively small-scale manually labeled datasets by introducing a few knowledge-driven learnable parameters into the input space, aiming to guide models to extract more crucial features of brain data spanning a large range of age. (Sec. <ref>). 


 




 §.§ Vision Pre-training
  
In this section, we utilize datasets with sub-optimal labels to establish models for segmenting brain MR images into tissue and structure, respectively. Specifically, we select one cutting-edge segmentation model as the backbone. Using the T1-weighted MR image as input, we first train the model to predict the brain tissue segmentation mask, which is supervised by the Dice loss function formulated as Eq. (<ref>). Subsequently, the predicted tissue segmentation results as input to train another model for brain structure parcellation, which is supervised by both Dice and Focal loss functions
as expressed in Eq. (<ref>). 


    Dice Loss = 1 - 2 × |P ∩ G|/|P|+|G|,
 
where P is the prediction result, and G denotes the ground truth.


    Loss = Dice Loss - α× (1 - p_t)^γ×log(p_t),
 
where α and γ are set to 100 and 0.2 based on experience.  p_t is the probability of the predicted result.   




 §.§ Fine-tuning with Knowledge-wise Prompt
   
To utilize specific textual information reflecting brain properties to enhance the discriminative learning of fine-grained brain anatomical structures, we introduce a novel KGPL approach. The procedure is illustrated at the bottom of Fig. <ref>.  




  §.§.§ Knowledge Prompt Generation.
  
Brain morphology, such as cortical thickness, shows a strong correlation with biological attributes like brain age, sex, and disease status <cit.>. This textual information comes from the corresponding data acquisition centers and official websites. 
To leverage this prior knowledge for guiding feature extraction, we initially categorize subjects into age groups every decade, considering that brain morphology remains relatively similar at close ages. 
Then individual attribute details are embedded into a template to generate a descriptive sentence (e.g., "This is a brain magnetic resonance image acquired from a male with mild cognitive impairment at fifty years old"). Finally, this sentence is encoded by a pre-trained text encoder from BiomedCLIP to obtain knowledge embeddings with dimensions (B, N, D), where B is the batch size, N is identical to the number of word tokens, and D is a fixed hidden dimension of 768.   



  §.§.§ Learnable Prompt Pre-initialized by Knowledge Prompt.
 
To effectively integrate prior knowledge into models, we first pre-define multiple learnable tokens (i.e., learnable embeddings) with zero values, matching with the dimension of knowledge embeddings. Then, the learnable embeddings are pre-initialized by knowledge embeddings through addition operation, referring to knowledge-wise learnable embeddings. 



  §.§.§ Model Fine-tuned Guided by Knowledge-wise Prompt.
 
Inspired by VPT, we serve knowledge-wise learnable embeddings as input to deep-level encoder layers, considering knowledge features are considered high-level. 
Specifically, we denote the collection of p knowledge-wise learnable tokens as P = { P_k ∈ℝ^M || k ∈ℕ, 1 ≤ k ≤ p}, where p is the number of word tokens. 
To ensure that they can interact with image embeddings, we reshape them on the hidden dimension. 
For U-Net or Swin UNETR as the backbone, knowledge-wise learnable embeddings are reshaped via the Adaptive Average Pooling (AAP) operator, transitioning from (B, N, D) to (B, N, 1). Following this, they pass through a linear layer with shape (B, C, N), where C is the number of channels; 
For UNETR, knowledge-wise learnable embeddings are first transposed from (B, N, D) to (D, N, B), then processed by a linear layer to match image embeddings' shape, except for the sequence length dimension. Next, they are inverse-transposed to restore the original dimension order. Finally, they are concatenated with image embeddings to calculate information interaction. The shape is (B, C, (N+S)), in which S represents the dot product of the feature map's length, width, and height. 
The procedure can be formulated as:  


    [X_i, _] = L_i-1 ([P_i-1,X_i-1]),
 


    [X_i+1, _] = L_i ([P_i, X_i]),
  
where X_i∈ℝ^B × C × S is image embeddings extracted by the i-1-th encoder layer L_i-1. 
P_i denotes knowledge-wise learnable embeddings at L_i's input space.
During the fine-tuning step, only the image embeddings are directed to the decoder.
Notably, we initialize the model of the target domain using weights learned from the source domain, by updating only the learnable parameters and the decoder while freezing the encoder.




§ EXPERIMENTS AND RESULTS
  


 §.§ Datasets and Metrics
  
Our datasets covering various age stages are collected from multiple repositories, including CBMFM (643 subjects, aged 18-81), Alzheimer’s Disease Neuroimaging Initiative (ADNI) (1285 subjects, aged 20-97) <cit.>, Autism Brain Imaging Data Exchange (ABIDE) (1062 subjects, aged 9-64) <cit.>, and Adolescent Brain Cognitive Development (ABCD) dataset (458 subjects, aged 9-11) <cit.>. Detailed analysis of demographic information can be found in the supplementary material.
Most datasets processed by FreeSurfer have sub-optimal labels. To obtain accurate labels for fine-tuning and model inference, FreeSurfer is initially applied, followed by manual annotations. Each dataset is randomly partitioned into training, validation, and testing sets in an 8:1:1 ratio. 
Each subject has a T1-weighted MR image, brain tissue segmentation mask, and structure segmentation mask (106 regions). T1-weighted MR images are pre-processed by skull stripping, bias field correction <cit.>, and intensity normalization. 
Performance evaluation utilizes two quantitative metrics: Dice Similarity Coefficient (DSC) and Average Surface Distance (ASD). 




 §.§ Implementation Details
 
Experiments are conducted using MONAI <cit.> on a single NVIDIA TITAN RTX GPU with 24GB memory. Training spans 1000 epochs with early stopping, utilizing an AdamW optimizer with a learning rate of 1e-4 and regularization weight of 1e-5. We adopt a linear warmup cosine annealing learning rate scheduler. Data augmentation involves randomly flipping along three axes with a probability of 0.5. Images are cropped to (128,128,128) based on foreground regions and a resolution of 1×1×1 mm^3 using linear interpolation. 




 §.§ Quantitative Comparison Analysis
  


  §.§.§ Effectiveness analysis of backbone architecture.
 
We validate the expansibility of our method based on three representative networks: (1) U-Net, a pure CNN; (2) UNETR, combining CNN and ViT; and (3) Swin UNETR, blending CNN and Swin Transformer. Their original architectures are used in our study, i.e., encoders have four CNN-based or twelve transformer-based blocks, decoders have four or five CNN-based blocks, and skip connections between the encoder and the decoder are included. 

 


Tables <ref> and <ref> present quantitative comparison results of brain tissue and structure segmentation using three backbones, respectively. Due to page constraints, we show partial brain structure segmentation results. The complete comparison results are shown in the supplementary material.
It is evident that our approach can significantly improve the performance using different backbones, especially with Swin UNETR obtaining a high prediction accuracy compared with other backbones. 
This indicates that our method is robust in applying to various network architecture types. 


 



  §.§.§ Comparison with full fine-tuning methods.
 
Quantitative analysis of Tables <ref> and <ref> shows that KGPL using Swin UNETR as the backbone improves average DSC and ASD values by 1.36% and 6.99%, 1.15% and 6.78% for brain tissue and structure segmentation, respectively.
Statistical significance of this improvement is confirmed by a paired t-test (p-value  0.05). Notably, our method achieves optimal performance with minimal trainable parameters, particularly in UNETR, where the parameter count is reduced from 92.78M to 4.46M without sacrificing performance. 
This suggests that KGPL aids models to prioritize essential information for brain image segmentation, thereby improving efficiency and adaptability. 



  §.§.§ Comparison with random-prompts-only methods.
 
Table <ref> of brain tissue segmentation shows that our proposed KGPL using UNETR as the backbone can offer 1.87% and 3.08% improvements of average DSC and ASD, respectively. In Table <ref>, KGPL using Swin UNETR shows improvements of 1.71% and 7.47% in average DSC and ASD for right hippocampus segmentation, respectively. Importantly, our proposed method is computationally faster (3×), i.e., extending the time from 3 days to 1 day. 
This emphasizes that knowledge-wise prompts can direct models to better comprehend and process brain image data with different characteristics and variations, thus promoting efficient convergence.


 



 §.§ Qualitative Comparison Analysis
  
To more intuitively illustrate the superiority of our method, we display qualitative comparison results of brain tissue and structure segmentation. 

 

In Figs. <ref> and <ref>, our KGPL method yields more accurate segmentation across different backbones, with predictions closer to ground truths compared to full or random prompt-only fine-tuning.
For example, Fig. <ref> demonstrates our brain tissue segmentation closely matching the ground truth, particularly in complex brain regions like sulci and gyri. In Fig. <ref>, full fine-tuning backbones exhibit significant errors in regions like the hippocampus and cerebral cortex, while our predictions display greater precision, suggesting potential for aiding in brain disease diagnosis, such as left/right hippocampal sclerosis detection.




§ CONCLUSION
 
This work presents a segmentation framework employing knowledge-guided prompt learning for brain MRI analysis throughout the lifespan. 
It stands as a pioneering effort in utilizing biomedical attribute knowledge as guidance to achieve precise segmentation for brain data. Experimental results indicate that knowledge benefits models in better learning structural variations of brain data across different age stages. 
The introduction of knowledge-wise learnable embeddings facilitates the extraction of fine-grained anatomical features and enhances training efficiency, even with small-scale datasets. 
Hence, the proposed method holds promise for clinical applications such as brain disease diagnosis. 
Moving forward, we plan to extend our methodology to infant phase datasets, enabling segmentation across the entire lifespan continuum. 





  §.§.§ 
 This study was partly funded by National Natural Science Foundation of China (grant numbers 62131015, 62250710165, U23A20295), the STI 2030-Major Projects (No.2022ZD0209000), Shanghai Municipal Central Guided Local Science and Technology Development Fund (grant number YDZX20233100001001), Science and Technology Commission of Shanghai Municipality (STCSM) (grant number 21010502600), and The Key R&D Program of Guangdong Province, China (grant numbers 2023B0303040001, 2021B0101420006).



  §.§.§ 

The authors have no competing interests to declare that are relevant to the content of this article. 



splncs04 
 



 Paper Title: PIXELMOD: Improving Soft Moderation of Visual Misleading Information on Twitter.
















: Improving Soft Moderation of
 Visual Misleading Information on TwitterThis paper is accepted for publication at the 2024 USENIX Security Symposium. Please cite accordingly.
    Pujan Paudel^, Chen Ling^, Jeremy Blackburn^, and Gianluca Stringhini^

^Boston University, ^Binghamton University

{ppaudel,ccling,gian}@bu.edu, jblackbu@binghamton.edu

    
==============================================================================================================================================================================




















§ ABSTRACT 

Images are a powerful and immediate vehicle to carry misleading or outright false messages, yet identifying image-based misinformation at scale poses unique challenges.
In this paper, we present , a system that leverages perceptual hashes, vector databases, and optical character recognition (OCR) to efficiently identify images that are candidates to receive soft moderation labels on Twitter.
We show that outperforms existing image similarity approaches when applied to soft moderation, with negligible performance overhead.
We then test on a dataset of tweets surrounding the 2020 US Presidential Election, and find that it is able to identify visually misleading images that are candidates for soft moderation with 0.99% false detection and 2.06% false negatives.



: Improving Soft Moderation of
 Visual Misleading Information on TwitterThis paper is accepted for publication at the 2024 USENIX Security Symposium. Please cite accordingly.
    Pujan Paudel^, Chen Ling^, Jeremy Blackburn^, and Gianluca Stringhini^

^Boston University, ^Binghamton University

{ppaudel,ccling,gian}@bu.edu, jblackbu@binghamton.edu

    
==============================================================================================================================================================================







§ INTRODUCTION

Online users are constantly bombarded with false and misleading information, whether it is inaccurate and shared without malice (i.e., misinformation), or deliberately crafted to achieve a malicious goal, for example by state actors (i.e., disinformation) <cit.>. 
To help their users better distinguish between accurate and misleading or false information, online platforms like Twitter have introduced soft moderation measures, where they add labels to posts that include inaccurate information, with the goal of providing better context to these claims.
Unfortunately, little is known on how these moderation decisions are made by the platforms, and recent work found glaring issues with Twitter's soft moderation approach <cit.>.

While previous work on automated soft moderation <cit.> builds a solid foundation for the soft moderation of textual content, false information is not only spread through text, but images are commonly used to convey false and misleading narratives <cit.>.
While Twitter applies soft moderation decisions to posts containing images, the way in which this is done is not publicly known, and a preliminary analysis that we conducted shows that these labels are not applied uniformly and pervasively.
For example, in Figure <ref>, we show three tweets that contain identical images discussing the same debunked electoral map claiming Trump's landslide victory with 410 votes after elections servers were seized in Germany by the U.S military <cit.>.
The first tweet (Figure <ref>) received a warning label, while the other two tweets containing the same image (Figures <ref> and <ref>) did not receive any intervention, despite discussing the same misleading narrative. 

These observations highlight the need for effective automated approaches to identify image content that should receive soft moderation.
Moderating images, however, poses unique challenges.
First, the sheer amount of data published on social media makes this a particularly challenging task, especially on platforms like Twitter, where 50 million tweets are posted every day <cit.>.
Previous work has dealt with this problem by adapting perceptual hashing algorithms <cit.>.
These algorithms identify visually similar images by leveraging syntactic embeddings, which are lightweight to calculate.
Even if the calculation of hashes is relatively cheap, comparing an image against a large number of potential candidates is inefficient and does not scale.
In addition to the computational costs of comparing images, previous research has shown that image-based false information is highly contextual, and that similar images can be misinformation, satire, or even completely benign based on the context in which they are used <cit.>.
Existing approaches using perceptual hashing lack the nuance to determine this context, making them prone to false positives and unfit for the misinformation moderation use case.





Technical Roadmap.
In this paper, we aim to address the limitations of previous approaches and develop a scalable, performant, and effective system able to analyze images posted on social media and identify candidates for soft moderation. 
We present , an image search system designed to assist platform moderators by flagging visual content that is similar to content they have previously applied moderation labels to. 
takes as seed input a list of images initially moderated by Twitter and encodes the images using perceptual hashing <cit.>.
To allow efficient hash comparisons, leverages Milvus <cit.>, a vector database that is optimized for searching among millions of hash records.
After finding images that are visually similar, leverages OCR to identify the context in which an image is used, allowing it to rule out false positives.

successfully identifies both tweets in Figures <ref> and <ref> when queried with the image in Figure <ref>.
Additionally, not only identifies images identical to the seed images but also visual variants of the images such as memes and fauxtography, which are commonly used for spreading misleading information <cit.>.


Main Contributions & Findings.
addresses the limitations of previous image-based moderation approaches proposed in the academic literature, 
is platform-independent, and only requires a single misleading image already intervened by human moderators.
With the ability to identify thousands of posts spreading misleading images, we show that an image search system like can be used as a foundation for large-scale soft moderation intervention system for images.
While a large amount of research has been conducted on perceptual hashing and OCR, the novelty of our work lies in combining and tuning the two into a working end-to-end soft moderation system which achieves a much larger coverage than perceptual hashing alone.
Besides content moderation, the takeaways offered by our work can be helpful for researchers and practitioners in building image similarity search systems for related security problems like phishing <cit.> and CSAM detection <cit.>.


We compare with alternative approaches to assess visual similarity, showing that outperforms them in terms of F1-score.
In particular, the use of OCR to determine the context of false images allows to operate using a higher similarity threshold than existing approaches only based on perceptual hashing, increasing recall while keeping precision high as well, suffering only a negligible runtime performance hit in return.

We then test on Twitter data discussing the 2020 US Presidential Election.
Starting from a seed set of 959 tweets containing misleading images, we use to retrieve 40,244 tweets in the wild that are candidates for moderation.
Performing a thorough manual analysis, we find has a false negative rate of 2.06% and a false positive rate of 0.99%.
These findings are a positive step towards building automated systems that complement Twitter's existing systems to moderate misleading images and improve the state of content moderation. 

To better understand the way in which soft moderation was adopted by Twitter during that period, we perform a qualitative measurement study of the content flagged by , comparing it to tweets that received soft moderation labels from Twitter.
We categorize the images identified by based on Twitter's Platform Policy, finding that although different types of platform policies were moderated at different rates by Twitter, none of the policy violations exceeded a moderation rate of 5.96%.








§ OVERVIEW OF 


takes a query image that a platform wants to investigate for further moderation actions and retrieves images that are both visually and contextually similar.
This allows a platform to quickly identify a large, high-certainty candidate list of posts for moderation.
has four stages, as illustrated by Figure <ref>: i) generating image embeddings, ii) indexing image embeddings, iii) retrieving visually similar images through approximate nearest neighbor search, and iv) refining contextually matching images via Optical Character Recognition (OCR).
We demonstrate in the context of Twitter, identifying visually misleading information related to the 2020 US Presidential Election.
However, 's architecture is designed to be portable to any social media platform.



 §.§ Background

Identifying visually similar images given a query image is a reverse image search problem, which falls under the broader research area of Content-Based Image Retrieval systems <cit.>.
The goal is to efficiently collect, index, and search for visually similar images over an index
of millions of images, and there are several publicly available solutions (e.g., TinEye <cit.>, Google Vision AI <cit.>).
We conducted a preliminary exploration of these systems and identified two major limitations: 1) they require a paid subscription for programmatic access, and 2) they perform poorly for returning images from Twitter, Facebook, and other social media platforms because of platform restrictions on their crawlers.
To address these limitations, we built a custom image search system focused exclusively on social media that collects images, generates syntactic embeddings (pHash and PDQHash), and indexes them (using the Milvus <cit.> vector database).

Motivating the need for multi-modality in visual similarity search.
Prior works have discussed that incorporating contextual information within images is important when identifying when they have been manipulated or miscaptioned <cit.>.
Zanettou et al. <cit.> discuss similar existing shortcomings of image similarity systems and the need of incorporating semantic components like OCR in image detection pipelines to effectively identify image memes.
Another work studying misinformation images on WhatsApp in India <cit.> used semantic features like OCR from Google Cloud Vision to identify differences between different types of misinformation images (images taken out of context, photoshopped images, and memes) after retrieving images visually similar to fact-checked images.
However, no existing work uses contextual information as a part of the detection system or pipeline itself.

Existing detection systems can identify images that are very similar to misleading images upon querying, but are unable to verify if the identified images are used in the same misleading context as the query image.
We illustrate this case with an example in Figure <ref>.
When we query existing image similarity systems with Figure <ref>, they will typically return Figure <ref>, and Figure <ref> as matching results with strong confidence.
However, Figure <ref>, and Figure <ref> are not related to the query image under question, which contains the text of “Fraud. The Biggest Disgrace In Our History”, despite being visually similar to the query.
Figure <ref> is an image used in a different context, with no connotation to the narrative of “Election Fraud” during the 2020 Elections, whereas Figure <ref> is used in another context: Fox News projecting a win for Joe Biden.
Thus, the results returned are used in entirely different context than the query image, lacking the connotation of “Election Fraud” which makes the query image misleading.
A similar problem occurs when dealing with fauxtography, where images are manipulated with the inclusion of visual changes, overlay text, or are used out of context with the goal of misleading the user <cit.>.
This highlights the need for incorporating context when using image similarity systems to improve misinformation detection systems.

To address this issue, analyzes the overlay text included in images by using optical character recognition (OCR) to better capture the context in which an image is used and then incorporate this context as a part of our image similarity pipeline.
Including this context also helps overcome existing limitations of prior work using perceptual hashes that have to rely on lower similarity thresholds for matching <cit.>.
Incorporating context allows us to increase image similarity thresholds while at the same time achieving better recall without hurting precision.






 §.§ Generating image embeddings

There are two types of embeddings we can extract from images for retrieval purposes.
The first, syntactic embeddings, are fingerprints of an image's visual features, computed such that two visually similar images will have identical, or very similar fingerprints.
The second, semantic embeddings, capture features within the images (for example, if a given image is a chart or a portrait), and match images that are visually dissimilar, but still have similar meanings.

For our purposes, syntactic embeddings are more useful for two reasons.
First, images identified through them are minor variations of the query image, making them easy to interpret and take action on.
Second, semantic embeddings use complex deep learning architectures <cit.>, while syntactic embeddings are relatively fast and cheap to train and use.
This means we can scale to millions of images with relatively few resources (e.g., does not require a GPU.)

We generate our syntactic embeddings via perceptual hashing.
We make use of two different perceptual hashing embeddings: i) pHash <cit.>, and ii) PDQHash <cit.>.
pHash encodes images as 64-bit vectors by extracting geometry-preserving feature points using Discrete Cosine Transform (DCT).
The resulting vectors are invariant to simple image transformations and minor coloring differences.
PDQHash is an improvement over pHash, encoding images as 256-bit vectors.
Prior work studying the spread of misleading images in social media has used both pHash and PDQHash <cit.>.
We evaluate pHash and PDQHash with respect to precision, recall, as well as the time taken to generate an embedding, index it, and eventually query the index.



 §.§ Indexing image embeddings

The most similar image to a query is the one whose embedding is the shortest distance from the query image's embedding.
While we could perform a pairwise comparison with the query image and all candidate images, this is intractable at the scale at which social media platforms operate.
To address this, there are specialized vector databases that can be used to perform queries over embeddings.
For , we chose to use Milvus <cit.> a purpose-built vector management system.
Milvus abstracts away lower-level details (e.g., index creation and item insertion) via a higher-level API and Facebook's Faiss <cit.> library for efficient dense vector similarity search.

Milvus has two types of indexes: i)  and ii) .
 guarantees a 100% recall rate by exhaustively searching for matches to a query embedding.
 is quantization-based, dividing embeddings into multiple cluster units and comparing the query embedding to the center of each cluster.
 is faster than  but requires tuning hyperparameters to find an optimal recall rate.
Milvus recommends using  for datasets in the order of a few million vectors, which fits the datasets we use to evaluate 



 §.§ Retrieving visually similar images

Vector databases like Milvus scale by leveraging approximate nearest neighbor search across their indices.
Milvus determines the nearest neighbors by scoring candidates with a similarity metric.
We use Hamming distance, leaving us with the next challenge of choosing an appropriate threshold to treat as “visually similar.”
We select Hamming distance over cosine similarity as prior work on perceptual hashing have predominantly used this metric to assess visual similarity on both PHash and PDQHash ( e.g. baseline systems <cit.> and  <cit.>).
Additionally, we index the perceptual hashes on Milvus as binary hashes, and using Hamming distance as the similarity metric allows us to efficiently compare the binary representations of images.
Previous works using pHash to study misinformation on a variety of social media platforms have recommended a Hamming distance of six <cit.>, and eight <cit.>.
The developers of PDQhash suggest a distance of 32, which has been used to study misinformation on WhatsApp <cit.>.
Other work using PDQHash has used a threshold of 40 to study the spread of COVID-19-related media in Pakistan over WhatsApp <cit.>.
Ultimately, choosing a threshold depends on the dataset used and the particulars of how “visual similarity” is defined.
In Section <ref>, we experimentally evaluate the choice of the visual similarity threshold (θ_visual) by optimizing over both precision and recall.




 §.§ Refining matched images using OCR

The final component of involves processing the results retrieved as “visually similar” to the query image by comparing text extracted via an Optical Character Recognition (OCR) engine.
For the rest of the paper, we refer to the output text from an image extracted from an OCR engine as OCR label.
The advantages of applying this OCR driven post-processing to the matched results is two-fold: i) it eliminates the limitations of underlying perceptual hashing algorithms by filtering out their false positives and ii) it allows us to explore relatively larger distance thresholds for visual similarity matching, thus allowing us to minimize false negatives.
Prior applications of perceptual hashing algorithms for identifying visually similar images have used conservative thresholds <cit.> to minimize false positives.
In contrast, we aim to exploit larger thresholds to increase our recall as well as have a more tuneable mechanism with respect to precision trade-offs. 
In Section <ref>, we first validate the choice of Google Cloud Vision API as the underlying OCR engine <cit.>, and experimentally evaluate the calibration of the similarity metric for the extracted OCR labels as our OCR engine) and corresponding threshold for similarity (θ_textual).



§ DATASETS



We use three different datasets to evaluate (summarized in Table <ref>).
Our first dataset, , contains 7.6M tweets collected via the Twitter Streaming API using a set of 2020 US Election hashtags (e.g., #ballotfraud, #voterfraud, #electionfraud, #stopthesteal) curated by Abilov et al. <cit.>.
To comply with Twitter's terms of service, this dataset only provides tweet IDs, therefore, we need to retrieve the full tweet content from the Twitter API through a process called hydration.
Out of the 4,017,259 tweets we were able to rehydrate, about 218K have at least one “media” entity: either an image or a video (tweets with videos also contain a thumbnail image, which we include in our dataset).

Our second dataset, , consists of 13M tweets with media collected via the 1% Twitter stream from November 1 to December 31, 2020.
Unlike , we do not filter any keywords, therefore is a uniform 1% sample of Twitter.

Finally, we collect via Twitter's Academic Research full-archive search endpoint in late 2022, by querying context annotations <cit.> associated with the 2020 US Elections.
Twitter annotates tweets with context annotations by semantically analyzing their content and metadata, categorizing them into a nested structure of domains and entity labels.
The more than 80 domains cover things like politics and TV shows, while the nearly 145K entities cover details of the domains ranging from elections to festivals, to politics and media personalities.

We identify two context annotations related to the 2020 election by retrieving the context annotations of all tweets from : 1) “2020 US Election Day” and 2) “2020 US Presidential Election,” both in the Events domain.
We also found other context annotations from the tweets, but they were associated with “Political figures” and “Politics” in general, not the 2020 elections specifically.
To minimize the noise that can be introduced when using such a broad context, we do not include those annotations when building .
In the end, consists of the 6.9M tweets that we retrieve using the “2020 US Election Day” and “2020 US Presidential Election” context annotations and limiting our search to November 1st, 2020 to December 31st, 2020.

Index building.
We build two different Milvus indices for pHash and PDQHash embeddings, using all the images in our datasets.
The index size for both types of embeddings is 19.7M, after combining all the tweets from , , and .







§ SYSTEM VALIDATION



In this section, we present the validation strategy for .
First, we discuss how we build a ground truth dataset. 
Then, we describe our experimental setting to determine the best hashing mechanism, and the best string similarity algorithm for the OCR labels, and the corresponding θ_visual and θ_textual to use with the hashing algorithms and string similarity algorithms.
The goal is to experimentally determine the best operating values for by optimizing over both precision and recall for the overall system.

Building ground truth.
First, we randomly sample 50 images from .
For each of these 50 images, we query both the pHash and PDQHash indexes for similar images, limiting the results to the maximum threshold for each algorithm (10 and 90).
We select 90 as the maximum threshold for PDQHash as its developers experimentally verified it as the upper bound for images that are known to be different.
For pHash, we select 10 as the maximum threshold because previous work has found that higher thresholds produce results that are too noisy <cit.>.
This results in 11,825 unique images retrieved as similar across both indices for the same set of 50 query images.
Then we manually annotate the results using a pairwise image annotation tool developed by <cit.>.
Note that the goal of this annotation was to verify that images were visually and contextually similar to the original one (i.e., their overlay text contained similar words).
For this reason, we did not need to build a codebook and have multiple annotators agree on the results as we did for more subjective experiments later in the paper.
In the end, we find 9,785 images that are similar to our 50 source images.
We refer to these 9,785 images as .

Determining θ_visual. 
Next, we use the images in to determine the accuracy of the results returned by the two hashing algorithms when using different thresholds for θ_visual.
Our Milvus indexes return the closest set of embeddings for a query image, scored by the Hamming distance to the candidate image.
For pHash, we experiment with a range of Hamming distance thresholds from 4 to 10, which is in line with previous works that found an optimal threshold between 6 <cit.> and 8 <cit.>.
For PDQhash, on the other hand, it is recommended to use a range of pairwise distances for identifying similar images <cit.>.
Therefore, we test multiple threshold ranges, some used by previous work using PDQHash <cit.> (32, 48) as well as three additional ranges to take us up to the maximum possible threshold (64, 80, and 90).

Validating Google Cloud Vision API for OCR.
To validate the ability of Google Cloud Vision API to correctly extract the OCR labels in a query image, 
we sample 50 images from and create a ground truth of the text contained in the images.
These are examples of image occurring “in the wild,” which contain text in specialized fonts, small text in lower quality images, or text in artistic fonts, and are therefore a good test for the OCR component of .
On this dataset, the median Jaccard similarity of the ground truth text and the one extracted by Cloud Vision OCR is 1.0, and the mean is 0.95.
This validates that Google Cloud Vision API can be succesfully used as the underlying engine for identifying the contextual text contained in the images of .
Upon doing some error analysis of the OCR text, we find that the output of the Cloud Vision API is not missing any text present in the images, but is sometimes parsed in different order than the ground truth.
This is an artifact of how the OCR engine works on different regions of the image.
We argue that this would not pose a problem when using the system for as the method will work consistently across all the source images and the potential matches retrieved through perceptual hashing.


Determining θ_textual.
After validating that Cloud Vision API can be succesfully used to extract the contextual text contained in images, we aim to determine the appropriate text similarity metric and corresponding similarity threshold to compare the OCR labels of the retrieved visual matches with the query images.
We first check if the query image contains any OCR label, which we call label_query, using Google Cloud Vision API.
If label_query is not empty, then, for each visually matching image, we compute the corresponding OCR labels (label_match) using the same API endpoint.
We experiment with multiple text similarity metrics, and multiple similarity thresholds to assess if label_match is similar to label_query.
The similarity metrics we experiment with are: i) Normalized Levenshtein similarity, ii) Jaro-Winkler similarity, iii) similarity metric based on Longest Common Subsequence (Metric LCS), and iv) Jaccard-index similarity.
For each similarity metric, we measure F1 scores on the similarity threshold range of 0,0.05, 0.1, 0.15, …, 0.75, 0.8
Note that, for Jaccard index, we experiment with values of n-grams ranging from 1 to 5 as this algorithms convert string into  a set of n-grams when computing the similarity. 

Grid search setup for θ_visual and θ_textual to determine the best operating values for .
To identify the best set of components for , we experimentally determine the best hashing method, the corresponding distance threshold for that hash, similarity metric for the OCR label, and accordingly the similarity threshold for comparing label_query and label_match.
We perform a grid search over these four different components of , scoring the combinations of the components by their corresponding F1-score, evaluated on  .
We present the results of the grid-search experiment in Figure <ref>.
For space reasons, we only present the best performing text similarity method for each θ_visual and the two hashing methods.
We analyze how the F1 score of the embeddings with θ_visual change as we increase the θ_textual.

We can see that the combination of PDQHash with θ_visual of 90, and the OCR component using Jaccard similarity (ngram = 4) with θ_textual of 0.05 produces the best F1 score of 0.980.
This setting yields a precision of 0.990 and recall score of 0.979.
The next best performing metrics are PDQHash with θ_visual of 80 and pHash with θ_visual of 10 using normalized levenshtein similarities.
We observe the immediate returns of expanding the θ_visual to the maximum bound to retrieve as many relevant results as possible, without compromising on the false positives.
We will further evaluate this configuration with previous state of the art image detection methods in Section <ref>.
We also note that we can improve the performance of pHash embedding by using a wider θ_visual of 10, compared to the thresholds of 6 and 8 used in the prior works. 

In the rest of the paper, we set to use an image embedding of PDQHash with a θ_visual of 90 and an OCR post-processing component of Jaccard similarity (ngram = 4) with θ_textual of 0.05. 





§ EVALUATION

In this section, we first compare with various other image similarity systems as baselines, showing that is the best performing approach for content moderation.
Next, we use to identify misleading images during the 2020 US Presidential Election in the wild, and compare its effectiveness with the existing moderation system of Twitter.
Finally, we perform a qualitative characterization of the misleading images by analyzing the images through the lens of Twitter's platform policies in detail.



 §.§ vs. baselines


We compare the performance of against various image similarity systems as a baseline for evaluation.
We use four different families of image similarity systems for comparison.
First, we compare against semantic embeddings that use Deep Neural Networks (DNNs) based methods. 
These include features extracted from final layers of DNN architectures specializing in image classification such as Inception-v3 <cit.> and ResNet <cit.>. 
The features extracted from these methods have been shown to be useful for visual search and similarity tasks <cit.>.
The next family of methods that we compare against is image descriptors, which can be considered as “deeper syntactic hashes” <cit.>.
This set of methods is popular in computer vision tasks such as object recognition, scene recognition, and work by extracting a large number of keypoints in an image.
Matatov et al. <cit.> use ORB descriptors for identifying the spread of visual misinformation to assist journalists.
We use ORB <cit.>, SIFT <cit.>, and DAISY <cit.> as image descriptors.
Another family of method we compare with is the list of prior works that have used perceptual hashing algorithms for identifying images spreading misinformation.
These include Phash and PDQHash with multiple distance thresholds <cit.>.

Finally, we evaluate the performance of with multi-modal embeddings.
While there is a rich body of work and systems existing in the vision-language literature, the majority of these works are trained with an objective of explaining the visual concept contained in the image <cit.>.
On the other hand, OCR texts occur as semantic context to the image itself and can be much longer, noisy, and nuanced.
Thus it is not possible to directly adapt models such as Universal Image Text Representation Learning (UNITER) <cit.>, or Visual-BERT <cit.> that have been trained on vision-language tasks for our problem, in the same way adapting Inception-v3 or ResNet is possible.
However, to assess the potential of a multi-modal embedding that leverages text from an OCR engine, or text from the accompanying tweet with DNN based features, we experiment with “concatenated” vision-language embeddings.
First, we concatenate the embeddings obtained from Inception-V3 (a 2048 dimensional vector) and Sentence-BERT <cit.> (a 768 dimensional vector) to create a joint embedding from the two modalities.

We experimented with two different techniques for unifying the embeddings: i) concatenating, and ii) stacking, further reporting the best results.
While much more sophisticated mechanisms of combining embeddings such as attention mechanisms exist, the dataset size of limits us in evaluating methods that combine embeddings or modalities. 
Fully leveraging multimodal embeddings requires an end-to-end deep learning training setup, and accordingly large scale datasets designed for the problem of soft-moderation, which currently does not exist.

In the same way, we also combine textual and visual embeddings using the CLIP model from Open AI <cit.>.
CLIP consists of a text and an image encoder which encodes textual and visual information into a multimodal embedding space by using contrastive learning.
The model is trained on various text/images pairs from Web sources, and has demonstrated impressive zero-shot capabilities for classification purposes and have been used to detect hateful content on social media <cit.>.
Alongside the OCR text, we also experiment with using the tweet text as part of the multi-modal embedding to evaluate if using the tweet text as further context can improve model performance for image retrieval.
To this end, we concatenate the visual information encoded through the image encoder with three different variants of textual information: i) tweet text, ii) tweet text + OCR text, and iii) OCR text.
After extracting the joint embeddings using CLIP, we further normalize the multi-modal embeddings for evaluation.

For the neural network based image embeddings (Inception-v3, ResNet-50, ResNet-101, and ResNet-152) we experiment with multiple resolutions of input (1x, 2x, etc.), reporting the best performing F1 score for each method.
Similarly, for the neural network based image embeddings and the multi-modal embeddings, we experiment with multiple similarity thresholds (0,0.05, 0.1, 0.15, …, 0.75, 0.8) and report the best performing score for each method (except for the methods that use perceptual hashes with pre-determined similarity thresholds).
For the image descriptors, we experiment with a different number of features (30, 60, 90, and 120) and report the best-performing F1 score for each method.
We compare these methods on . 



The results of the comparison of with the baselines are presented in Table <ref>.
Image descriptor methods perform poorly when used for soft moderation. 
This is because these approaches are designed for higher level tasks like object and scene recognition.
Deep neural network approaches work well in identifying the subjects of moderated images (e.g., Donald Trump or Joe Biden) and therefore report a high recall.
At the same time, however, their precision is low, because they flag any image containing the same subjects as similar.
We find that leveraging multimodal embeddings slightly improves the performance over DNN based methods, but still has a very low precision (0.711) for Inception-v3+SentenceBERT.
In the same way, using concatenated embeddings from Open AI's CLIP significantly increases the precision over single-modality DNN methods, but has a very low recall (0.828).
Among the three different variants of CLIP embeddings leveraging different channels of modalities, we find that encoding the most simplest channel, i.e. CLIP (OCR) has the best performance. 
Surprisingly, encoding tweet text alongside the OCR text, i.e., both CLIP (tweet text) and CLIP (tweet text + OCR) does not improve the performance of the CLIP model.
This can be attributed to how state-of-the art multi-modal embeddings like CLIP are designed to match image and caption pairs, and therefore fail to capture the nuances of text and media co-usage in social media.
Perceptual hashing approaches, on the other hand, identify visually similar images, and therefore report a high precision.
At the same time, the need to exclude images that are visually similar but contain text that is contextually different forces these approaches to use low similarity thresholds, which limits their recall.



overcomes the limitations of all three types of approaches.
The use of perceptual hashing allows our approach to have a better precision than neural network based ones.
At the same time, the use of OCR to determine the context of an image allows to operate at a higher threshold than existing systems based on perceptual hashing, addressing the low recall reported by these methods.
While 's precision is slightly lower than the best-performing perceptual hash method (0.990 vs. 0.995), its recall is the highest among all tested approaches.
As a result of this, reports the best F1-score among the tested approaches, balancing false negatives and false positives better than previous work.

We also compare the runtime of different systems in Table <ref>.
For each method, we report the combined time of generating the image embeddings and indexing those embeddings on Milvus, averaged over 5 independent runs on identical system load.
The time for retrieving visually similar images from Milvus is not considered as Milvus optimizes the retrieval time across all embeddings of different sizes (average of 0.27 seconds) for all of the systems.
We can observe that the runtime of is around 8 times slower than DNN based methods, and about 10 times slower than peceptual hashing based methods.
Computing the OCR label of an image is the most time consuming operation of compared with other methods as it takes an average of 0.223 seconds per image.
While this is a large overhead incurred by , we need to keep in mind that its OCR component is only triggered when a similar image to a seed one is identified within the threshold of θ_visual, which only occurs for 0.973% of the images in .
When there is no match, the time overhead of is the same as PDQHash.
Therefore, we argue that this slowdown of 0.2 seconds every 100 images on average is an acceptable tradeoff, allowing to improve its F1-score by 8% over the second best performing algorithm and allowing for more comprehensive soft moderation.




To further check the impact of the OCR component of , we examine how the latency of OCR changes with increasing amount of text in the images.
We characterize the amount of text by using the percentage of image area covered by it.
Figure <ref> shows a scatter plot of the OCR runtime against the percentage of image covered by text in .
The Pearson's correlation coefficient (r) between the percentage of image covered by text and runtime is 0.401, suggesting a moderate positive correlation between runtime and amount of text contained in an image.
This moderate correlation indicates that the fraction of image covered by text plays a role in the OCR runtime.



 §.§ Detecting Visual Misleading Information on Twitter using 



In this section, we evaluate in the wild to further identify images that, while spreading false information, were not moderated by Twitter.
This showcases the utility of our approach while also highlighting the serious need for scalable and automated techniques to identify visually misleading information.
We first build a set of seed images spreading misleading information, using the 2020 US Presidential Election as a case study.
We then apply to find more images that are similar to the seeds that should have been moderated but were not.
Finally, we perform a thorough qualitative analysis of the types of images detected, providing a characterization of image-based misinformation in the context of the 2020 US Presidential Election and how they align with Twitter's platform policies.



  §.§.§ Identifying and Filtering 2020 US Presidential Election misleading images



In this section, we first describe how we select a dataset of images that received soft moderation from Twitter in the context of the 2020 US Presidential Election.
We then present the results reported by when looking for similar images in the wild, and compare these results with the coverage of the moderation applied by Twitter.

Building a seed set of misleading images.
To build a seed set of misleading images contained in tweets that were moderated by Twitter, we use datasets  and  . 
Following the methodology of <cit.>, we check the moderation status of all tweets in during late 2022 and find 1,133 tweets that were moderated by Twitter.
Unfortunately,  is too large (6.9M tweets) for us to check the moderation status of each tweet, given the limitations imposed by the Twitter API. 
Therefore, we only check for the tweets containing an image that occurred more than 5 times in  , based on their PDQhash, resulting in 542,700 candidate tweets.
From these, we identified a further 3,134 tweets with images that were moderated.
Combining the moderated tweets from these two sources, we end up with a final seed set of 4,267 images.

An issue that we face at this stage is that our dataset contains tweets that were moderated by Twitter, but we do not know if the moderation decision was made based on the image included in the tweet or based on the text.
In fact, it is rather common for social media users to include generic images (e.g., memes or GIFs) in their posts
that do not contain false information themselves.
Two examples of these types of tweets are presented in Figure <ref> (in the Appendix).
Neither image conveys false information, but the text is clearly supporting false claims of election fraud. 
If we kept these images in our seed dataset, would flag many unrelated tweets that just happen to include them, greatly reducing its effectiveness.



To avoid this issue, we manually annotate the 4,267 tweets with images that were moderated by Twitter, discarding those not directly related to false claims about the 2020 US Presidential Election.
We first sample 200 images from the 4,267 seed images and have two annotators independently label them as being relevant or not.
The annotators then discuss their results, finding that 186 out of 200 images were relevant with nearly perfect agreement (Cohen's Kappa score of κ = 0.9177 <cit.>).
This indicates that an image's relevance to the 2020 US Election is clearly understood between the two annotators.
We then split the remaining 4,067 seed images between them and wind up with a total of 959 relevant images.

Detecting misinformation images using .
Finally, we use to retrieve more tweets from the index that contain similar images to the seed misleading images.
To this end, we initially retrieve 50,439 images visually similar to the 959 seed misleading images, out of which 40,244 are further filtered to be contextually similar to the query images, and hence candidates for moderation.

To check for the accuracy of these moderation candidates identified by , we perform a false positive and false negative analysis. 
First, we randomly sample 50 query images from the 959 seed misleading images.
Since the 50 query images are already manually verified to contain misleading images related to the 2020 US Presidential Election, we can assume that visually and contextually similar images to the images will be misleading images as well.
We retrieve and annotate all the images from our index visually similar to the 50 query images following the similar methodology used to build in Section <ref>, resulting in a reference set of 10,387 images. 
Upon querying with the same 50 query images, it retrieves 10,172 images both visually and contextually similar to the query images as moderation candidates.
We find that 96 of these detected images are both visually and contextually dissimilar to the corresponding query images, resulting in a false detection rate of 0.99%.
This is due to the well-known limitations of perceptual hashing algorithms <cit.>, for example when dealing with images where a solid color background dominates.
On the other hand, we find that has a false negative rate of 2.06% as it could only succesfully detect 10,172 images out of the possible 10,387 images.
Based on these results, performs a lot better than Twitter's internal soft moderation approach, which moderated only 521 of those 10,387 images (94.98% false negatives).

Finally, we check if the candidate tweets containing the images identified by were also moderated by Twitter.
Following the methodology in <cit.>, we extract the relevant metadata related to soft moderation interventions for the tweet.
Out of the 40,244 tweets checked, we find that only 2,950 tweets received soft moderation interventions by Twitter.
We were unable to check the intervention-related metadata for 2,479 tweets as they were inaccessible during the time we conducted the study.  
This analysis indicates that Twitter's soft moderation approach is inadequate, with a large proportion of tweets (92.66%) containing misleading images left without moderation, and it showcases the utility of automated approaches like .



  §.§.§ Characterizing misleading images about the 2020 Presidential Election on Twitter



In this section, we perform an in-depth analysis of the images identified by .
To aid human annotation, we first group visually similar images into Image Stories.
We then code the extracted Image Stories according to Twitter's platform content policies <cit.>, with the goal of understanding how the detection performed by aligns with the platform's existing moderation guidelines.
Finally, we investigate whether misleading images violating different types of content policies get moderated differently by Twitter.

Aggregating misleading images into Image Stories.
To help manually code the images detected by , we aggregate them into Image Stories.
We define an “Image story” as a set of similar images that convey the same misleading message in the same context.
For example, the three images included in three different tweets in Figure <ref> are visually similar, and should be grouped together.
To this end, we follow a grouping approach similar to the one adopted by Zannettou et al. when studying image memes <cit.>.

First, we apply clustering to the PDQhash embeddings of the images detected by to group visually similar images by using the DBSCAN clustering algorithm <cit.>.
To use the DBSCAN clustering algorithm, we need to select parameters for two parts of the process: i) distance threshold for assessing similarity, and ii) minimum number of elements in a cluster.
Through the experimental validation performed in Section <ref>, we have already established that using PDQhash as the syntactic embedding, with a θ_visual of 90 is best suited for identifying visually similar images.
We use this empirically informed distance threshold as the Hamming distance difference threshold for the first clustering parameter and set the minimum number of elements in a cluster to be 1, to allow for misleading images that occur in isolation, without visual variants.  
Upon using the DBSCAN algorithm to cluster the 40,244 images detected by , we obtain 258 clusters of images.
These image clusters represent the misleading images aggregated into visually similar claims conveying the same misleading message.

Analyzing misleading images through the lens of Twitter's Platform Policies.
We present our codebook, which guides our annotation process for characterizing Image Stories used during the 2020 Presidential Election.
We develop the codebook such that it aligns with the platform policies of Twitter <cit.>.
We use two different policies outlined in Twitter's Platform Integrity and Authenticity resources: i) Synthetic and manipulated media policy <cit.> and ii) Civic Integrity Policy <cit.>.
Each policy contains multiple categories of violations, and each category is broken down into multiple rules.
The Civic Integrity Policy outlines how “Twitter should not be used for the purpose of manipulating or interfering in elections or other civic processes such as refereed, censuses.”
We include 3 different categories from the Civic Integrity Policy out of the four available, which we discuss in further detail.
The fourth category, “False or misleading affiliation” is related to the use of fake and parody accounts, and thus not related to misleading images.
Similarly, we include the “Synthetic and Manipulated Media Policy” as this policy covers usage of media on Twitter, which is the major modality of content in our study.
The details of the individual rules and categories are listed in Appendix <ref>.

To begin the process, we first randomly sample 100 Image Stories out of the 258 extracted to guide us in developing the codebook.
First, we take each of the 16 different rules of Twitter's Civic Integrity Policy <cit.>, divided into four major categories as the draft of our initial codebook.
Two annotators independently code the sampled images, considering additional contextual information learned from fact-checking articles associated with them.
We discuss these codes, repeating the process three times until the final codebook reached a point where further iterations would not improve it.
Note that we use the 16 different rules to help us identify the closest violation of Twitter's platform policy by the images while annotating the Image stories with the granularity corresponding to the four categories associated with the rules.
We reach a nearly perfect agreement (Cohen's Kappa score of κ = 0.9 <cit.>).

Positioning the extracted Image Stories alongside the platform policies of Twitter helps us re-evaluate the content moderation decision based on Twitter implementations and policies.
We present an example image violating each of the four categories discussed in Figure <ref> (in the Appendix).
In the rest of this section, we describe the four categories in our codebook in detail, along with their definitions and our evaluation for matching images with the corresponding category.




1. Misleading information about how to participate in civic processes:
This category refers to images that mislead people about “election participation procedures and requirements, cause confusion about officials, or discuss threats to voting locations.”
Note that we did not find any images related to threats at voting locations in the extracted image stories.
For election participation procedures and requirements, misleading images aim to spread false claims about the irregularity of the criteria by which votes are being cast.
For example, dead people are casting votes; votes are cast multiple times; votes are being cast after the closing of polls.
Misleading images may also claim that mail-in ballots are insecure, illegal, and a source of voter fraud.
Figure <ref> aims to intimidate voters from voting-by-mail, casting doubts over the security of mail-in ballots despite mail-in ballots proven to be safe to use <cit.>. 

2. Misleading information intended to intimidate people from civic processes:
This category contains images that mislead people about “how votes are being counted (or not counted), problems with ballot equipment, disruption at voting locations, and the closing of polls”.
Misleading images in this category aim to make claims about legitimately cast ballots getting invalidated, malfunction of voting machines, switching of votes between candidates, and deleting of votes on the machines.
We find images spreading misleading claims about disruptions at voting locations like poll workers filling ballots, poll workers forcing voters to use Sharpies, and ballot observers not being allowed to observe the counting process.
The example in Figure <ref> is spreading false claims about ballots using Sharpies in Arizona being invalidated and poll workers forcing people to use Sharpies <cit.>.

3. Misleading information about the outcomes of civic processes:
This category refers to the images that mislead people about “election rigging, ballot tampering, vote tallying, declaration of premature victory, casting doubt on the outcome of civic processes, calling for interference with the implementation of election results, and undermining public confidence in the methods and results of the election.”
Misleading images in this category aim to make claims about the illegal processing and handling of ballots, which include alteration, manipulation, destruction, forging, counterfeiting, stealing, and pre-filling ballots.
We also find false claims about voter registration and turnout, and visual and statistical anomalies in patterns of vote counting.
By casting doubt on the outcome of civic processes, misleading information aims to question the integrity of the election process, discredit different stages of election processes, and call for interference with the implementation of election results using slogans such as “Stop the Steal,” “Fraud.”
The misleading image in Figure <ref> casts doubt over bipartisan vote tallying process, suggesting that Trump ballots were discarded in garbage bags <cit.>.

4. Synthetic and Manipulated Media:
This category refers to images that are “significantly and deceptively altered, manipulated, or fabricated, and images shared with deliberate intent to deceive people”.
For this category, included images are significantly and deceptively altered, manipulated, or fabricated.
On the other hand, images shared with malicious intent include out-of-context tweets sharing the media.
For example, adding quotations shared in the past that are taken out of context to spread election misinformation or images from the past that are re-used in the present context, like discarded ballots from the 2018 election being repurposed to say ballots are being tampered with in the 2020 election.
Figure <ref> is a doctored image in connection to the “Bush-Gore Florida recount” during the presidential race in 2000 <cit.>, shared by a Trump campaign spokesman.


We assign each Image Story to one category.
There are some observations from our annotation process:
We notice that the use of memes is common in spreading misleading information.
They are designed for people to add text to them to make a variation.
During the annotation process, we first assess the overlay text on the memes to see if it falls into any of the categories other than the fourth: Synthetic and Manipulated Media. 
If it does not, we classify it as an image altered to be shared with malicious intent.

We also find that a large portion of images are used out of context and could potentially be categorized into the fourth category: Synthetic and Manipulated Media.
However, out-of-context information is widely used to push narratives related to the other three themes.
Thus, we take precedence of annotating the images into the first three categories if it suits any.
Otherwise, it is annotated as the fourth category.
We annotate these images according to their primary content to expand and deepen the qualitative analysis of these moderated images.
For example, a screenshot of a video clip in which ballot workers are transcribing ballots is used out of context and spread with the misleading content of “ballot stuffing.”
In this example, the medium of spreading the misleading image is taking it out of context. However, the primary content being spread is related to vote counting and ballot tampering, which falls under the Twitter Civic Integrity Policy.
Thus, we annotate it as Category Three: Misleading information about the outcomes of civic processes.

Moderation of misleading images by violation category.
After associating the misleading images with the corresponding platform policy violations, we want to understand whether Twitter is more likely to moderate images violating some of them.
 Table <ref> reports the breakdown of the rate of moderation by Twitter of images belonging to the different categories of violations, among the tweets identified by .
A high-level overview of the breakdown by category suggests that Twitter moderated specific types of misleading image stories more aggressively, while the moderation rate on other categories appears to be laxer.
Misleading image stories of the categories “Intimidation from Civic Processes” and “Participation in Civic Processes” are moderated the most, with 5.96%, and 4.16% of the images getting moderated.
Surprisingly, the most popular category of content violation among image Stories, “Misleading information about outcomes of civic processes” has the lowest moderation rate of 1.77%.
Finally, images violating the category “Synthetic and Manipulated Media” have the moderation rate of 2.76%.
While we can make overall observations that different types of platform policy violations might have been disproportionately moderated from Twitter's end, we find concerning results of overall moderation rate across the categories being very low.









§ RELATED WORK


Soft Moderation Approaches by Social Media.
Social media platforms namely Facebook, and Twitter have increasingly shifted their approach towards warning labels as a tool for content moderation, and additionally, provide surrounding  context to the users when interacting with potentially false information.
These platforms have applied warning labels on false information shared ranging from Covid-19 pandemic <cit.>, 2020 US Presidential Elections <cit.> to climate denial misinformation <cit.>.
Twitter has reported that approximately 74% of the tweet viewership happened after the warning labels were applied to the tweets and, warning labels were effective in decreasing users quoting the misleading tweets by an estimated 29%. 
Savvas Zannettou <cit.> performed empirical analysis on a sample of 2,244 tweets with warning labels related to the 2020 US Presidential Elections.
Paudel et al. proposed Lambretta, a system that aims to improve soft moderation of textual content on Twitter by leveraging Learning to Rank <cit.>.
To the best of our knowledge, is the first end-to-end approach to perform soft-moderation of images social media that has been proposed by the research community.

Applications of Perceptual Hashing in Computer Security.
Perceptual hashing techniques have been widely used in computer security, leveraging them to detect phishing websites <cit.>, scam websites <cit.>,fraudulent services <cit.>, and to identify impersonators on social media <cit.>.
Perceptual hashes have also been used for content authentication <cit.>, and tamper detection on images <cit.>. 
Alkhowaiter et al. <cit.> evaluated six types of perceptual hashes on their ability to detect image manipulation and transformation over two major social media platforms: Facebook and Twitter.
Historically, Microsoft developed PhotoDNA <cit.>, a type of syntactic embedding to identify and report the distribution of child exploitation material.
Facebook currently uses PDQHash in the “ThreatExchange” platform[https://developers.facebook.com/docs/threat-exchange/] to share “signals” of harmful media on their platform.


Image-based misinformation in social networks.
Several studies are focusing on developing automated detection methods to identify images containing misinformation, either manipulated images <cit.>, or images that are taken out of context or misinterpreted on social media <cit.>.
Most of the works on multi-modal misinformation tackle classifying or detecting a single image as misinformation, while very few works have focused on studying the spread, and diffusion of misleading images within and across social media.
The closest work to is a system called DejaVu <cit.>, which is designed to assist journalists in collaboratively addressing the spread of visual misinformation by using ORB descriptors <cit.> (another type of syntactic embedding) to encode the images, and FAISS <cit.> to index the images.
Additionally, other works study the spread of COVID-19 media through WhatsApp in Pakistan <cit.>, and other types of visual misinformation in WhatsApp <cit.>.
On the other hand, works by  <cit.> study the usage of images in state-sponsored influence campaigns.
Wang et al. <cit.> analyzed the spread of Fauxography images on social media, which are images that are presented in an incorrect or untruthful fashion, by using ground truth fact-checked images from fact-checked organization Snopes [https://www.snopes.com/fact-check/].
Similarly, Zannettou et al. used visual similarity (i.e., perceptual hashing) and images annotated by the website KnowYourMeme to study the evolution and diffusion of image memes posted on social media <cit.>.



§ DISCUSSION AND CONCLUSION



In this paper, we presented , a scalable system able to identify images that are candidates for soft moderation on Twitter.
overcomes the inability of perceptual hashing to discern the context of an image by incorporating OCR into the matching process.
Our results show that outperforms three types of image matching systems based on perceptual hashing, on image descriptors, and on deep neural networks.
With the highest F1-score among competitors, places itself as the state of the art in automated image-based soft moderation.

We believe that (which we make publicly available [https://github.com/idramalab/pixelmod]) will be an inspiring foundation for researchers and online platforms aiming to improve content moderation on social media.
In the rest of this section, we first discuss the ethical considerations of our work and design implications that online platforms should keep in mind when deploying .
We then discuss the limitations of our approach and some directions for future work.


Ethical Considerations.
Our work only uses publicly available Twitter data that was collected the official API while it was still open to academic researchers, and we do not interact with users.
As such, this work is not considered human subject research by our institution.
We also preserve the privacy of Twitter users as we do not analyze any personally identifiable information (e.g., location data, account names) and blur the regions of example tweets used in the paper containing identifiable (meta)data.
Additionally, we take steps to blur the example images used in the paper unless i) they are public figures, ii) they are an illustration, iii) they are stock images.
As any content moderation system, could be used for malicious purposes like censorship and surveillance.
We advocate that the system should be used with ethical principles in mind, following the respect for public interest and beneficence principles from the Menlo report <cit.>.

Design implications.
Our experiments found that Twitter soft moderation misses most images that should be labeled.
This means that leveraging would help Twitter's moderators cover more false information on their platform.
However, when deploying , Twitter or other online platforms should take several aspects into consideration.
First, while 's detection performance exceeds that of existing approaches, a false detection rate of 0.99% may still be considered too high by online platforms to consider its adoption as a fully automated soft-moderation system.
We envision PixelMod to be used by platform moderators as a tool to identify a set of candidates for soft moderation with limited false positives, which then receive manual vetting.
The ultimate decision on whether to apply the moderation labels, however, should remain with the human operator.
This is not dissimilar from what online platforms are doing already, but our approach would allow them to obtain a more comprehensive view of misleading content on their network.
Additionally, could help addressing the main pain points of relying on human moderation, which are the latency in decision making and having limited moderator resources <cit.>.

Second, is an inherently reactive system: it requires a set of images already identified by the platform as misleading.
This process could be streamlined by using example images that have been fact-checked by dedicated organizations <cit.>.
When curating the set of seed images in Section <ref>, we found that querying images directly from a list of moderated tweets can be tricky, since all of the moderated images might not be of misleading nature.
In such cases, moderators using should take an additional step to ensure the query images are misleading in nature, rather than ambiguous images not related to the events being studied, to get the most relevant results as candidates for moderation.

Misleading images can spread across multiple social media platforms, propagating with different contexts and forms <cit.>.
Platforms can use as a tool alongside an industry shared database of known misleading image hashes for tracking the spread of misleading images on their service and across other online communities.
Platforms already use shared databases for tracking Child Sexual Abuse Material (through the National Center for Misleading and Exploited Children) and terrorism content (through the Global Internet Forum to Counter Terrorism) <cit.>.
A tool like backed by a centralized database could easily be added to major social media platform's existing efforts to combat fraud and misinformation <cit.>.

Runtime implications.
As discussed in Section <ref>, the increased runtime overhead of is due to the OCR component, which only gets triggered for 0.973% of the images in .
This rate is 1.450% among the 19.7M images used in our ‘in-the-wild’ evaluation in Section <ref>.
It is to note that factors like image resolution, background and font complexity could also impact the runtime of the underlying OCR engine.
The overhead reported is an upper bound on the runtime of and the average runtime of our system is in the same order as other baselines.
This occasional slowdown is a tradeoff we make for increased recall for , which allows us to achieve a much higher coverage of soft-moderation candidates than the ones achieved by Twitter.
However, can be adapted according to the content moderation budget of the platform.
The initial set of results retrieved by (visual matches to query images) can be sorted or filtered through metadata such as popularity of the account posting the images, or other metadata aligning with specific content moderation strategies of a platform, before passing through the contextual similarity component.


Applying to other platforms and topics.
We could not test on other topics and platforms due to the lack of reliable datasets across platforms and topics.
First, we are not able to test our system on other social media platforms due to the lack of access to a uniform sample of posts (like the 1% sample that forms our dataset)
While datasets containing misleading images exist for other platforms like WhatsApp and Telegram, these are not suited for our evaluation since they only contain a single instance of labeled misleading images, making the visual similarity research process that is at the center of moot.
Even though Twitter applied warning labels on COVID-19 misinformation, these were unreliable and inconsistent <cit.>, which we independently confirmed in our preliminary analysis.
Despite the limitations on evaluation settings, we expect to generalize well across multiple platforms and topics.
The only requirement for a platform to apply to a new campaign is a set of seed images that are known to be misleading, and the system should generalize well to other platforms and topics, as the underlying image embeddings are syntactic in nature, and do not incorporate any domain-specific metadata (e.g. number of retweets, number of likes available on tweets).


Limitations.
Despite the promising performance of in identifying visually similar images at scale, the embedding used by (PDQHash) is vulnerable to adversarial manipulations <cit.>.
An adversary could modify images to have a PDQhash that is very far from the one of the corresponding seed image, generating a false detection by .
This is a serious risk, and future research should investigate defenses against these attacks.
Some potential avenues of defense are utilizing an ensemble of hashes: combining results from both pHash and PDQHash <cit.>, and adversarial training of embeddings <cit.>. 
Using an ensemble of hashes would force an attacker to jointly optimize the adversarial attacks against an ensemble of hashing methods opposed to a single method, thus increasing the operation cost on the end of adversaries.
Future works can also look at incorporating the contextual information (OCR text) contained in the images as part of generating the syntactic embeddings themselves, making it difficult for adversaries to modify the images without deviating from the contextual messaging of the images.
At the same time, the threat model here assumes centralized coordination by an adversary.
While this is within reach when dealing with state-sponsored disinformation actors <cit.>, it is not applicable to content spread autonomously and in good faith by regular users, for example in the wake of the uncertainty surrounding the COVID-19 pandemic.
In the case of crisis scenarios, misleading content with significant risk factors might proliferate as many different variants of the original content as a consequence of the broad and diverse vector of sharing by people.
This might render the initial set of seed hashes and similarity threshold being used ineffective in tracing the spread of the images.
One such example of this was the spreading of videos of the Christchurch mosque shooting, where Facebook reported that their systems were defeated by variants of the original videos as of bad actors started sharing it <cit.>.
Platforms could use methods from online learning to update the seed set of hashes to flag, and dynamically adjust the distance threshold in specific crisis cases to better handle such events.


Apart from applying soft moderation warning labels, Twitter outlines “reduced visibility” as a possible consequence for tweets violating their Civic Integrity Policy <cit.>.
This means it is possible Twitter could have identified all of the images subsequently identified by , and chosen to reduce visibility of tweets including them instead of applying warning labels.
However, we cannot analyze this phenomenon due to lack of any public metrics regarding visibility of tweets. 
Due to the nature of the evaluation datasets, images containing multi-lingual text could not be evaluated.
Therefore, future applications of the system in multi-lingual settings might require further configuration of the underlying OCR engine (e.g. specifying “languageHints” parameters in Cloud Vision OCR engine with the intended language of evaluation).

Future work.
In the future, we plan to apply to other online platforms.
The approach is platform-independent, and this could give interesting insights on how misinformation spreads across different online communities, and which communities are particularly influential in generating viral image misinformation.
For example, we believe quite strongly that because of its low computational overhead and high performance, it makes a good fit for deployment by decentralized social networks (e.g., Mastodon), an environment where recent work has shown the potential to actually help improve 's performance via federated model sharing <cit.>.
Since our approach does not require any platform-specific information, it is particularly interesting as a basis for further research, especially at a time where academic research is being seriously threatened by the discontinuation of the Twitter Academic API as we know it.
Regardless of the specifics of future work, our presentation of provides a roadmap for measuring, tuning, and benchmarking soft moderation systems; critical for any moderation tool's success.
We strongly believe that the computer security community has a lot to say in this space, and hope that more researchers will get into this space.

Acknowledgments. We would like to thank the anonymous reviewers and the anonymous shepherd for their feedback, and Primah Muwanga for her help setting up the artifact after the paper was accepted.
This work was supported by the NSF under grants CNS-1942610, CNS-2114407, CNS-2114411, CNS-2247867, CNS-2247868, and IIS-2046590, and by a grant from the Media Ecosystems Analysis Group (MEAG).



abbrv
























































































§ TWITTER'S CONTENT POLICIES AND VIOLATIONS.


We list the four different categories of Twitter's platform policy violations and the corresponding rules for each category below.


    
    
  * Misleading information about how to participate.

        

            
  * Images misleading people about participation procedures and requirements.

            
  * Images sowing confusion about officials and institutions.

            
  * Images discussing threats on voting locations.

        

    
  * Misleading information intended to intimidate people from civic processes.

     

            
  * Images about votes not being counted.

            
  * Images about Equipment Problems.

            
  * Images about disruptions at voting locations.

            
  * Images about closing of polls.

        

        
  * Misleading information about outcomes of civic processes.

    

            
  * Images undermining public confidence in methods and results of election.

            
  * Images with misleading claims about election rigging.

            
  * Images with misleading claims about ballot tampering.

            
  * Images with misleading claims about vote tallying.

            
  * Images with declaration of premature victory.

            
  * Images casting doubt on outcome of civic processes.

            
  * Images calling for interference with the implementation of election results.

        

    
  * Synthetic and Manipulated Media.


    

            
  * Images that are significantly and deceptively altered, manipulated or fabricated.

            
  * Images shared with malicious intent , including out of context tweets sharing the media.

        


























































































































































































 Paper Title: OmniParser for Pure Vision Based GUI Agent.



Extracting doubly-excited state lifetimes in helium directly in the time domain with attosecond noncollinear four-wave-mixing spectroscopy
    Stephen R. Leone
    August 5, 2024
==========================================================================================================================================




§ ABSTRACT 
   The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce OmniParser, a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. 
   We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements.
   
   OmniParser significantly improves GPT-4V's performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, OmniParser with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot. 




§ INTRODUCTION

Large language models have shown great success in their understanding and reasoning capabilities. More recent works have explored the use of large vision-language models (VLMs) as agents to perform complex tasks on the user interface (UI) with the aim of completing tedious tasks to replace human efforts <cit.>. Despite the promising results, there remains a significant gap between current state-of-the-arts and creating widely usable agents that can work across multiple platforms, e.g. Windows/MacOS, IOS/Android and multiple applications (Web broswer Office365, PhotoShop, Adobe), with most previous work focusing on limiting applications or platforms.

While large multimodal models like GPT-4V and other models trained on UI data <cit.> have demonstrated abilities to understand basic elements of the UI screenshot, action grounding remains one of the key challenges in converting the actions predicted by LLMs to the actual actions on screen in terms of keyboard/mouse movement or API call <cit.>. It has been noted that GPT-4V is unable to produce the exact x-y coordinate of the button location, Set-of-Mark prompting <cit.> proposes to overlay a group of bounding boxes each with unique numeric IDs on to the original image, as a visual prompt sent to the GPT-4V model. By applying set-of-marks prompting, GPT-4V is able to ground the action into a specific bounding box which has ground truth location instead of a specific xy coordinate value, which greatly improves the robustness of the action grounding <cit.>. However, the current solutions using SoM relies on parsed HTML information to extract bounding boxes for actionable elements such as buttons, which limits its usage to web browsing tasks. We aim to build a general approach that works on a variety of platforms and applications. 

In this work, we argue that previous pure vision-based screen parsing techniques are not satisfactory, which lead to significant underestimation of GPT-4V model's understanding capabilities. And a reliable vision-based screen parsing method that works well on general user interface is a key to improve the robustness of the agentic workflow on various operating systems and applications. We present OmniParser, a general screen parsing tool to extract information from UI screenshot into structured bounding box and labels which enhances GPT-4V's performance in action prediction in a variety of user tasks. 


We list our  contributions as follows: 

    
  * We curate a interactable region detection dataset using bounding boxes extracted from DOM tree of popular webpages. 
    
  * We propose OmniParser, a pure vision-based user interface screen parsing method that combines multiple finetuned models for better screen understanding and easier grounded action generation.
    
  * We evaluate our approach on ScreenSpot, Mind2Web and AITW benchmark, and demonstrated a significant improvement from the original GPT-4V baseline without requiring additional input other than screenshot. 
    




§ RELATED WORKS



 §.§ UI Screen Understanding


There has been a line of modeling works focusing on detailed understanding of UI screens, such as Screen2Words <cit.>, UI-BERT <cit.>, WidgetCaptioning <cit.>, Action-BERT <cit.>. These works demonstrated effective usage of multimodal models for extracting semantics of user screen. But these models rely on additional information such as view hierarchy, or are trained for visual question answering tasks or screen summary tasks. 


There are also a couple publicly available dataset that on UI screen understanding. Most notably the Rico dataset <cit.>, which contains more than 66k unique UI screens and its view hierarchies. Later <cit.>  auguments Rico by providing 500k human annotations on the original 66k RICO screens identifying various icons based on their shapes and semantics, and associations between selected general UI elements (like icons, form fields, radio buttons, text inputs) and their text labels. Same on mobile platform, PixelHelp <cit.> provides a dataset that contains UI elements of screen spanning across 88 common tasks. In the same paper they also released RicoSCA which is a cleaned version of Rico. For the web and general OS domain, there are several works such Mind2Web <cit.>, MiniWob++<cit.>, Visual-WebArena <cit.>, and OS-World <cit.> that provide simulated environment, but does not provide dataset explicitly for general screen understanding tasks such as interactable icon detection on real world websites. 

To address the absence of a large-scale, general web UI understanding dataset, and to keep pace with the rapid evolution of UI design, we curated an icon detection dataset using the DOM information from popular URLs avaialbe on the Web. This dataset features the up-to-date design of icons and buttons, with their bounding boxes retrieved from the DOM tree, providing ground truth locations. 



 §.§ Autonomous GUI Agent


Recently there has been a lot of works on designing autonomous GUI agent to perform tasks in place of human users. One line of work is to train an end-to-end model to directly predict the next action, representative works include: Pixel2Act <cit.>, WebGUM<cit.> in web domain, Ferret <cit.>, CogAgent <cit.>, and Fuyu <cit.> in Mobile domain. Another line of works involve leveraging existing multimodal models such as GPT-4V to perform user tasks. Representative works include MindAct agent <cit.>, SeeAct agent <cit.> in web domain and agents in <cit.> for mobile domain. 
These work often leverages the DOM information in web browser, or the view hierarchies in mobile apps to get the ground truth position of interactable elements of the screen, and use Set-Of-Marks<cit.> to overlay the bounding boxes on top of the screenshot then feed into the vision-language models. However, ground truth information of interactable elements may not always be available when the goal is to build a general agent for cross-platforms and cross-applications tasks. Therefore, we focus on providing a systematic approach for getting structured elements from general user screens. 






§ METHODS

A complex task can usually be broken down into several steps of actions. Each step requires the model's (e.g. GPT-4V) ability to: 1) understand the UI screen in the current step, i.e. analyzing what is the screen content overall, what are the functions of detected icons that are labeled with numeric ID, and 2) predict what is the next action on the current screen that is likely to help complete the whole task. Instead of trying to accomplish the two goals in one call, we found it beneficial to extract some of the information such as semantics in the screen parsing stage, to alleviate the burden of GPT-4V so that it can leverages more information from the parsed screen and focus more on the action prediction. 


Hence we propose OmniParser, which integrates the outputs from a finetuned interactable icon detection model, a finetuned icon description model, and an OCR module. This combination produces a structured, DOM-like representation of the UI and a screenshot overlaid with bounding boxes for potential interactable elements. We discuss each component of the OmniParser in more details for the rest of the section. 






 §.§ Interactable Region Detection


Identifying interactable regions from the UI screen is a crucial step to reason about what actions should be performed given a user tasks. Instead of directly prompting GPT-4V to predict the xy coordinate value of the screen that it should operate on, we follow previous works to use the Set-of-Marks approach <cit.> to overlay bounding boxes of interactable icons on top of UI screenshot, and ask GPT-4V to generate the bounding box ID to perform action on. However, different from <cit.> which uses the ground truth button location retrieved from DOM tree in web browswer, and <cit.> which uses labeled bounding boxes in the AITW dataset <cit.>, we finetune a detection model to extract interactable icons/buttons. 

Specifically, we curate a dataset of interactable icon detection dataset, containing 67k unique screenshot images, each labeled with bounding boxes of interactable icons derived from DOM tree. We first took a 100k uniform sample of popular publicly availabe urls on the web <cit.>, and collect bounding boxes of interactable regions of the webpage from the DOM tree of each urls. Some examples of the webpage and the interactable regions are shown in <ref>.  

Apart from interactable region detection, we also have a OCR  module to extract bounding boxes of texts. Then we merge the bounding boxes from OCR detection module and icon detection module while removing the boxes that have high overlap (we use over 90% as a threshold). For every bounding box, we label it with a unique ID next to it using a simple algorithm to minimizing the overlap between numeric labels and other bounding boxes. 




 §.§ Incorporating Local Semantics of Functionality

We found in a lot of cases where only inputting the UI screenshot overlayed with bounding boxes and associated IDs can be misleading to GPT-4V. We argue the limitation stems from GPT-4V's constrained ability to simultaneously perform the composite tasks of identifying each icon's semantic information and predicting the next action on a specific icon box. This has also been observed by several other works <cit.>. To address this issue, we incorporate the local semantics of functionality into the prompt, i.e. for each icons detected by the interactable region detection model, we use a finetuned model to generate description of functionality to the icons, and for each text boxes, we use the detected texts and its label.

We perform more detailed analysis for this topic in section <ref>. To the best of our knowledge, there is no public model that is specifically trained for up-to-date UI icon description, and is suitable for our purpose to provide fast and accurate local semantics for the UI screenshot. Therefore we curate a dataset of  7k icon-description pairs using GPT-4o, and finetune a BLIP-v2 model <cit.>  on this dataset. Details of dataset and training can be found in Appendix <ref>. After finetuning, we found the model is much more reliable in its description to common app icons. Examples can be seen in figure <ref>. And in figure <ref>, we show it is helpful to incorporate the semantics of local bounding boxes in the form of text prompt along with the UI screenshot visual prompt. 




















§ EXPERIMENTS AND RESULTS

We conduct experiments on several benchmarks to demonstrate the effectiveness of OmniParser. We start by a motivating experiments showing that current GPT-4V model with set of mark prompting <cit.> is prone to incorrectly assigning label ID to the referred bounding boxes. Then we evaluate on Seeclick benchmark and Mind2Web to further showcase OmniParser with local semantics can improve the GPT-4V's performance on real user tasks on different platforms and applications.   



 §.§ Evaluation on SeeAssign Task


To test the ability of correctly predicting the label ID given the description of the bounding boxes for GPT-4v models, We handcrafted a dataset SeeAssign that contains 112 tasks consisting of samples from 3 different platforms: Mobile, Desktop and Web Browser. Each task includes a concise task description and a screenshot image. The task descriptions are manually created and we make sure each task refers to one of the detected bounding boxes, e.g. 'click on 'settings”, 'click on the minimize button'. During evaluation, GPT-4V is prompted to predict the bounding box ID associated to it. Detailed prompt are specified in Appendix.  
The task screenshot images are sampled from the ScreenSpot <cit.> benchmark, where they are labeled with set of marks using OmniParser. The tasks are further divided into 3 sub-categories by difficulty: easy (less than 10 bounding boxes), medium (10-40 bounding boxes) and hard (more than 40 bounding boxes). 

From table <ref>, we see that GPT-4V often mistakenly assign the numeric ID to the table especially when there are a lot of bounding boxes over the screen. And by adding local semantics including texts within the boxes and short descriptions of the detected icons, GPT-4V's ability of correctly assigning the icon improves from 0.705 to 0.938.  

From figure <ref>, we see that without the description of the referred icon in the task, GPT-4V often fails to link the icon required in the task and the ground truth icon ID in the SoM labeled screenshot, which leads to hallucination in the response. With fine-grain local semantics added in the text prompt, it makes it much easier for GPT-4V to find the correct icon ID for the referred icon. 
























 §.§ Evaluation on ScreenSpot

ScreenSpot dataset <cit.>  is a benchmark dataset that includes over 600 interface
screenshots from mobile (iOS, Android), desktop
(macOS, Windows), and web platforms. The task instructions are manually created so that each instruction corresponds to an actionable
elements on the UI screen. We first evaluate the performance of omniparser using the this benchmark.  In table <ref>, we can see across the 3 different platforms: Mobile, Desktop and Web, omniparser significantly improves the GPT-4V baseline. Noticeably, omniparser's performance even surpasses models that are specifically finetuned on GUI dataset including SeeClick, CogAgent and Fuyu by a large margin. 
We also note that incorporating the local semantics (omniparser w. LS in the table) further improves the overall performance. This corroborates with the finds in section <ref> that incorporating local semantics of the UI screenshot in text format, i.e. adding OCR text and descriptions of the icon bounding boxes further helps GPT-4V to accurately identify the correct element to operate on.  
Furthermore, our findings indicate that the interactable region detection (ID) model we finetuned improves overall accuracy by an additional 4.3% compared to using the raw Grounding DINO model. This underscores the importance of accurately detecting interactable elements for the success of UI tasks. Overall, the results demonstrate that the UI screen understanding capability of GPT-4V is significantly underestimated and can be greatly enhanced with more accurate interactable elements detection and the incorporation of functional local semantics.

  



 §.§ Evaluation on Mind2Web

In order to test how OmniParser is helpful to the web navigation secnario, We evaluate on <cit.> benchmark. There are 3 different categories of task in the test set: Cross-Domain, Cross-Website, and Cross-Tasks.

We used a cleaned version of Mind2Web tests set processed from the raw HTML dump which eliminates a small number of samples that has incorrect bounding boxes. In total we have 867, 167, 242 tasks in the test set from Cross-Domain, Cross-Website, and Cross-Tasks category respectively. During evaluation, we feed both the parsed screen results and the action history as text prompt, and SOM labeled screenshot to GPT-4V similar to the prompting strategy in <cit.>. Following the original paper, we perform offline evaluation focusing on the element accuracy, Operation F1 and step success rate averaged across the task.

In the first section of the table (row 1-3), We report numbers from a set of open source VL models as it appears in <cit.>. Here CogAgent and Qwen-VL are not finetuned on the Mind2Web training set. More detailed information about model settings can be found in the Appendix<ref>. 

In the second section of the table (row 4-9) we report numbers from Mind2web paper <cit.> and SeeAct <cit.>
paper. In this section, all of the approaches use the HTML elements selected by a finetuned element proposal model on Mind2Web training set which produces top 50 relevant elements on the HTML page based on the user task. Additionally, GPT-4V+SOM and GPT-4V+textual choices corresponds to the SeeAct with image annotation, and textual choices grounding methods respectively. In GPT-4V+SOM, the set of mark (SOM) boxes are selected from the element proposal model, and are labeled with the ground truth location extracted from HTML. In contrast, GPT-4V+textual uses DOM information of the selected relevant elements directly in the text prompt, rather than overlaying bounding boxes on top of screenshot. The better performance of textual choice corroborates with the experiment results in <ref>. 

In the last section (row 10-11), we report numbers from OmniParser. We observe GPT-4V augumented with local semantics of icon functionality and the finetuned interactable region detection model (w. LS + ID) performs better than the model with raw grounding DINO model (w. LS + GD) in all of the categories. 


Further, without using parsed HTML information, OmniParser is able to outperform GPT-4's performance that uses HTML in every sub-category by a large margin, suggesting the substantial benefit of the screen parsing results provided by OmniParser. 
Additionally, OmniParser outperforms the GPT-4V+SOM by a large margin. Compared to GPT-4V+textual choices, OmniParser significantly outperforms in Cross-Website and Cross-Domain category (+4.1% and +5.2%), while underperforming (-0.8%) slightly in the Cross-Task category, which indicates that OmniParser provides higher quality information compared ground truth element information from DOM and top-k relevant elemnt proposal used by the GPT-4V+textual choices set-up, and make the GPT-4V easier to make a accurate action prediction. 
Lastly, OmniParser with GPT-4V significantly outperform all the other trained models using only UI screenshot such as SeeClick and Qwen-VL.  








 §.§ Evaluation on AITW

In additional to evaluation on multi-step web browsing tasks, we assess OmniParser on the mobile navigating benchmark AITW <cit.>, which contains 30k instructions and 715k trajectories. We use the same train/test split as in <cit.> based on instructions, which retain only one trajectory for each instructions and no intersection between train and test. For a fair comparison, we only use their test split for evaluation and discard the train set as our method does not require finetuing.  

In table <ref>, we report the GPT-4V baseline in <cit.> paper, which corresponds to the best performing set up (GPT-4V ZS+history) that uses UI elements detected by IconNet <cit.> through set-of-marks prompting <cit.> for each screenshot at every step of the evaluation. The detected UI elements consist of either OCR-detected text or an icon class label, which is one of the 96 possible icon types identified by IconNet. Additionally, action history is also incorporated at each step's prompt as well.  We used the exact same prompt format in <cit.> except the results from the IconNet model is replaced with the output of the finetuned interactable region detection (ID) model.
Interestingly, we found that the ID model can generalize well to mobile screen. By replacing the IconNet with the interactable region detection (ID) model we finetuned on the collected webpages, and incorporating local semantics of icon functionality (LS), we find OmniParser delivers significantly improved performance across most sub-categories, and a 4.7% increase in the overall score compared to the best performing GPT-4V + history baseline. 










§ DISCUSSIONS


In this section, we discuss a couple of common failure cases of OmniParser with examples and potential approach to improve. 



Repeated Icons/Texts From analysis of the the GPT-4V's response log, we found that GPT-4V often fails to make the correct prediction when the results of the OmniParser contains multiple repeated icons/texts, which will lead to failure if the user task requires clicking on one of the buttons. This is illustrated by the figure <ref> (Left) in the Appendix. A potential solution to this is to add finer grain descriptions to the repeated elements in the UI screenshot, so that the GPT-4V is aware of the existence of repeated elements and take it into account when predicting next action. 



Corase Prediction of Bounding Boxes One common failure case of OmniParser is that it fails to detect the bounding boxes with correct granularity. In figure <ref> (Right), the task is to click on the text 'MORE'. The OCR module of OmniParser detects text bounding box 8 which encompass the desired text. But since it uses center of the box as predicted click point, it falls outside of the ground truth bounding box. This is essentially due to the fact that the OCR module we use does not have a notion of which text region are hyperlink and clickable. Hence we plan to train a model that combines OCR and interactable region detection into one module so that it can better detect the clickable text/hyperlinks. 


Icon Misinterpretation We found that in some cases the icon with similar shape can have different meanings depending on the UI screenshot. For example, in figure <ref>, the task is to find button related to 'More information', where the ground truth is to click the three dots icon in the upper right part of the screenshot.  OmniParser successfully detects all the relevant bounding boxes, but the icon description model interpret it as: "a loading or buffering indicator". We think this is due to the fact that the icon description model is only able to see each icon cropped from image, while not able to see the whole picture during both training and inference. So without knowing the full context of the image, a symbol of three dots can indeed mean loading buffer in other scenarios. A potential fix to this is to train an icon description model that is aware of the full context of the image. 






§ CONCLUSION

In this report, We propose OmniParser, a general vision only approach that parse UI screenshots into structured elements.  OmniParser encompasses two finetuned models: an icon detection model and a functional description models. To train them, we curated an interactable region detection dataset using popular webpages, and an icon functional description dataset. We demonstrate that with the parsed results, the performance of GPT-4V is greatly improved on ScreenSpot benchmarks. It achieves better performance compared to GPT-4V agent that uses HTML extracted information on Mind2Web, and outperforms GPT-4V augmented with specialized Android icon detection model on AITW benchmark. We hope OmniParser can serve as a general and easy-to-use tool that has the capability to parse general user screen across both PC and mobile platforms without any dependency on extra information such as HTML and view hierarchy in Android. 




§ ACKNOWLEDGEMENT

We would like to thank Corby Rosset and authors of ClueWeb22 for providing the seed urls for which we use to collect data to finetune the interactable region detection model. The data collection pipeline adapted AutoGen's multimodal websurfer code for extracting interatable elements in DOM, for which we thank Adam Fourney. We also thank Dillon DuPont for providing the processed version of mind2web benchmark.



alpha





§ APPENDIX



 §.§ Details of Icon-Description Dataset


In figure <ref>, we see that the original BLIP-2 model tend to focus on describing shapes and colors of app icons, while struggling to recognize the semantics of the icon. This motivates us to finetune this model on an icon description dataset. For the dataset, we use the result of parsed icon bounding boxes inferenced by the interactable icon detection model on the ScreenSpot dataset since it contains screenshots on both mobile and PC. For the description, we ask GPT-4o whether the object presented in the parsed bounding box is an app icon. If GPT-4o decides the image is an icon, it outputs one-sentence description of the icon about the potential functionality. And if not, GPT-4o will output 'this is not an icon', while still including this in the dataset. In the end, we collected 7185 icon-description pairs for finetuning. 

We finetune BLIP-2 model for 1 epoch on the generated dataset with constant learning rate of 1e^-5, no weight decay and Adam optimizer. We show a few of the qualitative examples of finetuned model vs the original model in figure <ref>. 





 §.§ Training details of Interactable Icon Region Detection Model

As introduced in <ref>, we train a YOLOv8 model on the interactable icon region detection dataset. We collect in total of 66990 samples where we split 95% (63641) for training, and 5% (3349) for validation. We train for 20 epochs with batch size of 256, learning rate of 1e^-3, and the Adam optimizer on 4 GPUs. We show the training curve in figure <ref>. 






 §.§ Details of SeeAssign Evaluation



  §.§.§ Prompt Used for GPT-4V

GPT-4V without local semantics: 

Here is a UI screenshot image with bounding boxes and corresponding labeled ID overlayed on top of it, your task is task. Which icon box label you should operate on? Give a brief analysis, then put your answer in the format of “`Box with label ID: [xx]“`

GPT-4V with local semantics: 

Here is a UI screenshot image with bounding boxes and corresponding labeled ID overlayed on top of it, and here is a list of icon/text box description: parsed_local_semantics. Your task is task. Which bounding box label you should operate on? Give a brief analysis, then put your answer in the format of “`Box with label ID: [xx]“`



 §.§ Details of Mind2Web Evaluation

Here we list more details of each baseline in table <ref>.


SeeClick, QWen-VL SeeClick is a finetuned version of Qwen-VL on the Mind2Web training set and we report both of their numbers in their paper. 

CogAgent CogAgent number is taken from the SEEAct paper <cit.>, where they report cogagent-chat-hf checkpoint that is not fine-tuned on Mind2Web for experiments. 

MindAct(Gen), MindAct, GPT-3.5-Turbo, GPT-4 The numbers for these baseline are taken from the Mind2Web <cit.> paper, where they use HTML information to augument the corresponding web agent.

GPT-4V+som This model corresponds to the image annotation grounding method in SeeAct paper, where the som boxes extracted from the selelcted HTML elements are provided to GPT-4V to make action prediction.

GPT-4V+textual choice This corresponds to the best performing scenario in SeeAct paper (except the Oracle), that uses the selected HTML elments information in a multi-choice question format as input to the GPT-4V agent. 



  §.§.§ Qualitative Examples

We list a few more examples to demonstrate local semantics of icon function description helps GPT-4V make better action prediction in figure <ref>.


























 Paper Title: Benchmarking AIGC Video Quality Assessment: A Dataset and Unified Model.







Benchmarking AIGC Video Quality Assessment: A Dataset and Unified Model
    
Zhichao Zhang, 
Xinyue Li, 
Wei Sun, 
Jun Jia, 
Xiongkuo Min,
Zicheng Zhang, 
Chunyi Li, 
Zijian Chen, 
Puyi Wang, 
Zhongpeng Ji, 
Fengyu Sun, 
Shangling Jui, 
and Guangtao Zhai


    August 5, 2024
========================================================================================================================================================================================



§ ABSTRACT 
In recent years, artificial intelligence (AI) driven video generation has garnered significant attention due to advancements in stable diffusion and large language model techniques. Thus, there is a great demand for accurate video quality assessment (VQA) models to measure the perceptual quality of AI-generated content (AIGC) videos as well as optimize video generation techniques. However, assessing the quality of AIGC videos is quite challenging due to the highly complex distortions they exhibit (e.g., unnatural action, irrational objects, etc.). Therefore, in this paper, we try to systemically investigate the AIGC-VQA problem from both subjective and objective quality assessment perspectives. For the subjective perspective, we construct a Large-scale Generated Video Quality assessment (LGVQ) dataset, consisting of 2,808 AIGC videos generated by 6 video generation models using 468 carefully selected text prompts. Unlike previous subjective VQA experiments, we evaluate the perceptual quality of AIGC videos from three dimensions: spatial quality, temporal quality, and text-to-video alignment, which hold utmost importance for current video generation techniques. For the objective perspective, we establish a benchmark for evaluating existing quality assessment metrics on the LGVQ dataset, which reveals that current metrics perform poorly on the LGVQ dataset. Thus, we propose a Unify Generated Video Quality assessment (UGVQ) model to comprehensively and accurately evaluate the quality of AIGC videos across three aspects using a unified model, which uses visual, textual and motion features of video and corresponding prompt, and integrates key features to enhance feature expression. We hope that our benchmark can promote the development of quality evaluation metrics for AIGC videos. The LGVQ dataset and the UGVQ metric will be publicly released.



video generation, AIGC, video quality assessment, dataset, benchmark.




§ INTRODUCTION


With the rapid development of generative models and digital media techniques, artificial intelligence (AI)-generated content (AIGC) media, leveraging generative models to automatically create media content, has gained substantial attention in recent years. A typical form of AIGC media is text-to-video (T2V) <cit.>, also known as AIGC videos, where videos are generated entirely based on textual descriptions. Because of the highly simplistic generation process, AIGC videos can be widely used in industries including film <cit.>, gaming <cit.>, advertising <cit.>, and more. Although significant advancements have been made in the field of AIGC videos, they still face various quality issues. For example, AIGC videos may exhibit severe spatial and temporal distortions, such as blurred objects and backgrounds, poor action continuity, etc. Moreover, discrepancies between AIGC videos and the original text may impede their effectiveness in real-world applications. We show some typical distortions of AIGC videos in Fig. <ref>.

In the spatial domain, distortions include blurriness and irrational objects.  Blurriness results in poor sharpness, and irrational objects show forms that defy physical logic. Temporal distortions involve motion blur, causing significant distortion in rapidly moving parts, and frame jitter, leading to incoherent frames and severe shaking. Alignment distortions encompass event inconsistency, where the described action mismatches the video content, and context inconsistency, where the main subject or object in the prompt is inconsistent with the video.

Therefore, how to effectively evaluate the perceptual quality of AIGC videos is crucial for measuring the progress of video generation techniques, selecting the best AIGC videos from a set of candidates generated by T2V models and optimizing the video generation techniques <cit.>.



Video quality assessment has been applied in various fields <cit.>, most methods focus on User-Generated Content (UGC). However, there is still no fair metric for AIGC videos.
In previous video generation studies <cit.>, only a few metrics are utilized to evaluate the effectiveness of video generation methods, such as IS <cit.>, FID <cit.>, FVD <cit.>, KVD <cit.>, CLIP <cit.>, CLIPScore <cit.>, FCS <cit.>. However, FID, FVD, and KVD compare the distribution of Inception <cit.> features of generated frames with that of a set of real images/videos, thus failing to capture distortion-level and semantic-level quality characteristics. IS <cit.> does not require reference videos for comparison but relies on pre-trained classification models. Furthermore, motion generation poses a great challenge for current video generation techniques, yet FID <cit.> and FVD <cit.> are unable to quantify the impact of temporal-level distortions on visual quality. CLIP-based methods such as CLIPScore <cit.>, and FCS <cit.> are frequently employed to assess the alignment between the generated video and its prompt text.


Towards this goal, we establish a benchmark for AIGC-VQA to evaluate the effectiveness of existing quality assessment metrics in assessing the perceptual quality of AIGC videos. 
We first construct a Large-scale Generated Video Quality assessment (LGVQ) dataset to subjectively evaluate the three most critical quality dimensions (i.e., spatial quality, temporal quality, and text-to-video alignment) of AIGC videos. To enhance the utility of the benchmark, the generated video content should cover wide a range of real-world scenes as possible. So, we structurally partition the text prompts into three components: foreground, background, and motion. For each component, we include typical elements that frequently occur in daily life. Specifically, the foreground includes four categories: people, animals, plants, and man-made objects, and the background is categorized into indoor scenes, outdoor natural scenes, and outdoor man-made scenes. For motions, we focus on three types: static, dynamic, and local movement (e.g., watching belongs to static, running belongs to dynamic, and taking belongs to local movement). By combining different words or phrases from the aforementioned types, we obtain 468 text prompts for video generation. Then, six mainstream T2V algorithms are selected to generate 2,808 AIGC videos. To obtain the perceptual quality labels of these videos, we invite 54 subjects to provide their ratings of the spatial quality, temporal quality, and text-to-video alignment for each video. 


Subsequently, we bench existing quality assessment metrics on the LGVQ dataset to analyze their ability in assessing the quality of AIGC videos. Given that LGVQ provides spatial quality, temporal quality, and text-to-alignment labels for each AIGC video, we evaluate three categories of quality assessment metrics for the AIGC-VQA benchmark: image quality assessment (IQA) methods for spatial quality evaluation, video quality assessment (VQA) methods for temporal quality evaluation, and CLIP-based methods for text-to-video alignment evaluation. Specifically, we test 10 IQA methods, 10 VQA methods, and 8 alignment methods on LGVQ. 

Therefore, we further propose a Unify Generated Video Quality assessment (UGVQ) model to simultaneously evaluate the three aspects of the quality of AIGC videos. UGVQ leverages the visual and textual features of CLIP <cit.> and the motion features of SlowFast <cit.> as the quality-aware feature representation. 
We use visual CLIP <cit.> features and the motion features of SlowFast <cit.> as the frame-level spatial feature and the video-level temporal feature, respectively. 
For the text-video alignment, we use textual CLIP <cit.> features as the prompt semantics, and use the visual CLIP <cit.> features as the semantics of the frames. Then, we propose a video-level spatial feature extractor to extract spatial features from consecutive frames. 
To enhance the quality-aware feature representation, we propose a feature fusion module to fuse the spatial features, text features, and temporal features. Experiments show that UGVQ exhibits the best performance in assessing all three quality aspects of AIGC videos compared to existing quality metrics re-trained on the LGVQ dataset, suggesting that UGVQ can be used for an effective and comprehensive VQA metric for assessing the quality of AIGC videos.




§ RELATED WORKS





 §.§ Video Generation Techniques


With the development of artificial intelligence, video generation, especially text-to-video (T2V) generation, has made significant progress. T2V models can be categorized into three types: GAN/VAE-based <cit.>, Autoregressive-based <cit.>, and Diffusion-based <cit.>.

Initially, researchers used Variational Autoencoders (VAE) or Generative Adversarial Networks (GAN) for T2V generation. For instance, Li et al. <cit.> trained a conditional video generative model combining VAE and GAN to extract static and dynamic information from text. Deng et al. <cit.> proposed the introspective recurrent convolutional GAN.


Autoregressive models (e.g., Transformer) have also been explored for T2V generation. For instance, NÜWA <cit.> leverages a 3D transformer encoder-decoder with a nearby attention mechanism for high-quality video synthesis. NÜWA-Infinity <cit.> presents a “render-and-optimize” strategy for infinite visual generation. CogVideo <cit.> utilizes pre-trained weights from the text-to-image model and employs a multi-frame-rate hierarchical training strategy to enhance text-video alignment. Phenaki <cit.> uses a variable-length video generation method with a C-ViViT encoder-decoder structure to compress video into discrete tokens.

Diffusion models <cit.> have significantly advanced text-to-image generation and are now being explored for T2V generation <cit.>. The Video Diffusion Model <cit.> applies the diffusion model to video generation using a 3D U-Net architecture combined with temporal attention.
LVDM <cit.> introduces a hierarchical latent video diffusion model. Gen-1 <cit.> is a structure and content-guided video diffusion model, training on monocular depth estimates for control over structure and content. Tune-a-video <cit.> employs a spatiotemporal attention mechanism to maintain frame consistency. Video Crafter1 <cit.> uses a video VAE and a video latent diffusion process for lower-dimensional latent representation and video generation. NÜWA-XL <cit.> uses two diffusion models to generate keyframes and refine adjacent frames.

Despite these advancements, challenges like unrealistic motion, intermittent object appearances, and unrealistic phenomena remain <cit.>. Quality assessment for T2V videos is crucial for measuring progress and further promoting the development of T2V models.





 §.§ Quality Metrics for AIGC Videos


Spatial quality metrics aim to measure the frame-level visual quality of videos. IS <cit.> and FID <cit.> are the most frequently used metrics to evaluate spatial quality. However, many studies <cit.> have indicated that IS and FID exhibit poor correlation with human visual perception. On the other hand, image quality assessment (IQA) is design to quantify the perceptual quality of images. Many popular IQA methods, such as SSIM <cit.>, UNIQUE <cit.>, StairIQA <cit.>, MUSIQ <cit.>, LIQE <cit.>, etc., have demonstrated remarkable capability in measuring the perceptual quality of natural images. 




Temporal quality metrics are responsible for assessing the temporal coherence of AIGC videos.
Previous T2V studies utilize FVD <cit.> to gauge the disparity between features extracted by pre-trained Inflated-3D Convnets (I3D) <cit.> from generated videos and realistic videos. 
Similar to FID, FVD also demonstrates a weak correlation with human visual perception. As related research, user-generated content (UGC) VQA models <cit.>, such as SimpleVQA <cit.>, FastVQA <cit.>, DOVER <cit.>, OV-PSNR <cit.>, SSL <cit.>, etc., have attempted to utilize action recognition network (e.g. SlowFast <cit.>, Video Swin Transformer <cit.>) to represent the temporal quality feature. However, several studies <cit.> have shown that the current UGC VQA datasets pose little challenge to temporal quality analyzers in UGC VQA models



Text-to-video alignment metrics evaluate the consistency between the generated videos and textual descriptions. CLIP-based methods, such as CLIP <cit.>, BLIP <cit.>, and viCLIP <cit.> are frequently used to evaluate the consistency between the AIGC videos and their text prompts. While these methods are trained on large-scale text-image datasets <cit.> to maximize the similarity of positive pairs, recent studies <cit.> demonstrate that they have poor consistency with human visual perception. Hence, some studies have constructed human-rated text-to-image alignment datasets <cit.>. Based on these datasets, they develop alignment assessment models, like ImageReward <cit.>, PickScore <cit.>, HPSv1 <cit.>, HPSv2 <cit.>, etc., to evaluate the consistency between the text prompts and AIGC images. 




§ SUBJECTIVE QUALITY ASSESSMENT STUDY








 §.§ LGVQ Dataset

We first construct LGVQ, a large-scale generated video dataset consisting of diverse AIGC videos, to serve as the benchmark for testing the perceptual quality of AIGC videos subjectively and objectively. 




Prompts Selection.
Existing benchmarks such as VBench <cit.> and FETV <cit.> offer extensive but somewhat broad classifications. VBench, for example, categorizes text prompts into eight classes: animal, architecture, food, human, lifestyle, plant, scenery, and vehicles. Similarly, FETV identifies nine categories: people, animals, vehicles, plants, artifacts, food, building, scenery, and illustrations. These categories provide a solid foundation but often lack specificity in terms of motion and dynamic events. 

Generally, models perform significantly better on static prompts compared to dynamic ones. 
Our method introduces a detailed division into static, dynamic, and local movement. This refined classification allows for a more precise depiction of the inherent motion in each scenario, facilitating a more comprehensive and descriptive set of prompts. 



Specifically, we decompose the prompts into three components: foreground, background, and motion state. The foreground refers to the main subject of the event, including four categories: people, animals, plants, and man-made objects. The background refers to the environment and location where the event occurs, divided into indoor scenes, outdoor natural scenes, and outdoor man-made scenes. The motion state refers to the main motion pattern of the event, mainly divided into static, dynamic, and local movement. Each prompt needs to choose one or more elements from the components of foreground, background, and motion types. 



T2V Methods Selection.
We select six SOTA text-to-video models, including Gen-2 <cit.>, Hotshot-XL <cit.>, Video Fusion <cit.>, Video Crafter <cit.>, Text2Video-Zero <cit.>, and Tune-a-video <cit.>, to generate the videos for each prompt. So, there are a total of 2,808 AIGC videos in LGVQ.
The detailed information of videos generated by six T2V methods is shown in Table.<ref>. 



 §.§ Subjective Quality Assessment Experiment








We conducted a subjective quality assessment on the LGVQ dataset to derive quality labels for each AIGC video. This experiment targeted the investigation of main distortion types in AIGC videos on perceptual quality, as highlighted in Fig.<ref>, including three critical quality aspects: (1) Spatial quality focuses on the visual appearance of individual frames. (2) Temporal quality evaluates the coherence across video frames. (3) Text-to-video alignment assesses the correspondence between the video content and the accompanying text prompt.

Before the formal assessments, participants underwent a training session where they reviewed sample videos that were not included in the formal experiment. This session was intended to familiarize them with the evaluation criteria and the rating interface. Example distortions and quality attributes were discussed to calibrate participant expectations and rating standards. 

We follow the recommended method in  <cit.> to process the raw subjective ratings. First, the rating of an image is considered as an outlier if it is far from 2 (if normal) or √(20) (if nonnormal) standard deviations about the mean rating of that image. A subject with more than 5% outlier ratings is also rejected. 
The MOSs distribution is illustrated in Fig. <ref>, highlighting the variability and trends across the dataset. 



 §.§ MOSs Analysis


The MOSs of Different Generation Elements.
We calculated the average MOS across six T2V generation models to evaluate the strengths and weaknesses of current mainstream T2V models in generating various objects and actions. The results are shown in Fig. <ref>. For the foreground elements, the spatial and temporal quality of the human category is relatively low, which may be due to the complexity of human actions, postures, and expressions. In contrast, the spatial quality of animals, plants, and man-made objects is satisfactory. In terms of motion state, whether it is spatial, temporal, or T2V alignment quality, static objects clearly outperform locally moving objects, and fully moving objects perform the worst. This indicates that when the subject needs strong temporal changes, its temporal quality will decrease, and the spatial quality of this moving subject will also be affected. In terms of alignment quality, static and locally moving objects align better with the text prompts, whereas fully moving objects present increased challenges in alignment, leading to lower scores. For background elements, natural scenery performs best in terms of spatial and temporal quality, followed by indoor scenes and man-made outdoor scenes.






The MOSs of Different Generation Methods.
We also analyze the generation effect of six T2V models from three evaluation dimensions: spatial, temporal, and alignment. The results are shown in Fig. <ref>. For spatial quality, Gen-2 demonstrates outstanding performance across foreground, background, and motion state categories, surpassing all other models significantly. Text2Video-Zero follows closely, ranking second in all categories. Hotshot-XL and Video Fusion are comparable in quality. Video Crafter and Tune-a-video exhibit the poorest overall performance. For temporal quality, Gen-2 continues to excel overall. Following closely are Hotshot-XL and Video Fusion, which exhibit similar quality levels. Video Crafter performs moderately in terms of temporal quality. In addition, we found that the performance of the static motion of Hotshot-XL, Video Fusion, and Video Crafter is much higher than their performance of the dynamic motion, indicating that they are difficult to generate high-frequency movement. The Text2Video-Zero and Tune-a-video models perform the worst in this aspect. For text-to-video alignment, the differences between the 6 models are relatively small. Comparing all 10 aspects, the overall quality ranking of the 6 models from high to low is Gen-2, Hotshot-XL, Video Fusion, Text2Video-Zero, Video Crafter, and Tune-a-video.



§ AIGC VQA BENCHMARK


To provide a comprehensive and detailed evaluation for T2V models, we create this benchmark by incorporating a wide range of quality metrics. Our benchmark allows for an in-depth assessment of various aspects of AI-generated video, including spatial and temporal quality, as well as text-to-video alignment. This benchmark not only highlights the strengths and weaknesses of existing models but also drives innovation by identifying areas that require improvement.



 §.§ Compared Quality Metrics.
 

For spatial quality, we test two handcrafted-based IQA models, i.e. NIQE <cit.>, and BRISQUE <cit.>, and six deep learning-based IQA methods including UNIQUE <cit.>, MUSIQ <cit.>, StairIQA <cit.>, CLIP-IQA <cit.>, LIQE <cit.>,  CLIP-IQA+ <cit.>. 
For temporal quality, we test two handcrafted-based VQA models, i.e. TLVQA <cit.> and RAPIQUE <cit.>, and six deep learning-based VQA models, i.e. VSFA <cit.>, SimpleVQA <cit.>, FastVQA <cit.>, DOVER <cit.>, VIDEAL <cit.> and PatchVQ <cit.>. 
For text-to-video alignment, we evaluate four CLIP-based methods including CLIP <cit.>, CLIPScore <cit.>, BLIP <cit.>, and viCLIP <cit.>, and four text-to-image alignment assessment methods including ImageReward <cit.>, PickScore <cit.>, HPSv1 <cit.>, and HPSv2 <cit.>.





 §.§ Results
 We present the results in Table <ref>, from which we observe that across all three quality dimensions, the performance of all quality metrics is notably poor on the LGVQ dataset. 
For spatial quality metrics, the method MUSIQ, trained on the KonIQ-10K dataset, shows a notably higher SRCC value of 0.389, indicating relatively better performance in capturing the image quality perturbations specific to AI-generated content. Conversely, the method LIQE exhibits the lowest SRCC score,

possibly due to its reliance on less robust feature representations for complex image distortions typical of AIGC videos.
For temporal quality metrics, the VSFA model trained on the KonViD-1k dataset, leads with an SRCC of 0.295, suggesting mediocrity in dealing with temporal quality variations. On the other hand, the DOVER model scores lower with an SRCC of 0.254, which may stem from its approach that might be less adaptive to the specific temporal dynamics and distortion patterns present in AIGC videos.
For text-to-video alignment, the method viCLIP, leveraging a large-scale dataset InternVid-10M, achieves an SRCC of 0.280, showing poor performance in aligning textual descriptions with video content. However, the method BLIP shows a lesser SRCC value of 0.254, possibly indicating challenges in handling the intricate semantic relationships required for effective text-to-video mapping in AIGC scenarios.


Overall, the performance of all quality metrics is poor on the LGVQ dataset. This suggests that existing methods might not yet be fully equipped to handle the unique spatial and temporal distortion types associated with AIGC videos. Additionally, the lower performance across the board might also indicate a fundamental limitation in current methodologies to fully comprehend and evaluate the diverse and complex action concepts depicted in AIGC videos.








§ PROPOSED METHOD


In this session, we introduce UGVQ (Text-to-Video Unify Generated Video Quality), a structured framework designed to comprehensively assess video quality. The UGVQ comprises three key modules: a feature extraction module, a text-visual feature fusion module, and a quality regression module. 



 §.§ Quality-aware Feature Extraction


To accurately evaluate the quality of AIGC video regarding spatial, temporal, and alignment, we extract the features related to the quality of these three aspects.


  §.§.§ Spatial-aware Feature

Given a video v composed of n frames, v = {v_1, v_2, …, v_n}. First, we sample k key frames from n video frames. Then, we use the frame-level spatial feature extractor to get the spatial feature of k key frames. The set of the frame-level spatial features is shown in Eq (<ref>),

    {S_f(v_i), i∈{1,2,…,k}},

where S_f donate the frame-level spatial feature extractor. 


A video is composed of multiple frames (images), however, the spatial feature of a video is not a simple combination of the spatial feature of the frames.
Therefore, we train a video-level spatial feature extractor S_v to obtain the time series feature of a set of video frames as the spatial features of a given video v with n frames. The structure diagram is shown in Fig.<ref>. We split the features of the frames according to different scales, then use a feature extractor to obtain the video features of the two scales respectively, and finally use MLP to fuse these two scales. The formula is shown as Eq (<ref>),

    s_v = S_v({S_f(v_i), i∈{1,2,…,k}}),

where S_v represents video-level spatial feature extractor, and s_v represents video-level spatial feature of video v.



  §.§.§ Temporal-aware Feature

We use the pre-trained action recognition model as a temporal-aware feature extractor to obtain the motion features of each video. Action recognition networks <cit.> are commonly used to obtain features related to motion detection and motion classification, which can largely recognize the phenomenon of motion and frame discontinuity in AIGC videos. Therefore, for video v and action recognition network, we can get temporal features t_v by Eq (<ref>). 

    t_v = T(v),

where T donate the action recognition network.



  §.§.§ Alignment-aware Feature



For the AIGC video, the alignment of the prompt with the generated video content is an important aspect of quality assessment. Obtaining semantic features of the prompt and generated video is the key to judging the alignment. In Session <ref>, we mentioned that CLIP <cit.> is an effective method to evaluate the similarity of text and image. In this work, we also use CLIP-based methods <cit.> to obtain semantic features of prompt and video frame respectively. 
For a given prompt p and corresponding video v_p, we can obtain the text feature through text semantics extractor E_p, as shown in Eq (<ref>),

    e_p = E_p(p),

where e_p represent the semantic feature of prompt p.




 §.§ Feature Fusion Module





We propose the feature fusion module to fuse three kinds of features extracted in Section <ref> to enhance quality-aware feature representations. For a given prompt p and the corresponding generated video v_p, we can obtain the spatial feature s_v of video v by Eq (<ref>) and Eq (<ref>), the temporal feature t_v by Eq (<ref>), and the prompt semantic feature e_p by Eq (<ref>). 
Then, we fuse e_v and e_p by Eq (<ref>),

    j_1 = FFM(s_v, e_p),

which can enhance the feature expression of the alignment-aware features. 
In addition, we fuse temporal features of the video t_v and prompt semantics e_p to improve the performance of alignment and temporal quality, as shown in Eq (<ref>),

    j_2 = FFM(t_v, e_p),

where FFM denate the feature fusion module. 

We finally fuse the above two fuse features to obtain the fusion feature by Eq (<ref>)

    j = JFM(j_1, j_2),

where j donate the fused feature of j_1 and j_2.

The overall framework of the feature fusion module is shown in Fig(<ref>), and the fuse block is shown in Fig(<ref>). 







 §.§ Quality Regression Module


Through the above research, we obtained 3 quality-related features and one fusion feature, namely the spatial-aware feature s_v, the temporal-aware feature t_v, the alignment-aware feature e_p, and the fusion feature j. 
To maximize the performance, we concat all four features and input them into the quality regression model. The regression model was implemented by two MLP layers. The quality regression model will ultimately output three scores as spatial, temporal, and alignment quality of the prompt and the corresponding video, as shown in Eq (<ref>),

    s_s, s_t, s_a = QRM( concat( s_v, t_v, e_p, j ) ),

where QRM donates the quality regression module, and s_s, s_t, s_a represent the score of the spatial quality, temporal quality, and text-to-video alignment.



§ EXPERIMENTS




 §.§ Experiment Settings




  §.§.§ Test Dataset

We validate our proposed method on LGVQ, and the other two published datasets, FETV <cit.> and MQT <cit.>. The FETV dataset contains 2,476 videos, which were generated by 619 prompts and 4 text-to-video models
with 4 quality dimensions, static quality, temporal quality, overall alignment, and fine-grained alignment. 
In this paper, we use static quality, temporal quality, and overall alignment for our experiments. 
The MQT <cit.> dataset contains 1,005 videos, which were generated by 201 prompts and 5 text-to-video models
with 2 quality dimensions, alignment, and perception. In our experiments, we change the number of nodes in the last MLP in the quality regression module to predict the alignment and perception quality in MQT <cit.>.









  §.§.§ Training Protocols


To validate the performance of our proposed method comprehensively, all experiments use a training/validation/testing mode. The model is selected based on the best performance on the validation set, and the chosen model is then evaluated on the test set. The average results from all trials represent the final performance, reflecting the generalization ability of our method. In our experiment, the ratio of the training, validation, and testing sets is approximately 7:1:2.
Additionally, for all three datasets, there were duplicate prompts for video generation, meaning one prompt corresponded to multiple videos. To validate the generalization of our method in prompt-related tasks, all experiments use invisible prompt mode, which means that the prompt in the test or validation set does not appear in the train set.





 §.§ Performance Comparison


The performance of the IQA-based, VQA-based, and alignment-based models on proposed LGVQ databases are listed in Table. <ref>. 
The performance shows that the proposed method has excellent performance on temporal quality, which indicates that the UGVQ has a good effect on detecting the frame stabs and motion incoherence in AIGC video, and satisfies human subjective perception.
For spatial quality, we do not directly use IQA-related pre-trained models to generate spatial-aware features, but our method still performs well in terms of spatial quality. 
The alignment quality is relatively poor among the three aspects, which may be because it is inherently a cross-modal task, and eliminating the spacing between modalities is more difficult than the other two aspects. In addition, we directly used the pre-trained CLIP-based model to extract semantic features of video frames and prompts without training, resulting in a lack of discriminative features.


We conducted experiments on two other T2V datasets, MQT <cit.>, and FETV <cit.>, respectively. We compare IQA-based, VQA-based, and alignment-based models on FETV and MQT datasets, and the performance results are shown in Table.<ref>. Experimental results show that the UGVQ performs better on FETV and MQT datasets.




 §.§ Ablation Study








We conduct ablation experiments in this section to verify the effectiveness of the 3 main components in our UGVQ method, i.e., the overall architecture of UGVQ, the video-level spatial feature extraction, and the feature fusion module. Moreover, we further verify the effect of each feature on the performance of various qualities. The performance results are shown in Table.<ref>.

Through No.1, No.2, and No.3 experiments in Table.<ref>, we find that frame features, text features, and motion features all have a certain correlation with the quality of spatial, temporal, and text-video alignment. Among them, 

text features have a low correlation with the quality of all three aspects due to the lack of corresponding video or frame features. 
Through No.4, we found that the combination of text features and frame features significantly improved the performance in text-video alignment.

Through comparison with No.4 and No.5, it was found that the temporal performance in No.4 was still adversely affected due to the lack of motion features. In No.5, motion features significantly improved temporal performance, which indicates that the motion features extracted by the action recognition model have a good effect on detecting the frame stabs and motion incoherence in AIGC video.

Through No.5, No.7, and No.8, we found that our proposed video-level spatial feature extractor and feature fusion module effectively improved the performance of the three aspects of quality, and proved the effectiveness of the video feature extractor and feature fusion module.






 §.§ Cross-Dataset Evaluation

To verify the generalization performance of our proposed method, we conducted cross-dataset experiments on other datasets. We set up two sets of experiments. First, we train the UGVQ model on all the data in our proposed LGVQ dataset and then test the performance on all the FETV data. Second, we train the UGVQ model on the FETV dataset and then test it on LGVQ data. 

We run the same experiments on IQA-based, VQA-based, and alignment-based methods. The performance results are shown in Table.<ref>. Compared with the IQA-based, VQA-based, and alignment-based methods, we can find that our method has the best spatial, temporal, and alignment performance. However, numerically, it still needs to be improved.




§ CONCLUSION


In this paper, we establish a new quality assessment dataset (i.e. LGVQ) and benchmark for AIGC videos. LGVQ consists of 2,808 AIGC videos generated by 6 T2V generation methods using 468 text prompts. We conduct a subjective quality assessment experiment on LGVQ, evaluating the videos from three dimensions. Then, we test IQA, VQA, and text-to-image alignment methods on the LGVQ dataset, revealing their limitations in measuring the perceptual quality of AIGC videos. To address this issue, we develop a unified generated video quality assessment (i.e. UGVQ) model capable of simultaneously evaluating the three quality dimensions of AIGC videos.


IEEEtran










 Paper Title: ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models.




Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition
    Jiang Li, Xiaoping Wang, Senior Member, IEEE, and Zhigang Zeng, Fellow, IEEE
        
        
Manuscript received 30 July 2024; revised 30 July 2024; accepted 30 July 2024. This work was supported in part by the Interdisciplinary Research Program of HUST under Grant 2024JCYJ006, the National Natural Science Foundation of China under Grant 62236005 and 61936004, and the Fundamental Research Funds for the Central Universities of HUST under Grant 2024JYCXJJ067 and 2024JYCXJJ068. (Corresponding author: Xiaoping Wang.)
The authors are with the School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China, also with the Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan 430074, China, also with the Hubei Key Laboratory of Brain-Inspired Intelligent Systems, Huazhong University of Science and Technology, Wuhan 430074, China, and also with the Key Laboratory of Image Processing and Intelligent Control (Huazhong University of Science and Technology), Ministry of Education, Wuhan 430074, China (e-mail: lijfrank@hust.edu.cn; wangxiaoping@hust.edu.cn; zgzeng@hust.edu.cn).
Digital Object Identifier

    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
    
    
    In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through learnable latent variable optimization. We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MLP output during inference, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point.
    
    The results demonstrate that our method exhibits out-of-domain generalization and interpretability. Code: <https://github.com/mrwu-mac/ControlMLLM>.





§ INTRODUCTION

In recent times, there has been a surge in the development and adoption of large language models (LLMs), such as GPT-4 <cit.> and Llama <cit.>, showcasing remarkable capabilities in addressing a wide range of human-generated questions. The success of these models has sparked interest among researchers in exploring the integration of LLMs with visual inputs. Consequently, a new class of models known as Multimodal Large Language Models (MLLMs) has emerged <cit.>.
However, despite their widespread adoption, traditional MLLMs often face limitations due to their reliance on coarse image-level alignments. This restricts users to guiding MLLMs solely through text prompts for detailed region description and reasoning. However, text often fails to capture the intricate visual nuances present in an image. 

Addressing this challenge, recent efforts <cit.> have pioneered the integration of referring abilities within MLLMs, which enables users to provide input by pointing to specific coordinates of the objects or regions, as shown in Figure <ref> (left).  However, these endeavors typically entail substantial training costs to equip MLLMs with referring capabilities. Additionally, the model must undergo retraining to adapt to new data domains or new base MLLMs. 


In this work, we propose a training-free method to inject the visual prompts into the Multimodal Large Language Models via learnable latent variable optimization. 

The method originates from our observation of the attention maps from the MLLM decoder, which model the relationship between the pixels and text prompt tokens and encompass rich semantic relations that significantly influence the generated text.

However, MLLMs typically involve fine-tuning an MLP layer to bridge the gap between visual and linguistic representations, which means that the output of the MLP layer can indirectly impact the relationship between text prompt tokens and pixels in the attention layers of the MLLM decoder, thereby altering the model's output.

Thus, our key idea is that we can alter the outputs of MLLMs by adjusting the visual tokens from the MLP output during the inference process, controlling which text prompt tokens attend to which visual tokens in the attention layers. Specifically, we augment visual tokens with an additional learnable latent variable.

Subsequently, we optimize the learnable latent variable based on an energy function designed to enhance the strength of the referring regions in the attention map between the text tokens and the visual tokens.

Our method enables referring MLLMs with various visual prompts, including box, mask, scribble and point, and does not require model training, fine-tuning, or extra data.
We also demonstrate that our method exhibits out-of-domain generalization and interpretability. 









§ RELATED WORK





  
MLLMs
Motivated by the accomplishments of Large Language Models (LLMs) <cit.>, there is a burgeoning trend among researchers to develop a diverse range of Multimodal Large Language Models (MLLMs) <cit.>. These MLLMs typically comprise a visual encoder, a language decoder, and an image-text alignment module. The visual encoder and the language decoder are often sourced from pre-trained models, such as CLIP <cit.>, DINOv2 <cit.>, Llama <cit.>, and Vicuna <cit.>. Meanwhile, the image-text alignment module is trained on image-text pairs and fine-tuned through visual instruction tuning to enhance its visual conversation capabilities. These Multimodal Large Language Models (MLLMs) often confront limitations stemming from their reliance on coarse image-level alignments.



  
Referring MLLMs
In recent research, there has been a noticeable trend towards integrating foundation models with tasks involving referring dialogue. These models <cit.> introduce spatial visual prompts as extra input and are trained using region-text pairs. By leveraging this approach, they effectively bridge the gap between textual prompts and visual context, enabling comprehensive understanding of image content at the regional level. However, these methods inevitably require a substantial training burden.



  
Training-free Control in Text-to-Image
There are numerous works on controllable text-to-image generation, among which training-free methods <cit.> are most relevant to our research. Among them, Prompt-To-Prompt <cit.> explore the role of attention in text-visual interactions in Stable Diffusion <cit.> model, while Layout-Guidance <cit.> indirectly bias attention in Stable Diffusion model by optimizing an energy function. These contributions significantly inform our investigation into enhancing controllability and interpretability in MLLMs.



  
Visual Prompt
The visual prompt can be categorized into two main techniques: hard prompt and soft prompt.
The hard visual prompt works <cit.> that direct the model's attention to the region or enable visual grounding abilities in the Multimodal Models in a training-free and convenient manner by directly manipulating images. However, these methods inevitably compromise the structural information of the images, or a strong understanding of the corresponding patterns by the model is required. In contrast, the soft visual prompt works <cit.> integrate learnable visual prompts into models to adapt them for different downstream tasks. However, these methods do not support region guidance and require fine-tuning the model with downstream data.

In contrast, we optimize a learnable latent variable to support referring MLLM in the test time, without any downstream training data required, and TPT <cit.> is most related to our work. 








§ BACKGROUND








  
Multimodal Large Language Models (MLLMs):
The MLLMs typically consist of a visual encoder, an LLM decoder, and an image-text alignment module. Given an image I, the frozen vision encoder and a subsequent learnable MLP are used to encode I into a set of visual tokens e_v. These visual tokens e_v are then concatenated with text tokens e_t encoded from text prompt p_t, forming the input for the frozen LLM. The LLM decodes the output tokens y sequentially, which can be formulated as: 

    y_i = f(I, p_t, y_0, y_1, ⋯, y_i-1).


Considering LLaVA-liked <cit.> MLLMs, the LLM in MLLMs typically employs a transformer model <cit.> with the attention layer as its core. In such model, the attention maps represent the relationships between the visual tokens and the text prompt tokens. The attention map in attention layer τ, computed on the transformed visual-text concatenated embeddings [e_v, e_t]^(τ), is obtained as follows:




    A^(τ)=softmax([e_v,e_t]^(τ)· ([e_v,e_t]^(τ))^T/√(d_k)),

where d_k is a scaling factor. A^(τ) consists of A^(τ)_ij with i, j ∈{1, ⋯, n}, representing the relationship between the i-th token and the j-th token, and their impact on the output.



  
Training Referring MLLMs:
The objective of training referring MLLMs is to inject the visual prompt r into the MLLMs to achieve referring ability via model parameter learning. The visual prompt r can take various forms, such as a box, mask, scribble, or point, to indicate specific locations or regions within the image. 



Current referring MLLMs typically need to be fine-tuned on a training set with positional annotations before they can be effectively used. The fine-tuning process involves maximizing the log likelihood of generating the text conditioned on I, p_t, and r over the entire training dataset. This can be formulated as:

    θ^* = max_θ∑_i=1^U log P(y_i | I_i, p_t, r, y_0, y_1, ⋯, y_i-1; θ),

where θ represents the parameters of the model f, and U is the number of samples in the training set. This method significantly enhances the model's fine-grained understanding and interactivity. However, it incurs high training costs. Additionally, such fine-tuning strategies result in domain-specific behaviors, which have been shown to compromise the out-of-distribution generalization and robustness of MLLMs <cit.>. Therefore, when domain shifts occur, the model needs to be retrained, leading to a lack of flexibility.














§ METHOD



We aim to propose a training-free method to overcome the inconveniences of traditional training. Training-free referring MLLM maintains the model parameters θ frozen, eliminating the need for any training or fine-tuning with samples from the training set. During inference, the only information available is the single test sample without label information, as shown in Figure <ref> (right).


In this section, we explore and design a solution to address the challenges of Training-free Referring MLLMs. The key task is to flexibly embed visual prompts during the inference phase while maintaining the model's reasoning capabilities. To begin with, we delve into the mechanism of MLLMs (see Sec. <ref>), our key observation is the attention mechanism in LLM capturing the relationship between the model's output and the input pixels. Further, the visual tokens inputted into the LLM influence the values of the attention maps to indirectly control the model output. Building on this analysis, we propose the Latent Variable learning (a test-time prompt tuning strategy <cit.>, see Sec. <ref>) to edit the visual tokens, as shown in Figure <ref>. This method effectively integrates visual prompts into pre-trained MLLMs, enabling fine-grained visual reasoning.







 §.§ Analysis of the Attention in LVLMs


We begin by analyzing which factors in the model truly capture the relationship between input and output? In other words, we seek to understand how to interpret the association between the model's output and the input pixels. 


As demonstrated by Equation <ref>, Multimodal Large Language Models (MLLMs) fundamentally model the maximum likelihood output based on visual input and text prompts. By conditioning on the text prompt, the model can determine which parts of the image have the greatest impact on the output. Building on the discussions in the Sec. <ref> and illustrations in the Figure <ref> (top line), we can observe that the attention map models the influence of visual tokens on the output conditioned by the text prompt. Therefore, the attention map in MLLMs not only provides interpretability regarding the relationship between model output and input pixels but also facilitates guiding the model's output. 


A natural idea is that we can directly alter the model's output by editing the attention maps. Inspired by IBD <cit.>, we achieve this by adding an adjustment coefficient η to the attention related to the visual tokens corresponding to the referring region, which can be formulated as,





    A^(τ)=softmax([e_v,e_t]^(τ)· ([e_v,e_t]^(τ))^T/√(d_k) + M), 
        M_i =η   if  i  in  r    else  0,

where M is a mask with the same shape as the attention map, r denotes the referring region.
However, we have to carefully select a suitable coefficient η for each example. When the η is too small, it leads to ineffective control (as shown in Figure <ref> a), and when it is too large, it can impact the language capabilities of the LLM (as shown in Figure <ref> c). Additionally, we found that it is sufficient to manipulate the attention map at the 0-th step during model inference (as shown in Figure <ref> a,b,c), as it is most directly associated with the text prompt, and manipulating attentions step by step also affects the expression of the LLM (as shown in Figure <ref> d). Overall, directly manipulating attention maps is not a viable approach because it overlooks the relationships between attention layers and not all layers' visual tokens decide the output <cit.>.

We note that in the most MLLMs, typically the MLP layer is trained for image-text alignment. This implies that MLLMs indirectly affect the values of the attention map by learning the parameters of the MLP layer to alter the visual tokens. In other words, the visual tokens inputted into the LLM directly influence the values of the attention maps.

It is also worth noting that the input text prompt also directly influences the model's output, particularly regarding non-visual-related <cit.> output content. However, we aim to explain the correlation between the output and the input image. Therefore, we do not consider the direct impact of the text prompt on the output in our analysis.







 §.§ Manipulating Attention via Latent Variable Learning


Based on the analysis above, our core idea is to indirectly influence the attention maps by editing visual tokens, thereby focusing on the referred regions. We achieve this by optimizing a learnable latent variable based on an energy function <cit.>, which calculates the relationship between the input referring and the attention maps. To do this, we first need to determine which attention maps to use. One approach is to use attention maps between each text prompt token and all visual tokens. However, because visual tokens typically have a significant impact on the result based on only a few most relevant text prompts (referred to as highlight text tokens), using all attention maps would be computationally redundant. Yet, for users, identifying the highlight text tokens can be challenging. Therefore, we simply average pool the attention maps generated for each text prompt token to represent the global context of the text prompt (referred to as the context token) and its association with visual tokens. We found that this simple method of using context tokens produces attention maps similar to those generated by highlight text tokens, as shown in Figure <ref> (bottom line). We leave the optimization based on highlight text tokens for future work.


Specifically, our method supports four types of referring shapes, including box, mask, scribble, and point. We employ two types of energy functions to respectively support these referring shapes: a hard mask-based energy function for box and mask referring, and a soft mask-based energy function for scribble and point referring.



  
Hard Mask-based Energy Function
We first zero initialize a learnable latent variable p_v with the same shape as e_v, and add it to the e_v. Then we can get N attention maps from N attention layers which model the relation between the context token and the novel visual tokens. Given the referring box or mask, we first convert it into a binary mask. Then, we compute the mask-based energy function based on the mask and the attention map A^(ct), which is obtained by averaging pooling from N attention maps. The energy function can be formulated as:

    E(A^(ct), r)=(1-∑_i ∈ r A_i^(ct)/∑_i A_i^(ct))^2,

where r denotes the referring region. Then the gradient of the loss <ref> is computed via backpropagation to update the learnable latent variable:

    p_v ←p_v- α∇_p_v E(A^(ct), r),

where α>0 is a hyperparameter controlling the strength of
 the guidance. By optimizing p_v through the Equation <ref>, we indirectly guide the attention maps to produce higher responses in the referring region r, thereby increasing the influence of the visual content of region r on the output.



  
Soft Mask-based Energy Function
Since scribble and point lack the concept of the region, 
it is optional to use an extra SAM <cit.> model to obtain a mask for applying the Hard Mask-based Energy Function. However, this incurs additional inference cost, so we also provide an optional soft mask-based energy function based on a distance matrix D, which is computed via applying the OpenCV <cit.> distanceTransform function on the given scribble or point. Then the soft mask-based energy function can be formulated as: 

    E(A^(ct), r)=(1-∑_i ∈ re^-D_i^2/2 σ^2/√(2 π)σ
     A_i^(ct)/∑_i A_i^(ct))^2,

where σ is the standard deviation of the Gaussian function, which is set to 0.1. By optimizing p_v through the Equation <ref>, the closer the region of attention map is to the given scribble or point, the higher the response.



Finally, we iteratively optimize the learnable latent variable T times at the 0-th step of model inference. 
In addition, to prevent overfitting, we employ Early Stop (ES) and Exponential Moving Average (EMA) strategies to enhance model stability.


More details are shown in Appendix <ref>.













§ EXPERIMENTS



 §.§ Experiment Details


Unless explicitly stated otherwise, the MLLM we use is LLaVA-v1.5-7B <cit.>, T=4, α=400 and β=0.5. All experiments are conducted on two RTX 3090 GPUs with 24 GB of memory each. 




 §.§ Applications



  
Referring with Different Visual Prompts.
We first demonstrate referring QA with different visual prompts, including box, mask, scribble and point in the Figure <ref>.  Our method consistently demonstrates significant controllability with four types of visual prompts. And our method improves the interpretability compared to basic model (column 3 vs column 2), demonstrates a stronger correlation between the attention response areas and the generated descriptions.




  
Other Applications. 
First, guiding the model to focus on specific regions, potentially helps the model mitigate hallucination issues, as shown in Figure <ref> (c,d output in orange color). Second, our method can also perform referring scene text recognition (OCR), as shown in Figure <ref>.




 §.§ Comparisons




  
Comparison on Referring Object Classification Task.
Following Ferret <cit.>, we use the Referring Object Classification (ROC) task to evaluate whether our method can accurately pinpoint and understand the semantic of the referring region. The task requires the model to correctly identify the target within the referring region. We follow the setting of Ferret to form 1,748 questions (in which 1,548 for test and 200 for validation) based on LVIS <cit.> validation dataset, with corresponding box, mask, scribble and point. 
We consider the edit attention with η=10 (as Equation <ref> and Figure <ref> (b)) as the baseline model. And we compare several training methods <cit.>. 
We also evaluate the lower and upper limits of LLaVA's recognition capability by assessing LLaVA without referring region, as well as background blur outside the referring region, which are presented in gray. Additionally, we evaluate a method that highlights regions with color as a comparable training-free method.
More details and the input examples are shown in Appendix <ref>. 




The results are shown in Table <ref>. Our method shows a better performance than the training method GPT4-ROI with box referring (60.59 vs 58.59) and the Shikra-7B with point referring (58.85 vs 56.27). However, due to the limitations of LLaVA's capabilities (as shown in the results of the LLaVA+Blur), we present a performance gap compared to the latest training method Ferret <cit.>. Our method also demonstrates superiority compared to training-free color prompt-based method and baseline method. 



  
Comparison on Referring Text Classification Task.
We consider the Referring Text Classification (RTC) task as the out-of-domain task, to verify the model's out-of-domain transfer capability. Similar to the ROC task, we formulate the problem as a binary classification task and construct 1,372 questions based on the COCO-Text <cit.> dataset. Since point and scribble referring methods are not suitable for text due to the non-connectivity of the text, we only evaluate the RTC task with box and mask referring. 

The results are shown in Table <ref>. All the training methods we evaluated exhibited poor out-of-domain generalization performance. Specifically, Ferret achieves only 55.47% accuracy on the RTC task, despite its excellent in-domain performance as shown in Table <ref>. In contrast, our training-free method still demonstrates the best out-of-domain generalization performance. We also present comparative examples of out-of-domain tasks, as shown in Figure <ref> and Figure <ref>.




 §.§ Ablation Study

The ablation studies primarily focus on the box referred object classification. Furthermore, inspired by DIFNet <cit.>, we calculate a relevancy between the model's output and pixels within the referring region to assess the extent to which the model's output is influenced by visual content within the region. More details and additional experiments are shown in Appendix <ref> and <ref>.




























    
















































  
Impact of T and α.
As shown in Table <ref>, as T increases, the relevancy between the model's output and the referring regions also increases. However, the larger T results in a decrease in the model's accuracy on the ROC task, also with excessively large relevancy scores, showing that excessively large relevancy scores also indicate overfitting of the learnable latent variable. 
Therefore, the value of the relevancy score provides us with guidance to alleviate model overfitting, particularly when the relevancy score is around 0.18, typically resulting in better performance.
And the value of α affects the convergence speed of optimization, with larger α also leading to overfitting of the model.



  
Impact of EMA and ES.
As shown in Table <ref>, when equipped with a smaller β value, it effectively mitigates the overfitting issue associated with the learnable latent variable. For instance, with α=400 and β=0.3, the model's performance improves from 53.5 to 62.5. 
However, a smaller value of β also results in slower convergence of the learnable latent variable. Therefore, we combine the Early Stop strategy, allowing us to use a slightly larger β to accelerate the convergence of the learnable latent variable.
After incorporating the early stop strategy, 

we can opt for slightly larger T to ensure that the model is adequately optimized on challenging samples. The early stop strategy allows us to attain superior model performance while reducing the impact of overfitting. The additional experiment about ES is shown in Table <ref>.



  
Impact of Different Text Prompts and Size of Visual Prompt. 
As shown in Figure <ref> and Figure <ref>, combining with the clear and specific text prompt and suitable visual prompt size typically results in better controllability.




§ LIMITATIONS


While we have demonstrated visual prompt control by optimizing only visual tokens, our technique is subject to a few limitations. First, there is some additional inference overhead, while various engineering approaches (such as Ollama [https://ollama.com/]) can significantly speed up the process. Therefore, this limitation can be reasonably overlooked. Second, our method is applicable only to white-box models and relies on the basic capabilities of the models themselves. However, our approach is orthogonal to these ongoing advancements in foundational models. Third, currently, our method supports only a single region visual prompt, extending this to multi-region control is a direction for future work. Fourth, our current optimization strategy is relatively simple, and the selection of text prompts can also affect the optimization results. We plan to focus on improving this aspect in future research.



§ CONCLUSION

In this work, we present a training-free method to integrate visual prompts into Multimodal Large Language Models (MLLMs) through learnable latent variable optimization. By adjusting visual tokens during inference, our approach enhances the attention to referring regions, enabling detailed descriptions and reasoning without additional training costs. Our method supports various referring formats such as box, mask, scribble, and point. The results show that our approach demonstrates strong out-of-domain generalization and interpretability, making it a promising direction for embedding referring abilities into MLLMs.






This work was supported by National Science and Technology Major Project (No. 2022ZD0118201), the National Science Fund for Distinguished Young Scholars (No.62025603), the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. 62072389, No. 62302411), China Postdoctoral Science Foundation (No. 2023M732948), the Natural Science Foundation of Fujian Province of China (No.2022J06001), and partially sponsored by CCF-NetEase ThunderFire Innovation Research Funding (NO. CCF-Netease 202301).



plain








































§ BROADER IMPACT


The integration of Multimodal Large Language Models (MLLMs) has far-reaching implications across various sectors. These models enhance accessibility, improve education, advance healthcare, and revolutionize media and entertainment. They offer intuitive interfaces for diverse communication needs, assist in medical diagnosis and treatment, and enable immersive multimedia experiences. However, ethical considerations must be addressed to ensure equitable and responsible deployment. Collaboration among researchers, policymakers, and industry is essential to maximize the societal impact of MLLMs.



§ APPENDIX / SUPPLEMENTAL MATERIAL




 §.§ Details of EMA and ES




  
EMA. 
Model weight Exponential Moving Average (EMA) is a technique used to stabilize the training of deep neural networks by maintaining a smoothed version of the model parameters. It calculates the moving average of the model weights by giving more weight to recent updates while gradually decreasing the influence of past updates. Mathematically, EMA is defined as:


    θ_EMA^(t) = β·θ^(t) + (1 - β) ·θ_EMA^(t-1),


where θ_EMA^(t) is the EMA of the model weights at time t,
 θ^(t) is the model weights at time t,
 θ_EMA^(t-1) is the EMA of the model weights at the previous time step,
 β is the smoothing factor, ranging from 0 to 1.
EMA helps to stabilize the training process by reducing the variance of the parameter updates, which can prevent the model from diverging or oscillating during training. We employ the EMA on the learnable latent variable in our experiments during visual token optimization.



  
ES.  
Early Stop (ES) is a regularization technique commonly used during the training of machine learning models to prevent overfitting. The main idea behind ES is to monitor the performance of the model on a validation set during training and stop the training process if the performance begins to deteriorate. 
Specifically, in our experiments, ES tracks the loss on the test sample and halts the visual token optimization process if the metric does not improve for a certain number of consecutive epochs. Mathematically, ES can be described as follows:

Let L^(i) denote the loss at optimization process t, and let Δ represent a predefined threshold for acceptable loss degradation, then we

    Stop optimizing if  abs(L^(t) - L^(0)) / L^(0)≥Δ  for  T  consecutive process.

We set Δ=0.25 in our experiments. ES helps prevent overfitting by stopping the optimization process before the model starts to lose generalization ability. It provides a simple yet effective way to regularize the optimization process and improve the generalization performance of the model.






 §.§ Additional Experiment Details




  
Details of Relevancy Score.
To elucidate the influence of input visual pixels on outputs, we employ a technique known as the Relevancy Map. This map highlights the regions or features of the input data that contribute most significantly to the model's decision-making process. Specifically, the Relevancy Map assigns importance scores to different parts of the input, indicating their relative impact on the model's output. This interpretability tool not only enhances our understanding of the model's behavior but also facilitates error analysis and model debugging. More details can be found in the paper <cit.>.

In our experiments, we provide the relevancy scores alongside model predictions to provide insights into the model's decision-making process. In MLLMs, the typical use of Key-Value cache technique in LLMs, for convenience, we propagate the relevance from the model's first output token to the input of LLM ( e_v and e_t ). Then the relevancy scores corresponding to visual tokens are reshaped into a grid that matches the layout of the original image. Since relevancy maps are akin to attention maps and often exhibit significantly higher values in localized regions, taking the average may dilute their significance. So we extract the max value in the referring region of relevancy map as final relevancy score.

It is also worth noting that we found relevancy map plays a similar role to attention map, directly modeling the relationship between input and output of the model. This suggests that we may be able to directly control the model's output based on relevancy map. However, calculating the relevancy map requires computing the gradient for each relevant tensor in the model. For convenience, the approach presented in this paper only utilizes attention for implementation, while the relevancy map only be used to assess the extent of visual impact on the output.



  
Details of Implement.
We apply low-bit quantization to the basic model we implemented to further optimize memory usage. Nevertheless, we still achieve performance competitive with training-based methods (without quantization).



  
Details and Input Examples of ROC Task.
The box and mask are from the LVIS GT boxes and mask, the scribble and the point are randomly sampled inside the boxes. 
It is worth noting that we follow Ferret to choose negative object whose central point is close to the GT object. Although this is somewhat disadvantageous for us, we still achieve competitive performance compared to other methods. And we show the input examples of different methods on ROC task in Figure <ref>.





 §.§ Additional Experiments













  
Application on scene text recognition (OCR).
Results are shown in Figure <ref>. Our method can also perform referring region text recognition.





  
Examples on Out-of-Domain Task.
We present examples of the performance on out-of-domain tasks OCR and Screenshots. As shown in Figure <ref>, compared to Ferret, our method correctly identified the text in the referring region. Additionally, as shown in Figure <ref>, our method correctly recognized the app in the mobile screenshot, unlike Ferret.







  
Comparison on Referring Description Task.
We also validate our method through the Referring Description Task. Specifically, we construct the test set based on region-text pairs from the RefCOCOg test split and evaluate the method using traditional captioning metrics BLEU@4 (B@4) <cit.>, METEOR (M) <cit.>, CIDEr-D (C) <cit.>, and SPICE (S) <cit.>. However, it is important to note that these metrics are significantly influenced by the style and distribution of the model's output text. Typically, outputs that are more similar in distribution to RefCOCOg yield better results, while those with some unique styles lead to poorer results. Therefore, this experiment is only used for internal validation of the model and not for comparison between different models.
The results are shown in Table <ref>. Our method consistently improves the model's referring description performance.




  
Impact on Different MLLMs.
We validate our method on different MLLMs, including LLaVA-1.5-7B, LLaVA-1.5-13B, InstructBLIP-7B <cit.> and InstructBLIP-13B. The LLaVA-1.5-13B with a larger LLM with more layers compared to LLaVA-1.5-7B, and the InstructBLIP employs a cross-attention type image-text alignment module Q-Former. Specifically, in InstructBLIP, the interaction between visual tokens and text tokens occurs in Q-Former, so we utilize the cross-attention map from Q-Former for optimizing visual tokens. The results are shown in Table <ref>, our method exhibited poorer performance in LLaVA-1.5-13B, which may be attributed to the increased number of attention maps, making the optimization of visual tokens more challenging. Therefore, adopting different hyperparameters for different models may be essential. However, in InstructBLIP-7B and InstructBLIP-13B, our method consistently yielded performance improvements.



  
Inference Cost.
We compare the inference cost of our method and the basic LLaVA model. LLaVA + Ours model with T=5 and does not use Early Stop here. Results are shown in Table <ref>, when outputting only 7 tokens, our method noticeably adds approximately 2 seconds of inference time. However, when generating about 400 tokens, the proportion of the extra inference time produced by our method significantly decreases. When combined with an early stop strategy, the proportion of additional inference time will be further reduced. 



  
Impact of Different Text Prompt.
Results are shown in Figure <ref>. Different text prompts can significantly affect the performance of our method. For instance, our method fails when using an ambiguous text prompt like "describe the region in the image." However, it succeeds with a more specific referring text prompt such as "what is unusual about the region of the building?" Therefore, it is recommended to use clear and specific text prompts to achieve better control and performance.




  
Impact of the Size of Visual Prompt.
Results are shown in Figure <ref>. For box and mask prompts, when they do not fully cover the referring object, failure control may occur. This is because the highest attention response for the desired outcome may fall on any unexpected area of the object. Therefore, it is recommended to cover the object with a larger-sized visual prompt.




  
Comparing Highlight Token and Context Token based Optimization.
We compare Highlight Token and Context Token based Optimization as shown in Figure <ref>. Directly optimizing based on the highlight token is faster but also prone to overfitting. This may be due to the direct connection between the highlight token and visual token, while other text tokens contain redundant visual associations. However, this also ignores the potential influence of other text tokens.





  
More Examples of Referring MLLMs with Scribble and Point.
In Figure <ref>, we also show more examples of referring MLLMs with scribble (right) and point (left).










































 




























































































































    

























    






































































 Paper Title: MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions.


Instantons and the Large =4 Algebra
    Edward Witten
    
===================================



[1]These authors contributed equally to this work.

[2] Corresponding authors.
[3]Github repository:https://github.com/litwellchi/MMTrailhttps://github.com/litwellchi/MMTrail
[4]Project Page:https://mattie-e.github.io/MMTrail/https://mattie-e.github.io/MMTrail/



§ ABSTRACT 


Massive multi-modality datasets play a significant role in facilitating the success of large video-language models. 
However, current video-language datasets primarily provide text descriptions for visual frames, considering audio to be weakly related information. They usually overlook exploring the potential of inherent audio-visual correlation, leading to monotonous annotation within each modality instead of comprehensive and precise descriptions. Such ignorance results in the difficulty of multiple cross-modality studies.
To fulfill this gap, we present MMTrail, a large-scale multi-modality video-language dataset incorporating more than 20M trailer clips with visual captions, and 2M high-quality clips with multimodal captions. 
Trailers preview full-length video works and integrate context, visual frames, and background music. In particular, the trailer has two main advantages: 
(1) the topics are diverse, and the content characters are of various types, , film,  news, and gaming.
(2) the corresponding background music is custom-designed, making it more coherent with the visual context. 
Upon these insights, we propose a systemic captioning framework, achieving various modality annotations with more than 27.1k hours of trailer videos. Here, to ensure the caption retains music perspective while preserving the authority of visual context, we leverage the advanced LLM to merge all annotations adaptively.
In this fashion, our MMtrail dataset potentially paves the path for fine-grained large multimodal-language model training. In experiments, we provide evaluation metrics and benchmark results on our dataset, demonstrating the high quality of our annotation and its effectiveness for model training. 


   

    




§ INTRODUCTION


AI-driven movies and shot video production have a wide range of applications in people's daily lives.
Clearly, creating vivid videos requires more than just visual frame generation or individual modality-based ones. 
Thanks to various large-scale video-language datasets, numerous generative multimodal large language models have been developed to achieve this goal <cit.>.
However, the existing video-language datasets <cit.> typically focus on visual-based text descriptions, and they overlook the significance of the inherent visual-audio dependencies.
It presents a complex challenge that demands cohesive integration of multiple modalities, yet remains largely unexplored.






Collecting high-quality multi-modality source data that preserves consistency between different modalities is challenging. 
Unlike previous datasets that only provide visual frame-based caption <cit.>, multimodal datasets contain complex data formats (, music), resulting in more labor-intensive and time-consuming costs in data processing and annotation.
Moreover, achieving a high correlation between audio and visual content presents challenges.



r0.5
    


    < g r a p h i c s >

Distribution of video categories of MMTrail dataset.


    

Targeting to fill the dataset gap by creating a comprehensive and accurate multi-modality visual-audio dataset, we first notice trailers. As a precursor to a full-length work, the video trailer has emerged as a vital tool for artists to showcase and disseminate their creations. These short videos typically combine the most compelling visual shots with carefully selected music, have high cross-modality consistency, and hold significant potential in broader multimodal research. The topics are diverse, and the content characters are of various types, e.g., film, comedy, and gaming, as shown in  <ref>. Significantly, the trailer format represents a unique, high-quality, video-centric multimodal data source that benefits further multi-modality research exploration and analysis.





In this work, we propose MMTrail, which aims to unlock the potential of multimodal content understanding and generation for innovative applications in video content generation. We first recognize the immense value of trailers as a video-centric dataset, especially considering the music alongside the videos. MMTrail contains 20M+ video clips from 290k trailer videos encompassing various source categories as shown in <ref>. To ensure the quality of our dataset, we have carefully designed a robust data filtering and cleaning methodology. We also provide extensive statistics works to demonstrate the diversity and complexity of our dataset.


To address the multimodal to language annotation challenge, we have designed a multimodal captioning pipeline incorporating diverse state-of-the-art (SOTA) captioning models <cit.>. Furthermore, we propose a language model fusion strategy to generate fine-grained multimodal captions. We have performed small-scale annotations on the entire dataset, created a multimodal annotation subset of 3 million samples MMTrail-2M, and provided a testing set MMTrail-Test with manually-adjusted multimodal caption.


We present evaluation metrics and benchmark results on our dataset, demonstrating the high quality of our annotations and their effectiveness for model training. Through extensive experiments and benchmarking, we showcase the difficulty and diversity of our dataset using various evaluation metrics. We also conduct human evaluations to validate the quality of our multimodal captioning pipeline. Furthermore, we fine-tune understanding models <cit.> and generative models <cit.> on a subset of our dataset, providing evidence of its high quality and efficacy. Additionally, we evaluate video understanding models on the MMTrail-Test, highlighting the challenges posed by our dataset, and evaluate video-music-based models to demonstrate the effectiveness of cross-modality tasks.







§ RELATED WORK







 §.§ Video generation and understanding

Video understanding and text-to-video generation are inherently connected tasks. In recent years, there has been remarkable progress in understanding models <cit.>, which have greatly contributed to the advancement of text-based video generation techniques. The availability of large-scale datasets and diffusion models has revolutionized video generation, moving from pixel-level approaches like <cit.> to latent-level video diffusion models <cit.>. Concurrently, understanding models have also witnessed significant improvements. A series of MLLM-based understanding models <cit.> has reach a satisfied understanding abilities. Besides, some surveys give detailed summaries of this area from different application areas or aspects <cit.>. The iterative interaction between video generation and understanding has led to the development of excellent large-scale datasets and models encompassing diverse approaches. Panda <cit.> introduced an auto-caption model distilled from video understanding models like VideoLlaMA <cit.>, MiniGPT4 <cit.>. Furthermore, leading to a large amount of multimodal generation models <cit.>.







 §.§ Video-Language datasets


Captioned video datasets are essential for text-to-video generation and understanding tasks. MSR-VTT <cit.>, UCF-101 <cit.> are commonly used as evaluation sets. Anna et al. <cit.> presented 118,081 movie clips with descriptions. ActivityNet Caption <cit.> by Ranjay et al. is a benchmark involving event detection, natural language description, and event localization. WebVid <cit.>, VideoFactory <cit.>, and other works <cit.>, contain multilingual video descriptions, video clips, metadata such as titles, descriptions, tags, and channel names, and are used for tasks like video understanding, text-to-video retrieval, and audio-video captioning with weak annotations. Several datasets and approaches have utilized audio to enhance video captioning <cit.>. These datasets consist of movies, web videos, YouTube videos, and high-resolution videos from popular YouTube categories, providing transcribed audio descriptions, narrations <cit.>, ASR transcriptions <cit.>, and multiple captions generated through a auto caption model <cit.>.

However, existing works only use audio to enhance the video caption, using metadata or ASR to provide extra information for the video caption. A large-scale video-centric dataset that cooperates with high-quality music and multimodal caption is still lacking. 




§ MMTRAIL-20M DATASET


To boost the performance of multimodal generation tasks, we construct a high-quality multimodal dataset based on trailers, which offers a wealth of multimodal information and diverse categories that distinguish them from existing large-scale video-language datasets.









 §.§ Data Collection Pipeline


Trailers, as unique, well-made video types, have particular data distribution and exceptional quality. The trailer videos usually contain multiple themes, including movies, TV shows, games, etc. They are well organized with the most attractive clips of the corresponding videos. At the same time, most of them are accompanied by corresponding background voices and music. Such rich and high-density information is a complete challenge for the current Artificial Intelligence Generation Content(AIGC) task and brings considerable difficulties to data cleaning. To solve this challenging problem, we designed a comprehensive data collection and cleaning process as shown in Fig. <ref> to deal with such complex videos, as described in this section.





Collection Strategies
To enrich the diversity of our MMTrail, we first employ the keyword “trailer” to retrieve various in-the-wild trailer videos. Such general-purpose trailer videos encapsulate a wide range of artistic works and genres. Then, we tailor the keywords more specifically to explicitly collect trailer videos with divergent sources. Those keywords include “Movie Trailers”, “Video Game Trailers”, “TV Show Trailers”, “Documentary Trailers”, etc. Through these two methods, we collect 285,518 comprehensive trailer videos with a total duration of 94,911,802.8 seconds. 


Trimming
To facilitate the extraction of various video information in subsequent analyses, we cut the original videos into clips based on the scenes. As the most mature and practical tool currently available, PySceneDetect[https://github.com/Breakthrough/PySceneDetect] offers robust functionalities for this purpose. Thus, we use ContentDetector of PySceneDetect to compare the difference in content between adjacent frames and then cut the videos according to the predetermined threshold of 30. We finally generated 21,588,792 clips with an average duration of 4.6 seconds. 

Motion Filtering 
Evaluating motion quality is a crucial step in selecting high-quality video content. Motion vectors and optical flow are both mainstream motion quality evaluation methods. In our case, trailer videos often have rapid cuts and transitions between scenes, posing challenges for optical flow-based analysis. Other than optical flow, motion vectors are more robust to these quick changes, as they rely on larger block units' displacement rather than individual pixels' continuous flow. On the other hand, motion vectors are more lightweight and can be calculated more efficiently. Thus, we leverage motion vectors to filter out clips with problems like static frames, title sequences, and slideshow-like playback.


Diversity
We evaluate the diversity and richness of our dataset from three aspects: theme, objects, and backgrounds. While collecting, we first assess the categories from the Yotoube metadata provided by the video provider, as shown in  <ref>. Furthermore, we generate an object-level caption list and background by Llava-NexT <cit.> for a more accurate category-based generation. The word cloud of objects and backgrounds is shown in  <ref>.








OCR

Trailer videos often have text-heavy sections with high-quality text animations, like opening and ending credits. To identify these text-rich segments, we utilize OCR to detect the text content in the video frames and calculate the bounding box area of the text. This measurement reflects the amount of text in the clips. 


Quality Statistics
In addition to text detection, we considered image quality <cit.> and aesthetic scores <cit.> to enhance our analysis of trailer videos. These measures allowed us to evaluate frames' visual fidelity, clarity, and aesthetic appeal, providing more comprehensive insights for trailer analysis and editing. 


Audio Collection
We extract the audio from video source segments with a sampling rate of 44.1 kHz. We use the PANNs <cit.> algorithm to perform music event detection, and over 70% of the audio segments contain music.










 §.§ Video Captioning Pipeline


The MMTrail contains many complex themes, like subtitles and character animations, as shown in  <ref>, which brings extraordinary complex work for video captioning. At the same time, smooth transition shots also make it impossible for traditional single-frame annotation methods to convey semantics coherently. Therefore, this section introduces a multi-temporal and multimodal caption pipeline containing a detailed video description from frame, motion, and music levels. 

Frame Caption


The auto-captioning pipeline has proven efficient in cutting-edge video generation foundation models. SVD <cit.> and Pandas <cit.> have given promising results and demonstrated the importance of high-quality frame captions for the generation model. We initially performed image-level captioning on the individual frames of the data. We employed coca <cit.> for each video clip to generate separate captions for three frames(first, middle, and last), resulting in relevant captions.

Clip Caption
Having obtained concise captions for three frames that capture the essential information, we aimed to obtain fine-grained captions and variations between frames in the video. We concatenated multiple frames into a comic strip format and employed the LLaVA <cit.> image model to guide the description of the dynamic differences between frames. Additionally, leveraging a powerful multimodal language model, we incorporated OCR and more detailed summary descriptions to expand the information within the frame captions.


Categories and Background
Noticing the LLM-based caption has hallucinations when describing the frame, we further generate word-level labels to enhance the annotation of the main objects and background. Initially, we utilized LLaVA's QA capabilities to have the model answer questions about the background. Subsequently, through QA, we prompted the model to provide relevant category information. We conducted the word cloud in  <ref>. and certified caption quality by subjective experience in  <ref>.



Music Caption
Moreover, given that trailer music usually has a well-designed audio effect and background music, we applied the music caption on our dataset rather than a standard audio caption. In our work, we used MusicCaps <cit.>, an LLM-based music captioning model. The caption format is well designed with its description pipeline, which first describes its sound quality, a generated speech style, and a detailed description of its instrument and music style. More examples are shown in the Appendix. We further evaluate the generation tasks and the text-to-music generation based on the music caption, which shows the efficiency of our captioning and dataset.

OCR The trailer contains many text animations. Captioning the context inside the movie can also be a challenging task. In this task, we utilize the OCR ability by LlaVA <cit.> and caption the context for 5 frames of each video. We merge reliable text animation captions based on the previous method.

General caption Combining all the captions mentioned above, we use the language model llama2-13B <cit.> to merge all the captions and generate complete and high-quality multimodal captions. We evaluate the caption accuracy and quality by human preference in  <ref>.





 §.§ SubSet Separation




We applied the frame caption <ref> for the full MMTrailer-20M video clips with 20M+ clips. We included the scale comparison with other large-scale datasets in  <ref>, showing that our dataset is a large-scale video-language dataset. MMTrail has a resolution no smaller than 720p, and MMTrail-20M clips are 4.6s long on average.


High-quality Subset
, named MMTrail-2M, contains a detailed multimodal caption. Compared to the original distribution, we samples MMTrial-2M to create a high-quality subset using the following criteria:
1. We filter out clips with motion scores below 0.45 or above 50.
2. We only retain the clips within the top 85% of image quality scores.
3. We only keep the clips within the top 85% of aesthetic scores.
Furthermore, all clips in MMTrail-2M are longer than 4s and provided with all the captions, as shown in  <ref>, including categories, background, frame captions, music captions, merged captions, etc.
We also compare MMTrail-2M with Video-Audio datasets as shown in  <ref>, MMTrail has a larger scale than existing datasets (Audioset <cit.> and Vggsound <cit.>). 
 
High-quality test set
is extracted from the MMTrail-2M, we extract a fine-branded testing set that contains 1k video clips and multiple multimodal captions. Then, we manually adjust the merged caption to a manual caption to build a testing subset with trust-wise multimodal prompts. We test several tasks and models on the test set in  <ref> to show the complexity and difficulties of MMTrail. The test set has 98.2 words of caption on average and includes 3.2hr video clips.











§ EXPERIMENTS



This section presents comprehensive experiments on multiple tasks to demonstrate our dataset's effectiveness, diversity, complexity, and difficulty. 



 §.§ Multimodal Captioning




We present the results of our human evaluation of video caption quality in  <ref>. Ten videos were randomly selected from the MMTrail-Test dataset. They were rated on a scale of 0 to 10 based on general impressions, including aspects such as correctness, level of detail, richness, and fluency. The results of more than 100 sets of samples indicate that the manually adjusted prompts rating of 8.12 outperforms the auto-caption pipeline, while our merged captions achieve the second-best performance of 6.62. Despite being short and straightforward, object/background labels achieve a 6.02 evaluation score, demonstrating more correctness than other captions. Frame caption, music caption, and llaVA caption obtain 4.3, 4.6, and 5.4, respectively, and these findings demonstrate the effectiveness of our captions and highlight the quality of our labeled captions by human annotators.



 §.§ Video Generation









The model was fine-tuned on the VideoCrafter-2.0 <cit.> dataset using 8 Tesla-H800 GPUs with a batch size of 3 for 10,000 steps at a learning rate of 6e-6. The training data was randomly sampled from the MMTrail-2M dataset, using the video captions as input.
The evaluation results, shown in  <ref>, include 9 matrices on the VBench <cit.>, indicating that fine-tuning the model on the MMTrail-2M dataset led to improvements of 0.6 in motion smoothness and 1.77 in subject consistency, with a slight overall performance boost(0.12 higher) compared to the official VideoCrafter-2.0 checkpoint.
Visual examples of the generated content are provided in  <ref>, and additional demonstrations and experiment details will be included in the supplementary material.
This thorough evaluation and comparison of the tuned model's performance on critical metrics provides valuable insights into the effectiveness of the fine-tuning process and the potential benefits of leveraging the MMTrail-2M dataset for video generation tasks.




 §.§ Video Understanding



Experiment Setting
To evaluate the capability of our dataset in multimodal video understanding, we choose Video-LLaMA <cit.> as the baseline for the video captioning task. We use same model and training config as Video-LlaMA, which useVicuna-v0-7B as llama model <cit.>, ViT <cit.> and Q-Former  <cit.> as the video encoder and the linear projection layer from MiniGPT-4 <cit.>. We train 4 epochs by MMTrail-2M, each containing 2500 iters with batch size 32. We compare it with two official model weights: the pre-train Video-LlaMA weight on WebVid (2.5M video-caption pairs) and the fine-tuned Video-LlaMA.

Evaluation Metric
We evaluate video understanding models on the MMTrail-Test. As for the evaluation metric, we choose the commonly used metrics in text generation tasks-BLEU-4 <cit.>, ROGUE-L <cit.>, METEOR <cit.>, and CIDEr <cit.> to evaluate our result.
All the metrics are computed using the pycocoevalcap <cit.> package. We also use BERTScore <cit.> to evaluate the contextual similarity for each token in the ground truth and the predicted captions. The results are reported in  <ref>. The official weights show relatively low performance, highlighting the challenge of MMTrail, and the data distribution differs from their training data.

In addition, we also evaluated three checkpoints from Video-LlaMA <cit.> by human evaluation in  <ref> and found that the Video-LlaMA-MMTrail evaluation result slightly lags behind Video-LlaMA-Finetune but performs significantly better than Video-LlaMA-Pretrain. We provide further details in  <ref> for a more comprehensive understanding of our model.



 §.§ Music Generation


We used text-to-music generation to evaluate the effectiveness of the video-music pair data and the labeled video caption and music caption. 
We use MusicGen <cit.> to generate music based on our video caption (VideoCap2Music) and music caption (MusicCap2Music). We use Kullback-Leibler Divergence (KL), Inception score (ISc), Frechet distance (FD), and Frechet Audio Distance (FAD) <cit.> to evaluate the generated music. Besides, we use the ImageBind-AV score (IB) to evaluate the audio-visual alignment between the video and the generated music. For the model with text input in Tab. <ref>, compared with video caption, the evaluation results on music caption are 0.13 better in KL, 0.65 in ISc, 3.64 in FD, and 1.21 in FAD, showing the domain gap of multimodal descriptions.
This comparison shows that there is still a significant research gap between caption-music-video.







§ CONCLUSION


We introduce MMTrail, a comprehensive and accurate multi-modality visual-audio dataset to address the dataset gap. By utilizing the inherent value of trailers, which integrate visual, audio, and contextual elements, MMTrail offers detailed and precise multi-modality annotations. Our systematic captioning framework adaptively merges visual and musical perspectives, ensuring that the annotations capture the richness of multimodal content. Experimental results demonstrate the high quality of the MMTrail dataset, its effectiveness for fine-grained multimodal-language model training, and a variety of downstream applications. We believe this innovative dataset will unlock new possibilities in video content generation and significantly advance research in visual-audio understanding. The comprehensive and diverse nature of MMTrail makes it a valuable asset for the research community, paving the way for novel applications that leverage the power of multimodal learning.




§ CAPTIONING PIPELINE




 §.§ Music Caption


As trailer videos often contain a cacophony of audio tracks, which typically include background music and vocals, captioning audio poses a significant challenge. Therefore, to achieve optimal music captioning results, we initially utilize Demucs <cit.> for vocal separation on each audio clip. Subsequently, LP-MusicCap <cit.> is leveraged to caption the resultant audio, devoid of vocals, from the separation process.



 §.§ Speech Recognition



Besides, the speech is also an important track of the trailer audio. Therefore, we further turn the speech into the text.
Similar to our approach in music captioning, we use Demucs initially to perform vocal separation on each audio clip. Following this, Whisper is utilized to caption the separated vocal audio.



 §.§ Merge Caption




§ CAPTIONING PIPELINE


We use LlaMA-13B to merge the multiple captions, by constricting a pre-designed caption prompt as follows:
    
    There are some descriptions of a video, 
    including video caption, music caption, background caption and the main objects in the video. 
    Please combine all the descriptions into an overall description of the video in only one paragraph.
    Finally, please rewrite and polish it into an overall video description in one paragraph and give me only the content of the video. 
        Background: background
        Main objects: objects
        Video caption 1: image_cap
        Video caption 2: frame_cap
        Music caption: music_cap
    

As illustrated in Figure <ref>, we use the LLaMA-13B to merge frame captions, lava captions, music captions, objects, and background into a final merge caption. By merging multiple captions, the merge caption offers a comprehensive and detailed description of the audiovisual content of videos. Its components mutually complement one another, ensuring that every aspect of the narrative receives attention and providing unique insights into various facets of the video content.



 §.§ Frame Caption

We utilize LLava-v1.6-vicuna-7B as the image understanding model to generate captions for video clips. Initially, we filter out clips with durations of less than 1.0 seconds. Subsequently, we sample three frames at the fractions of 0.2, 0.5, and 0.8 of each clip's total duration. The prompt is as follows:
 
 Please describe the image in detail.
 






 §.§ LLaVA Caption

We use the image understanding model LLaVA-13b <cit.> to caption each of our video clips. Specifically, for each clip, we sample frames at positions 0.1, 0.3, 0.5, 0.7, and 0.9, and then horizontally concatenate them into a single image, which serves as input for LLaVA-13b to generate captions. We construct the caption prompt as follows:
    
    These are some keyframes of a video. 
    Please use one sentence to summarize the content of the video in detail. 
    Summarize the content of the entire video but not describe keyframes frame by frame.
    
Moreover, we observe that the results of LLaVA captioning often contained some redundant information, as illustrated by the green sections in Figure <ref>. Therefore, we use LLaMA-13b to refine the LLaVA captions, eliminating much of the extraneous content and rendering the final captions more in line with human expression. We construct the prompt as follows:
   
    [caption] This is a description of a video. 
    Please polish it to an overall video description in one sentence and give me only the content of the video. 
    Do not use the words 'frame' and 'video'. 
    Describe the content of the video directly, which means do not start with 'The video...' or something like that.
    Do not add extra information that is not included in the original description.
    Here is an example: A dancer in a vibrant orange skirt and gray jacket moves gracefully across the stage, her movements fluid and expressive. 
    

By combining frames and secondary polishing, we finally obtain high-quality captions that contain the main content information of the clips.






§ ADDITIONAL EXPERIMENTS




 §.§ Video Understanding Model

We use VideoLLaMA as our base model, which contains a llama model, a q-former, and a llama projection model. For LLM, we use llama vicuna-v0-7B.  
We also use model weight pretrained on minigpt4 to initialize our llama projection. In the training stage, we train from scratch and froze the Visual Transformer and LLM models. Only the visual q-former and the llama projection are trained in the pretraining stage. Following the video llama setting, we use a batch size of 32, and each video uniformly samples 8 frames with a 224 resolution. We train 5 epochs on our 2M training data. Each epoch contains 2500 iters with a linear warmup cosine lr. The weight decays are 0.05, and the warmup lr is 1e-6 with 2500 warmup steps, a 1e-4 init lr, and an 8e-5 minimum lr. The results can be seen in Table <ref>








 §.§ Video to Music Generation


To further assess the effectiveness of the video-music pair data, we conduct an extended video-to-music generation task by training VidMuse <cit.> on an individual music subset of MMTrail-2M. The results, shown in Table <ref>, demonstrate the high audio quality and strong audio-visual alignment between the video and the generated music achieved by our dataset.







§ DATASET DETAILS


Here is one metadata example of our dataset. We introduce the basic information of video and clips in the "basic" tag, including their duration, quality evaluation score, etc.  The useful caption and description are saved in the "scene" tag.


    "basic": 
        "video_id": "-3r7ptfObEs",
        "video_path": "group_33/-3r7ptfObEs.mp4",
        "video_duration": 71.73333333333333,
        "video_resolution": [
            720,
            1280
        ],
        "video_fps": 30.0,
        "clip_id": "-3r7ptfObEs_0000000",
        "clip_path": "video_dataset_33/-3r7ptfObEs_0000000.mp4",
        "clip_duration": 7.033333333333333,
        "clip_start_end_idx": [
            0,
            211
        ],
        "image_quality": 36.83453941345215,
        "of_score": 12.92151,
        "aesthetic_score": [
            3.010026454925537,
            3.664743423461914,
            3.994750499725342
        ]
    ,
    "camera": 
        "view_scale": "",
        "movement": "",
        "speed": ""
    ,
    "misc": 
        "frame_caption": [
            "a person standing in a room with a laptop on their lap. ",
            "a black and blue background with the name of the series. ",
            "a black and white image of the words manga and comics. "
        ],
        "music_caption": [
            
                "text": "This is an indie rock music piece. There is a male vocalist singing melodically in the lead. The main tune is being played by the electric guitar while the bass guitar is playing in the background. The rhythm is provided by a simple acoustic drum beat. The atmosphere is easygoing. This piece could be used in the soundtrack of a teenage drama TV series as the opening theme.",
                "time": "0:00-10:00"
            
        ]
    ,
    "scene": 
        "objects": [
            " Matheus Trindade",
            " de",
            " original",
            "Series",
            " Manga     Comics"
        ],
        "background": "Dark",
        "ocr_score": 0.07147466104497355,
        "caption": "The video is a series of original de Matheus Trindade comics, showcasing the artist's unique style and storytelling.",
        "polish_caption": "Original de Matheus Trindade comics showcase the artist's unique style and storytelling, featuring a series of vibrant and expressive illustrations that bring characters and scenes to life.",
        "merge_caption": "In this captivating video, Matheus Trindade's original comics come to life with vibrant and expressive illustrations, showcasing the artist's unique style and storytelling. Set against a dark background, the video features a series of dynamic and colorful images, including a person sitting in a room with a laptop, a black and blue background with the name of the series, and a black and white image of the words m̈angaänd c̈omics.T̈he indie rock music piece, featuring a male vocalist and simple acoustic drum beat, adds an easygoing atmosphere to the video, making it perfect for the opening theme of a teenage drama TV series."
    



plain










 Paper Title: Vision-Language Model Based Handwriting Verification.










Vision-Language Model Based Handwriting Verification
    
    Mihir Chauhan††   Abhishek Satbhai§   Mohammad Abuzar Hashemi¶   Mir Basheer Ali¶¶   Bina Ramamurthy‡   Mingchen Gao||   Siwei Lyu†   Sargur Srihari‡‡
 
    
    Department of Computer Science and Engineering 
 
    The State University of New York, Buffalo, NY, USA 
 
    {mihirhem††, bina‡, mgao8||, siweilyu†, srihari‡‡}@buffalo.edu 

    {ma.hashemi.786¶, abhishek07satbhai§, alimirbasheer¶¶}@gmail.com

    
=================================================================================================================================================================================================================================================================================================================================================================================================================================

empty



§ ABSTRACT 
Handwriting Verification is a critical in document forensics. Deep learning based approaches often face skepticism from forensic document examiners due to their lack of explainability and reliance on extensive training data and handcrafted features. This paper explores using Vision Language Models (VLMs), such as OpenAI’s GPT-4o and Google's PaliGemma, to address these challenges. By leveraging their Visual Question Answering capabilities and 0-shot Chain-of-Thought (CoT) reasoning, our goal is to provide clear, human-understandable explanations for model decisions. Our experiments on the CEDAR handwriting dataset demonstrate that VLMs offer enhanced interpretability, reduce the need for large training datasets, and adapt better to diverse handwriting styles. However, results show that the CNN-based ResNet-18 architecture outperforms the 0-shot CoT prompt engineering approach with GPT-4o (Accuracy: 70%) and supervised fine-tuned PaliGemma (Accuracy: 71%), achieving an accuracy of 84% on the CEDAR AND dataset. These findings highlight the potential of VLMs in generating human-interpretable decisions while underscoring the need for further advancements to match the performance of specialized deep learning models. Our code is publicly available at: <https://github.com/Abhishek0057/vlm-hv>


Keywords: Vision Language Model, Machine Vision, Handwriting Verification, Forensics








§ INTRODUCTION


Handwriting verification is crucial in pattern recognition and biometrics, focusing on authenticating and identifying individuals by their handwriting. It is vital in forensics, where experts analyze samples to verify documents, identify forgeries, and provide court evidence.

Historical Perspective: Handwriting Verification methods <cit.> <cit.> <cit.> have evolved from handcrafted features like GSC <cit.> to deep CNNs such as ResNet-18 <cit.>, and now Vision Transformers <cit.>, enabling comparison of both inter- and intra- writer variations in handwriting styles. 

Challenges: Despite the advancements, Forensic Document Examiners (FDE) remain skeptical due to the lack of interpretability in the model's decision making processes. Morever, these methods heavily depend on large datasets of labeled handwritten images, making dataset collection (X(x_q,x_k, y) with known x_k, questioned x_q handwritten samples and corresponding writer labels y) is costly and time-intensive. 


Why use VLM? Vision-Language Models (VLMs) integrate image and textual data to encode complex relationships between visual and linguistic information, enabling Forensic Document Examiners (FDEs) to interpret model decisions with clear, natural language explanations that enhance trust and reliability in forensic applications. They can adapt to the forensics domain in zero-shot scenarios without training examples and in few-shot scenarios with minimal examples by leveraging transfer learning capabilities, having been pre-trained on multiple tasks related to handwritten images and natural language understanding. Fine-tuning them with forensic-specific datasets further enhances their performance and applicability in real-world forensic use-cases. 

VLMs applied to Forensics: In <cit.>, the authors explore the capabilities of multimodal Large Language Models (LLMs) like GPT-4 in various digital forensics use-cases, including artifact understanding, evidence searching, anomaly detection, and education. More specific studies, such as <cit.>, demonstrate the effectiveness of GPT-4 specifically for the media forensics task of DeepFake image detection. Our research applies VLMs to handwriting verification. To our knowledge, we are the first to explore VLMs for the forensic task of handwriting comparison.












§ METHODS

Our study aims to use VLM in handwriting verification. We have chosen OpenAI’s GPT-4o VLM for its strong Visual Question Answering (VQA) capabilities. Using their API, we prompt GPT-4o <cit.> with 0-shot Chain-of-Thought (CoT) reasoning. This approach allows us to first generate human-interpretable explanations and then determine whether the given questioned and known handwritten samples were written by the same person or by different writers, as shown in Figure <ref>. Since fine-tuning GPT-4o is not generally available, we compared its performance to a recent open-source VLM, PaliGemma <cit.>, using 0-shot prompt engineering and parameter-efficient supervised fine-tuning (PEFT) on a training dataset with 100 examples.  

Data: The experiments were conducted using 1,000 sample pairs of known and questioned images from CEDAR Letter <cit.> and CEDAR AND <cit.> dataset. CEDAR Letter dataset contains a letter manuscript written by 1568 writers three times. CEDAR AND dataset is a subset of CEDAR Letter data which only contains the lowercase handwritten word “and” extracted from full letter manuscript. The evaluation dataset includes 1,000 sample pairs of known and questioned images written by 368 writers with writer ids greater than 1200 (writer ids below 1200 were used to train the baseline models). Of these pairs, 500 are from the same writer and 500 are from different writers.






§ EXPERIMENTS & RESULTS

CEDAR AND Baselines: As shown in Figure <ref> we use handcrafted features GSC features, CNN based ResNet-18 <cit.> and MaskedCausalVisionTransformer <cit.> (ViT) as our baseline feature extractors. GSC are 512-dimensional features extracted for the binarized “AND” images. All the three baselines were trained on 10% and 100% of known and questioned training pairs writers with writer ids less than 1200 resulting in 13,232 and 129,602 number of train pairs as shown in Table <ref>. The output of these feature extractors is fed into 2 fully-connected (FC) layers with 256 and 128 hidden neurons with ReLU activations. The final layer has 2 output neurons whose softmax activations represent similarity of samples with a one hot vector representation. We use categorical cross entropy loss given one-hot encoded logits compared to the target which is binary. 



Prompt Engineering with GPT-4o and PaliGemma: To effectively utilize the GPT-4o VLM for handwriting verification, we experimented with various prompts to optimize both the generation of human-interpretable explanations and the accuracy of verification decisions. Initially, we crafted prompts that directed the model to compare specific features of the handwriting samples, such as stroke width, slant, and letter spacing. For example, prompts like "Describe the similarities and differences in the stroke patterns between the two samples" and "Identify any matching unique characteristics in the handwriting" were used. However, we observed significant variance in the model’s responses, leading to inconsistent results. To address this, we adopted a Chain-of-Thought (CoT) reasoning approach. This method guides the model through a structured reasoning process, allowing it to generate more consistent and reliable explanations. Since no examples were provided to VLM in the prompt engineering phase, the number of training pairs is observed as 0 in Table <ref>. 



Additionally, we utilized prompt engineered both VLMs to identify and mark coordinates of similarities and dissimilarities within the two images. By prompting the model to highlight specific areas of interest, such as matching or differing stroke patterns, we were able to generate detailed coordinates that pinpointed these features as shown in Figure <ref>. This approach provided clear visual indicators of the reasoning behind each verification decision. By incorporating CoT reasoning, presenting known and questioned samples as different images, and using VLM to identify key coordinates, we achieved a balance where the model could generate detailed, interpretable explanations while accurately determining the authenticity of the handwriting samples.

Supervised Fine-Tuning with PaliGemma: 
We fine-tuned the pre-trained PaliGemma model on 100 curated examples of 0-shot CoT prompt engineered PaliGemma results. The PaliGemmaProcessor within Transformers library was loaded to process the inputs which involved the preparation of prompt templates and batching text inputs with images. The image tokens and pad tokens were set to -100 to be ignored by the model, and these preprocessed inputs were used as labels for the model to learn from. The model, PaliGemmaForConditionalGeneration, was loaded and set to fine-tune only the text decoder, leaving the vision encoder and multimodal projector frozen. We used BitsAndBytes config options with 4bit quantization and LoRA <cit.> to train only 11M parameters out of the 3B parameters of the network. The results of supervised fine-tuned PaliGemma on both CEDAR AND and Letter datasets are shown in Table <ref>.

Results: The results of using VLMs for handwriting verification are presented in Table <ref> on the CEDAR AND and CEDAR Letter datasets. For the CEDAR AND dataset, CNN based ResNet-18 architecture outperforms 0-shot CoT prompt engineering approach with GPT-4o as well as supervised-finetuned PaliGemma with 100% training pairs which suggest that although VLMs has a lot of potential for generating human interpretable decision making for the task of handwriting verification, still lags behind CNNs which aare fine-tuned on specific task of handwriting verification. This gap highlights the need for further improvements in fine-tuning regime of VLMs to enhance their effectiveness and reliability for specialized tasks like handwriting verification. For the CEDAR Letter dataset, the 0-shot CoT prompt engineering with GPT-4o also shows relatively higher performance than the supervised fine-tuning of PaliGemma but we observe high variability in the performance metrics because of low sample size. 



§ CONCLUSION

Our study explores the application of VLMs, specifically GPT-4o and PaliGemma, in the domain of handwriting verification. By leveraging the robust VQA capabilities of these models and employing 0-shot CoT reasoning through prompt engineering, we aimed to generate human-interpretable explanations for model decisions. Our experiments demonstrated that while VLMs offer significant improvements in interpretability and adaptability to diverse handwriting styles, they currently lag behind CNN-based architectures such as ResNet-18 in terms of performance. Specifically, ResNet-18 achieved an accuracy of 84% on the CEDAR AND dataset, outperforming GPT-4o's 70% accuracy and PaliGemma's 71% accuracy. These findings suggest that while VLMs hold great promise for enhancing transparency and trustworthiness in forensic handwriting verification, there is still a need for further advancements in their fine-tuning regimes to improve their effectiveness and reliability for specialized tasks. Moving forward, we aim to work with forensic document examiners to create a fine-tuning dataset of explanation reports using text and visual information to ensure the applicability of our approach in practical forensic settings.






apalike





 Paper Title: Segment anything model 2: an application to 2D and 3D medical images.


Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation
    Yixiao Wang1 Chen Tang1,2 Lingfeng Sun1 Simone Rossi3 Yichen Xie1Chensheng Peng1Thomas Hannagan3Stefano Sabatini3Nicola Poerio4Masayoshi Tomizuka1Wei Zhan1
    August 5, 2024
===============================================================================================================================================================



§ ABSTRACT 
Segment Anything Model (SAM) has gained significant attention because of its ability to segment a variety of objects in images given a prompt. The recently developed SAM 2 has extended this ability to video inputs. This opens an opportunity to apply SAM to 3D images, one of the fundamental tasks in the medical imaging field. In this paper,



we provide an extensive evaluation of SAM 2's ability to segment both 2D and 3D medical images.
We collect 18 medical imaging datasets, including common 3D modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) as well as 2D modalities such as X-ray and ultrasound.
We consider two evaluation pipelines of SAM 2: (1) , where prompts are provided to one or multiple slice(s) selected from the volume, and (2) , where prompts are provided to each slice. The former is only applicable to 3D modalities, while the latter applies to both 2D and 3D modalities.
We learn that SAM 2 exhibits similar performance as SAM under , and has variable performance under depending on the choices of slices to annotate, the direction of the propagation, the predictions utilized during the propagation, etc. 










§ INTRODUCTION

Medical image segmentation is a crucial task for multiple clinical applications such as disease diagnosis and clinical analysis <cit.>. Despite the advancements in medical imaging technologies, segmentation remains a challenging task due to the labor-intensive nature of data annotation and the complexity of medical images <cit.>. 



Segment Anything Model (SAM) addresses these challenges in multiple directions. On one hand, SAM has demonstrated impressive zero-shot segmentation performance with prompt inputs, significantly reducing the need for extensive manual data annotation <cit.>. On the other hand, several works fine-tune SAM to specific tasks and demonstrate improvements over standard segmentation techniques <cit.>, such as nn-UNet <cit.>.

Despite these advancements, SAM's limitation to 2D images restricts its applicability to scenarios that require three-dimensional understanding <cit.>. 

Existing work addresses this challenge by introducing extra components to SAM to enable its 3D segmentation capability. For example, SAM3D <cit.> combines the SAM encoder with a lightweight 3D CNN decoder; 3DSAM-A <cit.> modifies the original prompt encoder and mask decoder to operate in 3D; SAM-Med3D <cit.> introduces an additional 3D convolution before the image encoder and replaces 2D positional encoding layers with 3D one. On the other hand, the recently introduced SAM 2 <cit.> solves this limitation fundamentally by extending the backbone of SAM to 3D. Specifically, SAM 2 proposes a memory bank that retains information from past predictions and allows it to make predictions on slices without prompts based on the information.

This feature motivates us to examine SAM 2's ability to segment 3D medical images since video segmentation can be transferred to 3D segmentation seamlessly, i.e., each slice is treated as a frame. Note that we will use the terms “slice” and “frame” interchangeably throughout the paper.

In this paper, we extend the previous evaluation experiments on SAM <cit.> to SAM 2, aiming to explore the model's effectiveness in a more complex, three-dimensional context. Specifically, we consider two evaluation pipelines: and . In <cit.>, prompts are provided to the object of interest on each slice, whereas in , prompts are provided on one or a few slices selected from the volume. We further consider some unique challenges in , such as the selection of the slice(s) to be annotated, the direction of propagation, i.e., predicting slices without prompts based on ones with prompts, and the selection of prediction during propagation. Our experiments are conducted across 18 datasets, covering 5 modalities (magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), X-ray, and Ultrasound) and various body locations. We observe several trends of SAM 2 under both evaluation pipelines and summarize them in Section <ref>.






§ METHODS




In this section, we discuss the two evaluation pipelines, and , in detail. 
Intersection over Union (IoU) is the evaluation metric throughout the paper. To have a comparable performance between 2D and 3D segmentation, IoU is only computed over non-empty slices.



 §.§ Evaluation of Single-Frame 2D Model Testing on 2D Images

During , SAM 2 exhibits the same behavior as SAM in segmenting the object of interest based on prompts.
For datasets with 2D modalities, we run SAM 2 naturally at the image level. For datasets with 3D modalities, we simulate prompts for each slice of the volume.

Following previous work <cit.>, we design in a non-iterative manner in which all prompts are determined without feedback from any prior predictions. 

Specifically, the following four 2D prompting modes (P-Mode) are used:

    
  * P-Mode 1: One point prompt placed at the center of the largest connected region of the object of interest.
    
  * P-Mode 2: One point prompt placed at each separate connected region of the object (up to three points).
    
  * P-Mode 3: One box prompt placed at the center of the largest connected region of the object of interest.
    
  * P-Mode 4: One box prompt placed at each separate connected region of the object (up to three boxes).

These strategies cover the most efficient and straightforward annotation strategy that provides one point or box prompt to each connected region of the object of interest. 


    





 §.§ Evaluation of Multi-Frame 3D Model Testing on 3D Volumes

SAM 2 differs from SAM mostly in its new ability to segment videos, which can be seamlessly transferred to the 3D image segmentation task.

In this section, we are mostly interested in SAM 2's semi-supervised segmentation ability, where we only provide prompts for one or a few frames in advance and use SAM 2 to predict other frames. 

Initial Frame Selection. In video segmentation, the initial frame to be annotated is typically the first frame of video as it arrives first in the time stream and the object of interest usually does not change shape or size dramatically between consecutive frames <cit.>.<cit.> 
This is also the strategy used by SAM 2 for the semi-supervised video object segmentation task. However, such a strategy is usually infeasible in 3D medical imaging because, in most scenarios, the boundary of the object of interest appears first, and the change between different frames can be significant. 

Instead, when annotating medical volumes, annotators tend to start with the slice in which the object of interest is most salient.
To approximate this behavior, we consider the following three frame modes (F-Mode):

    
  * F-Mode 1: Choose the slice with the largest object of interest.
    
  * F-Mode 2: Choose the slice at the center of the object of interest.
    
  * F-Mode 3: Uniformly select 3 slices from the entire object of interest.













Prompts Simulation. When the slice to be annotated is selected, we only utilize the same point-based strategies (P-Mode 1 and 2) for simplicity because SAM 2's video predictor does not have the function of adding boxes yet. 





Propagation Direction. When the first frame is annotated, SAM 2 supports propagation to predict other non-conditioned frames (frames without prompts). 
This propagation step involves the newly introduced memory structures: memory attention, memory encoder, and memory bank. Memory attention conditions the current slice's features on features from previously predicted slices and prompts. The memory encoder compresses these predictions into smaller embeddings and stores this information in the memory bank.







Note that since new prediction relies on the output from existing predictions, the order of propagation matters. 

Although the order of propagation might have less impact on video predictions in which the consecutive frames are similar and the object of interest appears consistently from beginning to end, it can have a more significant impact on 3D medical volumes in which the change between slices can be large and the object of interest does not occupy the full volume.
To take the order of propagation into account, we consider the following 2 direction modes (D-Mode):

    
  * D-Mode1: Front-to-End Propagation: start from the first slice of the volume and propagate forward through the volume.
    
  * D-Mode2: Start from the first annotated slice and propagate backward to the first slice; then restart from the annotated slice and propagate forward to the end of the volume.

    
    
    
    
    
Predicted Mask Selection. Due to the ambiguity of providing click prompts, it is common to predict various levels of objects based on a single or a few clicks. For example, if we place a point prompt on a tumor inside the brain, the object of interest can either be the tumor or the whole brain.

In the 2D setting of both SAM and SAM 2, the mask predictor can create multiple mask outputs from which humans can select the one closest to their needs. If there is no human involvement in the selection step, we can simply pick the first prediction channel, which, according to previous SAM empirical results, tends to represent the smallest area among the three channels <cit.>. Alternatively, we can select the channel with the highest confidence based on the IoU calculated by SAM's mask decoder.

In 3D mode, to prevent propagating ambiguity across frames, SAM 2 chooses to pick the prediction with the highest confidence and only utilizes this prediction when estimating the object of interest on other frames. However, based on previous experiments with SAM <cit.>, selecting the predictions with the highest confidence can be sub-par when compared to selecting the first channel. This observation motivates us with the following predicted mask selection strategies:

    
  * Selecting the predicted mask with the largest estimated IoU, i.e., confidence. 
    
  * Selecting the first channel.



In summary, under , we cover 3 slice selection modes, 2 prompt-simulation modes, 2 predicted mask strategies, and 2 propagation modes, resulting in a total of 24 different experimental designs. Notably, we do not enumerate all possible choices. For some options, when we identify a consistently better choice for one stage compared to others, we trim some subsequent combinations applying this choice to reduce the total number of experiments.
























§ DATASET


Consistent with the previous experimental study on SAM <cit.>, this study utilizes 18 diverse medical datasets to evaluate the performance of SAM 2. Specifically, all datasets are evaluated during , and twelve datasets with 3D modalities (MRI, CT, and PET) are evaluated during . The “US-Nerve” dataset<cit.>, which was included in the previous SAM evaluation paper<cit.>, has been removed from this project due to licensing regulations. We also updated the MRI-Breast dataset to include more annotated volumes. The pre-processing steps are consistent with the previous study, except that for SAM 2 3D mode, we have to convert the input images to JPG format instead of PNG. We also provided visual representations of the annotations masks for each dataset in Fig. <ref>.



 §.§ 2D Datasets

We included 6 2D datasets, 2 X-rays, and 4 ultrasounds, covering 7 different anatomical objects. Specifically, the X-ray datasets cover chest and hip joint segmentation, and the ultrasound datasets encompass a broader range of regions, including breast, kidney, muscle, and ovarian tumor segmentation. Detailed information on these datasets can be found in Table <ref>. To keep the format consistent with 3D datasets, the images of 2D datasets are also converted to JPG using the same pipeline. 





 §.§ 3D datasets

We included 12 3D datasets, 5 MRI, 6 CT, and 1 PET-CT, covering 20 different anatomical objects. The structures of the datasets are modified to fit the input requirements of SAM 2 3D, where each individual volume has its own folder, and each slice is numbered according to its position within the volume. Table <ref> demonstrates more detailed information.












§ EXPERIMENTAL RESULTS



 §.§ Results of SAM 2 on Single-frame 2D Segmentation testing compared with previous SAM

In this section, we present the performance of SAM 2 under scenario across four prompting modes (refer to Section <ref>). Fig. <ref> and Fig. <ref> show the results for 2D and 3D datasets, respectively. The findings indicate that SAM 2's single-frame 2D segmentation capability is comparable with that of SAM (a comparison of single-frame 2D segmentation results between SAM and SAM 2 is provided in Fig. <ref>). Similar to SAM, the performance of SAM 2 varies significantly across different datasets. For instance, SAM 2 achieves an impressive IoU of 0.907 on the Xray-Hip dataset for ilium but performs poorly with an IoU of 0.277 on the MRI-Spine dataset for GM.

Comparing the performance for different prompting modes, both the results on SAM and SAM 2 highlight the superiority of box prompts over point prompts. Moreover, prompts that indicate each separate part of the object individually are generally superior to those that indicate only one part or outline all parts in a single box, which is also consistent with the findings in previous SAM evaluations <cit.>. 






 §.§ Results compared with 3D model's Prediction Selection Strategy


We investigate the impact of prediction selection strategies across three initial frame selection modes: F-Mode 1, F-Mode 2, and F-Mode 3, all within the single point click setting (P-Mode 1) and using the front-to-end propagation direction (D-Mode 1). The results, depicted in Figure <ref>, demonstrate that for the annotated slice, opting for the first channel's prediction achieves performance that is comparable to or better than SAM 2's default method of choosing the most confident slice. Notably, for the MRI-Spine: SC dataset, selecting the first channel leads to an approximate 4% improvement in DSC when starting from the largest slice, and a 14% improvement when starting from the center slice and 17% when starting from 3 uniform slices.

Thus, we find that selecting the first channel from the annotated slices is a better choice compared to the highest confidence selection. Consequently, we will primarily focus on this selection method in the following sections.





 §.§ Results compared with different Initial frame selection and Prompt simulation strategies

As shown in Figure <ref>, we compare the results of providing prompts on the largest slice, the center slice, and three uniformly distributed slices.
We find that providing only one prompt for the entire volume results in unsatisfactory performance. For the front-to-end propagation (subplot row 1, first two plots), selecting the largest slice yields an average DSC of 6% across all datasets, while selecting the center slice yields an average DSC of 14%. For the bidirectional propagation (subplot row 2, first two plots), these numbers are 8% and 19%, respectively. These results show a dramatic decrease in performance compared to the previous 2D evaluation results across all datasets.

For different prompt modes, we do not observe a significant performance change. Providing one or more prompts (as shown by the different colors of the bars in each plot) yields consistent results. This is expected because additional prompts are only provided when there is more than one disconnected region. In most medical image datasets, the target object usually appears as a single connected region.

Comparing the performance of starting with annotations on three slices versus one slice (the three plots in the first row), we observe a slight improvement: around 10% compared to annotating the largest slice and 2% compared to annotating only the center slice. These results are surprising at first glance. However, upon deeper examination, we find that when annotating three uniformly distributed slices, the IoU from the center slice is usually the highest, while the IoUs from the first and last slices are lower (likely because the objects around the 1/4 and 3/4 quarters are smaller). Thus, the center slice's prediction mainly contributes to the final performance. Through these findings, we conclude that placing prompts on a few slices is not significantly helpful.

The low performance of SAM 2 raises some concerns since expanding the predictions from a single slice to a full volume is a common task in both the medical and natural image fields. 

To further explore whether the low performances stem from the propagation step (i.e., the annotated slice's performance not transferring well to other slices), we calculate the average IoU for the prompted slices as well on F-Mode 2 on two different propagation ways, resulting in 41.2% and 37.1% average IoU. We find that the segmentation performance on the prompted slices is quite consistent with previous 2D segmentation results. Therefore, the low results are more likely caused by the errors accumulated during the propagation. 

We hypothesize that there are two reasons for the unsatisfying behavior of SAM 2 during propagation: (1) SAM 2 was trained on a frame width of 8 frames, and thus the memory attention is most effective when the propagated frame is within 8 frames of the annotated frame. Medical volumes range from 10 to a few hundred slices. Consequentially, the performance will be significantly affected when the volumes are deeper. (2) SAM 2's primary objective is to segment any videos. Although this function can be transferred seamlessly between video segmentation and 3D segmentation, videos and medical volumes can differ significantly. Namely, the change between consecutive slices can be large for medical volumes due to the nature of medical imaging technology. In this case, we believe that direct usage of SAM 2 for 3D medical imaging segmentation tasks is not yet feasible. Further fine-tuning and redesigning of the memory strategy are needed.






 §.§ Impact of the propagation mode

Comparing the first and second rows in Figure <ref>, we find that applying bi-directional propagation can slightly improve the performance. When selecting the initial frame with F-Mode 1, i.e., the slice with the largest object of interest, the performance of SAM 2 improves around 2% IoU; and under F-Mode 2, i.e., the middle slice of the object of interest, SAM 2's performance improves around 5% IoU. 

These findings suppose that, though SAM 2 supports prompting on any frame as a condition to predict new frames, it is better to start propagating from adjacent slices as they are first added to the memory bank. Though the original SAM 2 was mainly examined on 1-N propagation mode in their experimental settings, the performance of reverse propagation (saying from K to K-1) can still work well given that the order of medical images is not relevant to the time order. 



 §.§ Results compared with 2D model and 3D model

When evaluating the 3D datasets, we can compare it on testing on 2D mode and testing on 3D mode. 
this should be done last and is optional. We can just describe the comparison verbally in 4.3






§ CONCLUSION

In this work, we conduct an initial investigation of the performance of SAM 2 in the medical imaging field. 
In addition to evaluating the 2D segmentation task, we investigate its ability to do the 3D segmentation task thanks to SAM 2's ability to segment videos. 
We find that under :

    
  * SAM 2 exhibits similar performance to SAM.
    
 
Under , we observe that:

    
  * When deciding the initial frame selection, selecting the center slice of the object of interest is better than selecting the slice with the largest object of interest.
    
  * Providing 1 or 3 points to the selected frame has little effect on the final performance. 
    
  * During propagation, it is better to utilize the first predicted mask instead of the one with the largest confidence.
    
  * Propagating predictions bidirectionally, starting from the annotated slice, works better than propagation from beginning to end.
    
  * The best average performance of SAM 2 in the 3D segmentation is 0.19 IoU when only one point prompt is provided to one volume.



splncs04



 Paper Title: MaskUno: Switch-Split Block For Enhancing Instance Segmentation.


[


    
    
    
    
    
    MaskUno: Switch-Split Block For Enhancing Instance Segmentation 

    
    
    


Jawad Haidar, Marc Mouawad, Imad Elhajj, Daniel Asmar




Vision and Robotics Lab (VRL), Maroun Semaan Faculty of Engineering and Architecture, American University of Beirut, Beirut, Lebanon


   



]


§ ABSTRACT 
Instance segmentation is an advanced form of image segmentation which, beyond traditional segmentation, requires identifying individual instances of repeating objects in a scene. Mask R-CNN is the most common architecture for instance segmentation, and improvements to this architecture include steps such as benefiting from bounding box refinements, adding semantics, or backbone enhancements. In all the proposed variations to date, the problem of competing kernels (each class aims to maximize its own accuracy) persists when models try to synchronously learn numerous classes. In this paper, we propose mitigating this problem by replacing mask prediction with a Switch-Split block that processes refined ROIs, classifies them, and assigns them to specialized mask predictors. We name the method MaskUno and test it on various models from the literature, which are then trained on multiple classes using the benchmark COCO dataset. An increase in the mean Average Precision (mAP) of 2.03% was observed for the high-performing DetectoRS when trained on 80 classes. MaskUno proved to enhance the mAP of instance segmentation models regardless of the number and type of classes or the type of architecture in which it was applied.




instance segmentation, competing kernels




§ INTRODUCTION
 
Instance segmentation is a fundamental task in computer vision, which combines mask segmentation and object detection, and whose goal is to perform a per-pixel classification of different objects in a scene. Instance segmentation has a wide range of applications such as in medical diagnostics <cit.>, agricultural analysis <cit.>, and autonomous driving <cit.>. Its abundant use in practical scenarios necessitates accurate and robust predictions. The problem of instance segmentation is difficult and faces challenges such as occlusion, scale changes, and cluttered backgrounds  <cit.>.   

With an eye toward enhancing the performance of instance segmentation, some researchers adopted the cascade methods that rely on progressive refinement <cit.>. For example, one could pipe the output of the the bounding box prediction from a Stage 1 cascade into a region extractor (Stage 2), and so on, such that the progressive flow of the refined bounding boxes guarantees better-localized feature extraction. As a result, a higher accuracy is achieved in delineating masks and bounding boxes. 
    
Another research direction concentrates on designing backbone networks that can detect features that are better suited for the downstream task of instance segmentation. There exists a trade-off between the depth of a backbone model and the spatial resolution of features. Contrary to image classification, instance segmentation requires high spatial resolution of features, and so research on this front is concerned in architecting backbone networks that can identify features with high spatial resolutions
     <cit.>.  



Most previous work in instance segmentation adopts the pipeline of Mask R-CNN <cit.>: after extracting features using a backbone, a Region Proposal Network (RPN) localizes the regions that encompass the sought-after objects. The localized regions are then extracted using a region of interest block and fed to detection and mask generation branches. To avoid competition among classes, Mask R-CNN replaces the multi-nomial cross entropy with a per-class sigmoid and binary loss <cit.>, keeping in mind that the weights are common between different classes given that they belong to the same branch.      
To tackle this gap, our method works on top of most instance segmentation methods by simply isolating the branches corresponding to different classes. Specifically, our pipeline consists of two main parts: detection and per-class segmentation. The output of the detection is fed to a switch that directs the bounding boxes to their corresponding segmentation branch. Consequently, the optimization of each branch is done independently to guarantee non-competing kernels between different classes. These simple modifications significantly improve the performance of the baseline Mask R-CNN <cit.> and the high performing CNN based model Detectors <cit.>.

The remainder of this paper is organized as follows. In the next section, we will delve deeper into the recent methods in the field of instance segmentation. In Section <ref>, the MaskUno proposed method is introduced. In Section <ref>, experiments are conducted and results demonstrate the enhancements obtained by MaskUno when applied to various instance segmentation models, trained and tested on the standard COCO dataset benchmark. Finally, Section <ref> concludes the paper.




§ RELATED WORK


The architecture of an instance segmentation model contains different blocks, each contributing in a different manner to the end result. First, the backbone is responsible for extracting high-level features of the image. One can usually add a Feature Pyramid Network (FPN) that can help the network gain multi-scale context, which is crucial for detecting objects of different sizes. Next, comes the RPN, which generates from a feature map potential region candidates that might contain objects. The ROI-Align layer ensures that all the regions of interest are of similar size. Finally, the regions of interest are fed to a terminal block which classifies them and predicts bounding boxes and binary masks for each instance. In the following section, we will review the various interventions at the level of the backbone, object detection, and cascade. 



 §.§ Backbones
 

The pillar of every deep learning model is the feature extraction block (named backbone) and most of them are built around a CNN. The downside of such models is that spatial resolution decreases with depth, which is problematic for instance segmentation models that require high spatial resolution for pixel-wise mask prediction. To address this issue, SpineNet <cit.> uses Neural Architecture Search to learn scale permuted features as well as cross-scale connections. DetectoRS <cit.> proposes a recursive FPN  that mimics thinking twice at a macro level; more specifically, the outputs of the FPN are fed to the bottom-up backbone. Additionally, they introduce the looking twice concept at the micro level, which is achieved by adding Switchable Atrous Convolutions.



 §.§ Object detection
 
Object detection is an essential block for instance segmentation as it delineates the bounding box surrounding each of the detected objects in a scene. 

Object detection methods can be based either on one stage<cit.>, or two stages <cit.>, where the former achieves better inference speed and the latter provides higher accuracy. The high inference speed in one-stage detectors is ascribed to their simple architecture; for instance, YOLO <cit.> divides the image into a grid and places various anchors having different scales and aspect ratios. Then, the anchors are directly classified and adjusted after a feature extraction stage. For the sake of predicting small-scale objects, SSD <cit.> performs detection at different stages of the feature extraction. RetinaNet <cit.> proposes the focal loss to attenuate the effect of the abundant easy negatives, resulting in improved accuracy over the standard cross entropy loss. To get rid of anchors, CornerNet <cit.>  expresses the bounding boxes by two points, the top-left and bottom-right corners. CenterNet <cit.> uses the corners as proposals, and the center to verify the class of the object. A more accurate object representation is proposed by ExtremeNet <cit.> that replaces the corners with the objects' extreme points.  


On the other hand, two-stage object detectors include an RPN, whose role is to first propose regions with high objectiveness probability, and output class-agnostic bounding boxes of possible objects. These bounding boxes are then used by the region of interest extractor.

R-CNN <cit.> uses a non-learning-based proposal generator to propose possible objects, and then each of these regions of interest is passed through a CNN to extract corresponding features. This is a slow operation as each box is passed independently. On the other hand, Fast R-CNN <cit.> applies a feature extractor to the entire image and then extracts the corresponding features. Finally, Faster R-CNN <cit.>, replaces the non-learning-based proposal method with a learnable RPN. Mask R-CNN <cit.> adds a mask prediction head to support instance segmentation, and additionally proposes a region of interest alignment block that takes into account the neighboring pixels surrounding the bounding box.
 



 §.§ Cascade
 

Cascade is the process of iterating the output of a model to achieve progressive refinement. Cascade R-CNN <cit.> proposes to add a simple cascade architecture on top of Faster R-CNN. A Stage 1 bounding box prediction is fed to a region of interest extractor and has the effect of superior localized bounding boxes that can contribute to enhanced feature extraction. Likewise, Cascade Mask R-CNN applies this concept to enhance mask predictions. Hybrid Task Cascade (HTC) introduces direct connections between the different stages of mask branches, thereby strengthening the flow by integrating complementary features across stages <cit.>.






To summarize, the most significant enhancements to instance segmentation in the literature include interventions at the level of the cascade, loss function, or backbone. In this paper, we propose to intervene at the level of the last prediction layer, in a manner to mitigate the problem of competing gradients between different classes. As far as we know, none of the previous methods do this. 

The contributions of this paper include the following:  


  * Proposing a  modular Switch-Split block that can replace multi-class prediction heads in most instance segmentation methods. 

  * Ensuring no competing kernels between the different classes, which leads to richer representations as no trade-off between classes is permitted.

  * Enhancing the accuracy of instance segmentation models on the standard COCO dataset benchmark.





§ PROPOSED SYSTEM
 



Recalling from the previous section, at the output of an instance segmentation model, detected regions of interest are streamed into a terminal block consisting of three different branches: the first one is usually a classifier indicating the presence of an object in this region, and if so classifies it. The second branch predicts refined bounding box coordinates for each proposed region, adjusting the initially generated region proposals. The third branch takes the ROI-aligned features as input and predicts a binary mask for each object instance within the proposed regions. These masks indicate the pixel-level segmentation of the objects.

To our knowledge, all instance segmentation architectures in the literature include the aforementioned terminal block (i.e., including the three branches).  An interesting observation we came to realize is that this block in its current form suffers from what we refer to as competing kernels. The competing kernels problem arises when a model tries to synchronously learn multiple classes having different features. During training, the kernels learned for each class compete with others from another class in an attempt to maximize their own segmentation accuracy. 

The main intuition driving our proposed MaskUno system is that each class should be learned on its own in a block we name the Switch-Split block.



 §.§ Switch-Split block

The idea behind adding a switch and split block is to first benefit from the bounding box refinement and then specialize the learning per class. In other words, we first pass the bounding box head output through the ROI Align layer, then use it as an input to the Switch-Split block (see Fig.<ref>). The output of the classifier is given as an input to the switch, then the classifier, based on the input, classifies the refined ROI which then turns the switch to the corresponding class inside the splitting block. Finally, each ROI is assigned to a specific one-class mask head. 

The split block is a set containing N+1 blocks, each trained to classify and predict the bounding box and the mask of one specific class.

This idea applies to any architecture based on Mask-RCNN, and to any class. In this section, we will show how to adapt it to the baseline Mask-RCNN, the Cascade Mask-RCNN, and the hybrid task cascade. 



  §.§.§ Mask-RCNN

instead of having the bounding boxes and the masks predicted in parallel as is usually done in Mask-RCNN, in MaskUno it is done sequentially. In other words, the bounding boxes are first predicted, and then based on these refined boxes the mask is estimated. Figure <ref> presents an example of the methodology applied to this architecture.




  §.§.§ Cascade Mask-RCNN and Hybrid Task Cascade


this architecture already includes bounding box refinements from one layer to the other, meaning that the output of the refined bounding box of Layer i+1, is fed as an input to the Switch-Split block of Layer i.
The split block is a set containing N+1 blocks, each trained to classify, and predict the mask of one specific class. Figure <ref> shows how each block is represented: they all include a mask head, all having the same functionality as the branches of the basic Mask-RCNN pipeline. In terms of hybrid-task cascade, we also replace every mask-head with a Switch-Split block, taking as input the refined classified bounding boxes as shown in Fig. <ref>.






 §.§ Loss functions


Every one of these architectures is allocated a set of three loss functions, one for every sub-part of this block. Hence, there is the categorical cross-entropy loss defined by (<ref>), where y_i is the true class label, and ŷ_i is the predicted class label, and N is the number of classes. 


    L_cls = - ∑_i=1^N y_i log(ŷ_i)


The second loss function is the smooth L1 loss defined by (<ref>) where x represents the difference between the predicted and the true value.


    L1(x) = 
    0.5x^2,    if  |x| < 1 
    
    |x| - 0.5,    otherwise


For the mask prediction part, the loss function is a pixel-wise binary cross entropy defined by (<ref>).


    L_mask = - ∑_i,j [ y_true(i, j) log(y_pred(i, j)) + 
            
    (1 - y_true(i, j)) log(1 - y_pred(i, j))

where y_true is the true pixel label, and y_pred is the predicted pixel label. 



 §.§ Training

A typical model contains its own set of loss functions [L_cls,  L1,  L_mask] that it will try to minimize, while eliminating the issue of competing kernels. The mask head of class i will have its own L_mask_i representing the pixel-wise loss function that will be minimized. Having different loss functions for every class is also essential to solve the issue of competing kernels as it makes the weights of every mask head independent of one another also in terms of cost.

The training schedule and hyper-parameters are kept the same for every block to avoid overfitting on a specific class. 



 §.§ Generalizing to other pipelines for instance segmentation

The switch split method should be applicable to any architecture, by first identifying the blocks where classification, bounding box regression, and mask prediction, are implemented, and then splitting the segmentation block into multiple ones, each specialized for the concerned class.

As a proof of concept, one can even, perform sequential training for every architecture, meaning train the model on Class 1 to obtain a specialized mask-head for this class. Then, train a new mask-head using the original checkpoint of the imported model on Class 2, up until all needed classes have a specialized mask-head. 

In Section IV, we perform experiments on models in the literature to validate our proposed method.










§ EXPERIMENTATION AND DISCUSSION
























 §.§ Datasets and metrics



  §.§.§ Data used

Experiments are performed on the Common Object in Context (COCO) dataset <cit.> because it is challenging due to the numerous classes it contains, and its high number of samples, but also to make a fair comparison with the state-of-the-art models trained using this benchmark. We use the train2017 and val2017 data which contain a total of 115,000 and 5,000 images respectively. The data is split per-class for both training and validation. For the bounding box and mask training, instance annotations are used, and for the semantics, the COCO-stuff annotations.





  §.§.§ Evaluation


to evaluate the model performance, we use the mean average precision taken at different IOU thresholds (at 0.5 and 0.75). Additionally, different scales are taken for the data: small, medium, and large.  We report the AP, AP_50, AP_75, AP_s, AP_m, and AP_l.


 §.§ Experimentation


To validate the generalization of MaskUno we test it on various classes and various architectures. The process starts by removing the multiclass mask prediction head from a pre-trained model with a plateau in its accuracy, followed by adding the Switch-Split block and finally training the new weights on the sought classes. Afterward, we compare its mean Average Precision (mAP) with the one obtained before training the model. The hypothesis is that its mean average precision increases for every class using the MaskUno method.
The experimentats are divided accordingly. First, using the COCO dataset, and the library MMDetection <cit.>, a similar experiment is conducted on four different models: the baseline Mask-RCNN <cit.>, the Cascade-Mask-RCNN <cit.>, the Hybrid Task Cascade <cit.>, and the DetectoRS backbone <cit.> on a list of 10 classes. Since the pre-trained models have more classes compared to the MaskUno models, we separate the validation data into different sub-datasets, each responsible for validating a specific class. 
Furthermore, we experiment on a total of 80 classes and compare the results for Mask-RCNN and DetectoRS. Mask-RCNN is used since it is the baseline model for instance segmentation. DetectoRS is an architecture that contains a change in backbone, as well as a hybrid task cascade architecture. 
Note that, given the methodology, we care about the mask prediction results for all models and classes.



 §.§ Results and discussion


We discuss in this section the results obtained after applying our method to different models and a variety of classes. Table <ref> shows the results of the four models on every class. The experiments consist of training each of these previously assumed saturated models, on a list of ten classes, shown in the column of the table. The models are chosen in a way to include the different techniques found in the literature, including (1) a change in the backbone (DetectoRS), (2) a cascade architecture (Cascade mask-RCNN), and (3) a hybrid task (HTC). It is worth mentioning that DetectoRS is currently the best-performing CNN-based model for instance segmentation. 
Moreover, we can see comparative results of the metrics before and after training, and for all architectures, there is a significant increase in the mAP which varies from 0.5% to 5%.   


Furthermore, for some classes, we did not observe a significant increase in the metrics. A possible explanation would be the number of samples available. One example is the class 'sheep' which contains a small number of samples (765 samples for training). Additionally, very few entries show a slight decrease after applying our method.


This experiment supports the fact that MaskUno can be applied to a wide set of methods for instance segmentation. 
We extend this experiment by training 80 classes for both the baseline Mask-RCNN and DetectoRS. For both models, we observe an average increase in the mAP (Fig.<ref>).  This enhancement infers a different conclusion for every model. For Mask-RCNN, the mAP increases by 4.8%, which confirms that MaskUno is not affected by a specific choice of class. As for DetectoRS, the mAP increases by 2.03%, which validates that solving the competing kernels is complementary to a cascade architecture or a backbone enhancement. 
 
Figure <ref> shows a few examples at inference time, for multiple classes. Besides successful detection and segmentation, one can note that the degree of confidence for every instance is significantly high.







 










§ LIMITATIONS AND FUTURE WORK



  §.§.§ Limitations

Our experiments demonstrate that MaskUno outperforms previous models when evaluated on a per-class basis, with each class evaluated exclusively on its positive images. However, it's important to note that this experimental setup does not account for negative frames where false positives may cause the switch to misdirect certain images to incorrect branches.


  §.§.§ Further experiments


Another experiment would be to investigate the effect of splitting the bounding box regression block on the mAP, compare it to classical bounding box refinement, and merge both together. A new version of MaskUno where splitting is done on the branch responsible for bounding box prediction, followed by the mask split. This would not only enhance bounding box refinements but also the segmentation branch benefiting from this refinement. 



  §.§.§ Transformer-based models


Instance segmentation models based on transformers proved to surpass CNN-based models. They are usually used as a backbone for various computer vision downstream tasks. A similar approach could be implemented, where the bounding box regression and segmentation branches of such models would be split into multiple parts to specialize the training per-class. 



§ CONCLUSION


To address the issue of conflicting gradients, we offer MaskUno, a Switch-Split block that can be used for most instance segmentation algorithms. It enhances the accuracy of instance segmentation models based on Mask-RCNN that use multi-class segmentation methods and has a complementary effect to tasks such as cascade or hybrid task. MaskUno provided a 2.03% improvement over one of the best-performing CNN-based instance segmentation models "Detectors" on the standard COCO dataset. Moreover, for future work, applying MaskUno on transformer-based models, while benefiting from a split of both bounding box regression and mask prediction, has the potential to reach a new state-of-the-art accuracy.



§ ACKNOWLEDGMENT

This work was supported by the Didymos Horizon Europe project, grant number 101092875–DIDYMOS-XR.

ieeetr



 Paper Title: Dynamic Object Queries for Transformer-based Incremental Object Detection.




Kramers-Kronig relations via Laplace formalism and L^1 integrability
    Leonardo Ricci
    August 5, 2024
====================================================================

 
[1]These authors contributed equally to this work. 
[2]Corresponding authors. 


§ ABSTRACT 
Incremental object detection (IOD) aims to sequentially learn new classes, while maintaining the capability to locate and identify old ones. As the training data arrives with annotations only with new classes, IOD suffers from catastrophic forgetting. Prior methodologies mainly tackle the forgetting issue through knowledge distillation and exemplar replay, ignoring the conflict between limited model capacity and increasing knowledge. In this paper, we explore dynamic object queries for incremental object detection built on Transformer architecture. We propose the Dynamic object Query-based DEtection TRansformer (DyQ-DETR), which incrementally expands the model representation ability to achieve stability-plasticity tradeoff. First, a new set of learnable object queries are fed into the decoder to represent new classes. These new object queries are aggregated with those from previous phases to adapt both old and new knowledge well. Second, we propose the isolated bipartite matching for object queries in different phases, based on disentangled self-attention. The interaction among the object queries at different phases is eliminated to reduce inter-class confusion. Thanks to the separate supervision and computation over object queries, we further present the risk-balanced partial calibration for effective exemplar replay. Extensive experiments demonstrate that DyQ-DETR significantly surpasses the state-of-the-art methods, with limited parameter overhead. Code will be made publicly available.





§ INTRODUCTION





Humans inherently possess the ability to incrementally learn novel concepts without forgetting previous ones, capable of acquiring and accumulating knowledge from past experiences. Traditional object detection models <cit.> rely on supervised learning with fixed data, where all classes are predefined and known beforehand. However, real-world data continuously evolve over time, leading to non-stationary distributions. Due to the stability-elasticity dilemma, fine-tuning models directly on new class data leads to catastrophic forgetting <cit.>, whereas joint training is expensive in both computation and storage. Therefore, incremental object detection (IOD) has attracted increasing attention in both research and practical applications. 


Recent advances for IOD adopt knowledge distillation and exemplar replay to address forgetting. Knowledge distillation-based methods <cit.> typically involve distillation on the non-background predictions of the old model to circumvent the imbalance issue caused by an excessive number of background predictions. Exemplar replay-based methods <cit.>, on the other hand, operate by preserving a small subset of samples from past training data (exemplars), replaying them in subsequent phases to mitigate the forgetting of old data. Despite the progress, there still exist limitations. First, due to the fixed model capacity, severe conflicts between preserving knowledge of old classes and incrementally learning new ones exist. This not only impedes the accommodation of new class knowledge but also leads to the overwriting and forgetting of old class knowledge. Second, exemplars with incomplete labels are inadequately equipped for replay. Even if pseudo labeling is an intuitive way, the low-quality supervision inherited from old knowledge will hamper the adaption.


To address the above concerns, we propose Dynamic object Query Assembly based DEtection TRansformer, abbreviated as DyQ-DETR, for incremental object detection. Inspired by dynamic networks for incremental classification <cit.> , we motivate to investigate the dynamic architecture for expanding model capacity in IOD. Specifically, the object queries serve as the class-specific representative essence for memorization. The object queries from old classes can be assembled with learnable new queries for dynamically expandable representations with limited memory overhead and acceptable time overhead. As demonstrated in Fig.<ref>, we first obtain the visual features of the whole image with a CNN attached Transformer encoder. Then the set of object queries from previous steps are assembled with learnable ones corresponding to new classes in each incremental step. Moreover, we disentangle the self-attention and isolate the bipartite matching to remove the interaction between object queries from different steps. Besides, we propose the risk-balanced partial calibration to tackle incomplete annotations for effective exemplar replay.


Generally, we circumvent the catastrophic forgetting in IOD from the perspective of dynamic networks, with the inspiration of incrementally adding class tokens has been proven effective in incremental learning <cit.>. Notably, for object detection, multiple objects of a single class may co-exist in one image (sometimes even many). We adopt a many-to-many rather than one-to-one matching of dynamic queries, for which sparse object queries implicitly associate with the content and reference positional information of one or multiple seen old classes. We propose to memorize the set of object queries for sequentially arrived classes of data. An isolated group of queries is responsible for detecting the objects from a set of classes arriving at the same time step. By incrementally aggregating the category-wise object queries from previous steps and learning new class embedding, we decouple the representations for old and new class knowledge with lightweight query embeddings, simultaneously maintaining the stability and plasticity. As for exemplar replay, we propose to reserve images with moderate matching losses as exemplars, further perform partial calibration on the outputs of corresponding queries with only the annotated classes. Such risk-balanced partial calibration avoids over focusing on classes from any particular stage and eliminates the reliance on low-quality pseudo labels. Through dynamic object queries and risk-balanced partial calibration, our proposed DyQ-DETR significantly alleviates the forgetting in IOD.  
The main contributions are three-fold: 





  * We propose a novel approach to integrate dynamic object queries into Detection Transformer for incremental learning. By dynamically incorporating object queries into DETR, our approach provides a simple yet effective way to expand the model capability compatible with new knowledge adaption and old knowledge preserving. 

  * We propose the disentangled self-attention for dynamic object queries. For dynamic knowledge expansion, the isolated bipartite matching over the object queries from different phases further decouples the representation learning of old classes and new ones.

  * A risk-balanced selection mechanism is proposed to explore informative and reliable exemplars. The partial calibration is further associated to tackle the incomplete annotations for incremental detection with exemplar replay.


Extensive experiments on public benchmarks demonstrate the superiority of our proposed DyQ-DETR, outperforming the state-of-the-art methods by a large margin. It achieves the average 4.3% AP improvement in non-exemplar scenarios and 2.9% AP improvement with exemplar replay.




§ RELATED WORK



Incremental Learning (IL). 

Prevailing IL methods can be broadly divided as regularization-based, distillation-based and structure-based ones. Regularization-based methods<cit.> estimate parameter importance and penalize updating of crucial parameters to maintain previous knowledge. Distillation-based methods build the mapping between the old and new model by matching logits<cit.>, feature maps<cit.>, or other information<cit.>, which leverage the knowledge transfer to prevent forgetting. Structure-based methods<cit.> dynamically expand the representative network, e.g., backbone, prompt, to fit the evolving data stream. 



Incremental Object Detection (IOD).  As the typical extension of incremental learning, IOD involves multiple objects belonging to the old and new classes appearing simultaneously. This co-occurrence makes knowledge distillation an inherently effective strategy for IOD, since it allows for the utilization of old class objects from new training samples to minimize the discrepancies in responses between the previous and current updating model. As a pioneering work, ILOD<cit.> distills the responses for old classes to counteract catastrophic forgetting on Fast R-CNN<cit.>. The idea of knowledge distillation is then extended to other detection frameworks, such as CenterNet<cit.> (SID<cit.>), RetinaNet<cit.> (RILOD<cit.>), GFLV1<cit.> (ERD<cit.>), Faster R-CNN<cit.> (CIFRCN<cit.>, Faster ILOD<cit.>, DMC<cit.>, BNC<cit.>, IOD-ML<cit.>) and Deformable DETR<cit.> (CL-DETR<cit.>). Built on Deformable DETR instead of conventional detectors such as Faster R-CNN, DyQ-DETR can efficiently expand queries rather than inefficiently enlarging the backbone or specific convolutional layers. Note that DyQ-DETR also uses knowledge distillation techniques. 
As for exemplar replay,<cit.> proposes maintaining an exemplar set and fine-tuning the model on the exemplars after each incremental step.<cit.> proposes an adaptive sampling strategy to achieve more efficient exemplar selection and <cit.> proposes distribution-preserving calibration, which selects exemplars to match the training distribution. They usually finetune directly using the exemplar set with incomplete annotations and overlook the amount of information and reliability of the annotated objects. 




Transformer-based Object Detection.
The infusive work DETR (DEtection TRansfomer)<cit.> formulates object detection as a set prediction problem, with an elegant transformer-based architecture. It captures global context and reasons object relations with attention mechanism. With a small set of learnable object queries and Hungarian bipartite matching<cit.>, it eliminates the need for the complex non-maximum suppression and many other hand-designed components in object detection while demonstrating good performance. Deformable DETR<cit.> introduces sparse attention on multi-level feature maps, thereby accelerating the convergence of DETR and improving the performance, particularly for small objects. There also exists many other DETR variants<cit.> designed to accelerate convergence speed and enhance detection performance. Without loss of generality, we build our method on the commonly used Deformable DETR.




§ METHODOLOGY



Preliminaries
In the paradigm of IOD, object detection is performed in multiple steps from sequentially arrived training data to recognize and localize objects of all seen classes in test images. Let D be a dataset with samples (x,y), where x is the image, y is the set of object class labels and the associated bounding boxes. Suppose there are T steps. At time step t, the incoming dataset is denoted as D_t=(x_t, y_t) and the objects belong to seen classes in C_t. Specially, the images in D_t are only annotated for objects in C_t. Moreover, the class sets at different time steps are mutually exclusive from each other, i.e., C_t∩ C_τ=∅, t≠τ. Since D_t only contains the annotations with classes in C_t, each class can be learned only once in a specific step. Due to the missing annotations of old classes {C_1, C_2, ⋯, C_t-1} in current data D_t, detection models are prone to forget the knowledge of old classes and bias towards new ones. To overcome  the catastrophic forgetting, we introduce the dynamic object queries into Detection Transformer and propose DyQ-DETR for IOD.


We construct the DyQ-DETR based on Detection Transformer for incremental object detection. In particular, we elaborate the details of dynamic object queries with the structure design and training strategy in Sec. <ref>. The risk-balanced exemplar selection and partial calibration are in Sec. <ref>.







 §.§ Transformer-based Incremental Object Detection



We build up the DyQ-DETR based on the DETR architecture <cit.>. Other than the backbone, a DETR consists of an encoder, a Transformer decoder, and the predictor to generate object classes and locations. The encoder takes the images as inputs and outputs visual features. Then those visual features and learnable object queries are fed into Transformer decoder for prediction. Notably, we propose to aggregate the object queries from up-to-far learning steps to resist the forgetting of old class knowledge. Due to the absence of annotations from previous seen classes in incremental step t, knowledge distillation is applied for preserving the class-specific old knowledge. As in Fig.<ref>, we select the non-background predictions of old classes by thresholding. Pseudo labels ỹ_τ (1 ≤τ<t) are generated by the last previous model. Instead of mixing the pseudo labels ỹ_τ (1 ≤τ<t) with the ground-truth labels y_t, we use ỹ_τ and y_t separately to supervise the model training. The pseudo labels for old classes and the real ground-truths are used to guide the learning of object queries being Q_τ and Q_t, respectively. Besides, object queries from different groups share the weight parameters of the Transformer decoder. To allow for computational complexity to grow linearly rather than quadratically, the self-attention over different groups of object queries are eliminated.


As for IOD with exemplar replay, where a small number of exemplar images ϵ_t from dataset D_t in different time steps are stored, we introduce a risk-balanced selection mechanism. At time step t, we use the trained model Φ_t to score the annotated objects from images in D_t. The computed loss from the partial bipartite matching is considered as the risk score for exemplar selection. We choose the sample images with moderate risk score for the tradeoff between the annotation importance and quality. Specifically, to build the exemplar dataset ϵ_1:t, we select the samples falling into the middle part after sorting, because they are informative and reliably annotated. Considering the image in ϵ_1:t is incompletely annotated for specific classes, we adapt the partial calibration. We leverage the incomplete annotations to calibrate the output for corresponding object queries in each group. Since ϵ_1:t is balanced, the partial calibration will prohibit the model being biased towards certain classes.
 




 §.§ Dynamic Object Query Assembly




Existing DETR models employ a fixed set of object queries (i.e., learnable embedding) as the inputs of Transformer decoder. These object queries are progressively optimized to map into object instances in images. Despite various designs, the learnable object queries are highly relevant with the specific classes. For IOD, the object queries are expected to be associated with objects belonging to sequentially arrived classes. Since new classes are considered as backgrounds in previous time steps, the preservation of old knowledge naturally contradicts the knowledge updating from new data learning, especially from the perspective of object queries. Moreover, the conflict between a fixed network and the continually emerged class-specific information severely undermines the performance of incremental learning, especially in non-exemplar scenarios. 


To cope with incremental classes without extra modules in network architecture, we focus on tackling the forgetting issue with dynamic object queries. At time step t, for a set of new classes C_t, a new set of learnable object queries Q_t is added. The newly added object queries Q_t are aggregated with previous sets of queries Q_τ, 1 ≤τ<t. The assembly of object queries {Q_1, Q_2, ⋯, Q_t} serves as the input of Transformer encoder in step t. The visual embeddings e_t corresponding to the newly expanded queries Q_t are used to predict the objects of new classes C_t, and the old classes C_τ (1 ≤τ<t) is detected with the embeddings e_τ of old queries Q_τ. By dynamically expanding object queries, the new and old classes are segregated by class embeddings, significantly alleviating the conflict between the old knowledge and continuously evolving new knowledge.


Based on the dynamic assembly of object queries, we further investigate the decoder design to restrict the computational burden. In standard DETR, object queries interact visual features with cross-attention for refinement. Besides, those object queries interact with each other by self-attention. 
Through self-attention, duplicated detections can be removed, but the computational complexity increases quadratically with the number of object queries. Considering that the object instances from different class sets rarely overlap, we disentangle the self-attention in Transformer decoder. The self-attention is computed among separate set of object queries, as:

    a_i,j=
        
        Softmax(Q_m,i· Q_n,j^T/√(d))   m=n
    
        0   m≠ n,

where a_i,j denotes the attention weight between two queries Q_m,i and Q_n,j from the query set Q_m and Q_n, respectively. By eliminating the attention interaction between queries of different groups as shown in Eq.(<ref>), we achieve a linear growth in computational complexity almost for free. Upon the addition of new queries, the capability of old queries to detect old classes is fully preserved. We perform the decoder forward passes as many as the time steps, obtaining the embedding vectors from different sets of queries as: 

    e_τ=decoder(f_t, Q_τ), 1 ≤τ≤ t,

where f_t denotes the visual feature extracted from image x by the CNN and Transformer encoder. Each decoder forward pass is executed with a different set of queries Q_τ, resulting in different task-specific embeddings e_τ to obtain detection predictions of the corresponding classes C_τ.


We further adapt the knowledge distillation for incremental detector training. The foreground predictions with the pseudo labels generated from old model are kept for distillation and used for supervision. As in <cit.>, the foreground predictions with high confidence from the old model are selected. A probability threshold θ_p (typically 0.4) is set over the prediction scores. An additional IoU threshold θ_IoU (typically 0.7) is used to restrict the predictions not too close to the ground-truth bounding boxes of new classes. It helps filter out incorrect predictions about new class objects that are misclassified as old classes. The highly-confident predictions after filtering are used as pseudo labels ỹ_τ (1 ≤τ < t), which contain two parts of annotations (i.e., the predicted object labels and bounding boxes). Notably, 
instead of merging the pseudo labels with real grounding truths, we compute the separate bipartite matching losses with different set of object queries in an independent way. The loss for retaining old class knowledge ℒ_τ^DETR (1 ≤τ < t), as well as the loss for learning new class knowledge ℒ_t^DETR can be formulated as in Eq.(<ref>):

    ℒ_τ^DETR = ℒ^DETR(e_τ,ỹ_τ), 1 ≤τ < t ; 
      ℒ_t^DETR = ℒ^DETR(e_t, y_t),

Note that the specific embeddings e_τ, 1 ≤τ≤ t are only supervised with the corresponding pseudo labels or real annotations, leading to decoupled representations. The total loss is the weighted sum as:

    ℒ_total=∑_τ=1^tw_τℒ_τ^DETR.

To tackle the varying number of classes in each step and prevent the model from being biased towards class sets with fewer classes, we set the weight w_τ of the class set C_τ to |C_τ|/|C_1:t|.


r0.5
  

  
    < g r a p h i c s >

  Illustration of risk-balanced exemplar selection. We choose the middle part with moderate risk score to serve as the exemplars.
  

  




 §.§ Risk-balanced Partial Calibration



For exemplar replay based IOD paradigm, an exemplar memory is formed to store a small number of samples for subsequent incremental learning. At time step t, the exemplar set ϵ_t is a subset of the entire dataset D_t. Let ϵ_1:t=ϵ_1 ∪⋯ϵ_t. The exemplars in ϵ_t are used to represent the samples with objects in C_t. We intend to choose the class annotated images which are substantial for detector training. An intuitive way is to directly compute the loss between the model output and the real label y_t for sample selection. However, since the image x ∈ D_t only contains annotations for the classes C_t of interest, this will result in the loss being dominated by the absence of old classes, posing challenges to foreground-background balancing. 

Benefiting from the internal decoupling of dynamic object queries, we are able to exclusively detect new classes C_t using the corresponding queries Q_t. Considering that the image is only annotated for the specific classes, the partial loss is more reliable. For computing the partial loss, the old classes are considered as backgrounds by Q_t, which is compatible with the incomplete real labels y_t. After each incremental step of training, the partial loss from Eq.(<ref>) is considered as the risk score to guide the subsequent selection of exemplars.

    Υ←ℒ_partial=ℒ_t^DETR(e_t, y_t).









The risk score can measure the quality of incomplete class labels and bounding boxes. Additionally, it also takes the number of annotated objects like in <cit.> into account. As in Fig. <ref>, images with low risk, which constitute a high proportion, provide limited information for optimization, while those with high risk are likely to be outliers with incorrect annotations. Based on the risk estimation, we construct the exemplar set ϵ_t by sorting and selecting the middle part of risk-balanced samples in D_t. The exemplar set ϵ_t is merged with ϵ_t-1 to form a set ϵ_1:t for partial calibration of the model.


In incremental step t, the exemplar sets ϵ_1:t are used to finetune the model after training with D_t. Previous IOD methods typically finetune the model directly with dataset ϵ_1:t, without addressing the issue of missing labels. That is, an image x in the balanced exemplar set ϵ_1:t is only annotated for a specific class subset from {C_1, ...,C_t-1, C_t}, with the absence of annotations for other classes. The confusing, even contradictive supervision hinders the process of prediction calibration. An intuitive way is to use pseudo labels, yet the quality of pseudo labels is hard to guarantee. Thanks to the dynamic object queries and associative disentangled computation, we propose to perform partial calibration that relies only on the incomplete real labels. Specifically, we calculate the partial loss between the outputs of the corresponding queries and the ground-truth annotations in ϵ_1:t. This type of partial calibration further mitigates the forgetting.




§ EXPERIMENT







 §.§ Experimental Setup


Dataset and evaluation metrics. Following<cit.>, we conduct experiments on the widely-used COCO 2017 dataset<cit.>, which consists of images from 80 object categories in natural scenes. The standard COCO metrics as AP, AP_50, AP_75, AP_S, AP_M, AP_L are used for performance evaluation.



Protocols. We evaluate DyQ-DETR with two protocols: 1) traditional protocol <cit.> (Tab.<ref>-left) and 2) revised protocol proposed by<cit.> (Tab.<ref>-right). Protocol 2) avoids observing the same images at different stages. Therefore, we adopt protocol 2) for all subsequent experiments, and the details of protocol 1) can be found in the appendix. For protocol 2), we adopt both two-phase and multiple-phase settings, which can be formulated in the form of c_1+c_2+...+c_T, where c_t represents the number of new classes in incremental step t (c_t=|C_t|) and the sum of c_t is denoted as c (c=|C_1:T|). At time step t, we observe a fraction c_t/c of the training samples with c_t new categories annotated. We test settings c_1+c_2+...+c_T = 40+40, 70+10, 40+20 × 2, and 40+10 × 4. Following <cit.>, we also set the total memory budget for the exemplars to be 10% of the total dataset size.


























































































Implementation details. Following <cit.>, we build DyQ-DETR on top of Deformable DETR <cit.> without iterative bounding box refinement and the two-stage variant. The backbone is ResNet-50<cit.> pretrained on ImageNet<cit.> and the training configurations for the initial stage are consistent with<cit.> to maintain uniform performance in the initial phase. We denote the initial number of queries for the detector as N (typically 300). For both two-phase and multi-phase settings, we dynamically expand by N queries at each incremental step t and the initial parameters of Q_t are inherited from Q_t-1. At step t, old queries Q_1:t-1 are frozen during the incremental training and unfrozen in subsequent exemplar replay. 
We train the model for 50 epochs, and for an additional 20 epochs during fine-tuning. All the experiments are performed on 8 NVIDIA GeForce RTX 3090, with a batch size of 8.




 §.§ Quantitative Results


Two-phase setting. We compare our DyQ-DETR with LwF<cit.>, iCaRL<cit.>, RILOD<cit.>, SID<cit.>, ERD<cit.>, and the previous SOTA method CL-DETR<cit.>. For each setting, we provide the performance of different methods with/without Exemplar Replay (ER). The metrics by joint training are also presented as the upper bound for reference. Tab.<ref> shows that, in the two-phase settings, our proposed DyQ-DETR consistently outperforms the aforementioned methods under different protocols with significant margins. For protocol 2), with exemplar replay, the DyQ-DETR achieves the AP of 39.7% and 41.9% under 40+40 and 70+10 settings. It surpasses CL-DETR by 2.2% AP and 1.8% AP under the 40+40 and 70+10 settings, respectively. Compared with the upper bound, the DyQ-DETR obtains an average performance gap of 1.4%, which is much smaller than the 3.4% gap of CL-DETR. To evaluate the model's ability to preserve old knowledge and learn new knowledge, we conduct a comparison with CL-DETR in the 40+40 setting, where both capabilities are equally important. CL-DETR achieves an old class AP of 39.7% and a new class AP of 36.3%, while our method achieves much better results with 41.3% AP for old classes and 38.6% AP for new classes, respectively. This validates that our proposed method achieves better stability and plasticity, thus addressing the catastrophic forgetting effectively.



Additionally, Tab.<ref> include a performance comparison without exemplar replay (w/o ER) to show the effectiveness of dynamic object queries. In non-exemplar scenarios, DyQ-DETR demonstrates a more significant advantage, outperforming CL-DETR by 2.9% AP and 5.6% AP in the 40+40 and 70+10 settings, respectively. It is noteworthy that the performance of our method without ER is comparable or even exceeds that of the existing method with ER. For example, the performance of our DyQ-DETR w/o ER is 1.6% AP higher than that of CL-DETR w/ ER in the 40+40 setting.

Multiple-phase setting. We conduct experiments in the more challenging 40+20×2 and 40+10×4 settings. The changing AP and AP_50 along with time steps are presented in Fig.<ref>. Our DyQ-DETR consistently outperforms other IOD methods. Moreover, in both settings, the AP improvements of DyQ-DETR become more pronounced with the increase of incremental steps.


r0.5
  

  
    < g r a p h i c s >

  
  Comparison of parameter (left) and complexity overhead (right) with the addition of 100 queries at each step. DSA denotes disentangled self-attention.
  

  


Scalability. As shown in Fig.<ref>-left, by expanding queries rather than the network structure, the additional parameter overhead of our method is almost negligible. As illustrated in Fig.<ref>-right, the computational overhead of our method grows linearly because of the removed inter-group query interaction. Since the computational load of the decoder (excluding the portion shared by different query groups) constitutes a small portion (∼6%) of the entire model's computation, our computational complexity increases at a slow linear rate. Specifically, with 20 stages and an increment of 100 queries per stage (note that the standard Deformable DETR has 300 queries), DyQ-DETR only increases the parameters and GFLOPs by 2% and 39% respectively compared to the standard Deformable DETR, confirming its scalability.







 §.§ Ablation Study


Effect of dynamic object queries. Tab.3 illustrates the ablation study over dynamic object queries in the 70+10 setting. Compared to the baseline, the naive way of expanding queries (+Nat Query) increases the AP to 34.5% and AP_50 to 52.7%, by 1.1% and 4.2%, respectively. This can be attributed to model capacity. By equipping the dynamic object queries (+Dy Query) with isolated matching on independent query set, we further obtain 3.6% AP and 3.5% AP_50 improvements. It validates that dynamic object queries is effective to retain old knowledge during the learning of new knowledge. Furthermore, freezing the task-specific old queries (+DyFro Query) during the incremental training leads to slightly more improvement of 0.4% AP.
  

Components for exemplar replay. We introduce the risk-balanced exemplar selection (RS) and partial calibration (PC) for exemplar replay. The ablative results of the two components in the 70+10 setting are provided in Tab.<ref>. We compare the RS and PC with the distribution-preserving selection (DS) and direct calibration (DC) proposed in CL-DETR <cit.>, respectively. 
By separately applying RS and PC, we increase the AP by 0.5% and 0.9%, respectively. Applying the whole risk-balanced exemplar selection and partial calibration obtains 41.9% AP, exceeding the baseline by 1.5%.








Effect of disentangled self-attention. The comparison in Tab.<ref> under both the 40+40 and 70+10 settings shows that removing the self-attention interaction between different query groups has almost no impact on performance. This is reasonable because the class sets detected by different query groups do not overlap, thus eliminating the need for self-attention interaction to remove duplicate predictions. Combined with Fig.<ref>-right, by disentangling the self-attention computation, the computational complexity can be reduced from quadratic to linear growth without performance drop.

Effect on the number of expanded object queries. Tab.<ref> presents the APs with changing number of object queries. It indicates that the expanded queries number has a trivial impact. By expanding fewer queries, we can decrease complexity with negligible harm to performance.


Visualization of decoupled queries. In Fig.<ref>, we visualize the decoupling behavior of  dynamic queries on the test set, based on the 40+20×2 setting. The classes "bicycle", "person", and "car" appear in phases 1, 2, and 3, respectively. They are detected by queries in Q_1, Q_2 and Q_3 accordingly, in a decoupling manner. It can be observed that Q_1 detects the "bicycle" class accurately, while Q_2 and Q_3 consider it as background. Once a query group Q_t learns to detect the class set C_t at step t, its class-specific knowledge remains unchanged thereafter, requiring only the maintenance of old knowledge, which significantly improves the performance of IOD.







§ CONCLUSION



In this paper, we propose DyQ-DETR for incremental object detection. Distinct from the mainstream methods focus on the distillation mechanism, we address the catastrophic forgetting by taking inspiration from dynamic networks for model capability expansion. In particular, we propose the dynamic object queries with incremental assembly of new object queries, disentangled self-attention computation and isolated bipartite matching over object queries from different time. The DyQ-DETR can alleviate the conflict between outdated background knowledge and continually emerged classes, thereby achieving stability-plasticity tradeoff. Benefiting from the isolated supervision of dynamic object queries, we further propose risk-balanced partial calibration for effective exemplar replay, with the idea to select exemplars based on risk and partially finetunes the model without relying on low-quality pseudo labels. Extensive experiments demonstrate that our proposed DyQ-DETR surpasses existing IOD methods by a large margin, with quite limited memory overhead. Besides of dynamic query, we hope that more various ways of model expansion can be explored in IOD.












§ APPENDIX



In Sec.<ref>, we present the algorithmic pipeline of DyQ-DETR. In Sec.<ref>, we introduce a simple but strong baseline with only exemplar replay to emphasize the importance and rationale of the non-exemplar setting. In Sec.<ref>, we provide more detailed experimental results. This includes details of the traditional protocol (Sec.<ref>), more fine-grained results about old classes and new classes (Sect.<ref>), and more ablation results (Sec. <ref>). In Sec.<ref>, we show more visualization results, including images with different levels of risk, behavior of decoupled queries, and performance comparison with CL-DETR. Please note that all experiments, except for those in Sec.<ref>, are based on the revised protocol, using Deformable DETR on COCO 2017, and we will not specify this further.



§ ALGORITHMIC PIPELINE OF DYQ-DETR












To get a more comprehensive understanding of the overall framework, we describe the training pipeline of DyQ-DETR in Algorithm <ref>. In the τ-th phase, we dynamically add a new set of queries, which are responsible for detecting and recognizing new classes. We disentangle self-attention modules in the decoder, ensuring that queries between different groups remain relatively independent. In the incremental training, we use a score threshold to filter out background predictions, resulting in the pseudo-labels y^pseudo. Instead of simply merging pseudo-labels y^pseudo and incomplete real labels y, we divide the completed annotations of an image according to the incremental phase/class set. For an image, annotations corresponding to different class sets act on the outputs of respective queries, thus resulting in the decoupled loss ℒ_total^DETR for training.

After the incremental training, we utilize the trained model to calculate the partial loss as the risk score for images in D_τ. We sort D_τ based on the risk score and simply select the middle 10% of these samples to serve as exemplars. Then, for an image in the exemplar set ϵ_1:τ, we identify the specific 
stage corresponding to its annotation, and we use the respective output and real labels to compute the partial loss ℒ_partial for calibration.




§ BASELINE WITH ONLY EXEMPLAR REPLAY



We introduce a simple baseline, which only combats forgetting by saving exemplars of the same size. As shown in Tab.<ref>, it completely forgets past knowledge after the incremental training, but its performance significantly improves after the simple exemplar replay. The total memory budget for exemplars is typically set as 10% of the total dataset size. This relatively large proportion may obscure issues in the incremental training and potentially diminish the advantages of superior methods. Therefore, we believe that comparisons in non-exemplar settings are more suitable for evaluating the performance of different methods. 




§ MORE DETAILED EXPERIMENTAL RESULTS






 §.§ Traditional protocol and results

















































The main difference between the traditional protocol<cit.> and the revised protocol<cit.> lies in data partitioning. In the traditional protocol, the model can observe all images containing at least one object of the currently interested classes, which may result in an image appearing across multiple phases. Formally, let D={(x,y)} denotes a dataset with images x and corresponding object annotations y. First, we divide the total class set into non-overlapping parts {C_1, C_2,..., C_T}, one for each incremental phase. For each phase τ, all samples in D retain annotations of C_τ and drop others. The incremental training dataset in phase τ consists of images that contain at least one annotation for C_τ. Additionally, the revised protocol shuffles the class order, while the traditional one does not, leading to different class partitions.

Tab.<ref> shows that our proposed DyQ-DETR consistently outperforms the state-of-the-art (SOTA) method in the traditional IOD protocol, especially in the non-exemplar settings. Specifically, without exemplars, our DyQ-DETR surpasses the SOTA by 2.2% AP and 3.7% AP in the 40+40 and 70+10 settings, respectively. With exemplars, the advantage decreases to 0.4% AP and 2.0% AP, respectively. The substantial advantage of our DyQ-DETR in non-exemplar settings demonstrates the effectiveness of dynamic query, which is the core of our method. 







 §.§ Results about old classes and new classes


Tab.<ref> presents fine-grained results in the 40+40 setting about old classes and new classes, which respectively represent the model's stability and plasticity. The results show that, compared to the SOTA, our DyQ-DETR achieves improvements of 1.6% AP for old classes and 2.3% AP for new classes, indicating enhanced stability and plasticity in our method. 



 §.§ More ablation results



Effect of the score threshold. In the 40+40 setting, we set the score threshold to 0.6, while for other settings, it is set to 0.4. This is because, in other settings, there is a higher proportion of old classes where stability is preferred, and more old class pseudo-labels help combat forgetting. The ablation study on the score threshold is shown in Fig.<ref>(a), where the optimal threshold for the 40+40 setting is 0.6. We also find that when the score threshold is low, the pseudo-label noise is significant, leading to poor performance in non-exemplar settings. However, after exemplar replay, the performance greatly improves, further illustrating that exemplar replay may mask issues in the incremental training.



Effect of the selected risk interval. After sorting the images by risk in ascending order, the results of changing the selection interval are illustrated in Fig.<ref>(b). It can be observed that selecting images with moderate risk is optimal. This selection strategy is straightforward, and our contribution lies more in demonstrating the feasibility of using the model to optimize the selection.



§ MORE VISUALIZATION RESULTS




Images with different levels of risk. As illustrated in Fig.<ref>, images with low risk are considered as simple samples containing only a few annotated objects, while images with high risk are characterized by a dense distribution of objects, accompanied by severe occlusion or inaccurate annotations. Samples in the middle part are considered informative and reliable, 
and such an active selection strategy promotes more effective exemplar replay.









Behavior of decoupled queries. As depicted in Fig.<ref>, our designed dynamic query set achieves the goal of only detecting the corresponding class set. The old query set is responsible for memorizing knowledge of old classes, while the new query set focuses on learning new knowledge. This decoupling significantly improves the performance in non-exemplar settings.

Performance comparison with CL-DETR.
Fig.<ref> shows the visualized detection results of CL-DETR and our DyQ-DETR in non-exemplar and exemplar-based settings. In both settings, DyQ-DETR presents more robust performance for remaining old knowledge and learning new knowledge, thus greatly mitigating catastrophic forgetting.





§ DISCUSSION OVER OPEN VOCABULARY OBJECT DETECTION



Open vocabulary object detection (OVD) leverages visually related language data as auxiliary supervision to bridge the gap between base categories and novel label spaces<cit.>. By using image-caption pairs, it acquires an extensive vocabulary of concepts, allowing the model to learn object detection with annotations only for certain base categories. Consequently, the model can detect both base classes and novel classes that were not present in the training annotations. OVR-CNN<cit.> is the pioneering work that introduced the concept of open vocabulary object detection and established a comprehensive framework to address this problem. Subsequently, with the introduction of large-scale visual language pretraining models such as CLIP<cit.>, novel approaches have emerged that utilize these pretrained models to enhance the capability of open vocabulary object detection. Some works<cit.> employ knowledge distillation, using large model text encoders to align visual features with the large model and transfer new class information. Other works address the gap in object detection region localization by converting image-text level pre-training to region-text level<cit.>, while some complete this during the detector training stage after pre-training<cit.>.

The difference between IOD (Incremental Object Detection) and OVD (Open Vocabulary Detection) lies in their focus. IOD emphasizes the continuous learning and updating process of the model, while OVD focuses on the model's generalization ability. IOD aims to improve the model through multiple learning steps, whereas OVD aims for the model to generalize perfectly to other problems with just training once. Specifically, after training, OVD can detect some objects in a new category space, even if these objects have not been seen in the annotations of base categories, due to the rich semantic concepts obtained during the image-caption alignment process. However, despite the richness of the learned semantic concepts, OVD is difficult to handle newly generated concepts due to the dynamic nature of the environment and requires re-alignment of image-caption pairs to learn these new concepts. In contrast, IOD simplifies this process by allowing incremental learning of new concept based on newly collected and annotated data, thus avoiding the need for retraining the entire model.

IOD and OVD are not mutually exclusive but complementary. They are both important for open-world practical applications. As mentioned above, OVD still shows deficiency in handling the emergence of new concepts without retraining, whereas IOD simplifies this process, saving time and computational resources. However, IOD struggles to resist catastrophic forgetting of old class semantics during continuous model updates. OVD relies on pre-learned rich semantic concepts for novel classes, without consideration of forgetting issue.




§ LIMITATIONS AND BROADER IMPACTS


We propose a dynamic object queries-based detection transformer (DyQ-DETR) to address catastrophic forgetting in incremental object detection. While it scales well to tasks with an incremental step of 20, DyQ-DETR may face challenges in scaling to tasks with longer incremental steps.

High-performance incremental object detection systems have tremendous potential to have a significant impact on various fields and inspire innovative research approaches in robotics, autonomous driving, and beyond. For example, our proposed DyQ-DETR can progressively enhance the recognition and manipulation capabilities of robots, thereby improving efficiency and productivity in industries and healthcare. Furthermore, considering the significant overhead of joint training, DyQ-DETR helps detection models avoid retraining from scratch, providing an effective and efficient approach for incremental updates of large visual models. As for potential widespread impact, our work has not shown any negative social consequences.




















§ NEURIPS PAPER CHECKLIST







  * Claims
    
   Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
    
   Answer:  
    
   Justification: We have clearly claimed the scope, contribution and related experiments in the abstract and introduction sectoin.
    


  * Limitations
    
   Question: Does the paper discuss the limitations of the work performed by the authors?
    
   Answer:  
    
   Justification: We have discussed at the end of the appendix.
    


  * Theory Assumptions and Proofs
    
   Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
    
   Answer:  
    
   Justification: We do not include theoretical results.
   

    
  * Experimental Result Reproducibility
    
   Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
    
   Answer:  
    
   Justification:  We have included details in the beginning of the experiments section and in the appendix.
    



  * Open access to data and code
    
   Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
    
   Answer:  
    
   Justification: We are currently organizing the code and we guarantee that it will be open-sourced.
    



  * Experimental Setting/Details
    
   Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
    
   Answer:  
    
   Justification: We have included details in the beginning of the experiments section and in the appendix.
    


  * Experiment Statistical Significance
    
   Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
    
   Answer:  
    
   Justification: The experimental results are quite stable, so error bars are not necessary.
    


  * Experiments Compute Resources
    
   Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
    
   Answer:  
    
   Justification: We have included details in the beginning of the experiments section and in the appendix.
    
    

  * Code Of Ethics
    
   Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics <https://neurips.cc/public/EthicsGuidelines>?
    
   Answer:  
    
   Justification: We have read and conformed the the NeurIPS Code of Ethics.
   



  * Broader Impacts
    
   Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
    
   Answer:  
    
   Justification: We have discussed at the end of the appendix.
    
    

  * Safeguards
    
   Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
    
   Answer:  
    
   Justification: We will include this in the open-sourced code in the future.
    


  * Licenses for existing assets
    
   Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
    
   Answer:  
    
   Justification: We have cited all the original papers.
   


  * New Assets
    
   Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
    
   Answer:  
    
   Justification: We do not release new assets.
    


  * Crowdsourcing and Research with Human Subjects
    
   Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 
    
   Answer:  
    
   Justification: We do not involve crowdsourcing nor research with human subjects.
    


  * Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
    
   Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
    
   Answer:  
    
   Justification: We do not involve crowdsourcing nor research with human subjects.
    






 Paper Title: Improving Image De-raining Using Reference-Guided Transformers.




Diff3DETR: Agent-based Diffusion Model for Semi-supervised 3D Object Detection
    Jiacheng Deng10000-0003-2838-0378 Jiahao Lu1 0009-0006-3280-9447 Tianzhu Zhang1, 2 † 0000-0003-1856-9564
    
============================================================================================================


§ ABSTRACT 


Image de-raining is a critical task in computer vision to improve visibility and enhance the robustness of outdoor vision systems. While recent advances in de-raining methods have achieved remarkable performance, the challenge remains to produce high-quality and visually pleasing de-rained results. In this paper, we present a reference-guided de-raining filter, a transformer network that enhances de-raining results using a reference clean image as guidance. We leverage the capabilities of the proposed module to further refine the images de-rained by existing methods. We validate our method on three datasets and show that our module can improve the performance of existing prior-based, CNN-based, and transformer-based approaches.










Image de-raining, transformers




§ INTRODUCTION




Image de-raining is an essential task in computer vision as rain streaks can decrease visibility and deteriorate the robustness of most outdoor vision systems. De-raining has been widely applied in a wide range of practical applications, including autonomous driving <cit.> and surveillance systems <cit.>, as an essential pre-processing step. 






Early approaches that solve the task with hand-crafted priors such as sparse coding <cit.> and Gaussian mixture model <cit.> are formulated to explicitly model the physical characteristics of rain streaks. However, they often fail under complex rain conditions and show over-smoothed images <cit.>.




The advent of Convolutional Neural Networks (CNNs) has led to substantial advances in single image de-raining <cit.>. CNNs, however, have limited receptive fields, which means that the pixel value estimation for each spatial location primarily depends on small local surroundings. Therefore, due to the limited capacity for modeling long-range spatial context <cit.>, CNN-based methods often struggle with accurately detecting heavy rain streaks, leading to blurred results <cit.>.
Transformer-based methods <cit.> have emerged as a promising alternative as they can better capture non-local information, enhancing image reconstruction quality. However, these approaches do not model local image details well, which are crucial for achieving clear image restoration <cit.>. This limitation arises from the self-attention mechanism in Transformers, which does not adequately handle the local invariant properties, in contrast to CNNs.
While single image de-raining methods have made significant progress, there remains room for improvement in their ability to handle diverse and challenging rain conditions.[More results and the code: <http://ziiihooo.com/blog/2024/derain/>]

















In this paper, we propose a novel framework for image de-raining. We use existing de-raining models as baselines and present a reference-guided de-raining filter that extracts useful feature information from a reference clean image to compensate for the baseline results. The key insight is to transfer useful features from a reference clean image. Our framework consists of a feature extractor, a feature attention module, and a feature fusion module. Given a rainy image and a reference rainy image as input, we first estimate the de-rained images using an existing de-raining model. We use these results with a reference clean image as input for the feature extractor that extracts the multi-scale features of each image. The feature attention module uses these features as input to estimate useful features from a reference rain/clean image. At this point, the feature attention module computes the most relevant feature patch from the reference clean image. Finally, we introduce the feature fusion module to aggregate multi-scale features. 

 




















We summarize our main contributions as follows:





  * We propose a novel framework to integrate existing de-raining methods into a reference-guided de-raining filter that captures useful features by leveraging reference images.

  * Our method can be used with a wide range of existing methods in a plug-and-play manner.

  * Experimental results show that our method improves the performance of existing methods, from a prior-based to a state-of-the-art method.



 








§ PROBLEM FORMULATION
















Given an input rainy image X_r, existing learning-based single image de-raining methods aim to learn a model f_θ_S(·), parameterized by θ_S, that can generate an estimated clean image X̂_c = f_θ_S(X_r) in which the rain streaks are removed. The model f_θ_S(X_r) is learned by minimizing the error between X̂_c and the ground-truth clean image X^gt_c using a loss function ℒ as:




    min_θ_S ℒ(f_θ_S(X_r), X^gt_c).



In this paper, we propose to further employ a reference clean image R_c as guidance to improve the result of existing de-raining models and generate the enhancement output X̂^out_c. We present a model g_θ_R(·), parameterized by θ_R, that aims to extract useful feature information from R_c to compensate for X̂_c, resulting in the final enhancement output X̂^out_c = g_θ_R(X̂_c, R_c). Namely, our model learns to minimize the following loss function:  


    min_θ_R ℒ(g_θ_R(X̂_c, R_c), X^gt _c).








§ METHOD


Figure <ref> shows the overview of our framework. The proposed reference-guided de-raining filter (RDF), g_θ_R(·), is designed to extract useful feature information from R_c to compensate for X̂_c obtained from an existing baseline model. 
Given X̂_c, we first perform image retrieval to find R_c from an image database. We then obtain the synthesized rainy image R_r by synthesizing the rain streaks to R_c <cit.>. By using R_r as input to the baseline model, we can estimate the reference de-rained image R̂_c. By capturing the similarity between the two de-rained images, X̂_c and R̂_c, RDF aims to transfer the useful information from a reference clean image R_c to X̂_c, generating the enhancement output X̂^out_c.


RDF mainly consists of three components: feature extractor, feature attention and feature fusion.  
The feature extractor first projects the images X̂_c, R̂_c, and R_c into the features P_X̂_c, P_R̂_c, and P_R_c, respectively. The feature attention module firstly computes the attention weights taking P_X̂_c as query (Q) and P_R̂_c as key (K).  Attention embedding module outputs the highest relevance as soft attention maps S at patch level and indexes corresponding to highest relevance at patch level as hard attention maps H. In the hard attention module, H is further used to select the most relevant patch from the paired value (V), P_R_c, estimating the feature P, the useful feature extracted from the reference clean image. In the feature fusion module, P is re-weighted at the patch level using the soft attention maps S. The re-weighted feature is then integrated with the de-rained image through the Cross-Scale Feature Integration (CSFI) stage. Finally, the fused feature is back-projected into the image space to produce the enhancement output X̂^out_c.





 §.§ Feature extractor








The feature extractor module, including several convolution blocks, is designed to project images into a feature space. Specifically, the feature extractor maps a single image into three distinct feature levels. The Level-1 feature,  P_{·}^1 ∈ℝ^(B, C, H, W), preserves the input image size but with a higher-dimensional channel space C. The Level-2 and Level-3 features are represented at lower resolutions and increased channel dimensions, specifically denoted as P_{·}^2 ∈ℝ^(B, 2C, H/2, W/2) and P_{·}^3 ∈ℝ^(B, 4C, H/4, W/4), respectively. In this context, the feature P_{·}≜{ P_{·}^1, P_{·}^2, P_{·}^3} aggregates these three-level features extracted from an input of the feature extractor.



 §.§ Feature attention

The feature attention module receives three different features, P_X̂_c, P_R_c, and P_R̂_c. This module maps the query P_X̂_c to useful feature patches in the context of the key P_R̂_c and value P_ R_c, subsequently outputting the useful features P along with the corresponding relevance maps at the patch level. During the attention embedding stage, only the Level-3 features P_X̂_c^3 and P_R̂_c^3 are utilized to compute relevance for patches, as they encapsulate more abstract information and a wider receptive field. Following this, the hard attention module selects the most relevant feature patch of P_R_c for the patches of P_X̂_c, using the relevance just computed at every feature level.
In Figure <ref>, we visualize the attention maps, where we could see that similar areas are highly noticed, which are used to compensate for de-rain results later.




 §.§ Feature fusion


In Figure <ref>. the CSFI module represents a well-established method for blending features with various scales <cit.>. In this paper, we leverage this approach to fuse P, the useful features extracted from the reference, with the original de-rained image. The useful feature is incrementally added to the original de-rained image X̂, guided by relevance maps. During the compensation stage, the CSFI module is employed in conjunction with the residual block to facilitate information sharing across all feature levels. The level-1 feature, characterized by its relatively precise information and intricate details, is compensated to the image first. Conversely, the level-3 feature, which encapsulates more abstract information, is compensated to the image last. This systematic process results in a projection from the feature space to the image space, generating the final output X̂_c^out.
















 §.§ Loss

During the training phase, our approach consists of initialization and fine-tuning stages. In the initialization stage, the model is trained to transfer useful features from clean object images using an L1 loss function. The model learns the transformation from a derained to a clean image in this stage, ensuring the model has the capability to perform deraining on simple rainy images. In the fine-tuning stage, the model is further trained to transfer useful features from reference clean images to better simulate real-world application scenarios. For the independent objectives of these two stages, we employ the loss functions tailored to each stage's specific requirements. During the initialization stage, the loss function is defined as the ℓ1 reconstruction loss:


    ℒ (X̂_c^out, X_c) = X̂_c^out - X_c_1.


In the fine-tuning stage, we apply the MS-SSIM-L1-Loss <cit.>:


    ℒ (X̂_c^out, X_c) = α_1 X̂_c^out - X_c_1 + α_2 (1 - SSIM(X̂_c^out, X_c)),

where α_1 and α_2 are the hyperparameters to control the effect of each loss.






































§ VALIDATION





 §.§ Experiment setup



In our framework, we use existing de-raining models as baselines. We adopted three baseline models, including the prior-based (GMM <cit.>), CNN-based (PReNet <cit.>), and transformer-based (Uformer <cit.>) models. Except for the prior-based method that does not require training, we used public codes for training each baseline on each dataset.


















For the dataset, we require both a clean/rainy image pair and a reference clean/rainy image pair that contain similar scenes. However, existing rain benchmarks, such as Rain100L <cit.> and DID <cit.>, have limited similar scenes, providing unreasonable reference images. Therefore, we constructed the dataset as follows:



  * BDD100K-Rain: We used BDD100K <cit.>, a large-scale driving scene dataset, and synthesized the rain streaks by using SyRaGAN <cit.> and obtained 256× 256 images. 

  * Cityscapes-Rain/KITTI-Rain: This dataset includes 256× 1024 images and constructed by <cit.>, which renders rain streaks to evaluate bad weather. 




For image retrieval, we implemented the reference image retrieval method at the image hash level. First, all images in the dataset are projected into the image perceptual hash space. Then, for every image that requires compensation, the nearest neighbor image is selected as the reference image. 




We utilized 8 A100 GPUs and PyTorch for our experiments. The channel numbers for levels 1, 2, and 3 are set to 64, 128, and 256, respectively. The convolution blocks in the feature extractor module are initialized with VGG-19 <cit.>. At the feature attention stage, the patch sizes for levels 1, 2, and 3 are set to 12, 6, and 3, respectively. Within the feature fusion module, the features for levels 1, 2, and 3 are configured as (64, H, W), (128, H/2, W/2), and (256, H/4, W/4), respectively. To ensure that our model can capture useful features and compensate accurately, we initially employ the input clean image X_c as the reference image for initialization training using the loss functions in (3). The initialization training is done for each model on each dataset. After the initialization training stage, X_c is replaced with the actual reference image in the subsequent training stage for fine-tuning. In the fine-tuning stage, we set α_1 = 0.6 and α_2 = 0.4 and use the loss function in (4). 








 §.§ Discussion


Table <ref> presents the quantitative results of the baselines and the improvements achieved using our method. Although each baseline employs a different methodology for image de-raining, our method can universally enhance the performance of these baseline models. This suggests that our reference-guided de-raining filter effectively extracts useful features from reference images. The improvement is most significant on the KITTI-Rain dataset, as this dataset provides better reference images. Our pipeline also demonstrates various degrees of compensation effects on different baselines. For Uformer <cit.>, which achieves the best results among the three baseline models, our method shows relatively small improvements. On the other hand, for GMM <cit.>, which is the earliest method among the three baseline models, our method shows more substantial improvements.











		
  
We further evaluate the effect of reference images on the BDD100K-Rain dataset, specifically to analyze how a reference image can contribute to the de-raining process. Using PReNet as a baseline model, we analyze the effect of reference images using three different image types: ground truth clean images, Gaussian-noise images, and our reference images collected using image retrieval. As shown in Table <ref>, ground truth images, which encapsulate all the useful information, yield the best results and significantly enhance the performance. Noise images, that include less relevant information, produce the worst outcomes. Reference images containing similar scenes provide results that fall between the upper bound set by the ground truth images and the lower bound established by the noise images. The results indicate that our model can transfer useful features from reference images, and the degree of enhancement primarily depends on the quality of the reference images.






§ CONCLUSION


This paper introduces a novel framework for image de-raining that leverages a reference-guided de-raining filter, a transformer network that enhances existing de-raining results using a reference clean image as guidance. 
Our framework, as plug-and-play de-raining enhancement, shows performance improvements of prior, CNN, and transformer-based models across multiple datasets. As future work, we will integrate our method with text-to-image generation models that can synthesize clean images and use these images as references for de-raining. 
















IEEEbib




 Paper Title: Skeleton-Based Action Recognition with Spatial-Structural Graph Convolution.






Skeleton-Based Action Recognition with Spatial-Structural Graph Convolution
    Jingyao WANG
LIMOS, CNRS UMR 6158 
 LPC, CNRS UMR 6533 

Université Clermont Auvergne 
jingyao.wang@uca.frIssam FALIHLIMOS, CNRS UMR 6158 
Université Clermont Auvergne

issam.falih@uca.frEmmanuel BERGERET LPC, CNRS UMR 6533 
Université Clermont Auvergne 
emmanuel.bergeret@uca.fr

    August 5, 2024
============================================================================================================================================================================================================================================================================================




§ ABSTRACT 
[Code is available at: <https://github.com/jingyaojade/SpSt-GCN>] 
Human Activity Recognition (HAR) is a field of study that focuses on identifying and classifying human activities. Skeleton-based Human Activity Recognition has received much attention in recent years, where  Graph Convolutional Network (GCN) based method is widely used and has achieved remarkable results. However, the representation of skeleton data and the issue of over-smoothing in GCN still need to be studied. 1). Compared to central nodes, edge nodes can only aggregate limited neighbor information, and different edge nodes of the human body are always structurally related. However, the information from edge nodes is crucial for fine-grained activity recognition. 2). The Graph Convolutional Network suffers from a significant over-smoothing issue, causing nodes to become increasingly similar as the number of network layers increases.
Based on these two ideas, we propose a two-stream graph convolution method called Spatial-Structural GCN (SpSt-GCN). Spatial GCN performs information aggregation based on the topological structure of the human body, and structural GCN performs differentiation based on the similarity of edge node sequences. 
The spatial connection is fixed, and the human skeleton naturally maintains this topology regardless of the actions performed by humans. However, the structural connection is dynamic and depends on the type of movement the human body is performing. Based on this idea, we also propose an entirely data-driven structural connection, which greatly increases flexibility.  
We evaluate our method on two large-scale datasets, i.e., NTU RGB+D and NTU RGB+D 120. The proposed method achieves good results while being efficient. 



Human Activity Recognition, Deep learning, Skeleton Data, Graph Convolutional Network.






§ INTRODUCTION



Human activity recognition (HAR) is a research field that spans computer science and electronic engineering. Its goal is to identify and classify human activities using algorithms. Human Activity Recognition (HAR) has received significant attention in recent years and has been applied in various fields such as healthcare, sports, security, smart homes, and wearable devices. The aim is to enhance human well-being, safety, and productivity by offering insights and information for decision-making, monitoring, or feedback purposes<cit.>. The HAR process involves several steps: data collection, data preprocessing, feature extraction, and activity recognition<cit.>. 

Various data modalities are used for different scenarios. They could be divided into two groups: visual modality and non-visual modality. RGB and skeleton data are two common types of visual modalities. RGB data is typically captured with a camera and contains a wealth of color and contour information. This data is widely used in security cameras, intelligent robots, and other applications. Skeleton data is usually extracted from RGB data or collected by depth cameras. Compared to RGB data, 3D skeleton data effectively captures spatial-temporal properties. It contains less data but has higher computational efficiency<cit.>. At the same time, it protects privacy. Skeleton data is a good choice when computing resources are limited or privacy protection is required. Acceleration data is typically non-visual data. The acceleration sensor is small, lightweight, and designed to protect privacy. It can accurately and comprehensively measure the spatial acceleration of an object without prior knowledge of its direction of motion. This sensor is commonly used in low-cost and low-power-consumption portable devices, such as smartwatches. Each of the various data modalities has its suitable application scenario. In this article, we focused on visual data, specifically on skeleton data. 

Currently, the predominant approach involves using spatial-temporal convolutional networks to extract features from skeleton data. Researchers utilize GCN to extract spatial features within frames, and then utilize TCN to extract temporal features between frames. The researcher designs the skeletal connection diagram based on the human skeleton's topology. When performing graph convolution, the adjacency matrix is utilized to depict the connections between nodes and perform message passing between neighboring joints<cit.>. The issue is that human beings have symmetrical structures, and there should be not only spatial connections but also structural connections between joints. 

The action of "clapping hands" requires both hands to come together. There is a strong connection between the two hands. However, a mere spatial connection would not adequately represent the connection between the hands, and we need to depict this connection with a structural link. We believe that distinguishing fine-grained activities requires a focus on the activities of edge nodes. Taking typing on the keyboard and writing as an example<cit.>, the most critical metric to distinguish between these two actions is the fine-grained movement of the hand. In terms of spatial structure, edge nodes are the least connected to other nodes and farthest away from the center node. This hinders their ability to effectively aggregate information from other nodes. To address this issue, we propose adding extra structural connections to the edge nodes.


The spatial connection is fixed, and the human skeleton naturally maintains this topology regardless of the actions performed by humans. However, the structural connection should be dynamic and depends on the type of movement the human body is performing. There is a strong connection between the hands when clapping, but not when checking the time (from watch) [19]. Based on this idea, we propose a data-driven method to initialize the structural connections based on the similarity of edge node sequences for various samples, so that each sample has its unique structural connection. 
GCN aggregates node information through edge connections to generate new node representations, which is effective but can lead to the problem of over-smoothing. Different from the normal GCN way of aggregating information, we propose performing node information differentiation through the structural branch. Our differentiation method can enhance the contrast between the edge nodes, and also reduce the issue of over-smoothing to some extent.


The main contributions of this paper are summarized as follows:

    
  * We propose a spatial-structural graph convolution method which can better represent symmetrical human structures and edge nodes.
    
  * Instead of initializing the adjacency matrix solely based on the count of adjacent nodes, we adopt a differential approach to initialize the structural connections of various samples. This differential initialization is contingent on the similarity of edge node sequences, enhancing the flexibility of graph convolution. 
    
  * To address the over-smoothing problem inherent in GCN, we utilize the inverse adjacency matrix to discern information pertaining to edge nodes.


The remainder of this paper is organized as follows: Section 2 presents the state of the art papers related to our work. Our proposed method is highlighted in section 3. Section 4 presents the experimental part including data set description, experimental settings, evaluation metrics, experimental results and their analysis and the conclusion is given in section 5.



§ RELATED WORKS


HAR is a classic temporal classification problem. In earlier years, traditional CNN and RNN models were dominant.
Zhang et al.<cit.> designed a two-stream CNN-RNN model to overcome view dependency. The main idea of this paper is to use view adaptation modeling to obtain a variable observation coordinate, allowing for the generation of skeleton data from different viewpoints. An RNN-based view adaptation subnetwork is integrated into the VA-RNN stream, while a CNN-based view adaptation subnetwork is integrated into the VA-CNN stream. Then, an LSTM classification network is applied in the VA-RNN stream, while a CNN classification network is applied in the VA-CNN stream. 
Caetano et al.<cit.> encoded the magnitude and orientation values of the skeleton joints into the image, and then utilized a CNN network for classification. 


In recent years, several Transformer-based and GCN-based models have appeared.
Plizzari et al. <cit.> developed a two-stream spatial-temporal transformer network. On the S-TR stream, a Spatial Self-Attention module (SSA) is used to analyze intra-frame interactions between different body parts, followed by a 2D convolution on the time dimension (TCN). On the T-TR stream, a Temporal Self-Attention module (TSA) is used to model inter-frame correlations, while spatial features are extracted by a standard graph convolution (GCN). 

Skeleton data used for human activity recognition (HAR) tasks are sequential data that contain consecutive frames. Skeleton data represents natural topography. In addition to being used as a matrix, skeleton data could also be utilized as a graph by a graph neural network <cit.>. In addition, the skeleton data used for action recognition is sequential data. GCN and TCN blocks are often used to capture spatial-temporal features, where Graph Convolutional Network (GCN) is used to capture intraframe spatial information by aggregating information from neighboring nodes, and Temporal Convolutional Network (TCN) is used to capture interframe temporal information.
Yan et al.<cit.> propose the Spatial-Temporal Graph Convolutional Networks (ST-GCN) model, which applies alternating convolution of GCN and TCN, and has become the baseline in the field of skeleton-based action recognition. 
Song et al. <cit.> developed an efficient graph convolutional network. They extended the convolutional layer in CNN to the GCN network to extract temporal dynamics and compress the model size. They then employed a compound scaling method that adjusts the width (number of channels) and depth (number of layers) factors to make the model flexible and effective. In addition, Spatial Temporal Joint Attention (ST-JointAtt) is incorporated into each block, and three input branches are fused early in the network. The above strategies enable the network to achieve good results at a low computational cost. 





 §.§ Skeleton Data Representation


Skeleton data consists of a set of joint coordinates, usually represented as 3D points or 2D projections, along with additional information like joint velocities or orientations. These joint coordinates are often normalized to a consistent scale or relative to a specific reference point to enhance generalizability across individuals and improve recognition accuracy. Skeleton data removes video backgrounds, people textures and outlines, which reduces lots of calculations. 3D skeleton data has a good view-invariance. It is possible to change the data view by rotating the skeleton in 3D space. 

Some researchers use matrices or images to represent skeleton data. Yang et al. <cit.> proposed a Tree Structure Skeleton Image (TSSI) which encodes skeleton data by depth-first tree traversal order to image. Banerjee et al. <cit.> extracted four features from skeleton data (Distance feature, Distance velocity feature, Angle feature, Angle velocity feature). These features are then encoded into single-channel grayscale images and are fed to the CNN. 


Skeleton data is a natural topography. In addition to being used as a matrix, skeleton data could also be represented as a graph. Some researchers represent skeleton data as an undirected graph, which effectively illustrates the connections between joints. The joint information will be aggregated bi-directionally. Yan et al.<cit.> represented the skeleton data using natural bone topology based on prior experience. Instead of using a fixed graph, Shi et al. <cit.> propose a data-driven method to enable the model to adapt to various data samples. According to the pattern of human movement, the central joints always drive the peripheral joints. Shi et al. <cit.> represented the skeleton as a directed acyclic graph to better incorporate joint and bone data. 

In addition to representations based on the original topology, some researchers utilize the Node2vec method to obtain graph embeddings. Laines et al. <cit.> conducted a Depth-First-Search (DFS) on the skeleton tree to obtain a new spatial embedding.  Sun et al. <cit.> utilized the breadth-first search (BFS) algorithm, which commences from the root joint (hip center joint) to rearrange the joints.

In addition to utilizing fixed topological structures, some researchers have employed dynamic structures to represent the skeleton data. Ye et al.<cit.> proposed a Context-encoding Network (CeN) to automatically learn the skeleton topology, incorporating contextual features from the remaining joints in a global manner when learning the dependency between two joints. Cheng et al.<cit.> proposed a non-local shift graph operation, which enables each node to have a receptive field covering the entire skeleton graph, allowing for adaptive learning of the relationships between joints. Chen et al.<cit.> proposed Channel-wise Topology Refinement Graph Convolution (CTR-GC), which learns a shared topology across all channels and refines it with channel-specific correlations for each channel.





§ PROPOSED APPROACH

In this work, we utilize the GCN-TCN block to extract spatial-temporal features, where the GCN layer and TCN layer are alternately employed to capture spatial and temporal dependencies. Temporal convolutions are essentially 2D convolutions with a kernel size of 1, designed to capture the temporal dependence of each node. In this section, we introduce our proposed spatial-structural graph convolution and the generation of the structural adjacency matrix. 



 §.§ Spatial-Structural Graph Convolution (SpSt-GCN)

For GCN-based methods,  the input skeleton data is represented as (V, E) with N joints and T frames. Here, V represents joint points, and E represents connections between nodes, often represented by an adjacency matrix. 

The conventional spatial graph convolution is shown below, where f_in and f_outSpatial represent the input and output features, respectively. A_j represents the adjacency matrix at distances j, Λ_j is a weight vector used to normalize A_j. The parameters W_j can be optimized during the training process. 

    f_outSpatial = ∑_jW_jf_inΛ_j^-1/2A_jΛ_j^-1/2

To increase the flexibility of graph convolution, we add a parameterized matrix B as described in <cit.>. The matrix B is initialized to an all-zero matrix of the same size as adjacency matrix A and optimized together with the other parameters in the training process.

    f_outSpatial = ∑_jW_jf_in(Λ_j^-1/2A_jΛ_j^-1/2+B_j)



Many researchers commonly employ the skeletal topology structure for establishing connections among joints, a methodology known for its efficiency albeit lacking in precision. It is crucial to recognize that the human skeleton, characterized by symmetry, presents a nuanced structure wherein even widely spaced symmetrical joints exhibit robust structural correlations. While prevailing graph convolution techniques generally enable non-edge nodes to efficiently aggregate information from both nearby and distant nodes, limitations arise when considering edge nodes. These edge nodes possess only a singular connection and, as a result, can solely aggregate information from nodes in close proximity to the central point. This asymmetry in information flow becomes particularly evident when comparing edge nodes to central nodes, as the former receives comparatively less neighbor information, thereby impeding the formation of optimal node representations. To address this challenge and enhance the representation of both symmetric and edge nodes, we propose a novel approach named spatial-structural graph convolution. 

This innovative method involves the integration of two branches: a spatial branch responsible for aggregating information from spatially adjacent nodes and a structural branch focused on consolidating information between edge nodes. By combining these branches, the spatial-structural graph convolution aims to achieve a more comprehensive and accurate representation of the underlying skeletal structure, thereby advancing the state-of-the-art in joint connectivity modeling <ref>.
The structural graph convolution is formulated as follows:

    f_outStructural = ∑_jM_jf_inAs_j

where f_in and f_outStructural represent the input and output features, respectively. As_j represents the structural adjacency matrix for distance j. The parameters M_j can be optimized during the training process. In this work, we set the distance as 1.  Then, we use element-wise addition to combine the outputs of these two branches and obtain the final output. 

    f_out = f_outSpatial+f_outStructural






 §.§ Structural Adjacency Matrix


In contrast to the conventional approach of spatial graph convolution, where all samples share a uniform adjacency matrix, we recognize that the structural connection between edge nodes is intricately linked with the specific actions performed. To capture these nuanced relationships, we propose a data-driven structural adjacency matrix, wherein the strength of connections between edge nodes is determined solely by the characteristics of each sample.

In the context of the NTU60 dataset's skeleton data, we observe five edge nodes, each corresponding to a distinct time series. In standard graph convolutional networks (GCN), the adjacency matrix is normalized based on the number of adjacent nodes. However, for structural connections, creating a conventional adjacency matrix between edge nodes is impractical as edge joints lack tangible spatial connections. Aggregating them without differentiation would compromise the unique characteristics of each edge node. Based on this idea, we use the similarity of edge node sequences instead of the number of adjacent nodes to represent the strength of the connection.

Euclidean distance and cosine similarity are widely employed distance calculation methods in deep learning. Euclidean distance quantifies the straight-line separation between two points in Euclidean space, making it well-suited for sequences with consistent lengths and where element order is crucial. Cosine similarity gauges the cosine of the angle between two vectors, treating sequences as vectors in a high-dimensional space. Cosine similarity is suitable for situations involving text data, document comparison, and situations where vector magnitudes are not critical.

In the context of skeleton data, the interplay of human muscles and bones results in a strong correlation among joint movements. However, the execution of force progresses incrementally. Consider playing badminton as an example: the upper arm propels the forearm, the forearm propels the wrist, leading to the final stroke of the shuttlecock. The transmission of the movement unfolds from the core to the periphery, with movement speed and amplitude gradually accelerating. Despite the collective coordination of the entire arm for a given motion, the time and range of movement for different joint points vary. In such cases, neither Euclidean distance nor cosine similarity proves suitable.

Taking inspiration from similarity calculation methods in natural language processing, we turn to the utilization of Dynamic Time Warping (DTW) to compute distances between each pair of edge nodes. DTW is a technique adept at measuring the similarity between two sequences that may exhibit variations in time or speed. It excels in scenarios where sequences share similar patterns but may be temporally misaligned. The DTW algorithm constructs an optimal warping path between the sequences, assessing similarity between corresponding points and allowing for time axis stretching or compressing as needed. The primary objective is to minimize the overall cost of warping while aligning similar features. Notably, DTW accommodates discrepancies in movement time among different joint points and considers the magnitude of movement, making it highly suitable for assessing movement similarity between edge nodes.

We finally use FastDTW for efficiency. We calculate a distance matrix D for each sample. To discern the correlation between different edge nodes, we leverage the inverse of the distance matrix D^-1 as a representation of the correlation matrix. Larger values in D^-1 signify greater similarity in actions between the two edge nodes, indicating a stronger correlation.

Addressing the over-smoothing issue inherent in GCN, where data from different joints tends to become overly similar with each aggregation step, structural GCN takes a different approach. When employing the D-1 approach directly, heightened similarity among adjacent nodes leads to an aggregation of information, thereby exacerbating the likeness of edge nodes, a condition contrary to the desired outcome. Rather than engaging in information aggregation from neighboring nodes, this method prioritizes the preservation of node-specific information by delineating each edge node from others through the utilization of -D-1. Consequently, the application of -D-1 fosters greater differentiation among edge nodes, thereby facilitating the capture of nuanced information.
To achieve this, an identity matrix (I) is introduced to retain the distinct characteristics of each node. The dynamic structural adjacency matrix, denoted as I-D^-1, is then calculated for each sample, providing a nuanced representation of the structural connections within the data.


For a visual representation, we refer to the algorithm in <ref>, illustrating the steps involved in the dynamic structural adjacency matrix calculation. Additionally, Figure <ref> provides visualizations of the spatial-structural connections for three distinct samples—throw, writing, and stand up—underscoring the effectiveness of our proposed approach in capturing the intricacies of fine-grained activity recognition. Figure <ref> provides visualizations of matrix D^-1. Figure <ref> provides visualizations of learned spatial and structural adjacency matrix.














 §.§ Data Preprocessing

In this work, we adopt the same data preprocessing as described in <cit.>. The shape of each action sequence x is (C_in,T_in,V_in), where C_in denotes coordinates , T_in denotes frames, and V_in denotes joints. We extract three input features from the input data: joint features, velocity features, and bone features. 

For joint features, we first calculate the relative joint positions j_r of all joints i (i∈{1,2,...,V_in}) to the center joint c, then concatenate the original joint positions with the relative joint positions. For the NTU RGB+D Dataset and NTU RGB+D 120 Dataset, we selected joint 2 (middle of the spine) as the central joint. Then we concatenate the original joint positions x  with the relative joint positions j_r as joint input features.

    j_r = x[:,:,i]-x[:,:,c]


    j = x ⊕ jr

For velocity features, we calculate slow motion v_s for time t (t∈{1,2,...,V_in}) using one frame as the time interval, and fast motion v_f for time t (t∈{1,2,...,V_in})  using a two-frame time interval.  Then we concatenate the slow motion v_s  with the fast motion v_f as velocity input features.

    v_s = x[:,t+1,:]-x[:,t,:]


    v_f = x[:,t+2,:]-x[:,t,:]


    v = v_s ⊕ v_f

For bone features, we first calculate the lengths of the bones b_l, which consist by joint i (i∈{1,2,...,V_in}) and its adjacent joint i_adj (i_adj∈{1,2,...,V_in}) . Then we calculate bones' angles b_a,w for three coordinates w (w∈{x,y,z}). We concatenate the lengths of the bones b_l with the bone angles b_a,w as bone input features.

    b_l = x[:,:,i]-x[:,:,i_adj]


    b_a,w = arccos(b_l,w/√(b_l,x^2+b_l,y^2+b_l,z^2))


    b = b_l ⊕ b_a





 §.§ Model


For the overall efficiency of the network structure, we utilize the GCN block introduced by <cit.> as the fundamental unit <ref>. However, instead of the conventional GCN layer, we have integrated our novel spatial-structural graph convolution layer. The network architecture begins with an initial block designed to extract features at the outset, followed by the stacking of four GCN blocks to progressively extract higher-dimensional features. Finally, a Classifier block, which incorporates a Global Average Pooling (GAP) layer, a Dropout layer, and a Fully Connected (FC) layer, is employed to generate the output for each modality, as illustrated in <ref>. We then combine the results from three inputs (joint features, velocity features, bone features) to get the final output, where leveraging multiple modalities to enhance the model's capacity for nuanced feature extraction and robust predictive performance.








§ EVALUATION




 §.§ Dataset




  §.§.§ NTU RGB+D Dataset

"NTU RGB+D" dataset<cit.> contains 60 action classes and 56,880 video samples captured from 40 different people using the Microsoft Kinect v2. The 60 action classes can be categorized into three main groups: 40 daily actions, 9 health-related actions, and 11 mutual actions. As mentioned, the NTU RGB+D dataset is a multi-modal dataset that contains RGB videos, depth map sequences, 3D skeletal data, and infrared (IR) videos for each sample. In this work, we only utilize skeleton data. Each frame contains two skeletons, and each skeleton contains 25 joints. 

We evaluated our method using two benchmarks proposed by the authors of this dataset: cross-subject validation (X-sub) and cross-view validation (X-view). In cross-subject validation, we divided the subjects into two groups: twenty subjects in the training data group and another twenty subjects in the validation group. In cross-view validation, the data collected by cameras 2 and 3 are recognized as training data, while the data collected by camera 1 is used for validation. 



  §.§.§ NTU RGB+D 120 Dataset

The "NTU RGB+D 120" dataset<cit.> is an extension of the "NTU RGB+D" dataset, captured from 106 different individuals, with an additional 60 classes and 57,600 video samples. The 120 action classes can be categorized into three main groups: 82 daily actions, 12 health-related actions, and 26 mutual actions. We evaluate our method using two benchmarks proposed by the authors of this dataset: cross-subject validation (X-sub) and cross-setup (X-set120). In cross-subject validation, 53 subjects are divided into a training data group and another 53 subjects into a validation group. In cross-validation, 16 setups are used for training, and the remaining 16 setups are used for testing. 




 §.§ Experimental Setting

In our experiments, we set the training epoch to 50. We apply the same settings as described in<cit.>. Our models are trained with gradient descent (SGD) with a Nesterov momentum of 0.9 and a weight decay of 0.0001. The learning rate is set to 0.1 and decays with a cosine schedule after the 10th epoch. The probability in the dropout layer is set to 0.25. We perform a data transformation as described in <cit.> for the X-view benchmark. The batch size is set to 16 for both the NTU RGB+D dataset and the NTU RGB+D 120 dataset. All our experiments are performed on one NVIDIA GeForce RTX 3090 GPU.




 §.§ Comparisons

 



  §.§.§ Effect of Structural Graph Convolution

 In this part, we will discuss the impact of three input modalities and the impact of our structural branch on X-sub benchmark <ref> and on X-view benchmark <ref>. We can see that all three modalities are essential, and that fusing the scores of the three modalities outputs in the highest accuracy for both X-sub and X-view benchmarks. In addition, using the same network structure, there is a significant improvement in accuracy when employing the spatial-structural graph convolution layer instead of the spatial graph convolution layer.







  §.§.§ NTU RGB+D Dataset

We compare our models with the state-of-the-art methods on the NTU RGB+D dataset in <ref>. We classify the commonly used methods for skeleton-based human activity recognition into two categories: CNN-RNN-LSTM-based methods and GCN-based methods. Notably, the GCN-based method exhibits superior performance compared to CNN-RNN-LSTM-based methods. In recent years, there has been a growing preference among researchers for GCN-based approaches in skeleton recognition tasks, showcasing better overall performance than their CNN-RNN-LSTM counterparts. 
ST-GCN is one of the most well-known GCN-based models for skeleton recognition. Our method greatly exceeds it by 10.1% on the X-sub benchmark and 7.5% on the X-view benchmark. EfficientGCN is another popular model for its high efficiency and high accuracy. In this work, we use the same GCN block and the same number of blocks as EfficientGCN-B0. The accuracy of our model exceeds it by 1.4% on the X-sub bencnmark and 0.9% on the X-view benchmark. CTR-GCN learns topologies dynamically, which demonstrates commendable outcomes. However, it overlooks critical considerations such as the symmetry inherent in human body structure and the constraints associated with edge nodes. We posit that integrating the adaptable topology of CTR-GCN with our refined graph representation will likely yield superior results. We consider that the results of StSp-GCN exceed those of most models due to our precise expression of the skeleton data and the flexibility of graph convolution by introducing unique structural connections for each sample.






  §.§.§ NTU RGB+D 120 Dataset


Compared with the NTU RGB+D data set, the NTU RGB+D 120 dataset is larger and has more classes. This means that there are more similar actions that need to be distinguished, which poses a greater challenge to the model. The overall performance of the GCN-based model is still better than the CNN-RNN-LSTM-based method. Compared to ST-GCN, our method greatly exceeds it by 17.1% on the X-sub benchmark and 15.6% on the X-view benchmark. Our method outperforms EfficientGCN-B0 by 1.2% on the X-sub benchmark and 3.8% on the X-view benchmark. In comparison to CTR-GCN, our introduced structural connections augment the delineation of topology across diverse samples. It is our conjecture that amalgamating our structural links with the inherently flexible topology of CTR will yield more favorable outcomes. Similar to the results on the NTU RGB+D dataset, the performance of SpSt-GCN is mainly attributed to the precise expression of the skeleton data and the flexibility of graph convolution.





  §.§.§ Model Complexity


Considering the importance of efficient resource utilization, we not only assess model accuracy but also consider computational complexity. We compare the model complexity (FLOPs and the number of parameters) with state of art methods on the X-sub benchmark in <ref>. Compared with most models, our model has a higher accuracy while having low computational complexity and parameter quantity. Specifically, our model's FLOPS and parameter count are only 30.02% and 15.48% of ST-GCN, respectively. The computational complexity and parameters of our method have increased relative to EffciientGCN-B0, mainly due to adjustments in model parameters. Compared with Sp-GCN, the FLOPs the number of parameters of SpSt-GCN have only increased by 15.3% and 9.1% respectively, which indicates that the structural branch is very efficient.








§ CONCLUSION

In this article, we propose a Spatial-Structural Graph Convolutional Network (SpSt-GCN) for skeleton-based action recognition to better represent symmetrical human structures and edge nodes and address the issue of over-smoothing in Graph Convolutional Networks. The spatial branch aggregates information based on the topological structure, while the data-driven structural branch performs differentiation based on the similarity of edge node sequences. Our approach yields favorable results while maintaining efficiency. In the future, we will explore how to improve the flexibility of structural connection and how to extend this method to other graph recognition tasks, e.g. In this article, we have only considered the structural connectivity of edge nodes. The representation of structural connectivity between non-edge nodes is also worth exploring.

plain




 Paper Title: Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal Language Model.



Quantized electrical, thermal, and spin transports of non-Hermitian clean and dirty two-dimensional topological insulators and superconductors
    Bitan Roy
    August 5, 2024
==============================================================================================================================================

§ ABSTRACT 
Multimodal language models (MLLMs) are increasingly being implemented in real-world environments, necessitating their ability to interpret 3D spaces and comprehend temporal dynamics. Despite their potential, current top models within our community still fall short in adequately understanding spatial and temporal dimensions.
We introduce , a simple, training-free, effective, and general-purpose visual prompting method to elicit 3D and temporal understanding in multimodal LLMs.
Our method uses a lightweight tracking model to find object correspondences between frames in a video or between sets of image viewpoints. It selects the most frequent object instances and visualizes them with markers with unique IDs in the image.
With this simple approach, we achieve state-of-the-art results on 3D understanding benchmarks including ScanQA (+20.5%) and a subset of OpenEQA (+9.7%), and on long-form video benchmarks such as EgoSchema (+6.0%). We also curate a small diagnostic dataset to evaluate whether MLLMs can reason about space from a described viewpoint other than the camera viewpoint. Again, improves spatial perspective-taking abilities but we highlight that MLLMs struggle with this task.
Together, we demonstrate that our simple prompting method can significantly aid downstream tasks that require 3D or temporal reasoning.





§ INTRODUCTION



Multimodal LLMs (MLLMs) today are part of larger applications in the physical world through devices like smartphones, smart glasses, and robots <cit.>. Augmented with vision encoders <cit.>, proprietary models like GPT-4o <cit.> and Gemini-Pro <cit.> have recently been demoed to understand users' physical environment as well as reason about their actions over time.
Despite their promise, research benchmarks have continued to highlight that our community's best models still lack sufficient 3D understanding of space and time <cit.>.
Popular 3D (e.g. ScanQA <cit.> and OpenEQA <cit.>) and video (e.g. EgoSchema <cit.>) benchmarks report that MLLMs struggle when asked to reason about spatial relationships between objects (“Is the couch to the right of the door”?) or remember temporal events in long videos (“How does [the person] ensure the completion of their primary goal in the video?”)

Many recent efforts have begun incorporating 3D or temporal information to improve reasoning.
For instance, new methods finetune MLLMs with specialized structures or representations <cit.>. 
Unfortunately, these methods require an open-sourced model with weights that can be modified.
In parallel, visual prompting mechanisms (e.g. set-of-marks <cit.> and 3DAxiesPrompts <cit.>) augment image inputs with visualizations to guide the MLLM's attention to specific 2D regions in an image. Visual prompting methods so far show improvements for 2D understanding but not yet for 3D and temporal reasoning.
Worse, these efforts have focused either on 3D understanding or temporal understanding—not both. 
Even if visual prompting methods could add useful information, encoding that information  meaningfully in a video for an MLLM is difficult:
Proprietary MLLMs are developed primarily using image inputs and not videos; when presented with a video, they sparsify and preprocess videos before sending it to the MLLM.



To address the above challenges, We introduce , a propose a simple, effective, and general-purpose visual prompting method to evoke 3D and temporal understanding in multimodal LLMs. Given a video or a set of images of a space, transforms the images/frames to help MLLMs reason.
First, we use a lightweight tracking model <cit.> to extract object correspondences for the same instance across multiple frames at the video's original high frame rate.
Next, we sparsify the videos to reduce the number of frames we pass to the MLLM. 
Finally, we visualize the top-K objects that appear in the most number of frames, sorted by the number of pixels they occupy. We use marks and segmentation outlines with unique IDs to highlight these objects (Figure <ref>).
Notice that these images are modified independent of any underlying questions or tasks.
Finally, we pass these modified images, along with the question to an MLLM.
In our experiments, we apply our visual prompting method to GPT-4V, GPT-4O, and other MLLMs.

Our framework consistently achieves significant improvements over state-of-the-art results on ScanQA <cit.> and OpenEQA <cit.>, two 3D understanding benchmarks, and EgoSchema <cit.>, a long video understanding benchmark. On ScanQA, our framework allows GPT-4V <cit.> to surpass specially designed and fine-tuned methods <cit.> in a zero-shot manner with fewer views. 
Additionally, the rapid advancement of MLLMs provides the potential for our method to elicit other emergent understanding abilities. For instance, with GPT-4O, our method enables 3D spatial understanding from extremely sparse views, reducing the computational cost and expense associated with MLLMs.

To further test for 3D spatial reasoning, we create a small diagnostic benchmark called . 
Current benchmarks assess a model's 3D spatial awareness from the viewpoint of the camera-holding observer. In cognitive science, the spatial orientation test (SOT) is commonly used to evaluate spatial intelligence in children <cit.>. It measures spatial perspective-taking, which is the capacity to envision how an object or scene looks from a different viewpoint <cit.>.
contains 10 scenes with videos from different perspectives. Using these scenes, we test whether MLLMs can reason about spatial relationships from different perspectives.
We show that improves the GPT-4 models' abilities. But overall, we find that their performance is at random chance.


In summary, we propose , a general visual prompting method that combines lightweight tracking models to induce 3D and temporal reasoning in any MLLMs. Our results across 4 3D, temporal, and our new spatial cognition benchmarks verify that our method works out-of-the-box. 
We also demonstrated that even the most advanced MLLMs have significant shortcomings in spatial understanding that are not easily overcome. We hope this work can help MLLMs better understand the physical world we live in.


§ RELATED WORK



Multimodal language models
Multimodal LLMs<cit.> integrate vision encoders <cit.> into large LLMs <cit.>, allowing them to directly reason over visual input. Many proprietary models, such as GPT-4 <cit.>, Gemini <cit.>, and Claude <cit.>, as well as open-source models like the LLaVA series <cit.> and BLIP series <cit.>, have made significant progress in 2D vision-language tasks like image captioning <cit.> and visual question answering (VQA) <cit.>. Beyond these language-related tasks, many newer attempts applying MLLMs to applications such as autonomous driving <cit.> and robotics <cit.>. Many of these tasks require understanding the 3D space in which they are deployed and reason about how things are changing temporally. We improve the 3D space-time capabilities of such models.

Visual prompting.
Effective prompting has been widely proven to improve LLMs across multiple domains. Methods, such as chain-of-thought prompting <cit.>, force the model to reason before answering a question.
For multimodal LLMs, methods such as Red-circle prompting <cit.> and Set-of-marks <cit.> can enhance the grounding abilities of CLIP <cit.> and GPT-4V.
PIVOT <cit.> employs visual prompting combined with iterative VQA to induce GPT-4V to generate outputs for robotics control. 3DAxies <cit.> enhances GPT-4V's ability to use numerical expressions to describe 3D relationships of objects in a single image by annotating a scaled 3D coordinate system on the image.
Unlike these works, prompts MLLMs to understand the spatial relationships within a complete 3D scene from an image sequence. 

Video understanding.
Videos carry rich information about both the 3D structure as well as temporal changes in the physical world. To perform better long-horizon reasoning, work has begun incorporating video inputs into MLLMs. 

By replacing the image encoder in MLLMs with a video encoder <cit.>, recent work has improved performance on video dense captioning <cit.> and videoQA <cit.>.
To further advance the understanding of temporal relationships in videos, EgoSchema <cit.> introduced a benchmark for long video understanding, which is more challenging than previous video-language benchmarks.
Meanwhile, understanding 3D spatial relationships in videos received relatively less attention. 3D-LLM <cit.> converts multiview images into 3D point clouds and then feeds them into LLMs, demonstrating better results on the ScanQA <cit.> benchmark for 3D understanding. OpenEQA <cit.> is also a benchmark dedicated to evaluating MLLM's understanding of 3D physical space, with outputs that are more open-vocabulary compared to ScanQA.
In this paper, we propose a framework that does not require any training in modifying MLLMs; it extracts meaningful information from videos using off-the-shelf tracking models and achieves state-of-the-art performance on the benchmarks mentioned. 

Visual correspondences.
Visual correspondences have been a vital area of research in computer vision for a few decades. Applications such as Structure-from-Motion<cit.> utilize correspondences to better reconstruct 3D scenes. In the past, we relied on handcrafted features like SIFT <cit.> or SURF <cit.> to obtain good correspondence. Today, features extracted from deep models <cit.> can also provide increasingly accurate correspondences. Generally, people aim to achieve precise geometric and semantic correspondences at the pixel level. However, in this paper, we use coarse visual correspondence to prompt MLLMs, which can be easily obtained from off-the-shelf video tracking models <cit.>.





§ METHOD

We introduce , a visual prompting method that allows MLLMs to reason about 3D space and time. 

Problem formulation. Given a question 𝒬 and a sequence or set of observations in an environment [I_1, …, I_n], 
our aim is to design a visual prompt 𝒫(…) that modifies the input image set. These image inputs don't have to be a video. They can also represent a set of images of a scene from multiple viewpoints.
We evaluate the prompt by measuring its utility in prompting an MLLM ℳ:

    [I_1^', …, I_n^'] = P([I_1, …, I_n])
       𝒜̂ = ℳ(([I_1^', …, I_n^']), 𝒬)

We compare the generated answer 𝒜̂ with the ground truth 𝒜.


In our framework, the MLLMs can be any general-purpose model without requiring any special architecture or training for 3D or temporal understanding. Our aim is to develop a prompting strategy that allows models to improve such capabilities without any training (Figure <ref>).




 §.§ Coarse Correspondence

Our prompting method, , contains four steps: (1) tracking correspondences, (2) sparsify frames, (3) selecting, and (4) visualizing coarse correspondences. 

(1) Tracking correspondences. Given n input images, [I_1, …, I_n], we first use an off-the-shelf video object tracking model, such as Tracking Anything <cit.>. This model extracts class-agnostic instance segmentation masks (M_1, …, M_n ) for each image. Each M_i is a H × W dimensional matrix where H and W are the height and width of the input image I_i. Each pixel location in M_i contains an instance ID, indicating which instance the pixel at that position belongs to within the image sequence. 


(2) Sparsify frames.
Since most MLLMs contain a large number of parameters, directly using them to process long image sequences is very computationally intensive. Additionally, proprietary MLLMs like GPT-4O can also incur significant costs if the number of image tokens that need to be processed increases. Reducing the number of input images might lose vital information necessary for MLLMs. 

strikes a balance in this tradeoff by extracting meaningful video object tracks (a relatively cheaper operation) from high-frame-rate image sequences, and then samples a few image inputs along with the tracks, to retain—and even improve—performance while reducing the MLLM's computation cost.
From this extracted video object tracks, we perform temporal downsampling, retaining only m << n uniformly sampled images and their corresponding masks, denoted as [I_s_1, …, I_s_m] and [M_s_1, …, M_s_m], where s_i ∈{1, …, n}.
This downsampling reduces the number of images we feed into ℳ. 


(3) Selecting coarse correspondences.
Prompting an MLLM with all the detected correspondences results in information overload. In fact, our ablations (discussed in Sec <ref>) find that adding all the correspondences reduces the MLLM's performance.
Therefore, we select a subset of prominent instances to retain. We select the prominent instances of the top-K objects that co-occur in the most number of frames.
We first calculate the occurrence frequency and area sum of each unique instance ID in the retained m masks using the following equation:

    ℱreq(ID)    = ∑_i=s_1^s_m1_{ID∈ M_i}, 
    𝒜rea(ID)    = ∑_i=s_1^s_m∑_p ∈ M_i1_{ID = p}.

Then, we first sort all instance IDs in descending order based on ℱreq(ID). If there are ties, we further sort based on 𝒜rea(ID). Finally, we retain the top k instance IDs as tracklets, denoted as [T_1, …, T_k], to visualize for MLLMs.

(4) Visualizing coarse correspondences.
For each set of obtained correspondence relationships, we visualize the correspondences directly in the image as a marker. Specifically, for each identified primary instance ID T_i, if it exists in the mask M_s_j of a retained image I_s_j, we overlay a mark with a fixed size and shape labeled with T_i at the position (x̅_ij, y̅_ij) on I_s_j to produce I_s_j^'. The specific placement position can be easily obtained by:

    (x̅_ij, y̅_ij) = ∑_(x,y) (x, y) ·1_{M_s_j(x,y) = T_i}/∑_(x,y)1_{M_s_j(x,y) = T_i}


Naturally, we can overlay not just the markers but also the segmentation outlines or even the segmentation masks associated with each retained prominent instance. We explore these ablations later. In the end, we obtain the prompted image sequence [I_1^', …, I_m^'], which is then used as the input to MLLMs. 

We refer to our method as Coarse because of the following: first, we only visually prompt for instance-level correspondences and not point-level correspondences. 
Second, the instance-level correspondences are extracted using off-the-shelf tracking models. Despite not being perfectly precise, they still help MLLMs build a better 3D model of the environment. Third, we only visualize a handful of prominent corresponding instances.











§ EXPERIMENTS

We evaluate the utility of across a number of tasks that require understanding 3D space (ScanQA <cit.> in <ref> and OpenEQA <cit.>) in <ref>) as well as temporal events (EgoSchema <cit.> in <ref>). 
Across all three benchmarks, we augment GPT-4V and GPT-4O with and evaluate its zero-shot performance.
We show that not only significantly improves the base GPT models but that the improvements establish new state-of-the-art results across all our evaluations. All experiments are conducted with A100 80G GPUs.


Existing benchmarks evaluate the 3D spatial awareness of a model from the perspective of the camera-wielding observer. To further demonstrate the utility of , we introduce a new benchmark: . Our benchmark tests whether models are able to reason about 3D space from the perspective of an imaginary observer at another location in their field of view.
Again, we show that significantly improves GPT-4{V,O}'s abilities.



 §.§ Spatial understanding with ScanQA







The ScanQA dataset <cit.>.
The ScanQA dataset consists of 41363 questions about 800 scenes, including 32337 unique questions. Its validation set contains 4675 questions about 71 scenes. Because the dataset is derived from the original ScanNet dataset <cit.>, it also contains 3D localization of all objects in the scene. 
Questions in ScanQA require basic recognition, 3D localization, and 3D embodied capabilities <cit.>. 
The validation set contains two ground-truth answers per question for evaluation with models that produce free-form answers.

Baselines. We evaluate by augmenting both GPT-4{V,O}, Gemini and Claude models. We also compare against other general-purpose multimodal LLMs such as LLaVA <cit.>, flamingo <cit.>, and BLIP2 <cit.>. We also consider 3D-LLM which was trained with 3D awareness <cit.>. 

Metrics. Following prior works, we adopt  BLEU <cit.> scores, METEOR <cit.>, ROUHE-L <cit.> and CIDEr<cit.> as our evaluation metrics

Results.
As shown in <ref>, compared with raw input to GPT-4V and GPT-4O, constantly improves the recently released GPT-4o's overall performance by a significant 5.7 BLEU-2, 3.2 METEOR, 6.5 ROUHE-L, and 15 CIDEr points. 

Compared with other vision-language models, it is clear that helps improve 3D understanding for different GPTs constantly; GPT-4V and GPT-4O both fail to achieve competitive performance without but surpass the state-of-the-art models with it. 






 §.§ Spatial understanding with OpenEQA



The OpenEQA dataset <cit.>.
OpenEQA is an open-vocabulary dataset benchmarking spatial environment understanding and embodied reasoning. We evaluate on OpenEQA's EM-EQA data split, which contains over 1600 high-quality human-generated questions collected from various real-world environments.
The subset tests the episodic memory of an agent moving through a 3D environment over time.
This task uses video inputs, from which we sample only K frames at equal intervals as input to the GPT models. We use K=8 frames for GPT-4V and only K=4 for GPT-4O, which we find to be the best performance empirically for the GPT models without . We adopt accuracy as our metric.

Baselines. We compare against language-only models to account for language bias (LLaMA2 <cit.>), commonly used general-purpose multimodal LLMs (GPT-4 <cit.>, Claude3 <cit.>, Gemini-Pro <cit.>, GPT-4V with 15 and 50 frames.

Results.
again achieves state-of-the-art performance while utilizing significantly fewer frames than all previous works (Table <ref>). This substantial reduction in the number of views highlights the efficiency of our approach during inference time. 
The improved performance, coupled with reduced computational overhead, charts a potential future for using for embodied AI tasks.
Despite our marks being generated unconditioned on the questions, they still improve performance.


  
  [t]0.48
    
      Models     Frame     Accuracy 

      LLaMA2 <cit.>     0     28.3 

      GPT-4 <cit.>     0     33.5 

      Claude3 <cit.>     20     36.3 

      Gemini-Pro <cit.>     15     44.9 

      GPT-4V <cit.>     15     54.6 

      GPT-4V <cit.>     50     55.3 

      Human     Full     86.8 

      GPT-4V     8     44.8 

      gray!20 GPT-4V+CC     8     58.5 

      GPT-4O     4     49.4 

      gray!20 GPT-4O+CC     4     59.1 

      
    tableComparisons on EM-EQA setting of OpenEQA . Our method further enhances the embodied ability of MLLMs and exceeds previous methods by a large margin.
    
  
  [t]0.48
    
      Models     Frame     Subset 

      LongViviT <cit.>     256     56.8 

      MC-ViT-L <cit.>     128+     62.6 

      LLoVi <cit.>     180     58.3 

      VideoAgent <cit.>     8.4     60.2 

      MVU <cit.>     16     60.3 

      VideoAgent <cit.>     -     62.8 

      LangRepo <cit.>     -     66.2 

      GPT-4V     8     64.2 

      gray!20 GPT-4V+CC     8     67.4 

      GPT-4O     8     67.2 

      gray!20 GPT-4O+CC     8     73.2 

      
    tableComparisons on EgoSchema validation set. improves existing MLLMs and surpasses previous finetuned models in a zero-shot manner.
    
  

  


 §.§ Temporal understanding



The EgoSchema dataset <cit.>. EgoSchema aims at benchmarking a model's long video understanding ability. Due to budget constraints, we limit this evaluation to 500 questions from the validation set. For the GPT models' input, we sample K=8 image frames uniformly from the video input.

Results. 
demonstrates state-of-the-art performance, significantly outperforms existing approaches in a zero-shot manner (Table <ref>) with fewer frames. The reduction in frame usage underscores 's ability to extract relevant temporal information more effectively, suggesting the possibility of using it for other video tasks in the future.




 §.§ The benchmark for Spatial Orientation Test





In cognitive science, the spatial orientation test (SOT) is a widely adopted examination for spatial intelligence in children <cit.>.
The SOT assesses spatial perspective-taking—the ability to imagine how an object or scene would appear from a perspective different from the current camera viewpoint. Numerous human studies <cit.> have shown that this ability is closely related to the development of spatial intelligence in children. In this section,
We adopt this test to evaluate multimodal LLMs.


Data curation. 
We manually curated ten real-world scenes, both indoor and outdoor, using different mobile devices at various viewpoints. We instructed 10 human participants to take two videos in their environment from two viewpoints. When in each viewpoint, they were asked to remain in place as they laterally pan their mobile devices to scan their 3D environment. 
From 20 collected scenes, we filtered to and retained 10 scenes that satisfied the following four criteria:
First, we could uniquely describe one viewpoint from the perspective of the other and vice-versa. For example, in Figure <ref>, we define the other viewpoints as 'a person stepping out of an elevator.'
Second, we ensured that no single frame captured the entire 3D space, ensuring that models can not short-cut answers using any single view.
Third, all scans move the camera from left to right.
Fourth, to avoid privacy concerns, we ensured that no people appeared in the videos.
Each scene scan lasts between 3 to 5 seconds. 

For each scene, we designed five carefully crafted questions, each asking the model to determine if one object is to the left or to the right of another from a specific viewpoint. The first three questions are from the observer's (camera's) perspective, while the final two describe the perspective in language, thereby, testing for a model's spatial perspective-taking ability. 

Human performance on these questions is 100%. 
We design questions to have a bias towards asking about relationships between objects that appear in the first last frame of the scan, ensuring that the has to use multiple frames to answer.
In total, across the 10 scenes, has a modest 50 questions.



Results. 

significantly improves GPT-4O in perspective-taking ability by a significant 21.2% (Table <ref>). Notably, when using only the first and last frames, as illustrated in Figure <ref>, our method enables GPT-4O to understand the 3D spatial structure represented by the images using minimal overlap, whereas GPT-4O alone performs only slightly better than random guessing.

We further investigate two striking findings. 
First, the GPT models exhibit a bias where they perform better when the camera moves from left to right. But if the order flipped, they perform worse. We report both the L->R and R->L camera pans and also report their harmonic mean. 
Second, when we isolate the performance on the 2 perspective-taking questions per scene in Figure <ref>. While improves GPT-4O's perspective-taking capability, these results are bittersweet since they still perform worse than random guessing.





§ ANALYSIS




Here, we explore the various design decisions implicit in our method.


How does differ from other visual prompting methods? Our proposed method calculates and highlights correspondences between images, aiming to elicit 3D and temporal understanding.
Other visual prompting methods (namely Set-of-Mark, 3DAxiesPrompts, and Chain-of-thought <cit.>) can also be viewed as alternative prompting methods. We compare against them on ScanQA qualitatively in <Ref>.

The orange part of <Ref> shows our Coarse Correspondence labels are recognized by GPT-4V. 
The output answer provides evidence that our coarse correspondence helps GPT-4V develop a mental 3D model of the scene.
Set-of-Marks <cit.> provides no spatial corresponding information and therefore is unhelpful.
The Axis labels in 3DAxies<cit.> can be easily misrecognized by GPT-4V, leading to misleading spatial information.
Though Chain-of-Thought <cit.> helps
identify objects, it fails to resolve the “spatial perspective-taking” issue. 




Why use coarse instead of dense correspondences?

Instead of filtering and retaining only a handful of coarse correspondences, one ablation we considered is the possibility of using all dense correspondence. Unfortunately, we find that excessively overlaying too many instance marks can degrade performance (Table <ref>) as they occlude the visual content in the images.

How large should the marks be?
We inject the correspondences into MLLMs by overlaying the marks into images. 
We empirically find an optimal mark size (where `px' represents the mark's diameter in pixels) in Table <ref>. Marks that are too small tend to be ignored while those that are too large occlude visual content.

What shape should the marks be?
We further studied the appearance of the marks. In addition to red circles with white text, we experimented with adding segmentation outlines and segmentation masks. As shown in Table <ref>, using segmentation outlines enhances object grounding. However, using segmentation marks occludes visual content and reduces performance.







§ DISCUSSION


Limitations.
We would like to point out two limitations of our method. First, our method relies on off-the-shelf video tracking models to obtain instance-level correspondences. 
Although the performance of tracking models has significantly improved with the advent of tools like SAM <cit.>, achieving good results on long-form in-the-wild videos remains challenging. 
This is particularly evident on the 180-second EgoSchema benchmark, where Track-Anything often loses track of objects after 100 seconds, leading to inconsistent instance segmentation masks between the beginning and end of the video clip. 
Despite observing consistent and significant improvements on EgoSchema, we believe that accurate correspondence would further enhance the benefits of our approach.
Second, we find that our method does not yet improve the 3D understanding of open-source models like LLaVA <cit.>. We speculate that there are two reasons for this: either LLaVA may not recognize the marks we overlay on the images or LLaVA may not effectively handle multiple image sequences appropriately. We find that LLaVA often focuses on the content in the last input image. There is a need to develop open-source models with visual promptable capabilities and with multi-image inputs.


Conclusion.
We propose a framework called prompting. By using off-the-shelf video tracking models at a high frame rate to obtain class-agnostic instance-level correspondences, we enable better recognition and reasoning with very few frames. 
Our method is easy to use, doesn't require any training, and performs zero-shot with strong models like GPT-4.
enables GPT to reason about the 3D structure from very sparse views.
We demonstrated consistent and significant improvements across multiple spatial and temporal understanding benchmarks.
We also identified that even GPT struggles with perspective-taking capability, a fundamental component of human visual intelligence.
Regardless, 's results highlight its potential for robotics and embodied AI, which require 3D or temporal understanding.











§ ACKNOWLEDGEMENT

We thank Xiaojuan Wang, Luming Tang, Ruitao Zhang, Yuntian Deng for helpful discussions, feedback and collecting data. This project is partially funded by Amazon Science. 

unsrt








§ APPENDIX



§ BROADER IMPACT

Our method aims at improving the trustworthiness and reliability of deployment of MLLMs in real world application, including but not limited to Vision Pro, autonomous driving, and also humanoid robots. To have a virtual assistant like JARVIS in Marvel films, it's necassry to align the understanding of vision-language model with human's understanding, so that we can ensure safe application of these applications. 
Further, we are committed to reducing the carbon emissions produced by these models. By employing our coarse correspondence prompting method, we use a much smaller tracking module to reduce the number of input used as input to large GPT model. Besides, we also improve the speed and lower the cost of calling OpenAI API to understand a 3d scene. This enables democratize MLLMs so that more people and small companies can create their own real-world applications based on GPT-4V. We hope our work can make large AI models more effectively used for social good.

Still, we would like to point out that with the development of MLLMs, increased reliance on advanced MLLMs could also lead to a reduction in human skills, especially in interpreting and interacting with visual content. Over-dependence on these models might erode critical thinking and analytical abilities in the long term.





§ MORE DISCUSSIONS

Relation to SlowFast
SlowFast <cit.> is a framework for video recognition that includes two parallel pathways: a Slow pathway that captures motion information at a high frame rate and a Fast pathway that captures semantic information at a low frame rate. The information from both pathways is fused through lateral connections for downstream video recognition tasks.
In a way, our coarse correspondence prompting can be seen as another form of SlowFast. However, unlike SlowFast, where the Slow and Fast pathways operate in parallel, our framework operates sequentially. First, it captures low-level, class-agnostic motion information at a high frame rate using a lightweight tracking model. Then, at a low frame rate, it performs recognition and reasoning requiring semantic understanding using larger MLLMs. The two stages are bridged through visual prompting.
Moreover, while SlowFast learns a representation of the input video for pure vision recognition tasks such as action classification and detection, our coarse correspondence framework aims to better understand the 3D spatial structure and temporal information contained in the input video to achieve spatiotemporal perception and reasoning simultaneously.

Eulerian vs Lagrangian If deep learning-based methods represent camera or object motion in videos from an Eulerian viewpoint—i.e., expressing how features at fixed locations evolve over time through a multi-dimensional tensor—then our framework adds a Lagrangian viewpoint to this representation. The Lagrangian viewpoint describes the trajectories of entities moving through space and time in the video. Previously, the Lagrangian viewpoint in video descriptions has been shown to better aid human action recognition <cit.>. Here, we demonstrate that it can more generally help 
MLLMs understand the 4D spatiotemporal context represented in videos.




§ QUALITATIVE CASE STUDY





To further demonstrate the effectiveness of our proposed Coarse Correspondence under sparse image input,
we defined two challenging tasks and one qualitative case study for each task.

The results of these case studies are shown in Fig. <ref>.
Detailed illustration of the results are provided in the figure captions.
The first case study is about the task of Duplicate Objects Counting,
where the model needs to count the number of objects in a 3D scene.
Only equipped with coarse correspondence can GPT-4V get a comprehensive understanding of the 3D scene, excludes the duplicate objects, and give the right answer.The second case study is about the task of Relative Location Modeling,
where the model needs to understand the relative location of objects in a 3D scene.
It is obvious that without the correspondence markers, GPT-4V fails to response from 3D perspective with only raw 2D images.These case studies demonstrate that our proposed Coarse Correspondence can elicit
MLLMs in understanding 3D scenes from sparse image inputs.







§ USER-FRIENDLY INTERACTIONS

We also prove that our Coarse Correspondence method works well with 
hand-crafted correspondence marks as shown in  <Ref>. 
This demonstrates that our method is highly user-friendly for utilizing proprietary multimodal language models, such as GPT-4O, in web interfaces. Users can easily complete prompts by marking correspondence relationships on images. Moreover, the marks can be diverse and flexible. This also proves the robustness of our method, as the marks are style-agnostic, as long as they convey the visual correspondence knowledge.












 Paper Title: Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion.





0000-0002-4238-7560

 ETH Zürich & DisneyResearch|Studios
 Zürich
 Switzerland
manuel.kansy@inf.ethz.ch

0000-0002-2315-5166

 DisneyResearch|Studios
 Zürich
 Switzerland

jacek.naruniec@disneyresearch.com

0000-0003-1473-1878

 DisneyResearch|Studios
 Zürich
 Switzerland

christopher.schroers@disneyresearch.com

0000-0002-6838-9775

 ETH Zürich & DisneyResearch|Studios
 Zürich
 Switzerland

christopher.schroers@disneyresearch.com

0000-0003-1196-5425

 DisneyResearch|Studios
 Zürich
 Switzerland

romann.weber@disneyresearch.com

§ ABSTRACT 

Recent years have seen a tremendous improvement in the quality of video generation and editing approaches. While several techniques focus on editing appearance, few address motion. Current approaches using text, trajectories, or bounding boxes are limited to simple motions, so we specify motions with a single motion reference video instead. We further propose to use a pre-trained image-to-video model rather than a text-to-video model. This approach allows us to preserve the exact appearance and position of a target object or scene and helps disentangle appearance from motion.

Our method, called motion-textual inversion, leverages our observation that image-to-video models extract appearance mainly from the (latent) image input, while the text/image embedding injected via cross-attention predominantly controls motion. We thus represent motion using text/image embedding tokens. By operating on an inflated motion-text embedding containing multiple text/image embedding tokens per frame, we achieve a high temporal motion granularity. Once optimized on the motion reference video, this embedding can be applied to various target images to generate videos with semantically similar motions.

Our approach does not require spatial alignment between the motion reference video and target image, generalizes across various domains, and can be applied to various tasks such as full-body and face reenactment, as well as controlling the motion of inanimate objects and the camera. We empirically demonstrate the effectiveness of our method in the semantic video motion transfer task, significantly outperforming existing methods in this context.









    
    < g r a p h i c s >

    1
    73.65mm
      [width=0.0875]7figures/teaser/body/frames_input/001014 
    
    73.65mm
      [width=0.0875, poster=3]7figures/teaser/body/frames/001014 
    
    62.55mm
      [width=0.0875, poster=3]7figures/teaser/body/frames_domain/001014 
    
    51.45mm
      [width=0.0875, poster=3]7figures/teaser/body/frames_multiple/001014 
    
    51.45mm
      [width=0.0875]7figures/teaser/body/frames_visualization/001014 
    
    35.6mm
      [width=0.0875]7figures/teaser/body_second/frames_input/001014 
    
    35.6mm
      [width=0.0875, poster=3]7figures/teaser/body_second/frames/001014 
    
    24.6mm
      [width=0.0875]7figures/teaser/face/frames_input/001014 
    
    24.6mm
      [width=0.0875, poster=6]7figures/teaser/face/frames/001014 
    
    13.7mm
      [width=0.0875]7figures/teaser/camera/frames_input/001014 
    
    13.7mm
      [width=0.0875, poster=6]7figures/teaser/camera/frames/001014 
    
    2.8mm
      [width=0.0875]7figures/teaser/handcrafted/frames_input/001014 
    
    2.8mm
      [width=0.0875, poster=8]7figures/teaser/handcrafted/frames/001014 
    
    We encode the motion of a reference video into a novel motion-text embedding using a frozen, pre-trained image-to-video diffusion model. This optimized motion-text embedding can then be applied to different starting images to generate videos with semantically similar motions. The general nature of our motion representation allows for successful motion transfer even when objects are not spatially aligned, across various domains, and for multiple objects. Additionally, our method supports multiple types of motions, including full-body, face, camera, and even hand-crafted motions. 

    1 Please refer to the supplementary material for corresponding videos for all figures of this paper. 
    A figure made up of two blocks. In the top block, the rough model architecture is shown. A motion-text embedding is extracted from a motion reference of a person doing jumping jacks, and this motion is then applied to three different input images and to an unconditional appearance (motion visualization). The generated videos all show a jumping jack motion even if the objects are not spatially aligned, from different domains and for multiple objects. In the bottom block, a number of motion input and generated output videos are shown for full-body reenactment, face reenactment, hand-crafted motions, and camera motions.
    



Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion
    Romann M. Weber
    August 5, 2024
===============================================================================





§ INTRODUCTION
 

Since the advent of diffusion models, video generation and editing methods have significantly improved. However, controlling the motions generated by these models remains challenging. For instance, Stable Video Diffusion <cit.>, a state-of-the-art image-to-video diffusion model, offers little practical control over motion. The primary methods to modify motion involve altering the diffusion process's random seed or adjusting micro-conditioning inputs like frame rate, but these options are not easily interpretable[The authors demonstrate camera motion control by fine-tuning motion-specific LoRA <cit.> modules; however, we consider this an extension rather than a feature of the base model.].

Existing methods for controlling motion with sparse control signals like text <cit.>, boxes <cit.>, or trajectories <cit.> are limited to simple motions in most practical scenarios. On the other hand, dense motion trajectories <cit.> may leak the motion reference video's spatial structure, thus often failing in unaligned scenarios.

Many reference-based motion transfer methods fine-tune model components (e.g., using LoRA <cit.>) on one or several motion reference videos <cit.>. This often results in appearance leakage, where the model overfits to the reference video's appearance and fails to generalize to new target object appearances. Motion transfer techniques that use the inversion-then-generation framework <cit.> attempt to replicate the reference video's structure in the generated video. However, this approach can be problematic when there are significant differences between the locations and geometries of the reference and target objects, leading to misaligned semantic features.

Most of the one-shot reference-based methods produce videos with motions that are mostly spatially aligned with the motion reference video, i.e., they follow the layout as well as the subject scale and position of the reference video. We thus argue that many of these works <cit.> can be considered as an advanced form of appearance transfer rather than motion transfer. 
Lastly, most existing methods are based on text-to-video models and thus cannot preserve the exact appearance and background of a given target image.

We propose to transfer the semantic motion of a motion reference video to a given target image. Specifically, given motion reference video V_R and target image I_T, we want to generate a video V̂ such that M_sem(V̂) = M_sem(V_R), where M_sem is the semantic motion, and A(V̂) = A(I_T), where A is the appearance and spatial layout. Importantly, the generated motion should match the semantics of the motion reference video rather than the exact same spatial motions. For example, we want to be able to generate a video of a given subject doing jumping jacks on the left side of the video even if the subject in the motion reference video is in the center.  Fig. <ref> shows exemplary results of our method, including motion transfers to multiple (misaligned) objects.

We identify two key challenges: appearance leakage from the motion reference video and object misalignment. To tackle appearance leakage, we employ an image-to-video rather than a text-to-video (or an inflated text-to-image) model and do not fine-tune the model. To the best of our knowledge, we are the first to use an image-to-video model for general motion transfer. To address object misalignments between the motion reference video and the target image, we introduce a novel motion representation that eliminates the need for spatial alignment by not having a spatial dimension in the first place.

Our motion representation is based on our observation that image-to-video models extract the appearance predominantly from the (latent) image input, whereas the text/image embedding injected via cross-attention mostly controls the motion. We therefore propose to represent motion with several text/image embedding tokens, together referred to as motion-text embedding, that we optimize on a given motion reference video. Thereby, our inflated motion-text embedding enables us to preserve the timing of the motion video very precisely, which is crucial for applications such as visual dubbing. Our approach, named motion-textual inversion, is general in nature and works for various types of motions and objects. Perhaps surprising at first, it turns out that while words are not ideal for describing motions, their embeddings can describe highly complex motions exceptionally well, as shown in our experiments.

To summarize, our contributions are:

    
  * We introduce the semantic video motion transfer task in an image-to-video setting.
    
  * We observe that text/image embeddings of image-to-video diffusion models store and affect motion and leverage them as a general and compact motion representation.
    
  * We propose motion-textual inversion, a novel method that optimizes multiple text/image embedding tokens on a motion reference video and transfers the learned motion to target images.
    
  * We demonstrate superior performance over existing motion transfer approaches.




§ RELATED WORK
 



 §.§ Domain-Specific Reenactment
 

Reenactment has been a significant research area, but much of the focus has been on domain-specific approaches like face reenactment <cit.> and human full-body motion transfer <cit.>. While these methods perform well, their architectures and training data are tailored to specific domains, making it challenging to adapt them for use across multiple domains. Current general image animation methods <cit.> also still require domain-specific training, so they cannot directly solve our task.

Since our aim was to introduce a general method, our design choices differ from domain-specific models. For instance, many reenactment techniques rely on keypoints, which are derived either from pre-trained, domain-specific landmark detectors <cit.> or learned in an unsupervised manner <cit.>. Although the latter approach is more flexible, it still requires a separate training for each domain, which is cumbersome. Additionally, this approach presents challenges in determining keypoint placement for unseen domains during inference (e.g., inanimate objects), cross-domain transfers (e.g., human-to-animal), and misaligned objects. We therefore choose an implicit motion representation rather than an explicit one. 

Given the impressive cross-domain translation capabilities of diffusion models, as demonstrated in works like <cit.>, we utilize a diffusion-based video foundation model for our general task to take advantage of its extensive and general priors.



 §.§ Video Generation


Following the rise of text-to-image diffusion models <cit.>, video generation models have also greatly improved in quality in recent years. Many text-to-video methods start with a pre-trained text-to-image model and inflate it by adding and training temporal convolution and attention blocks after each corresponding spatial block <cit.>. Similarly, many image-to-video diffusion models use a pre-trained text-to-image <cit.> or text-to-video <cit.> model as a starting point. They then adapt the model to the image-to-video task by conditioning the model on the image, e.g., by adding <cit.> or concatenating <cit.> it to the noisy input. The text embedding input from the pre-trained model is either kept <cit.> or replaced with an image embedding input <cit.>. While training a custom video generation model provides the most freedom in terms of design choices, it is very expensive in terms of computation and data. Even fine-tuning video models requires substantial resources, so we decided to use a pre-trained diffusion model, Stable Video Diffusion <cit.>, and keep it frozen. Additionally, we aim for our method to be applicable to a wide range of motions and subjects. In contrast, approaches that involve training the model often focus on a single type of motion, such as human full-body motion <cit.>.



 §.§ Video Motion Editing with Explicit Motions




  §.§.§ Based on Sparse Control Signals


In theory, the motion of all video generation models that have a text input can simply be controlled by text <cit.>, but this approach struggles with complex motions in practice. For more precise spatial control, recent methods use bounding boxes, either with training <cit.> or without <cit.>, and trajectories <cit.>, but they rely on consistent spatial alignment for effective motion transfer. Similarly, keypoints are another option for describing motions <cit.>, but they suffer from the challenges outlined in Section <ref>. Additionally, some methods focus specifically on camera motions <cit.> or combine camera and bounding box motions <cit.>. However, all these approaches are either limited to simple motions or require significant effort to specify complex ones. For instance, a bounding box can specify an object's location (e.g., a person) but not the detailed motion within it (e.g., doing jumping jacks), and modeling complex motion with part-based boxes quickly becomes impractical.



  §.§.§ Based on Dense Control Signals


Dense control signals, such as motion vectors <cit.> and depth maps <cit.>, allow for a more precise motion specification. However, using them for general motion transfer is challenging because they also encode information about image and object structure. This can result in unnatural motions when there is a mismatch between the structures of the target image and the reference video as shown in <cit.>.



 §.§ Video Motion Editing with Implicit Motions


This subsection covers methods for implicitly representing and transferring motion from a reference video. Fine-tuning approaches store motion in model weights, whereas inversion-then-generation methods store motion in model features and attention maps. Some techniques combine both paradigms. When the layout of the subjects in the reference and generated videos match, a given transfer can be seen as either changing the appearance to match the target image or altering the motion to match the reference video. Our focus is on motion transfer where the layouts do not align, a less explored area in the literature, as discussed in Section <ref>.



  §.§.§ Fine-Tuning
 

Many fine-tuning methods are inspired by image customization techniques like DreamBooth <cit.> and LoRA <cit.>. Loosely speaking, the idea is to fine-tune the parts of the model responsible for motion but avoid training the parts responsible for appearance. Tune-A-Video <cit.> inflates a text-to-image model by adding spatio-temporal attention and only trains some parts of the attention layers. Similarly, <cit.> only fine-tunes parts of the model and further focuses the training more on earlier denoising steps to emphasize learning the general motion rather than fine appearance details. MotionDirector <cit.> proposes a dual-path LoRA architecture and an appearance-debiased temporal loss to disentangle appearance from motion. Similarly, DreamVideo <cit.>, MotionCrafter <cit.>, and Customize-A-Video <cit.> have separate branches for appearance and motion. VMC <cit.> adapts temporal attention layers using a motion distillation strategy with residual vectors between consecutive noisy latent frames as the motion reference. 

Fine-tuning a model carries the risk of appearance leakage, which is why many of the aforementioned methods focus on preventing it. We find that using an image-to-video model instead of a text-to-video model largely avoids these problems. LAMP <cit.> is the most similar method to ours in that sense, but they adapt a pre-trained text-to-image model to the image-to-video task and fine-tune it only briefly. In contrast, we employ a pre-trained, large-scale image-to-video model to leverage stronger priors for better generalization.




  §.§.§ Inversion-then-Generation
 

The inversion-then-generation framework, initially developed for image editing <cit.>, involves first inverting a reference video into “noise” using methods like DDIM <cit.> to enable reconstruction through backward diffusion. Thereby, features such as self-attention maps are extracted from the reference video and then injected into the diffusion process of the video being generated. These features either directly replace existing features <cit.> or are incorporated into a loss function <cit.>, ensuring the generated video has a similar structure. Numerous methods have been proposed within this framework for video appearance editing <cit.> and video motion editing <cit.>, mainly differing in their inversion techniques and feature choices.

The methods mentioned above face several inherent issues in motion transfer tasks. Most notably, they often assume or enforce that the features of the reference and target videos are identical, which leads to problems when generating videos with different geometries or spatial layouts. Some methods attempt to address this by collapsing the spatial dimension of features before using them in a loss <cit.>, but they still typically produce motions with similar directions in pixel space. This limits control and diversity and can produce less natural results. Furthermore, these approaches require tuning numerous hyperparameters (choice of feature, layers, time steps) and necessitate inverting the video, which is challenging for high guidance scales <cit.> and when using few time steps <cit.>.




  §.§.§ With Different Spatial Layout
 

To avoid being restricted to the layout of a single motion reference video, some methods use multiple motion videos <cit.>. However, our goal is to transfer motion with precise temporal alignment to the reference video. This would require multiple temporally-aligned videos, which are often impractical to obtain. Additionally, many motion editing methods with spatial variations <cit.> use text to define the subject's appearance instead of an image, resulting in videos that only roughly match the input image. The approach in <cit.> is most similar to ours as it keeps the model frozen and learns a motion embedding like we do, but it also suffers from the above limitation.



§ METHOD



[width=0.09, poster=3]7figures/i2v_ignores_text/white_horse_frames/001014

[width=0.09, poster=3]7figures/i2v_ignores_text/white_horse_but_pink_text_frames/001014




[width=0.09, poster=3]7figures/horse_clip_swap_1024_576/white_horse_frames/0114

[width=0.09, poster=3]7figures/horse_clip_swap_1024_576/white_horse_with_pink_horse_emb_frames/0114

[width=0.09, poster=3]7figures/horse_clip_swap_1024_576/pink_horse_with_white_horse_emb_frames/0114

[width=0.09, poster=3]7figures/horse_clip_swap_1024_576/pink_horse_frames/0114



We propose to transfer the semantic motion of a motion reference video to a given target image by motion-textual inversion. We thereby optimize a set of text/image embedding tokens, which we refer to as motion-text embedding, for the motion reference video using a pre-trained image-to-video diffusion model. 



 §.§ Preliminaries
 

Diffusion models <cit.> consist of two processes. In the forward process, Gaussian noise is iteratively added to a clean data sample x_0 until it is approximately pure noise. In the reverse process, starting with pure noise x_T, a learnable denoiser D_θ iteratively removes noise to obtain a sample that matches the original data distribution p_data. We follow the continuous-time framework <cit.>, where the denoiser is trained via denoising score matching: 

    𝔼_(x_0, c) ∼ p_data(x_0, c),(σ, n) ∼ p(σ, n)[λ_σ ||D_θ(x_0 + n; σ, c) - x_0||^2_2],

where x_0 is a clean data sample and c an arbitrary conditioning signal from the original data distribution p_data; p(σ, n) = p(σ)𝒩(n; 0, σ^2), where p(σ) is a probability distribution over noise levels σ, and n is noise; and λ_σ: ℝ_+ →ℝ_+ is a weighting function. The denoiser D_θ is parameterized as

    D_θ(x; σ) = c_skip(σ)x + c_out(σ)F_θ(c_in(σ)x; c_noise(σ)),

where F_θ is the neural network to be trained; c_skip(σ) modulates the skip connection; c_out and c_in scale the output and input magnitudes respectively; and c_noise maps noise level σ into a conditioning input for F_θ. For more details, please refer to <cit.>.

Latent diffusion models <cit.> operate in the latent space rather than in pixel space to reduce computation and thus enable higher resolutions. First, an encoder ℰ produces a compressed latent z = ℰ(x). Then, we perform the diffusion process over z. Lastly, a decoder 𝒟 reconstructs the latent features back into pixel space.[To maintain consistency in notation, we use x for the diagrams and method description, even though the diffusion process actually occurs in latent space.]

Stable Video Diffusion (SVD) <cit.> is a video latent diffusion model trained in three stages: 1. A text-to-image model <cit.> is trained or fine-tuned on (image, text) pairs. 2. The diffusion model is inflated by inserting temporal convolution and attention layers following <cit.> and then trained on (video, text) pairs. 3. The diffusion model is refined on a smaller subset of high-quality videos with exact model adaptations and inputs depending on the task (text-to-video, image-to-video, frame interpolation, multi-view generation). For image-to-video generation, the task is to produce a video given its starting frame. The starting frame is supplied to the model in two places: as a CLIP <cit.> image embedding via cross-attention (replacing the CLIP text embedding from the text-to-video pre-training) and as a latent repeated across frames and concatenated channel-wise to the video input. Additionally, the model is micro-conditioned on the frame rate, motion amount, and strength of the noise augmentation (applied to first frame latent).



 §.§ Motivation
 

Transferring the motion of a reference video to a given target poses two key challenges, which our design solves quite naturally.



  §.§.§ Challenge 1: Appearance Leakage
 

When fine-tuning a text-to-video model to learn the motion from a single reference video, there is a risk of overfitting to the reference video's appearance, which can prevent the model from producing the correct target appearance during inference. We demonstrate that using a frozen image-to-video model can preserve the target appearance without any of the special mechanisms from literature described in Section <ref>. 

By design, image-to-video models generate videos given a starting frame, so generated frames tend to preserve the appearance of the input image. We observe that image-to-video models primarily derive the appearance from the image (latent) input, even with an additional text input, as shown in Fig. <ref>. This is likely because the model can directly copy (latent) pixels from the first frame instead of hallucinating them from the sparse text input. This strong reliance on the image input reduces the chance of the reference video's appearance leaking through. To further minimize the risk of appearance leakage, we keep the model's weights frozen, so they cannot possibly store the reference video appearance. This also helps retain the rich video understanding and generalization capabilities of the pre-trained model.





  §.§.§ Challenge 2: Handling Object Misalignment.
 

Our goal is to generate videos where subjects perform the same semantic actions, even if they are in different spatial locations or orientations. Handling misaligned objects is especially important when using image-to-video models because the subject's position is determined by the input image, which typically does not match the position in the motion reference video. 

As discussed in Section <ref>, existing methods using the inversion-then-generation framework inject features from the motion reference video into the generated video, making it closely follow the reference structure. Arguably, these methods do not copy the motion at its origin but rather the per-frame structure that results from a motion (e.g., rough object positions). For the general, unaligned case, these features would first need to be aligned spatially to avoid injecting the structure in the wrong place. This alignment is challenging since the final positions in the generated video are unknown during the diffusion process as they depend on the motion.

We forgo the alignment problem by representing motions with text or image embedding tokens that do not have a spatial dimension in the first place. Our novel motion representation was motivated by the observation shown in Fig. <ref>. While SVD generated walking motions for an image of a real horse, it generated no object but mostly camera motion for an image of a pink toy horse[Image was generated using <cit.>.], perhaps because the model learned that toys do not move. Recall that SVD has the first frame as input in two places: as image latent and as CLIP <cit.> image embedding. When using the image latent of the real horse but the CLIP embedding of the toy horse, the horse in the generated video does not move. Inversely, the toy horse starts walking when using the CLIP embedding of the real horse, implying that the CLIP embedding affects the motion. We believe that these embeddings are not just affecting the motion but are actually the main origin of the motion. 

Our intuition for why the text/image embeddings determine the motion (which may be surprising at first) is as follows: Videos can be divided into appearance and motion. Appearance is tied to the spatial arrangement of pixels, making it easier to extract it from spatial inputs like image latents. Motion depends on how pixels change over time, requiring a more global, semantic understanding. Thus, it is more effective to modify motion using image embeddings, which contain more semantic information, have no spatial dimension, and are injected in multiple places of the model. Furthermore, SVD was initially trained as a text-to-video model, with CLIP text embeddings describing motions like “standing”, “walking”, or “running,” incentivizing the model to control motion through cross-attention inputs to effectively denoise training videos.



 §.§ Motion-Textual Inversion
 

While using embeddings from different images can alter the generated motion, it does not transfer the motion robustly. Moreover, selecting a specific frame to define a desired motion is difficult since motion is rarely captured by a single frame. To address this, we propose optimizing the embedding based on a given motion reference video, which bears some resemblance to textual inversion <cit.>. In analogy to textual inversion, we name our method motion-textual inversion[In our implementation, it is actually an image embedding, but we refer to it as “motion-textual inversion” since SVD's image and text embeddings share the same CLIP space, and other I2V methods use text embeddings instead. Also, it feels more intuitive to represent motions as text rather than an image.]. Note, however, that our method has a completely different goal: using embeddings to encode video motion rather than image appearance.

Fig. <ref> shows a high-level overview of our method. Given a single motion reference video x_0 containing F frames, we optimize the motion-text embedding m directly by minimizing the diffusion model loss from Equation <ref>, keeping the diffusion model frozen:

    m^* = marg min    𝔼_(x_0, c) ∼ p_data(x_0, c),(σ, n) ∼ p(σ, n)
        [λ_σ ||D_θ(x_0 + n; σ, m, c) - x_0||^2_2],

where c encompasses all remaining conditionings of SVD (e.g., first frame latent, time/noise step, and micro-conditionings). All other symbols are defined in Equations <ref> and <ref>.



 §.§ Motion-Text Embedding and Cross-Attention Inflation
 



Cross-attention allows the model to dynamically attend to different tokens (∼ words in text-to-image and text-to-video) depending on the current features or context. It is computed as follows:

    Attention(Q, K, V) = M V = softmax(QK^T/√(d_a)) V, 
     
        Q = φ_i(z_t) W_Q,i,   K = m W_K,i,   V = m W_V,i,

where Q, K, V are the queries, keys, and values respectively; M is the attention map; d_a is the dimension used in the attention operation; φ_i(z_t) is an intermediate representation of the level i features with C_i channels; m is the motion-text embedding (or text/image embedding e in case of baseline SVD) with embedding dimension d; and W_Q,i∈ℝ^C_i × d_a, W_K,i∈ℝ^d × d_a, and W_V,i∈ℝ^d × d_a are learned weight matrices for queries, keys, and values respectively.

The image embedding of SVD only has one token. This results in a degenerate case for the cross-attention operation where all entries of the attention map M are 1, as shown in Fig. <ref>. The model thus attends 100% to that single token and applies its value to all spatial and temporal locations. 



  §.§.§ Multiple Tokens


Using the same value V for all locations limits the extent to which the embedding can change the motion. We therefore propose to have N tokens (instead of 1) to attend to, recovering the scenario from the text-to-image or text-to-video pre-training. This allows the model to dynamically attend to different tokens depending on the features, e.g., using different values for the background and foreground as seen in the spatial cross-attention maps in Fig. <ref>.


[width=0.08, poster=2]7figures/motion_vis/cond_000000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000000_frames_2/001014

[width=0.08, poster=2]7figures/motion_vis/cond_000200_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000200_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000200_frames_2/001014

[width=0.08, poster=2]7figures/motion_vis/cond_000500_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000500_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000500_frames_2/001014

[width=0.08, poster=2]7figures/motion_vis/cond_001000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_001000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_001000_frames_2/001014

[width=0.08, poster=2]7figures/motion_vis/cond_002000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_002000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_002000_frames_2/001014





  §.§.§ Different Tokens per Frame


For the spatial cross-attention, the default SVD broadcasts the image embedding across all frames. We argue that we can obtain a higher temporal granularity of the motion by using a different set of tokens for each frame, i.e., F × N tokens[Note that we always use the same F frames of the motion reference video when optimizing the motion-text embedding.]. Having different sets of tokens leads to different sets of keys and values for each frame. Different keys allow the model to attend to different spatial locations for different frames, e.g., attending to the arm of a person in one frame but to the leg in another frame. Different values allow the model to apply different changes to the features for different frames, e.g., applying a vector to inform the model to shift pixels in one direction in one frame but another direction in another frame. This is visualized in Fig. <ref>, where the spatial cross-attention maps differ greatly between frames because they use different tokens. 

For the temporal cross-attention, the default SVD broadcasts the image embedding across all spatial locations. Inflating this analogously to the spatial case would mean inflating the embedding across the spatial dimensions. Learning different tokens for different spatial locations is nontrivial (spatial dimensions change depending on the input resolution and the level i) and would likely cause issues with the alignment (see Section <ref>). Furthermore, in our empirical experiments, the temporal cross-attention seemed to have a smaller effect on the generated motion than the spatial cross-attention. Therefore, we decided to keep the number of tokens of our temporal motion-text embedding at N but learn them separately from the F × N tokens of the spatial motion-text embedding. Our combined motion-text embedding thus has (F+1) × N tokens per motion reference video. Section <ref> describes the tensor operations and dimensions of our motion-text embedding and cross-attention inflation in detail.

To give an intuitive analogy for our motion-text embedding inflation, think of building a house. Instead of using a single tool for every part of the house, it is more efficient to have N different tools depending on the spatial location on a given floor – like a hammer for the floor and a drill for the wall. Moreover, each of the F floors of the house might need a different set of tools. For example, the roof requires different tools compared to the walls. Similarly, in our approach, we use multiple tokens to handle different aspects of the motion. 





§ EXPERIMENTS




 §.§ Implementation Details


We base our implementation on the 14-frame version of Stable Video Diffusion (SVD) <cit.>, but our method could in theory be applied to other image-to-video methods that have a text/image embedding input. Per default, we use N=5 different tokens for each of the F=14 frames, so a total of (14+1) × 5 = 75 tokens for the motion-text embedding. Please refer to Section <ref> for the remaining hyperparameters and implementation details.

Our data consists of motion reference videos and target images from internal data sets as well as target images generated with Stable Diffusion XL <cit.>. We only use one reference video for the optimization per motion and can apply this motion to various target images.



 §.§ Motion-Text Embedding Analysis


SVD was pre-trained as a text-to-video model and dropped the image (latent) input for some percentage of training iterations for classifier-free guidance <cit.>. We find that SVD can produce somewhat reasonable videos with the image (latent) input zeroed out and only the CLIP <cit.> image embedding as input, especially if we increase the classifier-free guidance scale. We can use this to visualize our learned motion-text embedding with an unconditional appearance[Note that the visualization is not always easily interpretable, depending on the motion, the optimization iteration, and the seed.]. Fig. <ref> shows motion visualizations of our motion-text embedding for a “jumping jacks” motion after different numbers of optimization iterations and the generated videos for a given target image side-by-side. Starting around iteration 500, a person doing a “jumping jacks” motion can be seen in the visualizations. Beyond 1000 iterations, the motion visualizations become more abstract, but the generated motions in the conditional case remain of high quality. Notably, the appearance and position of the people do not match those of the motion reference video (from Fig. <ref>). Furthermore, the position of the people is different in the conditional and unconditional videos, but all videos have a similar semantic motion. This demonstrates that our motion-text embedding neither encodes the appearance nor the exact spatial positioning of the objects extensively, likely for reasons described in Section <ref>.



 §.§ Qualitative Evaluation



[width=0.08, poster=0]7figures/qual_eval/fullbody_jumping_jacks/frames_input/001014

[width=0.08, poster=0]7figures/qual_eval/face_nodding/frames_input/001014

[width=0.08, poster=0]7figures/qual_eval/camera_medium/frames_input/001014

[width=0.08, poster=4]7figures/qual_eval/fullbody_jumping_jacks/frames_svd/001014

[width=0.08, poster=3]7figures/qual_eval/face_nodding/frames_svd/001014

[width=0.08, poster=6]7figures/qual_eval/camera_medium/frames_svd/001014

[width=0.045, poster=4]7figures/qual_eval/fullbody_jumping_jacks/frames_vc/001014

[width=0.045, poster=3]7figures/qual_eval/face_nodding/frames_vc/001014

[width=0.045, poster=6]7figures/qual_eval/camera_medium/frames_vc/001014

[width=0.045, poster=4]7figures/qual_eval/fullbody_jumping_jacks/frames_md/001014

[width=0.045, poster=3]7figures/qual_eval/face_nodding/frames_md/001014

[width=0.045, poster=6]7figures/qual_eval/camera_medium/frames_md/001014

[width=0.08, poster=4]7figures/qual_eval/fullbody_jumping_jacks/frames_ours/001014

[width=0.08, poster=3]7figures/qual_eval/face_nodding/frames_ours/001014

[width=0.08, poster=6]7figures/qual_eval/camera_medium/frames_ours/001014



As our baseline, we use SVD <cit.> without any adaptations. Since it does not have a motion control input, we do not expect it to follow the correct motion given only the starting image in most cases. However, we show it anyway to showcase the quality and motions of typical SVD outputs. To our knowledge, our method is the first method to address the general motion transfer task in the image-to-video setting. Therefore, there are few methods to choose from that permit a direct and fair comparison. We considered the two most similar classes of methodology and compared our method with a representative of each class. Methods within these classes tend to have certain drawbacks in common:

    
  * Image-to-video methods with explicit motion representation: Without proper alignment, these methods transfer spatial but not semantic motion and may leak the reference video's structure. Since it is unclear how to automatically extract sparse motion inputs, we compare to VideoComposer <cit.>, which uses dense motion vectors.
    
  * Text-to-video methods with implicit motion representation: These methods do not directly take a target image input, compromising the preservation of the target's appearance and layout.  We compare to MotionDirector <cit.> since it learns the target image's appearance, which we expect to perform better than specifying the appearance with text alone.


We use the official implementations. For the models requiring a text input, we use BLIP <cit.> to generate image captions and SpaceTimeGPT <cit.> to generate video captions.

Fig. <ref> shows motion transfer results for three videos. The SVD baseline produces high-quality videos for the robot and waterfall, but their motions do not match the reference videos. For the face video, SVD introduces artifacts and changes the person's identity with head movement. In our experience, SVD often generates static objects with moving cameras, which users might find unexpected or frustrating. We suggest that motion transfer methods, like ours, can help generate more natural and diverse motions. VideoComposer fails to transfer motions successfully, alters appearances, and sometimes introduces strong artifacts. This issue is inherent to its class and arises from the dense motion inputs, which implicitly encode the reference video's structure that may not align with the input image. Since proper alignment is difficult to achieve in our general (and potentially cross-domain) setting, we find it easier to use an implicit motion representation without a spatial dimension, like ours, for general semantic video motion transfer. MotionDirector correctly transfers motion in the head-nodding example but fails to produce the correct motion for the other two. Furthermore, the appearance does not match the target image exactly. For example, the robot is rotated, the person's identity changes, and the waterfall also looks differently. This limitation stems from text-to-video methods not having a direct image input and relying on learning the appearance (or predicting it from text). Thus, we believe that image-to-video models have inherent advantages over text-to-video models when animating a given image. Our method uniquely preserves the input image's appearance and layout while successfully transferring the semantic motion of the video.



 §.§ Results



[width=0.09, poster=0]7figures/results/fullbody_drunk/frames_input/001014

[width=0.09, poster=0]7figures/results/fullbody_animal/frames_input/001014

[width=0.09, poster=3]7figures/results/fullbody_drunk/frames/001014

[width=0.09, poster=3]7figures/results/fullbody_animal/frames/001014

[width=0.09, poster=0]7figures/results/face_opening_mouth/frames_input/001014

[width=0.09, poster=0]7figures/results/face_tilting_up/frames_input/001014

[width=0.09, poster=3]7figures/results/face_opening_mouth/frames/001014

[width=0.09, poster=2]7figures/results/face_tilting_up/frames/001014

[width=0.09, poster=0]7figures/results/camera_birdseye_rotate/frames_input/001014

[width=0.09, poster=0]7figures/results/camera_object_following/frames_input/001014

[width=0.09, poster=3]7figures/results/camera_birdseye_rotate/frames/001014

[width=0.09, poster=3]7figures/results/camera_object_following/frames/001014

[width=0.09, poster=0]7figures/results/handcrafted_robot_jumping/frames_input/001014

[width=0.09, poster=0]7figures/results/handcrafted_passing_front/frames_input/001014

[width=0.09, poster=4]7figures/results/handcrafted_robot_jumping/frames/001014

[width=0.09, poster=5]7figures/results/handcrafted_passing_front/frames/001014



Our motion representation is highly versatile, allowing our method to work with nearly any type of object and motion, as demonstrated in Fig. <ref> and Fig. <ref>. Notably, we do not require a spatial alignment, so our motion-text embedding can be applied to images where objects have different positions and orientations from those in the reference video. For instance, in the sixth row on the right side of Fig. <ref>, the camera tracks the moving camper van in a similar way to how it tracks the car in the fifth row, despite their misalignment. Furthermore, our method applies the motion to all semantically reasonable objects simultaneously “for free”. It can even handle simple hand-crafted motions, enabling artists to sketch a motion (e.g., drawing stick figures) and apply it to complex scenes. For more results, including joint subject and camera motion as well as extreme cross-domain transfers and applying the same motion to multiple target images, please refer to Section <ref>. 



 §.§ Ablation Study



[width=0.07, poster=0]7figures/ablation_study/frames_input/001014

[width=0.07, poster=4]7figures/ablation_study/frames_same_clip_1/001014

[width=0.07, poster=4]7figures/ablation_study/frames_same_clip_15/001014

[width=0.07, poster=4]7figures/ablation_study/frames_clip_1/001014

[width=0.07, poster=4]7figures/ablation_study/frames_default/001014

[width=0.07, poster=4]7figures/ablation_study/frames_clip_15/001014




[width=0.07, poster=0]7figures/failure_cases/quality_svd/frames_input/001014

[width=0.07, poster=3]7figures/failure_cases/quality_svd/frames/001014

[width=0.07, poster=0]7figures/failure_cases/structure_leak/frames_input/001014

[width=0.07, poster=4]7figures/failure_cases/structure_leak/frames/001014

[width=0.07, poster=0]7figures/failure_cases/partial_motion_transfer/frames_input/001014

[width=0.07, poster=4]7figures/failure_cases/partial_motion_transfer/frames/001014

Our proposed motion-text embedding inflation is crucial for high-quality motion transfers. Fig. <ref> illustrates different settings for the motion-text embedding size. With only one token, only a small portion of the motion is transferred. Increasing the number of tokens but sharing them across all frames offers some improvement. However, the key factor is having different tokens per frame. Both rows 2 and 3 use 15 tokens, but the version with different tokens per frame performs significantly better. This is logical since the motion-text embedding can adapt to each frame, which is particularly beneficial for complex motions. Increasing the number of tokens per frame further improves performance slightly, but it eventually saturates, so we default to using N=5.



 §.§ Limitations


Fig. <ref> shows typical failure cases of our method. Since we do not fine-tune the model, our method inherits the priors and quality of our pre-trained image-to-video model. We observed that the SVD baseline often struggles with object motions, as can be seen in the head example in Fig. <ref>, where the appearance changes throughout the video. Our method's results have similar issues: in the first example of Fig. <ref>, the identity of the target person changes when he moves his head to the side. We believe our motion-text embedding does not exacerbate these issues or temporal inconsistencies, as it primarily instructs the model on the desired motion without altering the rest of the model. Often, it seems that the model attempts to produce the desired motion, but its priors are insufficient to generate a satisfactory result. SVD also does not seem to be able to handle some combinations of motions and given input images, likely because they fall outside of the range of the training data set. When the domain gap between motion reference video and target image is too large, our method may leak the structure of the motion reference video into the generated video. In the second example of Fig. <ref>, when applying a laid-back walking style to a kangaroo, the kangaroo starts walking, but its feet and overall structure become more human-like. Lastly, we found that some motions are not transferred or to a smaller extent. This is especially visible if a video has multiple motions, where the more fine-grained motion is sometimes not transferred. In the third example of Fig. <ref>, the person pretends to squat down and type on a keyboard. The dinosaurs in the generated video do squat down, but their hands do not move. We hypothesize that fine-grained motions are also a general limitation of SVD. Overall, we expect better results of our method as image-to-video models improve.

We believe our approach is more accessible than approaches that require substantial training or fine-tuning. Nevertheless, our approach does require an optimization procedure that takes around one hour on an NVIDIA Tesla A100 (80 GB) GPU per motion. We encourage future work in reducing this per-motion optimization time as well as trying to learn a model to directly predict motion-text embeddings from motion reference videos. 






§ CONCLUSION


We introduce the general task of transferring the semantic motion of a reference video to any target image. We observe and exploit inherent advantages of image-to-video over text-to-video models for this task and find that text/image embedding tokens are well-suited as a motion representation. Specifically, our method, motion-textual inversion, optimizes an inflated version of the text/image embedding for a given motion reference video. Due to its general nature, this motion can then be applied to a wide number of objects and domains. Our method thus enables completely novel applications and takes a significant step towards being able to reenact anything.



§ ACKNOWLEDGEMENTS


We would like to thank Michael Bernasconi, Dominik Borer, Jakob Buhmann, and Daniela Kansy for providing motion videos as well as all participants featured in our internal datasets. We also want to give special thanks to Michael Bernasconi, Vukasin Bozic, Karlis Briedis, Pascal Chang, Guilherme Haetinger, Christopher Otto, Lucas Relic, Seyedmorteza Sadat, and Agon Serifi for their valuable and insightful discussions throughout the project.



ACM-Reference-Format





§ BROADER IMPACT AND ETHICS


To the best of our knowledge, our method is the first that can reenact a wide array of objects and motions given a target image and motion reference video without training domain-specific models. We believe this represents a significant advancement in controllable video generation, as our approach can address multiple existing domain-specific scenarios within a single framework and even facilitate entirely new applications. That said, we acknowledge the potential for misuse of reenactment methods like ours, such as creating realistic deepfakes or videos depicting individuals or objects performing specified, potentially inappropriate actions. We strongly condemn such misuse and advocate for implementing safety mechanisms and procedures in real-world applications. Additionally, we support ongoing research into detecting fake videos to mitigate these risks.




§ IMPLEMENTATION DETAILS
 



 §.§ High-Level Overview of the Implementation


To aid in reproducibility, we list the main steps of our method's implementation below:


    
  * [Only once] Take pre-trained Stable Video Diffusion (SVD) <cit.> and adapt code to inflate motion-text embedding and cross-attention. See high-level description in Section <ref> and details in Section <ref>.
    
  * Initialize motion-text embedding of shape (F + 1) x N x d. See Section <ref>.
    
  * Repeat until convergence:
    
        
  * Load same F frames of reference video in data loader for each iteration.
        
  * Augment data. See Section <ref>.
        
  * Input noisy version of frames, motion-text embedding, and other inputs into SVD.
        
  * Apply loss from Equation <ref> to update motion-text embedding.
    
    
  * Save motion-text embedding.
    
  * For all target images:
    
        
  * Input learned motion-text embedding along with new target image to inflated SVD during inference to generate video with motion from reference video.
    




 §.§ Hyperparameters
 

Our implementation builds up on the diffusers implementation <cit.> of Stable Video Diffusion (SVD) <cit.>. We use the default parameters of the 14-frame version of SVD (e.g., micro-conditionings) unless specified otherwise. Like SVD, we generally employ a classifier-free guidance <cit.> scale that increases linearly from 1 to 3 across the frame axis. For the motion visualization (unconditional image input), however, we use a higher scale, i.e., increasing linearly from 1 to 10, to improve the visibility of the objects. We initialize the F=14 sets of N=5 tokens for the spatial cross-attention with the CLIP image embedding token of each corresponding frame and the N=5 tokens for the temporal cross-attention with the mean of the CLIP image embedding tokens across all frames. We additionally add Gaussian noise 𝒩(0, 0.1) to the combined motion-text embedding during initialization. In our experience, the initialization does not affect the results significantly, so other initializations are equally reasonable. During optimization, we always pick the same F frames of a given video and apply the same spatial and color augmentations to all frames[For horizontal camera motions, we turn of horizontal flipping]. Since most of the video motion is determined in noisy diffusion steps, we shift the noise schedule towards higher noise values (from P_mean=1.0, P_std=1.6 to P_mean=2.8, P_std=1.6 where log σ∼𝒩(P_mean, P_std^2) to speed up the optimization. We use Adam <cit.> with a learning rate of 1e-2 for 1000 iterations with a batch size of 1. 





 §.§ Hardware Requirements and Runtime


The optimization for a motion reference video with a resolution of 1024 × 576 takes around 45 GB of GPU memory and around one hour on an NVIDIA Tesla A100 (80 GB) GPU. The inference takes less than one minute per video. Note that our implementation is not optimized extensively for memory or runtime.



 §.§ Motion-Text Embedding and Cross-Attention Inflation
 



This section provides more implementation details for the motion-text embedding and cross-attention inflation described in Section <ref>. Fig. <ref> shows the spatial and temporal cross-attention layers of the default Stable Video Diffusion (SVD) <cit.> and our inflated version along with their tensor dimensions.

The image embedding of the default SVD consists of a single token and has dimensions B × 1 × d, where B is the batch size (in our implementation typically 1 when optimizing the motion-text embedding and 2 during inference because of classifier-free guidance) and d is the CLIP <cit.> embedding dimension. For spatial cross-attention, the image embedding is broadcast to dimensions (B*F) × 1 × d, i.e., the same token is used for all F frames. This results in an attention map M of dimensions (B*F) × (H_i*W_i) × 1 where H_i and W_i are the spatial heights and widths respectively, and C_i is the number of channels of level i of the diffusion model. Notably, due to the softmax operation and the last dimension being 1, every value of the attention map is 1. This means that each spatial location attends 100% to the single token. Similarly, for temporal cross-attention, the image embedding is broadcast from dimensions of B × 1 × d to dimensions (B*H_i*W_i) × 1 × d, eventually leading to an attention map M of dimensions (B*H_i*W_i) × F × 1  where every value is 1. Having only one token thus leads to a degenerate case of the cross-attention where Attention(Q, K, V) = V (broadcasted) and many of the components (e.g., queries and keys) have no effect on the result. 



  §.§.§ Multiple Tokens


To avoid the above degenerate case and instead be able to dynamically attend to different tokens, we extend the token dimension from 1 to N where N is a hyperparameter. For spatial cross-attention, this results in an attention map M of dimensions (B*F) × (H_i*W_i) × N where, in general, each spatial location has different values ≠ 1 for the N different tokens. Similarly, the temporal cross-attention map M has dimensions (B*H_i*W_i) × F × N with values ≠ 1. Since SVD was pre-trained using multiple text embedding tokens as input, the code can already handle multiple tokens, so mainly the initialization of the motion-text embedding as well as some input dimensions have to be adapted slightly. 



  §.§.§ Different Tokens per Frame


As explained in Section <ref>, we propose to learn different sets of tokens per frame for the spatial cross-attention to obtain a higher temporal granularity of the motion. The default SVD implementation broadcasts the embedding from dimensions B × N × d across all frames to (B*F) × N × d (where N=1 originally). We instead learn a larger spatial motion-text embedding of dimensions B × F × N × d and reshape it to (B*F) × N × d. We keep the dimensions of the temporal motion-text embedding at B × N × d and learn it separately. Therefore, the dimensions of the combined spatial and temporal motion-text embedding is B × (F+1) × N × d. 



§ ADDITIONAL RESULTS
 

Fig. <ref> shows that our method does not only apply the rough motion category but also its style, even in cases where the domains differ vastly, e.g., transferring the motion of a horse to a cereal box. Furthermore, these examples demonstrate that our method can transfer joint subject and camera motion. Fig. <ref> shows that our method generates the same semantic rather than spatial motion by applying the same learned motion to a flipped target image. Fig. <ref> shows additional results of our method, where we apply the same motion to different target images.  


[width=0.09, poster=0]7figures/style/trot/frames_input/001014

[width=0.09, poster=0]7figures/style/canter/frames_input/001014

[width=0.09, poster=3]7figures/style/trot/frames/001014

[width=0.09, poster=3]7figures/style/canter/frames/001014

[width=0.09, poster=3]7figures/style/trot/frames_2/001014

[width=0.09, poster=3]7figures/style/canter/frames_2/001014

[width=0.09, poster=3]7figures/style/trot/frames_3/001014

[width=0.09, poster=3]7figures/style/canter/frames_3/001014

[width=0.09, poster=3]7figures/style/trot/frames_4/001014

[width=0.09, poster=3]7figures/style/canter/frames_4/001014




[width=0.09, poster=0]7figures/alignment/frames_input/001014

[width=0.09, poster=4]7figures/alignment/frames/001014

[width=0.09, poster=4]7figures/alignment/frames_2/001014




[width=0.09, poster=0]7figures/results_extra/fullbody_jumping_jacks/frames_input/001014

[width=0.09, poster=0]7figures/results_extra/fullbody_animal_chewing/frames_input/001014

[width=0.09, poster=4]7figures/results_extra/fullbody_jumping_jacks/frames/001014

[width=0.09, poster=4]7figures/results_extra/fullbody_animal_chewing/frames/001014

[width=0.09, poster=4]7figures/results_extra/fullbody_jumping_jacks/frames_2/001014

[width=0.09, poster=4]7figures/results_extra/fullbody_animal_chewing/frames_2/001014

[width=0.09, poster=0]7figures/results_extra/face_nodding/frames_input/001014

[width=0.09, poster=0]7figures/results_extra/face_tilting_down/frames_input/001014

[width=0.09, poster=3]7figures/results_extra/face_nodding/frames/001014

[width=0.09, poster=4]7figures/results_extra/face_tilting_down/frames/001014

[width=0.09, poster=3]7figures/results_extra/face_nodding/frames_2/001014

[width=0.09, poster=4]7figures/results_extra/face_tilting_down/frames_2/001014

[width=0.09, poster=0]7figures/results_extra/camera_medium/frames_input/001014

[width=0.09, poster=0]7figures/results_extra/camera_left/frames_input/001014

[width=0.09, poster=4]7figures/results_extra/camera_medium/frames/001014

[width=0.09, poster=4]7figures/results_extra/camera_left/frames/001014

[width=0.09, poster=4]7figures/results_extra/camera_medium/frames_2/001014

[width=0.09, poster=4]7figures/results_extra/camera_left/frames_2/001014

[width=0.09, poster=0]7figures/results_extra/handcrafted_robot_jumping/frames_input/001014

[width=0.09, poster=0]7figures/results_extra/handcrafted_zooming_rect/frames_input/001014

[width=0.09, poster=4]7figures/results_extra/handcrafted_robot_jumping/frames/001014

[width=0.09, poster=4]7figures/results_extra/handcrafted_zooming_rect/frames/001014

[width=0.09, poster=4]7figures/results_extra/handcrafted_robot_jumping/frames_2/001014

[width=0.09, poster=4]7figures/results_extra/handcrafted_zooming_rect/frames_2/001014








 Paper Title: AMAES: Augmented Masked Autoencoder Pretraining on Public Brain MRI Data for 3D-Native Segmentation.






Augmented Masked Autoencoder Pretraining for 3D Brain Segmentation






A. Munk et al.




Department of Computer Science, University of Copenhagen, Copenhagen, Denmark Pioneer Centre for AI, Copenhagen, Denmark 

{asmu,jaam}@di.ku.dk
Equal contribution



: Augmented Masked Autoencoder Pretraining on Public Brain MRI Data for 3D-Native Segmentation
    Asbjørn Munk1,2* Jakob Ambsdorf1,2* Sebastian Llambias1,2 Mads Nielsen1,2

    August 5, 2024
==============================================================================================



§ ABSTRACT 
This study investigates the impact of self-supervised pretraining of 3D semantic segmentation models on a large-scale, domain-specific dataset. We introduce , a dataset of 44,756 brain MRI volumes from public sources, the largest public dataset available, and revisit a number of design choices for pretraining modern segmentation architectures by simplifying and optimizing state-of-the-art methods, and combining them with a novel augmentation strategy. The resulting AMAES framework is based on masked-image-modeling and intensity-based augmentation reversal and balances memory usage, runtime, and finetuning performance. Using the popular U-Net and the recent MedNeXt architecture as backbones, we evaluate the effect of pretraining on three challenging downstream tasks, covering single-sequence, low-resource settings, and out-of-domain generalization. The results highlight that pretraining on the proposed dataset with AMAES significantly improves segmentation performance in the majority of evaluated cases, and that it is beneficial to pretrain the model with augmentations, despite pretraing on a large-scale dataset. Code and model checkpoints for reproducing results, as well as the  dataset are available at <https://github.com/asbjrnmunk/amaes>.









§ INTRODUCTION





Large-scale self-supervised pretraining promises immense potential for medical image segmentation. Works in computer vision from the natural image domain demonstrate that state-of-the-art results are achieved by scaling model capacity and pretraining on large datasets, enabling increased generalization performance, few- or zero-shot predictions, and highly semantic embeddings <cit.>. 


To enable self-supervised representation learning in vision, various methods have been proposed. Early works involved reversing image degradations, such as denoising images <cit.>, solving jigzaw puzzles <cit.>, or re-colorizing images <cit.>.
Masked-Image-Modeling (MIM) proved to be an especially successful and scalable variant, where patches of the input image are masked and learned to be reconstructed either directly or indirectly by predicting an embedding <cit.>. A Masked Autoencoder (MAE) is an encoder-decoder architecture that reconstructs the entire image from a masked version <cit.>.
Another family of approaches applied contrastive learning, where semantically similar transformed inputs are mapped closely in a latent space, while dissimilar inputs are pushed apart <cit.>. Self-distillation is based on a related idea, enforcing similar embeddings for semantically close inputs for a student and teacher network <cit.>.
Recent approaches combined these image-level contrastive or self-distillation objectives with patch-level MIM to enhance the spatial representation performance for downstream segmentation tasks <cit.>.

Evidence from medical vision indicates that the benefits of pretraining are transferable to biomedical image segmentation <cit.>. Yet, the current method of choice for solving segmentation tasks is still a fully-supervised paradigm <cit.>. Due to a heavily optimized data augmentation pipeline, the fully-supervised approach can be effective even with limited labeled data <cit.>. However, to fully leverage the potential of large-scale models in medical image segmentation as witnessed in computer vision, more training data is required, and the supervised approach is inherently limited by labeling cost. Large-scale unlabeled medical imaging datasets could resolve this need, but they are not readily available to the public, hindering progress for methodological research on how to harness unlabeled data for downstream segmentation most effectively. To address the need for pretraining datasets in the brain MRI domain, we propose , to the best of our knowledge the largest public pretraining dataset for brain MRI to date[  is an order of magnitude larger than the pretraining dataset used in SwinUNETR, the current state-of-the art methodology.] comprising of 44,756 volumes collected from public sources.

Using this large-scale dataset, we are proposing the AMAES (Augmented Masked Auto Encoder for 3D Segmentation) framework for pretraining modern semantic segmentation architectures. AMAES combines the successful Masked Image Modeling (MIM) task <cit.> with augmentation reversal, both of which can be viewed as special cases of denoising autencoding  <cit.>, with the goal of learning local, detailed representations that are invariant to intensity shifts between different domains. The resulting framework balances factors such as GPU memory usage, pretraining runtime, and downstream task performance. We compare AMAES against the state-of-the-art pretraining method proposed for SwinUNETR <cit.>, which combines MIM with a rotation prediction task and a contrastive learning objective. We find that in practice, solving the rotation problem is trivial while the contrastive loss adds little benefit provided the relatively small batch sizes possible in 3D imaging. Our simplified framework results in a significant speedup and memory reduction while improving downstream performance.

We evaluate our method using two backbone architectures: (i) The U-Net, as the most popular semantic segmentation model, and (ii) MedNeXt, as a recently proposed modernization. The  dataset and proposed AMAES framework for pretraining segmentation models contribute to building a foundation for the next generation of large-scale biomedical image segmentation models. 







§  DATASET


We propose   a dataset of 44.756 Brain MRI volumes collected from public sources, featuring a diverse set of acquisition parameters and patient populations. The dataset is a compilation of data from four large non-labelled datasets (ADNI <cit.>, OASIS3 <cit.>, OASIS4 <cit.>, PPMI <cit.>) and five challenge datasets (MSD <cit.>, BraTS21 <cit.>, ISLES22 <cit.>, WMH <cit.>, MSSEG1 <cit.>). The resulting dataset is highly heterogeneous and assembled to simulate the characteristics of clinical data. Overview of the dataset can be found in Table <ref>. The dataset includes data acquired at both 1.5T and 3T. PPMI data is openly available from PPMI. ADNI data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. To compile the diverse data into a suitable pretraining dataset, we transformed all volumes to the RAS coordinates system, resampled to isotropic 1mm spacing, clipped values to the 99% percentile, and z-normalized on a volume level. Afterwards, each volume is cropped to the minimum bounding box of the brain. The preprocessing pipeline is based on the  framework for medical imaging <cit.>. Code to reproduce the final version of the dataset is provided in the project repository. Usage of the dataset needs to comply with the individual legal requirements of each source.




§ METHOD



In this section we briefly describe . Our approach is highlighted in Figure <ref>. The overall goal of the framework is to provide an effective and efficient pretraining strategy for large-scale datasets, requiring a balance between memory usage, runtime and downstream performance (see Figure <ref>).



 §.§ Pretraining


  is based on a Masked Autoencoder (MAE) architecture with a light weight decoder and a specific augmentation scheme. Importantly, the architecture assumes no specific choice of encoder and can be used with any encoder that is compatible with the skip connections of the U-Net decoder.

Pre-text task. Prior work has centered around 2D natural images <cit.>, however, previous work showed that a Masked Autoencoder works well in 3D by applying it to 3D CT Brains using the SwinUNETR architecture <cit.>. The SwinUNETR pretraining objective further included both a rotation and a contrastive loss. However we find that the rotation task was near-trivial for the models to solve and thus provided no tangible utility, while the contrastive loss resulted in the memory footprint during pretraining to be doubled, without significant performance gains (see Appendix <ref>). Threfore, we reduce the pre-text task for  to masked reconstruction, and instead increase the batch size and double the number of training epochs. For each 3D input patch, the input is divided into subpatches of size 4×4×4 voxels, and each subpatch is masked with probability 0.6, following prior work <cit.>.



Decoder architecture. Since the decoder is discarded after pretraining, it is desirable to use as small a decoder as possible. This not only decreases training time and memory usage, but also forces the network to learn expressive representations in the encoder.

We find that a light weight version of the standard U-Net decoder provides a good balance between flexibility, memory efficiency and parameter count. Table <ref> shows results of pretraining with two versions of the U-Net decoder: the standard 3D U-Net decoder and one with half the number of convolutional layers (denoted light weight decoder). Interestingly, the light weight decoder not only decreases pretraining time with 20%, but also further increases finetuning performance by 1.96 dice points.

Skip connections. While skip connections are used during finetuning, our proposed encoder-decoder architecture does not include skip connections during pretraining. We found that using skip connections during pretraining tended to slightly adversely affect downstream performance, with a finetuning dice score of 56.22 with and 56.65 without, on 6-fold cross-validation on the BraTS21 dataset. While the difference is not significant, we choose to not use skip connections in the pursuit of the simplest possible setup.

Augmentations. To learn representations which are robust to changes in the input domain, we apply an extensive augmentation pipeline to the data during pretraining. The full list of augmentations include all augmentations used in nnUnet <cit.> and is given in Appendix <ref>. We want our representations to be covariant to any spatial augmentations (rotation, scaling), while we seek to learn representations that are invariant to all other augmentations, for instance additive noise. We thus only apply spatial augmentations to the reconstruction target. This is visualized in Figure <ref>. Table <ref> shows the result of applying augmentations to the pretraining data: when applying augmentations during pretraining, we see a 1.24 Dice point increase in finetuning performance, compared to no augmentations, when finetuned with only spatial augmentations.



 §.§ Finetuning


For finetuning, we transfer the backbone encoder weights from pretraining and attach a new standard U-Net decoder with skip connections between the encoder and the decoder. We only use spatial augmentations (rotation, scaling) during finetuning, since this setup performed better empirically. Upon closer inspection, the learning curves showed signs of catastrophic forgetting during the second half of training when finetuned with the full augmentation package.









§ EXPERIMENTAL DESIGN



 §.§ Datasets

We evaluate   on three different segmentation tasks covering tumor (BraTS21 <cit.>), ischemic stroke lesion (ISLES22 <cit.>) and white matter hyperintensity (WMH <cit.>). To test the models few-shot learning ability, we evaluate the models in a low resource setting: we only use n_train = 20 labeled examples for finetuning, while we evaluate the models on the remaining data. We choose n_train = 20 to realistically model what one medical expert can manually label with limited effort. Since the pretraining dataset considers each sequence individually, we evaluate the model's ability to perform downstream tasks in a single sequence setup, and thus for BraTS21 and WMH we consider only the flair modality, for ISLES22 we consider only the DWI modality.

Hyperparameters were selected on 3-fold cross-validation splits of the BraTS21 dataset. We do not tune hyperparameters individually for each fold or for each dataset. All results are averages over 6 folds. For BraTS21, we evaluate the model on a 20% holdout set. For ISLES22 we follow <cit.> and provide results from the rotating cross-validation splits. For WMH we both perform cross-validation evaluation (denoted "WMH") and test set evaluation. The test set contains data from scanners not present in the training set, which are also not present in , and thus evaluates the models ability to generalize to unseen data. We call this dataset "WMH OOD" (out-of-distribution). 



 §.§ Configurations

Our baselines, pretraining and finetuning are implemented using PyTorch, PyTorch Lightning, and the   library for medical image analysis <cit.> as a back-bone. All models are trained using 16-bit mixed-precision to limit memory footprint. Models were trained patch-based, using an input patch size of 128 × 128 × 128. Models are pretrained for 100 epochs (one epoch is one patch from each volume) and trained supervised for 2500 steps on downstream datasets. The exact hyperparameters used for each model is given in Appendix <ref>. All models are pretrained on two H100 GPUs, and fine-tuned on a single H100 GPU.



 §.§ Baselines

We evaluate  along two dimensions: (i) How does   compare to the pretrained SwinUNETR?, and (ii) What is the impact of pretraining on downstream performance?

To evaluate (i), we pretrain SwinUNETR with the exact configuration given in <cit.> on , which includes both a contrastive loss, a rotation loss and a reconstruction loss, as well as a different choice of masking ratio and mask size. To ensure a fair comparison, SwinUNETR is pretrained for 100 epochs, similar to , and is pretrained with patch size 128^3, instead of patch size 96^3 and 90 epochs. To address (ii), we apply   to a set of convolutional backbones and evaluate the difference between training from scratch and finetuning the pretrained model. The backbone models include a U-Net in two sizes: XL (90M parameters) and B. The U-Net B is very similar in size to the one used in nnUNet (22M parameters, when trained with the default setting  of 12 GB). Further, we explore using the modernized U-Net architecture MedNeXt <cit.>, which uses depth-wise seperable convolution to introduce compound scaleable medical segmentation models. All MedNeXt models are trained with kernel size 3 and do not use UpKern. The networks are finetuned and trained from scratch using the same hyperparameters. Models trained from scratch use the full augmentation pipeline.







§ RESULTS AND DISCUSSION


The results of our experiments are given in Table <ref>. Both U-Net B, U-Net XL and MedNeXt models benefit from pretraining in all scenarios. MedNeXt M does not benefit from pretraining on BraTS21 and ISLES22, however exhibits very good performance on WMH both in and out of domain. We believe this is mainly due to strong baseline performance, before pretraining is added. Note that models do not use the full augementation pipeline during finetuning, while the baselines from scratch use all augmentations. All models trained with   outperform the pretrained SwinUNETR, except for the standard U-Net on WMH OOD. Generally, model size seems to be correlated with performance, with two notable exceptions: The SwinUNETR model seems to be worse than both MedNeXt M model and the standard U-Net, both of which have less than a third of the parameters of SwinUNETR. The results on WMH OOD demonstrate that pretraining also increases performance when applied to out-of-domain data – even with respect to the pretraining domain.

Limitations. This study only considers single sequence and low resource settings. We have not applied   to the multi-sequence versions of the dataset and neither have we applied it to datasets with more datapoints than n_train = 20. How to leverage pretraining for multi-sequence tasks is a relevant avenue for future work. This study neither tests the SwinUNETR framework using different backbones and nor tests   using a SwinUNETR backbone, since it is not readily compatible with a U-Net decoder.[Since it uses non-pretrained UNETR <cit.> encoder blocks in the interface between the Swin-Transformer and the convolutional decoder.]



§ CONCLUSION

We present , a framework based on Masked Image Modeling to pretrain 3D-native segmentation models which balances memory usage, runtime, and finetuning performance. We further introduce , the largest dataset of brain MRI for pretraining to date. We pretrain   on   and evaluate the models on three challenging datasets. We compare the framework to models trained from scratch as well as the pretrained SwinUNETR model. Our results demonstrate that when  is used to pretrain U-Net models or MedNeXt models, the models outperform similar models trained from scratch, as well as SwinUNETR models, both pretrained and not in the challenging tasks evaluated.
























    





Supplementary Material



§ HYPERPARAMETERS






§ AUGMENTATIONS







§ ADNI ACKNOWLEDGEMENT


Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimag- ing Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not par- ticipate in analysis or writing of this report. A complete listing of ADNI investigators can be found 
http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ ADNI_Acknowledgement_List.pdfhere.

The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner,MD. The primary goal of ADNI has been to test whether serial magnetic reso- nance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer’s disease (AD). For up-to-date information, see <www.adni-info.org>.

Data collection and sharing for this project was funded by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Depart- ment of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.;Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & John- son Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.;Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transi- tion Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih.org). The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer’s Therapeutic Research In- stitute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.




§ CONTRASTIVE AND ROTATION LOSS CURVES






§ RESULTS WITH STANDARD DEVIATION








