Paper Title: WAS: Dataset and Methods for Artistic Text Segmentation.





 



WAS: Dataset and Methods for Artistic Text Segmentation






X. Xie et al.




Huazhong University of Science and Technology, China
{xdxie,yzli12,yangliu1213,xbai}@hust.edu.cn Adobe, USA 

{zzhang,zhawang}@adobe.com, wxiongur@gmail.com

WAS: Dataset and Methods for Artistic Text Segmentation
    Xudong Xie1 Yuzhe Li1 Yang Liu1 Zhifei Zhang2 Zhaowen Wang2 Wei Xiong2 Xiang Bai1()
    August 5, 2024
=======================================================================================



Corresponding authorfootnote


§ ABSTRACT 
  Accurate text segmentation results are crucial for text-related generative tasks, such as text image generation, text editing, text removal, and text style transfer. Recently, some scene text segmentation methods have made significant progress in segmenting regular text. However, these methods perform poorly in scenarios containing artistic text. Therefore, this paper focuses on the more challenging task of artistic text segmentation and constructs a real artistic text segmentation dataset. One challenge of the task is that the local stroke shapes of artistic text are changeable with diversity and complexity. We propose a decoder with the layer-wise momentum query to prevent the model from ignoring stroke regions of special shapes. Another challenge is the complexity of the global topological structure. We further design a skeleton-assisted head to guide the model to focus on the global structure. Additionally, to enhance the generalization performance of the text segmentation model, we propose a strategy for training data synthesis, based on the large multi-modal model and the diffusion model. Experimental results show that our proposed method and synthetic dataset can significantly enhance the performance of artistic text segmentation and achieve state-of-the-art results on other public datasets. The datasets and codes are available at: https://github.com/xdxie/WAS_WordArt-Segmentationhttps://github.com/xdxie/WAS_WordArt-Segmentation.
  





§ INTRODUCTION


Text segmentation is dedicated to finely segmenting the strokes of text from complex scene images, discriminating whether each pixel belongs to the text foreground or the background. Accurate text segmentation results are the foundation for text-related generative tasks. For instance, tasks such as text image generation<cit.>, text style transfer <cit.>, and text removal <cit.> can produce excellent and practical generative outcomes based on text masks. However, although existing models have achieved outstanding performance in regular text segmentation tasks, they are difficult to accurately segment artistic text. Artistic text features complex appearances and shapes, making it difficult to distinguish from background patterns <cit.>. Therefore, artistic text segmentation is a challenging task that has not yet been studied by the academic community.

The first problem encountered in implementing this new task is the lack of datasets. The existing text segmentation datasets ICDAR13 FST <cit.>, COCO_TS<cit.>, MLT_S <cit.>, and Total-Text <cit.> suffer from the problems of low annotation quality and insufficient quantity. Moreover, these data are scene images with regular text. Although TextSeg <cit.> provides high-quality annotated data and some artistic text images, the number of these images is still insufficient to train a high-performance artistic text segmentation model with strong generalizability. Therefore, we construct a real WordArt Segmentation dataset called WAS-R, consisting of 7100 artistic text images with word-level annotations of quadrilateral boxes, masks, and transcriptions. Additionally, to further enhance the accuracy and generalization ability of the text segmentation model, we also propose a synthetic dataset called WAS-S. Our designed synthetic pipeline utilizes the popular large multi-modal model and diffusion model to achieve realism, accuracy, and diversity in the generated images.

Besides, artistic text segmentation presents two unique challenges compared to general object segmentation and regular text segmentation. (1) The strokes of artistic text have flexible and changeable local shapes, such as slender tails or twisted ligatures. (2) The global topological structure of the artistic text is very complex, with many holes and intricate connections within the text. In contrast, the local stroke shapes and the global structure of regular text are almost invariant, and the topological structure of general objects is very simple. Therefore, the task we propose has clear academic value and practical significance. 

There are currently few specialized models for text segmentation. Recent studies either require the aid of text detection modules <cit.> or the assistance of character-level recognizers <cit.>.
Moreover, these methods have not been specifically designed for artistic text. In view of this, we propose a WordArt segmentation model WASNet. To address the first challenge, we propose a Transformer decoder with the layer-wise momentum query. The input of the self-attention module is the momentum superposition of the masked queries from the current layer and the previous layers. This operation ensures that the model does not quickly fit a restricted regular mask area when updating attention, thereby ignoring some special-shaped stroke regions, which are precisely what the earlier layers are capable of capturing. To address the second challenge, we propose a skeleton-assisted head that enables the model to output both mask predictions and skeleton predictions simultaneously, guiding the model to capture the global topological structure. It enhances the decoder's ability to perceive the overall structure of the text.

We conduct extensive experiments to verify the effectiveness of the proposed method and the synthetic dataset on the task of artistic text segmentation. We also verified the generalizability on other public datasets <cit.>.
The results achieved state-of-the-art (SOTA) performance. More importantly, the model trained on the WAS dataset can be directly tested on other datasets without the need for fine-tuning and still achieve competitive results. This opens up a new experimental paradigm for the task of text segmentation.

In summary, our contributions are four-fold:


  (1) We present a new challenging task: artistic text segmentation, and construct a real dataset to benchmark the performance of various models.   

  (2) We design a training data synthesis strategy and generate a synthetic dataset consisting of 100k image-mask pairs.

  (3) We introduce the layer-wise momentum query to handle the changeable local strokes and skeleton-assisted head to capture the complex global structure.

  (4) We achieve new SOTA results in the tasks of artistic text segmentation and scene text segmentation, and simplify the experimental paradigm for text segmentation.









§ RELATED WORK






 §.§ Text Segmentation Method


Some early text segmentation methods relied on thresholding <cit.>, low-level features <cit.>, or Markov Random Fields (MRF) <cit.> to segment foreground text from the background, but these methods could only achieve limited success in document processing. 
With the continuous advancement of deep learning, the corresponding text segmentation methods have shown great potential in complex scenes <cit.>. For instance, SMANet <cit.> employs the encoder-decoder architecture of PSPNet <cit.> and achieves a multi-scale attention module for text segmentation. PGTSNet <cit.> employs a pre-trained detector to ground out text regions before segmentation, further enhancing the accuracy of segmentation. TexRNet <cit.> incorporates character recognition and attention-based similarity checking to aid the model in segmenting text. Building on these methodologies, Yu  <cit.> developed a model featuring a lightweight detection head and a Text-Focused Module, elevating text segmentation performance in complex scenes to a new level. Nevertheless, recent high-performance text segmentation methods either utilize extra bounding box annotations and rely on text detection, or employ character-level supervision. Also, these methods lack specialized design for artistic text.




 §.§ Text Segmentation Dataset


The construction of text segmentation datasets has not received enough attention in academia. ICDAR13 <cit.> and Total-Text <cit.> provide high-quality, pixel-level annotations for text segmentation, but their quantities are very limited with only 462 and 1,555 images, respectively. To address the issue of insufficient quantity, researchers have proposed a dataset COCO-TS <cit.> (14,690 images) based on COCO-Text <cit.> for text segmentation. Similarly, MLT_S <cit.> (6,896 images) is also a large-scale text segmentation dataset based on ICDAR MLT <cit.>. Both of these datasets use automatic annotation strategies, resulting in low-quality dataset annotations. In view of these problems, Xu  <cit.> introduced a larger-scale and high-quality text segmentation dataset TextSeg (4,024 images), which includes character-level and word-level annotations of masks, bounding boxes, and transcriptions. Moreover, different from the datasets mentioned above, BTS <cit.> is a bilingual text segmentation dataset. These real datasets are all derived from natural images but lack a dedicated dataset for segmenting artistic text. Although TextSeg contains some artistic text images, the quantity is insufficient to train a robust model for artistic text segmentation.




 §.§ Segmentation Dataset Generation

The construction of synthetic segmentation datasets plays an important role in studying visual perception problems. DatasetGAN <cit.> and BigDatasetGAN <cit.> only use a small number of manually labeled samples for each category to train the decoder and generate a large amount of new data. 
Diffumask <cit.> extends the text-driven image synthesis to semantic mask generation in Stable Diffusion <cit.> to create a high-resolution and class-discriminative pixel-wise mask. 
Dataset diffusion <cit.> leverages the pre-trained diffusion model and text prompts to generate segmentation maps corresponding to synthetic images. 
DatasetDM <cit.> is a generic dataset generation model that decode the latent code of the diffusion model as accurate perception annotations. 
MosaicFusion <cit.> is a diffusion-based data augmentation method that does not require training and does not rely on any label supervision, especially for rare and new categories. 
Then SegGen <cit.> integrates Text2Mask and Mask2Img synthesis to generate training data, improving the performance of state-of-the-art segmentation models in various segmentation tasks. 
These advanced methods often fail to align artistic text with their masks in images. In this paper, we avoid having the model generate both images and mask annotations. Instead, we pre-render the text masks and use ControlNet <cit.> to generate mask-conditioned images.








§ DATASET

As artistic text in the real world is incredibly diverse, we propose two new datasets: WAS-R composed of real-world text images, and WAS-S composed of synthetic text images. These multi-purpose artistic text datasets aim to bridge the gap between artistic text segmentation and real-world applications, accommodating the rapid advances in text vision research.



 §.§ WAS-R Image Collection

The WAS-R dataset is composed of 7,100 images sourced from a variety of contexts, including posters, cards, covers, logos, goods, road signs, billboards, digital designs, and handwritten text. Among these, 4,100 images serve as the training dataset, while the remaining 3,000 images constitute the test dataset. The artistic text can be categorized into two major types according to the way capturing images. A type of artistic text image is taken by cameras from various scenes, such as signboards. The other type is directly exported from design software, such as poster files.
During data collection, we specifically balances these two types to create a diverse dataset for research and development.



 §.§ WAS-R Image Annotation

The WAS-R dataset stands out due to its comprehensive annotations, surpassing existing datasets. Specifically, WAS-R provides minimum quadrilateral detection boxes with distinct segmentation mask labels for each word. It also provides text transcription for each word mask. Moreover, we annotates the word effects such as shadow, glow, 3D, which play a crucial role in distinguishing artistic text from conventional scene text and significantly impacts text segmentation. Fig. <ref> shows examples of collected images and their annotations in WAS-R. 









 §.§ WAS-S Synthetic Dataset Construction

Fig. <ref> shows the pipeline of generating synthetic text images. The core idea is that we build a text image generation model which can generate aligned text images from text masks and input prompts. To this end, we construct the training pipeline as illustrated in Fig. <ref> (a). Specifically, we first generate diverse and informative captions from the text images in the training set of WAS-R to obtain training triplets <caption, mask, image>. Following that, we train a ControlNet <cit.> with these triplets for generating diverse images that are pixel-wisely aligned with the input text mask. During inference, as shown in Fig. <ref> (b), we first construct diverse text masks using Mask Render, then use GPT-4 to extend the texts in the mask into a scene description caption. The constructed text mask and caption are send to our trained ControlNet to generate the synthetic text image. We describe the modeling details below. 





Training Pipeline. During training, we construct the  dataset for training ControlNet <cit.> based on the training set of WAS-R. To this end, we obtain the image captions of existing training samples in WAS-R using the advanced multi-modal large language model called Monkey <cit.>. Formally, let I_t represent the real image in the training set of WAS-R, and C_t denote the prompt generated from I_t, we have C_t = Monkey(I_t).

Having obtained the <caption, text mask, text image> training triplets, we train a ControlNet that maps the input prompt and text mask to a text image. Our goal is that the outline of the artistic text in the generated image should be well aligned with the input text mask. Moreover, the contents and styles of the generated images should be diverse enough as guided by the input prompts. 



Inference Pipeline.  During inference, as illustrated in Fig. <ref>(b), we first generate synthetic text masks denoted as M_syn using our proposed Mask Render technique. Specifically, for each mask, we randomly select 1-7 phrases from the 20 newsgroups dataset <cit.> based on the word distribution of each image in the real dataset WAS-R. These phrases are consist of 1-5 consecutive words. Additionally, we apply a random rotation in the range of -30^∘≤ϕ≤ 30^∘ to each phrase. The size of each phrase is limited to match the general width of the entire image, and we position them randomly within the image boundaries. Besides, we use 250 artistic fonts.
Finally, an affine transformation is applied to each phrase to introduce skewness and distortion. 

We use GPT-4 <cit.> to generate the prompts corresponding to the synthetic text masks. Specifically, we ask GPT-4 to mimic the style of captions we generate from the training set in WAS-R, and synthesize new prompts. Next, we incorperate the text information in the synthetic mask into the generated prompt to obtain the final prompt. Formally, we have: C_syn = GPT4(C_t, C_M), where C_syn is the caption we generate and C_M is the text in the synthetic mask. Fig. <ref> shows examples of our synthetic prompts. 

Following the construction of synthetic text mask and synthetic prompts, we use the trained ControlNet to generate the final text image. We have I_syn = ControlNet_θ(C_syn, M_syn), where I_syn is the image we generate, θ denotes the trainable parameters in ControlNet. Fig. <ref> shows examples of the final synthesized <text mask, prompt, image> triplet. More details can refer to the supplementary materials.










§ METHODOLOGY

In this section, we introduce our artistic text segmentation model WASNet. We first present the overall architecture, followed by detailed descriptions of the local and global designs.





 §.§ Overall Architecture

The overall framework of WASNet is shown in <ref>. We take an excellent semantic segmentation model Mask2Former <cit.> as the meta-architecture. It is a mask classification architecture that directly predicts multiple binary masks and corresponding category labels, instead of performing per-pixel classification.
We add a skeleton-assisted head and improve the Transformer decoder with a mechanism of layer-wise momentum query. The backbone extracts low-resolution features from an image. The pixel decoder upsamples the image features and generates a feature
pyramid. The multi-scale features are fed into the Transformer decoder, with each resolution corresponding to each layer's input of the decoder. Besides, each layer of the Transformer decoder also receives the mask prediction and query generated from the previous layer as input. Finally, the mask head and the skeleton head generate binary mask and skeleton predictions respectively, by decoding the per-pixel embeddings from the pixel decoder and object queries from the Transformer decoder. The ground truth of the skeleton is obtained by thinning the binary mask labels through a skeleton extraction algorithm <cit.>.





 §.§ Transformer Decoder with Layer-wise Momentum Query


Artistic text segmentation faces the challenge of the local stroke shapes being flexible and changeable. Due to designers using hundreds of different artistic fonts and applying various text effects, the local strokes of the same character can differ significantly. This results in some slender strokes spanning across other areas, as well as twisted ligatures leading to complex text edges.  In contrast, normal scene text typically utilizes regular printed fonts without special designs, and the stroke shapes are almost invariant. Therefore, it is necessary for the decoder to pay attention to these special local strokes.

First, we use the masked attention mechanism <cit.>, constraining cross-attention
to within the local text mask region for each query, instead of attending to the full feature map. This mechanism can be expressed as:

    𝐌𝐀_l=softmax(𝐌_l + 𝐐_l𝐈_l^T) 𝐈_l,

where l is the layer index, and 𝐐_l is the input queries. 𝐈_l ∈ℝ^H_l W_l × C is the image feature input to the l-th layer, which comes from the feature pyramid of the pixel decoder. H_l and W_l indicate the spatial resolution of the image feature and C is the feature dimension. 𝐌_l is transformed from the binary mask output of the previous layer, with the value of the text region being 0 and the value of the non-text region being -∞ <cit.>.
𝐌𝐀_l is the output of the masked attention module. We have omitted the residual connection and normalization here. 

Furthermore, since the masks predicted by each layer are different, the previous layers yield coarse masks that may include special-shaped stroke regions. However, the subsequent layers are inclined to predict more precise regions of regular strokes, overlooking those local special regions. Therefore, in order to prevent the model's attention from being quickly confined to regular regions, we design a mechanism of Layer-wise Momentum Query (LMQ). The momentum superposition of the masked queries from the current and previous layers is input to the self-attention module before the module gathers contextual information.  <ref> illustrates this mechanism.

    𝐌𝐐_l+1=α𝐌𝐐_l+(1-α)𝐌𝐀_l,

where α∈[0,1) is a momentum coefficient. 𝐌𝐐 is the momentum query that is input to the self-attention module. We ultimately use this decoder in WASNet with layer-wise momentum query.



 §.§ Skeleton-Assisted Head

Different from regular text and general objects, the global topological structure of artistic text is very complex, and there are many holes and intricate connections inside. This presents new challenges for the segmentation task. The model needs to capture the global structure of the text object rather than just a region. Inspired by DeepSkeleton <cit.> and DeepFlux <cit.>, 
we found that the skeleton is an effective representation to describe the shape and topology of text because it can extract the central axis of the object. Therefore, we use skeletons to assist text segmentation.

As shown in <ref>, we add a skeleton-assisted head to WASNet, enabling the model to simultaneously predict the mask and the skeleton, thus endowing it with the capability to perceive the global topological structure. Since the binary mask is a finely annotated label for semantic segmentation, the ground truth for the skeleton can be obtained by processing the mask with the classic Zhang-Suen <cit.> skeleton extraction algorithm. The algorithm progressively removes pixels that satisfy certain template structural conditions through an iterative process, until no more pixels meeting the conditions are deleted.

We use the binary cross-entropy loss and the dice loss <cit.> for our skeleton loss and mask loss:

    ℒ_skeleton=ℒ_mask= λ_ceℒ_ce + λ_diceℒ_dice.

We set λ_ce=λ_dice=5. The final loss is the combination of skeleton loss, mask loss, and classification loss:

    ℒ_final=ℒ_skeleton+ℒ_mask+ λ_clsℒ_cls,

where λ_cls=2 for
predictions matched with labels and 0.1 for predictions that have not been matched with any labels.

During the inference phase, it is unnecessary to output the predictions of the skeleton. Therefore, we follow the post-processing method in <cit.> to obtain the final output of text semantic segmentation.







§ EXPERIMENTS




 §.§ Implementation Details

Our experiments are mainly based on the MMSegmentation <cit.> toolbox. The overall hyperparameter configuration is the same as <cit.>. The pixel decoder is a multi-scale deformable attention Transformer <cit.> with 6 layers. The Transformer decoder consists of 9 layers, each with an auxiliary loss.
We use the AdamW <cit.> optimizer and the poly <cit.> learning rate schedule with an initial learning rate of 10^-4 and a weight decay of 0.05. The data augmentation strategies include random scale jittering, random color jittering, random cropping as well as random horizontal flipping. We use a crop size of 512×512 and a batch size of 16. The models are trained with 8 RTX4090 GPUs. 
If the model is only trained on the real dataset, we set the number of iterations to 100k. If the model needs to be pre-trained on the synthetic dataset WAS-S, we first pre-train the model for 50k iterations and then fine-tune it on the real dataset for 50k iterations.
For the momentum coefficient α in <ref>, we set α=0.8 by default.
Following the previous text segmentation methods <cit.>, we use foreground (text) Intersection-over-Union (fgIoU) as the major metric and F-score measurement on foreground pixels as the auxiliary metric. 






 §.§ Results of Artistic Text Segmentation



To verify the superiority of our method in the task of artistic text segmentation, we trained several representative models on our WAS-R dataset, including six semantic segmentation models and two text segmentation models. We use the officially released code for TexRNet <cit.>, DDP <cit.>, and TextFormer <cit.>, and the code reproduced by MMSegmentation <cit.> for other models. For a fair comparison, we did not apply the character-level glyph discriminator for TexRNet.

The experimental results in <ref> indicate that our WASNet outperforms all of these advanced models. Moreover, when we train the baseline models and WASNet with the synthetic dataset WAS-S, their performance can be further improved. Our final results have achieved a significant SOTA performance.




 §.§ Results of Scene Text Segmentation

To further verify the generalizability of WASNet, we also conducted experiments on three publicly available scene text segmentation datasets <cit.>, as shown in <ref>. We can draw the same conclusion as in <ref> regarding the effectiveness of WASNet and our synthetic dataset. It is worth mentioning that character-level annotations were used to train TexRNet on TextSeg. Extra bounding box labels were used to train the text detection module of TextFormer on all three datasets. However, we only use binary mask labels of the full images. Despite this, we still achieved competitive or state-of-the-art results. Due to the highly inaccurate annotation quality of COCO_TS <cit.> and the fact that Total-Text <cit.> contains only 300 test images, the conclusions drawn from the evaluation results of the models on these two datasets may be inconsistent.








Furthermore, we directly evaluate the performance of WASNet on the three datasets using the model trained on the synthetic and real WAS datasets, as shown in the last row of <ref>. Note that the results in this row have not been fine-tuned on specific datasets but are still competitive. Therefore, to simplify the experimental paradigm and evaluation process of text segmentation models, we encourage researchers to train on WAS and test directly on other datasets.



[t]0.45
captypetable



Ablation study on our proposed modules and datasets.


2*Methods     2cWAS 

    fgIoU     F-score 



Baseline <cit.>     80.21     0.890



+ LMQ     80.95     0.898 

+ Skeleton      82.11     0.901

+ WAS-S     84.18     0.913 





[t]0.48
captypetable


Ablation study on datasets.


2*Dataset     2cWAS 

    fgIoU     F-score 



WAS-S     84.18     0.913 



5w images    83.25     0.906 

20w images    84.01     0.912

BLIP2 <cit.>      83.07     0.906

1000 fonts     82.35     0.902 














 §.§ Ablation Study

In this section, we conduct the ablation study on the artistic text segmentation dataset WAS.
We first validate the effectiveness of our proposed modules and the synthetic dataset. As shown in <ref>, as we gradually apply the design of LMQ and the skeleton to the baseline, the performance of WASNet is incrementally improved. Pre-training WASNet on WAS-S can continue to enhance the performance of artistic text segmentation.



Therefore, the synthetic dataset is an important contribution of this paper. We conduct ablation experiments regarding some synthetic details of the dataset in <ref>. It is crucial to control the number of synthesized mask-image pairs. Less data will weaken performance, but more data will lead to a performance plateau.
The reasons will be analyzed in <ref>.
We also use other large multi-modal model BLIP2 <cit.> to generate the image captions, but the performance is limited. This is because the overall performance of BLIP2 is inferior to Monkey <cit.> we used. Besides, we applied more fonts to generate masks, but the performance actually decreased. The dataset of 1000 fonts includes a large number of regular fonts, which reduces the learning difficulty of the dataset.

Furthermore, we explored the impact of different momentum coefficient values α on the performance of WASNet in <ref> and found that the approximate optimal value is 0.8. A coefficient that is too large can cause the model to be overly influenced by the coarse predictions from earlier layers. A coefficient that is too small diminishes the positive effect of momentum queries.































 §.§ Further Analysis

To further verify the effectiveness of our proposed WASNet, we visualize the inference outputs of our baseline model Mask2Former <cit.> and WASNet in <ref>. According to <ref> (a), it is evident that WASNet can capture special-shaped stroke regions such as slender tails or twisted ligatures. This is attributed to our  Transformer decoder with the layer-wise momentum query. Additionally, according to <ref> (b), WASNet exhibits good scale adaptability. It can achieve fine results for both large-scale and small-scale text with complex structures. This is because the skeleton-assisted head can obtain the global topological structure of the text through the thinning operation, guiding fine segmentation. 

Once accurate text stroke masks are obtained, downstream text-related generative tasks can demonstrate excellent results. The application effects of text removal, text background replacement, and text style transfer are shown in the supplementary materials.




 §.§ Limitation


Although the proposed synthetic dataset can improve the performance of text segmentation models, the enhancement is limited and does not significantly increase. Even when we further increased the amount of synthetic data, the performance remained unchanged. This could be caused by the bottleneck encountered in the diversity and realism of the synthetic images. In the future, we are considering designing more advanced generative models.



§ CONCLUSION

The paper focuses on a new challenging task of artistic text segmentation. We propose a real dataset for this task to train the models and benchmark the performance. We also construct a synthetic dataset to further enhance the accuracy and generalization ability. In order to meet the challenges of this task, we introduce the layer-wise momentum query to handle the changeable local strokes and the skeleton-assisted head to capture the complex global structure. Experimental results have demonstrated the effectiveness and superiority of our method in the tasks of artistic text segmentation and scene text segmentation. We hope that more researchers can focus on this task in the future and that the dataset we propose can change the experimental paradigm of text segmentation.





§ ACKNOWLEDGEMENTS

This work was supported by the National Science Fund for Distinguished Young Scholars of China (Grant No.62225603).






splncs04




§ MORE DETAILS ON SYNTHETIC DATASET CONSTRUCTION

As stated in the paper, we first construct the training pipeline for a text image generation model, learning to generate text images spatially aligned with text masks. Then we construct an inference pipeline to input new masks and prompts into the trained generation model, generating new text images. Here we add the details of prompt generation in the training and inference pipelines.


 §.§ Training Pipeline

To train the text image generation model such as ControlNet <cit.>, it is necessary to obtain training data of <caption, text mask, text image> triplets. Text masks and text images are from our proposed real dataset. Captions should be detailed descriptions of the text images. To this end, we utilize a large multi-modal model, Monkey <cit.>, to caption the images. Monkey is an open-source model and can handle vision-language tasks with high-resolution input and detailed scene understanding. It performs well on Image Captioning and various Visual Question Answering (VQA) tasks. Therefore, we feed a text image and a prompt “generate the detailed caption in English” to Monkey and let it output a detailed description. The examples of the generated captions are shown in <ref>. We found that, in many cases, Monkey is able to recognize and describe the text in images. To ensure the accuracy of the descriptions and to highlight the importance of the text, we add a sentence after each caption: This image contains the text “text in the image”.







 §.§ Inference Pipeline

During the inference phase, we first need to produce new binary masks of text through the Mask Render introduced in the paper. Moreover, it is crucial to generate new prompts that describe more complex scenes. Combining the masks with rich descriptions of scenes, the trained model can generate new and realistic text images. We use GPT-4 <cit.> to generate the prompts. To ensure that the new prompts and the training prompts are in the same domain, and avoid domain gaps in the images generated by the model, we first provide GPT-4 with 50 caption examples produced by Monkey. Then we ask GPT-4 to mimic the style of these captions and synthesize new prompts. The instruction is Please follow the above caption examples and generate a similar caption, which must contain some double-quoted spaces “ ”. Next, we insert the text corresponding to each new mask into the quotation marks in the new prompt, forming the final prompt. The generated <prompt, mask, image> triplet is shown in <ref>.



§ APPLICATIONS



 §.§ Text Removal

Text removal refers to the process of erasing or deleting text regions from an image. The finer the text mask, the better the erasing performance, as it preserves more background pixels. Therefore, stroke-level text segmentation can greatly benefit this task. Text removal is essentially an image inpainting task, so LaMa <cit.> is employed and the results are shown in <ref>. 






 §.§ Text Background Replacement

Once we have obtained the fine mask of the text, we can freely replace the background of the image, embedding the text into various scenes. We use ControlNet <cit.> to replace the background and <ref> presents the results.





 §.§ Text Style Transfer

Text style transfer is a task that renders text in natural images into artistic text according to a style reference image while keeping the text content unchanged. It usually relies on accurate text masks. We use Intelligent Typography <cit.> as the style transfer model and input the predicted text masks to it. The stylized text is shown in the last column of <ref>.


 Paper Title: Translating Imaging to Genomics: Leveraging Transformers for Predictive Modeling.


 Enabling Next-Generation V2X Perception: 
 Wireless Rigid Body Localization and Tracking

    Niclas Führling0000-0003-1942-8691, Hyeon Seok Rou0000-0003-3483-7629, Graduate Student Members, IEEE,

Giuseppe Thadeu Freitas de Abreu0000-0002-5018-8174, David González G.0000-0003-2090-8481, Senior Members, IEEE, and Osvaldo Gonsa0000-0001-5452-8159




N. Führling, H. S. Rou and G. T. F. Abreu are with the School of Computer Science and Engineering, Constructor University, Campus Ring 1, 28759, Bremen, Germany (emails: [nfuehrling, hrou, gabreu]@constructor.university).

D. González G. and O. Gonsa are with the Wireless Communications Technologies Group, Continental Automotive Technologies GmbH, Guerickestrasse 7, 60488, Frankfurt am Main, Germany (emails: david.gonzalez.g@ieee.org, osvaldo.gonsa@continental-corporation.com).

    August 5, 2024
=========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
In this study, we present a novel approach for predicting genomic information from medical imaging modalities using a transformer-based model. We aim to bridge the gap between imaging and genomics data by leveraging transformer networks, allowing for accurate genomic profile predictions from CT/MRI images. Presently most studies rely on the use of whole slide images (WSI) for the association, which are obtained via invasive methodologies. We propose using only available CT/MRI images to predict genomic sequences. Our transformer based approach is able to efficiently generate associations between multiple sequences based on CT/MRI images alone. This work paves the way for the use of non-invasive imaging modalities for precise and personalized healthcare, allowing for a better understanding of diseases and treatment.

    


§ INTRODUCTION



For diseases like cancer, imaging modalities like MRI and CT are the preferred tools for diagnosing and screening patients. However, in the context of precision oncology,  treatment planning for each patient depends upon genomic markers. Gene expressions undergo drastic changes due to cancer <cit.>. Certain specific biomarkers determine the course of action for a particular cancer; for example, mutations of the Epidermal Growth Factor Receptor (EGFR) and  Kirsten Rat Sarcoma viral oncogene (KRAS)  are associated with the occurrence of Non-Small Cell Lung Cancer (NSCLC)   <cit.>, Hypermethylation of the O(6)-Methylguanine-DNA-Methyltransferase (MGMT) gene status determines the response of a patient to radiation therapy suffering from glioblastoma <cit.>. Genetic links can also determine the behavior and the tumor subtype. In 2021, WHO  
 proposed an improved classification criterion for central nervous system tumors, incorporating genetic information <cit.>. 

Therefore, the determination of gene expression and status is of paramount importance for oncology treatment selection. However, genomic testing is expensive and may not be always available. Numerous studies have been done that establish the link between genomic expressions and imaging features derived from pathological images <cit.>, CT <cit.> and MRI <cit.>. This allows us to predict the genomic expressions using the imaging modality alone. Deep Learning (DL) for predicting genomic expressions has also seen a considerable rise <cit.> with CNN-based methods outperforming all previous machine learning-based models. However, most of these studies rely on whole slide images (WSI) as the imaging modality <cit.> or to predict the mutation status <cit.>, not the complete RNA-Seq profile.

To the best of our knowledge, this work is the first to explore the prediction of complete genomic expression through the use of most commonly available imaging modalities like CT/MRI.








§ RELATED WORK



Several studies have demonstrated the effectiveness of DL-based methods for predicting RNA-Seq from imaging data. These studies rely on pathological WSI. He  <cit.>  used a DL-based method to predict 100 out of 30,612 gene expressions from breast cancer WSI. Zeng  <cit.> proposed a Hist2ST model comprising of Convmixer, Transformer, and GNN modules and outperformed state-of-the-art models by 10% for the HER2+ and cSCC datasets. Wang  <cit.> used a ResNet-based CNN for predicting gBRAC gene mutation in breast cancer patients. Saldanha  <cit.> integrated a self-supervised model <cit.> with attention
to make patient-level predictions for a pan-cancer dataset.

All the studies above utilize WSI; researchers have also used non-invasive modalities like CT/MRI for mutation status prediction. <cit.> used DL-based methods for predicting MGMT methylation status. Wei  <cit.> proposed a multi-modal model to predict IDH mutation in glioma patients, surpassing all state-of-the-art baselines. For NSCLC lung cancer,<cit.> 
proposed using associations between CT and genomic mutation to predict EGFR and KRAS mutation status. Although significant efforts have been devoted to integrating RNA-Seq with whole-slide imaging (WSI), there has been limited focus on predicting complete RNA-Seq from non-invasive methods. This work entails predicting entire subsequences solely based on CT/MRI imaging modalities.
















§ METHODOLOGY



 §.§ Datasets

The imaging data for the study is obtained from TCIA, while the genomic data is from the TCGA portal. We have used the TCGA-GBM <cit.> and TCGA-LGG <cit.> for brain cancer, TCGA-LUAD <cit.> for lung cancer and TCGA-BRCA <cit.> for breast cancer, respectively. We limited our study to public datasets for which CT/MRI, histopathological, and genomic data are available.

The imaging and genomic data are already anonymized, along with all the ethical clearances. For the pre-processing, CT and MRI scan slices were resampled with a slice thickness of 1 mm^3 and standard normalization operation was performed. Skull stripping was performed across all MRI scans. Also, we select only those scans that contain tumors. Following <cit.>, genes with a median of zero are removed, and only 31,793 genes remain. After pre-processing, we have a cohort of 550 patients with genomic, histopathological, and CT/MRI data available.



 §.§ Prediction Modelling


The prediction module consists of a transformer encoder followed by a prediction head for genomic prediction, as shown in Fig <ref>. We use TransUNet <cit.> for the encoding process, as it captures local and global features via UNet <cit.> and the Transformer model <cit.>. The encoder learns the embedding for every patient. The learned embeddings are passed to the prediction head comprising of a dropout layer and 1D convolution layer similar to <cit.>. Mean Squared Error (MSE) is used as the loss function here. 


§ RESULTS


We evaluate our model based on two criteria similar to <cit.> and compare the results with He2RNA<cit.> which uses WSI. Firstly, we find Pearson correlation coefficient <cit.> to determine the correlation, as shown in Fig <ref>. We also assess the number of genes predicted correctly using the Holm–Šidák (HS) correction discussed below.



For TCGA-GBM, we found 1731 genes with significant associations using our model, while using the He2RNA<cit.>, 3241 genes have strong correlations with WSI features. For TCGA-LGG, we found 1632 genes with significant associations, while the HE2RNA model reported 3232 genes. TCGA-BRCA 838 genes are predicted for the MRI with a statistically significant correlation; simultaneously, an average of 1627 genes are predicted with a statistically significant correlation using WSI features. For TCGA-LUAD, we saw 656 genes have significant associations, while the WSI-based model has 1539 genes with significant associations. Regarding the status of genes associated with particular forms of cancer, we found strong associations for BRAF, ALK, and KRAS for lung cancer, IDH for glioblastoma, and CHEK for breast cancer. 

The above results present strong proof of the association between imaging modalities such as CT/MRI and RNA-Seq and are encouraging enough to warrant further exploration and refinement. 





§ CONCLUSION


In this study, we present a simple method to leverage transformers to establish a relationship between non-invasive modalities like CT and MRI and molecular characteristics such as RNA-Seq. The results show strong associations between multiple gene sequences and imaging features, suggesting the need for deeper exploration in this domain.

    ieeenat_fullname
    





 Paper Title: Expressive Whole-Body 3D Gaussian Avatar.





 










Moon et al.




^1DGIST  ^2Codec Avatars Lab, Meta 

mks0601@dgist.ac.kr  {tshiratori,shunsukesaito}@meta.com 

<https://mks0601.github.io/ExAvatar>


Expressive Whole-Body 3D Gaussian Avatar
    Gyeongsik Moon1,20000-0003-0610-7936 Takaaki Shiratori20000-0002-1012-415X Shunsuke Saito20000-0003-2053-3472
    
=================================================================================================================



§ ABSTRACT 
Facial expression and hand motions are necessary to express our emotions and interact with the world.
Nevertheless, most of the 3D human avatars modeled from a casually captured video only support body motions without facial expressions and hand motions.
In this work, we present ExAvatar, an expressive whole-body 3D human avatar learned from a short monocular video.
We design ExAvatar as a combination of the whole-body parametric mesh model (SMPL-X) and 3D Gaussian Splatting (3DGS).
The main challenges are 1) a limited diversity of facial expressions and poses in the video and 2) the absence of 3D observations, such as 3D scans and RGBD images.
The limited diversity in the video makes animations with novel facial expressions and poses non-trivial.
In addition, the absence of 3D observations could cause significant ambiguity in human parts that are not observed in the video, which can result in noticeable artifacts under novel motions.
To address them, we introduce our hybrid representation of the mesh and 3D Gaussians.
Our hybrid representation treats each 3D Gaussian as a vertex on the surface with pre-defined connectivity information (i.e., triangle faces) between them following the mesh topology of SMPL-X.
It makes our ExAvatar animatable with novel facial expressions by driven by the facial expression space of SMPL-X.
In addition, by using connectivity-based regularizers, we significantly reduce artifacts in novel facial expressions and poses.




§ INTRODUCTION


Humans use all facial expressions, body motions, and hand motions to express our emotions and intentions, and interact with other people and objects.
In particular, facial expressions and hand gestures are one of the most powerful channels for non-verbal communication, and hand motions are necessary to interact with diverse types of objects.
Modeling the facial expression, body motion, and hand motion altogether is extremely challenging.
Several whole-body 3D human geometry models have been introduced <cit.>.
Among them, SMPL-X <cit.> is the most widely used one, which motivated a number of 3D whole-body pose estimation methods <cit.> and benchmarks <cit.>.



To represent 3D humans beyond the minimally clothed parametric models, personalized 3D human avatars have been recently studied.
The 3D human avatar is a representation that combines 3D geometry and the appearance of a certain person, which can be animated and rendered with novel poses.
However, most of existing 3D human avatars <cit.> modeled from a casually captured video only support body motions without facial expressions and hand motions.
Their avatars bake facial expressions and hand poses, and animating them is not possible.
A recent work <cit.> introduced a whole-body avatar that supports animation with facial expressions, and body and hand poses; however, it requires 3D observations, such as 3D scans or RGBD images with highly accurate SMPL-X registrations, with diverse poses and facial expressions.
Such an assumption does not hold for the majority of casually captured videos in daily life.





We present ExAvatar, an expressive whole-body 3D human avatar that can be made from a short monocular video.
ExAvatar is designed as a combination of the whole-body 3D parametric model (SMPL-X) <cit.> and 3D Gaussian Splatting (3DGS) <cit.>.
It utilizes the whole-body drivability of SMPL-X and the photorealistic and efficient rendering capability of 3DGS.
After the training, it is animatable with novel facial expression code and 3D pose of SMPL-X, as shown in Fig. <ref>.
Despite its desired properties, modeling ExAvatar is an non-trivial task with the following two challenges: 1) a limited diversity of facial expressions and poses in the video and 2) the absence of 3D observations, such as 3D scans and RGBD videos.
The limited diversity in the video makes a drivability with novel facial expressions and poses non-trivial.
In addition, the absence of 3D observations creates ambiguity in the occluded human parts, exhibiting noticeable artifacts in novel facial expressions and poses.



To address them, we propose a novel hybrid representation of the surface mesh and 3D Gaussians in ExAvatar.
Our hybrid representation treats each 3D Gaussian as a vertex on the surface, where the vertices have pre-defined connectivity  (i.e., triangle faces) between them following the mesh topology of SMPL-X.
Existing volumetric avatars <cit.> do not have the connectivity by the definition.
Also, previous 3DGS-based <cit.> works consider a set of 3D Gaussian points as a point cloud without considering the connectivity between them.


Using our hybrid representation, our ExAvatar becomes fully compatible with the facial expression space of SMPL-X.
Therefore, it can be driven with any facial expression code of SMPL-X even from a short monocular video without diverse facial expressions.
As our 3D Gaussians share the exactly same mesh topology with SMPL-X, we simply add the vertex offsets to our 3D Gaussian points to move them according to the facial expression as in FLAME <cit.> and SMPL-X <cit.>.
Hence, unlike previous works <cit.>, our drivability of the facial expression is not strictly limited by the number of training frames (e.g., 30 seconds of a short video).


Another benefit is that we can significantly reduce artifacts in novel facial expressions and poses using connectivity-based regularizers.
As the pose diversity in the training set is very limited, there could be human parts that are not observed at all in the video.
Without 3D observations, the ambiguous human parts could suffer from artifacts in novel poses.
While several point-based regularizers (e.g., L2 regularization of the underlying SMPL/SMPL-X template mesh) have been proposed <cit.>, they do not consider connectivity between vertices.
Such a lack of connectivity could introduce floating 3D Gaussians.
By considering the connectivity, we can naturally enforce local similarity, significantly reducing artifacts.


Throughout our experiments, 
our method substantially outperforms all previous 3D human avatars in various benchmarks.
Our contributions can be summarized as follows.


  * We present ExAvatar, an expressive whole-body 3D human avatar that can be made from a short monocular video without requiring 3D observations.

  * We propose a hybrid representation of the surface mesh and 3D Gaussians.
It allows ExAvatar to be animated with any novel facial expression code of SMPL-X even from a short monocular video without diverse facial expressions.

  * Using connectivity information between 3D Gaussians, we significantly reduce artifacts, especially in novel facial expressions and poses.








§ RELATED WORKS


3D human avatars.
Various 3D representations are used for modeling 3D human avatars.
Alldieck  <cit.> extended SMPL mesh with per-vertex offsets.
Bagautdinov  <cit.> achieved high-fidelity results using conditional variational autoencoder, which can be animated with incomplete driving signals.
Remelli  <cit.> propose to use texel-aligned features, a localized representation.
Motivated by neural radiance fields <cit.>, many volumetric and implicit representation-based avatars have been introduced.
Peng  <cit.> created a 3D human avatar from a capture studio, which provides accurate 3D pose and multi-view images for supervision.
Kwon  <cit.> improved the previous works by utilizing vertex-aligned features.
Shen  <cit.> created a whole-body 3D avatar, which supports whole-body animation with facial expression, from their capture studio dataset.
In contrast to the above works that make 3D avatars from capture studios, recent works focus on making 3D avatars from a short monocular video without requiring 3D observations, such as 3D scans, RGBD images, or multi-view images.
Jiang  <cit.> introduced a dataset and method for making a 3D human avatar from a short monocular video taken from in-the-wild environments.
Guo  <cit.> proposed a system that can decompose a scene and human with self-supervised learning.
Jiang  <cit.> introduced a system that can make a 3D human within several minutes.
Recently introduced 3DGS <cit.>, which achieves both powerful and efficient rendering capability, motivated several 3DGS-based avatars <cit.>.
Kocabas  <cit.> use the triplane for creating 3D avatar.
Hu  <cit.> introduced a robust system that takes a positional map of a posed SMPL mesh.
Moon  <cit.> presented universal hand model (UHM) to create authentic hand avatars from a phone scan.
Chen  <cit.> extended UHM of Moon  <cit.> for the relightability.


Except for a few works <cit.>, most of the above works only support body motions without hand motions and facial expressions.
X-Avatar <cit.> supports whole-body animation including facial expressions; however, it has two limitations.
First, it requires diverse facial expressions with accurate 3D geometry registration in videos to create avatars.
This is because they cannot directly utilize the facial expression space of FLAME <cit.>/SMPL-X <cit.>.
They need to transform the mesh-based facial expression space of FLAME <cit.>/SMPL-X <cit.> to their implicit representation using learnable modules.
To train the transformation module, they need training data with sufficiently diverse facial expressions and accurate 3D geometry registrations.
Second, it requires 3D observations, such as 3D scans or RGBD images with accurate SMPL-X registrations, for the training, hard to obtain from in-the-wild environments.
Due to the above two reasons, X-Avatar <cit.> is hard to apply to practical settings, such as short monocular videos.
Liu  <cit.> proposed another whole-body 3D avatar; however, their avatar is not animated with novel facial expressions.


Whole-body 3D human modeling and perception.
Modeling face, body, and hands at the same time is an extremely challenging problem as each human part has its own different characteristics.
Several whole-body 3D human models have been introduced <cit.>, which model 3D geometry of minimally clothed humans.
They are parametric models, parameterized by 3D poses, facial expression code, and shape parameter.
Among them, SMPL-X <cit.> is the most widely used one due to its completeness.
Motivated by the optimization baseline <cit.> and benchmarks <cit.>, a number of 3D whole-body pose estimation methods <cit.> have been introduced.






§ EXAVATAR




 §.§ Accurate co-registration of SMPL-X
 
We assume videos, which usually consist of 30 seconds of frames, are taken from an in-the-wild environment.
The video is from a single person with a natural backgrounds.
Before training our ExAvatar, we first preprocess the video.
Following previous works <cit.>, we first run an off-the-shelf SMPL-X regressor <cit.> and 2D pose estimator <cit.> to all frames.
Then, we additionally fit the regressed SMPL-X parameters (i.e., 3D poses θ∈ℝ^55 × 3, shape parameter β∈ℝ^100, and facial expression code ψ∈ℝ^50) and 3D translation 𝐭 to the estimated 2D pose of each frame.
The shape parameter is shared across all frames as all frames are from the same person.


r0.45[t]





    < g r a p h i c s >






Without the face offset Δ𝐕_face, the final 3D geometry of the avatar becomes totally inauthentic and inaccurate.
For each setting, normals of 3D Gaussian points and colors are used for the rendering.



One challenge during registering SMPL-X parameters to a video is accurate co-registration of body, hands, and face, unique challenges of the whole-body avatar.
The registration of hands and face can be negatively affected by a limited expressiveness of SMPL-X and registration accuracy of the body, which can limit the co-registration accuracy.
To achieve the accurate co-registration of body, hands, and face, we introduce two optimizable offsets initialized with zero and shared across all frames.
Both offsets are identity (ID)-dependent offsets and are not dependent on the poses and facial expressions.
Hence, they are added to the T-pose template mesh of SMPL-X before performing the linear blend skinning (LBS).


First, we introduce joint offset Δ𝐉, added to the joints in the T-pose space of SMPL-X.
The joint offset Δ𝐉, which affect both 3D skeleton and surface, are especially helpful to fit hands more perfectly as the shape parameter of SMPL-X has limited coverage of 3D hand skeleton, as shown in Fig. <ref> left.
Second, we introduce face offset Δ𝐕_face, per-vertex offset of the face region of SMPL-X, added to the face vertex in the T-pose space of SMPL-X.
To optimize the face offset Δ𝐕_face, we first fit the 3D face-only model (i.e., FLAME <cit.>) to 2D poses and images of the face by running DECA <cit.> and further optimizing it to 2D poses.
Then, we optimize the face offset by making a summation of the face offset and 3D face vertices of SMPL-X close to fitted FLAME vertices.
The optimization is straightforward as the face region of SMPL-X has exactly the same topology as that of FLAME.
The rationale is that 1) the face-only model has higher expressiveness in its shape space than the whole-body model and 2) the registration of the face-only model is not affected by the body registration.
Fig. <ref> right and  <ref> show the effectiveness of our face offsets.
Such a special treatment in the registration stage is greatly helpful for the final 3D avatar, not considered in previous whole-body avatars <cit.>.
Please refer to the supplementary material about the details of the fitting.





 §.§ Architecture
 
Fig. <ref> shows the architecture of ExAvatar.
We model ExAvatar on top of a canonical 3D human mesh, denoted by 𝐕̅∈ℝ^N × 3, where it has N=167K upsampled vertices and 335K upsampled triangle faces.
To obtain it, we first pass the optimized SMPL-X shape parameter β, joint offsets Δ𝐉, and face offsets Δ𝐕_face from Sec. <ref>, and a pre-defined neural pose (i.e., 大-pose) to the SMPL-X layer.
Then, we upsample it with the subdivision function of PyTorch3D <cit.>, which can upsample other 3D assets, such as facial expression blend shapes, in a consistent way.



Per-vertex Gaussian assets regression.
We initialize a learnable triplane <cit.> 𝐓∈ℝ^3 × C × H × W with zero, where C=32, H=128, and W=128 represent channel dimension, height, and width of the triplane, respectively.
Then, we prepare a positional encoding mesh 𝐏̅∈ℝ^N × 3 with a pre-defined neutral pose (i.e., 大-pose) and zero shape parameter.
We upsample the positional encoding mesh with the above subdivision function, which produces the same mesh topology as the canonical mesh 𝐕̅.
We extract the per-vertex feature from the triplane by orthogonally projecting 𝐏̅ to each plane and performing the bilinear interpolation.
The triplane is useful as it naturally enforces similarity between close vertices.
In practice, we construct another triplane dedicated to the face, as the face requires detailed geometry and appearance modeling with a small physical size.
The reason for not using the canonical mesh 𝐕̅ for the feature extraction is that it keeps changing during the training as we further optimize the shape parameter β and the joint offset Δ𝐉 during the training.
If the position of a certain vertex changes, the extracted triplane feature of that vertex could be one that was from other vertices, which can make the training unstable.





The interpolated features from the triplane are concatenated, denoted by 𝐅∈ℝ^N × 96.
We pass 𝐅 to two multi-layer perceptrons (MLPs), which regress 1) 3D offset Δ𝐕_tri∈ℝ^N × 3 and scale 𝐒_tri∈ℝ^N × 1 and 2) RGB values C_tri∈ℝ^N × 3 for the 3DGS, respectively.
The MLPs are shared across all vertices.
Motivated by Hu  <cit.>, for better generalization to novel viewpoints, we limit all Gaussian assets to be isotropic by limiting a degree of freedom of the scale to 1 and setting the rotation and opacity to identity and one, respectively.
Please refer to the supplementary material for the detailed architecture of the MLPs.
The regressed Gaussian assets (i.e., 3D offset, scale, and RGB values) are solely from the triplane, shared across all frames.
Hence, they represent identity (ID)-dependent and environment (e.g., lighting)-dependent assets without pose dependency as ID and environment are fixed in the input video, while pose changes for each frame.


To additionally model pose-dependent deformations, we employ two additional MLPs.
The first MLP takes 𝐅 and 3D poses θ without the root pose and outputs 3D vertex offset Δ𝐕_pose∈ℝ^N × 3 and scale offset Δ𝐒_pose∈ℝ^N × 1.
The second MLP takes 𝐅, 3D poses θ without the root pose, and the normal vector of each vertex and outputs RGB offset Δ𝐂_pose∈ℝ^N × 3.
The additional normal vector can 1) provide the view-dependent shading information to the network and 2) be useful to disentangle geometry and appearances <cit.>.
Thanks to our hybrid representation, we can easily obtain the per-vertex normal vector by averaging normals of triangle faces that include the vertex.
Instead of directly predicting pose-dependent Gaussian assets, ours output pose-dependent offsets.
This is helpful for the generalization to novel poses as Gaussian assets solely from the triplane already have reasonable expressiveness, which makes the role of the pose-dependent Gaussian assets small.
Such a design is especially important when making 3D avatars from a short video like ours as limited pose diversity makes generalization to novel poses challenging.




 §.§ Animation and rendering


Fig. <ref> shows examples of our animated and rendered avatars, made from short monocular videos.

Animation.
We need to animate Gaussian points from the canonical space with given facial expression code ψ and 3D poses θ of SMPL-X.
To this end, we first replace the pose-dependent vertex offset Δ𝐕_pose of hand and face vertices to those of SMPL-X.
This is because the hand and face are often naked; hence, we can directly utilize vertex offsets of SMPL-X.
Then, we add vertex offsets from facial expression code ψ of SMPL-X  to face vertices.
By directly using the facial expression offsets of SMPL-X, we do not have to learn a new facial expression space.
Such a direct utilizing is from our hybrid representation of the mesh and 3D Gaussians.
The below equations describe the above deformations in the canonical space.

    𝐕̅_tri = 𝐕̅ + Δ𝐕_tri + Δ𝐕_expr,


    𝐕̅_pose = 𝐕̅ + Δ𝐕_tri + Δ𝐕_pose + Δ𝐕_expr,

where Δ𝐕_expr represents the facial expression offset of SMPL-X, obtained from the facial expression code ψ.
Then, for the body vertices, we take the skinning weight of the nearest vertices from downsampled 𝐕̅, while for the hand and face vertices, we use the original skinning weight of the vertices.
This is because, for the body vertices, their semantic meaning could change due to the cloth geometry.
The final animated geometry, 𝐕_tri and 𝐕_pose, are represented with below equations.

    𝐕_tri = LBS(𝐕̅_tri, θ, 𝐖_tri)   and  𝐕_pose = LBS(𝐕̅_pose, θ, 𝐖_pose),

where 𝐖_tri and 𝐖_pose represent the skinning weight of 𝐕̅_tri and 𝐕̅_pose, respectively.




Rendering.
To render animated 3D geometry, we use 3DGS rendering pipeline <cit.> like the below equations.

    𝐈_tri = f(𝐕_tri, exp(𝐒_tri), 𝐂_tri, 𝐊, 𝐄),


    𝐈_pose = f(𝐕_pose, exp(𝐒_tri + Δ𝐒_pose), 𝐂_tri + Δ𝐂_pose, 𝐊, 𝐄),

where f, 𝐊, and 𝐄 represent rendering function of 3DGS, camera intrinsic, and extrinsic matrices, respectively.
As described above, following Hu  <cit.>, we restrict all Gaussian assets to isotropic for better generalization; hence, rotation and opacity of all Gaussian points are set to identity and one, respectively, not described in the equations.





 §.§ Loss functions

During the training of our ExAvatar, we optimize the triplane 𝐓, MLPs for the regression of Gaussian assets in Sec. <ref>, 3D pose θ of each frame, facial expression code ψ of each frame, 3D translation 𝐭 of each frame, the shape parameter β, and the joint offset Δ𝐉.
Also, we simultaneously optimize a 3DGS for the background following the original implementation <cit.> by segmenting out human regions using human masks from an off-the-shelf human segmentation model <cit.>.
Modeling background simultaneously produces better foreground mask <cit.>, as estimated masks often have errors, especially on hand parts.
We denote rendered images from a combination of Eq. <ref> and the 3DGS for the background by 𝐈_tri^*.
Likewise, we denote rendered images from a combination of Eq. <ref> and the 3DGS for the background by 𝐈_pose^*.
To train ExAvatar, we minimize the below loss functions.

Image loss.
Following 3DGS <cit.>, we minimize L1 distance, 1 - SSIM, and LPIPS <cit.> between rendered images (i.e., 𝐈_tri^* and 𝐈_pose^*) and the captured image. 
We found that the additional LPIPS is helpful for sharper textures.
To save the computation, we compute the image loss after cropping the human region.








Face loss.
Unlike other human parts, the face has its unique characteristics as there should be a strong consistency between geometry and texture.
For example, lip geometry usually has reddish textures.
If other face geometry has lip textures, in novel facial expressions or jaw poses, the lip would not properly change, which can lead to significant artifacts, as shown in Fig. <ref> (b).
Simply minimizing the above image loss does not guarantee the consistency between the geometry and texture of the face region.
To enforce the consistency, we minimize the L1 distance between the rendered face image with a standard differentiable mesh renderer and the captured image, where the texture for the mesh renderer is prepared by averaging the unwrapped UV texture of the face-only model <cit.> registrations from Sec. <ref>.
The UV texture is fixed, and the positions of 3D Gaussian points of the face region are adjusted to minimize the loss function.
Fig. <ref> (a) shows the effectiveness of our face loss functions.
Thanks to our hybrid representation of the mesh and 3D Gaussians, such a mesh-based loss function can be easily incorporated into our system.




Regularizers.
Due to the limited pose diversity in the training set, there can be human parts that are not observed in the video.
Such human parts suffer from occlusion ambiguity, which could result in artifacts in novel facial expressions and poses.
In addition, to utilize the facial expression offsets of SMPL-X, we need to make the face geometry similar to that of SMPL-X.
To address them, we utilize connectivity-based regularizers (i.e., Laplacian regularizer), motivated by the mesh modeling works <cit.>.
Fig. <ref> shows that our connectivity-based regularizer significantly reduces artifacts in novel facial expressions and poses.
We minimize the difference of the 1) Laplacian of deformed 3D Gaussian points in the canonical space (i.e., 𝐕̅_tri and 𝐕̅_pose) and 2) Laplacian of the canonical mesh 𝐕̅.
In this way, we can easily encourage the local similarity between 3D Gaussian points, which can prevent floating 3D Gaussians.
In particular, our connectivity-based regularizer is much more effective than the widely used L2 regularizer <cit.> that simply penalizes distance between 3D Gaussian points and underlying template mesh without considering the connectivity information. 
Due to our hybrid representation of the mesh and 3D Gaussians, the Laplacian regularizer, widely used in mesh modeling works, can be easily included in our system.
In addition to regularizing the 3D positions of 3D Gaussian points, we compute the same Laplacian regularizer for the scales and RGBs of 3D Gaussian points.
For other regularizers, please refer to the supplementary material.






§ EXPERIMENTS




 §.§ Datasets




NeuMan.
NeuMan <cit.> provides several short monocular videos taken from in-the-wild environments.
Each video contains a single person walking around for about 15 seconds.
Following previous works <cit.> we use bike, citron, jogging, and seattle videos that exhibit most human body regions and contain minimal blurry images.
We follow their official training and testing splits.

X-Humans.
X-Humans <cit.> provides 3D scans and RGBD videos of multiple subjects, captured from a studio.
Compared to NeuMan, X-Humans has more diverse facial expressions and hand poses.
There are two experimental protocols: 1) using 3D scans and 2) using RGBD images for creating avatars.
We create our avatar only with monocular RGB videos without depthmaps and compare ours against previous works <cit.> that use RGBD videos.
We use 0028, 0034, and 0087 subjects as their pre-trained weights of the RGBD protocol are publicly available.
We follow their official training and testing splits.













 §.§ Comparison to state-of-the-art methods

Tab. <ref> and  <ref> show that our ExAvatar achieves the best results on NeuMan <cit.> dataset regardless of whether the rendered background pixels are included or not.
All numbers are from papers <cit.> except NeuMan <cit.>, Vid2Avatar <cit.>, and 3DGS-Avatar <cit.> of Tab. <ref>, measured with their officially released code.
To exclude background pixels, we used an off-the-shelf segmentation network <cit.> following GaussianAvatar <cit.>.
Following previous works <cit.>, for the evaluation on NeuMan dataset, we fit SMPL-X parameters of testing frames while freezing all other parameters with the image loss of Sec. <ref>.














Fig. <ref> shows that ours produces photorealistic renderings in novel views and poses.
For example, prints on the shirts (the first and third rows) are significantly sharper and clearer than those of previous works.
Most importantly, ours produces faces and hands in novel views and poses substantially better than previous avatars.
As previous avatars do not have controllability on faces and hands, the averaged blurry textures are baked in (faces in the first row and hands in the second row and fourth row).
On the other hand, ours has sharp textures benefiting from the whole-body modeling.


Tab. <ref> and Fig. <ref> show that our ExAvatar outperforms previous whole-body avatar <cit.> on X-Humans <cit.> dataset even without using depth maps, while the previous work relies on it.
Our hybrid representation of the surface mesh and 3D Gaussians leads to stable training and shaper textures of faces and hands.
For X-Avatar's results, we used their officially released pre-trained weights and code.
Following Shen  <cit.>, we used given SMPL-X parameters without further fitting them to testing frames.







 §.§ Ablation study

In this section, we ablate the effectiveness of our hybrid representation of the surface mesh and 3D Gaussians, which enables us to incorporate Laplacian regularizer and face loss into our system.
Fig. <ref> and Tab. <ref> show that incorporating the Laplacian regularizer into our system brings significant performance boost and stability.
Fig. <ref> and Tab. <ref> show the benefit of the proposed face loss.
The numbers in Tab. <ref> are measured only for cropped face images when the face is visible to evaluate the effectiveness of the face loss.
The face visibility is decided by rasterizing SMPL-X meshes and checking the number of rasterized triangle faces of the face region.



§ CONCLUSION


Summary.
We present ExAvatar, an expressive whole-body 3D avatar that can be made from a short monocular video.
We propose a hybrid representation of the surface mesh and 3D Gaussians to address 1) the limited diversity of facial expressions and poses in the video and 2) the absence of 3D observations, such as 3D scans and RGBD images.
Our hybrid representation makes ExAvatar fully compatible with the facial expression space of SMPL-X and significantly reduces artifacts in novel facial expressions and novel poses.


Limitations.
First, as the inside of the mouth including the cavity and palm of the hands are often not observed in the video, our model hallucinates plausible geometry and textures.
Second, like previous avatars <cit.>, ours struggles in modeling dynamic clothes.
Material of clothes with motion information, such as velocity and acceleration, should be considered to properly model such dynamic clothes, out of our scope.


Future works.
To hallucinate unobserved human parts better, such as inside of the mouth, score distillation sampling <cit.> can be used to generate images and use them for supervision.
In addition, adding relightability to our ExAvatar is a promising and interesting future direction.





Supplementary Material for 
 

“Expressive Whole-Body 3D Gaussian Avatar"








   
   


In this supplementary material, we provide more experiments, discussions, and other details that could not be included in the main text due to the lack of pages.
The contents are summarized below:

    
  * Sec. <ref>: Details of co-registration of SMPL-X
    
  * Sec. <ref>: Detailed architecture of MLPs
    
  * Sec. <ref>: Regularizers
    
  * Sec. <ref>: Running time comparison
    
  * Sec. <ref>: 3D geometry comparison
    
  * Sec. <ref>: Comparison to generative AIs
    
  * Sec. <ref>: Implementation details
    
  * Sec. <ref>: Failure cases






§ DETAILS OF CO-REGISTRATION OF SMPL-X
 
This section describes details of the co-registration of SMPL-X of Sec. 3.1 of the main manuscript.

FLAME registration.
Given DECA <cit.>'s output FLAME parameters (i.e., face shape parameter, facial expression code, and jaw pose), we further fit them to 2D keypoints of face, where the 2D keypoints are from an off-the-shelf regressor <cit.>.
For the fitting, we minimize the below loss functions.

    L_FLAME = L_kpt + 0.1 L_init,

where L_kpt and L_init represent 1) L1 distance between projected and target 2D keypoints and 2) L1 distance between optimizing FLAME parameters and the initial regressed ones, respectively.
In this way, we can make FLAME's meshes pixel-aligned with the image while preventing them from being too far from DEAC's regressed ones.


SMPLX registration.
Given Hybrid-X <cit.>'s output SMPL-X parameters, we further fit them to 2D keypoints of the whole body, where the 2D keypoints are from an off-the-shelf regressor <cit.>.
We replace the facial expression code of Hybrid-X with that of DECA as DECA's output is more accurate.
As FLAME and SMPL-X share the same facial expression space, we share the facial expression code of FLAME and SMPL-X during the registration.
Before performing the LBS inside of the SMPL-X layer, we 1) add the joint offset Δ𝐉 to the T-pose joint locations of SMPL-X and 2) add the face offset Δ𝐕_face to the face region of the T-pose template mesh.
The joint offset allows us to obtain a more personalized 3D skeleton and surface than only using the SMPL-X shape parameter.
Please note that the joint offset affects both skeleton and surface as the added joint locations are used for following forward kinematics and LBS.


We minimize the below loss functions.

    L_SMPLX = L_kpt + 0.1 L_init + L_face + L_reg,


    L_face = 10 L_vertex + 10000 L_lap + L_edge,


    L_reg = 0.01 L_shape + 100 L_jo + L_sym,

where L_kpt and L_init represent 1) L1 distance between projected and target 2D keypoints and 2) L1 distance between optimizing SMPL-X parameters and the initial regressed ones, respectively.
We compute the L_kpt with and without the face offset Δ𝐕_face to prevent the face offset from representing any pose-dependent and expression-dependent things.
L_face is to optimize the face offset Δ𝐕_face.
L_vertex, L_lap, and L_edge represent 1) L1 distance between SMPL-X face vertices and FLAME vertices, 2) L2 distance between Laplacian of SMPL-X face mesh and FLAME mesh, and 3) L1 distance between edge length of SMPL-X face mesh and FLAME mesh, respectively.
We compute L_vertex with and without the pose and facial expression to prevent the face offset from representing any pose-dependent and expression-dependent things.
L_lap and L_edge are computed only for meshes without pose and facial expressions.
We compute L_face only when the face is visible.
To check the face visibility, we compute two vectors.
First, a vector from the face center to the middle of the eyes in the xz space of the camera-centered coordinate system.
Second, a vector from the camera position (origin) to the face center in xz space of the camera-centered coordinate system.
We decide the face is visible if the dot product between the two vectors is smaller than cos(135°).
Finally, L_reg is to regularize the SMPL-X shape parameter, joint offset, and face offset.
L_shape is a squared L2 norm of the SMPL-X shape parameter.
L_jo is a squared L2 norm of the joint offset, which prevents an extreme joint offset, and L_sym is for symmetricity of the joint offset and face offset, similar to Feng  <cit.>.





§ ARCHITECTURE OF MLPS
 
This section describes the detailed architecture of MLPs of Sec. 3.2 of the main manuscript.
The MLPs are briefly described in Fig. 4 of the main manuscript.



 §.§ MLPs without pose conditioning
 
The MLPs without the pose conditioning (the red MLPs of Fig. 4 of the main manuscript) consist of two types of MLPs.
The first MLP takes the triplane feature 𝐅 and outputs the geometry of 3D Gaussian points (i.e., Δ𝐕_tri and 𝐒_tri).
It consists of four fully connected layers with a hidden size of 128.
We use the group normalization <cit.> and ReLU activation function between each fully connected layer.
The second MLP takes the triplane feature 𝐅 and outputs RGB colors of 3D Gaussian points 𝐂_tri.
It has the same architecture as that of the first MLP except for the output dimension.




 §.§ MLPs with pose conditioning

The MLPs with the pose conditioning (the blue MLPs of Fig. 4 of the main manuscript) consist of two types of MLPs.
The MLPs have the same architecture as that of the MLPs of Sec. <ref> except for the input and output dimensions.
The first MLP takes the triplane feature 𝐅 and 3D pose θ of SMPL-X without the root pose and outputs geometry offsets of 3D Gaussian points (i.e., Δ𝐕_pose and Δ𝐒_pose).
The second MLP takes the 1) triplane feature 𝐅, 2) 3D pose θ of SMPL-X without the root, and 3) normal of each 3D Gaussian point and outputs RGB offset Δ𝐂_pose.







§ REGULARIZERS
 
This section describes regularizers, not introduced in Sec. 3.4 of the main manuscript.
First, to prevent 3D Gaussian points from being too far from underlying canonical mesh 𝐕̅, we penalize squared L2 norm of 3D Gaussian offsets Δ𝐕_tri and Δ𝐕_pose.
Second, to prevent 3D Gaussian from being too big, we penalize the squared L2 norm of 3D Gaussian scales.
Third, as hands often suffer from noisy colors due to their small scales and complicated articulations, we minimize the squared L2 distance between RGB colors of hand 3D Gaussian points and the average colors of hand 3D Gaussian points.
Finally, we use the same L_jo of Sec. <ref>.





§ RUNNING TIME COMPARISON
 
Tab. <ref> shows that ours achieves real-time animation and rendering speed while existing works <cit.> fail to.
Without the pose conditioning (the blue MLP of Fig. 4 of the main manuscript), ours achieves even faster speed.
For all methods, we measure the running time only for the human animation and rendering without the background rendering.
A single NVIDIA V100 GPU is used, and the rendering resolution is 1024 × 1024.








§ GEOMETRY COMPARISON
 
Fig. <ref> and Tab. <ref> show that our ExAvatar produces better 3D geometry than previous state-of-the-art 3D whole-body avatar <cit.>, especially around the face and neck.
It is not straightforward to extract actual 3D geometry from 3D Gaussians.
Instead, we render masks/depth maps/normal maps of Gaussians to multiple viewpoints and comparing them with ground truth.
However, when rendering depth/normals, even recent SOTA methods <cit.> simply take the depth/normal from the center of Gaussians and alpha-blend it in the screen space.
This may result in incorrect depth/normal maps as elongated Gaussians can deviate from the depth/normal values of the Gaussian centers.
Nevertheless, we report 3D geometry comparisons in the figure and table for reference.
We leave actual 3D geometry extraction from ExAvatar as a future work.





§ COMPARISON TO GENERATIVE AIS
 
Motivated by the powerful modeling capability of the diffusion models <cit.>, several human animation methods <cit.> have been introduced.
They can animate humans only from a single image using DensePose <cit.> or 2D pose as the driving signals.
Despite their photorealistic and plausible outputs, they lack authenticity, as shown in Fig. <ref>.
The person of the (b) generated image from MagicAnimate <cit.> has definitely a different face identity and clothes compared to the (a) captured image, although they look similar.
The result of MagicAnimate <cit.> is obtained using their official released code.
This is because of the hallucination, one of the biggest challenges for modeling generative AIs.
On the other hand, ours might lack plausibility, especially for unobserved human parts such as inside of mouth and cloth dynamics; however, ours has more authenticity.
We think combining the plausibility of generative approaches and the authenticity of our approach should be an interesting and future direction, as discussed in Sec. 5 of the main manuscript.


§ IMPLEMENTATION DETAILS
 
PyTorch <cit.> is used for implementation. 
For the training, we use Adam optimizer <cit.>.
The initial learning rate is set to 10^-3 and reduced by a factor of 10 at the 75 % and 90 % of total number of iterations.
We use a single V100 NVIDIA GPU for experiments, and the training takes 1.5 hours to 5 hours depending on the duration of the videos.
All other details will be available in our code. 






§ FAILURE CASES
 
Fig. <ref> shows that our rendered avatar does not have a smiling facial expression despite the person in the image is smiling.
This is because of the wrong regressed facial expression code ψ from the off-the-shelf regressor <cit.> as (b) rendered SMPL-X mesh also has the wrong facial expression.
As our ExAvatar is animated with the regressed facial expression code, ours also has wrong facial expression.
We think advanced facial models and high-fidelity regressors should be investigated to address this failure case.


Fig. <ref> additionally shows our failure case.
Although the person in (a) the image is frowning, our rendered avatar is slightly smiling.
This is because the subject of our avatar in the NeuMan dataset <cit.> is always smiling during the short video capture.
Hence, the smiling facial appearance is baked into our avatar.
Canonicalizing facial appearance from a short monocular is greatly challenging.
In particular, when the face takes a few pixels in the video like our case, 2D keypoints are often noisy, which makes the lip geometry of SMPL-X/FLAME are not fit perfectly in the co-registration stage (Sec. 3.1 of the main manuscript).
Such a misalignment error is propagated to our ExAvatar learning stage, which causes the failure case.
We think using priors of the canonical facial appearances using generative methods can be one way to address this failure case.




splncs04



 Paper Title: How Effective are Self-Supervised Models for Contact Identification in Videos.



How Effective are Self-Supervised Models


Gunawardhana et al.
Mohamed bin Zayed University of Artificial Intelligence: MBZUAI, UAE University of Moratuwa, Sri Lanka Weizmann Institute of Science, Israel
 *Corresponding author. Email: malithagunawardhana96@gmail.com




How Effective are Self-Supervised Models for Contact Identification in Videos
    Malitha Gunawardhana*1 Limalka Sadith1,2 Liel David3Daniel Harari3 Muhammad Haris Khan1

    August 5, 2024
============================================================================================


§ ABSTRACT 
The exploration of video content via Self-Supervised Learning (SSL) models has unveiled a dynamic field of study, emphasizing both the complex challenges and unique opportunities inherent in this area. Despite the growing body of research, the ability of SSL models to detect physical contacts in videos remains largely unexplored, particularly the effectiveness of methods such as downstream supervision with linear probing or full fine-tuning. This work aims to bridge this gap by employing eight different convolutional neural networks (CNNs) based video SSL models to identify instances of physical contact within video sequences specifically. The Something-Something v2 (SSv2) and Epic-Kitchen (EK-100) datasets were chosen for evaluating these approaches due to the promising results on UCF101 and HMDB51, coupled with their limited prior assessment on SSv2 and EK-100. Additionally, these datasets feature diverse environments and scenarios, essential for testing the robustness and accuracy of video-based models. This approach not only examines the effectiveness of each model in recognizing physical contacts but also explores the performance in the action recognition downstream task. By doing so, valuable insights into the adaptability of SSL models in interpreting complex, dynamic visual information are contributed.







§ INTRODUCTION





Automated video analysis has significantly evolved from basic frame-by-frame methods to sophisticated systems that understand the temporal dynamics in videos <cit.>. This shift marks a move from analyzing static images to interpreting complex, dynamic scenes, enhancing our understanding of visual content  <cit.>. Initially, video analysis relied on manually extracted features and simple models to understand temporal relationships, which worked well in controlled settings but struggled with real-world video complexity <cit.>. The introduction of deep learning, particularly Convolutional Neural Networks (CNNs), revolutionized video analysis, improving its applicability and performance. However, these advancements depend heavily on large, annotated datasets, which are often costly and scarce, posing scalability and adaptability challenges <cit.>.

The rise of self-supervised learning (SSL) offers a solution by using the data's inherent structure to create learning signals, eliminating the need for extensive annotations. This paradigm is particularly promising in video analysis as it allows models to learn from the data itself, for instance by leveraging the continuity and predictability of visual elements to build robust, context-aware systems <cit.>. This research direction promises to learn comprehensive, context-sensitive representations by exploiting temporal coherence <cit.>, spatial continuity <cit.> and the predictability of motion patterns <cit.>,


Distinguishing itself from image-based SSL methods, video-based SSL confronts unique challenges due to the intrinsic properties of videos <cit.>. Initially, researchers explored simple ordering-based methods to achieve promising results <cit.>. Subsequently, various strategies utilizing the Vision Transformer (ViT) <cit.> frameworks were proposed, introducing new areas to the field. The typical training path for SSL models involves a two-step process: a pretraining stage followed by a fine-tuning stage. During pretraining, models use supervisory signals to capture the intrinsic nature of the data. For instance, in video data, this could involve understanding common motion patterns, recognizing typical backgrounds, and identifying the relationship between different frames.  The trained models are then fine-tuned for specific downstream task such as action recognition and video retrieval. Most models undergo pretraining using the Kinetics-400 dataset <cit.>  and are evaluated on the UCF-101 dataset <cit.>, with evaluations focusing on action recognition or video retrieval tasks. Despite achieving notable performance, questions about the generalizability of these methods remain <cit.>. 



The ability to understand video content at intricate level has profound implications across various applications. This nuanced comprehension can substantially enhance surveillance systems by incorporating real-time alerting features that detect specific actions, such as distinguishing between a friendly wave and a distressed hand signal. Furthermore, it can redefine human-computer interaction in the sphere of robotics, enabling systems to interpret and respond accurately to subtle human behaviors. Among the  various challenges in this domain, the accurate identification of physical contact between objects or entities within video frames stands out as particularly critical. This aspect is essential for applications such as physical interaction detection <cit.>, and the creation of immersive augmented reality experiences <cit.>. This paper's focus is to thoroughly evaluate the efficacy of current CNN-based SSL models in identifying contact within videos. Such an examination is pivotal in advancing our understanding of visual content, setting the stage for significant progress in video analysis methodologies without relying heavily on extensively labelled datasets. The structure of this paper is outlined as follows.  First, we offer a comprehensive explanation of the models chosen for evaluation. Next, we describe the experimental setup. We then present the results and discuss their implications. The paper concludes with a summary of the findings in the conclusion section.







§ EVALUATED MODELS




For an unbiased and objective assessment, our evaluation exclusively encompasses CNN-based models only. These models have all been pre-trained on the Kinetic-400 dataset, ensuring a consistent foundation across the board. To further enhance the fairness and comparability of our evaluation, each model employs the R(2+1)D-18   <cit.> architecture as its backbone. This specific architecture choice aligns with our goal to maintain a uniform structure across models, minimizing variability that could arise from differing network designs. We have selected a total of eight CNN-based models that utilize SSL techniques for this evaluation. 


Selected models are AVID-CMA <cit.>, 
Catch the Patch (CTP) <cit.>, 
GDT <cit.>, 
MoCo <cit.>,VideoMoCo <cit.> , 
Pretext-Contrast (Pre-Con) <cit.>,
RSPNet <cit.>, and
TCLR <cit.>. 
  


AVID-CMA <cit.> is a multi-modal learning framework that takes advantage of both video and audio data. This method employs Audio-Visual Instance Discrimination (AVID) to foster a cross-modal similarity metric, effectively pairing video and audio instances that coexist. Furthermore, Cross-Modal Agreement (CMA) enhances this approach by clustering videos that exhibit strong similarity across both video and audio dimensions, thereby refining the selection of positive and negative samples for training.

Inspired by how the human eyes work during childhood, CTP <cit.> proposed an SSL pretext task known as "Catch the Patch".  The method involves tracking patches across video frames to learn consistent and robust feature representations. The core idea is that the temporal consistency of appearances and motions in videos offers a rich, unsupervised signal for learning. By focusing on patches rather than whole frames or objects, the method captures fine-grained details.

GDT <cit.> is also a multi-modal learning with contrastive learning. Authors generalize contrastive learning to a wider set of transformations and their compositions, aiming for invariance or distinctiveness in image representations. Previous work has shown that the choice and composition of transformations are crucial for performance in contrastive learning. However, these choices have been mostly driven by intuition, lacking formal understanding and generalization.  The authors propose a formal analysis of composable transformations in contrastive learning, providing principles for constructing training batches. They introduce a practical construction that satisfies the requirements of contrastive formulations. 


Momentum Contrast or MoCo <cit.> is one of the famous contrastive learning methods that focus on images. The proposed method, which consists of a dynamic dictionary and moving-average encoder, allows for the on-the-fly construction of a large, consistent dictionary, enhancing the effectiveness of contrastive unsupervised learning. VideoMoCo <cit.> extends MoCo concepts to videos. 

Rather than utilizing a single pretext task or contrastive learning method, the Pre-Con <cit.> method combines these two aspects. This joint optimization framework can improve performance rather than using one method.  They use 3D RotNet <cit.>, VCP <cit.> and VCOP <cit.> as the pretext task.


RSPNet <cit.> is another pretext task-based method which focuses on using speed for supervision. They use relative speed between two clips rather than using the exact speed.  To ensure the learning of appearance features, RSPNet also introduces an appearance-focused task, where the model is enforced to perceive the appearance difference between two video clips. 

TCLR <cit.> is another contrastive learning approach designed to emphasize the distinct of features over time. Unlike previous contrastive learning methods for video data, which did not specifically focus on temporal feature distinctiveness, TCLR uses clips from the same video as negative examples to promote diversity across time. The method introduces two innovative loss functions: the local-local temporal contrastive loss, targeting discrimination among non-overlapping clips from the same video, and the global-local temporal contrastive loss, which aims to enhance the temporal variance of features by distinguishing between different time steps within the feature map of a clip.

A summary of these models and their modality (based on <cit.>) is shown in Table <ref>. 












§ EXPERIMENTAL SETUP



 §.§ Dataset




We utilize two main datasets. The first dataset is the Something-Something V2 (SSv2) <cit.> dataset. The SSv2 contains 168913 training videos and 24777 validation videos. Each video is assigned to motion-centric action classes known as templates. There are 174 templates, including descriptions like "Holding something next to something" and "Digging something out of something," among others. 

The second dataset is the Epic-Kitchen-100 (EK-100) dataset <cit.>. This is a large-scale video dataset based on day-to-day activities in the kitchen. The dataset is divided into three main categories based on the annotations. Those are 1) Noun, 2) Verb and 3) Action. Videos are ego-centric and collected from 45 kitchens in four cities. A total of 97 verb classes and 300 noun classes are available in this dataset. Figure <ref> and Figure <ref> show an example of the EK-100 video.

These datasets were selected due to their unique characteristics, which align well with the objectives of this study. SSv2 captures a wide range of action categories and temporal dynamics, making it ideal for training models to recognize various contact-based activities. EK100, with its realistic context and ego-centric perspective, captures everyday activities involving numerous object interactions. The rich annotations, including action verbs and nouns, along with temporal segmentation, enhance the dataset's utility in identifying specific contact events.



We conduct an exhaustive review and analysis of each video template within SSv2 and each action within EK-100. For the purposes of this analysis, both video templates from SSv2 and actions from EK-100 are collectively referred to as "templates." These templates and videos are categorized into two primary groups: 1) videos depicting human interaction with objects, designated as the "True" category, and 2) videos lacking any human-object interaction, designated as the "False" category. It's important to note that the presence of contact between humans and objects is not immediately apparent through visual inspection alone. As a result, we employed optical flow analysis <cit.> for each videos to discern instances of contact. However, it was observed that within some videos, the presence of contact was inconsistent across videos. Therefore, these videos and templates were excluded from our analysis. Examples of these cases for the SSv2 dataset are shown in Figure <ref> and Figure <ref>. 










 §.§ Implementation Details



We implement these methods using the PyTorch 1.6.0 framework and train all models on eight Tesla A100 GPUs. Our evaluation methodology draws inspiration from <cit.>, adopting the same hyperparameters they utilized. For fine-tuning the SSv2 dataset, we employ a batch size of 32 and a learning rate of 0.0001, training each model for a total of 45 epochs. Similarly, for the EK-100 dataset, we maintain the same batch size but adjust the learning rate to 0.0025 and train over 30 epochs. In the context of linear evaluation, the SSv2 dataset, we set a batch size of 64 and a learning rate of 0.01 for 40 epochs, whereas, for EK-100, we reduce the batch size to 32 while retaining a learning rate of 0.0025 over 30 epochs.



 §.§ Evaluation Method













We assess our models under two main conditions: Overall performance and Template-based performance. Overall performance evaluates the models based on action recognition accuracy across the entire dataset. Template-based performance evaluation focuses on scenarios where there is contact between humans and objects. For overall performance on the SSv2 dataset, we utilize four accuracy metrics: Top-1, Top-5, Mean Top-1, and Mean Top-5. In contrast, the performance of the EK-100 dataset is evaluated using only Top-1 and Mean Top-1 accuracy metrics. During the Template-Based Performance evaluation, we apply only the Top-1 accuracy metric for both datasets. Moreover, only the verb class is considered for Template-based performance evaluation.

Top-1 accuracy can be defined as the correct number of predictions divided by total predictions. Top-5 accuracy is determined by the condition that at least one of the top five predictions with the highest probabilities corresponds to the actual outcome. The mean Top-1 accuracy refers to the average accuracy across all classes or datasets where, for each prediction, only the most probable outcome is considered correct. It's the mean of the Top-1 accuracies for each class or dataset. On the contrary, the mean Top-5 accuracy follows a similar concept. Still, it extends the criteria for a correct prediction to any of the top five most probable outcomes predicted by the model. It is the average of the Top-5 accuracies for each class or dataset.

In addition to the previously mentioned evaluation criteria, we evaluate both datasets under two main conditions. 1) full fine-tuning and 2) linear evaluation. During the full fine-tuning phase, the whole weights are modified throughout the training process. It allows us to modify the pre-trained model with small, incremental changes that are designed to enhance performance on the specific task. Linear evaluation keeps the pre-trained model static and learns a linear layer on top of it to map the model's output to the target task's output. In other words, we freeze the entire network except for the last linear/classification layer.


In the SSv2 dataset, there are a total of 24,777 videos distributed across 97 templates. Of these, 12,620 videos are labeled as 'false' and 7,812 as 'true'. The EK-100 dataset contains 9,668 videos, with 4,001 labeled as 'true' and 3,389 as 'false'. The remaining 4,345 videos in SSv2 and 2278 videos in EK-100 contain both true and false videos, and were excluded from the analysis. A summary of these statistics is presented in Table <ref> for both SSv2 and EK-100 datasets.








§ RESULTS


Table <ref>, Table <ref>, Table <ref>, and Table <ref> present the results for SSv2 full fine-tuning, SSv2 linear evaluation, EK-100 full fine-tuning, and EK-100 linear evaluation, respectively under overall performance evaluation.

Figure <ref> illustrates the template-based performance on the SSv2 dataset, detailing both full fine-tuning (see Figure <ref>) and linear evaluation scenarios (refer to Figure <ref>). Similarly, Figure <ref> demonstrates the template-based performance for the EK-100 dataset, with a specific focus on full fine-tuning (as shown in Figure <ref>) and linear evaluation settings (outlined in Figure <ref>). All values are presented as percentage (%) values in all figures and tables. 
























When evaluating overall performance across both datasets under two assessment conditions (full fine-tuning and linear evaluation), it's apparent that no single method consistently outperforms the others. Specifically, within the SSv2 dataset's full fine-tuning category (Table <ref>), TCLR achieves the highest Top-1 accuracy, whereas CTP secures the best Top-5 accuracy. For the linear evaluation (Table <ref>) within the same dataset, VideoMoCo stands out by leading in both Top-1 and Top-5 accuracy metrics. Comparing the outcomes for SSv2 across both evaluation methods, the linear evaluation showcases a broader range of results. The lowest Top-1 accuracy is noted at 7.21% with MoCo, and the highest reaches 19.66% with VideoMoCo, indicating a more significant variance than in the full fine-tuning approach, where Top-1 accuracy spans from a minimum of 45.26% with AVID-CMA to a maximum of 57.43% with TCLR.

In the EK-100 dataset, under the full fine-tuning scenario (see Table <ref>), VideoMoCo achieves the best Top-1 accuracy in verb recognition with a score of 44.53%. However, RSPNet outperforms other models across all evaluated metrics for both noun and action recognition. In the linear evaluation setting of the EK-100 dataset (see Table <ref>), VideoMoCo secures the highest Top-1 and Top-1 accuracy in verb recognition, whereas TCLR stands out by achieving the best Top-1 and Top-1 accuracy for noun and action recognition. Compared to the results on the SSv2 dataset, the EK-100 dataset shows lower performance in action recognition. Particularly in the linear evaluation, only the TCLR model achieves an action recognition accuracy greater than 1%, while models like MoCo, Pre-Con, and AVID-CMA achieve nearly 0% accuracy, with specific scores of 0.01%, 0.01%, and 0.02%, respectively. According to the results of both full fine-tuning and linear evaluation, noun and action recognition are significantly more challenging compared to verb recognition. 

Regarding the template-based performance evaluation, there is a clear difference between both datasets. False templates yield the highest performance in both full fine-tuning and linear evaluation modes in the SSv2 dataset (see Figure <ref>). In the full fine-tuning scenario, the TCLR model outperforms others in handling both true and false templates. Conversely, in the linear evaluation scenario, VideoMoCo leads in performance for both template types. Similar result variation as we observed in overall performance in SSv2 can be observed in template-based performance as well, where linear evaluation scores range from a minimum of 10.05% for false templates and 4.17% for true templates in MoCo to a maximum of 22.71% for false templates and 14.03% for true templates in VideoMoCo. In the context of full fine-tuning, performance spans from a minimum of 44.16% for false templates and 43.56% for true templates in AVID-CMA to a maximum of 58.57% for false templates and 54.80% for true templates in TCLR models.

In the EK-100 dataset, true templates show superior performance compared to false templates in both full fine-tuning and linear evaluation settings.  VideoMoCo shows the highest verb recognition accuracy (we only consider verbs as templates in the EK-100 dataset) in full fine-tuning setting, achieving 41.37 % and 56.39%, respectively, for both false and true templates. In the context of linear evaluation, VideoMoCo leads in recognizing false templates with a 22.66% accuracy, whereas CTP, MoCo, and Pre-Con exhibit superior performance in identifying true templates, each achieving a 48.41% accuracy. MoCo, CTP and Pre-Con could not correctly identify any verb in false templates in the linear evaluation setting. 

The observed disparity in the performance of noun and action recognition compared to verb detection within EK-100 dataset can be attributed to the limitations of the R(2+1)D-18 backbone, which struggles with complex actions. Interestingly, excelling in verb recognition does not necessarily guarantee similar success in recognizing nouns or actions. This indicates that these tasks require different skills, and a model's strength in one area doesn't automatically mean it will excel in another. For example, VideoMoCo does well with verbs but falls short in recognizing nouns, highlighting the challenge of adapting methods geared towards verb recognition to the subtle requirements of noun recognition in dynamic video content. On the other hand, RSPNet and TCLR show promise by performing well in both verb and noun recognition tasks. This suggests that these models have a more flexible or effective way of processing and learning from video data, which helps them meet the varied demands of different recognition tasks. Such versatility is key for creating advanced action recognition systems that can accurately identify both the actions being performed and the objects involved in those actions in complex visual settings.




§ DISCUSSION AND CONCLUSION




The study of video content using SSL models has emerged as a dynamic area of research, highlighting the complex challenges and distinct opportunities in this field. Among the methods that we explore, TCLR and VideoMoCo emerge as notable performers, showcasing their robust capabilities across various evaluation conditions. VideoMoCo focuses on capturing robust representations that are insensitive to temporal variations, while TCLR focuses on capturing the temporal variations within video instances. 
Both approaches employ discrimination-based learning objectives and focus on learning high-level cues, which likely contributes to their high performance.


However, finding the best-performing model remains complex, with no single model consistently outperforming across all datasets and conditions. This nuanced landscape of results underscores the inherent complexity and diversity of video analysis tasks, revealing the multifaceted nature of visual understanding and interpretation for contact identification.


A focal point of this exploration is the discernible variance in model performance between the EK-100 and SSv2 datasets. The EK-100 dataset, characterized by its ego-centric video perspectives, presents a formidable challenge that starkly contrasts the nature of the SSv2 dataset. This difference is primarily attributed to the ego-centric composition of EK-100 videos, which diverges significantly from the more generalized content found in the Kinetics-400 dataset. The ego-centric viewpoint captures a first-person perspective, often encapsulating complex, nuanced interactions with the environment that are inherently difficult to model. This complexity is increased by the need to accurately detect both verbs and nouns within these interactions, a task that has proven to be particularly challenging within the EK-100 dataset.


The challenge of detecting human contact between objects in the SSv2 dataset shows a major weakness in current SSL models. Although these models do well in identifying contact in the EK-100 dataset, they struggle with the unique challenges of SSv2, highlighting a shortfall in how they learn. Particularly, when using true templates—specific tests designed to see if the models can recognize physical contacts—the difference in performance is even more obvious. Generally, these models perform poorly in detecting contacts across different datasets, but they perform slightly better in the EK-100 dataset, which has a more complex, first-person perspective. This points to a complex issue: while the models have trouble applying what they've learned about contact detection to different datasets, they show a small improvement in the specific environment of EK-100 compared to the different scenarios in the SSv2 dataset. 

Expanding the range of datasets used in this field, the SAYCam dataset <cit.> stands out as an exciting opportunity for future research. This dataset is unique because it is shown from a child's point of view, capturing a variety of daily interactions and visual scenes. Although SAYCam is rich with diverse and unstructured content, it can be quite complex to analyze. This complexity poses challenges but also offers opportunities for SSL models. Using SAYCam could offer valuable insights into how these models process and understand visuals from a unique and very human perspective. This exploration fits well with the broader objective of improving how these models handle and make sense of varied, real-world visuals.

Among the broader challenges of video analysis, the task of accurately identifying contact interactions stands out as a critical area for advancement. The current discussion illuminates a potential pathway forward: integrating hand detection mechanisms to focus on contact-centric interactions. This approach proposes a targeted refinement of the models' capabilities, focusing on the specific, pivotal moments of physical contact within video sequences. By extracting and emphasizing these moments, models can develop a more refined understanding of interactions, potentially overcoming some of the limitations observed in datasets like EK-100. This strategy underscores a pivotal shift towards more specialized, context-aware models that prioritize the detection of meaningful, interaction-centric visual cues.

One of the major drawbacks of our research is that we only use CNN-based models. It would be worthwhile to investigate other architectures, especially ViT-based models in this aspect. As this field continues to evolve, the insights gain from these explorations will undoubtedly contribute to the development of more advanced, capable models. These models will be better equipped to navigate the complexities of real-world visual environments, marking significant progress in the quest for video interpretation and understanding.




splncs04




 Paper Title: Diff3DETR:Agent-based Diffusion Model for Semi-supervised 3D Object Detection.





 



Diff3DETR






[4]Corresponding Author


J. Deng et al.






MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China


Deep Space Exploration Lab

{dengjc,lujiahao}@mail.ustc.edu.cn,tzzhang@ustc.edu.cn


Diff3DETR: Agent-based Diffusion Model for Semi-supervised 3D Object Detection
    Jiacheng Deng10000-0003-2838-0378 Jiahao Lu1 0009-0006-3280-9447 Tianzhu Zhang1, 2 † 0000-0003-1856-9564
    
============================================================================================================




§ ABSTRACT 
3D object detection is essential for understanding 3D scenes. Contemporary techniques often require extensive annotated training data, yet obtaining point-wise annotations for point clouds is time-consuming and laborious. Recent developments in semi-supervised methods seek to mitigate this problem by employing a teacher-student framework to generate pseudo-labels for unlabeled point clouds.  However, these pseudo-labels frequently suffer from insufficient diversity and inferior quality. To overcome these hurdles, we introduce an Agent-based Diffusion Model for Semi-supervised 3D Object Detection (Diff3DETR). Specifically, an agent-based object query generator is designed to produce object queries that effectively adapt to dynamic scenes while striking a balance between sampling locations and content embedding. Additionally, a box-aware denoising module utilizes the DDIM denoising process and the long-range attention in the transformer decoder to refine bounding boxes incrementally. Extensive experiments on ScanNet and SUN RGB-D datasets demonstrate that Diff3DETR outperforms state-of-the-art semi-supervised 3D object detection methods. 







§ INTRODUCTION



3D object detection aims to localize and recognize 3D objects in 3D space to facilitate scene understanding, making it crucial for 3D applications such as autonomous driving <cit.>, AR/VR <cit.>, and robotic navigation <cit.>. The rapid development of deep learning-based methods <cit.>, including PointNet <cit.>, Transformer <cit.>, and DETR <cit.>, has significantly propelled advancements in 3D object detection. However, most existing approaches heavily rely on labeled point cloud data. Manually annotating vast amounts of 3D point cloud scenes is extremely time-consuming and labor-intensive, which could limit the potential for applying 3D object detection in larger-scale scenarios. 



To mitigate the dependence on annotated 3D point cloud data, semi-supervised methods <cit.> that utilize a small amount of labeled data alongside a large volume of unlabeled data are gaining attention and rapid development. Methods based on semi-supervised learning <cit.> leverage the untapped information in unlabeled point clouds to compensate for the information loss due to the scarcity of annotated data, thereby effectively enhancing the detection performance. Existing semi-supervised approaches <cit.> can be broadly categorized into two types: consistency-based methods <cit.> and pseudo-label-based methods <cit.>. For consistency-based methods, the core idea is to encourage consistency in the predictions for data augmented in different ways. For instance, SESS <cit.> enforces consensus on object location, semantic category, and size through three consistency losses between model outputs for differently augmented data. However, due to significant noise in model predictions, consistency constraints might lead to suboptimal results.



On the other hand, pseudo-label-based methods aim to select high-quality pseudo-labels from the model predictions on unlabeled data and then combine these with labeled data for model training, achieving more accurate detection results. 3DIoUMatch <cit.> and NESIE <cit.> equip detectors with global and multi-side localization quality estimation modules to assist in pseudo-label filtering and suppression strategies. Diffusion-SS3D <cit.> leverages a diffusion model <cit.> to randomly initialize noise sizes and labels to enhance the diversity and quantity of pseudo-labels and denoise the noisy boxes to improve the quality of pseudo-labels. The diffusion-based method <cit.> has achieved the best results, marking the diffusion model as a significant trend for the future development of semi-supervised 3D object detection.


Compared to 2D images, the expansive space and sparse distribution of point clouds in 3D scenes result in approximately 90% of spatial areas lacking point cloud coverage. The existing diffusion-based method <cit.> employs Farthest Point Sampling (FPS) on the point cloud to obtain a fixed number of object candidates,  thus avoiding sampling from empty areas which could lower the recall rate. However, compared to object queries in DETR <cit.>, these candidates struggle to learn sufficient content embedding across scenes. Moreover, the existing diffusion-based method <cit.> aggregates features within the initial boxes. Thus, noisy initial boxes can adversely affect feature aggregation, leading to sub-optimal outcomes. 



Based on the discussion above, we identify two critical aspects that need consideration and improvement for building more accurate diffusion-based semi-supervised 3D object detection models:

1) How to effectively model object candidates?

As shown in Figure <ref>(a), existing 3D object detection methods <cit.> primarily obtain candidates through two approaches: Farthest Point Sampling (FPS) and learnable object query <cit.>. FPS sampling is more likely to distribute candidates across areas where objects are located, reducing the rate of empty sampling. However, it cannot learn content embedding across scenes to aggregate object features effectively. On the other hand, learnable object queries can update and learn across scenes in the dataset but have a higher rate of sampling empty spaces, which lowers the object recall rate. Therefore, balancing the sampling position and the content learning is critical to modeling object candidates effectively.

2) How to aggregate the correct features to assist in the incremental refinement of noisy initial boxes?

Existing diffusion-based methods <cit.> generate initial bounding boxes using random Gaussian noise. However, as illustrated in Figure <ref>(b), these noisy initial boxes significantly differ from the ground truth object boxes. Consequently, the aggregated features often contain substantial errors, making it challenging to iteratively predict accurate object boxes. Therefore, it is crucial to appropriately expand the receptive field around the initial boxes to more accurately locate the correct target area and aggregate features, aiding in the denoising and correction of noisy boxes.


To achieve the above goals, we propose an agent-based diffusion model in a unified DETR architecture for semi-supervised 3D object detection, namely Diff3DETR, which consists of an agent-based object query generator and a box-aware denoising module.
Overall, our method employs the mean teacher framework, integrating the diffusion model and DETR architecture within a unified model. The randomness of the diffusion process generates a greater quantity of pseudo-labels. Simultaneously, the agent-based object query generator creates object queries that balance sampling locations and content embedding. In the box-aware denoising module, features are aggregated within a long-range perceptive field to iteratively optimize noisy boxes, achieving accurate object predictions.
More specifically, the agent-based object query generator initially establishes learnable object agents to obtain satisfactory content embeddings, where object agents could dynamically adapt to specific scenes through interaction with scene features. The object queries are derived through linear interpolating FPS-sampled locations with object agents. In the box-aware denoising module, the DDIM <cit.> denoising process and transformer decoder are ingeniously intertwined. Moreover, to ensure each object query focuses on the designated object area during the query process, we bind object queries with the noisy box locations, enhancing the positional dependency of object queries. 


To sum up, the contributions of this work can be summarized as follows: 

(1) We introduce an agent-based diffusion model within a unified DETR framework, which includes an agent-based object query generator and a box-aware denoising module. To the best of our knowledge, this is the first diffusion-based DETR framework in the semi-supervised 3D object detection field. 

(2) We develop an agent-based object query generator to generate object queries that better adapt to dynamic scenes while balancing sampling locations and content embedding. Additionally, we design a box-aware denoising module that leverages the incremental refinement capabilities of the DDIM denoising process and the long-range attention of the transformer decoder to denoising initial boxes for accurate 3D object detections. 

(3) Extensive experimental results on the ScanNet and SUN RGB-D datasets demonstrate that Diff3DETR achieves superior performance and outperforms existing state-of-the-art methods.



§ RELATED WORK


In this section, we provide a concise overview of methodologies related to diffusion models for perception tasks and semi-supervised 3D object detection.








Diffusion Models for Perception Tasks.

Diffusion models have demonstrated remarkable success in image generation <cit.>. Consequently, researchers have begun exploring the integration of diffusion models into perception tasks. Pix2Seq-D <cit.> utilizes the Bit Diffusion model <cit.> to conduct panoptic segmentation <cit.> tasks across images and videos.  <cit.> utilizes diffusion models, showcasing their efficacy in extracting discriminative features for classification tasks. DiffusionDet <cit.> frames object detection as a noise-to-box task, wherein high-quality object bounding boxes are generated by progressively denoising randomly generated proposals. In line with this work, Diffusion-SS3D <cit.> harnesses the power of diffusion models in semi-supervised 3D object detection settings, aiming to offer a novel approach to generate more dependable pseudo-labels. In this work, we extend the approach of Diffusion-SS3D by proposing the first diffusion-based DETR framework, which includes an agent-based object query generator and a box-aware denoising module.

Semi-supervised 3D Object Detection.

While fully supervised methods <cit.> have demonstrated superior performance, they are often constrained by the labor-intensive process of bounding box annotations, presenting significant practical limitations. Consequently, several semi-supervised 3D object detection techniques <cit.> have emerged to address this issue. SESS <cit.> represents the pioneering effort in semi-supervised 3D object detection. By enforcing consistency between the outputs of the mean teacher <cit.>, SESS effectively learns from unlabeled data. 3DIoUMatch <cit.> introduces confidence-based filtering and IoU prediction strategies to select high-quality pseudo-labels generated by the teacher model. However, many existing methods heavily rely on the teacher model for pseudo-label generation, limiting their ability to identify bounding boxes beyond the scope of teacher model predictions. To mitigate this challenge, models based on the Diffusion model have been proposed. Diffusion-SS3D <cit.> leverages the diffusion model for semi-supervised 3D object detection, treating the task as a denoising process to enhance the quality of pseudo-labels. In this work, we adhere to the diffusion-based methodology and employ the agent-based object query generator to produce object queries better suited for complex and dynamic environments. Additionally, we devise the box-aware denoising module to enhance refinement capabilities.




§ METHODOLOGY


In this section, we introduce the details of our Diff3DETR. Section <ref> covers the preliminaries. Section <ref> describes the Diff3DETR framework, focusing on how the teacher detector generates high-quality pseudo-labels to assist the student detector's training and the computational specifics of both models. Section <ref> elaborates on the structural details of the Diff3DETR detector.



 §.§ Preliminaries


Semi-supervised 3D object detection.
Given the point cloud of a scene as input, the goal of 3D object detection is to classify and localize amodal 3D bounding boxes for objects within the point cloud. The point cloud data is 𝐱∈ℝ^n × 3, where n denotes the number of points. Within the semi-supervised framework, we are provided with N training samples. The samples involve a set of N_l labeled scenes {𝐱_i^l, 𝐲_i^l}_i=1^N_l and a larger set of N_u unlabeled scenes {𝐱_i^u}_i=1^N_u. The ground-truth annotations 𝐲_i^l encompass K objects {𝐛_k^l, l_k}_k=1^K within 𝐱_i^l, with 𝐛 and l signifying a collection of bounding box parameters and semantic class labels with a total of N_cls classes. Specifically, the bounding box 𝐛 is formulated as 𝐛 = {𝐛_c, 𝐛_s, 𝐛_o}, where 𝐛_c = {c_x, c_y, c_z} delineates the centroid coordinates, 𝐛_s = {s_l, s_w, s_h} represents the object dimensions, and 𝐛_o is the object orientation along the upright-axis. 

Diffusion model.
Diffusion models, a class inspired by non-equilibrium thermodynamics <cit.>, present a novel approach in generating data by progressively introducing noise into the data samples. This process is mathematically modeled as a Markov chain <cit.> consisting of T diffusion steps, with the forward diffusion process described by:


    q(𝐳_T|𝐳_0) = 𝒩(𝐳_T| √(α̅_T)𝐳_0, (1 - α̅_T)𝐈),


where 𝐈 is identity matrix. In this equation, the original data sample 𝐳_0 is transformed into a noisy latent representation 𝐳_T through the application of additive noise. The noise scale α_s := 1 - β_s and α̅_T := ∏_s=1^Tα_s are computed as the product of individual noise scales from the first diffusion step to step T, and β_s is a predetermined variance schedule. A neural network f_θ(𝐳_T) is then trained to reverse this diffusion process by predicting the noiseless data 𝐳_0 from the noisy data 𝐳_T, optimizing an L_2 loss objective:


    ℒ_train = 1/2f_θ(𝐳_T) - 𝐳_0^2


During the inference phase, the model iteratively reconstructs the original data sample from its noisy counterpart by iteratively applying an update rule. The sequence of transformations 𝐳_T →𝐳_T-Δ→…→𝐳_0 refines the data sample at each step until the original data is retrieved. A detailed formulation of diffusion models is provided in the supplementary materials. 






 §.§ The Teacher-Student Framework of Diff3DETR



The proposed Diff3DETR is a novel diffusion-based framework for semi-supervised 3D object detection. As illustrated in Figure <ref>, Diff3DETR first processes the input point cloud through data augmentation of varying intensities, subsequently feeding them into two branches: the student model and the teacher model. The teacher model introduces Gaussian noise to the 3D bounding boxes and reverses the diffusion process as in DDIM <cit.> to generate reliable pseudo bounding boxes. 


For the student model, the student model takes both labeled and unlabeled point clouds as input and adds Gaussian noise to the ground truth boxes and pseudo boxes to obtain noisy boxes. The student Diff3DETR detector then takes these noisy boxes and the scene point cloud as input, directly predicting the final object boxes without undergoing a multi-round diffusion sampling process. The prediction results 𝐲̂^l and 𝐲̂^u are supervised by ground truths 𝐲^l and pseudo-labels 𝐲̃^u, respectively. Furthermore, the parameters of the teacher Diff3DETR detector are updated using an Exponential Moving Average (EMA) strategy based on the parameters of the student Diff3DETR detector.

To more clearly describe the computational processes of the teacher model and the student model, we detail the computation specifics for both in Algorithm <ref> and Algorithm <ref>, respectively. Compared to the teacher model, the student model mainly differs in the following aspects: 1) the generation of the noisy box's size and label is achieved by incrementally adding Gaussian noise to the GT/pseudo boxes; 2) instead of using a multi-step DDIM diffusion sampling process for denoising the noisy boxes, it directly predicts accurate boxes and computes the loss with ground truths and pseudo-labels. The overall loss function ℒ for the student model is defined as follows:

    ℒ = ℒ_l(x^l, y^l) + λℒ_u(x^u, ỹ^u),

where ℒ_l and ℒ_u are the detection loss functions for labeled samples and unlabeled samples, respectively. Both loss functions are used in 3DIoUMatch <cit.> for bounding box regression and classification, combined through a loss weight hyper-parameter λ. Model inference is completed through the student model by adding DDIM diffusion sampling for multi-step iterative denoising, similar to the inference step in Algorithm <ref>.





 §.§ The Detector of Diff3DETR


The overall architecture of the Diff3DETR detector is depicted in Figure <ref>. Initially, input point clouds are downsampled and feature-extracted using a PointNet++ encoder <cit.>. These downsampled point clouds are further processed with Farthest Point Sampling (FPS) to identify noisy centers, which serve as preliminary center points for object boxes. The coordinates of these noisy centers are then interpolated in an agent-based object query generator to form object queries for detecting scene targets. Noisy boxes are comprised of noisy centers, along with noisy sizes and noisy labels, which are directly initialized using Gaussian noise. Object queries and noisy boxes undergo prediction of the actual boxes and gradual iterative denoising of noisy boxes within a box-aware denoising module through a decoder layer and the DDIM <cit.> layer, respectively.

Agent-based object query generator.
The agent-based object query generator uniformly samples the normalized scene space at a resolution of L × W × H grid points and assigns a learnable variable to each grid point. The object agents are formed by adding each grid point's learnable variable to its grid point position coordinates encoded by a two-layer MLP. The initial object agents are denoted as G_0 ∈ℝ^N_G × C, where N_G = L× W × H denotes the number of object agents and C represents the number of feature channels.



 To facilitate dynamic adaptability of object agents across various scenes, we design an agent adaptor based on the attention mechanism <cit.>, which can be formulated as follows:


    Q = Attention(Q, K, V) = softmax(QK^T/√(d_k))V,


where Q, K, and V represent the queries, keys, and values, respectively, and d_k denotes the dimension of the keys. In this context, queries represent G_0, and keys and values correspond to the high-level point cloud semantic features from the encoder's intermediate layers. The updated object agents, denoted as Ĝ, are then interpolated through the noisy centers' coordinates 𝐛^noise_c to obtain the object queries Ô∈ℝ^N_Q × C, where N_Q represents the number of object queries and C is the number of feature channels.





Box-aware denoising module.
Noisy boxes comprise three components: noisy centers, noisy sizes, and noisy semantic labels. As shown in Figure <ref>, object queries and noisy boxes are fed into the decoder layer, where object queries are updated through self-attention and cross-attention mechanisms to predict accurate boxes. 

Given object query set Ô = {q_i}, noisy boxes 𝐛^𝐧𝐨𝐢𝐬𝐞 = {𝐛_c^noise, 𝐛_s^noise, 𝐛_o^noise} and point cloud features P = {p_k}, the output feature of the multi-head self-attention of each query element is the aggregation of the values that weighted by the attention weights, formulated as:

    Self-Att(q_i, {q_k}) = ∑_h=1^H W_h ( ∑_k=1^K A_h,i,k· V_h q_k ),


    A_h,i,k = exp( (Q_h (q_i+𝐌𝐋𝐏(b^noise_i)))^⊤ (U_h (q_k+𝐌𝐋𝐏(b^noise_k))) )/∑_k=1^Kexp(  (Q_h (q_i+𝐌𝐋𝐏(b^noise_i)))^⊤ (U_h (q_k+𝐌𝐋𝐏(b^noise_k))) ),

where h indexes over attention heads, A_h is the attention weight, Q_h, V_h, U_h, W_h indicate the query projection weight, value projection weight, key projection weight, and output projection weight, respectively. The output feature of the multi-head cross-attention of each object query are formulated as:

    Cross-Att(q_i, {p_k}) = ∑_h=1^H W_h ( ∑_k=1^KA̅_h,i,k· V_h p_k ),


    A̅_h,i,k = exp( (Q_h q_i)^⊤ (U_h p_k)+𝐑_i,k)/∑_k=1^Kexp( (Q_h q_i)^⊤ (U_h p_k)+𝐑_i,k),

where 𝐑_i,k represents the 3D Vertex Relative Position Encoding (3DV-RPE) of the ith object query's corresponding noisy box with respect to the kth point in the point cloud. Inspired by the V-DETR <cit.> approach, 3DV-RPE encodes a point by its relative position to the target object and is crucial for augmenting the transformers to capture the spatial context of the tokens. For further details, please refer to V-DETR <cit.>. 
The denoising process for noisy boxes occurs in the DDIM layer as detailed in Algorithm <ref>, predicting the noisy boxes for next step t_next based on the predicted boxes and noisy boxes at current step t_cur.  













§ EXPERIMENTS




 §.§ Datasets.

In our study, we conduct evaluations on two primary datasets: ScanNet <cit.> and SUN RGB-D <cit.>, employing evaluation protocols from existing semi-supervised 3D object detection literature. ScanNet is an established dataset for 3D indoor scene benchmarking and is composed of 1,201 training and 312 validation scenes, reconstructed from 2.5 million high-resolution RGB-D images. Our study places an emphasis on the 18 semantic classes as aligned with prior studies. The SUN RGB-D dataset, another significant 3D benchmark, consists of 5,285 training scenes and 5,050 validation scenes. We assess the models across 10 object classes.



 §.§ Evaluation metrics.

For the model evaluation, we split the datasets into partitions with various proportions of labeled and unlabeled data to support semi-supervised learning (SSL). Specifically, we allocate 5%, 10%, 20%, and 100% of labeled data for the ScanNet evaluation, and 1%, 5%, 10%, and 20% for SUN RGB-D. Performance metrics are calculated using the mean Average Precision (mAP). mAP@0.25 at an Intersection over Union (IoU) threshold of 0.25 and mAP@0.5 are reported. These metrics provide insights into precision at a granular level for object detection tasks. The evaluation is conducted across three random data splits to ensure robustness, and report averaged performance and the standard deviation.



 §.§ Implementation details

For the detector, we establish a grid of object agents with (L, W, H) = (16, 16, 4) and set the number of noisy centers and object queries to 128. 

For the diffusion process, we set the maximum timesteps to 1000. Like Diffusion-SS3D <cit.>, we set the mean of the sampling sizes to 0.25. We set the random sampling mean for noisy labels to the inverse of the respective class number. Our teacher model employs a dual-step DDIM sampling technique (T=2) to generate pseudo-labels and produce final evaluation results. 











 §.§  Comparison with State-of-the-art Methods

In this section, we compare our Diff3DETR with state-of-the-art approaches on ScanNet dataset and SUN RGB-D dataset.

Results on ScanNet dataset.
Table <ref> displays a comparison of our approach against the current state-of-the-art methods on the ScanNet validation dataset, achieving leading results across different ratios of labeled data. Through meticulous design in the generation of object queries and more accurate incremental refinement by the box-aware denoising module for noisy boxes, our method surpasses the best existing method by 1.6% mAP@0.25 and 1.3% mAP@0.5 with only 5% labeled data. Additionally, Figure <ref>(a) presents qualitative results of our method from the ScanNet dataset.

Results on SUN RGB-D dataset.
Table <ref> demonstrates the comparative performance of our method against the current state-of-the-art methods on the SUN RGB-D validation dataset, where it achieves leading results across different ratios of labeled data. Specifically, our method outperforms the best existing method by 1.8% mAP@0.25 and 1.3% mAP@0.5 with only 5% labeled data. Additionally, Figure <ref>(b) presents qualitative results from scenes within the SUN RGB-D dataset, showcasing the capabilities of our method.







 §.§ Ablation Study






Evaluation of the model with different designs. 
In Table <ref>, we present a series of ablation studies to validate the effectiveness of our designs. [A] represents the baseline IoU-aware VoteNet model <cit.> without employing DDIM iterative denoising <cit.>. [B] illustrates that the agent-based object query generator produces improved object queries which aid the model in achieving a 0.9% increase in mAP@0.25 and a 0.6% rise in mAP@0.5. The contrast between [C] and [B] confirms the significance of the agent adaptor for dynamically adapting to the scene. [D] further incorporates the proposed box-aware denoising module, which assists the network in aggregating features from the correct regions and incrementally refining predicted boxes, resulting in an effective increase of 0.8% mAP@0.25 and 0.8% mAP@0.5. [E] validates that 3D vertex relative position encoding is conducive to focusing the object queries on point cloud regions surrounding noisy boxes. [F] represents the complete Diff3DETR model which achieves the best performance among all variants.



Effectiveness of the agent-based object query generator. 
Object agents are initialized as learnable vectors uniformly distributed in normalized 3D space, and their density across the dimensions of length, width, and height significantly influences the quality of subsequent object query generation. Figure <ref> illustrates the visualization of grid point distributions under different [L, W, H] resolution settings for length, width, and height. Results from Table <ref> on the ScanNet dataset indicate that the [16, 16, 4] distribution achieves the best performance. This outcome is attributed to the fact that objects within scenes have a notably lower density distribution in vertical height compared to on the horizontal plane, hence allocating more object agents across the horizontal plane aids in modeling a wider variety of spatial semantic information.

Effectiveness of the box-aware denoising module.
The box-aware denoising module ingeniously integrates the DETR decoder architecture with the DDIM structure, where the number of blocks in both the decoder layer and DDIM significantly affects the model's ability to decode the scene's object boxes. Table <ref> presents ablation studies on the number of blocks in the decoder layer and DDIM, showing that, overall, more decoder blocks and DDIM iterations per cycle yield better detection results. However, increasing the number of blocks in both the decoder layer and DDIM considerably impacts the model's training and inference speed. Therefore, we seek a trade-off between accuracy and computational cost, selecting #DL=3 and #DDIM=2 as the final model configuration.






 §.§ Limitations


The diffusion model diffuses object boxes from ground-truth boxes to a random distribution, and the model learns to reverse this noising process. Applying the diffusion model to semi-supervised 3D object detection tasks offers several inherent merits of the diffusion model. First, the diffusion to a random distribution can generate more diverse pseudo-labels. Second, the denoising process coincides with the decoder process of the detection framework, promoting mutual enhancement. However, the slow denoising process of the diffusion model requires more computational resources for training and inference, hindering the algorithm's potential application in real-time devices and large-scale scenes. A more detailed analysis and discussion regarding model overhead are conducted in the supplementary materials.




§ CONCLUSION


In this paper, we introduce a novel agent-based diffusion model within a unified DETR framework for semi-supervised 3D object detection. Our proposed Diff3DETR comprises an agent-based object query generator and a box-aware denoising module. The agent-based object query generator is designed to produce object queries that effectively adapt to dynamic scenes while striking a balance between sampling locations and content embedding. Meanwhile, the box-aware denoising module utilizes the DDIM denoising process and the long-range attention in the transformer decoder to incrementally refine bounding boxes, thereby achieving better results. Extensive experiments on the ScanNet and SUN RGB-D benchmarks underline the superiority of our Diff3DETR.



§ ACKNOWLEDGEMENTS


This work was partially supported by the National Defense Science and Technology Foundation Strengthening Program Funding (Grant 2023-JCJQ-JJ-0219), the National Nature Science Foundation of China (NSFC 62121002), and Youth Innovation Promotion Association CAS.









splncs04



 Paper Title: MIST: A Simple and Scalable End-To-End 3D Medical Imaging Segmentation Framework.




MIST



A. Celaya et al.


The University of Texas MD Anderson Cancer Center, Houston TX 77030 Rice University, Houston TX 77005


MIST: A Simple and Scalable End-To-End 3D Medical Imaging Segmentation Framework
    
Adrian Celaya1,2 Evan Lim1 Rachel Glenn1 Brayden Mi1 Alex Balsells1,2 Tucker Netherton1 Caroline Chung1 Beatrice Riviere2 David Fuentes1

    August 5, 2024
==============================================================================================================================================


§ ABSTRACT 
Medical imaging segmentation is a highly active area of research, with deep learning-based methods achieving state-of-the-art results in several benchmarks. However, the lack of standardized tools for training, testing, and evaluating new methods makes the comparison of methods difficult. To address this, we introduce the Medical Imaging Segmentation Toolkit (MIST), a simple, modular, and end-to-end medical imaging segmentation framework designed to facilitate consistent training, testing, and evaluation of deep learning-based medical imaging segmentation methods. MIST standardizes data analysis, preprocessing, and evaluation pipelines, accommodating multiple architectures and loss functions. This standardization ensures reproducible and fair comparisons across different methods. We detail MIST's data format requirements, pipelines, and auxiliary features and demonstrate its efficacy using the BraTS Adult Glioma Post-Treatment Challenge dataset. Our results highlight MIST's ability to produce accurate segmentation masks and its scalability across multiple GPUs, showcasing its potential as a powerful tool for future medical imaging research and development.





§ INTRODUCTION

Medical imaging segmentation is a highly active area of research. Since its introduction in 2015, the U-Net architecture has received nearly 90,000 citations on Google Scholar <cit.>. Other architectures and frameworks like nnUNet have achieved state-of-the-art accuracy in several medical imaging benchmarks like the Brain and Tumor Segmentation (BraTS) and Medical Segmentation Decathlon (MSD) <cit.>. Since nnUNet's introduction in 2018, several innovative approaches for medical imaging segmentation have emerged. These innovations include new architectures like vision transformers <cit.> and loss functions like the boundary and generalized surface losses <cit.>. Despite these recent advances, there remains a lack of standardized tools for testing and evaluating different approaches for medical imaging segmentation. For example, several works like <cit.> report superior performance to nnUNet while <cit.> refutes these results. This inconsistency makes it difficult to evaluate and assess claims of state-of-the-art performance for new research in deep learning-based medical imaging segmentation.


To address this inconsistency, we propose the Medical Imaging Segmentation Toolkit or MIST, a simple, modular, and end-to-end deep learning-based medical imaging segmentation framework that allows researchers to seamlessly train, test, and evaluate existing and new methods on their data in a consistent and reproducible way. MIST has a rule-based data analysis and preprocessing pipeline that it makes available to multiple architectures, loss functions, and training parameters. Additionally, MIST has a standardized and flexible evaluation pipeline that computes multiple metrics like the Dice, Haudorff distance, and surface Dice for any user-defined segmentation class. This standardization of several pipelines allows for fair comparisons between methods and their effect on segmentation accuracy for any given dataset.



§ METHODS
 
In this section, we describe the required data format for MIST, the different pipelines in MIST, and the auxiliary features, such as evaluation, postprocessing, and test-time inference. We also provide the specific settings we use to train on the BraTS challenge data.



 §.§ Data Format
 
MIST uses the same file structure as the BraTS challenge datasets <cit.>. That is, MIST assumes that the train and test (optional) data directories are formatted so that each sub-directory corresponds to one patient and that each imaging modality (i.e., T1, T2, or CT) is its own NIfTI file. The labels should all be in a single NIfTI file in the same sub-directory for each patient. Once the dataset is in the correct format, the next step is to prepare a small JSON file containing the details of the dataset. These include information like the imaging modality, naming conventions for the images and masks, labels, and the final classes that we use for evaluation (i.e., labels 1, 2, and 3 for whole tumor). An example of this JSON file is provided in the documentation page https://mist-medical.readthedocs.io/en/latest/getting_started/here. MIST also provides support for converting datasets that are in CSV or MSD format. 



 §.§ Pipelines
 
MIST contains three main pipelines: Data Analysis, Preprocessing, and Training. These pipelines can all run at once with a single command or individually. We also provide several auxiliary pipelines for evaluation, postprocessing, and test-time inference.



  §.§.§ Data Analysis
 
The data analysis pipeline gathers parameters about the dataset including cropping to foreground, target spacing, patch size selection, normalization parameters, and is described further below. The results of this analysis are saved in a  file.



  
Cropping to Foreground Often, zeros or useless background information surround the region of interest. For example, zeros surround the region outside of the brain in the BraTS datasets, and the table and air surrounding the body in CT imaging do not provide relevant information. From a computational perspective, it is desirable to crop the images to the foreground to make them smaller. Additionally, cropping to the foreground also reduces the sparsity of the dataset. MIST crops each image by first windowing it with its 33rd and 99.5 percentile values, then using an Otsu filter to produce a foreground threshold. We then crop the image according to the binary mask resulting from this threshold (i.e., one if the voxel intensity is greater or equal to the threshold). To determine if we need to use cropping, we check to see if, on average, the resulting volume reduction is at least 20%.



  
Target Spacing To determine the target spacing for a dataset, we first look at the distribution of the voxel spacings along each axis. Our initial target spacing is the median spacing in each direction. However, suppose the resulting target spacing is anisotropic (i.e., the maximum and minimum spacing ratio is greater than three), then we take the 10th percentile spacing along the lowest resolution axis in that case. However, if resampling with this selected target spacing results in any image taking up more than 2 GB of memory, then we increase all components of our target spacing by 25%. We repeat this process until none of the resampled images take up more than 2 GB of memory. This last step is important because it conserves computational resources. Larger images take longer to load from disk and require significantly more resources to manipulate (i.e., patch extraction and augmentation).



  
Patch Size Selection We set the patch size for training from the median resampled image size by taking the nearest power of two less than or equal to each dimension in the median image size up to a maximum patch size. The default value for this maximum patch size is 256×256×256. However, users can adjust this maximum patch size based on their knowledge of their computing resources. 



  
Normalization Parameters If the dataset consists of MR or other non-CT images, then our windowing parameters are the 0.5 and 99.5 percentile values of either the entire image or the non-zero values of the image if, on average, the ratio of the number of non-zero to zero-valued voxels is less than 0.2. We use the mean and standard deviation with the same rules regarding the non-zero values for normalization. On the other hand, if we are dealing with CT images, then we compute the 0.5 and 99.5 percentile values, mean, and standard deviation of all of the voxels over the entire dataset corresponding to regions labeled as non-zero in the ground truth segmentation masks.




  
Other Checks In addition to the previous analysis steps, MIST checks to see that the header information in the images and masks agree and, if there are multiple modalities, that the headers in each image agree. If there is a disagreement between the headers, MIST prints a warning with the patient ID to the console and excludes the example from training. MIST also computes equal-weighted class weights for each class based on the amount of voxels belonging to each class over the entire dataset. These class weights are primarily informational but can be used for weighted loss functions. 



  §.§.§ Preprocessing
 
The preprocessing pipeline takes the information from the analysis pipeline to crop the images based on their foreground mask (if called for), reorient to right-anterior-inferior (RAI), resample to the target spacing, and window and normalize the intensity values. This pipeline can also compute the distance transform maps (DTMs) for each class in the ground truth masks or apply bias correction for MR images. Users also have the option to skip this pipeline if, for example, their NIfTI files contain preprocessed images already. Figure <ref> illustrates the workflow for this pipeline.





  
Reorient & Resample After optionally cropping or bias correcting each image, the preprocessing pipeline will reorient each image and mask to the RAI orientation. For resampling images to the target spacing, we use third-order spline interpolation. We apply one-hot encoding and use linear interpolation to each channel for masks. If the current image spacing is anisotropic, we first resample along the low-resolution axis using nearest-neighbor interpolation and then apply the appropriate interpolation along the other two axes <cit.>. We implement these operations using SimpleITK to optimize the speed and computational performance of these operations <cit.>. 



  
Window & Normalize Before normalizing the voxel intensities in each image, the preprocessing pipeline windows each image by either the image's 0.5 and 99.5 percentile or, if we are using CT images, by the precomputed windowing parameters. After windowing, MIST applies z-score normalization with either the image's mean and standard deviation or the precomputed normalization parameters for CT images. Again, in the case that the ratio of non-zero to zero-valued voxels is less than 0.2, then the mean and standard deviation are computed from the non-zero voxels only, and we multiply the resulting images by the non-zero mask (different from foreground mask) to preserve the zero-valued voxels.



  
Distance Transform Maps A DTM represents the distance between each voxel and an object's closest boundary or edge. The values in a DTM are positive on the exterior, zero on the boundary, or negative in the object's interior. As a result, they can provide helpful information during training. Several loss functions and shape-aware networks use DTMs to improve the accuracy of medical imaging segmentation methods <cit.>. MIST allows users to pre-compute the DTMs for each label in the ground truth segmentation masks. Each channel in the resulting DTM represents the DTM for that label (i.e., channel one is the DTM for label one). If a label does not exist in the image, the DTM will be a 3D array with all values being the diagonal distance of the image. Additionally, MIST allows users to output normalized DTMs whose values range from -1 to 1. 



  §.§.§ Training

Once preprocessing is complete, MIST uses a five-fold cross-validation to train and evaluate a model on the dataset. In other words, for each fold we set aside 20% of the data as an independent test set and use the other 80% of the data for training. By default, MIST sets aside 10% of the training data as a validation set, but users can adjust this percentage. Additionally, users have the option to adjust the number of folds and can even specify custom folds if, for example, they are running leave-one-institution-out cross-validation.


MIST's default behavior is to use all available GPUs. When MIST runs multi-GPU training, it uses data parallelism via PyTorch's DistributedDataParallel package <cit.>. Additionally, MIST uses the Nvidia Data Loading Library (DALI) to handle data loading, patch extraction, and random augmentation. This feature further optimizes its computational performance by running most of the operations on the GPUs. 



  
Network Architectures MIST supports six different network architectures. These architectures include nnUNet (default) <cit.>, U-Net <cit.>, Swin UNETR <cit.>, and PocketNet <cit.>. MIST also provides users with options like adding deep supervision <cit.> or variational autoencoder regularization <cit.> to most of these architectures. Other regularization features like L2 and L1 regularization and gradient clipping are available for all architectures. For more details about the supported architectures (and their differences), please refer to our https://mist-medical.readthedocs.io/en/latest/advanced_topics/#network-architecturesdocumentation page. Finally, MIST's modular design allows users to implement their own architectures and integrate them into MIST with minimal effort. 




  
Loss Functions Like with the choice of network architecture, MIST allows users to choose from a variety of loss functions like Dice with Cross Entropy (default) <cit.>, the Generalized Dice Loss <cit.>, and boundary-based loss functions like the Boundary, Hausdorff, and Generalized Surface losses <cit.>. Again, MIST's modular design allows researchers to implement custom loss functions and seamlessly use them within the MIST framework. 



  
Other Training Options Finally, MIST gives users the flexibility to choose different optimizers like Adam (default) <cit.>, stochastic gradient descent, and AdamW <cit.>. MIST also supports several learning rate schedulers, with a constant learning rate of 0.0003 being the default. Additional features including transfer learning are included in our documentation page https://mist-medical.readthedocs.io/en/latest/here.



  §.§.§ Auxiliary Pipelines




  
Evaluation Once training is complete, the evaluation pipeline launches to assess the accuracy of the predictions with respect to the final classes defined by the user in their dataset JSON file. By default, these metrics are the Dice similarity coefficient and the 95th percentile Hausdorff distance (HD95). Other metrics like the surface Dice and average surface distance are also available. The output of the evaluation pipeline is a CSV file called  where each row shows the accuracy for each class and metric for one patient. The last five rows give the mean, standard deviation, median, and 25th and 75th percentiles for all of the metrics and classes. The evaluation pipeline is also available as a stand-alone pipeline that can run on any set of predictions. 



  
Postprocessing Postprocessing is available as an optional stand-alone pipeline. This pipeline allows users to specify which labels to apply a user-specified set of operations. These operations include the removal of small objects, only keeping the top-k components (adjustable value of k), morphological cleaning, and filling holes with a specified value. Once the postprocessing pipeline finishes, the evaluation pipeline launches and compares the change in accuracy to the baseline results. MIST computes this change in accuracy as a weighted average of the specified metrics. 



  
Test-Time Inference Finally, MIST provides a stand-alone inference pipeline that takes as input one or more MIST models and a CSV or JSON file with the paths to test-time data. This pipeline writes test-time predictions to a user-specified output directory. 



 §.§ BraTS 2024 Training Protocols
 
Our choice of architecture is the Pocket nnUNet with deep supervision (two supervision heads) and with residual convolution blocks <cit.>. MIST automatically selects a patch size of 128×128×128. We use eight NVIDIA H100 GPUs with a batch size of 32 uniformly distributed across the GPUs. Additionally, we use L2 regularization with a penalty parameter equal to 1e-5. Our choice of loss function is the Dice with Cross Entropy loss. While the default number of epochs per fold for MIST is 1,000, we train for 15,000 epochs per fold. We use a cosine learning rate schedule with an initial learning rate of 0.001 <cit.>. Within each fold, we set aside 2.5% of the training data (27 patients) as a validation set to select the best model. Once training is complete, MIST runs inference on the challenge validation data using test time augmentation (flipping along each axis and averaging each prediction) for each model. Inference uses sliding windows with an overlap of 0.5 with Gaussian blending (σ=0.125). The predictions from all five models are averaged to produce a final prediction. All other options and hyperparameters are left at their default values. Please refer to MIST's https://mist-medical.readthedocs.io/en/latest/documentation for these values.



§ RESULTS
 


 §.§ Accuracy

We use MIST and the parameters described in Section <ref> to perform a five-fold cross-validation (CV) with the BraTS Adult Glioma Post-Treatment Challenge data <cit.>. Table <ref> gives the results of this five-fold CV in terms of the Dice score and HD95. Tables <ref> and <ref> give the accuracy of the predictions in the validation set in terms of the same metrics and their lesion-wise versions.









 §.§ Computational Performance

In addition to the accuracy of our predictions, we also report the scalability of MIST for different numbers of A100 and H100 GPUs for multiple batch sizes. Figures <ref> and <ref> show these results for A100 and H100 GPUs, respectively. In these figures, we see that MIST effectively scales for multiple GPUs. This scaling is especially evident for the H100 GPUs. It is also important to note that we observe a roughly two times speed up with the H100 GPUs, which is in line with what Nvidia reports in their comparisons between the two types of graphics cards. 







§ DISCUSSION

Our results indicate that models trained using MIST can produce accurate adult post-operative gliomas, achieving median Dice scores of at least 0.9 for all of the segmentation classes in our five-fold CV. We choose to submit the raw output of our MIST pipeline to the BraTS 2024 competition, omitting the use of post-processing. However, future submissions to the continuous evaluation and competitions will explore different post-processing strategies with MIST by using its built-in post-processing pipeline. Indeed, previous BraTS competitions cite replacing small objects belonging to the ET class with the SNFH label as beneficial <cit.>. Additionally, we may see improvements in accuracy by using information from DTMs using boundary-based losses like the Generalized Surface Loss. Finally, different architectures like the multi-grid inspired FMG-Net or vision transformer-based architectures like Swin UNETR are also available on MIST <cit.>. These architectures may also improve our results.

MIST is able to utilize multiple GPUs to speed up training efficiently. This efficient use of multiple GPUs is especially true for H100s, where we see a nearly six times speed up when going from one to eight GPUs. The ability to take advantage of multiple GPUs and compute nodes effectively will be essential to keep training times manageable as larger datasets like TotalSegmentor become available <cit.>. As the size and quality of publicly available 3D medical imaging datasets increase, the creation of foundation models for medical imaging segmentation will be beneficial to speed up training through transfer learning <cit.>. MIST already supports transfer learning with other MIST models. Given this capability and its scalability, MIST can be instrumental in the creation of future foundation models.

In conclusion, our results show that MIST is capable of achieving accurate segmentation results while also scaling well to multiple GPUs by effectively taking advantage of technologies like PyTorch's DDP and Nvidia's DALI loader. MIST's simplicity, accuracy, and scalability have the potential to make MIST a valuable tool in the development of foundation models for medical imaging tasks like segmentation and radiation treatment planning. MIST's modular design makes it easy for researchers to add new loss functions or architectures to its pipelines. This modularity and easy integration into MIST's existing pipelines allow for new research to be evaluated and compared to existing methods fairly and consistently. MIST is still under active development. We encourage researchers and the broader medical imaging community to contribute to its development by using it, providing feedback, or contributing new features. 



§ AVAILABILITY & ATTRIBUTION

MIST is an open-source package (Apache 2.0 license) and is available on https://github.com/mist-medical/MISTGitHub or https://pypi.org/project/mist-medical/PyPI. Please cite this manuscript, <cit.>, and <cit.> if you use MIST for your own publications.



§ ACKNOWLEDGMENTS

The Department of Defense supports Adrian Celaya through the National Defense Science & Engineering Graduate Fellowship Program. This research was partially supported by the Tumor Measurement Initiative through the MD Anderson Strategic Research Initiative Development (STRIDE), NSF-2111147, NSF-2111459, and NIH R01CA195524


splncs04




 Paper Title: Autonomous LLM-Enhanced Adversarial Attack for Text-to-Motion.


Rotational influence on fermions within negative curvature wormholes
     Abdullah Guvendi^ 1[ E-mail: abdullah.guvendi@erzurum.edu.tr  ] ,  Semra Gurtas Dogan^ 2[ E-mail: semragurtasdogan@hakkari.edu.tr (Corr. Auth.) ] ,  R. L. L. Vitória^ 3[ E-mail: ricardo-luis91@hotmail.com ]   

	^ 1 Department of Basic Sciences, Erzurum Technical University, 25050, Erzurum, Türkiye

	^ 2 Department of Medical Imaging Techniques, Hakkari University, 30000, Hakkari, Türkiye

	^ 3 Faculdade de Física, Universidade Federal do Pará, Av. Augusto Corrêa, Guamá, Belém, PA 66075-110, Brazil
    
============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
	Human motion generation driven by deep generative models has enabled compelling applications, but the ability of text-to-motion (T2M) models to produce realistic motions from text prompts raises security concerns if exploited maliciously. Despite growing interest in T2M, few methods focus on safeguarding these models against adversarial attacks, with existing work on text-to-image models proving insufficient for the unique motion domain. In the paper, we propose ALERT-Motion, an autonomous framework leveraging large language models (LLMs) to craft targeted adversarial attacks against black-box T2M models. Unlike prior methods modifying prompts through predefined rules, ALERT-Motion uses LLMs' knowledge of human motion to autonomously generate subtle yet powerful adversarial text descriptions. It comprises two key modules: an adaptive dispatching module that constructs an LLM-based agent to iteratively refine and search for adversarial prompts; and a multimodal information contrastive module that extracts semantically relevant motion information to guide the agent's search. Through this LLM-driven approach, ALERT-Motion crafts adversarial prompts querying victim models to produce outputs closely matching targeted motions, while avoiding obvious perturbations. Evaluations across popular T2M models demonstrate ALERT-Motion's superiority over previous methods, achieving higher attack success rates with stealthier adversarial prompts. This pioneering work on T2M adversarial attacks highlights the urgency of developing defensive measures as motion generation technology advances, urging further research into safe and responsible deployment.





§ INTRODUCTION




Human motion generation is a task aimed at producing natural and realistic human motions. It drives advancements in downstream applications such as animation and movie production, virtual human construction, robotics and human-robot interaction <cit.>. In recent years, with the development of deep learning, especially the growth of generative models such as Generative Adversarial Network (GAN) <cit.>, Variational Autoencoder (VAE) <cit.>, and diffusion model <cit.>, trained models have become capable of generating very natural motions <cit.>. Some models <cit.> even extend the generated motions for several minutes while satisfying given conditions. Among these motion generation models, text-to-motion (T2M) <cit.> gains particular attention from the community due to the user-friendly nature of text prompts that align with human expression.

Generating motions that exactly align with textual descriptions and are nearly the same as the real physical world is becoming increasingly feasible. However, allowing models to freely generate motions conditioned on arbitrary text prompts is even more dangerous than text-to-image (T2I). When they are applied to downstream tasks, such capabilities are maliciously exploited by attackers. For instance, in animation or movie production <cit.>, they are used to create more realistic harmful content involving pornography or violence. The risks are boosted when using the generated motions as humanoid controllers <cit.>, as they eventually are deployed on robots, posing potential threats to human safety.

Despite the growing focus on T2M tasks, there is currently a lack of research addressing the safety concerns specific to this domain. The most relevant line of work is on the safety of T2I <cit.>. These researches largely focus on how character-level or word-level modifications to benign text prompts could induce unintended outputs from the models. Early work <cit.> primarily explored the existence of this phenomenon, until the RIATIG <cit.> is inspired by them to propose targeted attacks against image generation models to raise awareness of potential security risks about T2I. However, these existing studies often search for adversarial attacks by modifying words to uncommon personal names, locations or other proper nouns, which is overlooked in image generation but would appear clearly out of place for motion tasks, making the attacks more easily detectable. Additionally, unlike the abundant image-text pairs available for image tasks, the limited data for motions makes it challenging to accurately measure the similarity between different motions, posing further difficulties for targeted attacks on T2M models. They make the findings from T2I safety difficult to directly apply to the motion domain.

To address the challenges of adversarial attacks on T2M models, we introduce ALERT-Motion, an autonomous large language model (LLM) enhanced adversarial attack against T2M models in a black-box setting. Unlike prior work, our ALERT-Motion leverages the knowledge about motions contained in LLMs to generate subtle yet powerful adversarial descriptions, whose outputs from the victim model closely match the desired motion. Crucially, the entire attack process is done automatically by LLM agent, using its own reasoning abilities to carry out the attack, without needing human-defined rules for operations like inserting, deleting or replacing characters or words.

As shown in Figure <ref>, previous state-of-the-art attack methods like RIATIG <cit.> for T2I models employ manually defined word or character-level modifications and prompt-level crossover, making it difficult to find natural and fluent adversarial text prompts. Such methods often result in obvious personal names or proper nouns like “Sebastian Hohenthal Fortec Motorsport” or “Boris Novachkov” appearing abruptly in descriptions of motions. In contrast, our proposed ALERT-Motion gives the modification of adversarial text prompts entirely to LLMs. It comprises two key modules: the adaptive dispatching (AD) module and the multimodal information contrastive (MMIC) module. In AD module, by simply designing instructions for different processes, LLM autonomously searches for adversarial text prompts that appear natural and fluent, avoiding the abrupt word insertions seen in RIATIG. However, as LLMs lack inherent capabilities for processing motion modality, we design MMIC module to obtain semantically similar information to the target motion, thereby assisting AD module in finding better adversarial text prompts. Through the coordinated operation of these two modules, ALERT-Motion generates adversarial text prompts that are not only natural and fluent but also query the victim T2M model to produce outputs closely resembling the target motions.

In summary, the key contributions are as follows:

1. To the best of our knowledge, we are the first to propose an adversarial targeted attack method, ALERT-Motion, for T2M models. We introduce an autonomous LLM-enhanced adversarial attacks on T2M models.

2. Our proposed ALERT-Motion consists of two key modules. A novel AD module constructs an LLM agent that incorporates the agent's inherent natural language and domain knowledge of motions into the automatic attack process. Additionally, MMIC module performs high-level semantic extraction of motion modalities and provides necessary semantical information to support AD's reasoning and decision-making.

3. We evaluate ALERT-Motion on two popular T2M models and compare it against two previous adversarial attack methods originally applied to T2I models. Experimental results demonstrate that our proposed ALERT-Motion achieves higher attack success rates while generating more natural and stealthy adversarial prompts that are difficult to detect.


§ RELATED WORK



  §.§.§ Text-to-Motion (T2M)
 T2M is a conditional motion generation task that aims to generate semantically matching and natural motion sequences from human-friendly natural language text descriptions. Its promising performance is driven by deep generative models such as GANs, VAEs, diffusion models, etc. One of the early works in this domain, Text2Action <cit.>, leverages GANs to create abundant realistic motions. Some research also explores the use of VAEs for generation, where Language2Pose <cit.> proposes an end-to-end text-to-pose generation framework that utilizes a VAE to model the latent space between text and motion. TEACH <cit.> further combines previous motions as extra inputs to the encoder module, enabling natural and coherent motion generation when handling multiple text inputs. With the rise of diffusion models in the generative domain, some studies have also employed diffusion models for motion generation. MDM <cit.> utilizes a diffusion model to predict the sample at each diffusion step rather than just the noise. MLD <cit.> adopts latent diffusion along with a VAE to generate motions, significantly boosting the generation speed without compromising quality. Additionally, there are studies that combine VQ-VAE with GPT-like transformers. TM2T <cit.> and T2M-GPT <cit.> utilize VQ-VAE to concatenate training T2M and motion-to-text modules. These works continuously improving the quality, coherence, and efficiency of motion generation from text descriptions. However, there has been no research focusing on attacks and defenses of the T2M model.


  §.§.§ Adversarial Attacks on Text-driven Generative Models
 Due to the convenience of text input for users, it serves as the most common driving condition for many multimodal generation models. However, the inherent complexity of text input inevitably introduces vulnerabilities to the generative models driven by it. Existing research on adversarial attacks in T2I models, such as <cit.>, attack T2I models by modifying the input text, causing abnormal outputs. The types of abnormal outputs may include degraded synthesis quality <cit.>, disappearance or alteration of objects in the image <cit.>. Among them, <cit.> manipulates words and characters, thereby causing the targeted objects specified by the attacker to be generated in the image by the victim T2I model. These studies indicate the lack of robustness of existing T2I models to input text. With the occurrence of LLMs, many studies also focus on the vulnerabilities of LLMs. A large portion of them focus on jailbreaking, making LLMs answer queries that violate safety policies. Jailbreaking strategies have evolved from manual prompt engineering <cit.> to LLM-based automated red-teaming <cit.>. Beyond these template-based jailbreaks aimed at identifying effective jailbreak prompt templates, a more general jailbreaking method called Greedy Coordinate Gradient <cit.> is recently proposed. It uses a white-box model to train adversarial suffixes that maximize the probability of an LLM producing affirmative responses. They <cit.> find that the identified suffixes transfer to closed-source off-the-shelf LLMs. The vulnerabilities of T2M models share similarities with the aforementioned security research on text-driven generative models. However, since the correspondence between motion and text involves the time dimension, the adversarial attack methods from the above studies cannot be directly applied to T2M.



  §.§.§ LLM Agents
 Research on using LLMs to enhance autonomous agents has seen a growing trend in recent years <cit.>. These LLM-powered agents, exemplified by HuggingGPT <cit.>, WebGPT <cit.>, and MM-REACT <cit.>, have been employed to tackle intricate tasks that demand effective understanding and planning from the agents. A considerable proportion of these studies leverage the rich commonsense knowledge inherently embedded within LLMs to execute downstream tasks with minimal or no additional training data. This approach helps to maintain the robust foundational world knowledge in LLMs. The demonstrated capabilities of LLMs encompass features such as zero-shot planning in real-world scenarios <cit.>. Inspired by these explorations, we introduce LLMs into the realm of adversarial attacks on T2M models, achieving an autonomous attack agent adept at crafting effective adversarial prompts.


§ METHODLOGY



We leverage an LLM to iteratively refine and enhance adversarial prompts towards a target motion. Initially, LLM generates alternative prompts semantically similar to the initial prompt to expand the search space. It then queries the victim T2M model with these prompts, recording the generated motions. The textual prompts and corresponding motions are unified into a suitable input format for LLM using MMIC. Exploiting its commonsense reasoning capabilities, LLM autonomously contemplates and updates the prompts based on the query results, iteratively steering them closer to the target motion. This process continues until the adversarial prompts evade detection while generating motions closely matching the target. Fig. <ref> overviews our ALERT-Motion attack framework.


 §.§ Problem Formulation

A T2M generative model G is essentially a function that maps the text prompt space P to the motion space M. Ideally, through training on semantically aligned text-motion pairs, a proficient model generates target motion m_t that is semantically consistent with a given target prompt p_t. The objective of an adversarial attack is to find an adversarial prompt p^⋆ such that G(p^⋆) closely approximates the target motion m_t. Simultaneously, the p^⋆ is semantically dissimilar from the target prompt p_t to avoid detection. The optimization steps outlined above are formulated as follows

    p^⋆=_p∈ P   s^ m(G(p), m_t),   s.t.   s^ p(p, p_t)<η

where s^ m represents the semantic similarity of motion, s^ p represents the semantic similarity between text prompts, and P is a promote set. η is the similarity threshold. As long as the similarity between the adversarial prompt and the target prompt is below η, we consider that our attack evades existing detection.


  §.§.§ Challenges

Implementing adversarial prompt generation from T2M models faces some challenges. First, T2M models need to go between natural language and physical motion, crossing the gap between language and motion domains. Different data types have different representation spaces, so integrating multi-modal information is needed. Second, the adversarial language prompts need to have high fluency in natural language and relevance to the target motion in their query results. But they also need to effectively fool the model. The space to search for good prompts is extremely large though. Autonomously generating optimal adversarial samples that meet these combined quality requirements is a big challenge.



 §.§ Multimodal Information Contrastive

Unlike most LLM agent-related researches, our task involves motion, which LLM cannot directly handle. Therefore, we design MMIC specifically to process information from different modalities in the task and organize it into textual information, making it convenient for LLM to understand and reasoning. As shown in Fig. <ref>, MMIC allows the adversarial prompts, refined through LLM, to query the victim T2M model, obtain corresponding motion and calculate the similarity with the target.

Nevertheless, measuring the similarity directly between two motion poses challenges. We consider semantically measuring the similarity of motion. RIATIG <cit.> employs the pretrained CLIP <cit.>, a model trained on a large-scale dataset of image-text pairs, to obtain semantic features aligned with textual descriptions. Similarly, we use the T2M alignment model proposed in <cit.> to extract semantic motion features and calculate the cosine similarity between the semantic features of motion as

    s^ m(G(p), m_t)=E_m(G(p)) · E_m(m_t)/E_m(G(p))E_m(m_t),

where E_m is a motion encoder <cit.>.

Subsequently, we organize this information into text and incorporate it into instructions, enabling LLM to contemplate and reason for better adversarial prompts.


 §.§ Adaptive Dispatching








In our proposed ALERT-Motion framework, the most critical module is the AD module. This module constructs an attack agent and plays a pivotal role in determining the effectiveness of adversarial prompts. In contrast to previous related researches, which predefine various operations to perturb semantics and then use a search algorithm to find prompts with higher scores, we directly convey complex task requirements using instructions, allowing LLM to automatically learn and execute all operations, with each step conducted in textual form. According to the purpose of instructions, we divide AD into three progresses: expansion, refinement, and update. The workflow of these three progresses and MMIC is outlined in Algorithm <ref>.

Due to the fact that AD responds to the current input in each round, similar to an agent in reinforcement learning, we borrow related concepts here to facilitate the definition of the processes within AD. We start by defining the state as the set of adversarial prompts and their corresponding information for each round, while the action is represented by various instructions sent to LLM. LLM is viewed as a function involving the next state S_k+1, current state S_k, and current action a_k, expressed as

    S_k+1← T(S_k, a_k) =  LLM(I,S_k),

where T is the state transition function and I represents the instruction text corresponding to a_k. It is important to note that the representation of state S differs between odd and even time steps, it is defined as

    S_k = {p_0}    if   k = 0   
    {p_1,…,p_n}    if   k   (mod  2)= 0 
    {p'_1,…,p'_n}    if   k  (mod  2)=1

where p_0 is initial adversarial prompt, {p_1, …, p_n} is the set of refined prompts, and p' are the expanded or updated prompts of p.


  §.§.§ Expansion

Initially, we begin with a single available adversarial prompt p_0. The current state is defined as S_0 = {p_0}. Without expansion, proceeding directly to the subsequent steps may lead the search into a local optimum. So we employ the expansion instruction text E to obtain expanded results through LLM. The next state is represented as

    S_1
    		
    		← LLM(I_E, S_0)


where I_E represents the expansion instruction and S_1={p_1, …, p_n} is the set of expanded prompts.


  §.§.§ Refinement

We find that when the instructions given to LLM are too long, its responses may sometimes fail to meet the attack requirements. New instructions are needed to emphasize the attack requirements in our task. Therefore, in this progress, we refine the adversarial prompts to ensure that they consistently meet the attack requirements, including being naturally fluent and relevant to motion. After refinement, the state of the agent is defined as

    S_2k LLM(I_R, S_2k-1)


where k∈{1,…,K} and I_R represents refinement instruction.


  §.§.§ Update

Unlike existing methods that relied on numerical scalar guidance to generate adversarial prompts, such as RIATIG, our AD utilizes the text information organized by MMIC to guide LLM in autonomously contemplating and generating adversarial prompts. This allows us to finely control the generated adversarial prompts more effectively in line with the attack requirements using richer information. Moreover, this control is automated, eliminating the need for continuously defining new operations, such as word or character insertion, deletion, replacement, and so forth, as in previous methods. In the update progress, as LLM contemplates and generates  adversarial prompts, the information organized by MMIC is also fed into LLM to assist in its decision-making process. After update, the state of the agent is defined as

    S_2k+1← LLM(I_U,  cat(S_2k,S'_2k))

where I_U is the update instruction and the function cat signifies string concatenation, and S'_2k={s_i^ m(G(p_i),m_ t),∀ i∈{1,…,n}} is obtained by Eq. (<ref>).

After K rounds of iteration, we choose the highest-scoring prompt among all candidates as the optimal adversarial prompt p^⋆ for the attack. The definition of p^⋆ is obtained by Eq. (<ref>).

	
		
		
		
	
Here P=∪ S_2k and S_2k={p_1,…,p_n}. In order to ensure that the adversarial prompt meets the constraints, we calculate the similarity between the adversarial prompts and target prompt as

    s^ p(p_t, p)=E_t(p_t) · E_t(p)/E_t(p_t)E_t(p),

where E_t is a text encoder <cit.> to extract features.




§ EXPERIMENT



 §.§ Experimental Settings



  §.§.§ Datasets.

We select target prompt texts and target motion from the
HumanML3D (H3D) <cit.>.  It includes 14,616 motion sequences from AMASS <cit.>, each with a textual description (totaling 44,970 descriptions). It also re-annotates AMASS and HumanAct12 <cit.> motion capture sequences. The dataset provides a redundant data representation involving root velocity, joint positions, joint velocities, joint rotations, and foot contact labels. It is used for both AMASS and HumanAct12 motion.


  §.§.§ Victim Models.

To assess the effectiveness of ALERT-Motion, we select two prominent publicly available T2M models: MLD and MDM. We employe their respective pretrained models from the official GitHub repositories, which were trained on H3D.







  §.§.§ Evaluation Setup.

In our experiments, all attacks are conducted in a black-box setting, meaning that we generate motion only by querying the model with prompts and obtaining the generated results. We utilize the “gpt-3.5-turbo-instruct” API with ChatGPT to implement our approach. The initial adversarial prompt text is a motion description randomly generated by ChatGPT. We set the number of iterations as 50, the size of the prompt set as 20. We set the similarity threshold η as 0.4. Examples for the attack were taken from the top 20 of the Dissimilar subset in the evaluation setup of <cit.>, where the model achieves the highest accuracy. During the attack process, we use the model from <cit.> to extract the motion features to compute cosine similarity and adopt the text feature extraction from <cit.>. The effectiveness is evaluated using T2M <cit.>.





  §.§.§ Baselines.

To the best of our knowledge, there is currently no targeted adversarial attack specifically designed for T2M generation. For comparison, we select two state-of-the-art targeted adversarial attack methods for text-to-image generation, MacPromp <cit.> and RIATIG <cit.>, as baseline methods. Since their tasks do not involve motion, we modify their task settings to match our task.




 §.§ Evaluation Metrics



  §.§.§ Motion Performance.

We utilize the R Precision, a widely-used metric in T2M <cit.> to evaluate generated motion. This assessment involves comparing each motion not only with its ground-truth text but also with a misaligned description. R Precision is determined through the Euclidean distance between motion and text features. Our evaluation centers on measuring the average accuracy among the top-k ranked descriptions. A ground-truth within the top-k candidates is considered a “True Positive” retrieval. Our approach involves a batch size of 20, encompassing 19 negative examples, and we explore the effectiveness of R at various values: 1 (R-1), 2 (R-2), 3 (R-3), 5 (R-5), and 10 (R-10). Furthermore, our study incorporates Frechet Inception Distance (FID) as a metric to assess the quality of generated motion. FID, a widely accepted standard for evaluating content quality <cit.>, involves comparing features extracted from generated motion and real motion. In our motion domain adaptation, we adopt an evaluator network to represent deep features, deviating from the original image-based Inception neural network. Smaller FID values are indicative of superior results. Additionally, we compute the Multimodal Distance (d_ MM), which is the mean Euclidean distance between the motions features and their corresponding textual descriptions features in the test examples <cit.>. A lower value indicates better alignment between prompts and their generated motions.


  §.§.§ Adversarial Similarity.



In adversarial attacks, it is essential for adversarial prompt  text to have low similarity with the target prompt text to evade detection. In line with previous studies, our initial step involves utilizing the Universal Sentence Encoder <cit.> for encoding both the adversarial sentence and the target sentence, resulting in high-dimensional vectors. Subsequently, we determine their adversarial similarity by computing the cosine score.


  §.§.§ Naturality.

To ensure the naturalness of adversarial examples, we measure perplexity (PPL) using GPT-2 <cit.>, trained on real-world sentences. PPL assesses the likelihood of the model in generating the input text, thereby indicating natural fluency of the adversarial prompts. Lower PPL values typically signify higher naturalness.


 §.§ Evaluation Results

The attack results on MLD and MDM are shown in Table <ref>. Compared to the baselines, ALERT-Motion achieves a higher R-precision. Although MacPromp achieves higher R-precision and lower FID and d_ MM in some cases, its direct translation of target prompts in various languages results in unnatural adversarial prompts. The perplexity is much higher than other methods, and, on the other hand, it closely resembles the target sentences, resulting in high adversarial similarity, making it less practical. RIATIG, compared to MacPromp, achieves similar or even higher R-precision, with a slight decrease in perplexity and adversarial similarity. However, as seen in Fig. <ref>, there are still incorrect words and some extra spaces.

From Table <ref>, it can be observed that our proposed ALERT-Motion performs better on most metrics across these models. Additionally, examining Figs. <ref> and <ref>, the adversarial prompts generated by ALERT-Motion are not only more natural but also relevant to the motion. In contrast, prompts obtained by other methods are mostly irrelevant to motion.


 §.§ Ablation Study



  §.§.§ Influence of Evaluation Models.

The current research on the evaluation of motion generation is still limited, with T2M <cit.> being widely recognized. Studies on motion generation, such as <cit.>, and <cit.>, adopt T2M to assess the quality of generated models. The latest research on the evaluation of motion generation is presented in TMR  <cit.>. Therefore, we also use it to evaluate our experiments. As shown in Table <ref>, under TMR model, ALERT-Motion exhibits significant superiority compared to other baseline methods, indicating that the excellent performance of our proposed ALERT-Motion is not influenced by the choice of evaluation models.


  §.§.§ Influence of Target Motion. 

To further analyze the selected target motion on the attack performance, we chose all 100 motion from the Dissimilar subset evaluation setting in TMR <cit.> as target motion. We conduct five experiments, each using a different set of 20 target motion, following the order specified in their setting. Table <ref> demonstrate that the superiority of ALERT-Motion performance over baseline methods remains consistent across different target motion. The overall performance of the attack method on these 100 target motion is presented in Table <ref>, demonstrating the consistent performance of our attack method.


§ DISCUSSION



  §.§.§ Safety Concerns.
 To prevent potential malicious uses, it is crucial to consider defense mechanisms for T2M models. There are valid concerns around malicious actors exploiting such models to produce explicit violent content <cit.>. Moreover, considering that these models serve as humanoid controllers <cit.> and may potentially be utilized for humanoid robots in the future, there are risks of the robots behaving in ways that endanger humans if not properly constrained. Although there is currently no research specifically addressing defense mechanisms for T2M generation models, we demonstrates that existing content moderation filters, if directly deployed on T2M models, are bypassed by our adversarial attack method. Moreover, the fact that our method creates adversarial prompts related to T2M task makes the attacks more challenging to defend against. Therefore, the safety risks of using motion generation models must be taken into consideration.


  §.§.§ Potential Defense Strategies.
 Several defense methods may be considered. Rule-based text filters might find our approach challenging to counter since the adversarial prompts we create seamlessly blend into normal text, remaining semantically related to motion. One potential defense mechanism involves leveraging larger datasets for training. The most extensive dataset in current motion generation research <cit.> comprises just over 10,000 text-motion pairs. We hypothesize that increasing the volume of training data boosts the alignment between the generated model and T2M tasks. Furthermore, established defense methods from the realms of adversarial attacks and NLP <cit.> is applied to strengthen T2M models, specifically through adversarial training.


§ CONCLUSION

In this paper, a novel method that involves conducting targeted adversarial attack against T2M models is proposed. Additionally, we introduce an autonomous LLM-enhanced adversarial attack method called ALERT-Motion, which comprises two modules: the multimodal information integration (MMIC) module and the adaptive dispatching (AD) module. Assisted by MMIC, AD, with the incorporation of LLM during progresses of expansion, refinement, and updating, autonomously learns and executes the search for optimal adversarial prompts. Our extensive experiments validate the ability to discover adversarial prompts that exhibit both fluency and related to motion. Moreover, these prompts trigger the victim T2M model to generate motion closely resembling the target, thus achieving successful attacks. The susceptibility of T2M models to our attacks suggest an urgent need to develop defensive methods and improve the robustness against adversarial exploitation.







    ieeenat_fullname
    







 Paper Title: Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition.















Corresponding author.
0000-0002-0217-9791

  Northwestern Polytechnical University
  Xi'an
  Shaanxi
  China

congqi.cao@nwpu.edu.cn


0009-0005-9387-8764

  Northwestern Polytechnical University
  Xi'an
  Shaanxi
  China

zhangyueran3@mail.nwpu.edu.cn


0009-0000-5504-9889

  Northwestern Polytechnical University
  Xi'an
  Shaanxi
  China

yatingyu@mail.nwpu.edu.cn


0000-0002-5359-5701

  Northwestern Polytechnical University
  Xi'an
  Shaanxi
  China

lvqinyi@nwpu.edu.cn


0000-0003-3970-7823

  Northwestern Polytechnical University
  Xi'an
  Shaanxi
  China

minlingtong@nwpu.edu.cn


0000-0002-2977-8057

  Northwestern Polytechnical University
  Xi'an
  Shaanxi
  China

ynzhang@nwpu.edu.cn













§ ABSTRACT 
  Existing works in few-shot action recognition mostly fine-tune a pre-trained image model and design sophisticated temporal alignment modules at feature level. However, simply fully fine-tuning the pre-trained model could cause overfitting due to the scarcity of video samples. Additionally, we argue that the exploration of task-specific information is insufficient when relying solely on well extracted abstract features. In this work, we propose a simple but effective task-specific adaptation method (Task-Adapter) for few-shot action recognition. By introducing the proposed Task-Adapter into the last several layers of the backbone and keeping the parameters of the original pre-trained model frozen, we mitigate the overfitting problem caused by full fine-tuning and advance the task-specific mechanism into the process of feature extraction. In each Task-Adapter, we reuse the frozen self-attention layer to perform task-specific self-attention across different videos within the given task to capture both distinctive information among classes and shared information within classes, which facilitates task-specific adaptation and enhances subsequent metric measurement between the query feature and support prototypes. 
  Experimental results consistently demonstrate the effectiveness of our proposed Task-Adapter on four standard few-shot action recognition datasets. Especially on temporal challenging SSv2 dataset, our method outperforms the state-of-the-art methods by a large margin.







  <ccs2012>
    <concept>
        <concept_id>10010147</concept_id>
        <concept_desc>Computing methodologies</concept_desc>
        <concept_significance>500</concept_significance>
        </concept>
    <concept>
        <concept_id>10010147.10010178.10010224.10010225.10010228</concept_id>
        <concept_desc>Computing methodologies Activity recognition and understanding</concept_desc>
        <concept_significance>500</concept_significance>
        </concept>
  </ccs2012>

[500]Computing methodologies
[500]Computing methodologies Activity recognition and understanding








    
  
    < g r a p h i c s >

  Paradigm comparison between classic few-shot action recognition methods and ours. (a) Existing methods fully fine-tune the feature extractor and combine it with temporal alignment module at feature level. (b) We advance the alignment module into the process of feature extraction particularly considering the internal information of the entire task by introducing tunable task-specific adapters into the frozen pre-trained model.
  figure description
  









Task-Adapter: Task-specific Adaptation of Image Models for
Few-shot Action Recognition
    Yanning Zhang
    
======================================================================================





§ INTRODUCTION

In recent years, large-scale pre-trained models <cit.> have demonstrated remarkable performance across various downstream tasks. However, manually collecting hundreds of millions of labeled data and training such large models from scratch for each specific task is impractical. Hence, researchers are increasingly focusing on fine-tuning large pre-trained models on specific tasks <cit.> to harness their generalization capability. In video understanding, the utilization of large-scale pre-trained models has significantly enhanced the performance of action recognition <cit.>. Yet, these works typically fine-tune pre-trained models on well-labeled video datasets <cit.>, where the testing and training categories are the same. In realistic scenarios, collecting sufficient and well-labeled video samples is inherently hard. Most problems manifest in a few-shot manner: there are very few annotated samples for new action categories. Few-shot learning <cit.> aims to achieve satisfactory classification performance on any few-shot manner unseen class data by training only on seen class data. Therefore, how to leverage the prior knowledge of pre-trained models to improve few-shot action recognition is an imperative challenge.

In few-shot action recognition, a common practice is following the metric based meta-learning strategy <cit.>, which computes classification probability according to the measured distance between query video feature and support prototypes. Consequently, previous works <cit.> design various temporal alignment modules at the feature level to better match the features of different videos. These works commonly use a pre-trained image model <cit.> and then fine-tune on the video dataset. An illustration of this traditional paradigm is depicted in Figure <ref> (a). However, this traditional paradigm has two drawbacks: First, the majority of works adopt full fine-tuning strategy, i.e., fine-tuning all parameters of the pre-trained model on seen classes and expecting to learn a robust feature extractor which maintains good generalization performance on unseen classes. However, this classic full fine-tuning strategy could overly destroy the prior knowledge of the pre-trained model, leading to catastrophic forgetting and overfitting to seen class data. The second drawback is that most existing works neglect the importance of extracting the task-specific information (i.e., inter-class distinctive information among classes and intra-class shared information within classes). Solely designing temporal alignment modules at highly abstracted feature level could not discover the most discriminative feature for a given few-shot learning task. 

To address the first drawback, we turn to the recently emerged Parameter-Efficient Fine-Tuning (PEFT) approach. Originating from the field of NLP <cit.>, PEFT aims to adapt large pre-trained models to downstream applications without fine-tuning all the parameters of the model, while attaining performance comparable to that of full fine-tuning. In the field of computer vision, a series of adapter-tuning works <cit.> have demonstrated the effectiveness of PEFT for video understanding. Inspired by these works, we argue such methods, which only fine-tune a small number of additional adapter parameters, are particularly well-suited for few-shot action recognition, since they enable better trade-offs between leveraging the generalizable prior knowledge of pre-trained models and incorporating domain-specific knowledge from downstream tasks. However, existing adapter methods mainly focus on temporal modeling and lack the consideration of enhancing task-specific features for the few-shot learning tasks, consequently not suitable to be directly introduced to the few-shot action recognition field. In this work, we propose a novel adapter-based method which further brings the capability of task-specific adaptation to the pre-trained model by reusing the frozen self-attention block to perform the task-specific self-attention across different videos within the given task to capture the most discriminative task-relevant information.

Task-specific adaptation of features has always been a key issue in metric based few-shot learning <cit.>. A common consensus is that the most discriminative features vary for different target tasks. For example, in distinguishing between “High Jump” and “Long Jump”,  the body movements performed by the actor is more crucial, whereas in distinguishing between “High Jump” and “Pole Vault”, it is more significant whether the actor is holding a long pole. In few-shot action recognition, most previous works <cit.> focus on the temporal alignment between different video features, but only a few <cit.> consider task-specific adaptation. Meanwhile, existing works solely apply task-specific methods at the level of well-extracted features. Different from previous works, we propose to perform task-specific adaptation during the process of feature extraction as shown in Figure <ref> (b). Specifically, for a Transformer backbone, e.g. ViT, in addition to the original spatial self-attention over the image tokens within the same frame, we further apply the frozen self-attention layer over the image tokens across different frames and different videos to perform both temporal attention and task attention across different videos within a given task. Experimental results reveal that performing task-specific adaptation during the feature extraction could better capture the discriminative task-specific feature compared to only adapting the final features. In summary, our contributions can be outlined as follows:

  
  * We introduce a novel parameter-efficient adaptation method for few-shot action recognition called Task-Adapter, effectively alleviating the issues of catastrophic forgetting and overfitting induced by full fine-tuning.
  
  * To the best of our knowledge, we are the first to propose performing task-specific adaptation during the process of feature extraction, which significantly enhances the discriminative features specific to the given few-shot learning task.
  
  
  
  * Extensive experiments demonstrate the superiority and good generalization ability of our Task-Adapter, which achieves new state-of-the-art results on four standard few-shot action recognition benchmarks.




§ RELATED WORK





 §.§ Few-shot Learning


Few-shot learning aims to learn a model that can quickly adapt to unseen classes using only a few labeled data. Existing works can be divided into three categories: augmentation-based <cit.>, metric-based <cit.> and optimization-based methods <cit.>. Augmentation-based methods use auxiliary data to enhance the feature representations or generate extra samples <cit.> to increase the diversity of the support set. Metric-based methods focus on learning a generalizable feature extractor and classify the query instance by measuring the feature distance. Optimization-based methods aim to learn a good network initialization which can fast adapt to unseen classes with minimal optimization steps.   Among these approaches, metric-based methods are commonly adopted thanks to its simplicity and superior performance. Additionally, task-specific learning has been explored in image-domain metric-based works <cit.> to achieve better few-shot learning performance. CTM <cit.> proposes category traversal module to obtain the most discriminative features within a given C-way K-shot few-shot learning task. FEAT <cit.> introduces set-to-set function to obtain task-specific visual features for the few-shot learning tasks. However, these methods all introduce task-specific modules at the level of well-extracted image features, which is insufficient for more complex video features due to the additional temporal dimension. In this work, we follow the metric-based paradigm and perform task-specific adaptation during the feature extraction process to achieve more comprehensive adaptation to video data.






 §.§ Few-shot action recognition


In few-shot action recognition, most of the previous works<cit.> fall into the metric-based methods. To solve the problem of temporal misalignment when comparing different videos, a wide series of temporal alignment modules are proposed. OTAM <cit.> improves the dynamic time warping algorithm to preserve the temporal ordering inside the video feature similarities. TRX <cit.> proposes to take the ordered tuples of frames as the least matching unit instead of single frames. CMOT <cit.> uses the optimal transport algorithm to find the best matching between two video features. HyRSM <cit.> treats the sequence matching problem as a set matching problem for better alignment. Except for the aforementioned works, there are also studies focusing on spatial relations <cit.> and multi-modal fusion <cit.>. However, there is still a lack of research on task-specific learning in few-shot action recognition. In this work, we emphasize the vital importance of task-specific adaptation in identifying the most discriminative spatial-temporal features within the provided video samples.




 §.§ Parameter-Efficient Fine-Tuning



Parameter-Efficient Fine-Tuning (PEFT) is derived from NLP <cit.>, aiming to fine-tune large pre-trained models on downstream tasks in an efficient way (i.e., only fine-tune a small number of parameters while freezing most parameters of the pretrained model), while achieving comparable performance to full fine-tuning. Recently, this method has been introduced to computer vision fields. CLIP-Adapter <cit.> adds a learnable bottleneck layer behind the frozen CLIP <cit.> backbone to learn new features for downstream tasks. ST-Adapter <cit.> proposes a 3D convolution based adapter to model the temporal information. AIM <cit.> reuses the frozen pre-trained attention layer and applies it to the temporal dimension of the input video. Vita-CLIP <cit.> introduces learnable prompt tokens into the frozen model to enrich the representation capability. However, the aforementioned PEFT methods do not consider the characteristic of the few-shot learning task, where task-specific adaptation plays a crucial role in distinguishing the query sample from irrelevant categories. In this work, we propose a novel PEFT method to maximize the generalization ability of the pre-trained model and equip the pre-trained image model with the capability of enhancing task-specific features for few-shot action recognition.












§ METHODOLOGY


In this section, we first describe how AIM <cit.> utilizes the pretrained image model ViT to perform action recognition in Section <ref>. Then, in Section <ref>, we extend this method to few-shot action recognition by introducing task-specific adaptation capability. The overview of our method is shown in Figure <ref>.



 §.§ Adapting ViT to Action Recognition


Given an RGB video sample v ∈ R^T× H × W × 3, a vanilla ViT takes the temporal dimension T as the batch dimension B, i.e., there is no exchange of information between frames. Specifically, ViT divides each frame into N = HW/P^2 patches, where P is the  patch size, and transforms the patches into D dimensional patch embeddings v_p ∈ R^T× N × D. A learnable [class] token is prepended to the patch embeddings v_p as v_0 ∈ R^T× (N+1) × D. To retain position information, a learnable positional embedding E_pos∈ R^(N+1) × D is added to v_0 as z_0 = v_0 + E_pos, where the E_pos is broadcast to T dimension to be shared among different frames and z_0 is the final input of the stacked ViT blocks. Each ViT block consists of LayerNorm (LN), multi-head self-attention (MSA) and MLP layers, with residual connections after each block, as shown in Figure <ref> (b). The computation process during each ViT block can be given by:

    z_l^'   =  MSA( LN(z_l-1))+z_l-1
    
        z_l    =  MLP( LN(z_l^')) + z_l^'

where z_l denotes the output of the l-th ViT block and all the MSAs are computed on patch dimension to learn the relationship of different spatial parts of the original frame. The [class] tokens of each frame from the last ViT block are concatenated as the final video feature F_v∈ R^T× D which is fed into the classifier and used to calculate the final classification score.

Inspired by PEFT techniques, AIM <cit.> tries to adapt pre-trained image model to video action recognition with minimal fine-tuning by freezing the original model and only fine-tuning additionally introduced Adapters shown in Figure <ref> (a). Moreover, AIM proposes to reuse the pre-trained MSA layer to do temporal modeling which is deficient in the vanilla ViT. As shown in Figure <ref> (c), a shared frozen temporal MSA (T-MSA) layer is prepended to the original spatial MSA (S-MSA) sequentially to exchange the temporal information between frames for temporal modeling, which can be achieved easily by permuting the patch dimension and the temporal dimension. The tunable Adapter is added after each MSA layer and parallel to the MLP layer to achieve temporal, spatial and joint adaptation as follows:

    z_l^t    =  Adapter( T  MSA( LN(z_l-1)))+z_l-1
    
        z_l^s    =  Adapter( SMSA( LN(z_l^t)))+z_l^t 
    
        z_l    =  Adapter(( LN(z_l^s)) +  MLP( LN(z_l^s)) + z_l^s

where z_l^t, z_l^s and z_l individually stands for temporal adapted, spatial adapted and jointly adapted output from the l-th ViT block. However, directly transferring this method to few-shot action recognition is inadequate due to the lack of considering the relationship between different videos within the few-shot learning task and the information gap between seen classes and unseen classes.





 §.§ Adapting ViT to Task-specific FSAR


Few-shot action recognition (FSAR) studies how to classify unseen class query videos with only a limited number of labeled samples available. Following the episodic training <cit.>, we randomly sample a C-way K-shot task, which has C action classes and K labeled samples in each class, from the training set 𝒟_tr for each episode during training. All the C × K video samples form the support set 𝒮 = {x_i}_i=1^C × K. Additionally, there is a query set 𝒬 = {x_i}_i=1^Q from the same C action classes. The learning objective of each task is to classify each query video q ∈𝒬 into one of the C classes. Although the temporal adaptation proposed by AIM gives vanilla ViT the ability of temporal modeling, it remains unaddressed that the ViT model could not attend to different videos and enhance the task-specific discriminative features. 

Next, we describe how we empower the frozen ViT with the ability to adapt embeddings with task-specific information for few-shot action recognition. As described in Section <ref>, we first map all the videos in the whole sampled task (both support set and query set) into their patch embeddings by Eq. <ref> and Eq. <ref>, then we get the input features of the first ViT block 𝒮_0 = [x_0^1, ..., x_0^CK] and 𝒬_0 = [x_0^1, ..., x_0^Q], where x_0^i ∈ R^T×(N+1)× D denotes the patch embeddings of the i-th video both for the support set and query set. Then, these patch embeddings are fed into several frozen ViT blocks to exchange spatial information between different tokens. It should be noted that we only introduce tunable adapters in the last L ViT blocks, which is a choice based on empirical study. We will further analyze the impact of this decision in the ablation experiment. Starting from the (N-L+1)-th layer, we introduce tunable adapter layers into the original model. Specifically, besides T-MSA and S-MSA, we further reuse the frozen MSA layer as Task-MSA after the temporal and spatial adaptation to perform task-specific self-attention across the tokens at the same spatial-temporal location in different videos (see Figure 3, upper part). Take the support features as an example:

  
       𝒮_l^t       = Adapter(T  MSA(LN(𝒮_l-1)))+𝒮_l-1

       𝒮_l^s       = Adapter(SMSA(LN(𝒮_l^t)))+𝒮_l^t 

       𝒮_l^task       = Adapter(TaskMSA(LN(𝒮_l^s)))+𝒮_l^s 

       𝒮_l       = Adapter(LN(𝒮_l^task)) + MLP(LN(𝒮_l^task)) + 𝒮_l^task
  
  

where S_l-1,S_l ∈ R^CK× T ×(N+1)× D individually denotes the input and output support features of the l-th layer (l ∈{N-L+1, N-L+2, ..., N}). 

The query features can be computed in the same way. To avoid exposing the target support class features to the specific query video feature, we isolate the information interaction between the support set and query set. The bottom right corner of Figure <ref> shows a conceptual view illustrating the changes that task-specific self-attention brings to the video features. As can be seen, the support video features of different classes are pushed away, and the query video features are pulled closer to the video features of the target support class based on the unique information specific to each task, resulting in varying effects across different tasks. As a result, we explicitly enhance the the most discriminative features in the given few-shot action recognition task with the introduced task-specific adaptation.


Finally, we concatenate the [class] tokens of each frame from the last ViT block to get the support set video representations ℱ_𝒮∈ R^C × K × T × D and the query set video representations ℱ_𝒬∈ R^Q × T × D. The final question is how to compute the distance between the query features ℱ_𝒬 and each class using the support features ℱ_𝒮. The existing works further design elaborate temporal alignment modules to construct class prototypes for each class, aiming to align the action semantics at the feature level. Thanks to our proposed task-specific adaptation, we have completed the semantic alignment during the process of feature extraction. Consequently, we directly average all the support video features as the class prototype for each class:

    ℱ̂_𝒮^c = 1/K∑^K_iℱ_𝒮[c][i]

where ℱ̂_𝒮^c ∈ R^C× T× D denotes the class prototype of the c-th support class. 

To calculate the classification probability, we measure the distance between the query features and the class prototypes by an arbitrary metric module, e.g., TRX <cit.>, OTAM <cit.> and Bi-MHM <cit.>, which can formulated by:


    𝒫^c_i =  Metric(ℱ_𝒬[i], ℱ̂_𝒮)


where 𝒫^c_i is the probability of the c-th class for the i-th query video in query set and ℱ̂_𝒮 = [ℱ̂_𝒮^1, ℱ̂_𝒮^2, ..., ℱ̂_𝒮^c] ∈ R^C× T× D stands for all the class prototypes of the whole support set. 

For the metric modules, we will demonstrate in our experiments that our proposed Task-Adapter consistently improves the performance of any metric methods. Besides that, owning to the superiority of our task-specific adaptation during feature extraction, excellent performance can be achieved simply by averaging the frame-to-frame cosine similarity without complex metric measurements. During the training stage, classification probabilities for each task serve as the logits for computing cross-entropy loss and backpropagating gradients. Importantly, we solely fine-tune the parameters of the additional adapter layers while keeping the original pre-trained model frozen. In testing, we freeze the entire backbone, including the adapters, and directly extract features for the unseen test classes.





§ EXPERIMENTS



 §.§ Experimental Setup


Datasets. We evaluate our method on four datasets that widely used in few-shot action recognition, namely HMDB51 <cit.>, UCF101 <cit.>, Kinetics <cit.> and SSv2 <cit.>. The first three datasets focus on scene understanding, while the last dataset, SSv2, has been shown to require more challenging temporal modeling <cit.>. Particularly, We use the few-shot split proposed by <cit.> for HMDB51 and UCF101, with 31/10/10 classes and 70/10/21 classes for train/val/test, respectively. Following <cit.>, Kinetics is used by selecting a subset which consists of 64, 12 and 24 training, validation and testing classes. For SSv2, we provide the results of two publicly available few-shot split, i.e., SSv2-Small <cit.> and SSv2-Full<cit.>. Both SSv2-Small and SSv2-Full contain 100 classes selected from the original dataset, with 64/12/24 classes for train/val/test, while SSv2-Full contains 10x more videos per class in the training set. 

Implementation details. As previously illustrated, we aim to adapt the image pre-trained ViT model to achieve task-specific few-shot action recognition. To show the applicability of our method for different pre-trained models, we respectively choose the ImageNet pre-trained <cit.> and CLIP pre-trained <cit.> ViT as our backbone. Specifically, for CLIP pre-trained backbone, we combine the zero-shot results and few-shot results in the same way as CLIP-FSAR <cit.> which uses a simple element-wise multiplication. For a fair comparison with the existing works <cit.>, we uniformly sample 8 frames for each video, and crop the frames to 224×224 as the input resolution. During the training stage, random crop is used for data augmentation. We freeze the original pre-trained model and only fine-tune the introduced adapters (as shown in Figure <ref> (c)) using the SGD optimizer with a learning rate of 0.001. For the testing stage, we freeze all the parameters of our backbone to extract task-specific discriminative video features for measuring the distances. All reported results in our paper are the average accuracy over 10,000 few-shot action recognition tasks.



 §.§ Comparison with state-of-the-art methods

Table <ref> presents a comprehensive comparison of our method with recent few-shot action recognition methods. As our Task-Adapter is designed for ViT-based architectures, we use CLIP <cit.> pre-trained ViT to make a fair comparison with recent few-shot action recognition works that use large-scale pretrained models. For the scene-related datasets, the background understanding is more crucial. Of particular note, by comparing the results of works that use CLIP pre-trained models and those that use ImageNet (IN-21K) pre-trained models, we can conclude that the performance of few-shot action recognition indeed benefits from large-scale pretraining, especially for scene-related datasets. As shown in Table <ref>, our Task-Adapter can achieve promising results on HMDB51, UCF101 and Kinetics. Especially for 5-shot tasks, our method outperforms the exsisting SOTA methods by 0.9% on HMDB51, 0.3% on UCF101 and 0.8% on Kinetics, respectively. Compared with full fine-tuning method CLIP-FSAR <cit.>, we can see our method has an overall improvement (from 0.2% to 6.5%) on all datasets. Compared with PEFT method MA-CLIP <cit.>, our method has an improvement of 0.9%, 0.8% and 0.8% respectively on HMDB51, UCF101 and Kinetics in 5-shot setting.
Note that while our method exhibits slightly lower 1-shot performance on Kinetics compared to MA-CLIP, it achieves a 1.5% improvement on UCF101 in the 1-shot setting. We attribute this to our underutilization of the semantic information in Kinetics, while MA-CLIP design an elaborate multimodal adaptation method with text-guided prototype construction module. We would like to consider more comprehensive multi-modal fusion as the future research.
As mentioned in Section <ref>, SSv2 requires more temporal modeling than other three datasets, making it much more challenging. In our experiments, we observe that our proposed task-specific adaptation significantly enhances the temporal alignment during the feature extraction process. From Table <ref>, we can see that our method outperforms all existing approaches on both SSv2-small and SSv2-Full, establishing new state-of-the-art results. Except for MA-CLIP <cit.> and ours, most of the previous works fully fine-tune the pre-trained models, resulting in inferior results due to the overfitting problem. Most relvant to our method, MA-CLIP <cit.> is also based on AIM <cit.>, but it focuses on the multimodal adaptation mechanism at the feature level, while our method emphasizes the importance of task-specific adaptation during the feature extraction. The results on the SSv2 datasets demonstrate the effectiveness of our proposed method, showing a significant improvement of 5.7% for the 5-shot task on SSv2-Small and 8.0% for the 1-shot task on SSv2-Full, respectively. Note that the improvement of the 1-shot task on SSv2-Small is smaller than that on SSv2-Full. We attribute this to the 10x increase in the number of videos for each class in SSv2-Full, which enables our Task-Adapter to learn low-shot task-specific adaptation through a variety of sampled tasks.









 §.§ Applicability of proposed Task-Adapter

In Table <ref>, we demonstrate the applicability of our method on different pre-trained weights. From the upper part of the table, we show the performance of different fine-tuning methods with IN-21K pre-trained weights. Note that the supervised pretraining on ImageNet does not follow the contrastive text-image pretraining paradigm used by CLIP. Consequently, we can only perform unimodal inference with pure visual information. As can be seen, just substituting the full fine-tuning with the PEFT method of AIM brings an improvement of 9.7%, which is in line with our insights that full fine-tuning limits the generalization capability of the pre-trained model. Furthermore, an another 3.0% improvement (totally up to 10.7%) is observed when using our Task-Adapter due to the fact that it enhances the capability of extracting task-specific information from the few-shot learning task. Also, we can see a 1.5% improvement of our Task Adapter over AIM and a 4.5% improvement over full fine-tuning under CLIP pre-trained weights. The consistent improvement over both full fine-tuning and AIM under different pre-trained weights show the strong generalization capability of our method. 

We also compare the performance of different fine-tuning strategies using different metric measurement methods to demonstrate the applicability of our Task-Adapter. Four commonly used methods are taken in this paper, namely TRX <cit.>, Bi-MHM <cit.>, OTAM <cit.> and ProtoNet (e.g., directly averaging the frame-to-frame cosine similarity over temporal dimension without complex metric measurements). Taking full fine-tuning strategy as the baseline, we observe a consistent improvement across all metric measurement methods, as shown in Figure <ref>. It should be noted that ProtoNet outperforms all the other methods when uniformly using Task-Adapter. We attribute this to the fact that with our Task-Adapter, the model is able to dynamically align the task-specific feature during the process of feature extraction corresponding to the given task. Therefore it is unnecessary to add extra complex alignment method at feature level. 














 §.§ Ablation Studies

Effectiveness of each component. To demonstrate the effectiveness of each component, we perform detailed ablation experiments in Table <ref>. The first row shows the results of a frozen pre-trained model without any learnable adapters, while the second row refers to the performance of full fine-tuning. Comparing the results of the third and fourth rows, we can see the effectiveness of PEFT over traditional full fine-tuning in few-shot action recognition. Note that the “Partial Adapting” in the fourth column of Table <ref> refers to a common strategy which only adds adapters in the last several ViT layers. We can observe from the table that the improvement of introducing task-specific adaptation into AIM is distinct (1.9%) when inserting adapters into all the layers of the pre-trained model. Additionally, using partial adapting strategy for AIM also brings an improvement of 1%. Even so, the improvement of introducing task-specific adaptation into AIM is notably enhanced under the partial adapting strategy, increasing from 1.9% to 5.9%. As mentioned in Section <ref>, AIM brings temporal modeling ability to the pre-trained image model, while our Task-Adapter further equip the model with the capability of performing task-specific adaptation. The result of the second-to-last row shows that we can still achieve comparable results only using task-specific adaptation without any temporal modeling, which demonstrates the importance of finding task-specific information for the few-shot learning task. Finally, the complete version of our method which only adds Adapters into the last several layers to the frozen model achieves the best performance.




Partial Adapting. 
In this section, we conduct ablation studies on the partial adapting strategy. Firstly, we study the effect of different inserting positions of our Task-Adapter in Table <ref>. The default setting is inserting adapters into all of the 12 ViT layers. Alternatively, we also evaluate the settings which respectively insert adapters to the bottom 6 layers and top 6 layers of the model (assuming forward propagation is from bottom to top). Inserting adapters only into the top 6 layers has been shown to achieve comparable performance with inserting to all layers in AIM <cit.>. However, inserting our Task-Adapter only into the top 6 layers outperforms inserting it into all layers by a large margin (5%) for few-shot action recognition. We attribute this to the hypothesize that bottom layers focus on generic features that do not require extensive adaptation, while the top layers focus on task-specific and discriminative features that benefit a lot from our task-specific adaptation module. Note that even with our Task-adapter inserted into all the layers, the increase in tunable parameters is small (only 14 M compared to 400 M frozen parameters of the pretrained model), which demonstrates the training efficiency of our method.
For the scene-related dataset, we choose the best partial adapting strategy by ablation experiments depicted in Figure <ref>. We can see from the figure that inserting our Task-Adapter into the top 2 layers is the best setting for all these three datasets. Since these scene-related datasets focus on background understanding and do not need much temporal modeling, a more lightweight setting with inserting adapters into only 2 top layers is more appropriate.

Position of Task-MSA relative to AIM. As illustrated in Section <ref>, the original AIM comprises two multi-head self-attentions (MSAs) individually for spatial modeling and temporal modeling. In this section, we study the optimal placement relative to the existing two MSAs of our Task-MSA for task-specific modeling. As can be seen in Table <ref>, inserting the proposed Task-MSA after the T-MSA and S-MSA achieve the best performance. It should be noted that regardless of where the Task-MSA is inserted, it consistently outperforms the baseline AIM <cit.>. The results indicate that compared with AIM, the introduction of Task-MSA indeed improves the performance for few-short action recognition. Furthermore, the Task-MSA after T-MSA and S-MSA can maximize the ability to extract task-specific discriminative features within the task.


 §.§ Visualizations

Figure <ref> presents the attention map visualizations for a given few-shot learning task obtained by baseline AIM and our Task-Adapter. We visualize the action “High Jump" and “Pole Vault” from UCF101 in Figure <ref> (a) and (b) respectively. As can be seen in Figure <ref> (a), the attention maps of AIM attend to several background areas that are unassociated with the action being performed. After introducing the task-specific adaptation proposed by us, the model begins to focus on the movements of the actor which are the most discriminative features for the category of “High Jump". In Figure <ref> (b), compared with the attention maps obtained by AIM, we can see that the areas of distinctive venue facilities gain a higher attention score (brighter color in the figure) with our Task-Adapter. For similar actions “High Jump” and “Pole Vault”, as can be seen, the model begins to attend to the specific information of the actions, e.g, the movements for “High Jump” and the long pole and large foam mats for “Pole Vault” when task-specific information is considered. The visualization results demonstrate that our method can enhance the most discriminative features for the few-short action recognition. 



§ CONCLUSION

In this paper, we propose a novel adaptation method named Task-Adapter to better adapt the pre-trained image models (including unimodal and multimodal pre-trained models) for few-shot action recognition. Instead of using the traditional full fine-tuning strategy, we propose to only fine-tune the newly inserted adapters which alleviates the issues of catastrophic forgetting and overfitting. To enhance the most discriminative features within the given few-shot learning tasks, we further propose to reuse the frozen self-attention layer for task-specific adaptation during the process of feature extraction, which remarkably improves the performance on four few-shot action recognition benchmarks and achieves the new state-of-the-art results, particularly on the challenging SSv2 dataset.








This work is supported by National Natural Science Foundation of China (No. 62376217, 62301434), Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), Key R&D Project of Shaanxi Province (No. 2023-YBGY-240), and Young Talent Fund of Association for Science and Technology in Shaanxi, China (No. 20220117).


ACM-Reference-Format




 Paper Title: On-the-fly Point Feature Representation for Point Clouds Analysis.

















  Singapore University of Technology and Design (SUTD)
  Singapore
  Singapore

jiangyi_wang@sutd.edu.sg



  Institute for Infocomm Research (I^2R), A*STAR
  Singapore
  Singapore
cheng_zhongyao@i2r.a-star.edu.sg


Corresponding authors.

  Singapore University of Technology and Design (SUTD)
  Singapore
  Singapore
na_zhao@sutd.edu.sg




  Institute for Infocomm Research (I^2R), A*STAR
  Singapore
  Singapore
cheng_jun@i2r.a-star.edu.sg


[1]

  Institute for Infocomm Research (I^2R), A*STAR
  Singapore
  Singapore
yang_xulei@i2r.a-star.edu.sg















§ ABSTRACT 
Point cloud analysis is challenging due to its unique characteristics of unorderness, sparsity and irregularity. Prior works attempt to capture local relationships by convolution operations or attention mechanisms, exploiting geometric information from coordinates implicitly. These methods, however, are insufficient to describe the explicit local geometry, e.g., curvature and orientation. 
In this paper, we propose On-the-fly Point Feature Representation (OPFR), which captures abundant geometric information explicitly through Curve Feature Generator module. This is inspired by Point Feature Histogram (PFH) from computer vision community. However, the utilization of vanilla PFH encounters great difficulties when applied to large datasets and dense point clouds, as it demands considerable time for feature generation. 
In contrast, we introduce the Local Reference Constructor module, which approximates the local coordinate systems based on triangle sets. Owing to this, our OPFR only requires extra 1.56ms for inference (65× faster than vanilla PFH) and 0.012M more parameters, and it can serve as a versatile plug-and-play module for various backbones, particularly MLP-based and Transformer-based backbones examined in this study.
Additionally, we introduce the novel Hierarchical Sampling module aimed at enhancing the quality of triangle sets, thereby ensuring robustness of the obtained geometric features.
Our proposed method improves overall accuracy (OA) on ModelNet40 from 90.7% to 94.5% (+3.8%) for classification, and OA on S3DIS Area-5 from 86.4% to 90.0% (+3.6%) for semantic segmentation, respectively, building upon PointNet++ backbone.
When integrated with Point Transformer backbone, we achieve state-of-the-art results on both tasks: 94.8% OA on ModelNet40 and 91.7% OA on S3DIS Area-5. 











<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224.10010225.10010227</concept_id>
       <concept_desc>Computing methodologies Scene understanding</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010224.10010240</concept_id>
       <concept_desc>Computing methodologies Computer vision representations</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Scene understanding
[500]Computing methodologies Computer vision representations

























On-the-fly Point Feature Representation for Point Clouds Analysis
    Yang Xulei
    August 5, 2024
=================================================================










§ INTRODUCTION

Point cloud analysis on robotics and automation application <cit.> has garnered substantial attention in recent years, driven by advancements in sensor technologies like LiDAR and photogrammetry. This growing interest attributes to two key advantages: 1) It can accurately represent complex objects with numbers of points. 2) It can be quickly created by using 3D scanning devices. Compared to 2D image data, point clouds provide a more powerful 3D sparse representation containing abundant geometry and layout information of the environment. 


Deep learning technology <cit.> has achieved significant improvements in various image processing tasks. However, the typical deep learning technology requires highly regular input data formats. The unordered and irregular point clouds bring great challenges to apply the image processing techniques directly. 
PointNet <cit.>, the pioneering work of network architecture that directly works with point clouds, overcomes the challenges of the unordered and irregular inputs. It uses point-wise shared-MLP followed by a pooling operation to extract global features from point clouds, but global pooling operation leads to the loss of valuable local information. 
PointNet++ <cit.> further proposes set abstraction (SA) to process local regions hierarchically. This step aggregates features from neighboring points, thereby capturing local information. However, it still learns from individual points without incorporating local relationships <cit.>. This could hinder the model from leveraging inherent point clouds geometric structures.


Local geometric structures are vital for understanding point clouds. In an effort to capture this information, some prior works attempt to learn local relationships from convolutions <cit.>, attentions <cit.>, or graphs <cit.>.
However, these methods require huge amount of labelled data to learn local geometry implicitly <cit.>, while getting large amount of labelled 3D annotations is difficult.
Recently, RepSurf <cit.> has emerged as a novel approach that explicitly learns geometric information based on umbrella surface <cit.>, which is a triangle set[In this paper, we refer triangle set to a collection of connected or disconnected triangles. For those connected ones, they form a surface.] with connected triangles formed by k nearest neighbors (k-NNs).
While triangle sets are effective in capturing location and orientation information, they often fall short in incorporating curvature knowledge, which is essential for accurate point cloud recognition <cit.>. 
Moreover, as depicted in the supplementary material, in certain k-NNs, the k points may come from different surfaces of the object. These “noisy points” can lead to the distortion of k-NN triangle sets, significantly impacting the quality of the obtained geometric features <cit.>.


To integrate curvature information explicitly, we draw inspiration from Point Feature Histogram (PFH) <cit.>, a notable hand-crafted feature descriptor for capturing regional curvature knowledge.
PFH exploits the histogram of curvature angles within local neighborhoods to characterize individual points. As shown in Fig. <ref>, these angles are calculated between the normal vector and the local coordinate system, which demand substantial computing resources. 
Nonetheless, many point cloud datasets <cit.> lack normal vectors and necessitate additional normal estimation <cit.>. Normal estimation poses significant computational challenges, particularly for dense point clouds, while its accuracy degenerates considerably for sparse point clouds.
These limitations can potentially lead to the breakdown of vanilla PFH approach, further underscoring the challenges of its direct integration with deep learning models. 


In view of PFH's potentials and limitations, we explore curvature information and propose On-the-fly Point Feature Representation (OPFR), which includes Local Reference Constructor module and Curve Feature Generator module. This provides an efficient way to leverage explicit curvature knowledge without the prerequisite of normal estimation, which inherently relies on the quality of triangle sets.
Additionally, we propose the novel Hierarchical Sampling module to mitigate the distortion of triangle sets that occurs in the naive k-NN approach. Our sampling method demonstrates the robustness against noisy points by employing a hierarchical sampling strategy and a farthest point sampling strategy. 
As a result, it can significantly improve the obtained geometric features.
These innovations confer the following properties:



    
  * Curvature Awareness. The usage of curvature information remains underexplored by prior works. Our proposed OPFR obtains the capability to explicitly capture not only location and orientation knowledge, but also curvature geometry via Curve Feature Generator module. 
    
  * Computational Efficiency. Vanilla PFH is computationally expensive due to normal estimation. Our proposed OPFR introduces Local Reference Constructor module, which approximates the local coordinate systems based on triangle sets to overcome the computational bottlenecks. 
    
  * Robustness. Naive k-NN sampling causes distortion of triangle sets, which compromises the obtained geometric features. In contrast, our proposed OPFR presents Hierarchical Sampling module to enhance the quality of triangle sets, ensuring robust geometric features for noisy points.



Moreover, our OPFR is backbone-agnostic, making it compatible with different 3D point clouds analysis architectures. We demonstrate its model-agnostic nature by adapting two representative backbones: PointNet++ <cit.> and Point Transformer <cit.>.
It serves as an efficient plug-and-play module, and achieves substantial performance improvements. 
Empirical results prove its compatibility with different backbones. When incorporating with Point Transformer backbone, our OPFR achieves state-of-the-art performance for both point cloud classification and semantic segmentation tasks. 












§ RELATED WORK





 §.§ Deep Learning on Point Clouds

Many prior works <cit.> learn from raw point clouds via careful network designs.
PointNet <cit.> pioneers this trail by handling coordinates of each point with shared-MLP and consolidating the final representation with a global pooling operation. However, it is susceptible to a deficiency in preserving local structures due to the use of global pooling operation.
PointNet++ <cit.> is an extension of the original PointNet architecture, which applies PointNet to multiple subsets of point clouds. It further leverages a hierarchical feature learning paradigm to capture the local structures. However, PointNet++ still processes points individually in each local region, neglecting explicit consideration of relationships between centroids and their neighbors. 


As PointNet++ establishes the hierarchical point clouds analysis framework, the focus of many works has shifted towards the development of local feature extractors, including convolution-based <cit.>, attention-based <cit.>, and graph-based <cit.> approaches. 
PointCNN <cit.> learns a χ-transformation from input point clouds, which attempts to re-organize inputs into canonical order. Subsequently, it utilizes vanilla convolution operations to extract local features.
Point Transformer <cit.> replaces conventional shared-MLP modules with Transformer <cit.> blocks, serving as feature extractors within localized patch processing.  
DGCNN <cit.> utilizes dynamic graph structures to enhance feature learning and capture relationships between points.
However, these works rely heavily on the learnability of feature extractors, potentially missing inherent local shape information.
More recently, RepSurf <cit.> leverages triangle sets with connected triangles formed by k nearest neighbors (k-NNs), to learn location and orientation-aware representations from geometric features explicitly.
Although location and orientation features are explicitly injected into network architecture in RepSurf, the usage of curvature information still remains underexplored.
Moreover, RepSurf relies on naive k-NNs to produce triangle sets and obtain geometric features, which are vulnerable to noisy points <cit.>.




 §.§ Hand-crafted Designs on Point Clouds







Many works in 3D computer vision attempt to build sophisticated feature descriptors <cit.>, which help to understand point clouds through hand-crafted features.
Point Feature Histogram (PFH) <cit.>, one of the feature descriptors, is commonly used in computer vision tasks like object recognition, registration and model retrieval <cit.>. 
It develops point cloud representations by summarizing the distribution of certain geometric attributes within a local neighborhood around each point.

We depict the workflow of PFH derivation for one of our interested point in Fig. <ref>. The whole process can be decomposed into two steps. Firstly, for each point pair within k nearest neighbors (k-NNs) of interested point, curvature features are characterized using angles calculated from normal vectors and relative positions. Secondly, for each angle, we achieve its histogram within k-NNs. The histograms of different angles are concatenated together, yielding the final PFH representation.
Unfortunately, many point cloud datasets <cit.> collected in real-world scenarios lack normal vectors, and estimating normal vectors <cit.> for sparse point clouds often leads to significant deviations from ground truth.
Nonetheless, PFH calculation involves establishing local coordinate systems and constructing curvature features, which is computationally expensive. As a result, the practical application of vanilla PFH is limited.







§ METHODOLOGY



The pipeline of On-the-fly Point Feature Representation (OPFR) is depicted in Fig. <ref>, and we illustrate the OPFR generating process for the right corner of table (highlighted in pink), which is one of our interested points.

Firstly, we propose Hierarchical Sampling module, which takes each point in the point cloud as input and outputs several clusters (highlighted in blue) and corresponding centroids (highlighted in orange).
This hierarchical sampling strategy improves the quality of triangle set for each point, thereby facilitating the development of subsequent geometric features.

Then, for each point pair within the clusters, we design Curve Feature Generator module to generate geometric features, including location, orientation, and curvature. 
The inclusion of explicit curvature information allows us to more effectively capture the local geometry surrounding these point pairs.

To enhance the efficiency and enable on-the-fly processing, we present Local Reference Constructor module. It approximates a local coordinate system (highlighted in red) for each point pair using adjacent points from triangle sets.

Lastly, these obtained geometric features are further fed into shared-MLP followed by a pooling operation, constituting our final OPFR representation 𝐫_i. The resultant OPFR representation 𝐫_i along with coordinate 𝐱_i can be directed into various point clouds analysis backbones, e.g., PointNet++ <cit.> and Point Transformer <cit.>, for end-to-end training.





 §.§ Hierarchical Sampling










As mentioned earlier, the connected triangle sets produced by naive k nearest neighbors (k-NNs) are susceptible to noisy points <cit.>, leading to significant distortion. 
Given that our Local Reference Constructor module inherently relies on triangle sets to approximate local reference frames, we propose the novel Hierarchical Sampling module to alleviate this distortion issue.
For each individual point, our Hierarchical Sampling module generates several clusters, forming a triangle set. Specifically, we firstly conduct k-NN algorithm to select the k_1 nearby points (highlighted in purple). Secondly, we utilize farthest point sampling <cit.> algorithm to identify k_2 surface centroids (highlighted in orange) from these k_1 nearby points. Lastly, for each centroid, we retrieve its k_3 nearest neighbors (highlighted in blue). The selected k_3 neighbors are used to further develop geometric features. The detailed implementation is presented in Algorithm <ref>.


As illustrated in Fig. <ref>, Hierarchical Sampling module is designed to decouple the right corner of the table into k_2 distinct clusters (e.g., table top and table leg). 
These k_2 clusters exhibit simpler geometric structures, allowing resultant triangle sets to better approximate the original local surface. Therefore, compared to the naive k-NN approach, our hierarchical sampling scheme effectively relieves the distortion issue of triangle sets, and ensures the robustness against original noisy points. 
As a result, it greatly enhances the development of subsequent geometric features.
We provide additional visualization examples comparing triangle sets generated by Hierarchical Sampling and those produced by k-NN sampling in the supplementary material.






 §.§ Local Reference Constructor


A local reference frame is a local system of Cartesian coordinates at each point <cit.>, which provides a reference for understanding local structures.
Denote a point set as 𝐗={𝐱_1,𝐱_2,⋯, 𝐱_N}⫅ℝ^N× 3, normal vector set as 𝐍 = {𝐧_1, 𝐧_2, ⋯, 𝐧_N}⫅ℝ^N× 3. Assume 𝐱_i as our interest point, and the objective is to extract geometric features for point 𝐱_i. Then, the local reference frame {𝐮,𝐯,𝐰} <cit.> for {(𝐱_i,𝐱_j), i≠ j} is defined as:



    𝐮 = 𝐧_i 
       𝐯 = (𝐱_j-𝐱_i)×𝐮||(𝐱_j-𝐱_i)×𝐮||
       𝐰 = 𝐮×𝐯||𝐮×𝐯||.


Although Equ. <ref> achieves the construction of local reference frames, it comes with two major problems. Firstly, it relies on normal vectors, which are often unavailable in many benchmarks <cit.> and real-life scenarios. 
Despite normal estimation <cit.> is feasible, its computational cost escalates significantly with dense point clouds, and its accuracy diminishes considerably with sparse point clouds.
Secondly, it involves multiple cross-product operations sequentially, which cannot be effectively parallelized in terms of tensor operations. This leads to the inevitable computational overheads.


To circumvent normal estimation and overcome the computational bottlenecks, we design approximated local reference frames through Local Reference Constructor (LRCon) module. Within each cluster generated by Hierarchical Sampling module, we establish point pairs between centroid and neighboring points. For each point pair, LRCon module leverages two adjacent neighbors along with their cross-product to serve as the approximate local reference frames. 
Denote number of neighbors as K, neighbors of centroid 𝐱_i as 𝐗_i={𝐱_i1,𝐱_i2,⋯,𝐱_iK}⊆ℝ^K× 3. Based on this setting, we can construct the approximated local reference frame {𝐮̂,𝐯̂,𝐰̂} for point pair {(𝐱_i,𝐱_ij),j=1,2,⋯,K}, which is defined as:



    𝐮̂ = 𝐱_ij^+-𝐱_i||𝐱_ij^+-𝐱_i||
       𝐯̂ = 𝐱_ij^--𝐱_i||𝐱_ij^--𝐱_i||
       𝐰̂ = 𝐮̂×𝐯̂||𝐮̂×𝐯̂||,


where 𝐱_ij^+, 𝐱_ij^- are the most adjacent points for 𝐱_ij in neighbor set 𝐗_i clockwise and counterclockwise. To maintain the consistency of local frame orientation, we apply clockwise cross-product <cit.> to compute 𝐰̂. 
When setting up the approximated local reference frames, the LRCon module basically finds the adjacent neighbors from the corresponding triangles in the triangle sets, i.e., Δ_1={𝐱_i, 𝐱_ij, 𝐱_ij^+} and Δ_2={𝐱_i, 𝐱_ij, 𝐱_ij^-}. 
As mentioned earlier, the Hierarchical Sampling module can boost the quality of triangle sets, ensuring the robustness against noisy points. This implicitly guarantees the reliability of the approximated local reference frames.


This approximation scheme allows us to establish local reference frames that are independent with normal vectors of point clouds. By re-ordering these neighbors in 𝐗_i based on their projected angles in xy-plane, we can efficiently derive approximated reference frames through tensor operations. Notably, with the integration of the LRCon module, our OPFR only requires an additional 1.56ms for inference, making it 65× faster than vanilla PFH.
Furthermore, since LRCon module eliminates the need of normal estimation, it is compatible with point clouds of varying densities.




 §.§ Curve Feature Generator













We propose to approximate the local curve at point (t,f(t)) by excluding high-order derivatives using Taylor Series <cit.>:



    f(x) ≈f(t)_location + f'(t)_orientation(x-t) + 1/2f”(t)_curvature (x-t)^2.


Intuitively, the derivatives f'(t) and f”(t) reflect how the local curve is oriented and skewed near point (t, f(t)) respectively. From Taylor approximation, it can be observed that, 1-order derivative information is inadequate for accurately characterizing local curves. 









To exploit 2-order curvature information, we propose the Curve Feature Generator (CFGen) module.
This module processes input point pairs along with their approximated local reference frames, generating geometric features that encompass location, orientation, and curvature. 
Denote the approximate local reference frames for (𝐱_i, 𝐱_ij) as {𝐮̂_ij,𝐯̂_ij,𝐰̂_ij}. The location and orientation can be naturally <cit.> characterized by relative position 𝐱'_ij=𝐱_ij-𝐱_i and frame cross-product 𝐧_ij=𝐮̂_ij×𝐯̂_ij, respectively.
Furthermore, we propose the curvature proxy 𝐩_ij for point clouds, which is an approximation of curvature definition <cit.> from differential geometry. We provide the theoretical analysis for this part in the supplementary material. The curvature proxy 𝐩_ij is defined as:



    𝐩_ij = 1/||𝐱'_ij||·arccos([𝐮̂_ij;𝐯̂_ij;𝐰̂_ij] ⊙𝐱'_ij/||𝐱'_ij||),


where ⊙ is the entry-wise dot product. Note that, curvature proxy 𝐩_ij approximates the limit definition of curvature from differential geometry, making it inherently curvature-aware. Intuitively, 𝐩_ij effectively captures how the surface is curved in three reference frames {𝐮̂_ij,𝐯̂_ij,𝐰̂_ij} in terms of normalized angles.



















 §.§ On-the-fly Point Feature Representation

Point Feature Histogram (PFH) <cit.> utilizes histogram operations to aggregate regional geometric features and generate final representation for each point. We argue that, these predefined transformation functions are task-agnostic, which making the final representations not fitting well for specific tasks. To this end, motivated by PointNet++ <cit.>, we employ shared-MLP to learn the final representations from point clouds. 
Therefore, the proposed OPFR representation 𝐫_i for point 𝐱_i is defined as:



    𝐫_i= 𝒜({ℱ([𝐱_ij';𝐧_ij;𝐩_ij]): j=1,2,⋯,K}),


where 𝒜 is a pooling operation (e.g., sum), ℱ is a shared-MLP, and [𝐱_ij';𝐧_ij;𝐩_ij] are explicit geometric features obtained from CFGen module for one point pair (𝐱_i, 𝐱_ij). By feeding OPFR representation 𝐫_i along with coordinate 𝐱_i into the backbone, the whole learning process can be achieved through end-to-end training.






In Fig. <ref>, the three-view drawing depicts the OPFR values of 1-st channel for an airplane. The blue hues represent areas with smaller OPFR values, typically the airplane body, while the red hues indicate larger OPFR values, primarily associated with the airplane wings. This color differentiation underlines our OPFR is sensitive to curvature variation across the airplane's structure, demonstrating the curvature-aware property of OPFR. 
We provide more visualization examples in the supplementary material.
Additionally, it is important to highlight that, as shown in Fig. <ref>, our OPFR can outperform vanilla PFH by a large margin with the help of learnable shared-MLP. Furthermore, the introducing of shared-MLP only increases 0.012M learnable parameters, which is approximately negligible for most popular backbones <cit.>.







§ EXPERIMENTS

We evaluate our OPFR on two primary tasks: point cloud classification and semantic segmentation. 
We choose two representative point cloud understanding models, PointNet++ <cit.> and Point Transformer <cit.>, as our backbones to evaluate the effectiveness and compatibility of OPFR representations across different backbone architectures.
Additionally, we carry out ablation studies to demonstrate the effectiveness of our OPFR network designs and quantitatively evaluate the efficiency and quality of OPFR feature representations.
Moreover, due to space constraints, we present qualitative results in the supplementary material.


Implementation details. For the Hierarchical Sampling module, we set k_1=20 and k_2=4 to control the number of candidate centroids and selected centroids respectively. 
The shared-MLP consists of three layers with 30 OPFR dimensions (𝐫_i ∈ℝ^30), followed by a sum pooling operation.
These are achieved via empirical studies, which will be further discussed in Sec. <ref>.
Following RepSurf <cit.>, we set k_3=8
, considering the trade-off between performance and efficiency. 
We use CrossEntropy loss and label smoothing <cit.> techniques with a ratio of 0.3 for both tasks. We provide more details about implementation in the supplementary material.





 §.§ Classification

We evaluate our OPFR on two commonly used benchmarks for point cloud classification: ModelNet40 <cit.> and ScanObjectNN <cit.>. 


Experimental setups. Following RepSurf <cit.>, we implement two versions to integrate OPFR with PointNet++ <cit.>, one standard version and one scaled-up version. The scaled-up version doubles the channels of standard version and exploits deeper networks. If not specified, we default to the standard version. We also apply the channel de-differentiation design <cit.> when integrated with PointNet++.
We opt Adam <cit.> optimizer with default parameters to train our models for 250 epochs with a batch size of 64 and initial learning rate of 0.002. We apply exponential learning rate decay scheme with decay rate of 0.7. The whole training and testing process are conducted through one NVIDIA Quadro P5000 16GB GPU. For evaluation metrics, we use overall accuracy (OA) and mean accuracy within each classes (mAcc). For efficiency metrics, we use number of learnable parameters (#Params) and floating point operations (FLOPs). For a fair comparison, we calculate FLOPs from 1024 input point clouds, and utilize single-scale grouping (SSG) set abstraction <cit.> for all PointNet++ based <cit.> methods.


Classification on ModelNet40. ModelNet40 <cit.> is one 
synthetic object classification benchmark, which contains 9843 training samples and 2468 testing samples. They contain 100 unique CAD models from 40 object categories. The experimental results are presented in Tab. <ref>. 
The results reveal that our OPFR significantly improves PointNet++ <cit.> backbone by 3.8% OA and 3.2% mAcc, with just an additional 0.012M more parameters and 0.15G more FLOPs. The scaled-up OPFR further attains a slight improvement of 0.1% OA and 0.2% mAcc.
Moreover, when integrated with transformer-based backbone, Point Transformer <cit.>, our OPFR achieves the state-of-the-art 94.8% OA and 92.0% mAcc (+1.1% OA and +1.4% mAcc). 


Classification on ScanObjectNN. ScanObjectNN <cit.> is a challenging, real-world object classification benchmark. It is composed of 2902 point cloud samples from 15 categories, including occlusion and background. Following the typical protocol <cit.>, we verify our OPFR on the hardest variant (PB_T50_RS_variant) of ScanObjectNN. In Tab. <ref>, the proposed OPFR achieves 85.7% OA and 83.8% mAcc (+7.8% OA and +8.4% mAcc) on PointNet++ backbone, which outperforms RepSurf <cit.> by a large margin of 1.4% OA and 2.5% mAcc with comparable model size. Our result surpasses PointMLP <cit.> by 0.3% OA as well, and utilizes 9× fewer parameters. Furthermore, we scale up our proposed OPFR and achieve 88.5% OA and 86.6% mAcc, which demonstrates a superiority of 0.8% OA and 0.8% mAcc compared with state-of-the-art MLP-based backbone, PointNeXt <cit.>. Our result is also comparable to prior state-of-the-art transformer-based backbone, SPoTr <cit.>, with around 2.1× fewer FLOPs. When integrated with Point Transformer, our OPFR attains a notable improvement of 5.8% OA and 5.6% mAcc, which only increases 0.003M more parameters and 0.04G more FLOPs.










 §.§ Semantic Segmentation

We evaluate our proposed OPFR representations on a challenging benchmark, S3DIS <cit.>, for semantic segmentation task. 

Experimental setups. When integrated with PointNet++ <cit.>, we apply the channel de-differentiation design <cit.>. We opt AdamW <cit.> with default parameters to train our models for 100 epochs with a batch size of 8 and initial learning rate of 0.006. Here, we employ multi-step learning rate decay scheme and decay at [60,80] epochs with a decay rate of 0.1. The whole training and testing process are conducted through two NVIDIA A40 48GB GPU. For evaluation metrics, we use mean of classwise intersection over union (mIoU), mean of classwise accuracy (mAcc), and overall accuracy (OA). For a fair comparison, we calculate FLOPs from 15000 input point clouds <cit.>, and leave test-time-augmentation <cit.> in absence.

Semantic Segmentation on S3DIS. S3DIS <cit.> encompasses 271 scenes which are distributed across 6 indoor areas, with each individual point being classified into one of 13 semantic labels. Following a common protocol <cit.>, we evaluate the presented approach in two modes: (a) Area-5 is withheld for training and is used for testing, and (b) 6-fold cross-validation. In Tab. <ref>, our proposed OPFR considerably enhances PointNet++ <cit.> by 14.7%/16.9%/3.0% (mIoU/mAcc/OA) on S3DIS 6-fold benchmark. Our result is comparable to PointNeXt-XL <cit.>, with around 40× fewer parameters and 11× fewer FLOPs. When integrated with Point Transformer <cit.>, the performance of OPFR exceeds previous state-of-the-art Superpoint Transformer <cit.> by 0.9%/0.1%/1.6% (mIoU/mAcc/OA) for S3DIS 6-fold. Meanwhile, on S3DIS Area-5, our OPFR attains mIoU/mAcc/OA of 72.6%/78.6%/91.7% (+2.2%/+2.1%/+0.9%), surpassing the prior state-of-the-art ConDaFormer <cit.>.


Furthermore, as shown in Tab. <ref>, we present quantitative segmentation results for each semantic class on S3DIS Area-5 in terms of mIoU.

In Tab. <ref>, the top performance gain comes from the most challenging columns semantic class for both PointNet++ and Point Transformer backbones. 
Within all classes columns exhibit a distinct columnar structure, which consists of two or three planes in S3DIS dataset. This multi-plane structure can be effectively captured by different clusters generated from the proposed Hierarchical Sampling module, which facilitates the recognition of column pattern with greater ease. We provide detailed qualitative results in the supplementary material.




 §.§ Ablation Study


We ablate some critical designs of our standard OPFR with PointNet++ <cit.> backbone on ModelNet40 <cit.> and ScanObjectNN <cit.> dataset for an insightful exploration.

Effectiveness of different OPFR modules. Shown in Tab. <ref>, as we remove Hierarchical Sampling module, Curve Feature Generator module, and 3-layer shared-MLP, the overall accuracy (OA) decreases by 1.17%, 2.02%, 1.53% and mean accuracy (mAcc) drops by 0.91%, 1.76%, 1.34% respectively. From this empirical study, we can confirm that, explicit geometric features are crucial for 3D object understanding, and shared-MLP is necessary as well to enhance the semantics of obtained geometric features. Furthermore, due to the use of Hierarchical Sampling module, we can effectively relieve the distortion of triangle sets, thereby improving the quality of geometric features. Additionally, we argue that, the Hierarchical Sampling module can be applied to RepSurf <cit.> to handle the distorted triangle sets from k nearest neighbors. Due to space limits, we provide the ablation study in the supplementary material.









Designs of OPFR network architecture. We ablate the designs of OPFR network architecture in terms of pooling operation 𝒜 and shared-MLP ℱ in Tab. <ref>. Empirical results demonstrate that, usage of summation pooling, batch normalization, and three-layer shared-MLP with 30 OPFR dimensions outperforms other options. From our experiments, we hypothesize that, the network tends to encounter overfitting issues as we increase the number of OPFR dimensions and shared-MLP layers.








Sensitivity of hyper-parameters. In Hierarchical Sampling module, we are required to determine number of surface centroid candidates k_1, number of selected surface centroids k_2 and number of neighbors k_3. Following RepSurf <cit.> design, we fix k_3 equal to 8 to construct OPFR and explore the relation between k_1 and k_2 in terms of overall accuracy (OA) in Tab. <ref>.
Generally speaking, our OPFR is relatively insensitive to the choices of hyper-parameters. As the value of k_1 increases, there is an initial rise in overall accuracy, which is subsequently followed by a slight decline. We hypothesize that, this phenomenon is attributed to the inherent trade-off between exploration and concentration. When k_1 is small, we are unable to capture the local region of point clouds effectively. Conversely, when k_1 is too large, we move far from the original point, leading to the deviation of obtained geometric features. Furthermore, our OPFR is insensitive to the change of k_2. We hypothesize that, this behavior primarily stems from these k_2 clusters may overlap with each other. To avoid computation overheads, we consider (k_1=20, k_2=4) as an ideal choice.









Efficiency of OPFR representations. Shown in Tab. <ref>, we evaluate the efficiency of our OPFR representations in terms of number of extra parameters and inference speed. Empirically, although vanilla PFH introduces no extra learnable parameters, it requires 102ms for each input sample to generate the final representation, rendering it impractical for online network training. 
The main computational bottlenecks lie in the estimation of point clouds normal vectors <cit.>. We propose novel Local Reference Constructor module to eliminate the needs of normal estimation and overcome the computational overheads. We achieve an impressive inference speed of 1.56ms (65× faster) with a marginal increase of 0.012M number of parameters. Therefore, OPFR can serve as a versatile plug-and-play module for various backbones.
Furthermore, the efficiency of our OPFR is close to the previous state-of-the-art plug-and-play feature representation RepSurf <cit.>, with only 0.004M more parameters and 0.44ms more inference time.









Quality of OPFR representations. Shown in Fig. <ref>, we compare the performance between PFH <cit.>, RepSurf <cit.>, and proposed OPFR using PointNet++ <cit.> backbone. All of them are injected to PointNet++ as extra features. By incorporating vanilla PFH, overall accuracy (OA) and mean accuracy (mAcc) are enhanced by 2.0% and 2.6% on ModelNet40, 4.7% and 5.3% on ScanObjectNN, emphasizing the effectiveness of regional curvature knowledge. This gain further escalates to 3.8% and 3.2% on ModelNet40, 7.8% and 8.4% on ScanObjectNN in OA and mAcc respectively, when equipped with the proposed OPFR. This demonstrates the significance of shared-MLP, which enriches the obtained geometric features. Furthermore, compared with the previous state-of-the-art feature representation RepSurf, our OPFR outperforms it dramatically on ScanObjectNN, with a considerable margin of 1.4% and 2.5% higher OA and mAcc. We hypothesize that, this phenomenon is attributed to the uses of explicit curvature knowledge and robust sampling strategy, which are underexplored in RepSurf.









§ CONCLUSION

We propose the novel plug-and-play module On-the-fly Point Feature Representation (OPFR) for various backbones. It explicitly captures local geometry including location, orientation and curvature through Curve Feature Generator module. We further develop the Local Reference Constructor module to improve efficiency and enable on-the-fly processing. 
Additionally, we introduce the Hierarchical Sampling module to mitigate the distortion of triangle sets that occurs in the naive k nearest neighbors sampling, thereby enhancing the robustness of obtained geometric features.
We evaluate the proposed OPFR on ModelNet40 <cit.> and ScanObjectNN <cit.> benchmarks for point cloud classification task, S3DIS <cit.> for semantic segmentation task. For both PointNet++ <cit.> and Point Transformer <cit.> backbones, our presented OPFR achieves the state-of-the-art results on different benchmarks.
The comprehensive empirical results demonstrate the backbone-agnostic nature of our proposed method.
We believe that our work can prompt consideration of how to better leverage geometric knowledge in network architecture designs for understanding point clouds.


This research work is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021).





ACM-Reference-Format









 Paper Title: A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gap.






CEFA











Both authors contributed equally to this research.

  School of Computer Science and Ningbo Institute, Northwestern Polytechnical University
  Xi’an, Shaanxi
  China

  National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean
  Xi’an, Shaanxi
  China
lijunzhang@mail.nwpu.edu.cn


[1]

  School of Computer Science and Ningbo Institute, Northwestern Polytechnical University
  Xi’an, Shaanxi
  China

  National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean
  Xi’an, Shaanxi
  China
suowei1994@mail.nwpu.edu.cn


Corresponding author

  School of Computer Science and Ningbo Institute, Northwestern Polytechnical University
  Xi’an, Shaanxi
  China

  National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean
  Xi’an, Shaanxi
  China
peng.wang@nwpu.edu.cn



  School of Computer Science and Ningbo Institute, Northwestern Polytechnical University
   Xi’an, Shaanxi
   China

  National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean
  Xi’an, Shaanxi
  China
ynzhang@nwpu.edu.cn











§ ABSTRACT 
  Human-object interactions (HOI) detection aims at capturing human-object pairs in images and corresponding actions. It is an important step toward high-level visual reasoning and scene understanding. However, due to the natural bias from the real world, existing methods mostly struggle with rare human-object pairs and lead to sub-optimal results. Recently, with the development of the generative model, a straightforward approach is to construct a more balanced dataset based on a group of supplementary samples. Unfortunately, there is a significant domain gap between the generated data and the original data, and simply merging the generated images into the original dataset cannot significantly boost the performance. 
  To alleviate the above problem, we present a novel model-agnostic framework called Context-Enhanced Feature Alignment  () module, which can effectively align the generated data with the original data at the feature level and bridge the domain gap.
  Specifically,  consists of a feature alignment module and a context enhancement module. On one hand, considering the crucial role of human-object pairs information in HOI tasks, the feature alignment module aligns the human-object pairs by aggregating instance information. On the other hand, to mitigate the issue of losing important context information caused by the traditional discriminator-style alignment method, we employ a context-enhanced image reconstruction module to improve the model's learning ability of contextual cues. Extensive experiments have shown that our method can serve as a plug-and-play module to improve the detection performance of HOI models on rare categories[https://github.com/LijunZhang01/CEFA].







<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224</concept_id>
       <concept_desc>Computing methodologies Computer vision</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Computer vision



































A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gap
    Yanning Zhang
    August 5, 2024
==========================================================================================





§ INTRODUCTION


The goal of Human-Object Interaction (HOI) detection is to localize humans and objects in an image and recognize their interactions. The interactions can be represented by a triplet Human, Verb, Object. HOI enables machines to understand human activities at a fine-grained level, which is an important step towards high-level advanced visual reasoning <cit.> and scene understanding <cit.>. However, real-world data have a long-tailed distribution such as human riding giraffe, which will lead to the presence of long-tail phenomena in HOI datasets. Long-tail datasets result in mediocre performance of HOI models on rare categories.

To address the long-tail problem, existing methods can be divided into two paradigms <cit.>. The first category is based on training strategies <cit.>, such as resampling and retraining techniques. Resampling and retraining are achieved by changing the sampling strategy for long-tail data or adjusting the loss function. The other research line focused on combinatorial learning <cit.>. They utilize the concept of combinatorial learning to recompose human-object pairs and interaction between different HOI instances as new training samples. In this way they promote knowledge generalization across HOI classes and form new triplets to mitigate the long-tail problem.

Although these methods have obtained great progress, they have not fundamentally addressed the long-tail issue caused by data imbalance. Recently, 

Artificial Intelligence Generated Content (AIGC) has made significant advancements in generating highly realistic images. This has opened up possibilities for using AI-generated data as a replacement for real data. Therefore, a straightforward idea is to utilize generative models to synthesize rare categories in the HOI dataset. 
Unfortunately, as shown in Fig. <ref> (a), we find that simply merging the generated images into the original dataset and training the model does not significantly improve the performance of the model. 
<cit.> have made some initial explorations about this, they alleviated the problem by designing different rules to filter out low-quality images that exhibit substantial deviation from the original dataset. However, these approaches lack scalability across different tasks and suffer from large human subjectivity.
In order to explore the critical reasons, as depicted in Fig. <ref> (b), we use t-SNE<cit.> to analyze the feature distribution between original data and generated data. From the visualization results, it can be found that there are significant distribution differences between the generated data and the original data. In other words, there exists a considerable domain gap between the original and generated data.

To address the aforementioned issues, we propose a novel yet simple feature alignment framework called Context-Enhanced Feature Alignment  () module, which aims to align the generated data with the original data at the feature level and bridge the domain gap.



On one hand, most existing feature alignment methods based on transformer architecture <cit.> mainly align encoder and decoder features through the discriminator, without considering the instance information.

While in the Human-Object Interaction tasks, the instance information of human-object pairs is critical for successful task execution. Since HOI triplets are predicted through the features of the decoder, we aim to pay more attention to instance alignment during the decoder feature alignment process. Considering that tokens in the decoder can often be repetitive or even erroneous, we propose a graph-based prototype instance alignment method. This approach views the high-scoring tokens as prototypes and constructs a graph network to align the human-object pairs by aggregating instance information.



On the other hand, some researches <cit.> have indicated that the above discriminator-style alignment method inevitably leads to losing important context clues. These context clues provide important contextual information to identify human-object interactions. 
For example, when the background is snow, the model should be more inclined to predict “people holding skis”.

Based on the above insights, we propose a context-enhanced image reconstruction module, aiming to improve the model's learning ability of contextual cues. 
 
 
Benefitting from the above two modules, as shown in Fig. <ref> (a) and (c),  our model effectively extracts more domain-independent features and bridges the domain gap.
 
 
In summary, the primary contributions of our paper are as follows:



  *  We propose a novel model-agnostic method  which can effectively bridge the domain gap between the generated and original domains.




  *  We build an instance feature alignment module to align the human-object pairs by aggregating instance information.  Meanwhile, we design a context-enhanced image reconstruction module to improve the model's learning ability of contextual cues.


  *  The proposed  is a plug-and-play module that can be conveniently applied to existing HOI models and boost the performance of rare categories.





§ RELATED WORK



  
Long tail solution in HOI detection.


To address long-tail phenomena in HOI, researchers have proposed various methods <cit.>, which can be summarized into two categories: training strategy-based methods and combinatorial learning-based methods.

Training strategy-based methods <cit.>  primarily improve the training process. They attempt to alleviate the long-tail problem by adjusting loss functions, sample sampling strategies, class balancing techniques, etc. 


Combinatorial learning-based methods <cit.> aim to enhance the performance of HOI models by recombining different representations of human-object pairs and interaction, forming new triplets to mitigate the long-tail problem. 


While existing methods have not fundamentally addressed the long-tail issue caused by data imbalance. AIGC offers a promising solution by using generative models to synthesize rare classes in the HOI dataset.
Recently <cit.> has made some initial explorations, while they suffer from the problem of large differences between the generated images and the original images. They addressed this by devising a set of rules to filter the generated images. These approaches lack scalability across different tasks and have relatively large human subjectivity. 
To this end, we propose  to align the generated data with the original data at the feature level.










  
Unsupervised domain adaptation.

To enhance the domain adaptation capabilities of the model, researchers have proposed various unsupervised domain adaptation techniques <cit.>. 

Next, we will introduce Unsupervised domain adaptation using object detection as an example.
For object detection, the goal of unsupervised domain adaptation is to narrow the domain gap between the source and target domains so that the source data can be utilized to train a better detector for the target data. 
Most existing domain adaptive detectors <cit.> employ a teacher-student model to generate pseudo-labels and utilize adversarial learning to encourage the model to extract domain-invariant features, thereby improving domain transferability. Among them, based on the transformer architecture <cit.> primarily align feature extracted by CNN and token sequences from the transformer encoder-decoder.

For instance, <cit.> aligns features at global and local levels to enhance performance in domain-adaptive object detection. 
While recent studies <cit.> have shown that the above discriminator-style alignment method will cause the model to focus on domain-invariant features while losing important context clues. We propose a context enhancement module to alleviate this problem.



§ PROPOSE METHOD


In this section, we introduce our Context-Enhanced Feature Align-ment  () framework, which aims to reduce the distribution difference between the generated domain and the original domain, thereby alleviating the HOI long-tail problem. As shown in Fig. <ref>, the  comprises an “Instance Feature Alignment Module” and a “Context Enhancement Module”.

Since our  is a plug-and-play paradigm that can be applied to the previous one-stage based HOI model, we begin with a detailed introduction to the foundational architecture of the HOI model in Section <ref>. Subsequently, in Section <ref>, we expound on the instance feature alignment module, which aligns the human-object pairs by aggregating instance information. In Section <ref>, we describe the context enhancement module, which employs a masked image reconstruction branch to strengthen the model's ability to capture contextual cues. Finally, in Section <ref>, we provide a comprehensive introduction to the loss function.






 §.§ HOI Model Overview


In this section, we first introduce the formal definition of the HOI detection problem. Given a human-object image I, the model is required to predict a set of HOI triplets S={(b_i^h,b_i^o,v_i),i ∈{1,2,...,N}}, where b_i^h, b_i^o and v_i denotes a human bounding-box, an object bounding-box and their corresponding action category, respectively. Since our  is a plug-and-play paradigm, we would first introduce the basic HOI model (e.g., <cit.>).

A basic HOI model can be divided into three components: visual feature extractor, human-object pair decoder, and relationship decoder. 
The visual feature extractor, consisting of a CNN and a Transformer encoder, is responsible for extracting image features and outputting encoder features X_e∈ R^(H^'× W^')× D_c. 

The encoder features X_e and the human-object pair query Q_ins∈ R^N_q× D_hq are fed into the human-object pair decoder to output X_ho∈ R^N_q× D_hd, which is used to detect the position of humans and objects. Then the encoder features X_e, the human-object pair decoder outputs X_ho are fed into the relationship decoder. The relationship decoder outputs X_ve∈ R^N_q× D_ve to detect relationship categories for the corresponding pairs of humans and objects.

Although the above basic pipeline has achieved promising results in HOI detection. Unfortunately, due to the natural bias from the
real world, existing methods mostly struggle with
rare human-object pairs and lead to sub-optimal results. Recently, with the development of the generative model, a straightforward approach is to build a more balanced dataset with a group of supplementary samples. However, as shown in Fig. <ref> (a), we find that simply merging the generated images into the original dataset cannot significantly improve the performance of the model. In order to reveal the potential reasons, as depicted in Fig. <ref> (b), we use t-SNE<cit.> to analyze the distribution between original data and generated data. From the visualization results, it can be found that there exists a considerable domain gap between the original and generated data. 

To bridge the domain gap and align features, we propose the instance feature alignment module and the context enhancement module. Next, we will provide a detailed introduction to these modules.



 §.§ Instance Feature Alignment Module


Many previous domain alignment techniques<cit.> based on the transformer architecture primarily align features extracted by CNN and token sequences from the transformer encoder-decoder. Most of them feed both CNN features and all tokens into discriminators for alignment. For CNN features and transformer encoder features, similar as in SFA<cit.>, we feed them into the discriminator for alignment, with the following loss:

    .89!L_enc=E_X_b∈ srclog D_b( X_b) + E_X_b∈ genlog( 1 - D_b( X_b))
       + E_X_e∈ srclog D_e( X_e) + E_X_e∈ genlog( 1 - D_e( X_e) ),

where src represents the images from the original domain and gen represents the images from the generated domain, D_b and D_e denotes the domain discriminator. X_b and X_e denotes the CNN features and transformer encoder features.

In the Human-Object Interaction tasks, the instance information of human-object pairs is critical for successful task execution. Since HOI triplets are predicted through the features of the relationship decoder, we aim to concentrate more on instance alignment during the relationship decoder feature alignment process. Considering that tokens in the relationship decoder can often be repetitive or even erroneous, feeding all relationship decoder tokens into the discriminator for alignment would lead to suboptimal results. Therefore, as shown in Fig. <ref>, we propose a prototype instance alignment module to aggregate instance information.

For the output of the relationship decoder, due to most of the predictions learned by the token are redundant or even incorrect, 

we select the top k sets of relationship tokens with the highest scores as human-object instances. 
Following <cit.>, each encoding can be viewed as a prototype, we consider the selected instances as prototypes. 



Considering the presence of redundant tokens, 
in order to better aggregate instance information, we aim to enable regular tokens to transfer instance information into the prototypes. 

Given the convenience of graph networks in specifying information propagation rules, we construct a graph network to aggregate instance information.
Specifically, we construct a graph 𝒢=(𝒱,A,X), where 𝒱={v_0, v_1, …, v_N_q}
represents the set of N nodes, A ∈ℛ^N× Nrepresents the adjacency matrix of the graph, and X ∈ℛ^N× D_ve represents the node feature matrix. 
Each graph node represents a token feature. 

We classify the nodes representing prototypes as prototype nodes, while the rest are referred to as regular nodes. 
To propagate information from regular nodes to prototype nodes for instance aggregation, there are edges connecting prototype nodes with regular nodes and other prototype nodes. There are no edges connecting regular nodes to regular nodes.

The Graph Convolutional Network (GCN) <cit.> can effectively capture the local and global context of nodes in the graph. By facilitating information transfer between nodes, it enables effective representation learning. So we utilize GCN for feature propagation among the nodes. Following <cit.>, the formula for the feature propagation is as follows:

    X = GCN( A,X),


    X_p = { X_i| i ∈ P},

where A represents the adjacency matrix, X represents the node feature matrix, P represents the set of prototype indices.

We extract the features of the prototype nodes as the instance features X_p and input them into the discriminator for feature alignment. 
The instance loss is defined as follows:

    L_inst = E_X_p∈ srclog D_dec( X_p)+  E_X_p∈ genlog( 1 - D_dec( X_p)),

where D_dec denotes the domain discriminator.

In summary, the adversarial optimization objective is formulated:

    L_adv = max_T min_D (L_inst+L_enc),

where T, D denotes the HOI detector and discriminators respectively. Gradient Reverse Layers(GRL)<cit.> is adopted for min-max optimization.



 §.§ Context Enhancement Module


The aforementioned feature alignment module aligns features from three different sources: CNN features, transformer encoder features and instance features. While studies<cit.> indicate that the above discriminator-style alignment method will cause the model to focus on domain-invariant features at the expense of losing important context clues. 
In the HOI task, context clues can provide rich scene information to help the model better identify human-object interactions. So we propose a context-enhanced image reconstruction module to improve the model's learning ability of contextual cues. 







As shown in Fig. <ref>, we randomly mask the generated image I_gen at a certain ratio σ to obtain I_mask. The original image I_src and the masked generated image I_mask are separately fed into the encoder to obtain the features X_src and X_mask. 
Considering that the generated images share the same semantic information as the original images with the same labels, we use the features of the original images as auxiliary information to aid the recovery process of the masked images. 
Firstly, we fed X_src and X_mask into the conditional controller. The role of the conditional controller is to learn the importance of different features from the original image for the image restoration process. It is composed of the decoder layer from the Transformer <cit.>. During this process, the X_src serves as the query as well as X_mask serves as the key and value. The output of conditional controller is denoted as D_sign. 
Secondly, Considering that not all features in the original image contribute to the recovery of the generated image, we select the D_sign that are greater than 0.5 using a gate mechanism. The final original image feature used for conditionally control X_src_ cond is obtained by element-wise multiplication between X_src and filtered D_sign.
In summary, the conditional controller and the gate mechanism work together to filter out the feature tokens irrelevant to image restoration, extracting only the useful features that facilitate the recovery of the image.  The above computations can be formulated as:














    D_sign = Condition( X_src,X_mask ),


    Sign = Gate( D_sign ) ,


    X_src_ cond = Sign⊙  X_src,

where Condition and Gate are conditional controller and gate, ⊙ indicates the element-wise multiplication.

We concatenate X_src_ cond with X_mask to obtain the fused features. These fused features are then fed into an auxiliary decoder similar to MAE <cit.> to reconstruct the original image. It can be formulated as:

    I_recon = Decoder(Concate( X_src_ cond,X_mask ) ).


Following <cit.>, our context-enhanced image reconstruction loss L_ctx is calculated by computing the Mean Squared Error(MSE) loss between the generated image and the reconstructed image. The calculation formula is as follows:

    L_ctx = 1/n∑_i=1^n ( I_recon^i-I_gen^i)^2,

where I_recon^i represents the reconstructed image patch and I_gen^i represents the generated image patch. 



 §.§ Loss


Our model has two types of inputs: generated images and original images. We denote L_sup as the supervised HOI loss of original images and L_unsup as the HOI loss of generated images. L_sup  is computed using labeled original images and L_unsup is calculated using generated images with pseudo-labels. 

The loss for the original images L_src includes the supervised HOI loss and the discriminator loss, defined as:

    L_src = L_sup + L_adv.

The loss for the generated images L_gen includes the HOI loss, the discriminator loss, and the context reconstruction loss, defined as:

    L_gen = L_unsup + L_ctx + L_adv.






Therefore, the total loss is given by:

    L_loss = L_src + L_gen.







§ EXPERIMENTS




 §.§ Experimental Setting



  
Datasets.
Following previous work, we evaluated our model on two common benchmarks, HICO-Det <cit.> and V-COCO <cit.>. HICO-Det consists of 47,776 images, with 38,118 images used for training and 9,658 images used for testing. It includes 80 object categories and 117 action categories, forming a total of 600 HOI triplets. Among these 600 HOI categories, any category with fewer than 10 training instances is defined as Rare, while the remaining categories are defined as Non-Rare.
V-COCO is a subset of the COCO dataset and comprises 10,396 images, with 5,400 images used for training and 4,964 images used for testing. It contains 29 action categories, including 4 body actions that do not involve interactions with any objects. The dataset shares the same 80 object categories as HICO-Det. The actions and objects together form 263 HOI triplets. Among the 29 action categories, we define action categories with less than 300 training instances as Rare, and the remaining action categories as Non-Rare.



  
Implementation details.
We use Blip-diffusion <cit.> to generate images with the template “a photo of human verbing object”, where verbing represents the present progressive form of the action in the HOI triplet. The threshold for pseudo-label with the model is set to 1.4 for GEN-VLKT and HOICLIP benchmark models, and 0.4 for the CDN benchmark model. 


During the training phase, we fine-tune the pre-trained weights of the existing HOI model and keep the human-object pair decoder fixed.


The learning rate for the backbone in the CDN benchmark is set to 1e-5, while the rest of the learning rates are set to 1e-4. For
the GEN-VLKT and HOICLIP benchmark, the backbone learning rate is set
to 1e-6, and the rest is set to 1e-4. The mask ratio σ is 0.8, and the number of prototype nodes k is set to 6.
We conduct all the experiments on 8 Tesla V100 GPUs, using a batch size of 12 per GPU.



  
Evaluation.
We evaluated our model using mean Average Precision (mAP). The HOI triplet predictions are considered true positives when they meet the following criteria: 1) the predicted human and object bounding boxes have an IoU greater than 0.5; and 2) both predicted HOI categories are accurate.
For HICO-Det, we evaluated three different category sets: the complete set of 600 HOI categories (Full), the 138 HOI categories with fewer than 10 training instances (Rare), and the remaining 462 HOI categories (Non-Rare).
For V-COCO, we also evaluated three different category sets: the complete set of 29 action categories (Full), the 4 action categories with fewer than 300 training instances (Rare), and the remaining 25 action categories (Non-Rare).





 §.§ Quantitative Evaluation

To evaluate the effectiveness of our method, we conducted combined experiments with three representative methods. Firstly, we performed experiments on the HICO-Det dataset, the results are shown in Table <ref>. Since the proposed method is a plug-and-play module, our method is combined with CDN, GEN-VLKT, and HOICLIP to achieve performance improvements. 
Specifically, when combined with CDN, our method achieved a relative improvement of 4.05% on rare categories. When combined with GEN-VLKT and HOICLIP, our method achieved a +1.49 mAP and +1.59 mAP improvement on rare categories. 
These results demonstrate the effectiveness of our method in handling rare class HOI tasks. 
We have also noted that merely merging the generated data with the original dataset for training does not lead to a direct and intuitive improvement in performance. This confirms that the domain gap between the original and generated data causes the model to deviate from the optimal solution during the fine-tuning process, thereby impeding the enhancement of performance. After applying our method, there is a significant increase in performance, which further attests to the effectiveness of our approach. Additionally, the compatibility of our method with various other methods also proves its excellent extensibility. We also compared our method to previous state-of-the-art (SOTA) approaches, it can be found that our method outperforms these works by a big margin.


Next, Table <ref> presents the performance of our method on the V-COCO dataset. The results showed significant improvements over the original models. 
Specifically, when CDN is combined with our method, we achieved a +1.02 mAP improvement for all categories and an impressive +2.23 mAP improvement on rare categories. When combined with GEN-VLKT and HOICLIP, we achieved a +1.64 mAP and +1.38 mAP improvement on rare categories.





 §.§ Ablation Studies







  
Model components ablation.
In this section, we conduct several ablation studies on the HICO-Det to systematically evaluate the contributions of different model components. 
As shown in Table <ref>, the “Instance” and “Context” represent instance feature alignment module and context enhancement module, respectively. Specifically, 

“Instance” indicates the process of aligning human-object pairs by aggregating instance information. “Context” represents using context-enhanced image reconstruction module to improve the model’s learning ability of contextual cues.

Due to HOICLIP <cit.> achieving state-of-the-art results in previous methods, we selected it as the baseline for comparison. As shown in rows 1-2 of Tabel <ref>, we follow the baseline and add an instance feature alignment module to align features. We can observe that the rare class mAP achieves an improvement of 1.06, which demonstrates that our instance feature alignment module is capable of effectively aligning human-object pairs, thereby promoting domain alignment.
In addition, in row 3 of Table <ref>, we follow the baseline and add the context enhancement module separately to extract important contextual information. The context enhancement module resulted in a relative improvement of 3.91% on rare categories, which shows that the proposed context enhancement module can effectively learn important contextual cues and alleviate the loss of contextual 
information. 





 §.§ Alternative Experiments






  
Alternative model settings.
In Table <ref>, we exploit some alternative experiments about the different model settings on the HICO-Det dataset. 

To validate the effectiveness of our proposed conditional controller in the context enhancement module, we removed it and used the original mask autoencoder to reconstruct the images. 

The results in the first row indicate that adding the original image as a conditional control can further guide the model to better reconstruct images and promote the learning of contextual clues.

In the second row of the table, we replaced the graph convolutional network with a transformer model in the instance feature alignment module. The results show that the transformer model is not well-suited for aggregating instance information. The main reason is that the transformer model does not consider the special relationship between prototype nodes and regular nodes. 

In the 3-4 rows of Table <ref>, we replaced the cross-attention with MLP and self-attention in the conditional controller, respectively. The results indicate that MLP cannot effectively serve as the conditional controller because it lacks the ability to fuse and control information compared to cross-attention. We observed that using both cross-attention mechanisms and self-attention mechanisms as conditional controllers yielded excellent performance, which proves that these two mechanisms can extract meaningful conditional control signals.

As shown in rows 5-6 of Table <ref>, we replaced the bidirectional graph with fully connected and directed graphs. Directed graphs refer to there are edges connecting from ordinary nodes to prototype nodes. In row 5 of Table <ref>, our bidirectional graph outperformed the fully connected graph by 0.92 mAP, indicating the importance of considering the special relationships between prototype and ordinary nodes. Row 6 indicates a slight impact of relationship propagation from prototype nodes to ordinary nodes.


  
Generate quantity settings.
As shown in Fig. <ref>, we studied the impact of the number of generated images on the HICO-Det.

We find that the performance of rare classes improves with an increasing number of generated images, but does not increase after the number of generated images exceeds 100.
In order to maintain a balance between performance and resource consumption, we decided to generate 50 images per rare class.



 §.§ Qualitative Evaluation

To more intuitively demonstrate the effectiveness of our approach, we visualize some examples of experimental results, as shown in Fig. <ref>. 

In the first row, we show the ground truth labels. The second row displays the predictions of the previous state-of-the-art method HOICLIP<cit.>. The third row represents the predictions of our  model.

In the figure, blue bounding boxes represent person, and green bounding boxes represent objects.

By observing the 1-2 columns, we find that the HOICLIP exhibits errors in the localization and classification of humans and objects, which subsequently affects the prediction of relationship interactions. In contrast, our model achieves more accurate localization and classification, enabling the correct prediction of relationship interactions.

Furthermore, by observing the the last three columns, we can observe that even the HOICLIP correctly detects humans and objects, errors can still occur due to failed relationship predictions. While our model is capable of improving this situation.




 §.§ Visualization of Feature Distribution

We present the distribution of features extracted by HOICLIP. As shown in Fig. <ref> (b), the generated image and original image features can be easily separated by domain. In contrast, our proposed  model is capable of learning domain-invariant features during the feature extraction phase. As illustrated in Fig. <ref> (c), after processing with our approach, the image features from different domains become significantly more aligned and are
not easily distinguishable.


§ CONCLUSION

In this paper, we introduce a model-agnostic framework that can be conveniently applied to existing HOI models and boost the performance of rare categories. Specifically,

we propose Context-Enhanced Feature Alignment  () module to align the generated data with the original data at the feature level.

The  helps the model overcome the domain gap, allowing the use of generated data for practical training. Experimental results demonstrate that our  achieves promising performance in terms of accuracy and scalability. We expect that this approach will provide a new paradigm for our community.





This work was supported by the National Natural Science Foundation of China (No.62102323), the National Natural Science Foundation of China (U23B2013) and the Innovation Capability Support Program of Shaanxi(Program No. 2023KJXX-142).

ACM-Reference-Format






 Paper Title: Learning Ordinality in Semantic Segmentation.















Learning Ordinality in Semantic Segmentation
    Rafael Cristino^1 Ricardo P. M. Cruz^1,2* Jaime S. Cardoso^1,2*
^1 Faculty of Engineering, University of Porto
^2 INESC TEC, Porto
 rpcruz@fe.up.pt, jaime.cardoso@fe.up.pt
    
===============================================================================================================================================================================



§ ABSTRACT 
Semantic segmentation consists of predicting a semantic label for each image pixel. Conventional deep learning models do not take advantage of ordinal relations that might exist in the domain at hand. For example, it is known that the pupil is inside the iris, and the lane markings are inside the road. Such domain knowledge can be employed as constraints to make the model more robust. The current literature on this topic has explored pixel-wise ordinal segmentation methods, which treat each pixel as an independent observation and promote ordinality in its representation. This paper proposes novel spatial ordinal segmentation methods, which take advantage of the structured image space by considering each pixel as an observation dependent on its neighborhood context to also promote ordinal spatial consistency.
When evaluated with five biomedical datasets and multiple configurations of autonomous driving datasets, ordinal methods resulted in more ordinally-consistent models, with substantial improvements in ordinal metrics and some increase in the Dice coefficient. It was also shown that the incorporation of ordinal consistency results in models with better generalization abilities.




§ INTRODUCTION


Semantic segmentation, or scene parsing, is the task of attributing a semantic label to each of the pixels in an image, resulting in a segmentation map. One common problem with these deep learning segmentation models is the lack of generalization ability, which means that the network fails to make appropriate predictions when parsing a situation that did not occur in the training dataset <cit.>. A hypothesis is that the neural network model does not have the necessary intrinsic domain knowledge of the task – it failed to infer appropriate high-level relations from the data used to train it. 

In many situations, there is an explicit ordering between the output classes, and by training the network with methods that uphold the ordinal constraints, the network may be able to learn better higher-level concepts, such as the ordinal relation between different objects (the lane marks and are inside the lane, etc) and the relative placement of objects (the sidewalk is to the side of the road, etc.).

Typically, ordinal problems have mostly been studied in the context of (image) classification <cit.>, where the task is to classify an observation (image) as one of 𝒞_1 ≺𝒞_2 ≺…≺𝒞_K ordered classes (for example, the severity of a disease), as opposed to nominal classes in the case of classic nominal classification. To our knowledge, only one work <cit.> attempted to introduce such ordinal relations for semantic segmentation, where 𝒞_1 ⊃𝒞_2 ⊃…⊃𝒞_K. However, the work does not explicitly perform spatial segmentation: it would be expected that an area labelled as 𝒞_k would only have a direct boundary to the areas segmented as 𝒞_k-1 and 𝒞_k+1, as exemplified in fig:ordseg_viz_example.






    
  * The ordinal relation between different objects, the lane marks and the lane itself – the lane marks are inside the lane;
    
  * The model should be more undecided between certain pairs of object classes, indecision between car/truck rather than between car/person;
    
  * The relative placement of objects, the sky or the sidewalk.





The transition from image classification to segmentation inherently evolves the prediction task from classifying independent samples (each image) to classifying structured dependent samples (each pixel in an image). The former can only take action in the representation space of each pixel, while the latter can also consider the structured space and the relations between the samples. Further developing this idea, the structured image space can be generalized to a graph, where each pixel is a vertex and is connected to its adjacent pixels.

The contributions presented in this paper can be summarized in:


  * The task of ordinal regression is formalized into ordinal consistency for each individual decision and ordinal consistency for the entire structure of the input, i.e. spatial consistency in the case of images (section <ref>).

  * Focusing on the latter case, two loss regularization terms (section <ref>) and one metric are proposed (section <ref>).

  * A thorough evaluation is performed for five biomedical datasets and an autonomous driving dataset.


The document is structured as follows: sec:sota introduces the state of the art; sec:proposal delineates the proposal; sec:experiments describes the conducted experiments; sec:results shows and analyzes the experimental results; and sec:conclusion revisits the proposed methods and summarizes the key conclusions.





§ STATE OF THE ART



Cross entropy is one of the most commonly used loss functions for image classification and segmentation problems. Defining cross entropy for a semantic segmentation problem,

    CE(y_n, p̂_n) = - 1/HW∑_i=1^H ∑_j=1^W ∑_k=1^K 1(y_n,i,j=k) log(p̂_n,k,i,j),

where p̂ is the model output as probabilities, in shape (N, K, H, W), where N is the batch size, K is the number of classes, and (H, W) are, respectively, the height and width of each image; y is the ground truth segmentation map, in shape (N, H, W), where each value y_n,i,j corresponds to the ground truth class k ∈{1,⋯, K} of the pixel at position (i,j) of observation n; and 1(x) is the indicator function of x.

Cross-entropy maximizes the probability of the ground truth class for each pixel in the observation, ignoring the distribution of the predictions for the other classes. This is a potential area where new loss functions can improve by restricting the probabilities of the non-ground truth class according to the domain knowledge of the task.



 §.§ Ordinal Classification Methods
 

Various research works seek to imbue deep neural networks with ordinal domain knowledge in the ordinal classification domain.



  §.§.§ Unimodality
 

The promotion of unimodality in the distribution of the model output probabilities has achieved good results in ordinal classification tasks <cit.>. This can be advantageous in ordinal problems because the model should be more uncertain between ordinally adjacent classes. For example, it would not make sense for a model to output a high probability for Low and High risk of disease, but a small probability for Medium risk of disease. fig:unimodality_pixel shows the difference between multimodal and unimodal distributions.



To promote unimodal output probability distributions, some authors have imposed architectural restrictions, which restrict the network output to a Binomial or Poisson probability distributions <cit.>. Other authors have promoted ordinality by penalizing the model when it outputs a non-unimodal distribution through augmented loss functions. One such case is the CO2 loss, which penalizes neighbor class probabilities if they do not follow unimodal consistency <cit.>,

    L_CO2(y_n, p̂_n) = L_CE(y_n, p̂_n) 
    + λ  L_O2(y_n, ŷ_n),

where L_O2 is the regularization term,

    L_O2(y_n, p̂_n)    = ∑_k=1^K-11(k ≥ y_n)  ReLU(δ + p̂_n,k+1 - p̂_n,k) 
       + ∑_k=1^K-11(k ≤ y_n)  ReLU(δ + p̂_n,k - p̂_n,k+1),

with δ being an imposed margin, assuring that the difference between consecutive probabilities is at least δ, and ReLU is defined as ReLU(x) = max(0, x).



  §.§.§ Ordinal Encoding


An approach to introducing ordinality to neural networks for classification involves regularizing the input data by encoding the ordinal distribution in the ground truth labels <cit.>. Defining k^⋆ as the ground truth class for a given sample, this input data encoding encodes each class as 1(k < k^⋆), whereas generic one-hot encoding encodes each class as 1(k = k^⋆).
This approach can be adapted for segmentation problems by similarly encoding the ground truth masks at a pixel level <cit.>. 

Using ordinal encoding for segmentation does not guarantee that the output probabilities are monotonous, the probability of ordinal class k, P_k, may be less than P_k+1. The consistency of the output class probabilities can be achieved by using,

    P(𝒞^+_k+1) = P(𝒞^+_k+1|𝒞_k^+) P(𝒞_k^+),
  
where P(𝒞^+_k+1|𝒞_k^+) is the (k+1)-th output of the network and P(𝒞_k^+) is the corrected probability of class k <cit.>. 

The state-of-the-art ordinal segmentation approaches have treated pixels as independent observations and promoted ordinality in their representation. However, this may be insufficient when applied to the structured image space, where pixels are dependent observations and a new level of consistency, spatial consistency, can be achieved.





§ PROPOSAL



Firstly, we present the formal foundation of our work (sec:foundation), then introduce the proposed ordinal segmentation methods, which are categorized into representation consistency (sec:proposal:pixel_wise), and structural consistency (sec:proposal:spatial). Finally, the ordinal segmentation problem is adapted to domains with arbitrary hierarchies (sec:proposal:hierarchies).



 §.§ Foundation



We start by recovering the definition of ordinal models, as introduced in <cit.>.
In a model consistent with the ordinal setting, a small change in the input data should not lead to a “big jump” in the output decision. Assuming f(𝐱) as a decision rule that assigns each input value 𝐱∈ℝ^d to the index ∈{1,2,…,K} of the predicted class, the decision rule is said to be consistent with an ordinal data classification setting in a point 𝐱_0 only if

    ∃ ε>0,  max_𝐱∈ℬ_ε(𝐱_0) f(𝐱)-min_𝐱∈ℬ_ε(𝐱_0) f(𝐱)≤1,

with ℬ_ε representing the individual feature-space neighborhood centered in 𝐱_0 with radius ε. Equivalently, the decision boundaries in the input feature-space 𝐱 should be only between regions of consecutive classes. Note that the concept of consistency with the ordinal setting is independent of the model type (probabilistic or not) and relies only on the decision region produced by the model. The state-of-the-art methods discussed in the previous Section focus only on this consistency (albeit often indirectly, by working on a related property, like the unimodality in the output probability space).



However, this consistency is only part of our knowledge in the ordinal segmentation setting.
In fig:ordinal-consistency, the feature description 𝐱∈ℝ at every pixel is mapped to the class according to the representation function

    f(x)=
    𝒞_1,  if    x<100
    𝒞_2,  if    100≤ x<200
    𝒞_3,  if    x≥ 200,

which is also depicted in fig:ordinal-consistency(a). Note that the model is ordinal-consistent in the pixel description space.
At most, a small change in x will change the decision to an adjacent class.

Consider now that the representation learned at each pixel v is given as 

    𝐱(v)=
    𝐱(v_0),      if   v∈ left half of the image 
    
    4𝐱(v_0),     if otherwise,

where 𝐱(v_0) is arbitrarily defined.
The model always sets the right half of the image to four times the values in the left half. Therefore, as illustrated in fig:ordinal-consistency(b), even being consistent at the pixel representation level, for some images (e.g., when the description of the pixels on the left takes the value 75), the decision will be 𝒞_1 for the left half and 𝒞_3 for the right half, which is to be avoided in an ordinal segmentation scenario. Therefore, we generalize the definition of ordinal-consistent models to models acting on structured observations, such as images or, more generally, graphs.

Consider each image as a graph 𝒢=(𝒱, ℰ), where 𝒱 denotes the set of graph vertices and ℰ⊂𝒱×𝒱 denotes the set of graph edges. 
Vertices in the image graph represent pixels, and because the values of a pixel are usually highly related to the values of its neighbors, there are undirected edges from a pixel to its neighboring pixels (often 4 or 8). The corresponding graph is then a 2D lattice.
Consider the goal of learning a function of signals/features which takes as input:


  * A feature description 𝐱_i = g(v_i) for every node v_i

  * A representative description of the graph structure 𝒢=(𝒱, ℰ)

and produces a node-level output f(𝐱_i).
Also, remember that the closed neighborhood of a vertex v in a graph 𝒢 is the subgraph of 𝒢, 𝒩[v], induced by all vertices adjacent to v and v itself, i.e., the graph composed of the vertices adjacent to v and v, and all edges connecting vertices adjacent to v.

We now define a model as ordinal consistent if the following two conditions are simultaneously met:


  * representation consistency: as before, if, for every point 𝐱_0 ∈ℝ^d, ∃ ε>0, max_𝐱∈ℬ_ε(𝐱_0) f(𝐱)-min_𝐱∈ℬ_ε(𝐱_0) f(𝐱)≤1, with ℬ_ε representing the feature-space neighborhood centered in 𝐱_0 with radius ε.

  * structural consistency: if, for every node v_i,
 max_v∈ N[v_i] f(g(v))-min_v∈ N[v_i] f(g(v)) ≤ 1

Figure <ref>(b) illustrates structural inconsistency and consistency – the representations of the pixels change smoothly over the image as given by

    𝐱(v)=
    𝐱(v_0),         if  v ∈left third of the image 
    𝐱(v_0) + 1,     if  v ∈middle third of the image 
    𝐱(v_0) + 2,     if  v ∈right third of the image.




 §.§ Representation Consistency for Ordinal Segmentation
 

In ordinal segmentation, ordinal representation consistency methods encompass those methods that act on the individual pixel representation 𝐱(v) and decision, they impose restrictions on the pixel, taking into account its own characteristics and disregarding the context of the neighboring pixels. Such methods include the ordinal pixel encoding and consistency methods discussed in the state-of-the-art analysis <cit.>. The previously introduced CO2 loss (<ref>) can be trivially adapted to segmentation by performing the regularization term for each pixel,

    L_O2(y_n, p̂_n) =    1/HW∑_i=1^W∑_j=1^H
    [ ∑_k=1^y_n,i,jReLU(δ + p̂_n,k,i,j - p̂_n,k+1,i,j) 
    
    +    ∑_k=y_n,i,j^K-1ReLU(δ + p̂_n,k+1,i,j - p̂_n,k,i,j) ].




 §.§ Structural Consistency for Ordinal Segmentation
 

The structural neighborhood concerns the neighborhood that relates one element with the others in the graph structure. In the case of semantic segmentation, the structural neighborhood is the set that contains all pixels connected to a given pixel. 



  §.§.§ Contact Surface Loss Using Neighbor Pixels
 





The contact surface loss is defined by penalizing the prediction of two neighboring pixels of non-ordinally adjacent classes. This penalization term is performed by multiplying the output probabilities for neighbouring pixels,

    L_CSNP(p̂_n,p̂_m) = ∑_n,m : 
    v_n∈ N [v_m]p̂_n^⊺ C p̂_m

where p̂_n^⊺ C p̂_n is a symmetric bilinear form, C is a K× K cost matrix, p̂_n∈ℝ^K is the vector of probabilities predicted at node v_n (similarly for 𝐩_m). C is defined as

    C_i,j = ReLU (|i-j| - 1) = 
    
    0,    if  |i-j|≤ 1 
    
    C_i,j = |i-j|-1,     otherwise,

to penalize ordinal-inconsistent neighboring probabilities,
where ReLU(·) is the Rectified Linear function, ReLU(x) = max(0,x). As an example, for K=4 classes, we obtain 

    C= [ 0 0 1 2; 0 0 0 1; 1 0 0 0; 2 1 0 0 ].

The intuition is to penalize high probabilities in spatially-close classes that are not adjacent.






  §.§.§ Contact Surface Loss Using the Distance Transform
 



Another approach is leveraging the distance transform, which is an image map where each value represents the distance from each pixel in the target image to its closest pixel with value 1, calculated with the customizable distance function d <cit.>. Defining the distance transform (DT) of the output probability map of class k, 

    = min_p_2 : p̂_n,k,p_2≥δ d(p_1, p_2),

where p = (i,j) and δ is the threshold parameter that selects the high-confidence pixels (typically 0.5), allowing the distance transform to be calculated for a high-certainty version of the output segmentation mask.




This provides a distance between pixels that have a high probability for a certain class and the closest pixel with a low probability of that class. That is, it provides an approximate distance between pixels of different classes. The model is trained to maximize this distance by multiplying the class probabilities map with the opposing class's distance transform,

    L_CSDT(p̂_n) = -1/|𝒮|∑_k_1, k_2 ∈𝒮 C_k_1, k_2   (p̂_n,k_1DT(p̂_n,k_2) 
        + p̂_n,k_2DT(p̂_n,k_1)),

where 𝒮 is the set of pairs (k_1, k_2) ∈{1,⋯, K} such that k_2-k_1 > 1.

At this stage, the  term maximizes the distance indefinitely. This is problematic because this may cause exploding distances, drawing the masks away from each other and possibly completely deviating from the ground truth. This can be solved by limiting the distance transform to a maximum distance, γ. This way, the loss only penalizes masks closer to each other than the γ value. For that reason, an updated distance transform was used, which saturates,

    ^γ = min (, γ).




 §.§ Partially Ordered Domains
 

The previous ordinal segmentation methods were proposed for ordinal domains with a (linear) total order in the set of classes, domains where 𝒞_1 ⊃𝒞_2 ⊃…⊃𝒞_K. This section proposes extending the previous work on ordinal losses to partially ordered output sets (the road may contain both lane marks and vehicles, but there is no relation between lane marks and vehicles).

Domains with a partial order in their set of classes must provide the segmentation methods with this partial order, the ordinal relations that motivate the ordinal constraints, as exemplified by Figure <ref>.



The ordinal segmentation metrics and losses proposed to total order sets can be extended to partially ordered sets.
Define first ℓ_m,n as the length of the shortest path from class m to class n in the Hasse diagram, or 0 if no such path exists. Note that ℓ_m,n≠ℓ_n,m and ℓ_m,n∨ℓ_n,m  = 0.

The previously introduced loss terms (L_O2, L_ and L_) are extended to arbitrary hierarchies as L_O2', L_' and L_'.

The L_O2' term can be redefined from (<ref>) by applying it between two neighboring classes, i.e. only when ℓ_m,n = 1, and if the edge (m,n) is part of a Hasse path that includes the ground-truth label k^⋆.

The C cost matrix in the L_' term can be defined as 

    C_m,n = ReLU(max(ℓ_m,n,ℓ_n,m)-1).
 

The adaptation L_' is similar to the adopted in L_O2'. 

Note that all these extensions revert to the base versions when applied to a totally ordered set.



 §.§ Evaluation Metrics




  §.§.§ Unimodal Pixels

We propose measuring the representation consistency using as a metric the percentage of Unimodal Pixels (UP), originally proposed by <cit.>, which consists of the fraction of times that the probability distribution produced by the model is unimodal.


  §.§.§ Contact Surface Metric

To evaluate the structural consistency, we propose to measure the percentage of ordinally invalid inter-class jumps between adjacent pixels, a metric for the contact surface between the masks of non-ordinally adjacent classes. Ordinally valid jumps are considered to be jumps between classes whose ordinal distance equals 1. If the ordinal distance between the classes of adjacent pixels exceeds 1, then that is an ordinally invalid jump. This requires that each pixel and its immediate neighborhood be examined during calculation. Defining the Contact Surface (CS) metric,

    CS(ŷ_n)    = 1/21 (CS_dx(ŷ_n)_i,j > 1)/1 (CS_dx(ŷ_n)_i,j > 0)
       + 1/21 (CS_dy(ŷ_n)_i,j > 1)/1 (CS_dy(ŷ_n)_i,j > 0),

where ŷ = _k=1^K(p̂), and CS_dx and CS_dy are the ordinal index variation, ordinal distance, from the current pixel (i, j) to the neighborhood, respectively, through the x and y axis,

    CS_dx(ŷ_n)_i,j = |ŷ_n,i, j - ŷ_n,i, j+1|
    CS_dy(ŷ_n)_i,j = |ŷ_n,i, j  - ŷ_n,i+1, j|







§ EXPERIMENTS
 



 §.§ Datasets



Various real-life biomedical datasets with ordinal segmentation tasks, where there is a clear ordering between classes, were identified from the literature <cit.>. table:sota:ordinal_datasets introduces the five biomedical datasets used to validate the proposed methods, along with a sample image and its corresponding segmentation mask.



To evaluate the ordinal methods on autonomous driving domains, the BDD100K <cit.> and Cityscapes <cit.> datasets were used:


  * BDD100K – is a multi-task, large-scale, and diverse dataset, obtained in a crowd-sourcing manner. Its images are split into two sets, each supporting a different subset of tasks: (1) 100K images – 100,000 images with labels for the object detection, drivable area, and lane marking tasks, and (2) 10K images – 10,000 images with labels for the semantic segmentation, instance segmentation, and panoptic segmentation tasks. The 10K dataset is not a subset of the 100K, but considerable overlap exists.

  * Cityscapes – is a large-scale and diverse dataset, with scenes obtained from 50 different cities. It provides 5,000 finely annotated images for semantic segmentation.


Two variants of the BDD100K dataset were considered: (1) BDD10K for the ordinary semantic segmentation task (10,000 images); and (2) BDDIntersected, which is the intersection of the 100K and 10K subsets and supports both the semantic segmentation and drivable area tasks (2,976 images). The models trained with BDD10K were subsequently tested with Cityscapes to validate the methods' generalization ability with out-of-distribution (OOD) testing. Furthermore, to evaluate how the proposed methods influence learning with scarce data, a dataset scale variation experiment was conducted with the BDD10K dataset.

To transpose semantic segmentation in autonomous driving to an ordinal segmentation problem, ordinal relations must be derived from the classes in the dataset. When analyzing an autonomous driving scene, fig:driving_scene1_bdd100k, we can, a priori, derive that, usually:



  * The vehicles will be on the road or in parking spaces;

  * The drivable area will be on the road;

  * The ego lane will be in the drivable area;

  * The sidewalk will be on either side of the road;

  * The pedestrians will either be on the sidewalk or the road;

  * The remainder of the environment surrounds the road.


Taking this domain knowledge into account, tab:segclasses introduces the ,  and  ordinal segmentation mask setups, including the ordinal relationship between classes in the form of a tree.  fig:driving_scene1_bdd100k(b) shows the  ordinal segmentation mask setup for the autonomous driving scene in fig:driving_scene1_bdd100k(a).







 §.§ Experimental Setup
 

The experimental results were obtained using the UNet architecture <cit.> with four groups of convolution blocks (each consisting of two convolution and one pooling layers) for each of the encoder and decoder portions[An open-source PyTorch implementation of the UNet architecture was used, <https://github.com/milesial/Pytorch-UNet>.]. All datasets were normalized with a mean of 0 and a standard deviation of 1 after data augmentation, consisting of random rotation, random horizontal flips, random crops, and random brightness and contrast. The networks were optimized for a maximum of 200 epochs in the case of the biomedical datasets and a maximum of 100 epochs in the case of the autonomous driving datasets, using the Adam <cit.> optimizer with a learning rate of 1e-4 and a batch-size of 16. Early stopping was used with a patience of 15 epochs. After a train-test split of 80-20%, a 5-fold training strategy consisting of 4 training folds and one validation fold was applied for each training dataset. For each fold, the best-performing model on the validation dataset was selected. An NVIDIA Tensor Core A100 GPU with 40GB of RAM and an NVIDIA RTX A2000 with 12GB of RAM were used to train the networks.

The metrics used were: (1) the Dice coefficient, which evaluates the methods with respect to the ground truth labels, (2) the contact surface, which evaluates the methods with respect to the spatial ordinal consistency, and (3) the percentage of unimodal pixels metrics, which evaluates the methods with respect to the pixel-wise ordinal consistency.

The cross-entropy loss and the methods by Fernandes  <cit.> were used as the baseline models. The methods to be evaluated had their parameterization, including the range of regularization term weights (λ), empirically determined. These methods are:



  * The semantic segmentation adaptation of the ordinal representation consistency L_CO2 loss function, with the imposed margin δ = 0.05, as recommended by the authors <cit.>;

  * The proposed ordinal structural loss for segmentation, L_CSNP
and L_CSDT, with the distance transform threshold δ = 0.5 and the maximum regularization distance γ = 10;

  * The mix of representation and structural methods, through the L_CO2 + L_CSNP loss combination of the two regularization terms.






§ RESULTS



Tables <ref>–<ref> depict the main results. Since each loss term proposed is actually a penalty that is weighted with cross-entropy (loss=CE+λ term), each loss was evaluated using grid-search, λ∈{0.1,1,…,10^4}, selecting the best value for each metric.

Table <ref> shows that the proposed losses generally offer an improvement, as judged by the Dice coefficient. Notice that this metric is computed for each pixel, explaining why CO2 (a loss that focuses on the pixel-wise ordinal representation consistency) generally performs better. Two additional tables are presented with the proposed ordinal metrics, evaluating the consistency for the representation space (% unimodality in Table <ref>) and the structural space (contact surface in Table <ref>). Naturally, CO2 again shows a very good performance for representation consistency, but the other losses start to become competitive as the structural consistency is evaluated using the contact surface metric.



Additional results are shown in plots that depict the influence of the weight of the regularization term, λ, on each respective metric. Figures fig:biomedical:plots:dice, fig:biomedical:plots:cs and fig:biomedical:plots:unimodal show, respectively, the Dice coefficient, contact surface, and unimodal metrics for the biomedical datasets. We can see that the ordinal methods successfully optimize the ordinal metrics, resulting in more ordinally consistent models. The dice coefficient metric does not improve much; however, excessive regularization can occur at some λ values, resulting in a lower Dice metric.

fig:bdd100k:plots:dice shows the Dice coefficient for the autonomous driving datasets, including the out-of-distribution domain testing. There, it can be seen that CO2 is the method with the highest impact on the generalization capability of the model, achieving a maximum absolute gain of 4.2% (11.5% in relative terms) at λ = 10, which means that using this loss helps the model generalize better to previously unseen scenarios. fig:results:co2:samples showcases a set of model inference outputs for this scenario, comparing cross-entropy outputs to CO2 with λ = 10.

To evaluate whether the inclusion of domain knowledge during the training of the models would help the network learn better with scarce data for autonomous driving scenarios, an experiment that consisted of varying the scale of the dataset used to train the models was performed. For each method, the best-performing lambda in the out-of-distribution scenario was selected, the rationale being that those models are the best at generalizing to unseen scenarios, which is useful when training with low amounts of data. For the L_CO2 and L_CSNP losses, λ = 10.0 was used, and for the L_CSDT, λ = 0.1 was used. fig:bdd100k:plots:scalevar:dice shows the Dice coefficient results for the dataset scale variation experiment.

When testing with the BDD10K dataset, the L_CO2 and L_CSNP losses achieve better Dice coefficient results at scales 0.25 and 0.5 when compared with the cross-entropy baseline, which suggests that, indeed, these losses help the network learn better when data is scarce. Especially when using the CO2 loss, resulting in absolute gains of 1.2% (5.7% in relative terms) in the Dice coefficient performance at scale 0.25 over cross-entropy. CO2 continues beating cross-entropy Dice performance through scales 0.1 and 0.05.

Cross-entropy generalizes better than the spatial methods when testing with the Cityscapes dataset in an out-of-distribution scenario for lower scales. However, the CO2 loss continues to generalize better than the cross-entropy throughout all scales, achieving a maximum absolute gain of 5.3% (15.7% in relative terms) in the Dice coefficient at scale 0.75. Still, it can be seen that the generalization ability of CO2 decreases at a higher rate than its performance on BDD10K – training with fewer data has a higher impact on the model's generalization ability.

Regarding the ordinal metrics, the scale variance does not significantly affect either metric.

In terms of compute complexity, since these are different losses, they do not affect inference time. Furthermore, any change in training time was too small to be detected.










 §.§ Discussion


A comparison between the biomedical and autonomous driving datasets results concludes that the biomedical results are significantly better in terms of the Dice coefficient performance – the performance gains with the autonomous driving datasets are still positive, but not as large, since these datasets are considerably more complex, with a greater variety of scenarios and segmentation classes.

The representation space, pixel-wise, L_CO2 performed well in both the biomedical and autonomous driving datasets, especially when considered in an out-of-distribution domain – it could potentially be applied in a real-world scenario in order to improve the generalization capabilities of perception algorithms.

The structured space, spatial, L_CSNP and L_CSDT losses, may not apply to the autonomous driving scenario in their current form, at least at high regularization weights. The scene perspective from the car makes it so that there is a large amount of valid contact surface between non-ordinally adjacent classes – in a 2D projection of the real world, the absolute minimization of these contact surfaces may not be the best solution since occlusions and different perspectives may originate legitimate contact between non-ordinally adjacent classes. Relaxed adaptations of these methods that consider this type of contact could be devised in the future. However, in the biomedical datasets, these methods performed well, with their greatest difficulty being the existence of occlusions.

As seen, the choice of λ value, regularization weight, is critical for the performance of the proposed methods. For this reason, in order to be applied to different domains, there should be an empirical study of the influence of the lambda value in the segmentation performance in that specific domain and the choice of a value that is a balance between the ordinal metrics and the Dice coefficient, a value that promotes some unimodality and spatial consistency but also does not hurt Dice performance to the point where it is unusable.





§ CONCLUSION



This paper explored ordinality in the context of structured data, proposing a formal definition of an ordinal model for generic graph structures. The concept was then specifically studied for semantic segmentation, where each image can be interpreted as a graph.

In this domain, two categories of loss functions for ordinal segmentation were studied: (1) an ordinal representation consistency loss, where each pixel is treated individually by promoting unimodality in its probability distribution, and (2) a structural consistency loss, where each pixel is considered in the context of its neighborhood and the contact surface between non-ordinally adjacent classes is minimized.

For that purpose, the following loss terms were proposed:


  * the segmentation adaptation of the representation L_CO2 loss;

  * the structural L_CSNP loss, which considers only the immediate neighbor pixels;

  * the structural L_CSDT loss, which considers the global neighborhood.

In addition, two metrics were proposed to evaluate the network output's ordinal consistency: (1) the percentage of unimodal pixels and (2) the contact surface between the segmentation masks of non-ordinally adjacent classes.

The proposed methods were initially validated on five biomedical datasets and two autonomous driving datasets, resulting in more ordinally consistent models without significantly impacting the Dice coefficient. To evaluate the methods' impact on the models' generalization capability, the resulting models were tested in an autonomous driving out-of-distribution scenario. Furthermore, the autonomous driving models were also tested with scaled-down versions of the BDD100K dataset to evaluate how the network learns with scarce data. The ordinal methods achieved maximum improvements in the Dice coefficient with an absolute value of 5.3% (15.7% in relative terms) in the out-of-distribution domain.

To summarize, incorporating ordinal consistency into semantic segmentation models showed promising results, including developing more generalizable models that exhibit improved learning capabilities with limited data availability. Since these are loss terms, they do not add time complexity to inference time.

Future research topics could include: (i) the development of more flexible spatial ordinal segmentation methods, allowing for limited contact between non-ordinally adjacent classes, such as in the case of occlusions and different perspectives; and (ii) the development of novel methods that leverage ordinal constraints not necessarily consisting of augmented loss functions.

Furthermore, the current work could contribute to other types of consistency. The partially ordered domains explored in this work were structural (e.g., a vehicle may be located on the road or in parking spaces), but a representational partially ordered domain could correspond to hierarchical segmentation <cit.>, where each pixel has a taxonomy (e.g., a motorcycle and bicycle are both two-wheels; two-wheels and four-wheels are both types of vehicles).

Also, part segmentation could be considered a different type of structural consistency, where parts of an object are next to each other. Super-pixels have been used to impose constraints, such as the “head” appearing above “upper body” or “hair” being above “head” <cit.>. Similarly, the function of Restricted Boltzmann Machines has been used to favor certain classes based on their spatial locations <cit.>. This work could be considered an alternative that promotes such structural consistency using loss penalties.



§ SOURCE CODE






IEEEtran




 Paper Title: Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection.






Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection]Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection













jhuang90@ur.rochester.edu

  University of Rochester
  Rochester
  NY
  USA


jpan24@ur.rochester.edu

  University of Rochester
  Rochester
  NY
  USA



wan.512@osu.edu

  The Ohio State University
  Columbus
  OH
  USA



hlyu5@ur.rochester.edu

  University of Rochester
  Rochester
  NY
  USA



jluo@cs.rochester.edu

  University of Rochester
  Rochester
  NY
  USA




§ ABSTRACT 
Recent advances show that two-stream approaches have achieved outstanding performance in hateful meme detection. However, hateful memes constantly evolve as new memes emerge by fusing progressive cultural ideas, making existing methods obsolete or ineffective. In this work, we explore the potential of Large Multimodal Models (LMMs) for hateful meme detection. To this end, we propose Evolver, which incorporates LMMs via Chain-of-Evolution (CoE) Prompting, by integrating the evolution attribute and in-context information of memes. Specifically, Evolver simulates the evolving and expressing process of memes and reasons through LMMs in a step-by-step manner. First, an evolutionary pair mining module retrieves the top-K most similar memes in the external curated meme set with the input meme. Second, an evolutionary information extractor is designed to summarize the semantic regularities between the paired memes for prompting. Finally, a contextual relevance amplifier enhances the in-context hatefulness information to boost the search for evolutionary processes. Extensive experiments on public FHM, MAMI, and HarM datasets show that CoE prompting can be incorporated into existing LMMs to improve their performance. More encouragingly, it can serve as an interpretive tool to promote the understanding of the evolution of social memes.[The source code will be made publicly available.] 

1.2Disclaimer. This paper contains offensive content that may be disturbing to some readers.










<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224.10010225.10010227</concept_id>
       <concept_desc>Computing methodologies Scene understanding</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Scene understanding










[
    Jiebo Luo
    
=============



§ INTRODUCTION




Hateful meme detection <cit.> is a crucial task in the field of multimodal research, aiming to identify content that combines text and images to propagate hate speech or offensive messages. 
Memes, as a widespread cultural phenomenon, proliferate on the Internet, blending images and texts to convey sophisticated meanings. 
Furthermore, the fusion of visual and textual elements in memes complicates the interpretation of their semantics and the identification of hateful undertones.
The combination of text and images presents significant challenges in hateful meme detection, especially for detecting and moderating hateful content <cit.>. 


With the advances in image-text pre-training, efforts to leverage and fine-tune CLIP <cit.> for hateful meme detection have demonstrated notable success in achieving high accuracy <cit.>. 
However, as shown in Figure <ref>(a), memes evolve similarly to biological organisms <cit.>, undergoing variation, mutation, and inheritance, which complicates the detection of hateful memes as they continuously evolve, merging with other cultural elements to form new expressions.

Traditional methods for hateful meme detection, as illustrated in Figure <ref>(b), suffer from limitations in interpretability and adaptability. 

These shortcomings hinder the understanding of memes' evolving nature and their contextual nuances, leading to overfitting on training datasets and diminished effectiveness in the dynamic and evolving real-world meme landscape.

To overcome this dilemma, we introduce Evolver, a novel approach that uses Large Multimodal Models (LMMs) <cit.> for hateful meme detection. By developing a benchmark specifically tailored for hateful meme detection based on LMMs, we aim to address the aforementioned limitations of traditional detection methods. Our approach not only enhances detection capabilities but also provides a more explainable and adaptable framework suitable for the evolving landscape of Internet memes. As shown in Figure <ref>(c), Evolver is an innovative approach to advance the state-of-the-art in zero-shot hateful meme detection significantly. Extensive experiments have shown the effectiveness of our method across three widely recognized hateful meme detection datasets, demonstrating a superior ability to identify and interpret hateful memes.

Overall, the main contributions of this work are:



  * We establish an LMM-based zero-shot hateful meme detection benchmark, which provides a comprehensive evaluation of the application of LMMs in social media.

  * We propose a simple yet effective Evolver framework, which advances LMMs with Chain-of-Evolution prompting. It expands LMMs with an evolution reasoning ability while offering good interpretability.

  * Extensive experiments on commonly used zero-shot hateful meme detection benchmarks with superior performance validate the efficacy and generalization of our method.




§ RELATED WORK

Hateful Meme Detection. Most works of hateful meme detection are based on pretrained vision-language models and fine-tune them on the hateful meme detection data. For example, <cit.> introduced the Hateful Memes Challenge and proposed a multimodal model that combines visual and textual features using a transformer-based architecture. Hate-CLIPper proposed a multimodal contrastive learning approach to improve the representation learning of memes by aligning the visual and textual modalities. These works demonstrate the effectiveness of using pretrained vision-language models. Differently, we discuss the potential of LMMs for hateful meme detection and introduce a knowledge-enhance LMM, Evolver.


Large Multimodal Models. 
With the release of the GPT-4V <cit.>, many researchers focus on the development and application of LMMs in processing and understanding multimodal data. Discuss groundbreaking models like GPT-4V and LLAVA-1.5 <cit.>, highlighting their capabilities in interpreting complex datasets that include a combination of text and images. Address the challenges these models face, such as sensitivity to noise and typographical errors, and the ongoing efforts to improve their accuracy and robustness in real-world applications. 

In this work, we adopt the LMMs to hateful meme detection, where set an LMM-based zero-shot hateful meme detection benchmark. Furthermore, we show that our Evolver further boosts the LMM's performance. 


Prompt-based Learning.
Prompt-based learning has evolved significantly from its inception with large language models (LLMs) to its current application in various tasks. The concept of prompt engineering emerged with the release of GPT-3, which demonstrated that LLMs could be guided to generate desired outputs through carefully crafted prompts. Inheriting the merit from LLM, prompt-based learning in large multimodal models (LLMs) has also gained extensive research attention. In our work, we present a novel chain-of-evolution prompting strategy to seamlessly boost the existing LMM for hateful meme detection. 





§ APPROACH

In this section, we first briefly describe the hateful meme detection task and LMM-based approaches. Next, we introduce the proposed Evolver.



 §.§ Background








  §.§.§ Problem Definition
 We define the task of hateful meme detection as a binary classification task. Meme, in our case, consists of an image-text pair. This task can be illustrated as follows: 

    ŷ_j = g({X^j_v, X^j_t}),
 where ŷ_j denotes the j-th prediction and the prediction ŷ_j ∈{0, 1} indicates the target image-text pair is hateful or not. g(·) is the multimodal model. {X^j_v, X^j_t} is the j-th image-text paired input.  






  §.§.§ Large Multimodal Models







Vision Encoder
serves as a translator to help language models understand visual content. It leverages frozen pretrained vision models such as CLIP <cit.> and ViT <cit.> to encode visual content so that the language model can understand visual content: 

    h_v = W·Enc_vis (X_v),

where h_v is the language embedding tokens. W is the projection to transform visual features into language embedding tokens. Enc_vis(X_v) denotes the visual feature extracted by pretrained model. 





Large Language Model Decoder generates a sentence given tokens <cit.>. The process of generation can be represented as: 

    h_t = Tokenize (X_t), 
     
        p(w) = ∏^n_i=1 p(w_i | w_<i, h_t, h_v),
where X_t is the input text, Tokenize(·) transform text into tokens, h_v is the image tokens, p(w) is the probability of generating a sentence by a language model, p(w_i | w_<i, h_t, h_v)  is the probability of generating a token at the i-th position given the previously generated tokens, input text, and image tokens. 
The visual tokens are incorporated with textual tokens and then fed to the language model. 





 §.§ Evolver: Chain-of-Evolution Prompting



To improve LMMs' understanding of the online hateful memes which are evolving in nature, we design a novel Chain-of-Evolution (CoE) prompt which has three components: (a) an evolutionary pair mining module to identify most relevant candidate memes; (b) an evolution information extractor to extract key information from the candidate memes; and (c) a contextual relevance amplifier for more effective hatefulness detection.
       



Evolutionary Pair Mining
It is unusual to find memes that have an evolutionary relationship with existing memes. Motivated by <cit.>, the evolution of a meme is defined as new memes that emerge by fusing other memes or cultural ideas. Therefore, the evolution of memes and old memes share similar textual and visual semantic regularities. We leverage this property to identify these old memes given the evolved meme. To this end, we can leverage the evolution of hateful memes to enhance LMMs' ability to understand hateful memes. In the implementation, we first generate the textual and visual embeddings from an external meme pool and target memes with CLIP and then fuse the textual and visual embeddings with a fixed ratio. Ideally, meme evolution has temporal ability, which means that the memes change with time. However, it is difficult to accurately locate the upstream and downstream of meme evolution with existing technology. Therefore, we use multiple evolutionary memes to find the common characteristics of this evolution. Finally, given a target image-text pair, we retrieve the top-K similar memes using cosine similarity:





    memes = {A_i | cos(𝐀, 𝐁)_i ∈Top_K(cos(𝐀, 𝐁)) },
 where 𝐀∈ℛ^n × d is embedding of n candidate memes and 𝐁∈ℛ^d is the  d-dimensional vector of the target meme. cos (·) return a similarity vector and Top_K(·) returns K highest values given the input vector. For each evolution meme, we pair K memes that the evolution meme is derived from. 





Evolution Information Extractor
To extract the information we are interested in (hateful component), we summarize paired memes with the help of a large multimodal model. The whole process can be expressed as follows:

    Info = LMM([memes_top_K, X_extract]),

where Info stands for our evolutionary information and LMM indicates the large multimodal model. memes_Top_K are the Top_K memes retrieved in the previous step, and X_extract is the instruction to guide the LMM to extract information. We present the  detailed instruction of X_extract as follows:



Contextual Relevance Amplifier
To enhance the in-context hatefulness information, we add a contextual relevance amplifier to the LMM during evolution information extraction and final prediction. The contextual relevance amplifier can help increase the search for hateful components. In our case, the contextual relevance amplifier is the definition of a hateful meme given by the dataset we use. Finally, we combine the extracted information and contextual relevance amplifier as the in-context enhancement and feed them to the model: 

    ŷ = LMM([memes_target, X_detect, Info, Amplifier]),

where Info stands for the information extracted previously, ŷ is the final prediction, memes_target is the memes we want to detect, and X_detect is the instruction to ask the large multimodal model to detect hateful memes. Amplifier refers to contextual relevance amplifier. The example of the amplifier is the same as the blue part in Table <ref>.










       






























































§ EXPERIMENT

We first describe three public datasets as well as the widely-used metrics and settings. Next, we introduce the baseline models including API-based (GPT-4V) and open-source LMMs (LLaVA) in Appendix <ref>. Finally, we present the evaluation and ablation study of our CoE prompting. 

To ensure a fair comparison, we create the LMM-based zero-shot hateful speech detection benchmark.
(a) For API-based LMMs, we select GPT-4V and Gemini-Pro-V, which are among the most popular models.
(b) For open-source LMMs, our comparison extends to include six widely used models: Openflamingo, LLaVA-1.5, MMICL, MiniGPTv2, BLIP-2, and InstructBLIP. 


GPT-4V <cit.> first creates a GPT-based LMM on the massive volume of datasets on the internet. They trained the language decoder, then aligned it with the vision encoder. 

Gemini-Pro-V <cit.> curates a large multimodal model with various image-text data on the web and then trains the model with image-text data from scratch.

Openflamingo <cit.> builds a LMM using CLIP as vision encoder and MPT/RedPajama as language decoder. The model is trained on open-source image-text datasets.

LLaVA1.5 <cit.> improves the performance of the model they propose previously <cit.> where CLIP is used as vision encoder and Vicuna as language decoder, with MLP projection to bridge the gap between vision encoder and language decoder. The model is pre-trained on the curated collected dataset. Furthermore, the model is instructionally fine-tuned with academic-task-oriented VQA data. 


MiniGPTv2  <cit.> connects the vision and language space with Q-former and an MLP projection. The model is pre-trained with a large collection of aligned image-text pairs and then instruction fine-tuned on curated high-quality image-text
pairs.

BLIP-2  <cit.> propose a model architecture using only Q-former to connect the vision encoder and language decoder to perform a series of multi-modal tasks. The pre-training of model has two stages. The initial stage initiates the learning of vision-language representations using a frozen vision encoder. Subsequently, the second stage advances the generative learning of vision-to-language from a stationary language model.

InstructBLIP  <cit.> further performs instruction fine-tuning based on BLIP-2 various converted multi-model datasets. 

MMICL <cit.> propose a framework 
using Q-former and an MLP layer to connect a vision encoder and language decoder. 




 §.§ More Experiment Details
 


Settings
We implement our Evolver based on the MMICL and LLaVA-1.5. For MMICL, we set the minimum length for the generation to 50, the maximum length for the generation to 80 during the Evolution Info Extractor, and the temperature to 0.2. During the final prediction, we set the minimum length of generation to 1, and the maximum length for generation to 50. For LLaVA-1.5, we set temperature to 0.2 and maximum generated tokens to 1024 for both stages. 

Effect of the Ratio of Textual and Visual fusion. 
The evolution of memes should consider both textual and visual information in a mixed form. We generate the textual and visual embeddings from the carefully curated meme pool and target memes with CLIP and then fuse the textual and visual embeddings with a 4:1 ratio in practice.

Effect of the External Curated Meme Pool. 
Generally, the external curated meme pool should have two characteristics (1) do not overlap with the test set. (2) have enough evolutionary information. To simplicity, we use the training set as the carefully curated meme pools, where the memes follow the same definition of hatefulness/harm/misogyny. 





Dataset Details. For more clarity, we show the detailed hateful meme detection dataset analysis in Table <ref>. Here, we give a detailed description of each dataset. 





 §.§ Datasets, Metrics, and Settings

Datasets We evaluate Evolver on three widely-used public datasets, namely, Facebook Hateful Meme dataset (FHM) <cit.>, Harmful Meme dataset (HarM) <cit.>, and Multimedia Automatic Misogyny Identification (MAMI) <cit.>.

Metrics 
We adopt ACC (accuracy) and AUC (area under the ROC curve) as evaluation metrics.

Settings
We implement our Evolver based on MMICL <cit.> and LLaVA-1.5 <cit.>. 

More details refer to the Appendix <ref>.












 §.§ Quantitative Analysis

Table <ref> shows the results of our method on three hateful meme detection datasets under the zero-shot setting. As we can see, our method consistently outperforms open-source LMM-based models across all datasets and metrics, which are closely aligned with API-based methods. Compared with the vanilla MMICL, our Evolver with the same architecture achieves higher accuracy (63.6% v.s. 60.4%) on the FHM dataset.




 §.§ Ablation Study

In Table <ref>, evolutionary pair mining (EPM) boosts the baseline with an improvement of up to 2.3%. Moreover, the evolutionary information extractor (EIE) and contextual relevance amplifier (CRA) significantly improve the generalization ability. Our full model achieves the best performance and outperforms the baseline by 8.5% in accuracy. This demonstrates that three parts of Evolver are beneficial by integrating with the evolution information.












 §.§ Qualitative Analysis

We visualize the paired evolution memes retrieved by the evolutionary pair mining modules in Figure <ref>, showing how the evolution information influences the final prediction.  The origin model (MMICL) without considering the meme's evolution, predicts non-hateful because it is not easy to detect the boy's disability or genetic conditions. After obtaining the evolution information like “discriminates against individuals with disabilities or genetic conditions.", 

our Evolver rectify the prediction, which supports the rationale of our method. 



 §.§ Computing Resource Requirements

Our Evolver is seamlessly integrated into the LMMs in the inference state, requiring minimal computational resources. For the main results, we conduct experiments on one GTX-3090-24G or one A100-40G. For the ablation study, we measure the comparison on one A100-40G.




 §.§ Impact of the Number of the Evolutionary Memes

As shown in Figure <ref>, we show the effect of the number K of the evolutionary memes. We find that our method significantly improves the baseline under all evolutionary meme number settings. Specifically, we set the K=5 to achieve the best performance in practice.




§ ADDITIONAL CASE ANALYSIS


To show the effectiveness of our method and align a better understanding of the hatefulness in the dataset, we show more examples generated by Evolver and the baseline model in Fig. <ref>.







§ CONCLUSION

We present Evolver to seamlessly boost LMMs for hateful meme detection via Chain-of-Evolution prompting. By integrating the evolution of memes, our method can adapt to unseen memes. Experimental results demonstrate the effectiveness of Evolver.



§ IMPACT STATEMENTS

Ethics Statement. Our Evolver aims to detect hateful memes by leveraging large multimodal models (LMMs) through a chain-of-evolution promotion strategy. While our objective is to mitigate the spread and impact of hate speech online, we recognize the potential for misuse or unintended consequences of this technology. Specifically, there are concerns about privacy, censorship, and the potential for biases within the AI model that could result in unfair or inaccurate content moderation. We are committed to ensuring that our technology is used ethically and responsibly.

Reproducibility Statement. We have clarified inference details including hyper-parameters, and the chain-of-evolution pipeline in Section <ref> and Appendix. In addition, all the datasets used in this paper are open-source and can be accessed online.

Limitation. While Evolver demonstrates significant advances in detecting hateful memes by leveraging the evolution of memes and large multimodal models (LMMs), it has some limitations. First, the effectiveness of our approach relies heavily on the quality and diversity of the curated meme pool used for seeking evolutionary memes. Furthermore, biases inherent in these datasets could potentially affect the model's ability to generalize across different cultural contexts and meme evolution patterns not represented in the related data. 




ACM-Reference-Format





 Paper Title: EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Head.





 



EmoTalk3D




Q. He et al.



^1 State Key Laboratory for Novel Software Technology, Nanjing University, China

^2 Fudan University, Shanghai, China   ^3 Huawei Noah's Ark Lab


EmoTalk3D: High-Fidelity Free-View Synthesis 
 of Emotional 3D Talking Head
    Qianyun He^1, Xinya Ji^1, Yicheng Gong^1, Yuanxun Lu^1, Zhengyu Diao^1, Linjia Huang^1, Yao Yao^1, Siyu Zhu^2, Zhan Ma^1, Songcen Xu^3, Xiaofei Wu^3, 
 Zixiao Zhang^3, Xun Cao^1, Hao Zhu^1
    
================================================================================================================================================================================================








§ ABSTRACT 
  We present a novel approach for synthesizing 3D talking heads with controllable emotion, featuring enhanced lip synchronization and rendering quality.  Despite significant progress in the field, prior methods still suffer from multi-view consistency and a lack of emotional expressiveness. To address these issues, we collect dataset with calibrated multi-view videos, emotional annotations, and per-frame 3D geometry.  By training on the dataset, we propose a `Speech-to-Geometry-to-Appearance' mapping framework that first predicts faithful 3D geometry sequence from the audio features, then the appearance of a 3D talking head represented by 4D Gaussians is synthesized from the predicted geometry. The appearance is further disentangled into canonical and dynamic Gaussians, learned from multi-view videos, and fused to render free-view talking head animation.  Moreover, our model enables controllable emotion in the generated talking heads and can be rendered in wide-range views.  Our method exhibits improved rendering quality and stability in lip motion generation while capturing dynamic facial details such as wrinkles and subtle expressions.  Experiments demonstrate the effectiveness of our approach in generating high-fidelity and emotion-controllable 3D talking heads. The code and dataset are released in <https://nju-3dv.github.io/projects/EmoTalk3D>.

  



§ INTRODUCTION



3D Talking heads refer to synthesizing a person-speaking animation given a speech, which has high applicability in various scenarios, such as digital humans, chatting robots, and virtual conferences.  
The core challenge of this task lies in accurately mapping speech signals to 3d lip movements, facial expressions, and, potentially, emotions.  So this task is highly complex and demands meticulous techniques along with sophisticated algorithms.

Despite numerous studies dedicated to the study of 3D talking heads, state-of-the-art methods still encounter evident challenges. In terms of rendering quality, there remains ample scope for enhancing lip synchronization accuracy. Additionally, the intricate facial expressions, including wrinkles and subtle expressions, are not accurately synthesized. These deficiencies result in artifacts within the rendered animations, thereby diminishing their overall fidelity. Furthermore, animations generated by previous 3D talking head methods often overlook emotional expression, thus restricting users' capacity for emotional communication. A key impediment to the research of emotional 3D talking heads is the lack of speech and 4D head datasets that incorporate emotion annotations. Though multi-view video datasets for emotion annotation are available <cit.>, they fall short in providing accurate camera calibration and dynamic 3D models.

In this paper, we proposed a novel approach for synthesizing 3D talking heads with controllable emotion and an emotion-annotated calibrated multi-view video dataset for training our model. As shown in Fig. <ref>, the model renders high-fidelity free-view animations given the audio. The emotion labels used for driving can be set artificially and changed freely in time sequence. The detailed wrinkles and shadings caused by facial motions have been vividly synthesized. 
Such excellent performance benefits from two innovations, which are described below.

Firstly, we present a `Speech-to-Geometry-to-Appearance' framework to map input speech to the dynamic appearance of a talking head. Specifically, a Speech-to-Geometry Network (S2GNet) is first used to predict 4D point clouds from audio features, where expression and lip motion are accurately recovered. Then, a 4D Gaussian model is established based on the predicted 4D points to represent the facial appearance efficiently. The appearance is disentangled into canonical Gaussians (static appearance) and dynamic Gaussians (facial motion-caused wrinkles, shading, etc). Geometry-to-Appearance Network (G2ANet) is introduced to learn the dynamic appearance from the multi-view videos and render free-view talking head animation. Experiments show that the proposed method generates more accurate and stable mouth shapes and models the dynamic details of facial motion, including wrinkles at specific facial expressions.

Secondly, we establish dataset, an emotion-annotated multi-view talking head dataset with per-frame 3D facial shapes. dataset are captured from 35 subjects, and the data for each subject contains 20 sentences under 8 specific emotions with two emotional intensities, summing to 20 minutes for each subject. 
We leverage an emotion extractor to parse emotion status from the input audio, then design an emotion-guided 4D prediction network to achieve emotion control over 3D talking head generation. The S2GNet is designed to be conditioned on emotion labels for emotion control, and G2ANet further synthesizes detailed expressions and dynamic details for specific input emotions. In this way, the emotion of our generated 3D talking head can be manipulated by manually set emotion labels.

The main contributions of this work can be summarized as:


    
  * We establish dataset, an emotion-annotated multi-view talking head dataset with per-frame 3D facial shapes. Based on this unprecedented dataset, we propose the first explicit emotion-controllable 3D talking head synthesis method.
    
  * A `Speech-to-Geometry-to-Appearance' mapping framework is introduced to enhance lip synchronization and overall rendering quality of a 3D talking head.
    
  * A 4D Gaussians model is proposed for representing the appearance of a 3D talking head, effectively synthesizing dynamic facial details such as wrinkles and subtle expressions. 





§ RELATED WORKS





 §.§ Audio-driven Talking Head


The key task of audio-driven talking heads is to establish correspondence between head motion and audio signals <cit.>, and can be categorized into 2D-based and NeRF-based method according to heads' representation.

2D-based Talking Head.
Early learning-based talking heads <cit.> leverage the encoder-decoder architecture to synthesize a talking head video.
To further improve the lip-synchronization, some works <cit.> extract disentangled appearance and semantic representations from speech or train an evaluation network targeted for synchronous lip movement. 
Later, generative models are introduced to produce talking heads directly from the extracted features or intermediate representation like facial landmarks <cit.>.
Other 2D methods study how to edit full-frame videos, including the portrait's head, neck, and shoulder, often in dynamic backgrounds. Most existing methods <cit.> synthesized the mouth-related area of the video and then blended it into the whole frame without altering other regions. 
Differently, other methods <cit.> apply the re-timing strategy to find the optimal target frame with the predicted mouth shape or leveraged landmark as an intermediate representation <cit.> with image-to-image translation networks to generate full-frame head animations.  
However, due to the lack of explicit 3D structure information, these methods fail to generate natural talking heads with consistent head poses.
 
NeRF-based Talking Head.
Recently, Neural Radiance Field (NeRF) <cit.> has gained much attention for generating photo-realistic rendering of objects by using neural networks to learn the shape and appearance of the object under different spatial locations and viewpoints. Notably, Guo  <cit.> proposed AD-NeRF, which utilizes audio-conditioned NeRF to synthesize the scene of a talking head for the first time. However, the head and the torso are learned from two sets of NeRF, leading to inconsistent results. To address this issue, Liu  <cit.> introduced SSP-NeRF, which incorporates an additional facial parsing brunch to improve the rendering efficiency and a torso deformation module to model the non-rigid torso deformations within a unified neural radiance field.  Additionally, DFA-NeRF <cit.> disentangled head pose, eye blink, and lip motion to synthesize personalized animation. To further accelerate training, DFRF <cit.> learned a base model for lip motion that can be quickly fine-tuned to different identities, achieving a few-shot talking head synthesis. ER-NeRF <cit.> resorts to optimizing the contribution of spatial regions by using a Tri-Plane Hash Representation. Differently, GeneFace <cit.> adopted a generative audio-to-motion model to improve the lip motion performance on out-of-domain audio. 
Different from all previous works, we leverage 3D Gaussian Splatting <cit.> to model dynamic facial motion while achieving superior visual quality with real-time rendering.



 §.§ Emotional Talking Head Generation


As a crucial factor in facial communication, emotion can enhance the authenticity of talking head animation. However, most previous researches focus on synchronizing lip motion with audio content, while neglecting facial expressions. Additionally, the absence of a large-scale audio-visual dataset with emotion annotations contributes to the challenge. To tackle this problem, Wang  <cit.> introduced the MEAD dataset, comprising multi-view audio-visual data with eight emotions. Building upon this dataset, Ji  <cit.> proposed EVP to decouple content and emotion information from the audio signal, facilitating facial emotion synthesis. Differently, some works resort to emotion labels <cit.> or extract facial expressions from additional emotional videos <cit.> for emotion retrieval. In this work, we also go beyond mouth motion and expand into synthesizing facial expressions by extracting emotion and content from the audio signal to achieve precise emotion control.



 §.§ 3D Talking Head Dataset


Most available talking head datasets have been captured in a controlled indoor environment. For example, Cudeiro  <cit.> collect VOCASET, a dataset of 4D head models created using 3D scanning and motion capture technology. Similarly, Wuu  <cit.> present Multiface, which contains high-quality human head recordings under various facial expressions using a multi-view capture stage. Meanwhile, Pan  <cit.> produce RenderMe-360 with rich annotations including different granularities. FaceScape <cit.> proposed to collect 3D faces in multiple standardized expressions for each subject to generate riggable 3D blendshapes, which can be driven by 3DMM coefficients to generate 3D talking face videos <cit.>. However, this strategy has poor accuracy in modeling lip and facial expressions, due to the limited representation ability of Blendshapes.
Other studies <cit.> have also presented multi-view audio-visual data with the potential to reconstruct 3D heads. However, only front and side view data were provided in Lombard <cit.>. While MEAD <cit.> recorded emotional audio-visual clips at seven views, they lack camera calibration data, which brings unstable reconstruction results. Another type of work like HDTF <cit.> collected a large in-the-wild audio-visual dataset and leverages the 3D morphable model (3DMM) to fit the monocular video, but the fitting results are typically coarse.  Different from all prior works above, our work presents the dataset, an emotion-annotated 4D dataset with speeches and per-frame 3D facial shapes. 



§ DATASET



We present dataset, an emotion-annotated multi-view face dataset with reconstructed 4D face models and accurate camera calibrations. The dataset contains 30 subjects, and the data for each subject contains 20 sentences under 8 specific emotions with two emotional intensities, summing to roughly 20 minutes for each subject. The collected emotions include `neutral', `angry', `contempt', `disgusted', `fear', `happy', `sad', and `surprised'. Except for `neutral', two intensities - `mild' and `strong' are captured for each emotion. We invited professional performance instructors to guide the subjects in expressing the correct emotions. 

For data acquisition, we built a dome with 11 video cameras in the same horizontal plane as the subject's head and evenly around the head in a 180^∘ degree focusing on the frontal face.  All cameras are temporally synced with a hardware synchronization system and are calibrated before capturing videos. State-of-the-art multi-view 3D reconstruction algorithm <cit.> is leveraged to reconstruct accurate 3D triangle mesh models, which are then transformed into topologically uniformed 3D mesh models <cit.>. The 3D vertices of the 3D surface model corresponding to each frame constitute the 3D points stream, namely 4D points, which are used for training our 3D talking face model. 

To the best of our knowledge, dataset is the first talking face dataset that contains both emotion annotations and per-frame 3D facial geometry. The meta parameters of our dataset and previous talking face datasets are compared in Table <ref>. The dataset has been publicly released for research purposes on our project page. More details about the dataset are explained in the supplementary material.







§ METHOD






As shown in Fig. <ref>, our method consists of the following modules: 1) the Audio Enocoder that encodes audio features and extracts emotional labels from input speech; 2) Speech-to-Geometry Network (S2GNet) that predicts dynamic 3D point clouds from the audio features and emotional labels; 3) Static Gaussians Optimization and Completion Module for establishing a canonical appearance model; 4) Geometry-to-Appearance Network (G2ANet) that synthesizes facial appearance based on dynamic 3D point cloud. The above modules together constitute the `Speech-to-Geometry-to-Appearance' mapping framework for emotional talking head synthesis.





 §.§ Speech-to-Geometry



As shown in Fig. <ref>, the speech-to-geometry module consists of two parts: an audio encoder that converts the input speech into audio features and the S2GNet that converts audio features and emotional labels into 3D mesh sequences. Concretely, a pre-trained HuBERT speech model <cit.> is used as the audio encoder. 
Drawing inspiration from Baevski 's method <cit.>, we employ a context network adhering to the transformer architecture <cit.> as the backbone of our emotion extractor. The model is initialized with the pre-trained weights from Baevski 's work and then fine-tuned with ground-truth emotional labels and speech audio in our dataset.   

The design of S2GNet follows FaceXHubert <cit.>, a cutting-edge audio-to-mesh prediction network. Specifically, S2GNet receives extracted emotion labels and encoded audio features as input, regressing vertex displacements based on the mean template mesh, ultimately producing the 4D talking mesh sequence. Rather than relying on the commonly used transformer-based networks, S2GNet opts for the Gated Recurrent Unit (GRU) <cit.> as its core architecture. Through experiments, we have verified that this approach achieves superior performance in lip-synchronization and speech generalization. 

We leverage FaceXHubert's pre-trained model to initialize the training of S2GNet, then finetune S2GNet on our dataset. The triangle mesh model predicted by S2GNet was quadrupled upsampled, which is then used as the input to the geometry-to-appearance module.  More details about the training and networks are illustrated in the supplementary material.



 §.§ Geometry-to-Appearance



After predicting the 4D facial points, we introduce the Geometry2Appearance Network (G2ANet) to synthesize the 3D Gaussian features <cit.> of the talking face by taking 4D points as input, as shown in Fig. <ref>. 

The design of G2ANet is founded upon two key observations. Firstly, we recognize that for a specific individual's talking head, facial movements produce relatively minor variations in appearance. With this in mind, we initially train a Gaussian model on a static, non-speaking head, referred to as Canonical Gaussians. Subsequently, a neural network is employed to predict appearance changes caused by taking motions, using Canonical Gaussians and 4D point-derived facial movements as inputs. These predicted variations are named Dynamic Gaussians. Secondly, given that there is no direct correlation between speech and non-facial features such as hair, neck, and shoulders, the S2GNet Network disregards predicting the geometric structure of these elements. Consequently, G2ANet learns to replicate the appearance of these non-facial parts and seamlessly integrates them with the face, as elaborated in Section <ref>.


Canonical 3D Gaussians.
Following 3D Gaussian Splatting (3DGS) <cit.>, we represent the static head as 3D Gaussians that are parameterized 3D points. Each 3D Gaussian is represented by the 3D point position μ and covariance matrix Σ, and the density function is formulated as:


    g(x) = e^-1/2(x-μ)^T Σ^-1(x-μ)


As 3D Gaussians can be formulated as a 3D ellipsoid, the covariance matrix Σ is further formulated as:


    Σ = 𝐑𝐒𝐒^T𝐑^T


where 𝐒 is a scale and 𝐑 is a rotation matrix. The 3D Gaussians are differentiable and can be easily projected to 2D splats for rendering. In the differentiable rendering phase, g(x) is multiplied by an opacity α, then splatted onto 2D planes and blended to constitute colors for each pixel. The appearance is modeled with an optimizable 48-dimension vector 𝐇 representing four bands of spherical harmonics. In this way, the appearance of a static head can be represented as 3D Gaussians 𝐆:


    𝐆←{μ, 𝐒, 𝐑, α, 𝐇}


where ← means G is a set of parameterized points, each represented by a parameter set on the right of the arrow.

In our approach, the canonical 3D Gaussians 𝐆_o represent a static head avatar and are learned from multi-view images of a moment without speech, usually the first frame of a video clip. The canonical 3D Gaussians 𝐆_o are denoted as:


    𝐆_o ←{μ_o, 𝐒_o, 𝐑_o, α_o, 𝐇_o}


Dynamic Detail Synthesis. To generate a 3D talking face with 3D Gaussians, we propose to predict the appearance at t moment with dynamic details. The dynamic details mean the detailed appearance due to the talking facial motion, such as specific wrinkles and subtle expressions. For a specific subject, the opacity α and scale 𝐒 remain unchanged and equal to α_o and 𝐒_o respectively. The Gaussians at t time 𝐆_t is formulated as:


    𝐆_t ←{μ_t, 𝐒_o, 𝐑_t, α_o, 𝐇_t}


As μ_t has been predicted in the speech-to-geometry network, only 𝐇_t and 𝐑_t are unknown and are predicted by FeatureNet and RotationNet, respectively:


    𝐇_t = FeatureNet(𝐇_o, e, p, δμ)



    𝐑_t = RotationNet(𝐑_o, e, p, δμ)


where e is the emotion vector; p is the position in the UV coordinate predefined for each point; δμ is defined as:


    δμ = μ_t - μ_o - 1/N∑_i ∈ G(μ_t^i - μ_o^i)


FeatureNet and RotationNet are 4-layer and 2-layer MLPs, respectively. The detailed architectures of the two networks are reported in the supplementary material. 

Through experiments, we find that combining canonical Gaussians with dynamic Gaussians enables the prediction of highly detailed dynamic facial appearance. These include wrinkles formed during angry or other expressions and nuanced appearance variations resulting from lip movements. No preceding talking head model has achieved such meticulous rendering of dynamic facial details.





 §.§ Head Completion
 


The speech-to-geometry network only predicts the 3D point cloud of the face, as the geometry other than the face, such as hair, neck, and shoulders, are not strongly correlated to the speech signals. Therefore, the regions other than the face do not have accurate initial points for optimizing Gaussians, so we add 85,500 uniformly distributed points in the space to constitute the regions other than the face by 3DGS optimization, as shown in Fig. <ref>.

The points other than the facial region (OTF points for short), including hair, neck, and shoulders, should follow the face point motion to a certain extent. We observed that these points tend to be stationary at points far from the face, such as the shoulders, while points close to the face will move together, such as the hair and neck. So we formulate the motion of the points except for the facial points δμ_otf as:


    δμ_otf = δμ_f· e^-α d


where δμ_f is the movement of a facial point that is closest to this OTF point; r is the distance between the OTF point and its closest facial point; α is the decay factor and is set to 0.1 in all our experiments.

In the training of dynamic Gaussians, we observed that training the Gaussians on hair and shoulders the same way as we did for faces would cause severe blur. It is because the facial points' position in our dataset is accurately recovered, so the Gaussian points can be more accurately aligned on every frame within a video clip. By contrast, the appearance of regions outside the face, including hair, neck, and shoulders, is optimized from random points. The 3D positions of these OTF points are unreliable, so these points will correspond to misaligned appearances at different frames, leading to a vague appearance predicted. On the other side, the complete head represented by the canonical Gaussians is clear, however, the appearance of the canonical Gaussians is static.

To solve this problem, we propose to treat the dynamic part (the face) and the static part (the area above the shoulders other than the face) separately, using dynamic Gaussians for the former and canonical Gaussians for the latter. Specifically, we pre-define a natural transition facial weight mask, which is applied to opacity α to achieve a natural fusion of dynamic and static parts. The weight W_p for point p is formulated as:


    W_p = {[                  0,          d<d_0; d-d_0/d_th,      d_0≤ d ≤ d_0+d_th;           1,          d_0+d_th < d ].


where d is the distance between point p and its closest facial point; d_0 and d_th are transition factors and are set to 5mm and 10mm, respectively. 
The opacity field for canonical Gaussians and dynamic Gaussians are multiplied by W_p and (1-W_p), respectively. 
In this way, the talking facial appearance and the other appearance (hair, neck, and shoulders) are fused to synthesize a clear and complete 3D talking head.



§ EXPERIMENTS





 §.§ Implementation Details



Data pre-processing. Before training our model, we crop and resize the raw frames into a squared image at resolution 512×512. A matting algorithm <cit.> is used to remove the background. For each subject, we randomly divided the data into a training set and a test set in a ratio of 4:1.

Training details. 
Adam<cit.> optimizer is adopted to train the speech-to-geometry network (S2GNet) and geometry-to-appearance network (G2ANet). 
In the training of G2ANet, we first obtain canonical Gaussians by training with the 1st frame of the video with neutral expression and closed mouth. The number of facial points and the OTF points are 100 and 200, respectively. The clone and splitting are prohibited to ensure a stable correspondence association of points between frames. Then, FeatureNet and RotationNet are trained to predict the dynamic details. Due to space limitations, more implementation details are explained in the supplementary material.

The renderings of our results at different views with different emotions are shown in Fig. <ref>.  It can be seen that high-quality frames are rendered at large-angle views with correct emotion synthesized. We recommend watching the supplementary video to evaluate lip-sync and rendering performance qualitatively.



We compare our method with several image-based methods (MakeItTalk <cit.>, EAMM <cit.>, SadTalker <cit.>, DreamTalk <cit.>), three 3D methods (AD-NeRF <cit.>, Real3D-Portrait <cit.>, Next3D <cit.>), and two speech-to-mesh methods (VOCA <cit.>, MeshTalk <cit.>). PSNR <cit.> and SSIM<cit.> are adopted to evaluate the overall image quality and LPIPS<cit.> is adopted to measure the perceptual similarity between the results and the ground truth. Landmark Distance (LMD)<cit.> is utilized to evaluate lip synchronization, and Cumulative Probability of Blur Detection (CPBD) <cit.> is used to measure the sharpness of the result images.







 §.§ Qualitative and Quantitative Evaluations


As shown in Fig. <ref> and Tab. <ref>, our results outperform all previous 2D image-based and 3D-based talking heads.  More importantly, our result can be rendered at -90^∘ - +90^∘ and explicitly emotion-controllable, as shown in Fig. <ref>, which all other methods cannot achieve. Our method also leads in the scores of PSNR, SSIM, LPIPS, and LMD, which indicates that our method improves rendering quality and lip-sync compared to previous works.

User Study. 
We conduct a user study to evaluate the quality of the synthesized portraits by comparing the real data with the generated ones from different approaches. Specifically, we sample 10 generated videos from the test set with various identities and viewpoints. 27 participants are invited to rate these videos with a score from 1 (worst) to 5 (best) w.r.t three aspects: speech-visual synchronization, video fidelity, and image quality. A more detailed explanation of these rating criteria is provided in the supplementary material. As VOCA and MeshTalk only produce mesh with no texture, the image quality for them is not evaluated. The results are shown in Table <ref>.  Our method obtains the highest score on all aspects, indicating better performance in speech-visual synchronization, video fidelity, and image quality.





 §.§ Ablation Study






We conducted three sets of ablation studies to analyze the effectiveness of each module in our method. The comparison results are shown in Fig. <ref>.

∙ (A) w/o Geometry. We compare the `speech-to-geometry-to-appearance' framework with the traditional `speech-to-appearance' framework to analyze the effectiveness of the former. In the `speech-appearance' framework, all parameters of 3D Gaussians, that is Dynamic 3D Gaussians 𝐆_t ←{μ_t, 𝐒_o, 𝐑_t, α_o, 𝐇_t}, are directly predicted from the audio features using several networks. We adopted the same design of FeatureNet and RotationNet to build the `speech-to-appearance' model, only made changes to how we obtain μ_t - a network with comparable parameters of S2GNet is used to predict δμ to calculate μ_t, and is trained in an end-to-end manner.  
As shown in Fig. <ref> and Tab. <ref>, introducing geometry leads to much higher image quality. Though PSNR and SSIM scores of `w/o Geometry' are higher, the rendering results clearly show that the facial details are all missing. We believe that this is because the introduction of geometry decomposes the complex Speech-to-Appearance mapping problem into two more stable and straightforward mappings - Speech-to-Geometry and Geometric-to-Appearance. This strategy reduces the complexity of 3D talking head synthesis, which makes it easier for neural networks to learn. 

∙ (B) w/o Emotion. The proposed method leverages explicit extraction of emotional components in speech for generating talking face animation (Section <ref>). We try to remove the emotion extractor and emotion labels in the S2GNet to evaluate the performance without emotion extraction and embedding.  As shown in Fig. <ref> and Table <ref>, the target emotion is not accurately synthesized in the generated videos after removing emotion-related designs, demonstrating the effectiveness of emotion extracting and encoding. 

∙ (C) w/o FeatureNet. In G2ANet (Section <ref>), we propose to leverage FeatureNet to model the dynamic appearance. Here, we remove the FeatureNet from the design and evaluate the performance. As shown in Fig. <ref>, the dynamic wrinkles are weakened and the expressions tend to be neutral. The overall facial appearance is degraded towards a canonical appearance, which demonstrates the effectiveness of the FeatureNet in G2ANet.



§ CONCLUSION



In this paper, we propose to synthesize a high-fidelity, emotion-controllable 3D talking head that can be rendered over a wide range of viewing angles. A `Speech-to-Geometry-to-Appearance' mapping framework with dynamic 4D Gaussians is proposed for better lip-sync and rendering quality. A multi-view video dataset with emotion annotation and per-frame 3D facial shapes is presented for learning 3D talking heads. Our data and algorithm can be the foundation for future research about emotion-controllable 3D talking heads.

Limitation. Our model is person-specific, which means only one identity can be generated by training a neural network. Therefore, our method relies on a well-calibrated multi-view camera system to collect videos for training, and cannot synthesize human appearance from a single image. In addition, our method fails to model dynamic flapping hair, which is also a challenging problem.

Acknowledgements This study was funded by NKRDC 2022YFF0902400, NSFC 62441204, and Huawei.










splncs04



 Paper Title: Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey.












Miyai et al.: 
Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey


§ ABSTRACT 
Detecting out-of-distribution (OOD) samples is crucial for ensuring the safety of machine learning systems and has shaped the field of OOD detection. Meanwhile, several other problems are closely related to OOD detection, including anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and outlier detection (OD). To unify these problems, a generalized OOD detection framework was proposed, taxonomically categorizing these five problems. However, Vision Language Models (VLMs) such as CLIP have significantly changed the paradigm and blurred the boundaries between these fields, again confusing researchers. In this survey, we first present a generalized OOD detection v2, encapsulating the evolution of AD, ND, OSR, OOD detection, and OD in the VLM era. Our framework reveals that, with some field inactivity and integration, the demanding challenges have become OOD detection and AD. In addition, we also highlight the significant shift in the definition, problem settings, and benchmarks; we thus feature a comprehensive review of the methodology for OOD detection, including the discussion over other related tasks to clarify their relationship to OOD detection. Finally, we explore the advancements in the emerging Large Vision Language Model (LVLM) era, such as GPT-4V. We conclude this survey with open challenges and future directions.



Anomaly Detection, Novelty Detection, Open Set Recognition, Out-of-Distribution Detection, Outlier Detection, Vision Language Model, CLIP, Large Vision Language Model, Large Multi-modal Model


Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey
    Atsuyuki Miyai, Student Member, IEEE, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, 
 Qing Yu, Member, IEEE, Go Irie, Member, IEEE, Shafiq Joty, Yixuan Li, Member, IEEE, Hai Li, Fellow, IEEE, Ziwei Liu, Member, IEEE, Toshihiko Yamasaki, Member, IEEE,  Kiyoharu Aizawa, Fellow, IEEE

A. Miyai, Q. Yu, T. Yamasaki, and K. Aizawa are with the Department of Information and Communication Engineering, The University of Tokyo, Japan, 1138656. Q. Yu is also with LY Corporation, Japan, 1028282.


E-mail: {miyai,yamasaki}@cvm.t.u-tokyo.ac.jp, {yu,aizawa}@hal.t.u-tokyo.ac.jp
J. Yang and Z. Liu are with 
S-Lab, School of Computer Science and Engineering, 
Nanyang Technological University, Singapore, 639798.


E-mail: {jingkang001,ziwei.liu}@ntu.edu.sg
J. Zhang, Y. Lin, and Hai Li are with the Department of Electrical and Computer Engineering, Duke University, Durham, NC, United States, 27708.


E-mail: {jz288,yueqian.lin,hai.li}@duke.edu
Y. Ming and S. Joty are with Salesforce AI Research, Palo Alto, CA, United States, 94301. S. Joty is also with NTU, Singapore, 639798 (on leave).


E-mail: {yifei.ming, sjoty}@salesforce.com
G. Irie is with the Department of Information and Computer Technology, Tokyo University of Science, Japan, 1258585.


E-mail: goirie@ieee.org

Y. Li is with the Department of Computer Sciences, University of Wisconsin-Madison, Madison, WI, United States, 53706.


E-mail: sharonli@cs.wisc.edu

Resources: <https://github.com/AtsuMiyai/Awesome-OOD-VLM>.


    Received July 31, 2024; accepted XX XX, XXXX
=========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ INTRODUCTION

A reliable visual recognition system should not only accurately predict known contexts, but also identify and reject unknown examples <cit.>. In critical applications like autonomous driving, the system must alert and cede control to the driver upon encountering unfamiliar scenes or objects not seen during training. However, most existing machine learning models are trained based on the closed-world assumption <cit.>, where the test data is assumed to be drawn i.i.d. from the same distribution as the training data, known as in-distribution (ID). Therefore, the development of classifiers capable of detecting out-of-distribution (OOD) samples is a crucial challenge for real-world applications. This challenge is precisely the focus of research in the field of OOD detection.

While OOD detection primarily focuses on semantic distribution shift, several other tasks share similar goals and motivations, including outlier detection (OD) <cit.>, anomaly detection (AD)  <cit.>, novelty detection (ND)  <cit.>, and open set recognition (OSR) <cit.>. Subtle differences in the specific definitions among these sub-topics have caused confusion in the field, leading to similar approaches being proposed across them. To address this issue, the generalized OOD detection framework was introduced <cit.>. The taxonomy of the generalized OOD detection framework is shown in Fig. <ref>. The generalized OOD detection framework classifies these tasks as special cases or sub-tasks under a unified taxonomy. This framework provides clear definitions and fosters a deeper understanding of each field.



In recent years, the emergence of Vision Language Models (VLMs), represented by CLIP <cit.>, has rapidly accelerated research in the field of Computer Vision. This has changed the paradigm of the recognition field, allowing for zero-shot <cit.> or few-shot learning <cit.> in various domains. VLMs have significantly influenced the aforementioned five problems (OD, AD, ND, OSR, and OOD detection), and the application of VLMs has become a highly notable research field <cit.>. However, alongside this remarkable progress, the paradigm shift with the advent of the VLMs has blurred the boundaries between the five problems. Due to the difficulty of a clear understanding of the distinctions and interrelations between these tasks, each community within the fields is facing significant challenges in identifying the optimal direction to pursue in this VLM era.

In this survey, we introduce a novel unified framework termed generalized OOD detection v2, which extends the previous generalized OOD detection framework and summarizes the evolution of these five problems in the VLM era. To create it, we systematically review the use of VLMs across these five problem areas, tracing their development from the start to the present, and summarize the evolutionary trajectory of each problem. Importantly, our framework reveals that a paradigm shift has caused some fields to become inactive or integrate with others, and the demanding challenges in the VLM era become AD and OOD detection, which is a remarkable finding for each community. In addition to the inter-field evolution, we elaborate on the important shifts in the definition of OOD detection as well as the problem settings and benchmarks, with the contrast of those for related tasks. Then, we conduct a thorough review of the methodology for OOD detection and related tasks in the VLM era, intending to clarify their similarities and differences and inspire future research in OOD detection.

Finally, we introduce the evolution of these problems in the emerging Large Vision Language Model (LVLM) era, such as GPT-4V <cit.> or LLaVA <cit.> (also referred to as Large Multi-modal Models or Multi-modal Large Language Models <cit.>). 
We summarize the definition of each evolving problem, the findings so far, and future challenges.





To sum up, in this survey paper, we make three contributions to the research community:


  * Unified Framework in the VLM era:
We systematically review the evolution of five closely related topics of OD, AD, ND, OSR, and OOD detection in the Vision Language Model era and provide an updated unified framework termed generalized OOD detection v2.
Our framework reveals that the paradigm shift has led to some field inactivity or integration, and the demanding challenges are AD and OOD detection.
We hope that these observations highlight the demanding challenges in the VLM era and foster collaborative efforts among each community.


  * A Comprehensive Survey for OOD Detection in the VLM era:
While comprehensive surveys on OD, AD, ND, OSR, and OOD detection methodologies have been published in recent years <cit.>, this survey is the first to comprehensively overview OOD detection methods specifically in the VLM era.
By connecting with other related tasks, we aim to provide readers with a holistic understanding of the developments and interconnections within each problem, particularly in the context of OOD detection.


  * An Introduction to the Evolution in the LVLM Era:
We further introduce the evolution of each problem in the Large Vision Language Model era. Despite the infant stage of these fields, this survey offers an in-depth introduction to each problem, aiming to facilitate future advancements in this area.


  * Future Research Directions:
We draw readers' attention to the future work necessary for advancing the field in the VLM and LVLM era. We conclude this survey with discussions on open challenges and opportunities for future research.






The paper content is organized as follows. 
In Sec. <ref>, we introduce the new version of generalized OOD detection by summarizing the evolution of the five related fields in the VLM era.
We then overview the two key problems (OOD detection and AD) that have evolved and remain active in Sec. <ref>, with a detailed breakdown of existing methodologies being presented in Sec. <ref> (CLIP-based OOD detection) and Sec. <ref> (CLIP-based AD).
In Sec. <ref>, we introduce early advancements of OOD detection and AD in the LVLM era.
Sec. <ref> and Sec. <ref> feature discussions on potential challenges and future directions.
Finally, we conclude with Sec. <ref>.


§ GENERALIZED OOD DETECTION V2


In this section, we introduce a novel unified framework termed generalized OOD detection v2, which summarizes the evolution of the five related fields in the VLM era. We first revisit the previous generalized OOD detection framework in Sec. <ref>. Next, we introduce the evolution of each problem.



 §.§ Background: Generalized OOD Detection V1


We first briefly revisit a previous generalized OOD detection, which encapsulates five related sub-topics: anomaly detection (AD), novelty detection (ND), open set recognition (OSR), out-of-distribution (OOD) detection, and outlier detection (OD).
These sub-topics can be similar in the sense that they all define a certain in-distribution, with the common goal of detecting out-of-distribution samples under the open-world assumption.
Previously, subtle differences existed among the sub-topics in terms of the specific definition and properties of in-distribution (ID) and OOD data. 

To provide a clear definition, a generalized OOD detection framework was proposed <cit.>. The taxonomy for generalized OOD detection is shown in Fig. <ref>. It is based on the following four bases: (1) Distribution shift to detect: The task focuses on detecting either covariate shift (OOD samples from a different domain) or semantic shift (OOD samples from a different semantic). (2) ID data type: The in-distribution (ID) data contains either a single class or multiple classes. (3) Whether the task requires ID classification: Some tasks require classification of the ID data, while others do not. (4) Transductive vs. inductive learning: Transductive tasks require all observations (both ID and OOD), while inductive tasks follow the common train-test scheme.
According to the above taxonomy, these five problems can be clearly categorized as shown in Fig. <ref>: Anomaly detection is categorized into sensory anomaly detection, which deals with covariate shift, and semantic anomaly detection, which deals with semantic shift. Novelty detection falls under the same category as semantic anomaly detection.
When addressing a multi-class scenario that necessitates ID classification, both open-set recognition and out-of-distribution detection are encompassed within this category. The main difference between OSR and OOD detection was the benchmark setup <cit.> (Sec. <ref> (c)).
Outlier detection belongs to a different category from the other tasks, as this problem is transductive (it has access to all observations).

For the detailed definition of each task, we refer the readers to the previous generalized OOD detection survey paper <cit.>.



 §.§ Evolution of Each Problem in VLM Era


We review how each problem has evolved in the VLM era. To make a fair judgment, we comprehensively investigated papers that use VLMs from top venues and summarized them in Table <ref>. 
Our survey revealed that CLIP <cit.> is predominantly used as the VLM for OOD detection and other sub-tasks, and other VLMs <cit.> are rarely utilized. Therefore, we focus on CLIP as the target VLM in this survey and refer to OOD detection using CLIP as CLIP-based OOD detection. Similarly, we will prefix other tasks with “CLIP-based" (CLIP-based AD). As OOD detection research is primarily focused on the image domain, we conduct a survey of other tasks within the image domain that are common and have strong connections to OOD detection research. For instance, our survey does not cover video domain tasks <cit.> due to their limited connection to OOD detection.



(a) Sensory AD → CLIP-based AD
Sensory AD has continued to develop as a common problem setting for CLIP-based AD,
inheriting the challenges of traditional sensory AD <cit.>. As shown in Table <ref>, the first appearance in a top venue was at CVPR 2023, and since then, a total of six papers have been published in top venues. In addition, there are numerous other papers <cit.>. Moreover, in terms of benchmarks, in addition to the commonly used MVTec-AD <cit.>, the largest industrial anomaly detection dataset VisA <cit.> has also become a standard benchmark in the field. Therefore, it is evident that sensory AD has become a highly active and noteworthy field in the VLM era.



(b) Semantic AD/ND  → Inactive
Research on semantic AD/ND appears to become inactive in the VLM era. 
As shown in Table <ref>, there are only two papers, TMLR 2022 <cit.> and CVPR 2024 <cit.>. However, the CVPR 2024 work <cit.> aims to build a generalist anomaly detector that solves many AD tasks, including sensory AD and semantic AD, and is not primarily focused on semantic AD. 
The reasons for the inactivity include saturation of performance for one-class semantic AD/ND, and incompatibility of methods with CLIP for multi-class semantic AD/ND. As for one-class semantic AD/ND, TMLR <cit.> exists, but the performances with common CIFAR and ImageNet-30 datasets have already achieved around 99%. As for multi-class semantic AD/ND, a common approach is to treat ID classes as a single class, but treating ID classes as a single class is less compatible with CLIP's class-wise discriminative capability.



(c) OSR  → CLIP-based OOD Detection 
We consider that OSR has been integrated into CLIP-based hard OOD detection. According to Table <ref>, there are no top venue publications on OSR research in the VLM era. Originally, the main difference between OSR and OOD detection was the benchmark setup <cit.>. OSR typically divides the classes in the one dataset into some known (ID) classes and unknown (OOD) classes, as seen in MNIST-4/6 <cit.> CIFAR-4/6 <cit.>, CIFAR-50/50 <cit.>, and TinyImageNet-20/180 <cit.>. However, in recent years, some works on CLIP-based OOD detection incorporate the benchmark setup of OSR and create new benchmarks such as ImageNet-10/ImageNet-20 <cit.> and ImageNet-protocol <cit.> for hard OOD detection. Therefore, the boundary between OOD detection and OSR has effectively disappeared, and all research in the VLM era has been integrated into OOD detection. 

Nevertheless, while pure OSR research is declining, some studies have used the term “open-set" in the context of domain generalization <cit.>. 
These studies deviate from the original scope of OSR research and are rather closely aligned with the field of domain generalization <cit.>. Therefore, within our generalized OOD detection v2, we do not classify these studies as falling under OSR research. We will discuss them in the context of full-spectrum OOD detection, a research field combining generalization and detection, in Sec. <ref>. 




(d) OOD Detection  → CLIP-based OOD Detection
OOD detection is a highly active research area in the VLM era.
As shown in Table <ref>, there are many papers in top venues, indicating a high interest from the community. Additionally, as mentioned above, OSR has been integrated with OOD detection as a field of hard OOD detection <cit.>. Therefore, it is expected that OOD detection will continue to grow and develop further.



(e) OD  → Inactive
OD has become less active in the VLM era. Previously, OD was used for open-set semi-supervised learning <cit.>, learning with open-set noisy labels <cit.>, and novelty discovery <cit.>.
The reason for the inactivity is that the use of CLIP led to a reduction in training costs and only a small amount of data needs to be collected, eliminating the need for large amounts of unlabeled data and reducing the need to consider noisy data.
However, recently, Liang  <cit.> proposed Unsupervised Universal Fine-Tuning, a new problem setting for CLIP-based OD in ICML2024. Unsupervised Universal Fine-Tuning assumes a more realistic problem setting for unsupervised tuning of the downstream task with CLIP where some OOD samples are included in the unlabeled samples.
With this new problem setting, there is still a possibility that OD will become active in the future.
However, as OD is not currently an active area, we exclude OD from the main discussion of this survey. Unsupervised Universal Fine-Tuning is deeply related to OOD detection and will be discussed in detail in Sec. <ref>.





 §.§ Discussion

Through Sec. <ref>, we found that previously mixed fields have been correctly organized in the VLM era, and that the focus should be on OOD detection and sensory AD. These fields are still developing, with an increasing number of methodologies and benchmarks, and are expected to become more active in the future. Note here that this does not mean that other fields have come to an end. For example, one reason why one-class semantic AD/ND has not been studied is the saturation of performance <cit.>. If more fine-grained and challenging datasets could be constructed, the field could be reactive. We put this in out-of-scope for this survey paper, but this is an important future challenge.



§ OVERVIEW OF EACH PROBLEM IN VLM ERA


In addition to the above inter-field evolution, we emphasize that the advent of VLMs has significantly changed the field of OOD detection itself. In this section, we present an overview of CLIP-based OOD detection, highlighting the key changes in the problem definition, the problem setting, and benchmarks. In addition, we also present an overview of CLIP-based AD in the hope that the understanding of each field will lead to a deeper understanding of CLIP-based OOD detection. For items that remain unchanged from traditional problems, such as background, applications, and evaluation, we refer the readers to the original generalized OOD detection paper <cit.>.



 §.§ CLIP-based Out-of-Distribution Detection


Definition
The definition of CLIP-based OOD detection differs significantly from that of conventional OOD detection.
Conventional OOD detection aims to detect test samples drawn from a distribution that is different from the training distribution. As another definition, OOD detection is defined as a task to detect test samples that the model cannot or does not want to generalize <cit.>. 
However, for CLIP-based OOD detection, CLIP has a vast amount of knowledge, so the OOD sample is completely unrelated to the distribution of the CLIP's pretraining data or the CLIP's own generalization ability. Therefore, traditional definitions cannot adequately describe the definition of CLIP-based OOD detection.

Unlike the previous definition, CLIP-based OOD detection is defined as follows <cit.>: CLIP-based OOD detection aims to detect samples that do not belong to any ID class text provided by the user. Given a pre-trained model, a classification task of interest is defined by a set of class labels , 
which we refer to as the ID classes. The semantic distribution is represented by the distribution P(). CLIP-based OOD detection aims to detect test samples that come from the distribution with the semantic shift from the ID classes, P()≠ P(). 
Following the definition of the generalized OOD detection framework <cit.>, ideal OOD detectors should keep the classification performance on test samples from ID class space , and reject OOD test samples with semantics outside the support of .



Problem Setting
CLIP-based OOD detection focuses on solving the image classification task in a computationally efficient way. Unlike traditional OOD detection settings, which primarily involve training an ID classifier with whole ID data, CLIP-based OOD detection primarily focuses on a zero-shot <cit.> (without utilizing ID images) or few-shot <cit.> (utilizing only a few ID images) setting. Each detailed definition of both settings are described later in Sec. <ref>. The field is advancing towards greater computational efficiency, requiring minimal or no training data.



Benchmark
Most recent works in CLIP-based OOD detection use high-resolution and large-scale datasets such as ImageNet <cit.>. 
The common ImageNet OOD benchmark uses ImageNet as ID and other datasets <cit.> as OOD. However, in this common benchmark, the semantics between ID and OOD are far, which may allow easy distinction between the ID and OOD. Therefore, recent works use more challenging OOD benchmarks where they split ImageNet classes into ID and OOD categories for hard OOD detection <cit.>. The representative datasets are ImageNet-20 <cit.>, ImageNet-10 <cit.>, and the recently proposed ImageNet-protocol <cit.> created by dividing into multiple variations of ID/OOD pairs from ImageNet-1K. This creation strategy initially focused on OSR but has recently been repurposed for OOD detection. These changes in the datasets shift OOD detection closer to the real world and make it a more challenging and practical task.





 §.§ CLIP-based Anomaly Detection


Definition
Unlike OOD detection, the definition of anomaly detection (AD) has not changed between conventional AD and CLIP-based AD. AD is intended for use in specific circumstances (industrial inspection), where samples that deviate from predefined normality are considered an anomaly <cit.>. Whether a model can generalize is irrelevant to the definition of “Anomaly". Therefore, even with the emergence of CLIP, the definition has not changed. 



Problem Setting
CLIP-based AD focuses on solving anomaly classification and segmentation in a computationally efficient way. 
Anomaly classification, like conventional AD, is a binary classification task that distinguishes between normality and abnormality. Anomaly segmentation, also following conventional AD, involves segmenting the location of anomalies. Like CLIP-based OOD detection, CLIP-based AD also primarily focuses on a zero-shot <cit.> (without utilizing images in the target dataset) or few-shot <cit.> (using only a few normal images in the target dataset) setting. Each detailed definition of zero-shot and few-shot settings is described later in Sec. <ref>. As another shift, conventional AD created separate models for each category  <cit.>, while CLIP-based AD creates a single unified model across multiple categories <cit.>, which leads to a more computationally efficient approach.

One key difference from CLIP-based OOD detection is that CLIP-based OOD detection does not involve localization tasks, while these are mainstream in CLIP-based AD. This will be discussed in detail in Sec. <ref>. 



Benchmark
Most works on CLIP-based AD tackle industrial inspection <cit.>. 
As for the benchmarks, besides the common MVTec-AD dataset <cit.>, the more challenging dataset VisA <cit.> has been newly employed <cit.>. The VisA benchmark includes objects with complex structures such as printed circuit boards and multiple instances with different locations within a single view, making it one of the most challenging datasets currently available in the open datasets. Since the pioneering work in CLIP-based AD (WinCLIP <cit.>) used MVTec-AD and VisA, many subsequent works have also used these datasets <cit.>.








§ CLIP-BASED OOD DETECTION: METHODOLOGY


In this section, we introduce the methodologies for CLIP-based out-of-distribution (OOD) detection. Fig. <ref> presents the timeline for representative methodologies for CLIP-based OOD detection. Table <ref> presents representative methods. We introduce methods for zero-shot OOD detection in Sec. <ref>, few-shot OOD detection in Sec. <ref>, and other research directions in Sec. <ref>. For each methodology, we categorize them by the type of training and whether additional OOD prompts were employed. 



 §.§ Zero-shot Out-of-Distribution Detection


Zero-shot OOD detection was proposed in 2021 by Fort  <cit.>. Since then, a growing number of methods have been proposed year by year.



Definition of Zero-shot OOD Detection
In zero-shot OOD detection, the term “Zero-shot" refers to the non-use of ID images during both training and inference phases. For instance, the method with additional training with auxiliary datasets (non-use of ID images) can be regarded as a zero-shot method <cit.>. The method with the pre-processing of the ID class texts can also be regarded as a zero-shot method <cit.>.





  §.§.§ Training-free Methods


a. With OOD Prompts
CLIP-based OOD detection started in this setting. 
The earliest work is ZeroOE <cit.>. ZeroOE feeds the potential OOD labels to the textual encoder of CLIP. However, the method of using known OOD labels is infeasible for real-world applications.
To solve this problem, ZOC <cit.> proposed to train an OOD label generator based on the visual encoder of CLIP and use the generated pseudo-OOD labels for OOD detection.
However, when dealing with large-scale datasets encompassing a multitude
of ID classes, the label generator may not generate effective candidate OOD labels, resulting in poor performance.
Building on these early works <cit.>, recent works focus on how to obtain high-quality OOD labels through either (i) OOD label retrieval <cit.> or (2) OOD label generation <cit.>. (i) One of the representative retrieval-based methods is NegLabel <cit.>. NegLabel selects high-quality OOD labels from extensive corpus databases by calculating the distance between an extracted OOD label and ID label. (ii) One of the representative generation-based methods is EOE <cit.>.
EOE utilizes Large Language Models (LLMs) to produce high-quality OOD labels. 
By modifying the prompts given to the LLM, EOE can be generalized to a variety of tasks, including far and near OOD detection.



b. Without OOD Prompts
In zero-shot OOD detection, many methods utilize OOD labels, but the difficulty and cost of creating these labels pose challenges. To address these issues, Ming  <cit.> proposed MCM, which uses only ID labels to detect OOD. MCM is a simple approach that devises softmax scaling to align visual features with textual concepts for OOD detection. Despite its simplicity, MCM has high effectiveness and scalability, and it serves as a crucial baseline in CLIP-based OOD detection.
Building on the concept of MCM, Miyai  <cit.> proposed GL-MCM, which extends MCM by just adding a local MCM score to enhance the fine-grained detection capability in local regions. SeTAR <cit.> enhances MCM and GL-MCM by changing the model's weight matrices using a simple greedy search algorithm.
We consider these methods to be post-hoc methods for CLIP-based OOD detection in that they directly employ an ID classifier for OOD detection. Due to their simplicity and high scalability, these post-hoc methods can bring fundamental performance improvements for many subsequent methods <cit.>. Therefore, we expect that this field should be developed further in the future, reflecting the trajectory of the field before CLIP emerged <cit.>.





  §.§.§ Auxiliary Training-based Methods


CLIPN <cit.> is the only auxiliary training-based method for zero-shot OOD detection. CLIPN aims to empower the logic of saying “no" within CLIP, and it designs a novel learnable “no" prompt and an additional “no" text encoder to capture negation semantics within images. To create an additional text encoder, CLIPN needs to be pre-trained on the CC-3M dataset <cit.>.
While the extensive pre-training of CLIPN may indeed lead to intensive computations and lower scalability, once pre-trained, it performs zero-shot OOD detection across a wide range of domains with comparable performance to few-shot OOD detection methods <cit.>.
In the future, within this field, there is potential for zero-shot open-vocabulary OOD detection to further advance the realm of zero-shot OOD detection, which will be discussed later in Sec. <ref>.



 §.§ Few-shot Out-of-Distribution Detection


Few-shot OOD detection was concurrently proposed by Miyai <cit.> and Ming <cit.> in June 2023. Since then, it has become the most active research area in CLIP-based OOD detection.



Definition of Few-shot OOD Detection
CLIP-based few-shot OOD detection aims to detect OOD images using only a few labeled ID images.
In few-shot OOD detection, the term “Few-shot" refers to the use of a few ID images during training or inference phases. For instance, the method with additional training with a few ID images can be regarded as a few-shot method <cit.>. Even without training, if a method uses a few ID images as a reference, we regard it as a few-shot method <cit.>. 
Regarding the number of shots, it is common to experiment with 1-shot to 16-shot <cit.>, following the closed-set setting <cit.>.



  §.§.§ ID Training-based Methods


a. Without OOD Prompts
Few-shot OOD detection began in this setting. Ming  <cit.> proposed PEFT-MCM for CLIP-based OOD detection, which demonstrates the effectiveness of combining parameter-efficient tuning methods (prompt learning <cit.> or adapter <cit.>) and MCM <cit.>. Concurrently, Miyai <cit.> proposed LoCoOp, a pioneer prompt learning approach for few-shot OOD detection. 
LoCoOp enhances CoOp's <cit.> OOD detection capabilities by performing OOD regularization with local OOD features. LoCoOp is the simplest prompt learning method and serves as a crucial baseline in few-shot OOD detection. 
Unlike LoCoOp, which utilizes non-ID local regions for OOD regularization, GalLoP <cit.> proposes an approach that utilizes local ID regions to enable a more fine-grained distinction between ID and OOD samples. GalLoP learns a diverse set of prompts by utilizing both global and local visual representations, thereby enhancing the detection capabilities.



b. With OOD Prompts
Similar to zero-shot OOD detection, recent works in few-shot OOD detection utilize additional OOD prompts <cit.>.
As representative methods, LSN <cit.> and NegPrompt <cit.> were proposed concurrently. They state that the simple negative prompts added “not” (“not a photo of a [cls]”) fail to capture the dissimilarity for identifying OOD samples. 
Therefore, by preparing negative prompts and training with them, LSN and NegPrompt can learn suitable negative prompts, enabling more accurate detection of OOD samples.
The difference between LSN and NegPrompt lies in their approach to the use of negative prompts. LSN prepares unique negative prompts for each class and learns suitable negative prompts for each class. In contrast, NegPrompt prepares multiple negative prompts common to all ID classes and trains them to learn generic templates representing the negative semantics of any given class labels. Also, NegPrompt tested the performances in the hard OOD detection setting with ImageNet-protocol <cit.>, outperforming LoCoOp and CoOp. 
Alternatively, IDPrompt <cit.> takes a different approach by introducing ID-like prompts, which are designed for OOD features that are close to the ID features. It extracts ID-like OOD regions in ID training images and trains ID-like prompts with these extracted OOD data. In a unique direction, LAPT <cit.> proposes an automatic sample collection strategy that retrieves or generates training ID images only with ID class names, which achieves high performance without image collection and annotation costs. LAPT then performs distribution-aware prompt learning, which distinguishes between ID class and OOD class tokens. LAPT is positioned within the context of more efficient few-shot OOD detection in this survey paper since it requires generating or retrieving “ID images" for the data collection.

In the context of few-shot OOD detection, recently, Li  <cit.> proposed a new problem setting called open-vocabulary OOD (OV-OOD) detection. 
While common few-shot OOD detection involves training on images from all ID classes during training, OV-OOD detection involves training on images from just a small subset of ID classes and performing OOD detection using all ID classes at inference time. Formally, we define a subset of semantic labels ⊂, where  represents all ID labels. Based on this subset of labels, we define a corresponding subset dataset ⊂. During training, only  is used. Then, at inference time, all ID classes  are used, and the goal is to detect OOD from a combination of all ID test data  with  and OOD test data  with . For this setting, existing few-shot OOD detection methods <cit.> can be easily applied by simply combining the rest of the ID classes. In particular, NegPrompt <cit.> learns general negative prompts that are not specific to the training ID classes, so it achieves high performance in OV-OOD detection.




  §.§.§ Training-free Methods


Training-free few-shot OOD detection is a novel research field, and only Dual-Adapter <cit.> falls under this category. Dual-Adapter adopts a prior-based method Tip-Adapter <cit.>, which leverages both textual and
visual features with a cache model and enhances performance without training. To adapt this to few-shot OOD detection, Dual-Adapter employs the concept of dual cache modeling and constructs Positive-Adapter and Negative-Adapter, and identifies OOD samples with the prediction difference with both adapters. 






 §.§ Other Important Research Directions




  §.§.§ CLIP-based Full-spectrum OOD Detection

CLIP-based full-spectrum OOD (FS-OOD) detection is a crucial challenge <cit.>. FS-OOD detection was proposed by Yang  <cit.> in 2022 as an important setting that considers both OOD generalization <cit.> and OOD detection simultaneously. 
Unlike standard OOD detection, which only focuses on semantic shifts between training and test distributions, FS-OOD detection further considers non-semantic covariate shift by including covariate-shifted ID images. As for the benchmarks, OpenOOD v1.5 <cit.> provides two large-scale benchmarks based on ImageNet-200 and ImageNet-1K, incorporating ImageNet-C <cit.> with image corruptions, ImageNet-R <cit.> with style changes, and ImageNet-V2 <cit.> with resampling bias as ID. As for CLIP-based methods, LSA <cit.> uses a bidirectional prompt customization mechanism, which adjusts discriminative ID and OOD boundary. 




  §.§.§ Other Tasks with CLIP-based OOD Detection

Unsupervised Universal Fine-Tuning
CLIP-based OOD detection is useful for a new task called Unsupervised Universal Fine-Tuning (UUFT) <cit.>.
UUFT is a problem of unsupervised learning for outlier detection (OD). Existing studies for unsupervised learning assumed that all unlabeled images belong to one of the ID classes <cit.>, but they require prior knowledge of exact class names linked to ground truth labels, which restricts their usefulness in various real-world situations. For a more realistic setting, UUFT assumes that OOD images are included in the unlabeled images. To detect OOD images during training, they developed MCM <cit.> and proposed UEO, which leverages sample-level confidence to approximately minimize the conditional entropy of confident instances and maximize the marginal entropy of less confident instances.



Open-world Prompt Tuning
CLIP-based OOD detection is useful for a new task called Open-world Prompt Tuning <cit.>. 
Open-world Prompt Tuning is a task that evaluates the classification accuracy on a mix of known and novel ID classes while training the model with known classes. To solve this problem, Zhou  <cit.> proposed DeCoOp which incorporates OOD detection into the inference pipelines and improves the base-to-new separability, preventing performance degradation on new classes. 


§ CLIP-BASED AD: METHODOLOGY


In this section, we introduce methodologies for
CLIP-based anomaly detection (AD) in the hope that the contrast with OOD detection clarifies the similarities and differences between each task and facilitates a deeper understanding of CLIP-based OOD detection. 



 §.§ Zero-shot Anomaly Detection


CLIP-based zero-shot AD was proposed in 2023 by Jeong  <cit.>. Although it started about two years later than OOD detection, many methods have been proposed up to the present.



Definition of Zero-shot AD
The meaning of the term “Zero-shot" for zero-shot AD is similar to that for zero-shot OOD detection.
In zero-shot AD, the term “Zero-shot" refers to the non-use of the images in the target domain during both training and inference phases. For instance, the method with additional training with auxiliary datasets can be regarded as a zero-shot method <cit.>. The method with the pre-processing of the target class texts can also be regarded as a zero-shot method <cit.>.





  §.§.§ Training-free Methods


With Anomaly Prompts
In zero-shot AD, a common approach is to utilize anomaly prompts to detect anomalies. This hypothesis is supported by several observations from existing work <cit.>. Firstly, the concepts of normality and anomalies are context-dependent states <cit.> of an object, with language playing a crucial role in defining these states. Secondly, language provides additional insights that help differentiate defects from acceptable variations in normality. 

The simplest zero-shot AD methods are (i) to perform anomaly classification with CLIP using text prompts for normality and anomalies as classes (“” vs. “”) and (ii) to calculate the similarity to the normal prompt (“”) as the score. These methods are called CLIP-AC <cit.>. Jeong  <cit.> reported that CLIP-AC with both normal and anomaly prompts outperforms that with only normal text prompts, which indicates the importance of the use of anomaly prompts.
However, the performances for this naive method are not yet satisfactory due to the wide range of variations of anomalies.
To solve this issue, Jeong  <cit.> proposed WinCLIP.
WinCLIP performs a compositional ensemble on a large number of pre-defined normal and anomaly templates and efficient extraction and aggregation of window/patch/image-level features aligned with the text. WinCLIP outperforms CLIP-AC by a large margin. Because of its simplicity and pioneering work, WinCLIP has become an important baseline for CLIP-based AD.
AnoCLIP <cit.> follows WinCLIP's approach of using a large number of pre-defined normal and anomaly templates but modifies the templates to be domain-aware (industrial photo) and contrastive state for normal and anomaly (perfect and imperfect).
However, it is noteworthy that the performance of the ensemble strategies of previous methods heavily depends on the text descriptions <cit.>. 
Also, it is observed that more descriptions are not always better <cit.>, which makes the previous approaches <cit.> using a naive ensemble of large templates somewhat uncontrollable and random in their applications. Therefore, SDP <cit.> proposes RVS, a representative vector selection paradigm, which makes the mechanism of extracting representative vectors from large templates controllable, allowing for a more diverse selection of representative vectors.




  §.§.§ Auxiliary Training-based Methods


Unlike CLIP-based OOD detection, all auxiliary training-based methods in zero-shot AD are open-vocabulary AD methods that are trained with auxiliary AD datasets and tested on unseen target datasets by simply changing the category prompt. Existing methods perform supervised training on the test set of one dataset and perform zero-shot testing on the other dataset <cit.> (training with MVTec-AD and evaluation with VisA.)

In recent years, the development of auxiliary training-based methods has received more attention than training-free zero-shot methods. There are two main reasons why training is necessary for AD: (i) The first is the domain gap between semantics and anomalies. CLIP is pre-trained to understand the semantics of images, so when applied in a zero-shot manner, it captures the semantics of the image. However, actual anomalies are not semantics, but rather represent the state of an object and appear only in local areas of the image. Therefore, without training, this domain gap between semantics and anomalies cannot be bridged. (ii) The second reason is that there are limitations to relying on a large set of manually crafted anomaly prompts. This incurs prompt creation costs and also makes it difficult to respond to unknown anomalies. Therefore, by replacing the anomaly prompts with learnable parameters, they aim to solve the high costs and limited adaptability to new anomalies.

To address the above issue (i), APRIL-GAN (also known as VAND) <cit.> was proposed. APRIL-GAN tackles the domain gap between semantics and anomaly by adding additional linear layers in vision encoders. These linear layers project image features at each scale into the text space, creating and aggregating anomaly maps at each stage. 
Similarly, SDP+ <cit.> also incorporates additional linear layers into SDP <cit.> to effectively project image features into the text feature space, addressing the misalignment between image and text.
To solve both the issue (i) and (ii), AnomalyCLIP <cit.> was proposed. AnomalyCLIP is a prompt learning-based method similar to CoOp. By replacing anomaly prompts with learnable parameters, it eliminates the need to prepare a large number of manually pre-defined prompts such as those in WinCLIP <cit.>.
Furthermore, unlike CoOp which learns object semantics, AnomalyCLIP learns object-agnostic text prompts that capture generic normality and abnormality in an image regardless of its semantics. To achieve this, AnomalyCLIP introduces object-agnostic text prompt templates for both normal and anomaly and performs global and local context optimization. A more recent approach, FiLo <cit.> leverages Large Language Models (LLMs) to generate fine-grained anomaly descriptions for each object category. This method replaces generic abnormal descriptions with LLM-generated specific anomaly content for each sample. By adding learnable prompts before the generated anomaly prompts, FiLo performs global and local context optimization, enhancing the ability to detect anomalies. 
As a unique direction from these methods, RWDA <cit.> proposes a data augmentation approach by utilizing CLIP's text embeddings as training data. RWDA adds randomly generated words into normal and anomaly prompts to generate a diverse set of normal and anomaly training samples and trains a regular feed-forward neural network with diverse text embeddings. 





 §.§ Few-shot Anomaly Detection


CLIP-based few-shot AD was proposed in 2023 by Jeong  <cit.>, concurrently with the development of zero-shot AD <cit.>.
Traditional few-shot AD research focuses on modeling the normal distribution from a limited number of normal samples to detect anomalies <cit.>. However, these methods often struggle to generalize to new domains, as they typically require retraining with the target datasets. With the advent of CLIP, the field of few-shot AD is shifting towards using a few target images only for inference at test time without training.



Definition of Few-shot AD
CLIP-based few-shot AD aims to detect anomaly images using only a few images in the target domain. The meaning of the term “Few-shot" is similar to that of few-shot OOD detection.
In few-shot AD, the term “Few-shot" refers to the use of a few normal images in the target domain during training or inference phases. For instance, the method with additional training with a few normal images in the target domain can be regarded as a few-shot method <cit.>. Even without training, if a method uses a few normal images in the target domain as a reference, we regard it as a few-shot method <cit.>. 



  §.§.§ Training-free Methods


The earliest approach in CLIP-based few-shot AD is WinCLIP+ <cit.>, an improved method of WinCLIP. WinCLIP, a base zero-shot AD method, cannot identify certain defects that can only be defined visually rather than textually. For example, the “Metal-nut" category in MVTecAD has an anomaly type labeled “flipped upside down," which can only be identified relative to a normal image. To address this, WinCLIP+ incorporates a few normal reference images into a memory bank <cit.> and calculates the anomaly score with the cosine similarity between the query image and its most similar image in the memory bank.



  §.§.§ ID Training-based Methods


There are few training-based methods for CLIP-based few-shot AD research, although CLIP-based few-shot OOD detection actively explores the training-based methods. 
This is likely because detecting anomalies in unknown classes is highly valuable in practical applications, and training on target data may oversimplify the task of few-shot anomaly detection. This oversimplification concern arises because the anomaly space for known categories is much more limited compared to that for OOD detection. Training with the target data could therefore make the task oversimplified, reducing its difficulty. The only existing work in this area is PromptAD <cit.>, a prompt learning method for one-class AD (where the normal class consists of one class). In one-class AD, traditional prompt learning methods for multi-class classification (CoOp <cit.>) do not work well. To address this, PromptAD creates a large number of anomaly prompts by adding a learnable anomaly suffix to the normal prompt. It then learns to bring the visual features closer to the normal prompt and further away from the anomaly prompts, enabling prompt learning for one-class AD.




  §.§.§ Auxiliary Training- and Reference-based Methods


We explore the methods trained on auxiliary datasets and utilize the normal images in the target domain as references during inference.
An early work in this category is APRIL-GAN (few-shot) <cit.>, which uses a linear layer trained with auxiliary datasets. Similar to WinCLIP+ <cit.>, APRIL-GAN (few-shot) utilizes a few ID reference images with a memory bank-based approach <cit.>.
More recently, Zhu  <cit.> proposed an in-context-learning-based method called InCTRL. InCTRL trains a model to discriminate anomalies from normal samples by learning to identify residuals or discrepancies between query images and a set of few-shot normal images (in-context sample prompts) from auxiliary data. During inference, InCTRL identifies anomalies by measuring the discrepancy between the features of the query image and a few in-context normal samples from the target dataset.




 §.§ Other Research Direction



  §.§.§ Anomaly Detection with Localization Models

Some works <cit.> tackle AD using SAM <cit.> or DINO <cit.>, foundation models for localization. The representative work is SAA and SAA+ <cit.>. SAA is a simple baseline approach and utilizes Grounding-DINO <cit.> for the anomaly region generator and SAM <cit.> for the anomaly region refiner. SAA+ is an improved method of SAA and it incorporates domain expert knowledge and target image context into SAA. 
Given that AD necessitates localization, it is expected that the number of works employing localization foundation models such as SAM will continue to increase.



  §.§.§ Medical Anomaly Detection

While most works on CLIP-based AD focus on industrial AD, recent studies have begun to challenge medical anomaly detection (medical AD) <cit.>.
CLIP-based medical AD is a more challenging area than industrial AD due to the larger gap between different data modalities. A representative work on medical AD is MVFA <cit.>. MVFA is a method specifically tailored for medical AD. It incorporates multiple residual adapters into the CLIP's visual encoder to reduce the domain gap, enabling a stepwise enhancement of visual features across different levels. 
The future progression of medical AD and industrial AD offers an intriguing perspective, exploring whether these fields will develop independently or influence each other.
However, when considering practical applications, it should be noted that medical AD faces the challenge that anomalies are not always describable in the language. Therefore, the development of medical AD methods that do not use CLIP is also important.






 §.§ Discussion


We discuss the similarities and differences between CLIP-based OOD detection and CLIP-based AD to deepen our understanding of CLIP-based OOD detection.



  §.§.§ Difference between Each Methodology

Differing Scopes of OOD
OOD detection and AD differ significantly in the scope of OOD (anomaly) they cover, which leads to differences in methodologies, particularly in the use of OOD prompts. As explained in Sec. <ref>, sensory AD is intended for specific use cases like industrial inspection, where samples deviating from predefined normality (defective products) are considered anomalies <cit.>. In other words, in sensory AD, the anomaly space is limited to damaged objects with shared semantics, and anomalies like images of dogs are not expected. This limited anomaly space allows even simple prompts to achieve decent performance. Therefore, as shown in Table <ref>, all AD methods utilize anomaly prompts.
Conversely, in OOD detection, as explained in Sec. <ref>, anything semantically different from the ID class is considered OOD. Thus, utilizing naive manual OOD prompts is forbidden (even if it improves benchmark performance). This vastness of the OOD space is the key factor differentiating the methodologies between the two fields.



Essential Features Learned for Detecting OOD
There is a significant difference in the features that need to be captured between CLIP-based OOD detection and CLIP-based AD. In CLIP-based OOD detection, 
it is ideal to learn a more compact ID decision boundary that produces low uncertainty for the ID data, with high uncertainty for OOD data <cit.>.
On the other hand, CLIP-based AD aims to learn the anomalies, instead of ID decision boundary for detecting anomalies <cit.>. 
The learned features are entirely different, which is helpful in developing each training-based method.



Difficulty of Localization Task
CLIP-based OOD detection and CLIP-based AD differ significantly in the difficulty of OOD (anomaly) localization tasks. In CLIP-based AD, anomaly segmentation is a mainstream task, often performed alongside classification in many papers. However, in CLIP-based OOD detection, there has been no research on object-level OOD detection/segmentation. Object-level OOD detection aims to detect OOD objects <cit.>. The inactivity is related to the size of the OOD space, and the too-vast space of OOD makes it difficult to identify OOD objects with prompts effectively. To pave the way for future development, the foundation models for localization such as SAM <cit.>, which can segment individual objects, have the potential to address object-level OOD detection/segmentation. Object-level OOD detection/segmentation using SAM is a promising future research direction.





  §.§.§ Similarity between Each Methodology

Each Problem Setting
The existing problem settings for CLIP-based AD and CLIP-based OOD detection are similar. Both primarily focus on zero-shot and few-shot settings and can be categorized into training-free, auxiliary training-based, and ID training-based methods. By examining each problem setting more closely, we can observe, for instance, that while open-vocabulary AD is predominant in CLIP-based AD, it has not been deeply explored in CLIP-based OOD detection. This provides valuable insights into future directions that are crucial for OOD detection.



History of Approaches
The history of the progress of methods for CLIP-based AD and CLIP-based OOD detection is similar. For instance, both problems initially started with naive methods with manual OOD prompts (ZeroOE <cit.> for OOD detection, WinCLIP <cit.> for AD). To address the issues with these initial approaches, subsequent methods emerged that replaced OOD prompts with learnable parameters (LSN <cit.> and NegPrompt <cit.> for OOD detection, and AnomalyCLIP <cit.> for AD). Therefore, by carefully examining each other's fields, there is potential for mutual enhancement and interaction in the future.


§ EVOLUTION IN LVLM ERA


In this section, we introduce the early advances in OOD detection and AD in the Large Vision Language Models (LVLM) era. While previous sections focused on VLMs such as CLIP, this section shifts our focus to the more emerging topic of “Large" VLMs.
Recent advancements in computer vision have led to the emergence of LVLMs such as GPT-4V <cit.> and LLaVA <cit.>. 
Although these fields are still in their early stages with limited papers, this survey provides a deep introduction to each problem in the hope that our detailed review can help foster further advancements in this area.







 §.§ Change of Each Problem

i. Sensory AD → Sensory AD
Sensory AD has continued to develop in the LVLM era <cit.>. The use of LVLMs has made AD applicable in many domains and modalities <cit.>. 



ii. OOD Detection → Unsolvable Problem Detection
In the LVLM era, OOD detection has evolved into a new task termed Unsolvable Problem Detection (UPD) <cit.>. UPD evaluates the LVLMs' ability to recognize and abstain from answering unexpected or unsolvable input questions, effectively expanding the scope of OOD detection into the context of Visual Question Answering (VQA) tasks. This shift to the VQA task has significantly broadened the concept of OOD detection to a wider range of AI tasks involving LVLMs.




 §.§ Unsolvable Problem Detection



  §.§.§ Summary of Problem

Background
Following the recent revolutionary development of LLMs <cit.>, LVLMs <cit.> have demonstrated remarkable capabilities in diverse applications <cit.>. However, a significant concern has arisen regarding the reliability of these models, specifically their ability to generate accurate and trustworthy information. These models frequently produce incorrect or misleading information, a phenomenon referred to as “hallucination” <cit.>.
Among the various hallucination issues <cit.>, the challenge of identifying out-of-place questions is crucial for deploying LVLMs in safety-critical applications. This challenge extends the concept of OOD detection to the VQA tasks for LVLMs and represents a specific aspect of LVLMs' trustworthiness.



Definition
Unsolvable Problem Detection (UPD) is a task to measure the trustworthiness of LVLMs, which is designed to evaluate models' capacity to withhold answers when faced with unsolvable problems. The UPD task can be categorized into three distinct problem types: Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Visual Question Detection (IVQD).
The details of each setting are as follows:

      
  * Absent Answer Detection (AAD): AAD evaluates the model's capacity to determine when the correct answer is absent from the provided options.
      
  * Incompatible Answer Set Detection (IASD): IASD assesses the model's ability to discern answer choices that are completely irrelevant to the given question and image.
      
  * Incompatible Visual Question Detection (IVQD): IVQD evaluates the model's capacity to discern whether a question and image are unrelated or mismatched.




Benchmark
Miyai  <cit.> created MM-UPD Bench for the UPD challenge. MM-UPD encompasses MM-AAD, MM-IASD, and MM-IVQD benchmarks for each UPD problem. Each benchmarks are created on the top of MMBench (dev) <cit.>, which is a systematically designed objective benchmark for evaluating various abilities of LVLMs. Following the definition of each ability in MMBench (“Coarse Perception: Image Scene” and “Logic Reasoning: Future Prediction”), MM-UPD evaluates the trustworthiness of LVLMs from various abilities.

Although MM-UPD is the main benchmark, the adaptation cost of creating UPD problems is not high, making it highly applicable to other benchmarks. For instance, the recently proposed MuirBench <cit.>, a benchmark for multi-image understanding, has incorporated the concept of UPD by adding unsolvable problems. 



Application
UPD has a wide range of applications, from everyday use of LVLMs to robot manipulation. Especially when incorporating LVLMs into safety-critical domains such as robot manipulation <cit.> and autonomous driving <cit.>, there is a risk of significant problems if the LVLM fails to identify erroneous user questions and makes incorrect predictions. UPD serves as a task to ensure safety in such safety-critical scenarios.




Evaluation
UPD introduces new evaluation metrics that incorporate the concept of the evaluation protocols for OOD detection, taking into account the prediction distribution for both standard (ID) and UPD (OOD) samples. 
The rationale is that ideal LVLMs should not only give correct answers for the standard problems but need to withhold answering in the UPD scenario where the problem is unsolvable. 
To better reflect the ideal behavior of LVLMs, UPD measures several metrics:
(i) Standard Accuracy: 
The accuracy on standard problems where the image, question, and answer sets are all aligned, and the ground-truth answer is always present within the provided options. (ii) UPD Accuracy: The accuracy of AAD/IASD/IVQD problems. (iii) Dual Accuracy: Accuracy of standard and UPD pairs. We count success only if the model is correct for both standard and UPD problems.




  §.§.§ Findings

In the following, we briefly summarized the findings of the UPD challenge <cit.>.



1. Most LVLMs Hardly Hesitate to Answer.
Most LVLMs, especially open-source LVLMs, have significantly low UPD accuracies, which indicates the difficulty of the UPD challenge. For example, LLaVA-1.5 <cit.> and CogVLM <cit.>, which are state-of-the-art LVLMs, completely fail to withhold answering. GPT-4V achieves higher performances than other LVLMs due to its safety training process <cit.>. However, there is still a performance gap from the upper bound scores.



2. Performance Tendency Differs a lot by Each Ability in the Benchmark.
The performance of LVLMs differs in each ability in the MM-UPD Bench. For instance, GPT-4V has its limitation in attribute comparison and LLaVA-NeXT-34B has its limitation in object localization.



3. Effective Prompt Strategies Vary Across Different LVLMs 
Effective prompt strategies vary across different LVLMs. In the original paper, they experimented with an option-based prompt approach that adds an option of “None of the above" and an instruction-based approach that adds an instruction “If all options are incorrect, answer None of the above". As a result, the effectiveness of each approach differs significantly depending on the type of LVLMs. This highlights the difficulty of finding an effective prompt strategy for all LVLMs. 




 §.§ Anomaly Detection in LVLM Era



  §.§.§ Summary of Problem

Background
Anomaly detection is a crucial task in a variety of domains and data types. However, existing anomaly detection models are often designed for specific domains or modalities <cit.>. Also, current AD methods only provide an anomaly score for the test sample and require a manual threshold to distinguish between normal and anomalous instances for each sample <cit.>. To facilitate real-world applications, developing a system capable of expressing anomalies in natural language across various modalities and domains is crucial for ensuring accessibility to a wider range of users.



Definition
The definition of AD remains consistent with conventional and CLIP-based AD, aiming to identify samples that deviate from predefined normality. The key distinction lies in the output. While previous methods produced an anomaly score requiring a manual threshold, AD with LVLMs aims to recognize and describe anomalies using text, eliminating the need for manual thresholds and enhancing human interpretability.



Benchmark
Since the field of AD with LVLMs is in its infant stage, there are still no unified benchmarks. AnomalyGPT <cit.> focuses on industrial image anomaly detection/localization and uses the standard benchmarks MVTec-AD <cit.> and VisA <cit.>. 
More recently, Cao  <cit.> extend the domain and modality and demonstrate the applications in industrial image anomaly detection/localization (MVTec-AD <cit.>), point cloud anomaly detection (MVTec 3D <cit.>), medical image anomaly detection/localization (Chest X-ray <cit.>, Head CT <cit.>), logical anomaly detection (MVTec LOCO <cit.>), pedestrian anomaly detection (UCF-Crime Dataset <cit.>), traffic anomaly detection (Kaggle Accident Detection <cit.>), and time series anomaly detection (Outlier Detection Dataset <cit.>).



Evaluation
Evaluation in anomaly detection with LVLMs is an open challenge.  AnomalyGPT <cit.> asks LVLMs the question “Is there an anomaly in this image?" and determines anomaly or normal based on the simple rule-based approach of whether the response contains a “yes" or “no". However, this rule-based approach is not robust, as a response is considered correct even if the explanation following “yes" is completely incorrect. On the other hand, Cao  <cit.> conducted only qualitative evaluations and left quantitative evaluations as an open challenge. Therefore, the evaluation of anomaly detection by LVLM is a future challenge.



  §.§.§ Findings

Cao  <cit.> described the observations of GPT-4V in the paper, so we briefly summarized them here.



1. GPT-4V Excels in Zero/One-shot Settings across Various Modalities and Fields.
GPT-4V shows proficiency in identifying anomalies in multi-modality (images, point clouds, X-rays) and multi-field (industrial, medical, pedestrian, traffic, and time series anomaly detection). In addition, GPT-4V demonstrates strong performance in both zero-shot and one-shot settings. 



2. GPT-4V can Understand Both Global and Fine-grained Anomalies.
GPT-4V can recognize both global and local abnormal patterns or behaviors, which indicates the ability to understand global and fine-grained semantics.



3. GPT-4V can be Enhanced with Increasing Prompts.
By giving more context and information, the model significantly improves its ability to detect anomalies accurately. 



§ POTENTIAL CHALLENGES


This section discusses potential challenges for CLIP-based OOD detection that may be highlighted by the widespread adoption of our framework. Since similar or ambiguous problem settings currently exist, we present these challenges to avoid future confusion for readers.





 §.§ Open-vocabulary OOD Detection
 

The existing setting for open-vocabulary OOD (OV-OOD) detection <cit.> involves training with images from a small subset of ID classes and evaluating with all ID class names, as explained in Sec. <ref>. However, this problem definition has several issues: (i) The OOD detection performance on ID classes not included in the training subset is unclear, (ii) This contradicts the settings of the existing open-vocabulary settings, including CLIP-based AD (Sec. <ref>), which assume no explicit class overlap between training and test ID data <cit.>, leading to confusion in the fields.
Therefore, for clear evaluation and common understanding in the field, we redefine the OV-OOD detection as in-class open-vocabulary OOD detection (in-class OV-OOD detection) and zero-shot open-vocabulary OOD detection (zero-shot OV-OOD detection). The illustration for each setting is shown in Fig. <ref>. Detailed explanations for each setting are as follows:



In-class OV-OOD Detection
In-class OV-OOD detection is the same as the original setting proposed by  <cit.>, involving training with images from a small subset of ID classes and evaluating on all ID class names including training classes.
In-class OV-OOD detection should be evaluated in the few-shot OOD detection setting to assess how well these methods retain their performance when trained with limited class data. In this setting, it is preferable to employ the existing benchmark setup that randomly selects subset ID classes from all ID classes <cit.>.



Zero-shot OV-OOD Detection
Zero-shot OV-OOD detection shares the training setting with in-class evaluation, using a small subset of ID classes. However, it differs in the testing phase, where it exclusively utilizes classes not included in the training subset. 
This problem setting is consistent with the common open-vocabulary setting <cit.> and the open-vocabulary setting in CLIP-AD <cit.>.
Zero-shot OV-OOD detection can be positioned within the area of zero-shot OOD detection, which would facilitate the research of zero-shot OOD detection. When selecting the subset of classes for training, we should ensure semantic dissimilarity between the classes in the evaluation and training sets to guarantee the validity of the zero-shot problem setting. For instance, when splitting ImageNet classes into training and evaluation subsets for zero-shot OV-OOD detection, it would be preferable to consider the hierarchical classes of ImageNet to avoid similar semantics between training and evaluation subsets.




 §.§ CLIP-based Full-spectrum OOD Detection






CLIP-based full-spectrum OOD detection is a well-established task with existing research <cit.>. However, with the future prevalence of hard OOD detection, which incorporates the OSR setup, the distinction between this task and open-set domain generalization (OSDG)  <cit.> may become ambiguous. OSDG shares a similar motivation of realizing models capable of both generalization and detection. The illustration of both problem settings is shown in Fig. <ref> (a)(b). OSDG assumes that only data with covariate shifts will be input during testing, and aims to classify input samples into one of ID classes if their semantics align with ID classes and detect OOD that exhibit semantic shifts.
OSDG is a research area with a much smaller number of works compared to the main topics of this survey (OD, AD, ND, OSR, and OOD detection), but, in recent years, some works have tackled OSDG using CLIP <cit.>.

To eliminate potential ambiguity, this paper defines a hard full-spectrum OOD (FS-OOD) detection, following our generalized OOD detection v2. The illustration of the problem setting for hard FS-OOD detection is shown in Fig. <ref> (c).
Hard FS-OOD detection expands the scope of existing FS-OOD detection by utilizing the conventional OSR setup. 
It introduces OOD samples from different categories within the training domain and OOD samples that share the same covariate shift as the target domain but exhibit semantic shifts. This task can be regarded as an extensive version of both existing FS-OOD detection and OSDG tasks. Hard FS-OOD detection represents a developing challenge with promising potential for future research.




§ FUTURE DIRECTIONS


In this section, we discuss the future directions of OOD detection and UPD.
For OOD detection, we explore not only OOD detection for VLMs but also single-modal OOD detection, with a specific focus on emerging challenges as VLMs evolve. 
For a discussion of the long-standing challenges in OOD detection, we can refer the readers to the previous generalized OOD detection paper <cit.>.



 §.§ OOD Detectin for Vision Language Models

a. Hard OOD Detection
Hard OOD detection will become increasingly important in the future due to its high practicality and the challenging nature of the problem. Hard OOD detection utilizes the OSR benchmark setup, where some classes within a single dataset are designated as ID and others as OOD. In this field, not only small datasets such as ImageNet-10 and ImageNet-20 <cit.> but also datasets with a larger number of classes such as ImageNet-protocol <cit.> have been proposed. Many existing studies, such as LoCoOp <cit.> and LSN <cit.>, primarily use the common ImageNet OOD benchmark, so hard OOD detection has not yet been well studied. This field will develop further in the future.



b. Post-hoc Methods
To propose post-hoc methods is important for the fundamental performance improvement of CLIP-based OOD detection. The methods of directly employing an ID classifier such as MCM <cit.> are called post-hoc methods. Prior to CLIP, various approaches were proposed  <cit.>.
However, CLIP-based post-hoc methods often underperform methods with additional OOD prompts, so they are not extensively researched in zero-shot OOD detection. However, we should focus on the scalability of post-hoc methods. The post-hoc methods <cit.> can be easily applied to many subsequent methods <cit.>, bringing fundamental performance improvements. Furthermore, very recently, post-hoc methods specifically tailored for prompt learning methods have also emerged <cit.>.
Therefore, proposing post-hoc methods and demonstrating the improvements not only in zero-shot but also in subsequent few-shot settings <cit.> is crucial, even if they underperform methods using OOD prompts in the zero-shot setting. This field should continue evolving, mirroring its growth before the advent of CLIP.



c. Bridging the Gap with Closed-set Classifiers.
OOD detection ensures the safety of ID classifiers, so it is crucial to bridge the gap between the advancements in existing closed-set classifiers and OOD detection. 
Currently, the representative method for few-shot OOD detection is LoCoOp <cit.>, a text prompt learning method based on CoOp. However, in closed-set settings, few-shot learning methods based on text prompt learning have been proposed other than CoOp <cit.>. Furthermore, text-based prompt learning methods only train the text prompts, so they cannot handle differences in image domains. Therefore, adopting methods that can handle image domain differences <cit.> for OOD detection is essential for bridging the gap with closed-set ID classifiers.



d. Training-free Few-shot OOD Detection
The research direction of training-free few-shot OOD detection is still in its infant stage, with only one existing study <cit.>. Given the prevalence of training-based methods in few-shot OOD detection, proposing methods that do not require training is crucial. Considering the advancements of training-free methods in CLIP-based anomaly detection, we anticipate a similar trajectory for CLIP-based OOD detection.
Future directions include refining adapter-based methods or leveraging external knowledge such as retrieval augmentation <cit.>.
Addressing training-free few-shot OOD detection is a pivotal step towards realizing more computationally efficient OOD detection in the future.



e. Full-spectrum OOD Detection
CLIP-based full-spectrum OOD (FS-OOD) detection is a promising research area <cit.>. In practical applications, there is a strong motivation to create models that can not only detect semantically shifted OOD inputs but also generalize to covariate-shifted data <cit.>. Within CLIP-based methods, OOD detection and generalization are often discussed in separate contexts <cit.>, resulting in a trade-off between detection and generalization performance <cit.>. Furthermore, there exists potential ambiguity in the distinction between FS-OOD detection and open-set domain generalization.
To eliminate the potential ambiguity, we have formulated a potential problem setting called hard FS-OOD detection.
We hope this survey inspires further advancements and developments for FS-OOD detection.



f. Open-vocabulary OOD Detection
Open-vocabulary OOD (OV-OOD) detection has a high practical potential, but it is still in its infant stages <cit.>.
In particular, as explained in Sec. <ref>, zero-shot OV-OOD detection is a potential research field. 
We hope that this survey paper will inspire future efforts in OV-OOD detection.



g. Object-level OOD Detection
Object-level OOD detection remains an unexplored area. As discussed in Sec. <ref>, this is due to the vastness of the OOD space, which makes it difficult to identify OOD objects using texts. 
To pave the way for future advancements in object-level OOD detection/segmentation, foundation models for localization, such as SAM <cit.>, offer a promising solution. By integrating these models with methods such as MCM <cit.>, we can potentially achieve object-level OOD detection and segmentation, opening up a new frontier in OOD detection research.




 §.§ Single-modal OOD Detection

a. Leveraging Large Pre-trained Models
Leveraging large pre-trained models is important for single-modal OOD detection. Numerous methods for OOD detection conduct experiments using backbones trained from scratch and do not utilize pre-trained models <cit.>. 
In a recent study, Miyai  <cit.> systematically investigated the impact of pre-training on OOD detection from both the perspectives of the types of OOD data and pre-training algorithms <cit.>.
Dong  <cit.> explored parameter-efficient learning for single-modal OOD detection and proposed DSGF, which leverages both fine-tuned features and original pre-trained features. While leveraging large pre-trained models <cit.> with lightweight tuning is an active area of research in single-modal closed-set classification <cit.>, there have been limited studies for single-modal OOD detection, which presents a promising avenue for future research.



b. Real-world Benchmarks and Evaluations
Considering the future development of CLIP-based OOD detection, there should be increasing focus on expanding the scope of benchmarks to encompass real-world scenarios where CLIP is less applicable.
For instance, recently, Baek  <cit.> introduced ImageNet-ES, consisting of variations in environmental and camera sensor factors by directly capturing 202k images with an actual camera in a controlled testbed, which bridges the gap between the common benchmark and the real-world scenario. Besides, utilizing datasets such as WILDS <cit.>, which consider real-world data shifts, or datasets for medical OOD detection <cit.>, can provide valuable insights, especially in safety-critical applications like autonomous driving and medical image analysis.




 §.§ Unsolvable Problem Detection

a. Exploring Effective Solutions
It is important to propose effective solutions for UPD. One of the potential approaches is to adapt the methodologies of OOD detection to UPD. For example, the perplexity of the LVLM's response could be used as a score to identify unsolvable queries. 
Furthermore, proposing model-agnostic post-hoc methods is also important to enhance the reliability of many LVLMs.
Therefore, incorporating the concepts of OOD detection techniques for UPD is an important direction for future work.



b. Extension to Diverse Benchmarks
MM-UPD Bench consists of general QA datasets. However, UPD can be incorporated into more diverse benchmarks including domain-specific knowledge for advanced reasoning <cit.> and multi-image understanding <cit.>. For example, MuirBench <cit.> incorporated the concept of UPD as a metric for robust evaluation. 
Integrating the concept of UPD into benchmarks is essential for evaluating the robustness and trustworthiness of LVLMs in their target tasks.



c. Theoretical Understanding of UPD
Theoretically understanding the reasons behind the difficulty of UPD could provide the community with valuable insights. Theorizing the behavior of LVLMs poses a shared challenge in the field, highlighting the importance of collaborative efforts within the community.



§ CONCLUSION


In this survey, we comprehensively review the evolution of the five problems including AD, ND, OSR, OOD detection, and OD in the VLM era, and propose a framework of generalized OOD detection v2. Our framework identifies OOD detection and AD as the primary challenges in the VLM era, which highlights the demanding challenges and fosters collaborative efforts among each community. 
By articulating the shifts in the definitions, problem settings, and benchmarks, we encourage subsequent works to accurately understand their evolving target problems in the VLM era.
By sorting out the methodologies, we hope that readers can easily grasp the mainstream methods, identify important baselines and novel problem settings, and propose future solutions.
By shedding light on recent studies in the LVLM era, we hope that researchers within each community can identify promising research directions in this emerging era.
By providing future directions, we hope that our survey will clarify the tasks to be tackled by future works in the VLM and LVLM era, thereby facilitating future advances in the right direction.


§ ACKNOWLEDGMENT

This work was supported by JST BOOST, Japan Grant Number JPMJBS2418 and JST JPMJCR22U4. We thank Toyooka Mashiro (AYM Lab at UTokyo) for his valuable assistance in designing the figures, and Kazuki Egashira, Yuki Imajuku, Takubon Son, and Zaiying Zhao (AYM Lab at UTokyo) for valuable feedback on the paper. 

ieeetr






[
    < g r a p h i c s >
]Atsuyuki Miyai
(Student Member, IEEE)
received his B.E. degree in Information and Communication Engineering in 2022 and his M.E. degree in Interdisciplinary Studies in Information Science in 2024 from The University of Tokyo. He is currently a Ph.D. student with the Department of Information and Communication Engineering at The University of Tokyo. His research topic is the safety of vision language models and foundation models. He is an IEEE Student Branch Chair at The University of Tokyo.




[
    < g r a p h i c s >
]
    Jingkang Yang is currently a PhD student at the College of Computing and Data Science (CCDS) at Nanyang Technological University (NTU), Singapore, working in the MMLab@NTU under the supervision of Dr. Ziwei Liu. His research interests include visual generalist models and AI safety for foundation models. He has published over 20 papers in relevant fields at top-tier conferences and journals, including CVPR, ICCV, ECCV, NeurIPS, ICLR, and IJCV. He serves as a reviewer of CVPR, ICCV, ECCV, NeurIPS, ICLR, etc., and is an outstanding reviewer in ICCV 2021 and CVPR 2024.




[
    < g r a p h i c s >
]Jingyang Zhang received his B.E. degree in Electronic Engineering from Tsinghua University, Beijing, China, in 2019.
Since then, he has been pursuing the Ph.D. degree at Dept. of Electrical and Computer Engineering at Duke University, supervised by Dr. Yiran Chen and Dr. Hai (Helen) Li.
His research spans from robustness of deep learning-based vision systems to (more recently) generative AI and multi-modal LLMs.




[
    < g r a p h i c s >
]Yifei Ming received his Ph.D. in Computer Science from the University of Wisconsin-Madison in 2024, advised by Dr. Yixuan Li. He is currently a Research Scientist at Salesforce AI Research. His research interests encompass reliable machine learning, vision-language models, large language models, and responsible foundation models. He has a prolific publication record and serves on the program committees of leading machine learning conferences and journals, including NeurIPS, ICLR, ICML, EMNLP, and IJCV.




[
    < g r a p h i c s >
]Yueqian Lin
received a B.S. degree in Data Science from Duke Kunshan University in 2024, graduating summa cum laude with signature work distinction. He is currently pursuing a Ph.D. in Electrical and Computer Engineering at Duke University under the supervision of Dr. Yiran Chen and Dr. Hai (Helen) Li. His research focuses on robustness in generative models.




[
    < g r a p h i c s >
]Qing Yu
(Member, IEEE) 
received his B.E. degree in Information and Communication Engineering and the M.E. degree in Interdisciplinary Studies in Information Science from The University of Tokyo in 2018 and 2020, respectively. He completed his Ph.D. in Information Science and Technology from The University of Tokyo in 2023. He is currently a Research Scientist at LY Corporation, Japan. His research interests include multimodal recognition and generation.




[
    < g r a p h i c s >
]Go Irie
(Member, IEEE) received his B.E. and M.E. in System Engineering from Keio University, Japan, in 2004 and 2006, respectively. He completed his Ph.D. in Information Science and Technology from The University of Tokyo in 2011. 
He is currently an Associate Professor at the Department of Information and Computer Technology, Tokyo University of Science, Japan. He was a research scientist at NTT Corporation, Japan, from 2006 to 2022, and a Visiting Research Scholar at Columbia University from 2012 to 2013. His research interests include pattern recognition, machine learning and media understanding.




[
    < g r a p h i c s >
]Shafiq Joty is currently a Research Director at Salesforce Research (Palo Alto, USA), where he oversees the NLP group's efforts in large language modeling (LLM) and generative AI. He also holds the position of a tenured Associate Professor (currently on leave) in the School of Computer Science and Engineering (SCSE) at NTU, Singapore. He was a founding manager of the Salesforce Research Asia (Singapore) lab. His research has contributed to over 30+ patents and 145+ papers in top-tier NLP and ML conferences and journals. He has served as the Program Chair of SIGDIAL-2023, as a member of the best paper award committees for ICLR-23 and NAACL-22, and in the capacity of a (senior) area chair for many of the leading NLP and ML conferences.





[
    < g r a p h i c s >
]Yixuan Li (Member, IEEE) received the PhD degree from Cornell University, in 2017, advised by John E. Hopcroft. She is an assistant professor in the Department of Computer Sciences, the University of Wisconsin-Madison. Subsequently, she was a postdoctoral scholar in the computer science department, Stanford University. Her research focuses on the algorithmic and theoretical foundations of learning in open worlds. She has served as Area Chair for ICLR, NeurIPS, ICML, and Program Chair for Workshop on Uncertainty and Robustness in Deep Learning. She is the recipient of the AFOSR Young Investigator Program (YIP) award, NSF CAREER award, MIT Technology Review Innovator Under 35, Forbes 30 Under 30 in Science, and multiple faculty research awards from Google, Meta, and Amazon. Her works received a NeurIPS Outstanding Paper Award, and an ICLR Outstanding Paper Award Honorable Mention, in 2022.





[
    < g r a p h i c s >
]Hai (Helen) Li
(Fellow, IEEE) received the Ph.D. degree from Purdue University in 2004. 
Dr. Li is currently the Clare Boothe Luce Professor and Department Chair of the Electrical and Computer Engineering Department at Duke University.
Her current research interests include neuromorphic circuits and systems for brain-inspired computing, machine learning acceleration and trustworthy AI, conventional and emerging memory design and architecture, and software and hardware co-design.
Dr. Li is a recipient of the NSF Career Award (2012), DARPA Young Faculty Award (2013), TUM-IAS Hans Fischer Fellowship from Germany (2017), and ELATE Fellowship (2020). She received 9 best paper awards and additional 9 best paper nominations from international conferences. Dr. Li is a Distinguished Lecturer of the IEEE CAS Society (2018-2019) and a Distinguished Speaker of ACM (2017-2020). 





[
    < g r a p h i c s >
]Ziwei Liu
(Member, IEEE) is currently a nanyang assistant professor with Nanyang Technological University, Singapore. His research interests include computer vision machine learning and computer graphics. He has published extensively on top-tier conferences and journals in relevant fields, including CVPR, ICCV, ECCV, NeurlPS, ICLR, ICML, IEEE Transactions on Pattern Analysis and Machine Intelligence, ACM Transactions on Graphics and Nature - Machine Intelligence. He is the recipient of ICCV Young Researcher Award, HKSTP Best Paper Award, CVPR Best Paper Award Candidate, ICBS Frontiers of Science Award and MIT Technology Review Innovators under 35 Asia Pacific. He serves as an area chair of CVPR, ICCV, ECCV, NeurlPS and ICLR, as well as an associate editor of International Journal of Computer Vision.





[
    < g r a p h i c s >
]Toshihiko Yamasaki
(Member, IEEE) received the Ph.D. degree from The University of Tokyo. He is currently a Professor at the Department of Information and Communication Engineering, Graduate School of Information Science and Technology, The University of Tokyo. He was a JSPS Fellow for Research Abroad and a visiting scientist at Cornell University from 2011 to 2013. His current research interests are computer vision, multimedia, pattern recognition, machine learning. He is a pioneer of attractiveness computing.





[
    < g r a p h i c s >
]Kiyoharu Aizawa
(Fellow, IEEE) received the B.E., the M.E., and the Dr.Eng. degrees in Electrical Engineering all from the University of Tokyo, in 1983, 1985, 1988, respectively. He is a professor with the Department of Information and Communication Engineering, and director of VR center, University of Tokyo. He was a visiting assistant professor with the University of Illinois from 1990 to 1992. His research fields are multimedia, image processing, and computer vision, with a particular interest in interdisciplinary and cross-disciplinary issues. He received the 1990, 1998 Best Paper Awards, the 1991 Achievement Award, 1999 Electronics Society Award from IEICE Japan, and the 1998 Fujio Frontier Award, the 2002 and 2009 Best Paper Award, and 2013 Achievement award from ITE Japan. He received the IBM Japan Science Prize in 2002. He is on Editorial Board of ACM TOMM. He served as the Editor-in-Chief of Journal of ITE Japan, an Associate Editor of IEEE TIP, TCSVT, TMM, and MultiMedia. He has served a number of international and domestic conferences; he was a General co-Chair of ACM Multimedia 2012 and ICMR2018. He is a Fellow of IEEE, IEICE, ITE and a member of Science Council of Japan. 






