Paper Title: Explainable and Controllable Motion Curve Guided Cardiac Ultrasound Video Generation.







Cardiac Ultrasound Video Generation




 


1National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Medical School,
Shenzhen University, China

xinyang@szu.edu.cn; nidong@szu.edu.cn 

2Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, China

3Marshall Laboratory of Biomedical Engineering, Shenzhen University, China

4Faculty of Applied Sciences Macao Polytechnic University, Macao, China 

5Shenzhen RayShape Medical Technology Co., Ltd, China




J.Yu and R.Chen et al.


Explainable and Controllable Motion Curve Guided Cardiac Ultrasound Video Generation
    Junxuan Yu1,2,3Junxuan Yu and Rusi Chen contribute equally to this work. Rusi Chen1,2,3⋆ Yongsong Zhou1,2,3 Yanlin Chen1,2,3 Yaofei Duan4 Yuhao Huang1,2,3 Han Zhou1,2,3,5 Tan Tao4 Xin Yang1,2,3() Dong Ni1,2,3()
    August 5, 2024
======================================================================================================================================================================================================================



§ ABSTRACT 
Echocardiography video is a primary modality for diagnosing heart diseases, but the limited data poses challenges for both clinical teaching and machine learning training.
Recently, video generative models have emerged as a promising strategy to alleviate this issue.
However, previous methods often relied on holistic conditions during generation, hindering the flexible movement control over specific cardiac structures.
In this context, we propose an explainable and controllable method for echocardiography video generation, taking an initial frame and a motion curve as guidance. 
Our contributions are three-fold.
First, we extract motion information from each heart substructure to construct motion curves, enabling the diffusion model to synthesize customized echocardiography videos by modifying these curves.
Second, we propose the structure-to-motion alignment module, which can map semantic features onto motion curves across cardiac structures.
Third, The position-aware attention mechanism is designed to enhance video consistency utilizing Gaussian masks with structural position information.
Extensive experiments on three echocardiography datasets show that our method outperforms others regarding fidelity and consistency. The full code will be released at https://github.com/mlmi-2024-72/ECMhttps://github.com/mlmi-2024-72/ECM.







§ INTRODUCTION

Echocardiography is a primary method that relies on dynamic video to obtain structural information for clinical diagnoses <cit.>. 
However, training radiologists with diagnostic skills and establishing machine learning models both suffer from limitations on video resources.
Recently, video generation models have demonstrated a promising ability to solve this problem, owing to their powerful capability in modeling data distribution <cit.>.
Several studies about video generation based on specific conditions have been investigated.
The typical ones relied on canny edges or depth maps <cit.> extracted from additional videos as conditions. Nevertheless, these conditions are non-editable and lack motion-driving information.
In contrast, Shi et al. <cit.> proposed the Motion-I2V framework to predict dense optical flow and guide video generation, maintaining both spatial and motion consistency.
Wang et al. <cit.> then employed a self-tracking training method.
Specifically, they specified the box positions of the first and last frames along with motion paths, to control the movement of objects.
However, these paths are basic and coarse, inadequate for ultrasound video synthesis that demands an accurate representation of intricate structure motions.





In the field of echocardiography video generation, 
Zhou et al. <cit.> proposed the OnUVS framework to synthesize ultrasound videos by animating source images and leveraging motion information from driving video. 
Nevertheless, OnUVS faced challenges due to the anatomical structure gap between the source image and the driving video, making accurate motion control difficult.
In addition, some studies have mined the structural information of the heart to guide the movement of videos.
For instance, Reynaud et al. <cit.> developed a Generative Adversarial Network (GAN) capable of generating echocardiography videos corresponding to the left ventricular ejection fractions (LVEFs).
They further employed a cascade video diffusion model conditioned on randomly sampled frames, enhancing the synthesis quality of echocardiography video <cit.>.
Van et al. <cit.> utilized segmentation masks of end-diastolic (ED) frames as a condition to generate four-chamber heart videos.
However, both LVEFs and ED masks are relatively sparse conditions, leading to an imbalance of information between intricate motions and limited conditions. 
This sparsity poses challenges for effectively controlling fine-grained cardiac structures, thus limiting the ability to capture the full complexity of heart movements and dynamics.

To address the above issues, we propose an explainable and controllable motion curve guided video diffusion model (ECM) that can synthesize video guided by the initial frame and motion curves (Fig. <ref>). 
Our contributions are threefold: 
(1) We innovatively mine the motion of the echocardiography video to obtain motion curves, which fully reflect the movement of each cardiac structure. This easily controlled approach enables the customization of videos through the modification (scaling and replacing) of the initial motion curve.
(2) As the curve lacks category information of the structure, we propose a Structure-to-Motion alignment mechanism. This mechanism extracts the semantic features of each cardiac structure and maps them with motion curve features, aiming to align visual and motion information effectively.
(3) We design position-aware attention masks based on the position of the cardiac structure movement, effectively enhancing the motion consistency of each structure.
To the best of our knowledge, ECM is the first study to apply cardiac motion curve guidance in echocardiography video generation. Extensive experimental results show that the proposed ECM is a flexible, controllable, and reliable method.




§ METHODOLOGY




Fig. <ref> shows the framework of ECM. During the training process, ECM takes an original video as input. 
A pretrained variational auto-encoder (VAE) from Stable Diffusion (SD) <cit.> is then utilized to downsample the video into latent features. 
By gradually adding noise to the latent and then learning to denoise it, the model can obtain the generated video with a pretrained VAE decoder. 
Remarkably, we develop the structure-to-motion alignment module to match cardiac structures with motion, yielding aligned features that condition the ECM model. 
Additionally, we employ Gaussian masks for each structure using a Position-aware Attention Mechanism, incorporating these into the spatial layers of the diffusion model. 
During the testing process, ECM generates echocardiography videos using an initial frame and motion curves as input. Users can customize the videos by replacing the initial frame or adjusting (scaling or replacing) the motion curves, highlighting the controllability and interpretability of our method.




 §.§ Extraction of Motion Curves

Previous studies <cit.> faced challenges in controlling the motion of specific cardiac substructures since they relied solely on the single sparse condition (LVEFs), while our approach aims to provide fine-grained control over the motion of each cardiac substructure.
Therefore, we extract motion curves for the key structures (e.g., Left Ventricle, Left Atrium, Mitral Valve, etc.) as conditions.

As illustrated in Fig. <ref>, the process of extracting motion curves is as follows: 
(a) We employ a well-trained anatomy detector (average accuracy=85%) to identify each substructure in each frame of the echocardiogram video. 
(b) Subsequently, we utilize the pixel coordinates of each substructure's bounding box (bbox) as the basis for encoding the motion curves, which form the basis for encoding the motion curves, represented as 
f_c^m∈ℝ^B × N × C × ( 4 × 2),
where N represents the number of frames and C represents the categories of the substructure. Notably, any missed detected structures would be treated as learnable parameters, initialized by the network.
(c) Since cardiac motion is periodic, we employ Fourier transformation (FT) to transform the pixel coordinates into a high-dimensional feature representation, denoted as 
f_c^m∈ℝ^B × N × C × E, 
where E represents the dimensionality of the features obtained from FT. 
(d) Finally, the motion embedding is passed through several multi-layer perceptron (MLP) layers. Overall, these motion curve features can be formulated as f_c^m∈ℝ^B × N × C × 1024.



 


 §.§ Structure-to-Motion Alignment Module


Although the motion curves of the echocardiography videos are captured, it is challenging for the model to distinguish the relationship between the motion curves and the semantic information of each substructure.
Recently, GLIGEN <cit.> has demonstrated its effectiveness in combining caption and bbox information, enhancing visual-language understanding to enable fine-grained control over specific objects in natural images.
However, our preliminary experiments suggested that GLIGEN struggles to effectively represent and interpret texts related to cardiac structures in echocardiography.



Innovatively, we replace texts with the cardiac structure features, denoted as f_i^s. 
As shown in Fig. <ref>, to obtain f_i^s, 
the regions of interest (ROI) that correspond to the cardiac substructures in the initial frame are cropped by the well-pretrained detector as mentioned above. 
Then, a pretrained CLIP image encoder <cit.> is utilized to transform the ROIs into structural embedding features.
Then, these features pass through several MLP layers,
resulting in an output denoted as f_c^s∈ℝ^B × N × C × 1024, which has the same shape with the extracted motion curve features f_i^m mentioned in Sec. <ref>.
Consequently, structural features and their corresponding motion curve features from the same category are concatenated to create aligned features. The aligned features are formulated as:

    F_c^a = Concat(f_c^s, f_c^m).


Note that any undetected structures are replaced with general features from the dataset.
Next, the concatenated features pass through an additional MLP layer for further integration.
Finally, to introduce the aligned motion curve features to guide the generation of echocardiography video, we then mapped 
F_c^a to the intermediate spatial layers of the UNet<cit.> via a cross-attention layer implementing cross attention, formulated as:

    CrossAtten(Q, K, V) = softmax( QK^T/√(d)) · V,

where Q, K and V represent the query, key and value respectively in the attention mechanism. Here, we regard noise latent feature F_l^m∈ℝ^B × 64 × 64 as query and the structure-to-motion embedding features as key and value. 






 §.§ Position-aware Attention Mechanism

Enhancing the consistency of cardiac motion is crucial in the task of cardiac video generation. 
To address this, we aim to inject the positional information of the cardiac structure into the cross-attention mechanism. Specifically, we design Gaussian masks that are generated based on the position of the cardiac structures. 
The Gaussian masks are defined as:

    M_g(x, y) = 1/2πσ^2exp(-(x - μ_x)^2 + (y - μ_y)^2/2σ^2),

where x and y represent the spatial positions of the mask.
μ_x and μ_y denote the center positions of the Gaussian distribution, which are the coordinates of the four corner points of the detected bbox. 
σ is the standard deviation to control the distribution width. 
Here, we set σ to 10 pixels as default.
Subsequently, the Gaussian masks are resized to match the dimensions of the latent feature map. 
The Gaussian-weighted cross-attention mechanism is as follows:

    CrossAttenMask(Q, K, V, M_g) = softmax( QK^T/√(d_k)⊙M_g ) V,

where ⊙ is the element-wise multiplication. Overall, this position-aware attention mechanism effectively integrates positional information of cardiac structures, enhancing the consistency and realism of generated cardiac motion.





§ EXPERIMENTS


Datasets and Implementations. 
To assess the performance of ECM, we gathered data from three sources: two private dataset from multiple hospitals and the publicly available dataset named EchoNet-Dynamic <cit.>.
The in-house dataset includes 144 apical four-chamber (A4C) and 100 apical two-chamber (A2C) heart videos, and the public one comprises 10,030 labeled A4C echocardiography videos.
For both datasets, the videos were randomly split into training (90%) and testing (10%) sets. During training, 12-frame clips were randomly sampled from each video, with a sampling interval ranging from 1 to 4 frames. 
For testing, videos were truncated according to different sampling intervals. 
The input videos from the private datasets were resized to 256x256 pixels, while those from the EchoNet-Dynamic dataset were kept at their original resolution of 112x112 pixels.
All methods were implemented in PyTorch using an NVIDIA RTX 4090 GPU under same settings. The Adam optimizer was used with a learning rate of 5e-3 and 60,000 training steps.

Evaluation Metrics.
Our evaluation metrics cover both image-level and video-level evaluation.
The image-level assessment includes Structural Similarity Index (SSIM) <cit.>, Mean Absolute Error (MAE) <cit.>, Peak Signal-to-Noise Ratio (PSNR) <cit.>, Fréchet Inception Distance (FID) <cit.> and Learned Perceptual Image Patch Similarity (LPIPS) <cit.>.
For video-level assessment, we only considered the commonly-used Fréchet Video Distance (FVD) <cit.>.
Notably, to assess the consistency of cardiac structures between the generated and target videos, we introduce a new indicator calculated by the Intersection over Union (IoU) between the bboxes of the original and synthesized videos.




Method Comparison.
The quantitative comparison of ECM and other methods is reported in Table <ref>.
The ECM model achieves the best performance across all metrics, indicating superior image quality and high fidelity compared to other methods.
It can be seen that the SEG Diffusion method, generating echocardiography videos without any control conditions, achieves poor quality.
Notably, ECM model markedly outperforms the SD method under bbox/text conditions, achieving 67.8%↓ in FVD. 
This demonstrates that synthesizing 2D images with rich conditions and stitching them into videos does not yield satisfactory outcomes. 
The integration of a position-aware attention mechanism significantly contributes to our superior performance and enhanced video consistency.
Furthermore, the ECM model shows better performance than 3D ControlNet. 
This disparity arises from the fact that natural images typically have clearly defined control conditions, whereas the motion in echocardiography videos is inherently more complex.
Similarly, for the EchoNet-Dynamic dataset, guided by motion curves, our proposed ECM outperforms the strong competitor <cit.> across most metrics, indicating its superior generation capacity.










Ablation Study.
We conducted the ablation study to test the contribution of each component in Table <ref>. 
The ECM model consistently outperforms others across most metrics for both the A4C and A2C datasets, achieving outstanding FVD scores of 189.27 and 208.12, and impressive IoU scores of 0.766 and 0.828, respectively.
This highlights that ECM effectively enhances video generation consistency while maintaining strong image quality.
Additionally, incorporating motion curves instead of traditional textual control leads to notable improvements. 
Specifically, using motion curves, the FVD decreased from 307.50 to 259.77, and the IoU increased from 0.691 to 0.748.
This indicates that motion curves provide precise control over the motion in echocardiography videos.
It can also be observed that the Structure-to-Motion alignment module (`+S2M') and the position-aware attention mechanism (`+Att') enhance image quality and video motion consistency, respectively. Consequently, the final ECM model demonstrates excellent performance at both the image and video levels.



Qualitative Results. 
Fig. <ref> demonstrates that ECM generates videos closely matching the target in both the A4C and EchoNet-Dynamic datasets by inputting the initial frame and motion curves. It effectively mimics heart structure motions such as chamber dilation, contraction, and diastolic opening and systolic closing
Furthermore, Fig. <ref> (a) illustrates ECM's controllability. Starting from the same initial frame, the second-row video uses the original motion curves from target video A, while the third-row video uses replaced curves from target video B. The mitral valve's systole and diastole differ after replacing the motion curves (as shown by yellow arrows), indicating ECM's effective control over motion curves.
More visualization results, including generated videos with replaced and scaled motion curves, can be found at http://106.53.160.189:8000/anonymous link.




§ CONCLUSION

In this study, We have presented ECM for generating echocardiography videos guided by motion curves, which reflect the movement of each cardiac structure. ECM enables to customize the generated videos by adjusting (scaling and replacing) the initial motion curve. Besides, to link the structure features with corresponding movement information, we propose the structure-to-motion alignment mechanism. Moreover, attention masks based on the position of the anatomical structures are introduced to enhance the motion consistency of each structure. Overall, our proposed ECM achieves state-of-the-art performance for generating echocardiography videos in terms of fidelity and consistency.




  §.§.§ 

This work was supported by the grant from National Natural Science Foundation of China (12326619, 62101343, 62171290), Science and Technology Planning Project of Guangdong Province (2023A0505020002), Science and Technology Development Fund of Macao (0021/2022/AGJ), and Shenzhen-Hong Kong Joint Research Program (SGDX20201103095613036).



  §.§.§ 

The authors have no competing interests to declare that are relevant to the content of this article.


splncs04




 Paper Title: Head360: Learning a Parametric 3D Full-Head for Free-View Synthesis in 360°.





 



Head360










Y. He et al.










^1 State Key Laboratory for Novel Software Technology, Nanjing University, China

^2 Fudan University, Shanghai, China   ^3 Tencent AI Lab, Shenzhen, China





Head360: Learning a Parametric 3D Full-Head for Free-View Synthesis in 360^∘

    Yuxiao He^1, Yiyu Zhuang^1, Yanwen Wang^1, Yao Yao^1, Siyu Zhu^2, 

Xiaoyu Li^3, Qi Zhang^3, Xun Cao^1, Hao Zhu^1
    
=====================================================================================================================




§ ABSTRACT 

Creating a 360^∘ parametric model of a human head is a very challenging task. While recent advancements have demonstrated the efficacy of leveraging synthetic data for building such parametric head models, their performance remains inadequate in crucial areas such as expression-driven animation, hairstyle editing, and text-based modifications. In this paper, we build a dataset of artist-designed high-fidelity human heads and propose to create a novel parametric 360-degree renderable parametric head model from it. Our scheme decouples the facial motion/shape and facial appearance, which are represented by a classic parametric 3D mesh model and an attached neural texture, respectively. We further propose a training method for decompositing hairstyle and facial appearance, allowing free-swapping of the hairstyle. A novel inversion fitting method is presented based on single image input with high generalization and fidelity. To the best of our knowledge, our model is the first parametric 3D full-head that achieves 360^∘ free-view synthesis, image-based fitting, appearance editing, and animation within a single model. Experiments show that facial motions and appearances are well disentangled in the parametric space, leading to SOTA performance in rendering and animating quality. The code and dataset are released in <https://nju-3dv.github.io/projects/Head360>.





§ INTRODUCTION



3D head modeling has been a longstanding and hot research topic in computer vision and graphics and is essential to many human-related downstream tasks. Generating a high-fidelity head model involves 3D surface reconstruction, material modeling, hair designing, rigging, etc, which is very complicated. To solve these challenges, classic pipelines rely on expensive systems (like light stage <cit.>) and the inevitable large amount of manpower (like hair design and rigging correction) to produce a high-fidelity head model. In recent years, the emergence of neural rendering and neural texture techniques has made it possible to implement this pipeline end-to-end with a sophisticated neural network, thus fully automated through a data-driven approach.


Overall, the head modeling methods based on neural rendering can be divided into two technical routings. The first routing leverages massive 2D face image sets for training and achieves highly realistic rendering, motion retargeting, and identity inversion <cit.>. The downsides are obvious, that is, it can only achieve high-quality renderings at a small angle range on the frontal side (<±30^∘), while larger poses will cause serious image quality degradation. The second routing is to learn neural heads from 3D models, which are scanned real faces <cit.> or models designed by artists <cit.>. These methods can synthesize 360^∘ high-quality renderings. However, the performance of motion animation and identity inversion is worse since too few 3D models are available for training. In summary, existing parametric 3D heads inevitably suffer from limited renderable angles (those trained on massive 2D images) or poor rigging/identity inversion (those trained on limited 3D head models).







In this paper, we go further along the routine of modeling human heads from artist-designed 3D models.
Since previous artist-designed datasets are not publicly available, we produce and schedule to open-source a high-fidelity 3D head dataset of 100 identities and 52 standard defined expressions per identity. To the best of our knowledge, this is the first publicly available large-scale high-fidelity 3D human head dataset. Based on the dataset, a hybrid representation is leveraged to model the 4D head with hair, adopting mesh and its parametric model to represent 3D shape/motion while adopting neural texture to represent the appearance. The shape and appearance are then transformed into a radiance field and are rendered via the volume renderer. We split the radiation field into two parts, representing the head and the hair, respectively, so that the hairstyle can be switched freely. With this framework, we train a parametric head model with hair that is 360^∘ renderable.

Based on this parametric model, we further extend its application for image-based fitting and appearance editing. A novel fitting method is designed to fit the shape and appearance of the whole human head based on a single image input.  The generated or fitted head can be stylized according to a text prompt by introducing the prior from the large vision-language model. We are surprised that the fitted or edited head can still be animated with standard blendshapes parameter streams in high quality. We consider the reason because the framework of `parametric 3D mesh + neural texture' effectively disentangles the head motion and appearance. Experiments show that our model can generate high-quality 3D head models that outperform previous parametric heads and are capable of fitting, animating, and text-based editing.


Our contribution can be summarized as:


    
  * We propose a parametric 3D mesh + neural texture for parameterizing 3D heads, which has proved effective in disentangling head motions and appearances.

    
  * A novel framework is proposed to detach hair and head, leading to the capability of hairstyle swapping and stronger fitting performance.
    
  * Image-based fitting and text-based stylizing schemes are proposed specialized for our parametric head fitting and editing.
    
  * A high-fidelity artist-designed 3D head dataset consisting of 100 identities, each with 52 standard expressions, is created and will be released upon publication.





§ RELATED WORK



3D Morphable Model (3DMM): 
3DMM is defined as a statistical model that transforms the 3D shape and texture of the heads into a vector space representation<cit.>. The parametric transformation is learned from a large set of heads in diverse shapes, appearances, and expressions, which are represented by registered 3D polygonal mesh models. According to the properties of parametric transformation, 3DMM can be further divided into linear models and nonlinear models. For a linear 3DMM, Principal Component Analysis (PCA) <cit.> is commonly used to learn the linear mapping from the 3D vertices and texture space of the head model to a low-dimensional space <cit.>. Linear 3DMM conversion calculation is straightforward and is widely used in face alignment, face reconstruction, and other tasks. For a non-linear 3DMM, a neural network or other non-linear model is adopted to model the mapping <cit.>, which is more powerful in representing detailed shape and appearance than linear mapping.
3DMM is widely used in 3D face reconstruction <cit.>, generation <cit.>, talking faces <cit.>, and other downstream applications. The main problem with 3DMM is that it cannot model hair, beard, and eyelashes, since 3D polygon mesh models can hardly represent these structures. Therefore, traditional 3DMM cannot be used to render highly realistic images. This problem is solved in methods 2D generative model and conditional NeRF by introducing photometric loss and differentiable rendering. We recommend referring to the recent survey<cit.> for a comprehensive review of 3DMM. By contrast,

2D Generative Head Model: 
Research on deep generative models has made significant progress in the last decade. The conditional generative models represented by the generative adversarial network (GAN) <cit.> have produced excellent results in the task of 2D face generation <cit.>. These methods learn a distribution of a large-scale 2D face image dataset through a deep generative network, which is trained by a min-max game between the generator network and the discriminator network. GAN-based models can not only synthesize 2D faces given a parameter but also obtain the parameters that fit a given 2D face by GAN inversion methods <cit.>.  As GAN-based models are commonly trained on a large amount of real-captured portraits, their generated faces are highly photo-realistic. The disadvantage of GAN-based models is that they lack 3D information, so large-angle view synthesis is not supported. We highly recommend reading the survey on GANs <cit.> and GANs for face <cit.> for a comprehensive understanding of this field.  Very recently, diffusion models <cit.> for face generation have shown appealing performance in cross-modal face generation <cit.>. However, similar to GAN-based generative models, they suffer from the lack of stable 3D information as well. 

Conditional Head NeRF: 
NeRF <cit.> provides an alternative representation for the parametric head by combining an implicit neural network with a volume renderer. 
A straightforward idea is to combine NeRF with GAN to create a conditional 3D GAN <cit.>. Early works <cit.> train generative 3D GANs by using massive 2D real portraits and are surprised to find that the model can learn 3D information from these 2D portraits with various identities and views. However, the learned 3D information is limited and can be rendered only at small angles in the front, while larger angles will lead to severe rendering degradation. HeadNeRF <cit.> proposes to train the model with both large-scale 2D head images and relatively few 3D models, but only extending the renderable view to about 60^∘. MoFaNeRF <cit.> only leverages high-quality multi-view head images for training and realizes a 180-degree free-view renderable parameter head model. However, the main problem is that existing high-quality multi-view 3D head data are commonly captured with hair tied up, so MoFaNeRF does not model hair. In later studies, RODIN <cit.> trains the parametric head with large-scale multi-view images rendered from artist-designed 3D models, and PanoHead <cit.> combined large-scale frontal 2D images with captured back-view hair images for training, both of which achieved a 360^∘ renderable parametric head model. The main problem with these two models is that their facial expressions and motions are not riggable. Unlike all the above models, our model is the first motion-riggable and 360^∘ renderable high-fidelity parametric head model.




§ METHOD







We present a parametric 3D head model with hair for free-view synthesis in 360^∘ and achieve single-image fitting, animating, and text-based editing. In this section, we will first introduce the artist-designed 3D head datasets in Sec. <ref>, which is used to train the parametric head. Then, the representation of the 3D head and the network design will be explained in Sec. <ref>. Finally, the training methods and the model manipulations, including fitting, animation, and editing, will be presented in Sec. <ref> and Sec. <ref>, respectively.




 §.§ Dataset




The quality and quantity of the training data are crucial to the performance of a learning-based parametric head, but capturing ideal training data is very hard and expensive. As shown in Tab. <ref>, previous multi-view or 3D head datasets have certain obvious flaws, like low 3D accuracy <cit.>, wrapped hair <cit.>, and lack of rigging <cit.>. Very recently, learning from large-scale synthetic 3D heads <cit.> has proved to be effective <cit.>. However, the synthetic 3D head dataset they used is not publicly released and is a very high cost.

Therefore, we create a high-fidelity synthetic 3D head dataset for free research use, containing 100 different subjects with diverse hairstyles and facial appearances, rigged into 52 standard blendshapes bases. The models were designed by the artists referring to FaceScape <cit.> and HeadSpace <cit.>, two public data sets, which cover the appearance of western and eastern races. 
The dataset encompasses 374,400 calibrated high-resolution images and 5,200 mesh models for each identity under 52 expressions. The ratio of males to females in the dataset is 1:1, and the age is fairly evenly distributed between 16 and 70. The 3D heads are rendered by 72 head-centric virtual cameras covering 3 pitch angles and 24 horizontal rotation angles. Both rendered images and 3D mesh models will be released. Regarding model quality, the models of dataset are more detailed and realistic than that of the Rodin dataset, as shown in Figure <ref>. Our models exhibit a greater level of skin detail, including pores, subtle textures, and the natural variations of skin irregularities. The rendering of light and shadow on facial features is more refined, reflecting a complex lighting environment that includes specular reflections off the skin's surface and soft shadow edges. More descriptions can be found in the supplementary material.










 §.§ Head Representation



As shown in Fig. <ref>, our model is represented by a neural radiance field <cit.> with hex-planes, which is conditioned on a generative neural texture <cit.> and a parametric 3D mesh model <cit.>. 

Parametric 3D Mesh. 
Given the shape code s and blendshapes code b representing facial motion, a head avatar mesh can be generated by a mapping defined by the parametric 3D mesh model. This mapping is established by reducing dimensionality to the synthetic dataset on the identity dimension. Specifically, given the vertices of the mesh models in our synthetic dataset containing 100 identities × 52 blendshapes bases, we first create a large matrix V_o ∈ℝ^(3N) × 100 × 52, where N denotes the number of the vertices. Then tucker decomposition <cit.> is used on the identity dimension, yielding a smaller core tensor C_r ∈ℝ^(3N)×50×52, the bilinear model face model for our model. In this way, the mapping between 3D models and parameters (s for facial shape and b for facial expressions) can be formulated as:


    V = C_r × s × b


where V is a N × 3 vector representing vertices of a head avatar model. 

Dual Hex-planes for Detaching Hair. 
Given the texture code t, the neural texture and the feature maps are synthesized by an image generator network. Following Next3D <cit.>, the neural textures with the generated 3D mesh are rasterized into multiple feature planes and rendered to the image given the camera parameters. Finally, RefineNet further improves the details of the rendered images. 
Based on the representation of tri-planes NeRF conditioned on neural texture and 3D mesh, we further propose three modifications.
First, instead of three feature planes to be rasterized <cit.>, we adopt six feature planes, namely hex-planes, to model the 360^∘ appearance of the head from front, back, left, right, top, and bottom. Experiments showed that the hex-planes improve the rendering quality and are more suitable for training on 360^∘ multi-view image data. Second, A two-branch network is introduced to model a bald head and hair separately. We propose a training strategy that decouples hair and head to achieve free-swapping of hairstyles and better fitting performance. The training strategy for disentangling hair and head will be detailed in Sec. <ref>.  Finally, a more accurate parametric 3D head model is used, which is connected to the rasterizing module. Though the parametric head model is not trained with neural texture, the connection enables the shape code s to be an optimizable parameter in the GAN inversion. 




 §.§ Model Training




Loss Functions.
The loss function to train the hex-planes conditioned on neural texture and parametric 3D mesh is formulated as: 


    ℒ_total = ℒ_photo + λ_1( ℒ_d-gan + λ_2ℒ_density)


where ℒ_photo is the photometric loss, which is calculated by L_1 loss between the renderings and the ground-truth image; ℒ_d-gan is the dual GAN loss presented in Next3D <cit.>; ℒ_density is the density regularization proposed in EG3D <cit.>. In our experiments, the loss weights λ_1 is set to 0.01, and λ_1 is set following the setting of EG3D. The learning rate l is related to the regularization interval R_i of the generator and discriminator, which is formulated:



    l    = l_base×R_i/Ri + 1



where the R_i for the generator is 4 and that for the discriminator is 16 in our experiments; Adam optimizer is used to train the model with the base learning rate l_base as 0.0025 and the batch size as 8. 


RefineNet.
A conditional GAN is introduced to further improve the details of the generated faces. Following MoFaNeRF <cit.>, we adopt pix2pixHD <cit.> as the backbone of our RefineNet. The input of the RefineNet is the generated images, which are rendered from the composited hex-planes. In the training phase, the overall network except for RefineNet is first trained, then RefineNet is trained with the other parts of the network detached. The loss function and hyper-parameters for training RefineNet are the same as pix2pixHD <cit.>.


Implementation Details.
Our model is trained on 4 NVIDIA 3090 GPUs for roughly 10 days. To achieve a balance between training speed and rendering quality, the images in our dataset are downscaled to a resolution of 512×512 before being fed into the network, and the rendering structure of the model also outputs images at a resolution of 512×512. 
As introduced in Sec. <ref>, dual hex-planes are generated for head and hair separately. The full network with two hex-planes trainable is firstly trained with the images with hairs in 3000k iterations, then is fine-tuned for hair detachment for additional 40k iterations.
Due to limited space, we put more training details in the supplementary material.





 §.§ Model Manipulation



Fitting to Single Image. 
A novel method is proposed to fit our model to a target image. Initially, we followed GAN-inversion <cit.> to keep the trained parameters of the model unchanged and optimize for shape code s and texture code t, but found the fitting results bad.  Therefore, three improvements are made to improve the fitting performance according to our observations and experiments.

Firstly, we believe that one major reason is that the solution space formed by s and t is too large to be optimized simultaneously.  We also tried to optimize t and s iteratively, again with no good results. Therefore, we propose to obtain a relatively faithful shape code s by minimizing the projection error of facial landmarks, according to Eq.  <ref> as presented by Yang  <cit.>. Then, the shape code is set unchanged, and only texture is optimized via GAN-inversion. 

Secondly, considering that our parametric head is trained on relatively small data with only 100 identities, the parametric space of the texture code t is very limited to cover diverse facial appearances. To address this issue, we propose to optimize for the Neural Texture rather than texture code t or mapped texture code w, which further boosts the generalization of the fitting. Specifically, L1 loss between the generated frontal views and the input image is adopted for the single-image fitting. The shape code s is optimized to obtain a fitted mesh that matches the input image. The fitted shape is then fixed and a random neural texture is selected as the initialization to optimize the neural texture. The image is firstly normalized to align the input face and the canonical position of our parametric 3D head. Then, Poisson blending <cit.> is used on the output image obtained from random texture initialization with wild real faces that have been aligned to dataset. By optimizing the hex-planes space to reduce the discrepancy between the pre-processed images and those generated with random textures, the neural texture and latent code for hair can be optimized. Then, a 360^∘ renderable and animatable head is generated to fit the appearance of the input image.


Thirdly, despite implementing the aforementioned improvements, we noticed that the fitting results for hairstyles remain unsatisfactory. We believe this is due to the extensive diversity in everyday hairstyles, which our dataset of 100 hairstyle sets is inadequate to represent. To alleviate this problem, we propose to first remove the hair from the target images by semantic parsing algorithm <cit.> and optimize for a bald head model. Then a hairstyle classifier based on ResNet-50 <cit.> is trained to classify the target's hairstyle into 30 categories that match the hairstyle categories in our dataset.  The model is trained by 6200 face images synthesized by StyleGanV2 with manual annotations of hairstyle. Finally, the predicted hairstyle is combined with the fitted bald head model to generate a high-quality head model with hair. 



Animation. Our parametric 3D mesh is defined following a standard 52 facial blendshapes, which is also adopted by ARKit <cit.>. Therefore, our generated or fitted head can be animated by many blendshapes streaming tools. Our model can synthesize appearance features not closely attached to the mesh surface (like hair, mouth, and eyelashes), which are temporal consistent. It means that our proposed learnable representation can reach the performance of complex traditional artist-designed head avatars, achieving high-fidelity animation. The animation results are shown in Sec. <ref> and the supplementary video.

Text-based Editing. 
Inspired by Instruct-NeRF2NeRF <cit.>, we incorporate text-based image-conditioned diffusion model <cit.> with our animatable 3D parametric representation for 3D head editing. 
Specifically, after a head is generated or fitted, the multi-view images are rendered and edited by a text-guided image-to-image translator. The edited images are then used to fine-tune our parametric 3D head model. Empirically, we sample patches on generated images to fine-tune our parametric model, leading to better performance than the original random sampling strategy. Our text-based editing strategy aims to add abstract features, such as `wearing makeup' and `aged', to a generated head, without changing his/her identity.
The edited head can still be animated by standard blendshapes parameters.










§ EXPERIMENTS



In this section, we first compare our fitting results and generated results with previous methods, then analyze the effectiveness of the proposed module through the ablation study, and finally show the results of animation, hairstyle-swapping, and text-based editing.




 §.§ Comparison of Fitting Results


We compare our method with state-of-the-art parametric or generative 3D head models in the task of single image fitting, including EG3D <cit.>, PanoHead <cit.>, HeadNeRF <cit.>, MoFaNeRF <cit.>, and Next3D <cit.>. Both original and re-trained models on our dataset are compared for a comprehensive comparison.

All the methods are compared quantitatively in Tab. <ref> and qualitatively in Fig. <ref>. 
Regarding original models, EG3D and Next3D are originally trained on large-scale in-the-wild 2D face datasets. The fitted head is of high quality at the frontal views but fails when viewing angles are larger than 90^∘. The PanoHead model is trained on both large-scale in-the-wild 2D images and in-house captured hair and head images. PanoHead synthesizes plausible results in 360^∘, but the resulting model cannot be animated like ours. The MoFaNeRF model is trained on studio-captured high-quality 3D face models. Its results are canonical heads without hair and fail to be rendered when the viewing angles are larger than 90^∘ as well. HeadNeRF leverages both in-the-wild images and studio-captured multi-view images for training. Its results are animatable but degrade severely when viewing angles are larger. In summary, the fitting results of all these methods are plausible at frontal views, but cannot be rendered in 360^∘ except for PanoHead. The major drawback of PanoHead is that its results are static and cannot be animated.

Regarding re-trained models on our dataset, we evaluate the performance of re-trained EG3D, PanoHead, and MoFaNeRF, as shown in Fig. <ref> and Tab. <ref>. The fitting method of all these three models is to optimize for the latent code by GAN inversion strategy. Since the appearances in our training set are few (100 identities), the performance of retrained EG3D, PanoHead, and MoFaNeRF is poor due to the limited appearance space. By contrast, our method optimizes the neural texture to fit the input image, which enables the model to fit in a much larger solution space. The visual comparison shows that our results are more plausible in 360^∘ than all the re-trained models, which is also verified by the quantitative evaluations.

























 §.§ Comparison of Generated Results



We compare our method with several pervious parametric or generative 3D head models, including EG3D <cit.>, PanoHead <cit.>, HeadNeRF <cit.>, and MoFaNeRF <cit.>.
The original models in their papers are trained on large-scale 2D image datasets or reconstructed 3D datasets, which is quite different from the synthetic dataset we used. For a fair comparison, we re-train EG3D, PanoHead, and MoFaNeRF on our synthetic dataset. The generated faces of the same identity in the training set are compared to evaluate the representation ability of these models.

The qualitative comparison is shown in Fig. <ref>. The aim of EG3D and PanoHead is modeling static 3D models that cannot be animated, which is relatively easier. The results of EG3D, PanoHead, and ours are of comparable quality, while our results are better in facial details. Our performance is also better than another animatable parametric 3D head, MoFaNeRF. 

The quantitative comparison is shown in Tab. <ref>, where PSNR <cit.>, SSIM <cit.>, LPIPS <cit.>, and CSIM <cit.> are for method are reported. For a fair comparison, all the models to be evaluated are re-trained on our artist-designed dataset. EG3D and PanoHead cannot be animated as the expression is not parameterized in their model, so only neutral expressions are used for evaluation. By contrast, the results of MoFaNeRF and our model are riggable.  Our method outperforms previous methods in all four metrics, which demonstrates the effectiveness of our representation in 360^∘ head modeling. 




 §.§ Ablation Study

We perform ablation studies on specific modules as follows:

    
  * Hex-planes ⟶ Vanilla  NeRF. The neural texture and hex-planes (Sec. <ref>) are replaced by a conditioned vanilla NeRF.
    
  * Hex-planes ⟶ Tri-planes. The hex-planes representation (Sec. <ref>) is replaced by a classic tri-planes representation.
    
  * w/o RefineNet. The RefineNet (Sec. <ref>) is removed.


As reported in Tab. <ref>, Fig. <ref>, adding RefineNet slightly decreases PSNR and SSIM scores, while enhancing LPIPS score. We think the reason is that RefineNet synthesizes photo-realistic details that conform to perceptual features (higher LPIPS). Though these details do not match the ground truth details (lower PSNR and SSIM), synthesizing such realistic details is visually better, as visualized in Fig. <ref>.
Additionally, ablation experiments are conducted to verify the effectiveness of the hair detaching module, as reported in Tab. <ref> and Fig. <ref>.





 §.§ Other Manipulations


Hair-Swapping. As the hair and head are detached in the hidden feature space, our model supports free hair-swapping for a generated or fitted head. As shown on the right side of Fig. <ref>, given a generated head on the left, the original hairstyle can be replaced with others. 


3D Animation.  The generated results can be driven by blendshapes parameter streams.
As shown in Fig. <ref>, the facial details like mouth, eyes, and motion wrinkles of a dynamic head are accurately modeled.


Text-based Editing Results. As shown in Fig. <ref>, our generated 3D model can be further edited given a text prompt. The subject's identity is maintained after the editing, while abstract features like `makeup' and `aged' are synthesized. 







§ CONCLUSION


In this paper, we have constructed a dataset of artist-designed, high-fidelity human heads, and developed a novel framework to learn a 360^∘ free-view renderable parametric model from this dataset. Our approach decouples the facial motion/shape and facial appearance, represented by a classic blendshapes model and neural texture, respectively. Notably, our model is the first of its kind — a parametric 3D head model that supports 360^∘ free-view synthesis, single-image fitting, and animation driven by a blendshapes parameter flow.

Limitations. 
Creating high-fidelity avatars comes at a considerable cost, thus limiting our dataset to 100 identities.
Considering the high cost of producing high-fidelity digital human data, we think it is meaningful to study how to learn a parametric head model under the condition of limited data amount.
Although some in-the-wild fitting results are plausible by learning from this finite training set, stability in fitting remains a challenge. Evidence of this can be found in the failure case presented in our supplementary material.  A potential solution might entail expanding the dataset and incorporating real-world images and models into the training process.
Moreover, our method does not decompose the material and lighting, which results in some highlights being baked into the texture. It reduces the photo-realism and application potential. This issue could be alleviated by utilizing the illumination and material information from the synthetic dataset.
We leave these considerations for future work.

Acknowledgement. This study was funded by NKRDC 2022YFF0902200, NSFC 62441204, and Tencent Rhino-Bird Research Program.














splncs04





 Paper Title: Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention.




Quantized electrical, thermal, and spin transports of non-Hermitian clean and dirty two-dimensional topological insulators and superconductors
    Bitan Roy
    August 5, 2024
==============================================================================================================================================




§ ABSTRACT 

Conditional diffusion models have shown remarkable success in visual content generation, producing high-quality samples across various domains, largely due to classifier-free guidance (CFG). Recent attempts to extend guidance to unconditional models have relied on heuristic techniques, resulting in suboptimal generation quality and unintended effects. In this work, we propose Smoothed Energy Guidance (SEG), a novel training- and condition-free approach that leverages the energy-based perspective of the self-attention mechanism to enhance image generation. By defining the energy of self-attention, we introduce a method to reduce the curvature of the energy landscape of attention and use the output as the unconditional prediction. Practically, we control the curvature of the energy landscape by adjusting the Gaussian kernel parameter while keeping the guidance scale parameter fixed. Additionally, we present a query blurring method that is equivalent to blurring the entire attention weights without incurring quadratic complexity in the number of tokens. In our experiments, SEG achieves a Pareto improvement in both quality and the reduction of side effects. The code is available at <https://github.com/SusungHong/SEG-SDXL>.






§ INTRODUCTION


Diffusion models <cit.> have emerged as a promising tool for visual content generation, producing high-quality and diverse samples across various domains, including image <cit.>, video <cit.>, and 3D generation <cit.>. The success of these models can be largely attributed to the use of classifier-free guidance (CFG) <cit.>, which enables sampling from a sharper distribution, resulting in improved sample quality. However, CFG is not applicable to unconditional image generation, where no specific conditions are provided, creating a disparity between the capabilities of text-conditioned sampling and sampling without text. This disparity results in a restriction in application, e.g., synthesizing images with ControlNet<cit.> without a text prompt (see the last two columns of Fig. <ref>).

Recent literature <cit.> has attempted to decouple CFG and image quality by extending guidance to general diffusion models, leveraging their inherent representations <cit.>. Self-attention guidance (SAG) <cit.> proposes leveraging the intermediate self-attention map of diffusion models to blur the input pixels and provide guidance, while perturbed attention guidance (PAG) <cit.> perturbs the attention map itself by replacing it with an identity attention map. Despite these efforts, these methods rely on heuristics to make perturbed predictions, resulting in unintended effects such as smoothed-out details, saturation, color shifts, and significant changes in the image structure when given a large guidance scale. Notably, the mathematical underpinnings of these unconditional guidance approaches are not well elucidated.




In this work, we approach the objective from an energy-based perspective of the self-attention mechanism, which has been previously explored based on its close connection to the Hopfield energy <cit.>. Specifically, we start from the definition of the energy of self-attention, where performing a self-attention operation is equivalent to taking a gradient step. In light of this, we propose a tuning- and condition-free method that reduces the curvature of the underlying energy function by directly blurring the attention weights, and then leverages the output as the negative prediction. We call this method Smoothed Energy Guidance (SEG).

SEG does not merely rely on the guidance scale parameter that cause side effects when its value becomes large. Instead, we can continuously control the original and maximally attenuated curvature of the energy landscape behind the self-attention by simply adjusting the parameter of the Gaussian kernel, with the guidance scale parameter fixed. Additionally, we introduce a novel query blurring technique, which is equivalent to blurring the entire attention weights without incurring quadratic cost in the number of tokens.

We validate the effectiveness of SEG throughout the various experiments without and with text conditions, and ControlNet <cit.> trained on canny and depth maps. Based on the attention modulation, SEG results in less structural change from the original prediction compared to previous approaches <cit.>, while achieving better sample quality.





§ PRELIMINARIES




 §.§ Diffusion models



Diffusion models <cit.> are a class of generative models that generate data through an iterative denoising process. The process of adding noise to an image 𝐱 over time t ∈ [0, T] is governed by the forward stochastic differential equation (SDE):

    d𝐱 = 𝐟(𝐱, t)dt + g(t)d𝐰,

where 𝐟 and g are predefined functions that determine the manner in which the noise is added, and d𝐰 denotes a standard Wiener process.

Correspondingly, the denoising process can be described by the reverse SDE:

    d𝐱 = [𝐟(𝐱, t) - g(t)^2 ∇_𝐱log p_t(𝐱)] dt + g(t)d𝐰̅,

where ∇_𝐱log p_t(𝐱) represents the score of the noisy data distribution and d𝐰̅ denotes the standard Wiener process for the reversed time.

Diffusion models are trained to approximate the score function with 𝐬_θ(𝐱, t) ≈∇_𝐱log p_t(𝐱). To generate an image based on a condition c, e.g., a class label or text, one simply needs to train diffusion models to approximate the conditional score function with 𝐬_θ(𝐱, t, c) ≈∇_𝐱log p_t(𝐱|c) and replace ∇_𝐱log p_t(𝐱) with it in the denoising process. To enhance the quality and faithfulness of the generated samples, classifier-free guidance (CFG) <cit.> is widely adopted. Accordingly, the reverse process becomes:

    d𝐱 = [𝐟(𝐱, t) - g(t)^2 (γ_cfg𝐬_θ(𝐱, t, c) - (γ_cfg-1) 𝐬_θ(𝐱, t))] dt + g(t)d𝐰̅.

Here, 𝐬_θ(𝐱, t) is learned by dropping the label by a certain proportion, and γ_cfg is a hyperparameter that controls the strength of the guidance. Intuitively, CFG helps us to sample from sharper distribution by conditioning on a class label or text.



 §.§ Energy-based view of attention mechanism



The attention mechanism <cit.>, which has been widely adopted in diffusion models <cit.>, has been interpreted through the lens of energy-based models (EBMs) <cit.>, especially through its close connection with the Hopfield energy <cit.>. In the modern (continuous) Hopfield network, the attention operation can be derived based on the concave-convex procedure (CCCP) from the following energy function <cit.>:

    E(ξ) = - lse(𝐗ξ^⊤) + 1/2ξξ^⊤,

where ξ∈ℝ^1× d, 𝐗∈ℝ^N× d, and lse stands for the log-sum-exp function, defined as lse(𝐯) := log(∑_i=1^N e^v_i). The quadratic term acts as a regularizer to prevent ξ from exploding <cit.>, while -lse(𝐗ξ^⊤) penalizes misalignment between 𝐗 and ξ.

Mathematically, it turns out that the attention mechanism is equivalent to the update rule of the modern Hopfield network <cit.>. Specifically, inspired by the Hopfield energy in (<ref>), and noticing that the first term depends on the attention weights, we propose the following energy function for entire self-attention weights in diffusion models:

Let 𝐐∈ℝ^(HW)× d be a matrix of query vectors and 𝐊∈ℝ^(HW)× d be a matrix of key vectors, where H, W, and d represent the height, width, and dimension, respectively. Let 𝐀∈ℝ^(HW)× (HW) := 𝐐𝐊^⊤. The energy function with respect to entire self-attention weights in diffusion models is defined as:

    E(𝐀) := ∑_i=1^H∑_j=1^W E'(𝐚_:(i,j)),
       E'(𝐚) := - lse(𝐚) = - log(∑_k=1^H ∑_l=1^W e^a_(k,l)).


Note that to explicitly denote the spatial dimension, we use the subscript (x, y) to represent the index of a row or column of the matrices. Despite using the definition in (<ref>) for the rest of the paper for simplicity, we additionally discuss the dual case, where we use the swapped indexing, in Appendix <ref>.

This view leads us to an important intuition: the attention operation can be seen as a minimization step on the energy landscape, considering that the first derivative represents the softmax operation which also appears in the attention operation. Building upon this intuition, we argue that Gaussian blurring on the attention weights modulates the underlying landscape to have less curvature, and we demonstrate this in the following sections by analyzing the second derivatives.



§ METHOD


Our aim is to theoretically derive the effect of Gaussian blur applied on the attention weights, which in the end attenuates the curvature of the underlying energy function. Then, utilizing this fact, we develop attention-based drop-in diffusion guidance that enhances the quality of the generated samples, regardless of whether an explicit condition is given. In Section <ref>, we claim some useful properties of Gaussian blur: that it preserves mean, reduces variance, and thus decreases the lse value. In Section <ref>, we find that the curvature of the energy landscape is attenuated by the attention blur operation, leading naturally to a blunter prediction for guidance. And finally, in Section <ref>, built upon this fact, we define Smoothed Energy Guidance (SEG) and propose the equivalent query blurring method, which can perform attention blurring while avoiding quadratic complexity in the number of tokens.



 §.§ Gaussian blur to attention weights



In this section, we derive some important properties of the Gaussian blur with the aim of figuring out the variation of the energy landscape. To this end, we start from some mathematical underpinnings on applying Gaussian blur to attention weights.

A 2D Gaussian filter is a convolution kernel that uses a 2D Gaussian function to assign weights to neighboring pixels. The 2D Gaussian function is defined as:

    G(x, y) = 1/2πσ^2 e^-(x-μ_x)^2 + (y-μ_y)^2/2σ^2

where μ_x and μ_y are the means in the x and y directions, and σ is the standard deviation.
The 2D Gaussian filter possesses symmetry, i.e., G(x, y) = G(-x, -y), and normalization, i.e., ∬ G(x, y)dxdy = 1. In practice, we use a discretized version of the Gaussian filter with a finite kernel size depending on σ, normalized to sum to 1.


Spatially applying a 2D Gaussian blur to the attention weights 𝐚 := 𝐐𝐤^⊤ preserves the average 𝔼_i, j[a_(i,j)]. In addition, the variance monotonically decreases every time we apply the Gaussian blur.



Proof sketch.
Applying a 2D Gaussian filter to the attention weights a_(i, j) yields the blurred values ã_(i, j):

    ã_(i,j) = ∑_m=-k^k∑_n=-k^k G(m, n) · a_(i+m, j+n)

where k is the filter size, G(m, n) is the Gaussian filter value at position (m, n), and a_(i+m, j+n) is the attention weight at position (i+m, j+n). Since the Gaussian filter is symmetric and normalized, it can be shown that the mean of the blurred attention weights is equal to the mean of the original attention weights. Similarly, we can show that the variance monotonically decreases when we apply a 2D Gaussian filter. See Appendix <ref> for the complete proof.

Note that this fact also implies that blurring with a Gaussian filter with a larger standard deviation causes a greater decrease in the variance of attention weights. This is because a Gaussian filter with a larger standard deviation can always be represented as a convolution of two filters with smaller standard deviations, due to the associativity of the convolution operation.

Finally, we show that applying a 2D Gaussian blur to attention weights increases the lse value in (<ref>), i.e., increases the energy in (<ref>). This provides a bit of intuition about the underlying energy landscape, yet it is more prominently utilized in the claims in the following sections.


Applying a 2D Gaussian blur to attention weights 𝐚 := 𝐐𝐤^⊤ increases the lse term when we consider the second-order Taylor series approximation of the exponential function around the mean μ := 𝔼_i,j[a_(i,j)]. Consequently, the maximum is achieved when the attention is uniform, i.e., a_(i,j) = a_(k,l) ∀ i,j,k,l. This corresponds to the case when we apply the Gaussian blur with σ→∞.



Proof sketch. Applying the second-order Taylor series approximation around the mean μ, and using Proposition <ref>, we show that the second-order approximation of lse(𝐚) is larger than or equal to that of lse(𝐚̃). Subsequently, we introduce Lagrange multipliers to find the maximum, which gives us the result, a_(i,j) = a_(k,l) ∀ i,j,k,l. We leave the full proof in Appendix <ref>.




 §.§ Analysis of the energy landscape



In this section, we demonstrate that applying a 2D Gaussian blur to the attention weights before the softmax operation results in computing the updated value with reduced curvature of the underlying energy function. To this end, we analyze the Gaussian curvature before and after blurring the attention weights. This is closely related to the Hessian of the energy function.


Let the attention weights be defined as 𝐚 := 𝐐𝐤^⊤. Consider the energy function in (<ref>). Then, applying a Gaussian blur to the attention weights 𝐚 before the softmax operation results in the attenuation of the Gaussian curvature of the underlying energy function where gradient descent is performed.



Proof sketch.
Let 𝐇 denote the Hessian of the original energy function, i.e., the derivative of the negative softmax, and 𝐇̃ denote the Hessian of the new energy function associated with blurred attention weights. Furthermore, let b_ij denote the i-th row, j-th column entry in the Toeplitz matrix 𝐁 representing the Gaussian blur. Calculating the derivatives, we have the elements of the Hessians, h_ij = (ξ(𝐚)_i - δ_ij)ξ(𝐚)_j and h̃_ij = (ξ(𝐚̃)_i - δ_ij)ξ(𝐚̃)_j b_ij. Using Lemmas <ref> and <ref> and under reasonable assumptions, we observe that |(𝐇)| > |(𝐇̃)|, which implies that the minimization step is performed on a smoother energy landscape with attenuated Gaussian curvature. The full proof is in Appendix <ref>.

To provide more intuition about what is actually happening and how we utilize this property in the later section, it is intriguing to consider the attenuating effect on the curvature in analogy to classifier-free guidance (CFG). CFG uses the difference between the prediction based on the sharper conditional distribution and the prediction based on the smoother unconditional distribution to guide the sampling process. By analogy, we propose a method to make the landscape of the energy function smoother to guide the sampling process, as opposed to the original (sharper) energy landscape.

From a probabilistic perspective, this process can be thought of as maximizing the likelihood of the attention weights in terms of the Boltzmann distribution conditioned on a given configuration, i.e., the feature map. Blurring the attention weights diminishes this likelihood as shown in Lemma <ref>, and also reduces the curvature of the distribution as shown in Theorem <ref>.




 §.§ Smoothed energy guidance for diffusion models



Based on the above observation that the Gaussian blur on attention weights attenuates the curvature of the energy function, we propose Smoothed Energy Guidance (SEG) in this section. For brevity, we redefine the unconditional score prediction as 𝐬_θ(𝐱, t), and the unconditional score prediction with the energy curvature reduced as 𝐬̃_θ(𝐱, t). Specifically, 𝐬̃_θ(𝐱, t) is the prediction with the attention weights blurred using a 2D Gaussian filter G with the standard deviation σ. We formulate the process as:

    (𝐐𝐊^⊤)_seg = G * (𝐐𝐊^⊤),

where * denotes the 2D convolution operator. Then, we replace the original attention weights with (𝐐𝐊^⊤)_seg and compute the final value as in ordinary self-attention.

For practical purposes when the number of tokens is large, we propose an efficient computation of (<ref>) using the property of a linear map, since the convolution operation is linear. Concretely, blurring queries is exactly the same as blurring the entire attention weights, and we propose the following proposition to justify our claim.


Let 𝐐 and 𝐊 be the query and key matrices in self-attention, and let G be a 2D Gaussian filter. Blurring the attention weights with G is equivalent to blurring the query matrix 𝐐 with G and then computing the attention weights.




Since the convolution operation is linear, we can always find a Toeplitz matrix 𝐁 such that:

    G ∗ (𝐐𝐊^⊤) = 𝐁(𝐐𝐊^⊤),

where ∗ denotes the 2D convolution operation. Using the properties of matrix multiplication, we can rewrite (<ref>) as:

    𝐁(𝐐𝐊^⊤) = (𝐁𝐐)𝐊^⊤ = (G ∗𝐐)𝐊^⊤.


Finally, SEG is formulated as follows:

    d𝐱 = [𝐟(𝐱, t) - g(t)^2 (γ_seg𝐬_θ(𝐱, t) - (γ_seg - 1) 𝐬̃_θ(𝐱, t))] dt + g(t)d𝐰̅,

where γ_seg denotes the guidance scale of SEG.

In a straightforward manner, as SEG does not rely on external conditions, it can be used for conditional sampling strategies such as CFG <cit.> and ControlNet <cit.>. For the combinatorial sampling with CFG, following <cit.>, we simply extend (<ref>) for improved conditional sampling with both SEG and CFG as follows:

    d𝐱 = [𝐟(𝐱, t) - g(t)^2 ((1-γ_cfg+γ_seg) 𝐬_θ(𝐱, t) + γ_cfg𝐬_θ(𝐱, t, c) - γ_seg𝐬̃_θ(𝐱, t))] dt + g(t)d𝐰̅,

which is an intuitive result, as the update rule moves x towards the conditional prediction while keeping it far from the prediction with blurred attention weights.

We are likely to get a result with saturation when using a large guidance scale, such as with classifier-free guidance (CFG) <cit.>, self-attention guidance (SAG) <cit.>, and perturbed attention guidance (PAG) <cit.>. This is a significant caveat since we need to increase the scale to achieve a maximum effect with these methods. Contrary to this, we can fix the scale of SEG as justified in Sec. <ref> and control its maximum effect through σ of the Gaussian blur, making the choice more flexible. For σ, two extreme cases are recognized. If σ→ 0, the blurred attention weights remain the same as the original, while when σ→∞, the attention weights merely adopt a single mean value across spatial axes. We find that even the latter extreme case results in a high-quality outcome, corroborating that we can control the quality to the limit without saturation.












§ DISCUSSION ON RELATED WORK




Classifier-free guidance (CFG) <cit.>, first proposed as a replacement for classifier guidance (CG) <cit.> is controlled by a scale parameter. The higher we set classifier-free guidance, the more we get faithful, high-quality images. However, it requires external labels, such as text <cit.> or class <cit.> labels, making it impossible to apply to unconditional diffusion models. Also, it requires specific traning procedure with label dropping and it is known that high CFG causes saturation <cit.>.

Tackling the caveats of CFG, unconditional approaches such as self-attention guidance (SAG) <cit.> and perturbed attention guidance (PAG) <cit.> have been proposed. SAG selectively blurs images with the mask obtained from the attention map and guides the generation process given the prediction. This indirect approach causes saturation and noisy images when given a large guidance scale, leading to the selection of a guidance scale less than or equal to 1. PAG guides images using prediction with identity attention, where the attention map is an identity matrix. However, the reliance on heuristics to make perturbed predictions results in unintended side effects. As an example of the side effects of replacing the attention map with identity attention, PAG changes the visual structure and color distribution of an image, as evidenced in Figs. <ref>, <ref>, and <ref>.

Contrary to these, we control the effect of SEG through the standard deviation of the Gaussian filter, σ. Moreover, while being theory-inspired, SEG is relatively free from unintended effects. In the following section, we corroborate our claim with extensive experiments.




§ EXPERIMENTS





 §.§ Implementation details


We build upon the current open-source state-of-the-art diffusion model, Stable Diffusion XL (SDXL) <cit.>, as our baseline, and do not change the configuration. To sample with SEG, we choose the same attention layers (mid-blocks) and guidance scale as PAG <cit.>. For SEG and PAG sampling, we use the Euler discrete scheduler <cit.>, while for SAG <cit.>, we instead use the DDIM scheduler <cit.> since the current implementation of SAG does not support the Euler discrete sampler. For SAG and PAG, we use the same configurations they used in the experiments with the previous version of Stable Diffusion, with guidance scales of 1.0 and 3.0, respectively. We set γ_seg to 3.0, except in the ablation study.















 §.§ Metrics


We use various metrics to evaluate quality (FID <cit.> and CLIP score <cit.>, calculated with 30k references from the MS-COCO 2014 validation set <cit.>) and to assess the extent of change due to applied guidance (LPIPS_vgg, alex <cit.>). The latter metric, calculated using the outputs of vanilla SDXL, measures the extent of side effects by comparing guided images to their unguided counterparts.



 §.§ Controlling image generation with the standard deviation


In this section, our aim is to demonstrate that with SEG, we can sample plausible images using vanilla SDXL <cit.> under various conditions and even without any conditions, as demonstrated in Fig. <ref>. Furthermore, without the risk of saturation, we can control the quality and plausibility of the samples. For the results, we use σ∈{1, 2, 5, 10}. Additionally, as mentioned in Sec. <ref>, we present two extreme cases, σ→ 0 (vanilla SDXL) and σ→∞ (uniform queries).



  
Unconditional generation

In this section, our aim is to demonstrate that with SEG, we can sample plausible images from the unconditional mode of the vanilla SDXL, which was originally trained on a large-scale text-to-image dataset. The results are presented in Fig. <ref>, Fig. <ref>, and Table <ref>. The results show a clear tendency to draw higher quality samples by utilizing the differences between the two energy landscapes with different curvatures derived from self-attention mechanisms.

In Fig. <ref> and Fig. <ref>, we show the effectiveness of generating more plausible images, while vanilla SDXL is unable to generate high-quality images without any conditions. The results show a clear tendency to draw higher quality samples by utilizing the differences between the two energy landscapes with different curvatures derived from self-attention mechanisms. When σ is larger, the definition and expression of the samples improve, as the difference in curvature becomes more pronounced.



  
Conditional generation

In Figs. <ref>, <ref>, <ref>, <ref>, and <ref>, we display sampling results conditioned on text, Canny, and depth map. Using text (Fig. <ref>), the vanilla SDXL without CFG is unable to generate high-quality images and produces noisy results. Canny and depth map conditioning on SDXL (Fig. <ref>, <ref>, and <ref>) is achieved through ControlNet <cit.>, trained on such maps. The results show that SEG enhances the quality and fidelity of the generated images while preserving the textual and structural information provided by the conditioning inputs. Notably, as σ increases, the generated images exhibit improved definition and quality without introducing significant artifacts or deviations from the original condition. The combination with higher CFG scales is shown in Figs. <ref>–<ref>.

In Table <ref>, we show the quantitative results for text-conditional generation in terms of σ. We observe a clear trade-off between image quality (represented by FID and CLIP score) and the deviation from the original sample (represented by LPIPS). We sample 30k images for each σ to compute the metrics.




 §.§ Comparison with previous methods

Since the results are visually favorable when we use σ = 10 and σ→∞, and they are the best in terms of CLIP score and FID, respectively, we adopt those configurations for comparison of unconditional guidance methods. The results are presented in Figs. <ref>, <ref>, <ref>, and Table <ref>. Notably, our method achieves better image quality in terms of FID, while remaining similar to the original output of vanilla SDXL as measured by LPIPS, implying a Pareto improvement.

r6.5cm



    < g r a p h i c s >



Ablation study on γ_seg and σ.








 §.§ Ablation study



In this section, we address two parameters, γ_seg and σ, and justify that fixing γ_seg is a reasonable choice. In Fig. <ref>, we present the results from our testing. The results reveal that increasing γ_seg does not generally lead to improved sample quality in terms of FID and CLIP score, due to various issues such as saturation. In contrast, increasing σ tends to improve sample quality and plausibility. This supports the claim that image quality should be controlled by σ, instead of the guidance scale parameter. We sample 30k images for each combination to calculate the metrics.







§ CONCLUSION, LIMITATIONS AND SOCIETAL IMPACTS




  
Conclusion

We introduce Smoothed Energy Guidance (SEG), a novel training- and condition-free guidance method for image generation with diffusion models. The key advantages of SEG lie in its flexibility and the theoretical foundation, allowing us to significantly enhance sample quality without side effects by adjusting the standard deviation of the Gaussian filter. We hope our method inspires further research on improving generative models, and extending the approach beyond image generation, for example, to video or natural language processing.



  
Limitations and societal impacts

The paper proposes guidance to enhance quality outcomes. Consequently, the attainable quality of our approach is contingent upon the baseline model employed. Furthermore, the application of SEG to temporal attention mechanisms in video or multi-view diffusion models is not addressed, remaining a promising avenue for future research. It is important to note that the improvements achieved through this method may potentially lead to unintended negative societal consequences by inadvertently amplifying existing stereotypes or harmful biases.


plain













§ FULL PROOFS




 §.§ Proof of Lemma <ref>



Let a_(i, j) denote the original attention weights and ã_(i, j) denote the blurred attention weights, as in the main paper. Assume that the original attention weights are properly padded to maintain consistent statistics. Then, the following shows that the mean of the blurred attention weights remains the same.

    𝔼_i,j[ã_(i,j)]    = 1/HW∑_i=1^H∑_j=1^Wã_(i,j) = 1/HW∑_i=1^H∑_j=1^W∑_m=-k^k∑_n=-k^k G(m, n) · a_(i+m,j+n)
       = ∑_m=-k^k∑_n=-k^k G(m, n) ·(1/HW∑_i=1^H∑_j=1^W a_(i+m, j+n)) 
       = ∑_m=-k^k∑_n=-k^k G(m, n) ·𝔼_i,j[a_(i,j)] = 𝔼_i,j[a_(i,j)] ·∑_m=-k^k∑_n=-k^k G(m, n) = 𝔼_i,j[a_(i,j)]


In addition, the variance of the blurred attention weights is smaller than or equal to the variance of the original attention weights.

    Var_i,j[ã_(i,j)]    = 1/HW∑_i=1^H∑_j=1^W (ã_(i,j) - 𝔼_i,j[ã_(i,j)])^2 
       = 1/HW∑_i=1^H∑_j=1^W(∑_m=-k^k∑_n=-k^k G(m, n) · (a_(i+m, j+n) - 𝔼_i,j[a_(i,j)]))^2 
       = ∑_m=-k^k∑_n=-k^k∑_r=-k^k∑_s=-k^k G(m, n) · G(r, s) ·Cov[a_(i+m, j+n), a_(i+r, j+s)]

Using the Cauchy-Schwarz inequality and the normalization property of the 2D Gaussian filter, we can show that the variance monotonically decreases when we apply Gaussian blur.

    Var_i,j[ã_(i,j)]    ≤∑_m=-k^k∑_n=-k^k∑_r=-k^k∑_s=-k^k G(m, n) · G(r, s) ·√(Var[a_(i+m, j+n)] ·Var[a_(i+r, j+s)])
       = (∑_m=-k^k∑_n=-k^k G(m, n) ·√(Var[a_(i+m, j+n)]))^2 
       ≤(∑_m=-k^k∑_n=-k^k G(m, n)) ·(∑_m=-k^k∑_n=-k^k G(m, n) ·Var[a_(i+m, j+n)]) 
       = ∑_m=-k^k∑_n=-k^k G(m, n) ·Var[a_(i+m, j+n)] 
       = Var_i,j[a_(i,j)]






 §.§ Proof of Lemma <ref>



Applying the second-order Taylor series approximation of e^x to our function f around the mean μ, we get:

    ∑_i=1^H∑_j=1^W e^a_(i,j)   ≈∑_i=1^H∑_j=1^W(e^μ + e^μ(a_(i,j) - μ) + 1/2e^μ(a_(i,j) - μ)^2) 
       = HW · e^μ + 1/2e^μ∑_i=1^H∑_j=1^W (a_(i,j) - μ)^2

In the last step, we used the fact that ∑_i=1^H∑_j=1^W (a_(i,j) - μ) = 0 because μ is the mean.

Similarly,

    ∑_i=1^H∑_j=1^W e^ã_(i,j)   ≈ HW · e^μ + 1/2e^μ∑_i=1^H∑_j=1^W (ã_(i,j) - μ)^2

Since Var[a] > Var[ã], we have:

    ∑_i=1^H∑_j=1^W (a_(i,j) - μ)^2 ≥∑_i=1^H∑_j=1^W (ã_(i,j) - μ)^2

Therefore, the second-order approximation of lse(𝐚) is larger than that of lse(𝐚̃).

Note that this fact also implies blurring with a Gaussian filter with a bigger variance causes more decrease in the variance of attention weights, because Gaussian filter with a larger variance can always be represented as a convolution of two filters with smaller variances, and the convolution operation is associative.

To find the maximum value subject to the constraint a_(1,1) + a_(1,2) + … + a_(H,W) = c for some constant c, we introduce Lagrange multipliers. Let g(a_(1,1), a_(1,2), …, a_(H,W)) = a_(1,1) + a_(1,2) + … + a_(H,W). The Lagrangian function is defined as:

    L(a_(1,1), a_(1,2), …, a_(H,W), λ) = e^a_(1,1) + e^a_(1,2) + … + e^a_(H,W) - λ(a_(1,1) + a_(1,2) + … + a_(H,W) - c)

Taking partial derivatives and setting them to zero yields:

    ∂ L/∂ a_(i,j) = e^a_(i,j) - λ = 0

Solving for a_(i,j), we obtain a_(i,j) = ln(λ) for all i = 1, 2, …, H and j = 1, 2, …, W
Summing these equations results in:

    λ = e^c/HW

Substituting λ back into a_(i,j) = ln(λ) gives a_(1,1) = a_(1,2) = … = a_(H,W) = c/HW.
Therefore, the minimum value of ∑_i=1^H∑_j=1^W e^a_(i,j) is achieved when a_(1,1) = a_(1,2) = … = a_(H,W).




 §.§ Proof of Theorem <ref>



Let 𝐚 = (a_1, …, a_n) denote the attention values before the softmax operation, and let 𝐚̃ = (ã_1, …, ã_n) denote the attention values after applying the 2D Gaussian blur. Let 𝐇 denote the Hessian of the original energy, i.e., the derivative of the negative softmax, and 𝐇̃ denote the Hessian of the underlying energy associated with the blurred weights.

The elements in the i-th row and j-th column of the Hessian matrices are given by:

    h_ij = (ξ(𝐚)_i - δ_ij)ξ(𝐚)_j,


    h̃_ij = (ξ(𝐚̃)_i - δ_ij)ξ(𝐚̃)_j b_ij,

respectively, where b_ij are the elements of the Toeplitz matrix corresponding to the Gaussian blur kernel, and δ_ij denotes the Kronecker delta.

Assuming ξ(𝐚̃)_i ξ(𝐚̃)_j ≈ 0 and ξ(𝐚)_i ξ(𝐚)_j ≈ 0 for all i and j, which is a reasonable assumption when the number of token is large, the non-diagonal elements of the Hessians approximate to 0 and the diagonal elements dominate. Therefore, the determinants of the Hessian matrices are approximated as the product of the dominant terms:

    |(𝐇)| ≈∏_i=1^n ξ(𝐚)_i,    |(𝐇̃)| ≈∏_i=1^n ξ(𝐚̃)_i b_ii


We have the following inequality:

    ∏_i=1^n ξ(𝐚̃)_i b_ii < ∏_i=1^n ξ(𝐚̃)_i    = e^∑_j=1^nã_j/(∑_j=1^n e^ã_j)^n
       ≤e^∑_j=1^na_j/(∑_j=1^n e^a_j)^n = ∏_i=1^n ξ(𝐚)_i,

where the first inequality follows from the property of the Gaussian blur kernel, 0 ≤ b_ii < 1, and the second inequality is derived from Lemmas <ref> and <ref>, which demonstrate the mean-preserving property and the decrease in the lse value when applying a blur. The monotonicity of the logarithm function implies that the denominator involving the blurred attention weights is smaller. Eventually, we obtain the following inequality:

    |(𝐇̃)| < |(𝐇)|.

This implies that the updated value is derived with attenuated curvature of the energy function underlying the blurred softmax operation compared to that of the original softmax operation.






§ DUAL DEFINITION



As we previously stated in Section <ref>, we have the dual definition regarding (<ref>), where we use swapped indexing. Importantly, the swapped indices can be interpreted as altering the definition of attention weights to 𝐀:=𝐊𝐐^⊤.

A similar conclusion can be drawn as in the main paper, except that query blurring becomes key blurring with this definition. To see this, Eq. <ref> changes slightly with this definition, using the symmetry of the Toeplitz matrix 𝐁:

    G ∗ (𝐊𝐐^⊤)    = 𝐁(𝐊𝐐^⊤) 
       = ((𝐊𝐐^⊤)^⊤𝐁^⊤)^⊤
       = (𝐐𝐊^⊤𝐁^⊤)^⊤
       = (𝐐(𝐁𝐊)^⊤)^⊤
       = (𝐐(G ∗𝐊)^⊤)^⊤
       = (G ∗𝐊)𝐐^⊤,

where ∗ denotes the 2D convolution operation. Empirically, this altered definition does not introduce a significant difference in the overall image quality, as shown in Fig. <ref>.




§ ADDITIONAL QUALITATIVE RESULTS


In this section, we present further qualitative results to demonstrate the effectiveness and versatility of our Smoothed Energy Guidance (SEG) method across various generation tasks and in comparison with other approaches.



  
Comparison with previous methods

Figs. <ref> and <ref> provide a qualitative comparison of SEG against vanilla SDXL <cit.>, Self-Attention Guidance (SAG) <cit.>, and Perturbed Attention Guidance (PAG) <cit.>. These comparisons highlight the superior performance of SEG in terms of image quality, coherence, and adherence to the given prompts. SEG consistently produces sharper details, more realistic textures, and better overall composition compared to the other methods.



  
Conditional generation with ControlNet

Figs. <ref> and <ref> showcase the application of SEG in conjunction with ControlNet <cit.> for conditional image generation. These results illustrate how SEG can enhance the quality and coherence of generated images while maintaining fidelity to the provided control signals. The images demonstrate improved detail, texture, and overall visual appeal compared to standard ControlNet outputs without prompts.



  
Unconditional and text-conditional generation

Fig. <ref> demonstrates the capability of SEG in unconditional image generation, showcasing its ability to produce high-quality, diverse images without text prompts. Fig. <ref> exhibits text-conditional generation results using SEG, illustrating its effectiveness in translating textual descriptions into visually appealing and accurate images.



  
Interaction with classifier-free guidance

Figs. <ref>–<ref> present a series of experiments exploring the combination of SEG with CFG. In these experiments, the SEG guidance scale (γ_seg) is fixed at 3.0, while the CFG scale is varied. The results demonstrate that SEG consistently improves image quality across different CFG scales without causing saturation or significant changes in the general structure of the images.














































 Paper Title: MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignment.









Language-Guided Semantic Segmentation with Mask-Text Alignment






A. Das et al.

Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignment
    Anurag Das Xinting Hu Li Jiang Bernt Schiele
    Received —-; accepted —-
========================================================================





§ ABSTRACT     
Recent approaches have shown that large-scale vision-language models such as CLIP can improve semantic segmentation performance. 
These methods typically aim for pixel-level vision-language alignment, but often rely on low-resolution image features from CLIP, resulting in class ambiguities along boundaries.
Moreover, the global scene representations in CLIP text embeddings do not directly correlate with the local and detailed pixel-level features, making meaningful alignment more difficult.
To address these limitations, we introduce MTA-CLIP, a novel framework employing mask-level vision-language alignment. 
Specifically, we first propose Mask-Text Decoder that enhances the mask representations using rich textual data with the CLIP language model. Subsequently, it aligns mask representations with text embeddings using Mask-to-Text Contrastive Learning.
Furthermore, we introduce Mask-Text Prompt Learning, utilizing multiple context-specific prompts for text embeddings to capture diverse class representations across masks. 
Overall, MTA-CLIP achieves state-of-the-art, surpassing prior works by an average of 2.8% and 1.3% on standard benchmark datasets, ADE20k and Cityscapes, respectively.


 




§ INTRODUCTION



Semantic segmentation, a fundamental task in computer vision, has been significantly advanced by the integration of language models,  particularly with the advent of large-scale vision-language models (VLMs) such as CLIP <cit.>. Recent studies <cit.>  
have demonstrated that pre-training with CLIP can considerably enhance the performance of semantic segmentation. 
The core strength of CLIP is its training on diverse datasets rich in visual and textual content, which enables a context-sensitive understanding of visual content. 
In semantic segmentation, the adoption of CLIP in training is defining new benchmarks, highlighting the benefits of cross-modal embeddings. 



The common practice of integrating CLIP into semantic segmentation adopts pixel-text alignment <cit.>. In this approach, each pixel feature is aligned with the text embedding of its corresponding ground truth class label as illustrated in <ref> (a).  
However, this approach faces several challenges. 
Firstly, due to computational constraints, pixel-text alignment often operates on low-resolution image features. This down-sampling of image features, caused by the network's image encoder stride and pooling, often results in the blurring of boundaries between different classes. This results in inaccurate alignment with the respective text embeddings.
As illustrated in <ref> (a), a region containing a “car” and a “person” is indiscriminately labeled as “car” after down-sampling. As a result, the text embedding for “car” does not accurately align with the pixel-level image feature over that region. Moreover, CLIP performs image-text matching <cit.>, resulting in text embeddings with global image-level semantics. It introduces mismatches when the text embeddings are aligned with the detailed and localized pixel features. As illustrated in <ref> (b), certain pixel features for “wall” can be easily confused with the text embedding of “person”. In contrast, when using mask features[we observe only one mask feature captures the entire class, for each of the classes in this sample. We provide more visualisations in the supplement.]
 for classes “wall” and “person”, the mask features align more effectively with the corresponding text embeddings.

To address the challenges posed by pixel-text alignment, our MTA-CLIP introduces the Mask-Text alignment.
Unlike pixel representations, each mask representation with inherent boundary information captures entire entities or regions within a class. Such embeddings are more in line with CLIP's pre-training approach and avoid class ambiguity compared to per-pixel alignment.
The key component of MTA-CLIP is the novel Mask-Text Decoder. 
Different from Mask-Text matching (<ref> <cit.>) that keeps the text embeddings fixed, our Mask-Text Decoder updates both mask and text embeddings. This enhances masks with rich textual data while enriching text embeddings with visual context using layer-wise attention mechanism. 
Furthermore, MTA-CLIP uses a contrastive learning approach within the decoder. This approach aligns the enhanced mask embeddings with corresponding text embeddings back into the CLIP feature space, leveraging the potential of CLIP's pre-trained model to regularize the mask representations.
Additionally, MTA-CLIP introduces Mask-Text Prompt Learning to effectively manage the diverse contexts within mask segments of the same class. This method utilizes distinct prompts for varying scenarios, thereby enhancing the model's adaptability and contextual awareness.

Overall, our MTA-CLIP achieves a Mask-Text alignment that  addresses the inherent limitations of pixel-text alignment. 
Our contributions are summarized as follows: 

    
  * We present MTA-CLIP, a novel CLIP based semantic segmentation framework that shifts from pixel-text to mask-text vision-language alignment, aligning more closely with CLIP’s holistic scene understanding and effectively resolving class ambiguities at boundaries.
    
  * MTA-CLIP introduces Mask-Text Alignment comprising of two key components, Mask-Text Decoder, which enhances and aligns mask representations with rich textual data, and Mask-Text Prompt Learning, which employs multiple context-specific prompts to capture diverse class contexts within masks. 
    
  * Extensive evaluation on standard datasets like ADE20k and Cityscapes shows that MTA-CLIP outperforms existing state-of-the-art methods, establishing new benchmarks in the semantic segmentation task with Mask-Text vision-language alignment.





§ RELATED WORK


Semantic Segmentation. 
Semantic segmentation involves the task of assigning a categorical label to each pixel within an image, a challenge that has been extensively addressed using Deep Neural Networks (DNNs)  <cit.>. Previous research has predominantly concentrated on two architectural paradigms, namely Convolutional Neural Networks (CNNs) and Transformer-based models. CNNs  <cit.>, with their ability to capture local spatial hierarchies have shown to be highly effective. Noteworthy methods within this domain emphasize the utilization of multi-scale features, exemplified by DeepLab's Atrous Spatial Pyramid Pooling (ASPP) <cit.>, PSPNet's Pyramid Pooling Module (PPM) <cit.>, Semantic Feature Pyramid Network (FPN) <cit.> with its Feature Pyramid Network, incorporation of shape information as an additional prior <cit.>, and integration of high-resolution features <cit.>. Conversely, transformers <cit.>, distinguished by their capacity to capture global context, have exhibited superiority over CNN counterparts for semantic segmentation. While the majority of prior works have focused on per-pixel segmentation, assigning a label to each individual pixel, our work aligns with mask-based segmentation networks <cit.>, aiming to partition an image into distinct masks with associated labels. Differently, we further enhance the mask representations with rich textual data following Mask-Text Alignment.

Open Vocabulary Semantic Segmentation.
To segment potentially unlimited classes, open vocabulary segmentation methods “align” pixel  <cit.> or mask features <cit.> with textual representations. We refer open-vocabulary based “alignment” as “matching”, given that the text features are not altered in such methods.
Pixel-Text Matching method <cit.>, introduced a cost-aggregation that improves CLIP finetuning generalisation for open vocabulary segmentation. Mask-Text Matching (<ref>) methods  <cit.>, propose a two-stage framework, where mask proposals are generated in the first stage, and are matched with CLIP text features in the next stage. 
Differently, we focus on closed-set semantic segmentation, where our “Mask-Text Alignment” enhances both text and visual features, essential for optimal performance (see supplement Tab. 5). 
Further, we also learn multiple contextual prompts for text embeddings to best align diverse class representations across masks, whereas the prior works <cit.> do not use learnable prompts and use the handcrafted prompts such as “a photo of”, “a painting of” instead. 

Language Guided Semantic Segmentation.
Recent works <cit.> have demonstrated that leveraging large-scale pretrained Vision-Language Models (VLMs) such as CLIP <cit.> surpasses ImageNet-based approaches, attributed to their enhanced transferability to downstream tasks.
DenseCLIP <cit.> initially proposed an extension of CLIP for downstream semantic segmentation tasks employing a pixel-text alignment method. Subsequently, PPL <cit.> expanded upon DenseCLIP by introducing additional probabilistic prompts designed to capture diverse attributes in the data. Notably, both DenseCLIP <cit.> and PPL <cit.> rely on pixel-text vision language alignment, particularly at lower resolution features. Diverging from their approach, our proposal introduces a Mask-Text vision language alignment, specifically addressing challenges associated with pixel-text alignment (see <ref>). 


Prompt Learning.
Adapting Vision-Language Models (VLMs) to downstream tasks involves the finetuning of the model, a process that can result in the degradation of learned representations and potentially impact model transferability. Moreover, with an increase in model size, finetuning becomes computationally expensive. The concept of prompt learning, as introduced in prior studies such as <cit.>, proposes an alternative approach by acquiring additional prompt tokens that tailor the large-scale VLM to specific downstream tasks. CoOp <cit.> optimizes the Contrastive Language-Image Pretraining (CLIP) model for downstream tasks by acquiring a set of input prompts for the VLM's language model. Building upon this, CoCoOp <cit.> further refines the approach by incorporating image conditioning. Additionally, while the majority of existing methodologies concentrate on learning a singular prompt for improved generalization, ProDA <cit.> advocates for the utilization of multiple probabilistic prompts designed to cater to diverse contexts. In the context of semantic segmentation, DenseCLIP <cit.> adopts a CoCoOp <cit.> style of image-conditioned prompt learning for pixel-level predictions. Similarly, PPL <cit.> extends the ProDA framework <cit.> by learning a probabilistic distribution of diverse prompts tailored to different image contexts. Both DenseCLIP and PPL employ pixel-text alignment to facilitate prompt learning. In contrast, our approach involves Mask-Text Alignment for the acquisition of distinct prompts catering to diverse contexts. 









§ MTA-CLIP



In this section, we first describe the preliminaries (<ref>), where we discuss mask-based segmentation method, Mask2Former <cit.> and Prompt Learning, as prior information required for our work. We then discuss the key component Mask-Text Alignment (<ref>), where we describe the two components, Mask-Text Decoder and Mask-Text Prompting, that effectively aligns the mask representations with the text embeddings. In the end, we present the training details (<ref>) for our framework.  



 §.§ Preliminaries


Mask2Former.
Mask2Former <cit.> is a mask-based segmentation method that segments an image into N binary masks and classify them to K classes. Different from standard pixel segmentation methods, it consist of an additional lightweight transformer decoder along with the segmentation backbone and a pixel decoder. The transformer decoder takes learnable tokens, m_1, …, m_N (also called mask tokens) and pixel decoder features as input and processes these mask tokens with pixel decoder features with masked attention layers. It utilises pyramid resolution features extracted from the initial three layers of pixel decoder for processing mask tokens, whereas use the last layer (fourth layer) for making predictions. Typically, the dot product of the mask tokens with the last pixel decoder layer gives the final binary mask segmentations, whereas, an additional classification MLP predicts the class for these masks. Differently, in this work, our Mask-Text Decoder jointly process the text embedding and mask tokens. In particular, it enhances and align the mask tokens with CLIP text embeddings in different Mask-Text Decoder layers. Further, we also utilize multiple context prompts that captures diverse class representations among different masks. 

Prompt Learning. 
CLIP <cit.> trained on 400M text-image pairs, learns powerful representations by aligning the image and text representations using contrastive learning. For better transferability on downstream tasks, prior works <cit.> learn contextual text prompts p, instead of using hand crafted prompts such as “a photo of”, or “a painting of”. The prompts p are appended to the class token, [p, ] and are fed to the frozen text encoder to obtain the context rich class embeddings 𝐭 = [t_1, …, t_C] for C classes. Different from  <cit.>, we learn prompts that align text features with mask representations. Moreover, we learn multiple context adaptive prompts to capture diverse class representations across different masks.   




 §.§ Mask-Text Alignment



The key component of MTA-CLIP is the novel Mask-Text Alignment mechanism, that combines class-specific language information from the CLIP text encoder with the mask representations. This alignment is achieved through two pivotal components: (1) Mask-Text Decoder, which effectively enriches and aligns mask representations with contextual language features to achieve more accurate and detailed segmentation, and  (2) Mask-Text Prompt Learning, which utilizes multiple context-specific prompts to capture the diverse representations of mask segments, thereby addressing the variability in real-world scenarios.



  §.§.§ 1. Mask-Text Decoder.


Our Mask-Text Decoder has two key components: 1) Text-Enhanced Mask Feature Learning, where we use the rich textual data from language models to enhance mask representations, creating more context-aware segmentation; and 2) Mask-to-Text Contrastive Learning, aiming to establish a class-specific alignment between the mask representations and text embeddings within the CLIP feature space using a contrastive learning strategy. 

Text-Enhanced Mask Feature Learning.

In Text-Enhanced Mask Feature Learning, we enhance mask queries with class-specific language information derived from the CLIP text encoder. The process begins with constructing a novel set of queries 𝐪, by concatenating mask queries 𝐦=[m_1, …, m_N] with text embeddings 𝐭=[t_1, …, t_C] for class labels from the CLIP text encoder, where N denotes the number of queries and C denotes the number of classes. Before concatenation, the text embeddings undergo dimension reduction through a MultiLayer Perceptron (MLP), adapting them to be compatible with the mask tokens in terms of dimension size and thus  enhances computational efficiency. This reduction is formulated as:

    𝐭̂ = (𝐭),   𝐭∈ℝ^C × 1024  and  𝐭̂∈ℝ^C × 256.

Consequently, the resulting concatenated queries 𝐪 are expressed as:

    𝐪=[𝐭̂, 𝐦],   𝐪∈ℝ^(C+N) × 256.

Through this concatenation, 𝐪 integrates both CLIP-based image features and text embeddings, serving as the input of the Mask-Text Decoder. 

Within the Mask-Text Decoder, which consists of L layers, 𝐪 undergoes a series of layer-wise attention refinements. At each layer i ∈{0, 1,⋯,L-1}, the process includes cross-attention and self-attention as follows:

    𝐪_ cross^(i)= (𝐪^(i), 𝐤^(i), 𝐯^(i)),
       𝐪^(i+1) = (𝐪_ cross^(i))

Here, 𝐤^(i), 𝐯^(i) are the i^th level image features from the pixel decoder. During the cross-attention phase, both mask and text queries interact with image features, adapting to the specific context within the specific image. In the self-attention phase, mask queries not only interact among themselves to enhance the understanding of the relationships and context between different image areas, but also interact with text queries to integrate visual information with corresponding textual information. 

Mask-to-Text Contrastive Learning.

Building upon the Text-Enhanced Mask Feature Learning phase, where mask queries are augmented with visual and textual information, Mask-to-Text Contrastive Learning aims to establish class-level alignment between the mask and text representations within the CLIP feature space. This alignment is achieved through a contrastive learning approach, which involves enhancing the mask-text similarity score of a “positive” pair against those of “negative” pairs.

We first project 𝐪, including both mask queries 𝐦 and the text queries 𝐭̂, back to the CLIP feature space after each Transformer decoder layer. This projection is conducted using MultiLayer Perceptrons (MLPs):

    𝐭' = (𝐭̂)  + 𝐭^0,   𝐦' = (𝐦)

Here, the addition of the initial CLIP text embedding 𝐭^0 to the projected text embeddings ensures the preservation of the original textual space from the CLIP model.

The essence of contrastive learning involves calculating the pairwise similarity between each projected mask query m_i', (i ∈{1,⋯, N}) and text query t_j', (j ∈{1, ⋯, C}). The similarity score S, a dot product normalized by a temperature parameter τ, is given by:

    S(m'_i, t'_j) = m'_i · t'_j/τ.

We select the text query with ground truth class for each mask query as the “positive” class, while “negative” pairs include the mask query and all other non-corresponding text queries.
The alignment is refined through a contrastive loss function, ℒ_sim, which is designed to maximize the similarity for the positive pair and minimize it for negative pairs. The loss is defined as:


    ℒ_sim = - 1/N∑_i=1^N ∑_j=1^C  [    y_ij·logexp( S(m'_i, t'_j))/exp( S(m'_i, t'_j)) + ∑_r ∈{1,⋯, C}
    r  j exp( S(m'_i, t'_r))]
 


Here, y_ij equals 1 for the mask-text pair where the text query belongs to the ground truth class for the given mask query, and 0 otherwise. We obtain the ground truth class for the given mask query using bipartite matching of the predicted binary mask with actual ground truth masks similar to <cit.>.
This contrastive learning process aligns each mask query with its corresponding class text query. Moreover, we apply ℒ_sim to 𝐦' and 𝐭' after each Mask-Text Decoder layer utilizing hierarchical feature pyramids from pixel decoder, enabling the model to accurately capture and segment across a range of mask sizes. 



  §.§.§ 2. Mask-Text Prompt Learning.


Our approach tailors text prompts to distinctively align with the diverse contexts of mask segments within the same class. By implementing K unique prompts shared across C classes, we create an extensive range of K × C text queries from the text encoder. This expanded set of text queries is specifically designed to cater to the varied contexts in which a class can appear. By aligning each mask segment with this extensive array of class-context combinations, our model achieves a fine-grained Mask-Text alignment, leading to more precise and context-aware masks. To accommodate this variety in contextual representations, we expand the query set in Text-Enhanced Mask Feature Learning from C+N to K× C+N. For Mask-to-Text Contrastive Learning, we adopt a refined approach: For each mask token m'_i, the text query among the different K instances from its ground truth class with the highest similarity score is identified as the “positive” query. Specifically, the positive text query for mask token m'_i with ground truth class j is defined as :
t'_j,k,  such that k = _k∈{1,⋯,K} S(m'_i, t'_j,k).
We employ two distinct strategies for selecting negative pairs:

MixNeg: All other text queries except the positive one (totaling K× C-1) are treated as negative,  with the loss modified as:



    ℒ_sim= - 1/N∑_i=1^N ∑_j=1^C  [    y_ij·logexp( S(m'_i, t'_j,k))/exp( S(m'_i, t'_j,k)) + ∑_r ∈{1,⋯ C}, k ∈{1,⋯ K}
    {r,k}{j,k}exp( S(m'_i, t'_r,k))]
 


SeperateNeg: Text queries not belonging to the ground truth class (totaling K× (C-1)) are treated as negative, with the loss modified as:


    ℒ_sim=  - 1/N∑_i=1^N ∑_j=1^C  [    y_ij·logexp( S(m'_i, t'_j,k))/exp( S(m'_i, t'_j,k)) + ∑_r ∈{1,⋯ C}, k ∈{1,⋯ K}
      r  jexp( S(m'_i, t'_r,k))]
 


As detailed in <ref>, both strategies exhibit improvement, with MixNeg emerging as more effective due to its superior capability in differentiating between closely related contexts. 



 §.§ Training and Inference


Our training process combines the strengths of both vision and language models to optimize the semantic segmentation performance of MTA-CLIP. The core of our training lies in the effective integration and alignment of mask representations with text embeddings through the Mask-Text Decoder. 

Multi-Layer Mask-Text Alignment: At each layer of the Mask-Text Decoder, we handle N mask tokens alongside K × C class tokens derived from our Mask-Text Prompt Learning. The height and width of the pixel decoder image feature used at each Mask-Text Decoder layer, denoted as h^(i) and w^(i), vary depending on the layer. Similar to <cit.>, these dimensions are set as (h^(i),w^(i))= (H/32, W/32),(H/16, W/16), (H/8, W/8) for layers i  3=0,1,2, respectively, where H and W are the height and width of the original image, and i ∈ [0,⋯, 8] for Mask-Text Decoder with 9 layers. This variation in feature resolution across layers ensures Mask-Text Alignment at different scales, important for segmenting classes of different sizes.

Overall Loss Function: We adopt a composite loss function that balances mask classification, mask segmentation, and Mask-Text Alignment. Specifically, we use cross-entropy loss ℒ_mask-cls for the classification prediction of mask tokens and binary cross-entropy loss ℒ_mask for the segmentation prediction following <cit.>. Additionally, we have the similarity loss ℒ_sim from Equation <ref> for aligning mask tokens with context-specific class tokens. The final loss function is a weighted sum of these components:

    ℒ = λ_1 ℒ_mask-cls + λ_2 ℒ_mask + λ_3 ℒ_sim,

where λ_1, λ_2, and λ_3 are weighting factors balancing the contribution of each loss component to the overall training objective.

Inference: During inference, we ignore the text tokens (t_1,...,t_C) and utilize only the mask tokens (m_1,...,m_N) from the final layer (9th layer) of the Mask-Text Decoder for final mask predictions. In particular, following <cit.> the text aligned mask tokens (m_1,...,m_N) are fed to an MLP layer, followed by dot product with pixel-decoder final layer features to give N masks for the mask tokens. Moreover, these text aligned masks tokens goes through a separate classification MLP layer to obtain the class prediction for these mask tokens. 




§ EXPERIMENTS

In this section, we discuss the extensive experiments we conducted to demonstrate the effectiveness of MTA-CLIP. We compare our framework's performance with state-of-the-art methods for semantic segmentation. Then, we discuss ablation experiments followed by qualitative results.   


Datasets and Evaluation Metric.
We evaluate MTA-CLIP on two standard dataset benchmarks i.e., ADE20k <cit.> and Cityscapes <cit.>. ADE20k has 150 categories, covering both indoor and outdoor scenes. It has around 20,000 images for training and 2,000 images for validation. Cityscapes, on the other hand, is an urban scene understanding dataset comprising 19 categories. It has 2,975 training images along with 500 validation images. Following prior works <cit.>, we report the mean Intersection over Union (mIoU) score for evaluation.

Implementation Details.
We employ Mask2Former <cit.> as our baseline segmentation network with CLIP pre-trained backbone for the image encoder. Following previous works <cit.>, we leverage all three available CLIP image backbones: ResNet-50 <cit.>, ResNet-101 <cit.>, and ViT-B <cit.>. The text encoder is based on the CLIP-trained model, and we maintain its frozen state during training. In line with <cit.>, we set the context length for each context prompt p to 8. Moreover, we conduct experiments with K=3 different contextual prompts, unless otherwise specified.
Following prior works in the field <cit.>, we set N=100 as the number of learnable mask tokens. Our final loss (ℒ) incorporates parameters λ_1 = 2.0, λ_2 = 5.0, and λ_3 = 1.0. Training hyperparameters, following <cit.>, are as follows: for the ADE20k dataset, we utilize a crop size of 512×512, a batch size of 16, and train for 160k iterations. On the Cityscapes dataset, a crop size of 1024 × 1024, a batch size of 8, and 90k iterations are employed. The AdamW optimizer <cit.> is used with an initial learning rate of 2×10^-4.


 
 





 §.§ Comparison with State-of-the-Art


We compare MTA-CLIP's performance on two standard benchmarks i.e., ADE20k in <ref> and Cityscapes in <ref> with different backbones. We make two key comparisons. First, we compare our framework's performance with prior state-of-the-art's that uses VLM such as CLIP for semantic segmentation task. These methods <cit.> employ pixel-text alignment for aligning the vision and language modalities. Second, we compare the performance of our frameworks with an intuitive baseline of Mask2Former <cit.> with CLIP trained image encoder, without any Mask-Text Alignment. We also compare MTA-CLIP performance with classical semantic segmentation methods with ImageNet pre-training. Due to space constraint, we show Params and FLOPs comparison in supplement (Tab. 4). 

Results for ADE20k. MTA-CLIP exhibits superior performance over the state-of-the-art Pixel-to-Text alignment segmentation method PPL <cit.>, achieving a performance gain of 4.4% and 3.4% for ResNet-50 and ResNet-101 backbones, respectively. This emphasizes the substantial impact of our Mask-Text Alignment approach compared to the Pixel-Text alignment approach for semantic segmentation tasks. Employing the ViT-B backbone, our framework achieves a performance of 52.3%, surpassing PPL's 51.6%. Furthermore, our method outperforms the prior state-of-the-art ImageNet pretrained Mask2Former by 1.9%, 2.1%, and 1.2% for ResNet-50, ResNet-101, and ViT-B backbones, respectively. 

Results for Cityscapes. MTA-CLIP exhibits superior performance compared to the Pixel-to-Text alignment-based segmentation method, DenseCLIP <cit.>, with gains of 2.2% and 1.1% for ResNet-50 and ResNet-101 backbones, respectively. Mask2Former with CLIP pretraining with our reimplementation improves over Mask2Former with ImageNet pretraining by 0.6% and 1% for ResNet-50 and ResNet-101 backbones, respectively. Our framework further surpasses this strong baseline by 0.7% and 1.0% for ResNet-50 and ResNet-101 backbones, respectively, showcasing the importance of Mask-Text alignment.

In summary, MTA-CLIP consistently outperforms previous methodologies in semantic segmentation by a substantial margin across standard dataset benchmarks. This observation shows the enhanced performance achieved through additional language alignment for masks. Additionally, our framework also exhibits superiority over previous semantic segmentation methods with ImageNet pre-training, emphasizing the significance of leveraging large-scale Vision-Language Models (VLMs) for network backbone pre-training. 


 
 





 §.§ Ablation Analysis

Ablation with different components.
We conduct ablation studies on the components of our network, as detailed in <ref>. `Baseline' denotes Mask2Former trained with the CLIP ResNet-50 backbone. We first enhance mask representations following Text-Enhanced Mask Feature Learning (Text-Enhanced, <ref>) resulting in a performance gain of 0.6% mIoU over baseline. Subsequently, by aligning mask representations with text representations with similarity loss using Mask-to-Text Contrastive Learning (MTCL, <ref>), we observe a further increment of 0.5% in mIoU. Finally, we use Mask-Text Prompt Learning (MTPL, <ref>) with MixNeg to learn distinct prompts, yielding an additional boost of 0.5% in mIoU. The Mask-Text Decoder, which combines the learning of enhanced mask representations (Text-Enhanced) followed by aligning them with text representations (MTCL), gives a significant boost of 1.2% in the mIoU score compared to the baseline, showing the importance of such alignment. 

Results with other backbones.
We extend our evaluation by comparing the performance of MTA-CLIP with alternative backbones, as presented in <ref>. Specifically, we utilize ImageNet <cit.> pretrained ViT-S <cit.> and MiT-B4 <cit.> backbones for this experiment. We demonstrate the versatility of our framework, indicating its applicability beyond Vision-Language Model (VLM)-trained backbones to also include ImageNet trained alternatives. In the case of ViT-S, DenseCLIP <cit.> based on pixel-text alignment achieves of 47.8% mIoU. Our MTA-CLIP framework, relying on Mask-Text Alignment, surpasses DenseCLIP by 3.6% in mIoU and also outperforms the strong baseline Mask2Former by 0.8%. Moreover, our framework exhibits effective compatibility with Mix Transformer backbones <cit.>, demonstrating a performance improvement of 1.4% compared to SegFormer <cit.>, which employs a similar backbone. Collectively, these results affirm the adaptability of our framework across diverse backbones.


 
 



Enhancing Mask with Textual Data. 
In this analysis (<ref>), we underscore the significance of enhancing mask representations with textual data from the pre-trained CLIP language model. Specifically, we modify the SelfAttn operation outlined in <ref> to ensure that the text embeddings t̂ and mask queries m do not attend to each other. Instead, individual SelfAttn operations are applied to the text embeddings t̂ and mask queries m, denoted as t̂ = SelfAttn(t̂) and m = SelfAttn(m). It is important to note that, in this experiment, only the SelfAttn operation is modified, while all other operations remain the same. The results reveal a performance decline of 0.5% when mask queries m are not attended by the text embeddings t̂. This observation shows the importance of attending mask queries with text embeddings for optimal performance.

Comparison between Contextual Prompts.

As discussed in <ref>, we employ two distinct strategies, namely SeparateNeg and MixNeg, to derive specialized text prompts aligned with the diverse contexts of masks for a given class. In this experimental analysis, we first compare the efficacy of the two strategies, SeparateNeg and MixNeg. Subsequently, we assess our framework's performance across different numbers of text prompts (K). Overall, both approaches show effectiveness (see <ref>), with MixNeg exhibiting a performance gain of 0.5%, compared to 0.3% achieved by SeparateNeg, over the baseline that utilizes a single prompt. Furthermore, our observations indicate that employing K=3, involving three learnable unique prompts shared across all classes, yields the optimal performance of 49.1% with MixNeg. Similarly, for SeparateNeg, the best performance is achieved with K=4 across varying values of K.

 
 










 §.§ Qualitative Analysis

We present the qualitative comparison of our framework in <ref>. We compare our framework with Pixel-to-Text Alignment method DenseCLIP <cit.> and mask-based semantic segmentation baseline Mask2Former <cit.> for ResNet-50 backbone on ADE20k dataset. Overall, our framework segments the classes better than DenseCLIP and Mask2Former. Additional results with ResNet-101 backbone and on Cityscapes dataset is in supplement (Sec. 2). 




§ CONCLUSION

This work introduces a novel framework, MTA-CLIP, designed for the semantic segmentation task, which implements innovative mask-level vision-language alignment. The proposed framework enhances mask representations by incorporating and aligning them with class-specific language features. Additionally, multiple context-specific prompts are employed to capture diverse representations of the mask segments. The framework is systematically evaluated using various backbones and pretraining strategies, consistently surpassing prior state-of-the-art approaches across diverse dataset benchmarks in the semantic segmentation task. We hope that this work stimulates further innovation in this challenging yet promising research direction.
splncs04




 Paper Title: Generalized Tampered Scene Text Detection in the era of Generative AI.



[
     R. Zamini[Corresponding author: rahelehzamini1361@yahoo.com] ^1,  S. Ghafouri^2  and F. Goodarzi^3

	^1  Department  of Mathematics, Faculty of Mathematical Sciences and  Computer, Kharazmi University, Thehran, Iran. 

	^2   Department of Mathematics, Faculty of Science, Arak University, Arak, 384817758, Iran.

^3   Department of Statistics, Faculty of Mathematical Sciences, University of Kashan, Kashan, Iran.

    
===================================================================================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
The rapid advancements of generative AI have fueled the potential of generative text image editing while simultaneously escalating the threat of misinformation spreading. However, existing forensics methods struggle to detect unseen forgery types that they have not been trained on, leaving the development of a model capable of generalized detection of tampered scene text as an unresolved issue. To tackle this, we propose a novel task: open-set tampered scene text detection, which evaluates forensics models on their ability to identify both seen and previously unseen forgery types. We have curated a comprehensive, high-quality dataset, featuring the texts tampered by eight text editing models, to thoroughly assess the open-set generalization capabilities. Further, we introduce a novel and effective pre-training paradigm that subtly alters the texture of selected texts within an image and trains the model to identify these regions. This approach not only mitigates the scarcity of high-quality training data but also enhances models' fine-grained perception and open-set generalization abilities. Additionally, we present DAF, a novel framework that improves open-set generalization by distinguishing between the features of authentic and tampered text, rather than focusing solely on the tampered text's features. Our extensive experiments validate the remarkable efficacy of our methods. For example, our zero-shot performance can even beat the previous state-of-the-art full-shot model by a large margin. Our dataset and code will be open-source.








§ INTRODUCTION


The rapid development of deep models sparks an generative AI revolution in computer vision, demonstrating remarkable progress in controllable editing <cit.>. However, the advancement of generative AI also facilitates the spread of malicious fake information on text images, posing serious risks to social information  security <cit.>. Consequently, the detection of AI-tampered text has become an vital topic in recent years <cit.>. It is crucial to develop effective methods for detecting AI-tampered text.

Recently, the Tampered-IC13 dataset <cit.> has been introduced to benchmark tampered scene text detection methods. Some promising methods have also been proposed for this task, such as frequency domain feature extraction <cit.> and masked image modelling pre-training <cit.>. Although significant progress has been achieved, the existing techniques are far from being adequate in real-world scenarios, and the reasons for this are analyzed as follows:

1) The inability of the existing Tampered-IC13 dataset to reflect model performance in real-world scenarios. There are numerous new text editing methods continuously emerging in this generative AI revolution, producing more and more realistic forgeries. However, all of the tampered texts in the Tampered-IC13 dataset were forged by the oldest text editing model, SRNet <cit.>, which is unlikely to be the real-world preference due to its relatively worse performance. Moreover, as the development of generative scene text editing is quite rapid, it is impractical to fit the forensics model to all types of the generative forgeries. Consequently, the ability to generalize across unseen tampering methods and unseen scenarios becomes an vital indicator in measuring the models' practicality in real-world scenarios. However, this open-set generalization ability totally cannot be evaluated by the Tampered-IC13 benchmark.

2) The challenging nature of the tampered scene text detection task. As constructively introduced by the pioneering work <cit.>, there are already two main challenges for tampered scene text detection. First, the limited size of high-quality training data. Existing text editing methods have a high probability of producing unsatisfactory outputs, requiring manual refinement in post-processing for visual consistency. Training models on the purely generated samples without manual refinement will lead them to overfit to the obvious visual inconsistency, thereby making them difficult to generalize to real-world forgeries. Second, fine-grained perception. After being generated by text editing models and further improved by human labour, the tampered scene texts are visually consistent with the authentic texts <cit.>. Only subtle texture anomaly may exist, and such subtle anomaly is difficult to capture. The above two challenges are far from being sufficiently solved in previous works.

3) The poor open-set generalization ability of existing tampered scene text detection methods. Forensics models trained on seen types of forgeries mostly suffer from significant performance degradation on unseen types of forgeries, this is well known in almost all related fields, such as face anti-spoofing <cit.>, deepfake detection <cit.>, image manipulation localization <cit.>. Undoubtedly, no exception for the tampered scene text detection. However, none of the previous works paid attention to the open-set generalization of tampered scene text detection methods, resulting in a huge gap between the real-world application.

To address these issues and to bridge up the gap between real-world scenarios, we propose the following techniques:

1) To address the shortcomings of the existing benchmark, we manually construct a comprehensive high-quality benchmark for open-set tampered scene text detection, termed as Open-set Scene Text Forensics (OSTF). The tampered text in our OSTF benchmark is tampered by eight text editing models, successfully covering up all three types of scene text editing methods (conventional deep models, font rendering model and diffusion models) and catching up with the recent development of generative AI, as shown in Figure <ref>. Except for the cross tampering methods test settings, our OSTF benchmark also includes the cross source data test settings, thus can better evaluate open-set generalization ability.

2) To simultaneously address the scarcity of high-quality training data, and facilitate the models' abilities of fine-grained perception and open-set generalization, we propose a novel pre-training paradigm termed as Texture Jitter. In this paradigm, we slightly change the texture of the randomly selected text regions with different operations to produce different types of texture anomalies, and train the model to localize the processed text. Since the proposed Texture Jitter does not change the macro appearance, it can be applied directly to any scenario and will always produce perfectly visually consistent results. Therefore the scarcity of high-quality training data can be significantly alleviated. In addition, since models trained with our Texture Jitter are forced to capture the subtle anomaly on the texture, their ability of fine-grained perception can be considerably improved. Moreover, the common characteristic of different text editing methods is that they cannot perfectly restore the original texture and always cause texture anomaly. By guiding the model to detect the texture anomaly rather than a specific type of tampering clue left by a particular tampering method, our Texture Jitter can also notably improve the model's open-set generalization. 

3) To further facilitate the models' ability of open-set generalization, inspired by the development of unsupervised anomaly detection <cit.>, we propose a novel pluggable framework termed as Difference-Aware Forensics. The key idea is to learn a compact, robust representation for authentic text, and identify whether the input text is tampered or not by comparing its features with the learned authentic representation. By paying attention to the feature differences rather than just the features of the input text, models can have less confusion on unseen forgeries.


We conduct extensive experiments on both the proposed OSTF benchmark and the Tampered-IC13 <cit.> benchmark. Our method demonstrates strong generalization ability in these experiments. For example, the proposed method leads to a gain of 27.88 mean F-score on the open-set generalization ability in the OSTF benchmark. Moreover, the zero-shot version of our method even outperforms the full-shot version of the previous SOTA method UPOCR <cit.> by 10.46 mean IoU on the Tampered-IC13 <cit.> benchmark.

In summary, the contributions of this paper are as follows:


  * We propose a novel task, open-set tampered scene text detection, to meet the crucial demands. We manually construct a comprehensive high-quality benchmark for it.

  * We propose a novel pre-training paradigm for tampered scene text detection. The proposed paradigm significantly mitigates the scarcity of high-quality training data, and notably improves the models' capabilities for both fine-grained perception and open-set generalization.

  * We propose a novel pluggable framework that further improves open-set generalization on unseen forgery.

  * In-depth analysis and extensive experiments have verified the effectiveness of the proposed method.




§ RELATED WORKS

Scene Text Editing. Since scene texts are sparse and have complex styles, various sizes, it is difficult to edit them to target content by copy-paste <cit.>. The actual scene text editing is achieved by neural networks. Existing scene text editing methods can be divided into three types. (1) Conventional deep models, which edit scene text in an E2E manner without utilizing diffusion models. SRNet <cit.> is the first model to achieve E2E scene text editing. STEFANN <cit.> proposed to edit scene text at char-level. MOSTEL <cit.> improved the visual quality of text editing with stroke-level masks and self-supervised learning on non-synthetic data. (2) Font-rendering based methods, which edit text with digital font files. DST <cit.> inpainted the original text region and rendered new text on it with the corresponding digital font file. This is very similar to the process of manually editing the target text using image processing software. (3) Diffusion-based methods, which leverage the power of diffusion models for realistic text tampering.  TextDiffuser <cit.> generated text images with the given prompts, while DiffSTE <cit.> manipulated the specific parts of the images with target texts. AnyText <cit.> introduced Auxiliary Latent Module for higher visual quality. UDiffText <cit.> improved scene text editing with large-scale training data and text embedding. The rapid development of generative scene text editing techniques brings huge challenges and risks to social security <cit.>. Therefore, it is essential to develop forensics models that can achieve open-set generalization on the text tampered by unseen methods.

Tampered scene text detection aims to localize tampered text on the given image. Wang. et al. <cit.> did the first work for tampered scene text detection, they proposed the first tampered scene text detection benchmark Tampered-IC13, they also proposed the S3R strategy and frequency domain modelling. However, this type of frequency domain modelling is likely to suffer from significant performance degradation on unseen types of forgery <cit.>. Qu. et al. <cit.> introduced Selective Tampering Synthesis and DTD to improve tampered text detection especially in documents. However, these methods are not suitable for scene text due to the board variety in sizes and appearance. They also introduced the DocTamper synthetic dataset for documents, but it does not contain any text tampered by AI model and is unable to benchmark model performance on AI-based tampering. Benefiting from more training data and more lenient evaluation metrics, Peng. et al. <cit.> achieved the highest mean F-score on the Tampered-IC13 benchmark by averaging real text detection scores. However, due to the lack of a specialized design, their performance for tampered text detection is still unsatisfactory. None of the existing work has explored open-set tampered scene text detection. To meet with the real-life requirements, we manually construct a comprehensive high-quality benchmark for open-set tampered scene text detection and propose novel simple-yet-effective method.



§ OSTF DATASET AND BENCHMARK


Motivation. In this era of generative AI, numerous new text editing models continuously emerge <cit.>. However, the existing benchmark Tampered-IC13 <cit.> only includes text tampered by the oldest text editing method SRNet <cit.>, which can hardly be a real-world preference due to its relatively inferior performance. Moreover, the ability to detect the text tampered by unseen text editing model and that on unseen scenario is essential for forensics model in this era of generative AI. However, this ability totally cannot be evaluated in the Tampered-IC13 benchmark. To address the above issues, we manually construct a comprehensive high-quality new benchmark for tampered scene text detection, termed as OSFT, which includes text tampered by various latest text editing methods and cross source-dataset evaluation settings.



 §.§ Dataset Construction

Tampering Methods. We take all the three types of generative tampering methods (conventional, font rendering, diffusion) into account, and select eight text editing methods, including SRNet <cit.>, STEFANN <cit.>, MOSTEL <cit.>, DST <cit.>, DiffSTE <cit.>, AnyText <cit.>, UDiffText <cit.>, Textdiffuser <cit.>, as shown in Table <ref>. Given that the Tampered-IC13 dataset <cit.> already has high-quality forgeries tampered by SRNet, we take it as our `SRNet` part.

Data Source. We forge the text images from the ICDAR2013 <cit.> with the selected eight text editing methods. To enable the cross source dataset evaluation, we further edit the text images from the TextOCR <cit.> validation set with UDiffText, and the text images from the ICDAR2017 <cit.>, ReCTS <cit.> validation sets with TextDiffuser.

Manual Improvement. To ensure the high quality of the tampered texts in our dataset, we edit all of the text instances from the selected source datasets using the selected methods, and manually pick up the most successful outputs. We further manually and elaborately improve the visual consistency of the picked texts using PhotoShop and GIMP. To ensure that the annotations are of high-quality. We manually update the bounding box for each tampered text to match its new appearance. We also manually check the labels via visualization to avoid errors.



 §.§ Dataset Statistics, Benchmark Settings, Highlights

Dataset Statistics. As shown in Table <ref>, there are a total of 5018 tampered texts and 1980 tampered images in our OSTF dataset. The detailed statistics of it are shown in Table <ref>. 

Evaluation Settings. 
As shown in Table <ref>, there are 9 sessions in our dataset (ICDAR2013 tampered by 7 methods, TextOCR tampered by UDiffText, ICDAR2017 and ReCTS tampered by TextDiffuser). To evaluate both closed-set performance and open-set generalization, the models are trained on the training set of 1 session and tested on the testing sets of all the 9 sessions. As a result, there are 9×9=81 test settings, enabling all the three evaluation protocols: cross tampering methods, cross source dataset, cross both of tampering methods and source dataset. 


Dataset Highlights. The comparison between our OSTF dataset and previous dataset is shown in Table 1. Some of the samples in our dataset are shown in Figure <ref>.
The main highlights of our dataset are as follows:

(1) Comprehensive. Our dataset includes all of the three types of generative text editing methods, keeping pace with the generative AI revolution. Our benchmark includes both the cross-AI-model and cross-dataset evaluation settings.

(2) High Quality. The tampered texts in our OSTF dataset are manually selected and elaborately manually enhanced for better visual consistency. The bounding boxes are manually adjusted to match the tampered texts.

(3) Board Diversity. The images in our OSTF dataset have various styles and resolutions, the texts in these images have various fonts and backgrounds, and are tampered by 8 different generative methods.













§ PRE-TRAINING WITH TEXTURE JITTER


Motivation. With optimal text editing methods, we can produce abundant high-quality tampered scene text. However, current generative text editing methods often generate noticeable visual inconsistencies <cit.>. This is especially evident when dealing with complex fonts, unfamiliar languages, and diverse styles. Failure to generate good tampered text results in model's overfitting to obvious visual inconsistency. Consequently, trained models struggle to extend their applicability to real-world scenarios where manipulated text's appearance is refined through human intervention and has minimized visual inconsistency. 

Method. Based on the above observations, we propose a simple-yet-effective method called Textual Jitter, that is, just slightly changing the texture of randomly-picked texts while keeping their macro appearance unchanged (the processed texts are almost the same as the original ones). Once the processing is done, we train the models to localize the text regions and to identify whether their texture has been changed or not. Without changing the macro appearance, this approach can be applied directly to any text image and can always produce high-quality training data that is visually consistent and similar to the elaborately tampered text. Importantly, our proposed Texture Jitter can simultaneously address multiple major bottlenecks in tampered scene text detection, including data scarcity, fine-grained perception, and open-set generalization, by simply applying our Textual Jitter to the existing text detection datasets.

Pipeline. The pipeline of our Texture Jitter is shown in Figure <ref>. Given an input text image, we randomly select some text instances, and apply a random texture processing operation or the combination of multiple operations. The texture processing operations we consider include random blur, reverse blur, random image compression, reverse image compression. 
By doing this, various types of texture anomalies are created. The processed text instances are then spliced into the same positions as the original ones, with a smooth transition at the edges to avoid obvious visual anomalies. Moreover, to ensure a balance between the learning difficulty and a natural fusion between the original and processed regions, we also propose to adaptively adjust the intensity of the Texture Jitter and the edge smoothing with rules based on the size of the target text. More details are given in the appendix. The full implementation will be open-source. With the proposed adaptive intensity, our Texture Jitter can always produce satisfactory outputs.



Despite its simplicity, the clear advantages and interpretability of the proposed Texture Jitter are manifold, simultaneously addressing all of the three major issues for tampered scene text detection:

(1) The scarcity of high quality training data is significantly alleviated. The main reason for the data scarcity issue is the difficulty to always maintain visual consistency with existing text editing models. However, our Texture Jitter can always automatically output visually consistent text with only obscure texture anomaly. Therefore, our Texture Jitter can be directly applied to any scenario and foreign language and will always produce diverse high-quality data, successfully addressing the data scarcity issue. 

(2) The ability of fine-grained perception is considerably improved. The text processed by our Texture Jitter is visually consistent with other texts and its anomaly is obscure and difficult to capture. By training forensics models with our Texture Jitter, the models' ability to detect the subtle anomaly can be continuously improved.

(3) The ability of open-set generalization is notably lifted. The common characteristic of different text editing methods is that all of them cannot perfectly restore the original texture. After being trained with our Texture Jitter, the model learns to identify tampered text by judging whether the texture is abnormal, rather than relying on a specific feature of a tampering method, and thus achieves better open-set generalization on unseen forgery.





§ DIFFERENCE AWARE FORENSICS


Motivation. Forensics models often suffer from significant performance degradation on unseen forgeries <cit.>. Especially in this era of generative AI, advanced generative models are increasingly emerging and they can edit the texts in images in the ways that the forensics models have never seen during training. Therefore, constructing a robust forensics model that generalizes across unseen forgeries is critical in real-life scenarios.

Analysis. The performance degradation on unseen forgeries is mostly caused by the training objective, a common binary classification task <cit.>. As shown in the top left of Figure <ref>, during the training process, the models simply learn the specific features for the seen authentic class (blue circle) and the tampered class (yellow circle). When text is tampered by an unseen text editing method, the editing styles are never seen before and thus the features of the tampered text (red circle) are different from the seen ones. Consequently, the classifier is confused by new features from unseen forgeries, leading to poor performance.

Key idea. Although diverse text editing models will produce various tampering features, the features extracted from the real text should remain similar. Moreover, the features of the tampered text can be regarded as anomalies since they are different from those of the authentic text. Inspired by this, we propose to model the difference between the input feature and the authentic feature for text forensics, instead of just relying on the individual input feature, as shown in Figure <ref>. Both the seen and unseen forgery features can be distinguished from the authentic feature, thus the confusion caused by the unseen forgery can be alleviated.

Solution. Specifically, we propose a novel framework DAF, as shown in Figure <ref>. Our DAF builds on the widely used detection models (e.g., Faster R-CNN <cit.>) by simply adding a forensics branch, and is trained and tested in an E2E manner. The forensics branch shares the same backbone model as the original detection head, and extracts features for real/fake classification with an extra FPN network <cit.>. This follows the S3R <cit.>, which performs text localization and forensics with separate networks. In the forensics branch, an authentic kernel is learned by pulling together the features of authentic text and pushing away the features of tampered text, this is supervised by L2 loss during training. More details are given in the appendix. This authentic kernel is approximately equal to the average of the features extracted from the real texts in the training set, and it can be regraded as the common representation for the real texts. For each input image, the learned authentic kernel is modulated by the global representation of the image with a linear layer to adapt to the image style. We subtract between the modulated authentic kernel and the RoI feature vectors. Finally, the subtracted features are fed into the classifier for final prediction.

























§ EXPERIMENTS



Implement Details
We conduct experiments on both the well-acknowledged Tampered-IC13 benchmark <cit.> and our proposed OSTF benchmark. We first pre-train our model for 12 epochs with the proposed Texture Jitter on  the same training sets as UPOCR <cit.>, including LSVT <cit.>, ReCTS <cit.>, ICDAR2013 <cit.>, ICDAR2015 <cit.>, ICDAR2017 <cit.>, TextOCR <cit.>, ArT <cit.>. The AdamW optimizer <cit.> with a learning rate initialized at 6e-5 and decaying to 1e-6 is used in the experiments. We then fine-tune the model with the training sets of the Tampered-IC13 <cit.> and the OSTF dataset respectively for 15k iterations with a batch-size of 8. We adopt Swin-Transformer (Small) <cit.> as the backbone model following the previous work <cit.>. Input image is resized to ensure that the shortest edge ≤ 1024 and the longest edge ≤ 1536.


Evaluation Metric.
For a fair comparison with the segmentation-based methods, we convert the bounding box predictions of our model into segmentation maps (detailed in the appendix) and calculate the Precision (P), Recall (R) and F1-score (F) at the pixel-level with the built-in functions of mmsegmentation <cit.>  following the previous works <cit.>. For a fair comparison with the detection-based methods, we adopt P, R, F calculated at the instance-level with the official scripts provided by the Tampered-IC13 <cit.>. We use these detection-based metrics for the ablation experiments.



 §.§ Comparison Study

Comparison with the segmentation models. The comparison results on the T-IC13 benchmark <cit.> are shown in Table <ref>. Evidently, even the zero-shot versions of the our models notably outperform the full-shot version of the previous SOTA model UPOCR <cit.> by more than 10 points mIoU. The `zero-shot` means that only the pre-trained model is used, which has never seen any tampered text. The strong zero-shot generalization ability of our models is attributed to the effectiveness of our methods. Our full-shot models achieve higher performance.

Comparison with the detection models. The comparison results are shown in Table <ref>, the zero-shot versions of our models also outperform the full-shot version of the state-of-the-art (SOTA) model by about 3 points mean F-score. The full-shot versions of our models further significantly outperform the state-of-the-art model (e.g. the full-shot version of `Ours+Cascade R-CNN` achieves 88.62 mean F-score, about 12 points higher than the SOTA of 76.66), demonstrating considerably strong generalization ability.






 §.§ Ablation Study
 
The ablation results on the OSFT dataset are shown in Table <ref> and Figure <ref>. In Table <ref>, the `mP`, `mR`, `mF` are calculated by averaging the P, R, F of all the 81 test settings in the OSTF dataset. In Figure <ref>, the four matrices are from the ablation settings (1), (3), (7), (8) in Table <ref>, respectively.

In Table <ref>, settings (1), (3), (5) and (2), (4), (8) are designed to verify the effectiveness of our Texture Jitter paradigm. The model pre-trained with our Texture Jitter (setting (5)) gets 71.96 mF-score on tampered text detection, 26.18 points higher than the baseline pre-trained with the same training configuration but only with the common text detection task (setting (3)), and even 37.14 points higher than the baseline initialized with the official COCO detection pretrained weights (setting (1)). Similarly, huge improvement on Cascade R-CNN <cit.> can also be observed by comparing between settings (2), (4), (8). This demonstrates the surprising effectiveness of the proposed Texture Jitter. Settings (5), (6), (7) are designed to verify the effectiveness of our DAF framework. By adding single-center-loss <cit.> to setting (5), setting (6) gets only tiny improvements. However, by equipping setting (5) with our DAF framework, setting (7) achieves much larger improvements. Setting (7) and setting (8) show that our methods are robust to different base detectors, and that a better base detector can mostly lead to better performance.

In Figure <ref>, each matrix has 9×9=81 cells, corresponding to the F-scores of the 81 test settings on the OSFT dataset. Different rows of the matrices denote different training sets and different columns denote different test sets. Darker color indicates better performance. In Figure <ref>, the top-left matrix is the result of the baseline model (setting (1) of Table <ref>), this matrix shows an obvious diagonal, indicating that the common model only performs well when tested on seen forgeries. For example, in the top-left matrix, the model trained on the text tampered by `STEFANN` gets 85.14 F-score when tested on `STEFANN` (row 2, column 2), but its F-score drops to 0 when tested on the unseen forgery tampered by `SRNet` (row 2, column 1). This poor open-set performance issue is not addressed when the model is pre-trained on common text detection task with the same datasets and configuration as ours but without our Texture Jitter (setting (3)). In contrast, the two matrices at the bottom of Figure <ref> are the results of our methods, these matrices have much less obvious diagonal, much more uniform and darker colors, indicating that the models trained with our methods gain much better open-set generalization ability. The qualitative results for visual comparison are shown in Figure <ref>, which further validate the proposed methods.

The ablation results on the Tampered-IC13 dataset are given in the appendix, where both of our Texture Jitter and DAF framework contribute towards higher performance. 

Discussion. In Figure <ref>, models have relatively worse open-set performance when being tested on `STEFANN` and `TextOCR`, for the following reasons: (1) For STEFANN, there is a huge difference in texture appearance between the output of STEFANN and other editing methods. Texts edited by STEFANN have almost binary texture appearance, while texts edited by other methods have richer and more realistic texture details (Figure <ref>). Therefore,  generalization is much more challenging. (2) For TextOCR, it has many tiny and fuzzy texts that are challenging to detect (Figure <ref>). When models are trained on other source data, they are adapted to big clear text and thus have performance degradation on such tiny and fuzzy texts.
The tiny and fuzzy texts are also the main reason for false positives in other subsets (Figure <ref>). Despite these challenges, our proposed methods still significantly improve the performance (according to Figure <ref>, 33.07 points improvement for `STEFANN` and 28.31 points improvement for `TextOCR` on average).






§ CONCLUSION

In the era of the generative AI, this paper introduces a novel task of open-set tampered scene text detection, designed to meet the demands for generalized forensics analysis. This task challenges forensics models with both seen and previously unseen forgeries, aiming to thoroughly evaluate their generalization. To facilitate this, we have manually developed a comprehensive high-quality benchmark, named OSTF, which stands out by including text images tampered by various editing models. Further, we introduce an innovative pre-training paradigm, termed Texture Jitter, which effectively mitigates the lack of high-quality training data and significantly enhances the models' capabilities in fine-grained perception and open-set generalization. The proposed Texture Jitter is the first fine-grained perception pre-training paradigm, which differs from previous works that are coarse-grained perception. To further enhance open-set generalization, we also present a novel pluggable framework DAF, which focuses on identifying the feature difference between authentic and tampered texts. This differs from previous works that rely only on the specific input features. Extensive experiments validate the effectiveness of our methods. With these advances, our work achieves a significant step forward in the field of tampered text detection.





 Paper Title: GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models.










2676 















A]Ali Abdollahi
[Equal contribution.]

A]Mahdi 	Ghaznavi

A,D]Mohammad Reza Karimi Nejad
 
A]Arash Mari Oriyad
A]Reza Abbasi
A]Ali Salesi
B,C]Melika Behjati
A]Mohammad Hossein Rohban


A]Mahdieh Soleymani Baghshah




[A]Sharif University of Technology
[B]ecole polytechnique fédérale de lausanne (EPFL)
[C]IDIAP research institute
[D]Shamsipour Technical and Vocational College



§ ABSTRACT 
Vision-language models (VLMs) are intensively used in many downstream tasks, including those requiring assessments of individuals appearing in the images. While VLMs perform well in simple single-person scenarios, in real-world applications, we often face complex situations in which there are persons of different genders doing different activities. We show that in such cases, VLMs are biased towards identifying the individual with the expected gender (according to ingrained gender stereotypes in the model or other forms of sample selection bias) as the performer of the activity. We refer to this bias in associating an activity with the gender of its actual performer in an image or text as the Gender-Activity Binding (GAB) bias and analyze how this bias is internalized in VLMs. To assess this bias, we have introduced the GAB dataset with approximately 5500 AI-generated images that represent a variety of activities, addressing the scarcity of real-world images for some scenarios. To have extensive quality control, the generated images are evaluated for their diversity, quality, and realism. We have tested 12 renowned pre-trained VLMs on this dataset in the context of text-to-image and image-to-text retrieval to measure the effect of this bias on their predictions. Additionally, we have carried out supplementary experiments to quantify the bias in VLMs’ text encoders and to evaluate VLMs’ capability to recognize activities. Our experiments indicate that VLMs experience an average performance decline of about 13.2% when confronted with gender-activity binding bias.













§ INTRODUCTION



Nowadays large multi-modal models have shown tremendous potential in various tasks, from information retrieval systems to image captioning, visual question answering, image generation, and visual reasoning.
As a branch of multi-modal models, Vision-Language Models (VLMs) <cit.>
that provide a shared cross-modal embedding space between text and image modalities, 
have been intensively investigated by researchers recently and are used in many real-world applications <cit.>. 








Despite their wide range of success in downstream tasks, VLMs are subject to many critical biases, which can consequently affect their performance, especially in sensitive applications. For example, <cit.> provides a comprehensive analysis of biases and their implications in the CLIP Models, revealing how this model is exposed to gender biases and how challenging it is to trust it in real-world applications.
As <cit.> and <cit.> state, the bias is introduced in visual-linguistic pre-training due to unfairness in the training data, and additionally, at inference time, the visual and linguistic contexts that is used for few-shot applications can also promote bias. 





One important type of bias, which is highly discussed in the literature, is the gender bias. Gender bias demonstrates the undesirable association of a factor with a gender according to the model. For instance, due to the high occurrence of data in which men are repairing devices, this activity is usually associated with men by deep models, which leads to failure when faced with women repairing devices.
 <cit.> suggests that different VLMs do not perform equally well on determining gender for an occupation, and do not assign equal retrieval likelihood to images of male and female professionals. Also, <cit.> and <cit.> show that CLIP <cit.> shows biased behavior in retrieval based on gender-neutral queries. Please note that, in line with common practice in the literature, our focus is on two genders, masculine and feminine. This approach streamlines the process of generating the dataset and analyzing the experiments.



In this research, we delve into the issue of gender-activity binding in retrieval tasks. We aim to retrieve the corresponding caption or image from two given captions or images, each depicting an activity performed by a different gender, based on a provided image or caption that shows one of the genders performing the activity. We refer to this bias in associating gender and activity as the Gender-Activity Binding (GAB) bias. This bias arises due to ingrained gender stereotypes in the model or other types of sample selection biases <cit.>. Our research indicates that VLMs do not display a substantial bias in retrieval tasks when both text and image modalities depict only one gender. This is because the identification of the performer’s gender can directly lead to the correct output, eliminating the need to bind the performed activity with the performer’s gender. Though, text encoders have shown a considerable bias towards the expected gender. However, The bias becomes more pronounced when the scene is more complex, i.e., when two individuals of different genders are present in the image. We observe a drop in retrieval accuracy in scenarios where the activity performer is the unexpected gender. 
For example, in the context of the “repairing” activity, the retrieval accuracy of VLMs in identifying the performer decreases when a woman is repairing a device and a man is also present in the scene. This is in comparison with scenarios where the performer is a man, or there is no man present in the scene to associate the activity with him.

The significance of this bias escalates when we consider its underlying implications. Not only does it perpetuate societal biases and fairness concerns related to gender stereotypes, but it can also lead to serious complications if incorporated into judgement and decision-making systems. The potential for such bias to inadvertently influence outcomes underscores the importance of addressing it.

















































We have created a dataset known as the Gender-Activity Binding (GAB) dataset as shown in Figure <ref>(a). This dataset comprises images depicting various activities performed by men or women in different contexts (with or without the presence of an individual of the opposite gender), each accompanied by a descriptive caption.


Given that our scenarios are uncommon due to biases and stereotypes, suitable real-world images are scarce for evaluation purposes. To address this issue, all images are generated using DALL-E 3[https://openai.com/dall-e3]. We employ extensive prompt enhancement techniques to ensure the diversity and quality of the generated images. We also evaluated the generated images on diversity, quality, and realism. We performed multiple quantitative assessments consisting of Fréchet Inception Distance (FID) for both quality and diversity assessment, Structural Similarity Index (SSIM) 
and Learned Perceptual Image Patch Similarity (LPIPS) for diversity assessment, and human feedback for a final qualitative assessment to ensure the final results accurately represent the intended activities and their related contexts.

The activities included in GAB dataset are sourced from various places to encapsulate societal gender stereotypes and other activities that are statistically biased towards a specific gender, as observed in the LAION-400M dataset <cit.>.






We benchmark the performance of 12 renowned multi-modal foundation models and assess their performance in terms of text-to-image retrieval, image-to-text retrieval, and recognizing activities on GAB dataset as shown in Figure <ref>(b). 
Specifically, in the image-to-text retrieval task, we observed that while an unexpected gender is doing a stereotypically biased activity, the average performance of VLMs declined by approximately 33.2% due to the presence of the expected gender in the scene. Moreover, most VLMs undergo an average accuracy reduction of approximately 13.2% when encountering gender-activity binding bias. On the other hand, in the text-to-image retrieval task, most VLMs achieve an accuracy of approximately 50%, indicating that their performance is nearly random, and they fail to recognize the performer of the activity based on the text.


















As far as we are aware, this research is the first to delve into the origins of the association between genders and activities in vision-language models by conducting a thorough analysis both image-to-text and text-to-image retrieval accuracy of VLMs. It encompasses not only straightforward scenarios involving single individuals, where VLMs demonstrate high accuracy for both genders <cit.>, but also more complex situations with individuals of different genders <cit.>. This comprehensive approach allows us to isolate the impact of the complexity of the scenario on performance decline <cit.> and more accurately measure the Gender-Activity Binding bias. Additionally, the study carries out supplementary experiments to investigate the bias in VLMs’ text encoders and their proficiency in recognizing activities.



To summarize, our contributions are as follows:

    
    
  * We have created a novel dataset, known as the gender-activity binding dataset. This dataset includes AI-generated images of various activities that are typically associated with a specific gender, and are excellent in terms of diversity, quality, and realness (Section <ref>).

    
    
    
    
    
    
    
    
    
    
  * We conduct an intensive performance benchmark of well-known vision-language models in retrieval tasks to assess their robustness against the Gender-Activity Binding bias. Our findings reveal that when two individuals of different genders are present, VLMs exhibit a bias towards binding the activity with the gender that is expected to perform it, resulting in a 13.2% drop in retrieval accuracy. However, in scenarios involving a single individual, these models demonstrate high image-to-text retrieval accuracy. We also demonstrate that VLMs lack the ability to bind gender and activity in text-to-image retrieval tasks (Section <ref>). 

    
  * We carry out additional experiments to investigate the binding bias in VLMs’ capability to comprehend activities and the bias present in their text encoders (Section <ref> and the Appendix). We delve into the insights that can be derived about how VLMs internalize this bias and its impact on the shared embedding space of VLMs (Section <ref>).

    
    







§ RELATED WORK







 §.§ Vision-Language Models

VLMs apply contrastive training to bridge the image and text embedding spaces. Using this shared embedding space, VLMs have exhibited a remarkable performance on a notable number of multi-modal downstream tasks such as image captioning, text-to-image and image-to-text retrieval, visual question answering, and text-to-image generation. As <cit.> and <cit.> mentioned, VLMs can be categorized into three main groups based on their architectures: I) Fusion VL-encoders such as FLAVA <cit.> and LXMERT <cit.>, II) Dual-stream encoders like CLIP <cit.>, GLIP <cit.>, and FILIP <cit.>, and III) VL Encoder-decoder including BLIP <cit.>.



 §.§ Gender-activity Binding Bias

Previous works discussed that VLMs have many categories of biases <cit.>, especially gender-related ones <cit.>. 

For example, <cit.> studied bias in VLMs as a task of measuring associations between entities and gender in visual-linguistic models using template-based masked language modeling. Moreover, <cit.> measured gender bias in VLMs by comparing the MASK token prediction probabilities of factual and counterfactual samples and concluded that VLMs inherit human stereotypes from the training data.

On the other hand, there exist a number of studies including a restricted assessment of how a VLM understands a particular activity <cit.>. For example, <cit.> has introduced a small dataset called Winoground containing objects, attributes, and activities to evaluate the ability of VLMs in compositional reasoning. Moreover, <cit.> designed VL-CheckList framework to study the capability of VLMs in understanding objects, attributes, and relations containing activities. However, to the best of our knowledge, there is no systematic study on the ability of VLMs in associating an activity with a gender.













§ GENDER-ACTIVITY BINDING BIAS






Despite the recent increase in large-scale data collection, it’s important to note that these datasets are not necessarily devoid of biases. These biases can manifest in large datasets derived from real-world distributions, primarily due to sample selection bias.

The issue causes VLMs to have varying levels of accuracy in recognizing the performer of an activity depending on the gender of the individual involved <cit.>. 
We refer to this bias in associating an activity with the gender of its actual performer in an image or text as Gender-Activity Binding Bias. 


Consider an image-to-text retrieval task, where we have two pieces of text C_1 and C_2, each describing an activity performed by a different gender. We also have an image I_1 depicting a person (either a man or a woman) performing an action. The task is to match the image with the correct piece of text (Figure <ref>-(a) left). Also, consider the text-to-image retrieval task, where we have two images I_1 and I_2, each showing an activity being performed by a specific gender, and we want to retrieve the correct image based on a given piece of text C_1 (Figure <ref>-(a) right). The Gender-Activity Binding bias manifests as a decline in the retrieval accuracy of VLMs when they incorrectly identify the gender of an individual performing activities typically associated with a different gender.


Let's denote the probability of a VLM correctly identifying the performer of an activity in a data i (image or text) as p, the activity as a, and the gender as g. Ideally, the probability p(g|a, i) should be equal for all g. 
However, because of the co-occurrence of gender g of the performer and the activity in the dataset, which is referred to as sample selection bias <cit.>, the gender of the performer becomes correlated with the activity. Thus, we often find that p(g_1|a, i)> p(g_2|a, i), where g_1 and g_2 represent different genders and g_1 shows the expected gender for the performer of a. It is also 
In simple terms, VLMs tend to incorrectly select the text or image because they are biased towards associating the activity with a specific gender. This is not always accurate, leading to errors in scenarios where the activity is performed by the less expected gender. 












The accuracy of models in image and text retrieval tasks is assessed in two scenarios: one where only a person of an unexpected gender (in terms of the bias) is depicted or described in the text or image, and another where both genders are present in the data, but the activity is performed by only one of them. Consequently, the impact of the bias can be observed as a decrease in accuracy in two situations: when the expected gender is absent or present, and when the activity is performed by the unexpected or expected gender. These cases together provide better insights into how these models handle gender-activity associations under different conditions. The first scenario allows us to understand how well the models can recognize and correctly associate an activity with an unexpected gender when the other gender is not present. This helps us gauge the models’ ability to break away from societal stereotypes and biases. The second scenario, on the other hand, tests the models’ ability to correctly identify the performer of an activity when both genders are present, which is a more complex and realistic situation. This indicates these biases affect the models’ performance in real-world scenarios. Together, these cases provide a comprehensive understanding of the models’ strengths and weaknesses in handling gender-activity binding, paving the way for targeted improvements to mitigate the gender-activity binding bias.






































































§ DATASET




This section details the creation process of the Gender-Activity Binding (GAB) Dataset, which serves to evaluate various VLMs against the gender-activity binding bias we have identified and VLMs ability to bind activities and performers. This dataset comprises AI-generated images that illustrate a variety of activities with potential subjects of both genders.





The GAB dataset divides images into four distinct groups: In two groups, both genders are present in the scene. In one of these groups, the expected gender is performing the activity while the other gender is just present in the image, and in the other group, the unexpected gender is performing the activity while the other gender is also in the scene.
In the next two groups, only one gender is present in the scene. In one of them, the expected gender is performing the action, and in the other one, the scenario is reversed, with the unexpected gender acting. 

For each activity, with respect to these grouping, we can consider four groups based on the performer of the activity (expected (E) or unexpected (U)) and the number of persons (with different genders) in the scene (1 or 2): E1, E2, U1, U2.






For each image in the dataset, we have templates that are replaced with activity or gender names, such as "man" and "woman," in accordance with the designed experiments. Details of the experiments and these replacements can be found in Section <ref>. The template for images containing two genders is: "a man/woman is doing activity and a woman/man is in the scene" and for images containing one gender, the template is: "a man/woman is doing activity". This specific template can also be used for images with two genders.





Each of the mentioned groups contains at least 26 images for each activity, resulting in a total of 5500 images.






This dataset structure facilitates the evaluation of gender-activity binding bias and the ability of VLMs to bind and recognize activities by definition of different retrieval tasks, as you can see in Section <ref>.

Pipeline of the dataset creation is shown in Figure <ref>(a)




 §.§ Selection of Activities




We proposed three methods to identify the biased activities, all utilizing Masked Language Modeling (MLM) as part of their approach to detecting biased activities. We utilized the 'RoBERTa-large' <cit.> model for its superior performance in masked token prediction, which was attributed to its objective function.

The sentences are formatted as "a mask is doing activity".[mask is the special mask token of the RoBERTa model] We then calculate the probabilities associated with "man" and "woman"
(as well as related masculine and feminine identifiers like "boy" and "girl"), from which we calculate the log ratios, as specified in Section <ref>.



  §.§.§ Stereotyped Activities from GPT Combined with MLM


Our first method involved using GPT-4 to request activities stereotypically associated with men or women in society. We provided over 100 activities from GPT-4 and formed sentences by replacing the activity token in the template of 'a mask is doing activity'. We then inputted these sentences into the RoBERTa model and calculated the following equation for the sentences.

    bias(sentence) = log(P(mask = man|sentence)/P(mask = woman|sentence))

We applied this methodology separately for activities stereotypically associated with each gender. For men, this resulted in a normal distribution[all the tests including log ratios formed normal distributions that we tested using Shapiro–Wilk test for normality] with a positive mean (μ > 0), from which we chose the sentences at the rightmost end. Similarly, for women, it also resulted in a normal distribution but with a negative mean (μ < 0), from which we then chose the sentences at the leftmost end.




  §.§.§ Everyday Activities from GPT Combined with MLM


In our second approach, we used GPT-4 to compile a list exceeding 1,000 sentences related to hobbies, jobs, and everyday activities. We then masked the subjects of the sentences just like in the first method and asked the RoBERTa model to fill the mask. We stored the log ratios of predicted probabilities. The log ratios formed a normal distribution, from which we selected sentences at each end.




  §.§.§ Activities from LAION-400M Dataset Combined with MLM


In the third method, we used the LAION-400M <cit.> dataset, which consists of 400 million image-caption pairs. We filtered out the NSFW pairs and sampled 20 million pairs randomly. We then extracted the verbs, subjects, and objects of all caption sentences using spaCy <cit.>, a natural language processing toolkit. We counted the number of masculine and feminine subjects for each verb-object pair and calculated their log ratios. This also formed a normal distribution.

We then used the Z-test to determine whether each verb-object pair had a significantly different ratio compared to the mean of all verb-object pairs. This test is appropriate due to the large dataset size of LAION-400M. By checking if differences are statistically significant, we identify biased activities. This method enhances the reliability of our activities by ensuring they are not a result of random variance or outliers. Then, we created sentences with the biased verb-object pairs as mentioned in Section <ref>, masked the subject, and submitted them to the RoBERTa model to fill the mask, conducting the same experiment as in the previous methods to select the biased activities.




















 §.§ Generation


The generation process consisted of two phases. The first involved generating a good prompt that helped the image generation system meet the criteria given below, derived from the base sentence. The second involved generating the image from the prompt. The generated images should accurately represent the original activity, look natural and high quality, and be diverse.


  §.§.§ Prompt Enhancement


We used a Large Language Model (LLM) to generate diverse prompts while also keeping the original activity intact (Details about the utilized LLM are provided in the Appendix.). We then created a list of predefined settings like the environment of the house, the skin color of the person, etc. Although we could have sampled N prompts from each setting to generate diverse images, we chose to create a list of different combinations of these settings and encode them using MiniLM <cit.> sentence transformer <cit.>. We then clustered them into K[K = 6 which resulted in sufficient diversity in our experiments] clusters and sampled N/K samples from each cluster to enforce more diversity semantically. We then generated a diverse prompt by instructing Llama2 to generate diverse prompts that maintain the original activity while describing the environment and the settings using the selected samples. Using these prompts, the generated images were significantly improved in quality and detail.



  §.§.§ Image Generation

We initially experimented with several diffusion models, as mentioned in the Appendix, to generate images. The generated images were realistic and diverse. However, they lacked representation of the intended activity, especially in the context of compositional generation. We then explored DALL-E3 <cit.> using OpenAI API. This approach resulted in significant improvement to the quality of compositional generation while also keeping the reality and the clarity of the image. We chose DALL-E3 as our final image generation solution.




 §.§ Image Filtering Methodology

Our image filtering methodology employs a combination of automated metrics and human evaluation to ensure the generated images are of high quality, relevant, and diverse. The process comprises three steps: quality assessment, diversity metrics, and human feedback, detailed as follows:



  §.§.§ Quality Assessment

The quality of generated images was assessed using the Fréchet Inception Distance (FID) <cit.> alongside criteria emphasizing the representation of human figures, as identified by the YOLOv8 <cit.> and MTCNN <cit.> models. The FID score aims to measure the similarity between the generated images and a set of reference images from the COCO <cit.> dataset, calculated as:


    FID(x, y) = ||μ_x - μ_y||^2 + Tr(Σ_x + Σ_y - 2(Σ_xΣ_y)^1/2),

where μ_x, μ_y are the feature-wise mean of the real and generated images, and Σ_x, Σ_y are the covariance matrices of the real and generated images, respectively. To achieve high-quality images, we initially filtered out images with high Fréchet Inception Distance (FID) scores. After this filtering process, we successfully reduced the mean FID score to 11.9. Furthermore, for an image to pass our quality assessment, individuals must constitute at least 15% of the detected objects and must have a detectable face, as confirmed by the MTCNN model. 
It’s worth mentioning that the FID scores for both expected (E1 and E2) and unexpected (U1 and U2) categories of images are 12.1 and 11.7 respectively, signifying an acceptable level of quality across all scenarios.



  §.§.§ Diversity Metrics



To ensure image diversity, we employed the Structural Similarity Index (SSIM) <cit.> and the Learned Perceptual Image Patch Similarity (LPIPS) <cit.> metrics. The SSIM index measures the similarity between two images; in our context, lower scores indicate greater diversity. It is defined as:


    SSIM(x, y) = (2μ_xμ_y + c_1)(2σ_xy + c_2)/(μ_x^2 + μ_y^2 + c_1)(σ_x^2 + σ_y^2 + c_2),

where μ_x, μ_y are the average intensities, σ_x^2, σ_y^2 are the variances, and σ_xy is the covariance. c_1 and c_2 are constants added to stabilize the division and avoid numerical instability with small denominators. Our target mean SSIM threshold across all image pairs is 0.046. It’s also worth mentioning that the SSIM for the expected (E1 and E2) and unexpected (U1 and U2) categories of images are 0.046 and 0.045, respectively.
This indicates a satisfactory level of diversity in both image categories.

LPIPS measures the perceptual similarity between two images, using deep network features to closely mimic human visual perception. To indicate greater diversity, we aim for higher LPIPS values, setting the mean target for all pairs at 0.66. Also, the LPIPS scores for the expected (E1 and E2) and unexpected (U1 and U2) categories of images are 0.66 and 0.65, respectively.





  §.§.§ Human Feedback

After quantitative evaluations, we performed a qualitative analysis with human feedback. This involved scrutinizing images for alignment with the desired activities and contexts. Images that did not align with the activities or did not depict the intended subjects were removed. Additionally, images with unrealistic or nonsensical elements, and those appearing abnormal to an external observer were excluded.

Following this step, a list of images that have passed both quantitative and qualitative evaluations is obtained. As a result, the images are diverse, high-quality, and realistic while also meeting our criteria for the generation phase.






§ EXPERIMENTS





























 We designed several experiments on our proposed GAB dataset to evaluate Vision-Language Models in text-to-image and image-to-text retrieval tasks as shown in Figure <ref>(b). Based on these experiments, we assessed the defined gender-activity binding bias and the ability of VLMs to recognize gender-biased activities. Finally, we evaluated the bias in the text encoders and image encoders of different VLMs separately.






 §.§ Setup





  §.§.§ Task Definitions

The experiments we have designed to assess the performance of VLMs on the aforementioned aspects are grounded in their zero-shot performance on both text-to-image and image-to-text retrieval tasks.


 






More formally, consider the space of images as ℐ and the space of captions as 𝒞.

 
Also consider a VLM with an image encoder E_𝒞:𝒞→ℝ^D and a text encoder E_ℐ:ℐ→ℝ^D, that map images and texts respectively to a shared embedding space, which is shown in the middle box in Figure <ref> (b). We can define matching score function s:ℐ×𝒞→ℝ^+, which measures the similarity between a caption and an image, as the cosine similarity between the embedding of an image I and the embedding of a caption C:


    s(I, C)=E_ℐ(I).E_𝒞(C)/‖ E_ℐ(I) ‖‖ E_𝒞(C) ‖.


Text-to-image retrieval involves finding the image that best matches a given text from a collection of images. Conversely, image-to-text retrieval is about finding the text that best describes a given image. We compute the similarity using the previously defined similarity measure in the upcoming experiments. For the text-to-image retrieval task, we have a caption C and make comparisons between two images: I, which shows an image corresponding to the caption C, and I^R which corresponds to C^R (obtained by reversing the gender of subjects in C). Similarly, for the image-to-text retrieval task, we have an image I and make comparisons between two captions C and C^R. 









  §.§.§ Selected Vision-Language Models



In our experiments, we benchmark 12 VLMs and present the results of various tests.



For comparing the effects of patch size and backbone size, we report the results for 4 CLIP models released by OpenAI <cit.>, each with different backbones and patch sizes.


Another model selected for evaluation is NegCLIP <cit.>, a version of the CLIP-ViT-B-32 model by OpenAI <cit.> which is fine-tuned on a modified subset of the Visual Genome Dataset <cit.> annotations. NegCLIP is fine-tuned to purportedly enhance the base model's performance on tasks requiring relational and activity understanding.


We also report results for several other notable VLMs that have been published in recent years. These include Eva01 <cit.> and Eva02 <cit.>, FLAVA <cit.>, ALIGN <cit.>, COCA <cit.> and AltCLIP <cit.>.










 §.§ Results

This section presents the outcomes of experiments designed to evaluate model performance across various tasks, including Image-To-Text and Text-to-Image retrieval, and activity recognition(Detailed results and descriptions of activity recognition experiments are provided in the Appendix.). Additionally, this section specifically addresses and reports on the bias observed in the text encoder.



  §.§.§ Image-to-Text Retrieval



We assess the performance of the models discussed in Section <ref> on the GAB dataset and calculate their accuracy in retrieving the correct caption as outlined in Section <ref>. We employ a caption template of the form: “a man/woman is doing activity and a woman/man is in the scene”. The results for activities that are stereotypically biased are presented in Figure <ref>, while the results for the other two groups of activities are provided in the Appendix. 













In this experiment, we assessed image-to-text retrieval across three distinct scenarios. The first scenario involves images where an individual of an unexpected gender is performing a typically biased activity with no other individuals present in the scene. The second scenario comprises images where both genders are present, and the activity is performed by the gender typically associated with it. The third scenario mirrors the second, but in this case, the activity is performed by the gender not typically associated with it.


Performance Drop Due to Presence of Expected Gender:
As depicted in Figure <ref>, there is a noticeable drop in the accuracy of the models when the scene includes two genders compared to scenarios where only the unexpected gender is present and performing the activity associated with bias. The chart reveals that most VLMs experience an average performance decline of approximately 33.2%. However, a few models demonstrate only a slight decrease in performance.


Performance Drop Due to Gender-Activity Binding Bias: 
In the scenarios where there are two individuals present at the scene, the majority of models display a substantial decline in retrieval accuracy when the action is performed by an individual of the unexpected gender. As indicated in Figure <ref>, the VLMs undergo an average accuracy reduction of approximately 13.2% when encountering gender activity binding bias. 

Note that, as illustrated in Figure <ref>, the models demonstrate satisfactory performance in the Image-to-Text retrieval task on both image scenarios of U1 and E1, indicating that the gender bias adversely affects the performance of the models only when there is more than one individual in the image. Additionally, the accuracy of the models remains consistent, regardless of whether the expected or unexpected gender is performing the activity. The models' average performance in both scenarios is approximately 80%.



  §.§.§ Text Encoder Bias

Given an activity a, consider e,u∈{man, woman} where e is the expected gender of the individual performing a and u is the unexpected one. To separately assess the bias of the text encoder, we determine the frequency at which the embedding of a gender-neutral sentence "a person is doing activity" is closer to the embedding of "a e is doing activity" than the embedding of "a u is doing activity". The outcomes are presented in Table <ref>. It can be observed that in all categories of the gathered activities, the majority of activities exhibit a bias towards the expected gender.














  §.§.§ Text-to-Image Retrieval




Figure <ref> illustrates the accuracy of VLMs in the task of text-to-image retrieval. For each activity, we form random pairs of images from E2 and U2[Pairing is in the form of one-to-one correspondence.]. Each model is then evaluated for its ability to retrieve the image that best matches captions formatted as "a man/woman is doing activity and a woman/man is in the scene". As can be seen in Figure <ref>, VLMs achieve an accuracy of approximately 50%, indicating that their performance is nearly random in this task.
If we replicate this experiment, altering only the captions to a gender-neutral phrase "a person is doing activity", and designate the image where the expected gender is performing the activity as the true label, the VLMs achieve an accuracy of approximately 50% again, as illustrated in Figure 4 in the Appendix.

In essence, the benchmarked models do not seem to comprehend the properties of the given images that would aid in retrieving the correct one given embedded information from the caption, i.e., they fail to recognize the performer of the activity based on the text.









































 











































§ DISCUSSION



 §.§ Image and Text Embedding in VLMs

To gain a deeper understanding of how texts and images are encoded into the shared embedding space, consider these points:

    
  * Given that the image retrieval accuracy is nearly 50% for both expected and unexpected caption groups (as shown in Figure <ref>), we can infer that the cosine similarity (as per Eq. <ref>) between both sets of captions and images (for expected and unexpected groups) is almost identical.
    
  * As the average text retrieval accuracy exceeds 60% for E2 and falls below 50% for U2, it suggests that text embeddings are somewhat closer to the image embeddings in E2.

A plausible interpretation of these observations is that VLMs encode images more closely to the encoding of the more expected text. However, the image encoders of VLMs embed images from both groups closely together, resulting in nearly equal cosine similarity with any caption embedding.



 §.§ Bias Mitigation

Multiple works have previously discussed approaches for mitigating bias from VLMs, such as orthogonal projection <cit.>, making unbiased datasets <cit.>, representation correction <cit.>, and prompt tuning <cit.>. Several of these approaches could be applied also for robustness to gender-activity binding bias. For more detail on these works, refer to the Appendix.




§ FUTURE WORKS

Study Other Social Biases.
In this work, we focused solely on gender bias; however, the experimental approaches described can be extended to other social biases, such as race and age. In future research, we plan to explore how VLMs perform across these dimensions using the experiments outlined in this study.

Study the Source of Bias in the Training Data.
Due to limited accessibility to the training sets, especially for models with unpublished datasets, this study could not investigate the source of bias in the training data directly. We plan to address this limitation in future work by examining the sources of bias in publicly available training datasets for VLMs and analyzing the origins of these biases within those datasets.






§ CONCLUSIONS





In this study, we have explored a significant yet overlooked bias in VLMs known as the gender-activity binding bias. We used a unique dataset named GAB, which includes about 5500 images. We found that the capacity of VLMs to link an activity with the gender of its performer notably decreases when another person of a different gender is in the scene. We discovered that while the bias is evident in image-to-text retrieval tasks and in the text encoder alone, the models perform randomly in text-to-image retrieval tasks. This suggests that the gender-activity binding bias is primarily absorbed by the text encoder rather than the image encoder in VLMs. Furthermore, we noted that while VLMs have difficulty with gender-activity binding, they do have some ability to recognize activities. We believe that the ability of VLMs to comprehend activities and perform compositional reasoning in complex scenes are other crucial factors contributing to the gender-activity binding bias that could be investigated in future research.
















































§ DATASET






 §.§ Detailed Report

The method of selecting activities is detailed above, ultimately resulting in three categories of biased activities. comprehensive report of each category provided in Table <ref> , Table <ref> , Table <ref>.











































































































 §.§ Experimented Models For Image Generation


Initially, we explored several diffusion<cit.> models including the original Stable Diffusion<cit.>, Stable Diffusion XL<cit.>, and its checkpoints like (Juggernaut XL[https://civitai.com/models/133005/juggernaut-xl] and DreamShaper XL[https://civitai.com/models/112902/dreamshaper-xl]). We tried Fooocus[https://github.com/lllyasviel/Fooocus], an image generation tool that includes its prompt rewriting system using GPT-2<cit.>. The generated images were realistic and diverse; however, they lacked representation of the intended activity, especially in the context of compositional generation. Upon using DALL-E 3 with OpenAI API, we observed a significant improvement in the quality of compositional generation while also keeping the reality and the clarity of the image. We chose DALL-E 3 as our final image generation solution.




 §.§ Used LLM For Prompt Enhancement


We evaluated various large language models (LLMs) for this task. Due to our hardware constraints, we ultimately selected the Llama2 model, which has 70 billion parameters, to generate prompts <cit.>.




§ BIAS MITIGATION

Multiple works have previously discussed approaches for mitigating bias from VLMs that could be applied also for robustness to gender-activity binding bias. 

In <cit.>, an orthogonal projection is applied to the end embeddings, making them orthogonal to the embeddings of spurious prompts, effectively mitigating bias introduced by these prompts. This alteration might also strip away useful nuances from the embeddings, potentially degrading the model’s overall performance on tasks where those nuances are beneficial and also this might not hold true in complex real-world scenarios where biases are not strictly linear.

In <cit.>, a method is introduced that utilizes additive residual image representations to correct biases in the original representations. However, training these residuals demands extensive and diverse data, making the process resource-intensive. Moreover, incorporating these residuals may alter other inherent characteristics of the original representations, potentially affecting the overall performance of the model.

In <cit.>, learnable prompts integrated into the text input are used to mitigate bias, optimized through the training of an adversarial classifier that evaluates similarity scores between outputs from two encoders. This method employs both adversarial and contrastive losses to preserve the quality of the joint representation while aiming to reduce bias. However, the effectiveness of this adversarial training depends on the availability of a large and diverse dataset; otherwise, there is a risk of overfitting to adversarial objectives. Additionally, since VLMs are trained with various objective functions, making adjustments to mitigate bias could potentially lead to trade-offs in other performance metrics.

Both <cit.> and <cit.> address the creation of fair datasets to mitigate bias in VLMs, though each method has its limitations. <cit.> uses AI to generate datasets for fine-tuning VLMs, which is computationally intensive and can introduce new biases due to the limitations of AI generation tools. On the other hand, <cit.> employs counterfactual data—images and texts modified to alter specific attributes. While this method targets specific biases directly, it is limited to those attributes it explicitly modifies and may not adequately address all potential biases.

In <cit.>, the authors address bias by calculating the mutual information between each dimension of the encoder outputs and the gender attribute, clipping dimensions with the highest mutual information to reduce their influence. While this approach effectively reduces gender bias in embeddings, it faces limitations similar to other inference-time interventions. Notably, it may not adapt well to a wide range of biases and could potentially degrade the accuracy of models in other downstream tasks due to the removal of critical information during the clipping process.







§ RESULTS






 §.§ Image-to-Text Retrieval


The performance of text retrieval for texts featuring two genders (one of them performing the activity) is demonstrated in Table <ref> and Table <ref>. This is akin to the experiment discussed in Section 5.2.1,
In this experiment, we assessed image-to-text retrieval across three distinct scenarios. The first scenario involves images where an individual of an unexpected gender is depicted performing a typically biased activity with no other individuals present in the scene. The second scenario comprises images where both genders are present, and the activity is performed by the gender typically associated with it. The third scenario mirrors the second, but in this case, the activity is performed by the gender not typically associated with it. In Table <ref> the experiment is reported for gender-biased activities from LAION-400M and in Table <ref> the experiment is reported for gender-biased everyday activities.








 §.§ Text-to-Image Retrieval


The performance of image retrieval for images featuring a single individual (the one performing the activity) is demonstrated in Figure <ref>. This is akin to the experiment discussed in Section 5.4 of the main text, where images in E1/U1 are categorized and the models are evaluated based on their ability to accurately retrieve the correct image using captions that follow the template “a man/woman is performing an activity”. The models exhibit an average accuracy of approximately 90%, indicating that they can effectively identify the image that correctly represents the gender of the individual performing the activity, provided there are no individuals of a different gender present in the scene.
Also, you can see the results of experiment mentioned in Section 5.4 of the main text in Figure <ref>.






 




 §.§ Activity Recognition


A crucial inquiry concerning the bias in binding gender with certain activities is whether this bias originates from a misunderstanding of the activity itself, and if this misunderstanding leads to a failure in binding the gender and the activity. To gauge the model’s understanding of activities, we collect several (at least 9, 17.25 on average) other different activities that are performed in the same environment as the reference activity for each activity in the GAB dataset using GPT-4 <cit.>. For each image in the GAB dataset, we create a caption from the reference and associate collected activities and arrange them based on their matching score to the image. We show the mean reciprocal rank (MRR), recall@1, and recall@3 for the true caption (reference activity) in Table <ref> for stereotypical activities. (results for other categories of activities is depicted in Table <ref> and Table <ref>).

The table indicates that the model can, to some extent, retrieve the true activity from a list of activities performed in a similar environment, demonstrating that VLMs can recognize the performed activity in image-to-text retrieval.




























 Paper Title: Enriching thermal point clouds of buildings using semantic 3D building modelsenriching thermal point clouds of buildings using semantic 3D building models.















	1 Professorship of Photogrammetry and Remote Sensing, 
	2 Chair of Engineering Geodesy, 
    3 Chair of Geoinformatics, 

 TUM School of Engineering and Design, Technical University of Munich, 80333 Munich, Germany 
 - (jingwei.zhu, olaf.wysocki, christoph.holst, thomas.kolbe)@tum.de











XX, YY 
XX/YY 
   



Thermal point clouds integrate thermal radiation and laser point clouds effectively.
However, the semantic information for the interpretation of building thermal point clouds can hardly be precisely inferred.

Transferring the semantics encapsulated in 3D building models at LoD3 has a potential to fill this gap.
In this work, we propose a workflow enriching thermal point clouds with the geo-position and semantics of LoD3 building models, which utilizes features of both modalities:

model point clouds are generated from LoD3 models, and thermal point clouds are co-registered by coarse-to-fine registration.
The proposed method can automatically co-register the point clouds from different sources and enrich the thermal point cloud in facade-detailed semantics.
The enriched thermal point cloud supports thermal analysis and can facilitate the development of currently scarce deep learning models operating directly on thermal point clouds.




Enriching Thermal Point Clouds of Buildings using Semantic 3D building Models
    Jingwei Zhu1, Olaf Wysocki1, Christoph Holst1,2, Thomas H. Kolbe3 
    
=============================================================================







§ INTRODUCTION




A thermal point cloud combines synchronized thermal radiation data and the laser point cloud, effectively capturing the thermal characteristics of objects.
In the case of building objects, the radiation and temperature differences can be caused by building operation, material variation, aging, and physical damage.
Therefore, thermal point clouds of buildings can be applied in hidden structure detection, energy inspection, and heritage protection <cit.>.
When interpreting a building's thermal point cloud, we must consider the element types and comprehensive geometric descriptions.


Notably, geometry-induced thermal variations such as cracks and distortions can be estimated from dense point clouds.
However, directly inferring facade elements, such as doors, windows, roofs, and walls, from thermal point clouds proves challenging.




Semantic information can be obtained by manually labeling thermal point clouds or by performing semantic segmentation. 
Directly projecting TIR images to a 3D building model for thermal point cloud generation will render unrelated objects, such as vehicles and pedestrians, to the facade, leading to misinformation. 
Therefore, it is crucial to synchronize and align the laser point cloud with TIR images to ensure accurate radiometric representation.
However, directly labeling the laser point clouds is time-consuming and requires extensive familiarity with the study area. 
Additionally, existing methods for point cloud semantic segmenting facade elements often lack accuracy, especially for categories like clutter and windows <cit.>.
Given these challenges, it is advisable to augment the facade-level semantic information with data from reliable sources.










A potential to fill such data-scarcity gap exhibit worldwide-available semantic 3D city models.
Rich building-related semantics are encapsulated in semantic 3D building models at LoD3, characterized by highly-detailed and object-wise semantics at the facade level <cit.>.
Moreover, such LoD3 models possess highly accurate absolute georeferencing accuracy, reaching up to the cm-level <cit.>.
Recent trends imply growing availability of LoD3 building models, since there are new LoD3 datasets emerging [https://github.com/OloOcki/awesome-citygml] as well as novel methods investigating automatic LoD3 reconstruction <cit.>.

Assuming the infrequent occurrence of substantial structural and morphological changes in urban architecture, we believe thermal point clouds can be semantically enriched by fusion with LoD3 models.
However, three factors need to be considered.


First, the LoD3 model and point clouds are organized in different data formats <cit.>.
The structured LoD3 geometric representation frequently follows the boundary representation (B-Rep), as per the CityGML standard <cit.>. In contrast, point clouds are commonly represented as a set of unstructured points (x, y, z).
Second, the shared overlap is different.
The LoD3 model contains envelopes of the complete building, including all the outer-observable details.
The buildings in the point clouds may be incomplete due to occlusions and scanning platforms.
Besides buildings, point clouds include all the objects in the scene, such as vehicles, traffic signs, and pedestrians.

Moreover, the objects' geometric features are represented differently.
Although the LoD3 models are highly detailed, they represent a generalized building geometry. 
In contrast, point clouds provide non-generalized, raw data representing
events and states occurring only at the time of recording, such as opened doors and blinded windows, which is also crucial for thermal analysis.


Considering all the factors, we propose a workflow to transfer the semantic information from the LoD3 model to the thermal point clouds.
We first transfer the LoD3 model to semantic point clouds and then co-register the point clouds.
The semantic labels are enriched in thermal point clouds according to registered building models.
The enriched thermal point clouds disclose thermal attributes for different building elements for analysis.
The co-registered point clouds, on the other hand, can help to validate the geometric information of LoD3 models and enrich the details.
The contributions of our work include:


    
  * We propose a feasible workflow to enrich the semantic information of point clouds from LoD3 models.
    
  * We propose an algorithm to automatically co-register the laser point clouds and point clouds from the LoD3 model.
    
  * Our experiments validate the results of enriched information from the model and application to the thermal analysis.



The structure of this paper is organized as follows:
In Section <ref>, we summarise the related work, and our proposed methods are presented in detail in Section <ref>.
Then, the data and experiments are described in Section <ref>, and then the results are discussed in Section <ref>.
Finally, some conclusions are drawn in Section <ref>.




§ RELATED WORK




 §.§ Semantic 3D building models


Semantic 3D city models comprehensively describe structures, taxonomies, and aggregations on a city, regional, and even national scale. 
Internationally, the standard CityGML, established by the OGC <cit.>, is utilized for the representation and management of city models. 
CityGML facilitates the modeling of urban objects with their 3D geometry, appearance, topology, and semantics at four different LoD. 
The latest data model of CityGML 3.0 adheres to the ISO 191xx series of geographic information standards, and CityGML datasets are commonly encoded using either the GML or CityJSON <cit.>.

Since urban dwellings are the cornerstone of each city, most existing semantic 3D city models comprise buildings <cit.>.
LoD1 and LoD2 building models are currently widely available, as underscored by the example of approximately 220 million models available in Germany, Japan, the Netherlands, Switzerland, the United States, and Poland [https://github.com/OloOcki/awesome-citygml].
This broad adoption owes mainly to the robust 3D reconstruction algorithms and available building footprints combined with aerial observations <cit.>.
Although LoD1 and LoD2 possess building semantics, they lack detailed facade semantics, which is pivotal for facade-level point cloud labeling. 
This gap can be filled by LoD3 building models, characterized by descriptive facades, composed of objects such as windows, doors, balconies, and even underpasses <cit.>. 
Currently, the automatic LoD3 reconstruction is an active field of research proposing various promising methods and input datasets to solve the challenge <cit.>.




 §.§ Model to Point Clouds Registration


Registration of 3D models and point clouds is typically done by feature matching.
Point clouds offer accurate and detailed geometry information about existing structures. 
Therefore, they are widely used as data providers for 3D model reconstruction and manual modeling.
Extracted planes are used as features for co-registration due to their relatively simple representation and frequent occurrence in man-made objects.
<cit.> propose the semi-automatic method to register construction sites with BIM models. Three corresponding planes are required to be manually selected for coarse registration. 
<cit.> also focus on planes but generalize BIM faces as TLS patches with a point and a normal vector.
The detected faces and planes are organized manually by connected relation.
Then, the model faces are co-registered to the point cloud patches for monitoring the construction process. 
Besides man-involved work, automatic methods are also investigated.
<cit.> group the detected parallel planes as descriptors to register as-built point clouds and as-plan BIM models.
The use of planes avoids setting control points for registration, but a sufficient number and unique patterns are required.
Another often-used set of primitives is lines.
<cit.> propose a fully automated method to register photogrammetric point clouds to a building model with lines for the indoor scene, where sufficient and well-distributed corresponding line features from images are required.
<cit.> conducts a coarse-to-fine registration from BIM to the point cloud.
Raw camera poses are used to coarsely align the model, and then an adopted ICP to achieve the fine registration. 


Despite registering point clouds to BIM models, which targets to monitor and detect the changes, only a limited number of research groups have delved into the realm of coregistration between street-level point clouds and semantic 3D city models in our extensive investigation.

<cit.> primarily centers around point clouds generated from images through the SfM algorithm, utilizing radiometric features for prefiltering, which may inadvertently eliminate valid building features.
<cit.> detect the footprint of buildings from point clouds and LoD model. A mixed integer linear program is employed to identify correspondences between 2D lines and points.
In these cases, sufficient line and point features are necessary to form unique patterns for registration.
<cit.> consider only façades points by random forest and register to LoD1 model for trajectory optimization.
This approach, however, requires training data and initial transformation information.










 §.§ Point cloud co-registration

Point cloud registration has long been a research topic.
It involves aligning point clouds from different sources, with low overlap, or from metrically inaccurate datasets. 
The standard registration, such as ICP <cit.>, can deal with common situations with sufficient overlap and similar point density but can hardly handle complex situations.
Using control points (targets) could solve this issue leading to a high and traceable accuracy <cit.>, but requires manual field work and is thus not scalable to large data sets and it lacks autonomy.
Therefore, most point cloud registration pipelines inherently have a workflow containing two steps: coarse registration for initial transformation and refined transformation with denser correspondences <cit.>.
The coarse transformation uses sparse feature-based correspondences, and it is crucial for non-georeferenced point clouds.
Key points <cit.>, lines <cit.>, and planes <cit.> can all be used as features for registration.
Moreover, a combination of the features is also used, like 4PCS <cit.> and Super4PCS <cit.>.
When the initial relative poses are given by GNSS or manually processed, the coarse registration is typically unnecessary.
The fine registration can adjust the geometric transformation to achieve better accuracy.
It usually iteratively updates the transformation matrix to minimize the point distances with denser correspondences.
The ICP method is widely adopted in the field owing to its simplicity and efficiency. 
Numerous algorithms have stemmed from the ICP framework, exemplified by references such as <cit.>.

Recently, deep learning <cit.> methods have become increasingly prevalent for point cloud registration.
However, a notable challenge persists in managing the intricacies of large study areas and meeting the demand for extensive training datasets.
This issue underscores the current limitations in the application of deep learning techniques to point cloud registration, particularly in addressing the complexities posed by expansive geographical contexts.


 

§ METHOD


Our work aims to transfer the semantic information from the LoD3 model to the corresponding thermal point clouds.
The general workflow is shown in Figure <ref>. 
First, the point clouds are generated from laser point clouds and TIR image sequence, and an LoD3 model respectively.
Then, thermal point clouds are aligned to the model point clouds to obtain the transformation matrix.
After registration, the semantic information from model point clouds is transferred to the thermal point clouds for analysis.

 



 §.§ Point cloud generation







To cope with the cross-domain gap, we opt to homogenize the two distinct representations. 
The approach of generalizing the point cloud feature to match the model primitives is challenging, as shown by the 3D building reconstruction research <cit.>.
The occlusion and incompleteness of laser point clouds lead to inefficiency and false matching.
Therefore, LoD3 geometry is sub-sampled to a set of points corresponding to the representation of the thermal point cloud.







Thermal point cloud generation
The thermal point clouds are generated by projecting the thermal texture from TIR images onto the MLS point clouds with position information <cit.>.
The MLS point clouds and TIR image sequences are captured with the same platform. 

When the relative poses of the thermal camera are estimated, the points in the point clouds find its corresponding points in the TIR image by co-lineary equation (eq. <ref>) <cit.>.




    u_i=K[R|T]X_i



 
where  $̄u_i= image coordinatesX_i= point cloud coordinatesK= camera parameters, including aspect ratios,
 focal lengthfand principle points(c_x, c_y)R=3×3rotation matrixT= translation matrix

    K=
        [ s· f    0  c_x;    0    f  c_y;    0    0    1 ]
         R=[ α_11,α_12,α_13; α_21,α_22,α_23; α_31,α_32,α_33;                ],T=[ t_1; t_2; t_3 ]

When the matrixK,RandTare obtained from pose parameters, the corresponding intensity values of point clouds are calculated from the image and rendered to the point cloud.

Model point cloud generation
The semantic 3D building models follow the paradigm of boundary representation (B-Rep), where each modeled object has its geometrical, outer-observable surface explicitly described by a set of vertices <cit.>.
Moreover, each object in the model has assigned semantics and shall not overlap with other objects.

As illustrated in Figure <ref>, we leverage these traits to our advantage by performing object-oriented point surface sampling on a regular grid (purple) at the given sampling rates_r; the parameter is chosen accordingly to the expected thermal point cloud density. 
Each sampled point inherits the semantic class of a leaf objectlabel_iand its associated unique identifierid_i.
This results in a point cloudMC_iextended by a scalar valuelabel_iandid_i, as in Equation <ref>.


    MC_i = [x_i, y_i, z_i, label_i, id_i]


 §.§ Co-registration

Owing to the point cloud generation step, we formulate the alignment as the cloud-to-cloud co-registration problem.
In our work, only rigid transformations between point clouds are considered, as shown in eq. <ref>.


    MC_i=[R_p|T_p]TC_i

    where $̄TC_i= coordinates of thermal point cloudsMC_i= points in the model point clouds



Considering the limited overlapping areas, the features and details of building facades are important for co-registration with point clouds.
Unlike LoD1 and LoD2, the LoD3 building models comprise 3D facade elements <cit.>.

The increased information makes it possible to locate the corresponding features with point cloud facades, especially in highly repeated patterns.
However, the features of the windows and doors may not have the same representation in measured point cloud and the model point cloud of the LoD3 model. 

One-to-one corresponding features cannot be guaranteed. 
Under this condition, we propose a coarse-to-fine registration by combining a feature-based method using FGR <cit.> and our adopted version of plane-based ICP <cit.> as fine registration.

FGR uses features to calculate the correspondences and estimate the transformation matrix.
It first calculates the FPFH (Fast Point Feature Histograms) features of the point clouds.
The initial corresponding points are established by feature matching with nearest neighbor pairs. 
It uses the Reciprocity test and Tuple test to improve the inlier ratio of the correspondences set.
Due to the differences in feature representation, noisy correspondences cannot be avoided.
Then, the pose are optimized such that distances between corresponding points are minimized.
The optimization function for the optimal transformation matrix estimation is expressed as in (eq. <ref>):


    E([R|T]) = min_∑_(p,q)∈ Kρ‖ p_i - [R|T] · q_i ‖

    where $̄p_i= point coordinates in thermal point cloudsq_i= corresponding point coordinates 
 in the model point cloudsρ= robust penalty

FGR uses a scaled, well-chosen German-McClure estimator to reduce the computation, and Black-Rangarajan duality is used to optimize eq. <ref> with a line process over the correspondences.
Then, the optimization objective can be turned into a least-squares objective and the Gauss-Newton method is used to find the solution.

After coarse registration, a fine registration is further updated by point-to-plane ICP.
Although the FGR provides the initial transformation result, it is insufficient for detailed analysis due to the false matching from different target and source point cloud feature representation.

We adopt the point-to-plane ICP variant <cit.> and model-based height rectification <cit.> with both height and center point rectification.
Assuming that the model and point clouds are all related to the ground, the thermal point cloud is lifted to the same basic height of the model.
The same applies for the center points of the planes.
We leverage the algorithm to align two point clouds while minimizing the distances between corresponding points belonging to the target and source plane.
Since thermal point clouds only capture the facades of buildings, which follow planar-like shapes, the main plane of the buildings are extracted as a base for the registration.
We perform the plane extraction using RANSAC algorithm <cit.> where the main planes are detected in both point clouds. 
Our approach assumes that the closest planes are detected and aligned from the coarse registration.
The algorithm initiates with an initial alignment represented as identity matrix, where the target point cloud is approximately aligned with the source point cloud using an initial transformation matrix of from coarse registration. Subsequently, a nearest neighbor search is conducted for each point in the source cloud to find its closest counterpart in the target cloud with optimization eq. <ref>
    E([R|T]) = min∑_(p,q)∈ K‖ p_i - [R|T] · q_i ‖ n_p

    where $̄n_p= normal of the pointpThe maximal corresponding distance corresponds to thed_max.
The convergence criteria are met if the RMSE reachest_rmse(eq.<ref>) threshold and performst_ititerations.


    t_rmse = √(1/N∑_i=1^N (d_i)^2)


Upon meeting these conditions, the algorithm concludes, providing the final alignment result in the form of a transformation matrix. 
The updated transformation parameters are used to refine the position of the source point cloud. 



 §.§ Semantic enrichment


 In Section <ref>, the transformation matrix is calculated to register the thermal point clouds to the model point clouds.
 The geo-reference coordinates of thermal point clouds can be calculated by applying the estimated transformation matrix.
 After the transformation, the thermal point cloud is aligned to the model point cloud.
 Assuming there are no changes in building details for the laser point clouds and model; the semantic labels for windows and doors will remain the same.
 Therefore, the points in thermal point clouds should have the same labels as in the model point clouds.
 Considering the differences in sampling rate and locations, a threshold distance is set to minimize the false correspondence.
 For each point in the thermal point clouds, the closest point in the model point clouds is calculated, and the label is given to the laser point.
 If the nearest neighbor points do not have corresponding labeled points in the model point cloud within a certain threshold, they can be regarded as noise and labeled as "unlabeled".
 This avoids mismatched labels from other objects, such as trees and pedestrians, while keeping the labels for the buildings.
 

 

§ DATA AND EXPERIMENTS



The test site is around the main campus of the Techinical University of Munich (TUM). 
The thermal point clouds were generated by combining thermal image sequences and MLS point clouds from the TUM-MLS dataset <cit.>.
The TUM-MLS dataset was measured using a mobile platform, which includes two laser scanners and a thermal camera.
The poses of TIR images were estimated from the GNSS and the IMU system from the integrated platform.
The LoD3 model was selected from the TUM2TWIN dataset  [https://github.com/tum-gis/tum2twin].
We selected one building from the test dataset close to the main gate characterized by a partial coverage (the so-called building 23), as shown in Figure <ref>.
We set the sampling rate ats_r = 0.1, which resulted in a uniformly sampled point cloud of a 0.1 m distance.
Figure. <ref>(a) shows an example of the TUM-MLS point cloud with intensity, and (b) demonstrates the LoD3 building model.



The generation of thermal point clouds and semantic labeling was done using c++ and pcl library(1.81) <cit.>.
With 32G RAM, and an i7-6000 @3.4 GHz CPU, it takes approximately 346.06s for labeling.
The FGR was performed using the code from <cit.>.
Further experiments were performed using the Feature Manipulation Engine (FME)  version 2020.01 and Open3D <cit.>.
The implementation is available in a public repository[https://github.com/tum-pf/LoD3toTCld].

 

§ RESULT AND DISCUSSION


The generated thermal point cloud and model point cloud are shown in Figure <ref>.
Thermal point clouds (Figure. <ref> (a)) include building facades and other objects in the TIR images, such as traffic lights, pedestrians, and vehicles.
Thermal point clouds show the geometry of building elements, including different shapes of windows.
The moldings, balconies, and special decorations are also recorded as they are.
However, the rooftop and some corners (e.g., the upper right corner) are missing due to scanning mode and height limitations.
Compared to the raw point clouds, thermal point clouds keep the original geometry features while attaching thermal attributes as intensity for the points.
The different intensities represent the temperature and can reveal inner structures like heating pipes.
The higher intensity around windows shows wooden frames and some indoor rooms with higher thermal temperatures.
Model point clouds (Figure. <ref> (b)) generated from the LoD3 model describe the semantic information with different colors for windows, doors, walls, roof, and ground. 
Unlike laser point clouds, where the laser penetrates the window glasses and leaves empty spaces, the model point clouds block the window areas with in-depth planes.
Moreover, all the functional segments are labeled, but non-functional decorations are simplified compared to the laser point clouds.



Then, we registered the thermal point clouds to the model point cloud after point cloud processing.
The results are shown in Figure. <ref>.
Figure. <ref>(a) illustrates the transformation of thermal point clouds with initial pose after FGR.
The transformed thermal point cloud shows that it is roughly transformed to the model point cloud within a certain overlap.
Although the thermal point cloud has mainly limited facades with occlusions, the corresponding main planes with most of the windows were attached. 
The redundant objects on the front street and inner noisy point clouds of facades influenced the registration result.
Figure. <ref>(b) presents the results after fine registration of the point clouds aligned with the model.
After fine registration, the thermal point clouds are correctly aligned to the facade of model point clouds. 
The corresponding windows with different shapes are equivalent, showing a satisfactory co-registration result.





    fitness = # inlier correspondences/# points in target





To estimate the co-registration result, we calculate the RMSE distances (eq. <ref>) and the fitness (eq.<ref>) (threshold = 2m).
As we show in Table <ref>, our fine-registration approach can reach a high improvement rate.
In the case of our tested sample, the RMSE has decreased approximately five times (1.46m vs. 0.33m), while the fitness score has improved by approximately 65% (0.54 vs. 0.88).
To further validate our result, a comparison experiment was conducted by manually selecting corresponding points and estimating the transformation matrix.
Six corresponding points were manually selected from model point clouds and thermal point clouds and were transformed with the estimated matrix (Figure <ref>).
The RMSE between the ground truth and fine registration was 0.4m.
Regarding the fitness and RMSE, our proposed method achieved a comparable level of accuracy and better fitness to the reference (Table <ref>).



With the calculated transformation matrix, the thermal point clouds can be enriched from semantic model point clouds.
By applying the transformation, the thermal point clouds are aligned to the LoD3 model and georeferenced to the reference coordinate system w.r.t. the LoD3 model.
The semantic labels for each laser point are calculated as described in Section. <ref>.
As shown in Figure. <ref>, the facades of the building are labeled with semantic labels while the uncovered areas (from the street) remain unlabeled.
To further analyze the behavior of different details in the building, the statistical number is calculated based on the frequency for each class, as shown in Figure. <ref>.
For each classes, we calculate the average intensity and the standard deviation to investigate the thermal difference for different classes, which correspond to different materials and structures, in the same environment.
The average intensity value and distribution vary across the classes.
The number of wall points and windows were the prevalent classes in the point clouds.
The limited roof points below the eave show relatively lower temperature and variance compared to walls and windows.
The windows show relatively higher changes with higher standard deviation.
The initial results from this experiment show that thermal intensity varies for different materials, with potential applications such as material differentiation and anomaly detection.






§ CONCLUSION


In this paper, we propose a feasible workflow to enrich the semantic information of a thermal point cloud given a LoD3 model.
The proposed method converts the LoD3 model to a point cloud and registers a thermal point cloud to the model through point cloud co-registration.
With the proposed coarse-to-fine registration, the thermal point clouds can be registered to semantic model point clouds regardless of limited overlap and feature differences.
The co-registration results have comparable accuracy to the referenced manually registered results.
Finally, the semantic labels from the LoD3 model are assigned to the thermal point clouds for analysis.
This work is not limited to thermal point clouds but also to all the co-registration tasks, from laser point clouds to building models requiring semantic label transfer.
The enriched results improve the point cloud labeling pace by giving knowledge and enhancing the efficiency of semantic data generation.
The thermal point clouds with LoD3 labels can serve as supportive data for further urban study and algorithm development, such as testing and training deep learning models.
With the given pose of the image, the labels can be back-projected to the TIR images for processing and supportive analysis, as shown in Figure <ref>.
In this work, we combine point cloud geometry for thermal anomaly interpretation, but also a bi-directional information exchange can be pursued: The thermal properties might be mapped onto the LoD3 objects enriching the LoD3 in radiometric thermal features for visualization and building operation monitoring <cit.>


For future work, it is worth further investigation into the topic of robust methods for model and point cloud co-registration, especially in large-scale datasets.
Though this work proposed the initial tasks for single building co-registration and semantic enrichment results, how to improve the efficiency is to be explored.
How to use the features in the laser point clouds and LoD3 model while minimizing the effect introduced by different feature representations is still a problem.
The co-registration results can help enriching and fusing information from different datasets or localize and compare the changes from other models or time stamps.



 

§ ACKNOWLEDGEMENT

We would like to express our gratitude to the Leonhard Obermeyer Center (LOC) at the Technical University of Munich (TUM) for their support in facilitating this work. 
Additionally, we extend our appreciation to tum2twin team at TUM for their valuable insights and for generously providing access to the CityGML datasets.

1.17

 Paper Title: SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving.



simplellm4ad-an-end-to-end-vision-language-model-with-graph-visual-question-answering-for-autonomous-driving


§ SIMPLELLM4AD: AN END-TO-END VISION-LANGUAGE MODEL WITH GRAPH
VISUAL QUESTION ANSWERING FOR AUTONOMOUS
DRIVING


Peiru Zheng, Yun Zhao, Zhan Gong, Hong Zhu, Shaohua Wu

IEIT Systems

abstract


 §.§ Abstract


Many fields could benefit from the rapid development of the large
language models (LLMs). The end-to-end autonomous driving (e2eAD) is one
of the typically fields facing new opportunities as the LLMs have
supported more and more modalities. Here, by utilizing vision-language
model (VLM), we proposed an e2eAD method called SimpleLLM4AD. In our
method, the e2eAD task are divided into four stages, which are
perception, prediction, planning, and behavior. Each stage consists of
several visual question answering (VQA) pairs and VQA pairs interconnect
with each other constructing a graph called Graph VQA (GVQA). By
reasoning each VQA pair in the GVQA through VLM stage by stage, our
method could achieve e2e driving with language. In our method, vision
transformers (ViT) models are employed to process nuScenes visual data,
while VLM are utilized to interpret and reason about the information
extracted from the visual inputs. In the perception stage, the system
identifies and classifies objects from the driving environment. The
prediction stage involves forecasting the potential movements of these
objects. The planning stage utilizes the gathered information to develop
a driving strategy, ensuring the safety and efficiency of the autonomous
vehicle. Finally, the behavior stage translates the planned actions into
executable commands for the vehicle. Our experiments demonstrate that
SimpleLLM4AD achieves competitive performance in complex driving
scenarios.

introduction


 §.§ 1. Introduction


Autonomous driving has garnered significant attention from both academia
and industry over the past decade.<cit.> The promise of
safer roads, reduced traffic congestion, and increased mobility for all
populations has driven rapid advancements in this field. Traditional
approaches to autonomous driving often rely on a modular pipeline
consisting of perception, prediction, planning, and control. However,
these methods can suffer from compounding errors across modules, leading
to suboptimal performance in complex and dynamic driving environments.

With the advent of large language models (LLMs),<cit.> a
new opportunity arises to redefine the approach to autonomous driving.
LLMs, particularly when integrated with vision-language models
(VLMs),<cit.> have shown remarkable capabilities in
understanding and generating human-like text based on visual inputs.
These capabilities can be harnessed to create more coherent and
integrated autonomous driving systems, capable of nuanced reasoning and
decision-making.

In this work, we introduce SimpleLLM4AD, an end-to-end autonomous
driving (e2eAD) method that leverages the power of VLMs. Our method
reimagines the traditional autonomous driving pipeline by structuring
the task into four interconnected stages: perception, prediction,
planning, and behavior. Each stage is framed as a series of visual
question answering (VQA) pairs, which are interlinked to form a Graph
VQA (GVQA). This graph-based structure allows the system to reason about
each VQA pair systematically, ensuring a coherent flow of information
and decision-making from perception to action.

The perception stage in SimpleLLM4AD utilizes ViT models to process raw
visual data, extracting meaningful features and identifying objects in
the driving environment. These visual insights are then translated into
a format that the language model can interpret, allowing for a more
sophisticated analysis of the scene. In the prediction stage, the system
forecasts the future states of the identified objects, considering their
potential movements and interactions. This predictive capability is
crucial for anticipating potential hazards and planning safe maneuvers.
The planning stage involves synthesizing the information gathered from
the previous stages to develop a driving strategy. This strategy is
designed to optimize safety and efficiency, taking into account the
dynamic nature of the driving environment. Finally, the behavior stage
converts the planned actions into executable commands for the vehicle,
ensuring precise control and responsiveness.

Our experiments demonstrate that SimpleLLM4AD not only achieves
competitive performance in driving benchmarks but also exhibits enhanced
robustness in complex scenarios. The integration of VLMs enables the
system to make context-aware decisions, significantly improving its
reliability and safety. Our key contributions are summarized as follows:

• We leverage the logical dependency of GVQA by utilizing the answers to
associated questions as contextual information for the current question.
This approach has been shown to significantly enhance the capabilities
of LLMs in terms of accuracy (ACC) and language score.

• We refine the prompts to further boost the performance of LLMs. We
optimize the simple question-and-answer (Q+A) format of the contexts to
simplify the contextual information, making it easier and more efficient
for LLMs to leverage previous knowledge.

• Finally, we introduce object detection branches into the LLM
optimization process, which include object localization, color
identification, and categorization. These additional branches provide
LLMs with a richer set of contextual cues, further enhancing their
performance.

related-work


 §.§ 2. Related work


large-language-models


  §.§.§ Large Language Models


LLMs have rapidly evolved in recent years, demonstrating unprecedented
capabilities in natural language understanding and generation. These
models, exemplified by OpenAI's GPT-3<cit.> and
GPT-4<cit.>, and other open-sourced transformer-based
architectures such as the LLaMA series,<cit.>
Vicuna,<cit.> Baichuan,<cit.>
Qwen,<cit.> Yuan 2.0,<cit.> Yuan
2.0-M32,<cit.> leverage vast amounts of data and
sophisticated training techniques to produce human-like text and
comprehend complex linguistic structures.

One of the significant breakthroughs with LLMs has been their ability to
perform a wide range of tasks with minimal fine-tuning. This versatility
arises from the models' pre-training on diverse corpora, enabling them
to capture extensive world knowledge and linguistic patterns. The
emergence of LLMs has led to remarkable advancements in various fields,
including natural language processing (NLP), question answering, machine
translation, and text summarization.

vision-language-models


  §.§.§ Vision-language Models


VLMs represent a significant advancement in the field of artificial
intelligence, aiming to bridge the gap between visual and textual data.
The architecture of VLMs typically involves two main components: a
visual encoder and a language decoder. The visual encoder processes the
image to extract features, which are then combined with text
representations generated by the language decoder. This integrated
approach enables the model to perform complex reasoning tasks that
require both visual and linguistic understanding. These models combine
the strengths of vision models, such as convolutional neural networks
(CNNs)<cit.> and ViT<cit.>, with the
capabilities of language models, allowing for more comprehensive
understanding and interaction with multimodal inputs.

Efforts in VLMs focused on tasks like image captioning and VQA. Image
captioning involves generating descriptive textual annotations for
images, while VQA requires the model to answer questions about the
content of an image. Nowadays, more and more VLMs have demonstrated
impressive capabilities in these areas by effectively integrating visual
and textual information. Flamingo<cit.> utilizes visual
and language inputs as prompts, demonstrating exceptional few-shot
performance in visual question answering. GPT-4,<cit.> the
LLaVA series<cit.> introduces visual instruction tuning
to enhance the instruction-following capabilities of VLMs. Concurrently,
models like KOSMOS-2<cit.>, and
Qwen-VL<cit.> have advanced VLMs by incorporating visual
grounding capabilities, facilitating tasks such as region description
and localization. Furthermore, PaLM-E<cit.> and
EmbodiedGPT<cit.> represent significant strides in
adapting VLMs for embodied applications, vastly expanding their
potential use cases. These advancements illustrate the rapid progress in
the development and application of VLMs, showcasing their ability to
handle increasingly complex multimodal tasks. As the integration of
vision and language continues to evolve, VLMs are poised to play a
pivotal role in various AI-driven domains, including autonomous driving,
healthcare, and interactive robotics.

language-grounded-driving


  §.§.§ Language-grounded
Driving


Language-grounded driving is an emerging research area that combines the
principles of NLP with AD technologies. This interdisciplinary approach
leverages the understanding and generation of natural language to inform
and enhance the decision-making processes of autonomous vehicles. By
grounding driving actions in language, these systems can achieve greater
interpretability, flexibility, and user interaction capabilities.

Recent advancements in LLMs and VLMs have propelled the field of
language-grounded driving. Models like GPTDriver,<cit.>
LLM-Driver,<cit.> and LMDrive<cit.> offer
new possibilities for integrating language grounding into autonomous
driving. These models enable more nuanced and context-aware
interpretations of driving environments and instructions.

Language-grounded driving represents a promising frontier in autonomous
vehicle technology. By integrating natural language understanding with
driving systems, researchers are developing more interactive,
interpretable, and adaptable autonomous vehicles. Our SimpleLLM4AD
method builds upon these advancements, utilizing language grounding to
enhance each stage of the e2eAD pipeline, from perception and prediction
to planning and behavior. This integration not only improves the
vehicle's decision-making capabilities but also facilitates better
human-vehicle interaction and scenario-based training.

method


 §.§ 3. Method


overall-architecture.


  §.§.§ 3.1. Overall architecture.


The pipeline of our method is outlined in Figure 1. The overall
architecture comprises two main modules: a vision encoder that processes
images, and an LLM decoder that handles questions.

Vision Encoder. We choose InternViT-6B as the vision encoder.
InternViT-6B, a vision transformer with 6 billion parameters, was first
introduced by Chen et al.<cit.> It was pre-trained using
web-scale image-text data from various sources to align with large
language models. The Query Model serves as a bridge between the vision
encoder and the LLM decoder, aligning the vision and text modalities.
This vision-text alignment component is initialized with a
multilingual-enhanced LLaMA.<cit.>

LLM Decoder. We employ Vicuna-13B as the LLM decoder.
Vicuna-13B is an open-source LLM that was fine-tuned from LLaMA using
user-shared conversations collected from ShareGPT.31 Although different
questions share the same LLM decoder model, we have devised a GVQA
strategy to enhance the language model's capabilities and crafted
tailored prompts based on different problem types.

In our method, each key frame in the nuScenes dataset undergoes a series
of QA pairs. Initially, the six images of the nuScenes key frame are
encoded into feature maps using InternViT-6B. Next, in the Query module,
the image features interact with 96 learnable queries and the question
text from the QA pair. The Query module transforms the image tokens
generated by InternViT-6B into representations that are aligned with the
LLMs. Finally, the output from the Query module is fed into Vicuna-13B
to generate an answer. Notably, the answer generated in the previous
step is combined with the subsequent question to form a context-rich
question. By iterating this process, the pipeline progresses round by
round to accomplish the end-to-end autonomous driving (e2eAD) tasks step
by step.

This modular approach guarantees that each processing stage—from
visual encoding to vision-text alignment and ultimately language
generation—is optimized for its distinct function while maintaining
seamless integration throughout the entire pipeline. This architecture
not only elevates the system's capability to handle intricate visual and
linguistic inputs but also ensures a coherent flow of information,
enabling more precise and context-aware decision-making in autonomous
driving scenarios.


    < g r a p h i c s >


Figure 1. Overall architecture.

gvqa-logical-dependency


  §.§.§ 3.2. GVQA logical
dependency


The SimpleLLM4AD method encompasses a sequence of four stages, each
intricately linked by the logical dependencies of the QA pairs they
contain. As depicted in Fig. 2, the logical dependency of GVQA is
graphically represented, illustrating the interconnected nature of the
QA pairs throughout the process. The diagram consists of two primary
elements: nodes (N) and edges (E). Nodes (N) symbolize individual QA
pairs, and the edges (E) that connect them denote the logical
dependencies that exist between them. Specifically, the answer from a
preceding node (NP) serves as contextual information for
the subsequent node (NS).

In the context of AD, understanding this logical dependency is crucial
for the system's decision-making process. For instance, during the
Perception stage, the system must identify key objects in the current
scene that will be considered for future reasoning and driving
decisions. This initial identification sets the stage for the Prediction
stage, where the system assesses the motion status of objects and
predicts their potential future states.

The Planning stage then utilizes the information gathered during
Perception and Prediction to formulate a safe and effective course of
action for the ego vehicle. This involves considering the potential
actions that could be taken in response to the identified objects and
predicting the outcomes of those actions, including the likelihood of
collisions and the safety of the proposed maneuvers.

Figure 2 elucidates the logical flow from one stage to the next, with
each node representing a critical decision point or informational
milestone. For example, (c1,CAM_FRONT,714.3,503.6) represents the
identification and initial assessment of an object captured by the front
camera. The subsequent nodes then build upon this information, asking
questions about the object's movement status and how it might interact
with other objects in the scene, such as node
(c3,CAM_FRONT,1300.8,531.7), which could represent another vehicle or
significant obstacle.

The edges connecting these nodes indicate the flow of logic and the
progression of the system's thought process. For instance, the question
of whether object `c1' will be in the moving direction of object `c3' is
predicated on the answers derived from the previous nodes. Similarly,
the decision-making process regarding the ego vehicle's actions is
contingent upon the predictions and assessments made in the preceding
stages.

By mapping out these logical dependencies, the SimpleLLM4AD method
ensures a coherent and systematic approach to autonomous driving
decision-making. This structured methodology not only aids in the
development of more sophisticated AI systems but also enhances the
transparency and reliability of the decisions made by these systems.


    < g r a p h i c s >


Figure 2. GVQA logical dependency.

experiments


 §.§ 4. Experiments


datasets-and-metrics


  §.§.§ 4.1. Datasets and Metrics


dataset


  
Dataset

The datasets used for fine-tuning and evaluation is
DriveLM-nuScenes.<cit.> DriveLM-nuScenes is a
comprehensive dataset specifically designed for developing and
benchmarking autonomous driving models. It consists of a training set of
4072 frames and a validation set of 799 frames, providing a robust
foundation for training and testing. DriveLM-nuScenes incorporates both
scene-level descriptions and frame-level QA pairs, divided into three
categories: perception, prediction, and planning. This structure ensures
a comprehensive understanding of the driving scenarios by addressing
various aspects of the driving process.

Perception: This category involves questions related to the thorough
examination of the entire frame. While some questions are manually
annotated, prompts are designed to generate questions based on the
observational facets of objects within the scene, leveraging ground
truth from nuScenes and OpenLane-V2 datasets.

Prediction: This category includes inquiries regarding the future states
of key objects and the ego vehicle in the current frame, focusing on the
reasoning process behind these predictions. Due to the complexity of
these predictions, the answers are manually annotated.

Planning: This involves questions related to the subsequent actions of
the ego vehicle within the current frame. Similar to prediction,
planning questions are challenging and require manual annotation for the
reasoning process.

Each key object within the QA pairs is encoded as c tags in the format
c, CAM, x, y, where c is the identifier, CAM
indicates the camera, and x, y are the coordinates of the object's 2D
bounding box in the respective camera's coordinate system. The dataset
also includes a dictionary in each key frame, recording basic
information about key objects such as bounding box size, category,
moving state, and visual description.

metrics


  
Metrics

To evaluate the performance of our models, we use a set of
well-established metrics tailored to different tasks within the
autonomous driving domain. The metrics are divided into three main
categories: P1-3 VQA Metrics, Behavior Task Metrics, and Motion Task
Metrics.

P1-3 VQA Metrics: These metrics assess the performance of
perception, prediction, and planning question-answering (QA) tasks.
Common Visual Question Answering (VQA) metrics are employed, alongside a
new GPT score for a more nuanced semantic evaluation.


BLEU: Measures n-gram overlap between generated text and
reference text, focusing on precision but often insensitive to semantic
nuances.

ROUGE_L: Focuses on recall by calculating the longest common
subsequence between model outputs and reference answers.

METEOR: Considers precision, recall, stemming, synonymy, and
word order, offering a detailed alignment between model outputs and
references.

CIDEr: Utilizes a combination of BLEU and vector space models,
calculating semantic consistency through cosine similarity of n-gram
TF-IDF vectors.

SPICE: Parses text into a syntactic dependency tree and maps it
to a scene graph, computing F-scores to evaluate the alignment between
predicted and reference scene graphs.

GPT Score: Provided by ChatGPT, this metric assesses prediction
quality through a more rational score based on robust reasoning
capabilities, assigning a score between 0 and 100.


Behavior Task Metrics: These metrics evaluate behavior
predictions, focusing on classification accuracy of behavior, behavior
speed, and behavior steer.


Classification Accuracy: Evaluates the accuracy of behavior
predictions by comparing the predicted behaviors of the ego vehicle to
the ground truth. The ground truth trajectory points are categorized
into speed and steering behaviors, and the accuracy is calculated based
on the agreement between predicted and actual behaviors.


These metrics ensure a comprehensive evaluation of the model's
performance across various aspects of autonomous driving, from
perception and prediction to planning and behavior prediction.

implementation-details


  §.§.§ 4.2. Implementation
Details


In our method, the SimpleLLM4AD model is finetuned with DriveLM-nuScenes
dataset. We use the pretrained weights of InternViT-6B and keep it
frozen. QLLaMA and 96 queries are trainable during finetuing. The
pretrained Vicuna-13B could be totally frozen or tuned by
parameter-efficient fine-tuning (PEFT)<cit.> methods such
as LoRA<cit.>. The image resolution is set to 224 × 224.
We finetuned the model on NVIDIA GPU with a learning rate of 1e-4 and a
global batch size of 16.

test-results-on-drivelm-nuscenes


  §.§.§ 4.3. Test Results on
DriveLM-nuScenes


All models in Table 1 are fine-tuned on the training split of the
DriveLM-nuScenes dataset and tested on the test set of the same dataset.
The DriveLM baseline,<cit.> which fine-tunes
LLaMA-Adapter-V2<cit.>, achieves a final score of 32.4,
primarily due to its poor performance on the accuracy (acc.) metric. Cho
et al. fine-tune LLaVA-1.5<cit.> and
Cube-LLM,<cit.> resulting in a significant improvement
compared to the baseline.

Our method, SimpleLLM4AD, demonstrates a marked enhancement in both the
accuracy and language scores, leading to a final score of 52.7. The
partial result presentation is illustrated in Figure 3. This substantial
improvement highlights the effectiveness of our approach. The detailed
methodology and analysis of this result will be expounded in the
following section.


    < g r a p h i c s >


Figure 3. Result Presentation.

Table 1. DriveLM-nuScenes Benchmark. (Test set)

[]@llllllll@
Method     Acc.     ChatGPT     Match    
BLEU1     ROUGEL    
CIDEr     Final ScoreDriveLM baseline<cit.>     0.0     65.1     28.3     5.0     8.4    
9.9     32.4LLaVA-1.5<cit.>     38.5     53.5     26.1     15.8     14.3     30.3    
36.1Cube-LLM<cit.>     38.5     89.4     39.0    
16.3     20.4     31.3     50.1SimpleLLM4AD     66.5     57.4     35.3     75.7    
73.4     15.3     52.7

ablation


  §.§.§ 4.4. Ablation


During our exploration, we conducted training and inference of
SimpleLLM4AD under various settings. The differences among these
solutions primarily involve the treatment of prompts and the methods for
detecting key objects. The baseline performance shown in Table 1 was
obtained by using LLaMA-Adapter-V2 for evaluation on a validation set
that we created. This validation set was extracted from the original
training set by selecting one scene out of every six.

Table 2. DriveLM-nuScenes results of different settings. (Validation set
split by us)

[]@llllll@
Method     Acc.     Match     BLEU1    
ROUGEL    
CIDErBaseline     60.1     1.16     57.6     65.9     3.3Version A     62.7     2.94     66.8     68.4     14.1Version B     63.4     2.94     72.2     72.4     17.0Version C     67.7     3.43     74.8     71.9     15.0Version D     67.5     3.55     80.5     75.1    
33.2Version E     -     24.6     -     -    
-

Notice: In Table 2, the "match" only measures the box coordinate
matching; unlike in Table 1, where the "match" includes both box
coordinate matching and the ChatGPT score.

chain-of-thought


  
Chain of Thought

Chain-of-Thought (CoT) is an approach for prompting, in which one
includes the intermediate steps of reasoning within the prompt
(intermediate “thoughts”), besidesthe task
input/output.<cit.> CoT was shown to significantly improve
the capability of LLMs to solve problems without resorting to any model
updates.

In SimpleLLM4AD, we employ CoT by use the answer of NP
as contextual information for NS, and NS
would always be the next QA pair of NP in the dataset.
Comparing to the DriveLM baseline, Version A that utilizing CoT show a
remarkable improvement in acc. And language score.

In the DriveLM-nuScenes dataset, `What are the important objects in the
current scene?' question would always be the first question in each
frame, which we call it N0. In Version B, we apply
N0 + NP as context for each
NS.

graph-of-thought


  
Graph of Thought

While CoT only support unidirectional single branch of thought when
prompting LLMs step by step, graph of thought (GoT) enable current task
use any previous QA pairs as context and would allow any subsequent
questions refer to the results of current task. After trying different
logical dependency graphs, the prominent one is shown in Fig. 2. By
using GoT to arrange context like this, Version C get a prominent
improvement.

prompt-optimization


  
Prompt Optimization

Refined prompt would contribute to the performance of LLMs. In the
solution of DriveLM baseline, the context that pass to the subsequent
question would be previous Q + A, making the context redundant and hard
for LLMs to understand. Therefore, we optimize the simple Q + A format
of the contexts to simplify the context, making it easier and more
efficient for LLMs to use the previous information.

The original answer of N0 is:

`There is a red car to the front of the ego vehicle, a white SUV to the
front of the ego vehicle, a white sedan to the front of the ego vehicle,
a black sedan to the front of the ego vehicle, and a red light to the
front of the ego vehicle. The IDs of these objects are
c1,CAM_FRONT,714.3,503.6,
c2,CAM_FRONT,993.3,503.3,
c3,CAM_FRONT,1300.8,531.7,
c4,CAM_FRONT,892.5,507.5, and
c5,CAM_FRONT,712.6,361.8.'

As you can see, the color/class information and the coordinate
information of the same object are separately described in two sentence,
which could make LLMs confused. Therefore, when using it as the context
for subsequent question, we format the answer as:

`c1,CAM_FRONT,714.3,503.6 is a red car to the
front of the ego vehicle.'

We only give the information of the object mentioned in the current
question instead of all objects detected in N0.

Besides, we modify other QA pairs into declarative sentence when used as
a context. For example, QA pair like `Would
c1,CAM_FRONT,714.3,503.6 be in the moving
direction of the ego vehicle? No.' would be modified to
`c1,CAM_FRONT,714.3,503.6 would not be in the
moving direction of the ego vehicle.' when used as a context.

By reformatting the QA pairs, brief and informative context provide
improvement of the language score in Version D.

detection-of-the-key-objects


  
Detection of the Key
Objects

Format_instruction is "Input six images in turn. The first of six
images is CAM_FRONT, which is in the front of
the ego vehicle. The second of six images is
CAM_FRONT_LEFT, which is in the front left of
the ego vehicle. The third of six images is
CAM_FRONT_RIGHT, which is in the front right
of the ego vehicle. The fourth of six images is
CAM_BACK, which is in the back of the ego
vehicle. The fifth of six images is
CAM_BACK_LEFT, which is in the back left of
the ego vehicle. The sixth of six images is
CAM_BACK_RIGHT, which is in the back right
left of the ego vehicle. number, number is
object box center coordinate in the image (1600*900)."

To enhance the performance, we have intergrated Format_instruction to
guide LLM. To accurately detect and classify objects, we leverage
existing, well-established detection networks, such as dinov2, which
provide us with robust target detection output. In addition, we have
trained a specialized detection classification network that goes beyond
mere object detection. This network is fine-tuned to identify key
attributes of the detected objects, including their color, precise
position, and even orientation. This approach allows us to generate
detailed descriptions such as "These is a red car to the front of the
ego vehicle, and the box center is [714.3,503.6]."

By incorporating this rich information about the location, color, and
class of objects within the ego vehicle's environment, we have achieved
a significant improvement in the match score of Version E. Especially,
the positioning accuracy of the box center has significantly improved to
24.6%.

conclusion


 §.§ 5. Conclusion


In this paper, we present SimpleLLM4AD, a multi-modal language model
that achieves competitive performance in complex driving scenarios. The
integration of VLM allows for more nuanced and context-aware
decision-making, significantly improving the robustness and reliability
of the autonomous driving system. This approach also highlights the
potential for LLMs to enhance multimodal AI applications, paving the way
for further advancements in the field of e2eAD. In the future, we will
continue to optimize box positioning as well as the alignment of
multi-frame vision-language large models and other related directions.


unsrt

<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>
<cit.>







 Paper Title: Robust Box Prompt based SAM for Medical Image Segmentation.





 

Y. Huang et al.


1National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Medical School, Shenzhen University, China

nidong@szu.edu.cn 

2Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, China

3Shenzhen RayShape Medical Technology Co., Ltd, China

4Centre for Computational Imaging and Simulation Technologies in Biomedicine (CISTIB), University of Leeds, UK

5First Affiliated Hospital, Southern University of Science and Technology, Shenzhen People’s Hospital, China

Robust Box Prompt based SAM for 
 Medical Image Segmentation
    Yuhao Huang1,2, Xin Yang1,2 Han Zhou1,2 Yan Cao3 
Haoran Dou4 Fajin Dong5 Dong Ni1,2()
    July 2024
==========================================================================================



§ ABSTRACT 
The Segment Anything Model (SAM) can achieve satisfactory segmentation performance under high-quality box prompts.
However, SAM's robustness is compromised by the decline in box quality, limiting its practicality in clinical reality.
In this study, we propose a novel Robust Box prompt based SAM (RoBox-SAM) to ensure SAM's segmentation performance under prompts with different qualities.
Our contribution is three-fold.
First, we propose a prompt refinement module to implicitly perceive the potential targets, and output the offsets to directly transform the low-quality box prompt into a high-quality one.
We then provide an online iterative strategy for further prompt refinement.
Second, we introduce a prompt enhancement module to automatically generate point prompts to assist the box-promptable segmentation effectively.
Last, we build a self-information extractor to encode the prior information from the input image.
These features can optimize the image embeddings and attention calculation, thus, the robustness of SAM can be further enhanced.
Extensive experiments on the large medical segmentation dataset including 99,299 images, 5 modalities, and 25 organs/targets validated the efficacy of our proposed RoBox-SAM.






§ INTRODUCTION


The Segment Anything Model (SAM) has marked a significant breakthrough in image segmentation, due to its powerful zero-shot performance <cit.>.
SAM was trained on a large annotated dataset with 11M images and 1B masks based on the Vision in Transformer (ViT).
It supports different types of prompts, including point, box, text, and mask. 
Recent studies have widely explored the potential of applying SAM to the medical imaging domain <cit.>, which demonstrates that SAM can outperform the task-specific state-of-the-art conventional deep models after fine-tunning with additional medical images <cit.>.

Most previous studies equipped the SAM with high-quality prompts, e.g., the center point or tight box of the target.
However, this assumption typically cannot be held in clinical practice, where numerous low-quality prompts commonly occur.
As shown in Fig. <ref>, SAM prompted with high-quality boxes can achieve good performance, whereas it will easily generate poor segmentation after randomly shifting the promoting boxes. 
Hence, enhancing the robustness of SAM against imprecise prompts is crucial for its reliability.





Segmentation ability improvement of SAM.
Fine-tuning SAM using medical datasets to enhance medical image segmentation efficacy is an important field of research. 
Ma et al. ﬁxed the prompt encoder, and tuned the parameters in both image encoder and mask decoder during training <cit.>.
Wu et al. then introduced the adaptor modules with few trainable parameters to capture medical knowledge <cit.>.
Xie et al. reconstructed the mask decoder within SAM to develop a few-shot fine-tuning strategy <cit.>.
Moreover, Semantic-SAM <cit.> and HQ-SAM<cit.> were proposed to improve the segmentation quality. 
However, these studies aimed to improve the segmentation performance of SAM in different scenarios. 
They assumed to directly use the high-quality tight box as prompt, lacking the exploration of the low-quality prompt.


Robustness of SAM. 
Several studies have evaluated SAM's robustness under various corruptions and adversarial attacks <cit.>. 
However, their robustness tests were conducted at the image level rather than the prompt level. 
In <cit.>, the authors randomly shifted the tight box to mimic the practical operation.
They concluded that the SAM is sensitive to prompt randomness, especially box prompts.
The higher the random shifts, the more serious the performance degradation.
Although they have noticed the weak robustness of SAM under different prompt shifts, they did not introduce solutions to solve this problem.

One study observed that when subjected to low-quality prompts, SAM's mask decoder encounters an attention drift issue, leading to the cross-attention module activating inaccurate regions <cit.>.
To address this issue, stable SAM (SSAM) leveraged a deformable sampling plugin to shift attention to the target region.
However, SSAM only predicted the indirect feature offset to enhance the robustness.
Besides, it was developed and validated on natural images characterized by clear target boundaries, distinguishable foregrounds and backgrounds, and color encoding. 
These characteristics significantly differ from those of medical images, which may present unique challenges to the SSAM.


In this study, we introduce a novel robust box prompt based SAM named RoBox-SAM, aimed at making SAM robust to various prompt qualities.
Our contribution is three-fold.
First, we introduce an intuitive prompt refinement module (PRM) designed for potential target perception and directly predict box prompt offsets to refine low-quality prompts.
We also design an online refinement strategy to optimize the box prompt quality iteratively, and further achieve robustness improvement.
Second, we propose the prompt enhancement module (PEM), and leverage the movement information between the noisy and optimized prompts to output the potential point prompts automatically.
Finally, we extract intrinsic features from the input image via a self-information extractor (SIE), to enrich the image embeddings and refine the cross-attention calculation within the decoder. 
Validated on the large medical segmentation dataset, we demonstrate that RoBox-SAM is robust to box prompts of varying qualities, and general for different medical modalities and targets, achieving satisfactory results.





§ METHODOLOGY


Fig. <ref> shows the schematic view of our proposed Robox-SAM.

In stage 1, to eliminate the effects of low-quality prompts, the proposed PRM first predicts the offsets to transfer the input box prompt into a high-quality one.

In stage 2, we adopt the prompt correction knowledge to output potential points via the proposed PEM.

In the last stage, we leverage the prior knowledge encoded by SIE to optimize the image embeddings, and update the attention calculation in the mask decoder.
This can further optimize the feature activations under suboptimal box prompts, thus, improving the robustness of SAM.



Original SAM.
SAM mainly consists of three parts: image encoder, prompt encoder, and mask decoder <cit.>.
The image encoder is used to extract the image embedding, while the prompts are encoded using different methods (e.g., point and box: positional encoding <cit.>, text: CLIP <cit.>). 
Before being fed to the mask decoder, the image embedding should add the positional encoding, while the prompt tokens should concatenate with two output tokens, i.e., IoU token and mask token, to form the embedding (e) and tokens (t), respectively.
The mask decoder mainly contains two Transformer layers, and each layer contains four modules: 1) self-attention of tokens, 2) token-to-image cross-attention, 3) Multi-Layer Perception (MLP) on tokens, and 4) image-to-token cross-attention.
For example, the token-to-image cross-attention can be formulated as C_Att(t,e)=softmax(Q(t),K(e)^T/√(d_k))· V(e), where d(k) represents the dimension of the keys.
Finally, the mask token is separated from tokens and operated with the final up-scaled image embedding to predict multiple outputs.
Besides, the IoU token predicts the IoU score for each mask.



PRM for Target Perception and Box Prompt Correction.
Inaccurate box prompts may cause the SAM's mask decoder to activate imprecise regions, and the cross-attention between image features and tokens may further accumulate the errors, resulting in unstable segmentation <cit.>.

We propose the PRM to learn and perceive the target location, and predict the offsets to update the input prompts.
Given the image and prompt features (f_img, f_p), cross-attention can be performed to obtain f^*=f_img+C_Att(f_img,f_p).
Then, f^* will be inputted to the light-weighted PRM, outputting the offsets Δ(o)={Δ x_1, Δ x_2, Δ y_1, Δ y_2}.
The input box prompt (p_c={x_1,x_2,y_1,y_2}) can be updated to p_c^*=p_c+Δ(o).
The loss function can be written as L_o=L_1(p_c^*, p_gt), where p_gt represents the coordinate of GT tight box.
This supervision can drive the development of SAM's target perception ability, outputting reliable offsets to make the input box prompt move and match the potential target.
Moreover, during testing, the box refinement process can be interactive. Specifically, the output refined box in the current step can be considered as the input of the next step.
The iteration will terminate when ∑Δ(o)=0 or iterating over k=5 steps.
This interactive refinement strategy can further improve the model's accuracy and robustness.


PEM for Automatic Point Prompt Generation.
We only consider the box as the initial input prompt. 
However, we believe that the point prompts can assist the box prompts in some cases, especially negative points for restraining over-segmentation.
The above box correction process is intuitive and effective in revealing the potential region for point prompt generation.
Refer to stage 2 (right-down corner) in Fig. <ref>, given the original (green, A) and refined (red, B) boxes, and the annotated ground truth (GT) mask (orange, C), the non-GT region (i.e., (A∪ B)-C) indicated by yellow arrows can be treated as the potential negative area.
This region can be flattened into a one-dimensional vector and n uniformly sampled points can be taken as GTs (p_gt).
The CNN-based PEM (ℳ) can learn the potential points by inputting image-prompt attention features before and after refinement (p_pre=ℳ(f_r^*-f^*)).
Specifically, it generates n=5 points and matches them with sampled GT points one by one to find the combination with the smallest distance errors to calculate the L2 loss: L_p=||(p_pre,p_gt)||_2


Self-information Mining for Feature and Attention Optimization.
Utilizing the prior knowledge of the input image can enhance the accuracy and robustness of the segmentation model.
In the self-information pool, we provide several components that contain useful shape and edge information.
As shown in the green block of Fig. <ref> (stage 3), yellow arrows in the canny-based sketch <cit.> and SLIC-based superpixel <cit.> represent the location and boundary of the target.
We also adopt the fast Fourier Transform to transfer the image to the frequency domain <cit.>, including amplitude and phase parts.
Specifically, the low-frequency amplitudes are masked to highlight the details and edges of the image.
This information was encoded by the proposed SIE, and the output dimension was mapped to the same as the image embedding.
Then, the image embedding can be re-calculated by f'=f^*+f_s to fuse and emphasize the self-information.
Finally, f' will be input into the cross-attention module for attention re-calculation and optimization.
Thus, though the refined box prompts may still include noise, the inherent self-information can help remove it via attention update, and assist in improvement in terms of both accuracy and robustness.







§ EXPERIMENTAL RESULTS


Materials and Implementations. 
We validated the RoBox-SAM on a large medical segmentation dataset, including 99,299 images/slices, 5 modalities, and 25 types of targets.
This dataset contains 14 public and 5 in-house datasets (marked with ^*).
Specifically, for CT datasets <cit.>, the targets include chest (heart, lung, trachea) and abdomen (aorta, bladder, gall bladder, kidney, liver, pancreas, spleen, stomach).
MRI datasets contain the brain <cit.>, brain tumor <cit.>, knee (femur and tibia) <cit.>, and prostate <cit.>.
Ultrasound datasets have nine objects, including breast nodule^*, eyes^*, femur^*, heart <cit.> (left atrium, left ventricle endocardium, left ventricle epicardium), kidney pelvis^*, nuchal translucency^*, and thyroid nodule <cit.>.
We also considered two other targets from two modalities, i.e., Colonoscopy: polyp <cit.> and Fundus: OpticDisc <cit.>.
Approved by the local IRB, the in-house ultrasound datasets were collected and annotated by an experienced sonographer under strict quality control.
CT/MR volumes were sliced into 2D images similar to <cit.>.
The whole dataset was split into 6:1:3 for fine-tuning, validation, and testing, respectively.


We implemented RoBox-SAM in Pytorch, using an NVIDIA A40 GPU.
The batch size, learning rate, and total epoch are set to 16, 1e-4, and 20, respectively.
SAM's backbone was ViT-H.
GT box was generated by the adjacency box of the annotated mask.
Input box prompts were randomly shifted by 0-30% based on the GT box to evenly mimic the human operations with varying experiences/annotation preferences, with the condition that their IoU>0.5.
We followed the SAM to use cross-entropy and dice losses to optimize the model via AdamW optimizer.
We further added L_o and L_p to assist the model learning.
The proposed PRM, PEM, and SIE are two-layer CNNs with linear projection.
They encode the high-dimensional features and output offsets, points, or low-dimensional features to satisfy specific requirements.
As shown in Fig. <ref>, during fine-tuning, the original components in SAM were frozen, whereas other parts were learnable.
During testing, all the parameters were frozen, and the mask decoder will output three predictions with corresponding predicted IoU scores.
The mask with the highest IoU prediction was selected as the final output.






Quantitative and Qualitative Analysis.
We performed N=5 times random testing, and reported the average performance to obtain stable and reliable results.
Evaluation metrics in terms of accuracy (DICE, %) and robustness to prompt (PR, %: ∑_i=1^N(DICE(M_i, M_u))/N, similar to <cit.>), were used to validate the segmentation performance.
M_i represents the mask in i-th testing, while M_u is the union mask region of all N masks.
The higher the PR score, the more consistent the segmentation results are across box prompts of different qualities.


Table <ref> compares the RoBox-SAM with 7 methods, including SAM <cit.> (different model sizes: ViT-B/L/H), MedSAM <cit.>, MedSAM_boxaug, MedSAM_offset, and HQ-SAM <cit.>. 
All the MedSAM-based methods and HQ-SAM are based on ViT-H and fine-tuned using our medical datasets.
MedSAM_boxaug adds the random box augmentation strategy during fine-tuning, and MedSAM_offset trains an additional detector to predict the offsets for prompt correction before inputting to SAM. 
Based on the first five experiments, we concluded that enlarging the model size, fine-tuning using medical images, and augmenting with random boxes can optimize model robustness in most cases.

However, comparing MedSAM with MedSAM_offset, we found that adding naive offset prediction cannot steadily improve robustness, with 3/5 modalities having performance drops.
RoBox-SAM achieves the best performance among all the competitors (p<0.05), including the strong HQ-SAM, showing its efficacy and superiority in model robustness improvement.
Besides, RoBox-SAM is economical, has low training learnable computational overhead (<1M params), and takes only ∼0.6s per testing image.






We also performed ablation studies to evaluate the contribution of each proposed module in Table <ref>.
Specifically, we treated MedSAM as baseline, and progressively added the proposed modules to observe the performance changes.
It can be seen that PRM contributes ∼6% and ∼8% improvements to DICE and PR scores, respectively.
Adding PEM, the metrics can be increased further.
This illustrates that the generated point prompts can effectively assist box prompts in achieving better segmentation. 
By aggregating the prior self-information (RoBox-SAM), the model's performance can achieve consistent improvement across the five modalities.
It is highlighted that the performance improvement brought by each module is statistically significant with p<0.05.





In Table <ref>, we further decreased the quality of box prompts by progressively increasing the random shift max to 30%.
This can verify the model's performance under extreme conditions.
The lines GT mean that the tight boxes of objects are taken as input box prompts.
In this situation, although the PRM may not output large offsets, point prompt enhancement and self-information mining can still improve the segmentation.
Thus, RoBox-SAM shows a slight improvement compared with MedSAM under high-quality tight box prompts.
As the box prompt quality continued to decline, the accuracy and stability of MedSAM severely degraded.
For example, CT targets dropped about 50% in DICE values (89.06%→49.98%).
The performance of RoBox-SAM was less affected by low-quality box prompts (e.g., CT: 5.26%↓ in DICE).
This shows that RoBox-SAM can cope well with extreme box prompts and output more stable results than MedSAM.
Moreover, we provided the results of RoBox-SAM (Iter), and found that the interaction strategy can help boost the DICE and PR scores, especially for extremely low-quality box prompts (>1% significant improvement).

Fig. <ref> and Fig. <ref> are qualitative results, with DICE scores shown in the right-down corners of the images.
Fig. <ref> visualizes the segmentation results of compared methods and our proposed RoBox-SAM.
It can be observed that the original SAM shows poor performance under low-quality prompts.
RoBox-SAM performs better than other methods, approaching the upper-bound (RoBox-SAM_GT) and GT.
Fig. <ref> (a)-(i) shows the box refinement and performance comparison (in each sub-figure, from left to right shows GT, MedSAM, and RoBox-SAM results).
Given low-quality box prompts, MedSAM produces poor prediction, while RoBox-SAM can refine the box and output satisfactory results.
Fig. <ref> (j)-(k) shows the interactive prompt refinement process.
RoBox-SAM performs worse in the first-round refinement, but achieves good correction finally due to the iterative refinement strategy.
Fig. <ref> (l)-(m) reveals the robustness of RoBox-SAM. In each case, given three different random prompts (orange boxes), RoBox-SAM can refine them to the green ones, and output stable results. 





§ CONCLUSION


In this paper, we propose RoBox-SAM for improving SAM's robustness to box prompts of different qualities.
We first directly refine the input box with poor quality via the proposed PRM to obtain a high-quality box prompt.
Then, we automatically generate potential point prompts through REM to assist the box-promptable segmentation.
Last, we mine the self-information in the image to further ensure robustness via attention optimization.
Extensive experiments on the large medical dataset validate the efficacy of RoBox-SAM.
In the future, we will extend the framework to more modalities and 3D segmentation tasks.




  §.§.§ 

This work was supported by the grant from National Natural Science Foundation of China (12326619, 62101343, 62171290), Science and Technology Planning Project of Guangdong Province (2023A0505020002), and Shenzhen-Hong Kong Joint Research Program (SGDX20201103095613036).



  §.§.§ 

The authors have no competing interests to declare that are relevant to the content of this article.


splncs04




 Paper Title: Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation.





 



OptTrajDiff







Y. Wang et al.




University of California, Berkeley, USA The University of Texas at Austin, USA Stellantis, France Stellantis, Italy 

{yixiao_wang, wzhan}@berkeley.edu



Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation
    Yixiao Wang1 Chen Tang1,2 Lingfeng Sun1 Simone Rossi3 Yichen Xie1Chensheng Peng1Thomas Hannagan3Stefano Sabatini3Nicola Poerio4Masayoshi Tomizuka1Wei Zhan1
    August 5, 2024
===============================================================================================================================================================




§ ABSTRACT 
 Diffusion models are promising for joint trajectory prediction and controllable generation in autonomous driving, but they face challenges of inefficient inference steps and high computational demands. To tackle these challenges, we introduce Optimal Gaussian Diffusion (OGD) and Estimated Clean Manifold (ECM) Guidance. OGD optimizes the prior distribution for a small diffusion time T and starts the reverse diffusion process from it. ECM directly injects guidance gradients to the estimated clean manifold, eliminating extensive gradient backpropagation throughout the network. Our methodology streamlines the generative process, enabling practical applications with reduced computational overhead. Experimental validation on the large-scale Argoverse 2 dataset demonstrates our approach's superior performance, offering a viable solution for computationally efficient, high-quality joint trajectory prediction and controllable generation for autonomous driving. Our project webpage is at <https://yixiaowang7.github.io/OptTrajDiff_Page/>
  






§ INTRODUCTION


The diffusion model is a class of generative models capable of representing high-dimensional data distributions. In particular, it has demonstrated strong performance in trajectory prediction and generation for autonomous driving <cit.>. In contrast to traditional trajectory prediction <cit.> and generative models<cit.>, the unique advantage of diffusion models lies in their ability to deform the generated trajectory distribution to comply with additional requirements at inference stage via gradient-based guided sampling. Notably, it is achieved without extra model training costs. This ability to perform controllable trajectory generation enables various useful applications, such as enforcing additional realism constraints on predicted trajectories, generating directed and user-specified simulation scenarios. 

However, computational efficiency is a crucial bottleneck hindering the practical application of diffusion models in autonomous driving. Real-time inference is essential for trajectory prediction, as it provides timely forecasts of surrounding agents' behavior, enabling safe and efficient planning in dynamic traffic scenarios. The high demand for inference speed, coupled with limited onboard computational resources, makes it infeasible to deploy diffusion models for onboard trajectory prediction. While simulations do not occur onboard, lightweight models are still preferred to streamline the closed-loop training and evaluation pipelines. The heavy computational cost is mainly attributed to the following two aspects:

Computationally Expensive Reverse Diffusion. At the inference stage, the diffusion model samples from standard Gaussian distribution and gradually denoises the sample through dynamics described by a Stochastic Differential Equation (SDE) <cit.>, aiming to eventually obtain a sample as if drawn from a target data distribution. The target data distribution can be significantly different from a standard Gaussian distribution, necessitating a large number of denoising steps to yield good performance. Prior works have attempted to reduce the reverse diffusion steps through adaptive noise schedule <cit.>, fast samplers <cit.>, or distillation <cit.>. However, the fixed standard Gaussian prior poses a challenge in accelerating the reverse diffusion process without violating the SDE, which can inevitably compromise the generation quality.

Computationally Expensive Guided Sampling. Controllable generation is typically achieved by guiding the denoising process with the gradient of a guidance cost function. The guidance cost function encodes the desired characteristics of the generated data, which is typically defined on the clean data manifold in trajectory prediction and controllable generation problems. However, guided sampling intends to inject the gradient of the guidance cost function into the series of noisy data at intermediate diffusion steps. It requires a forward pass to estimate the clean data first and then back-propagating throughout the entire network to estimate the gradient with respect to the intermediate noisy data <cit.>, which is very computationally intensive.

Targeted at these challenges, we take a step further to improve the computational efficiency of diffusion models while maintaining their performance for joint trajectory prediction and controllable generation tasks. Specifically, we propose two novel solutions for efficient reverse diffusion and guided sampling respectively. First, we present Optimal Gaussian Diffusion (OGD) to accelerate the reverse diffusion process. At the inference stage, instead of a standard Gaussian distribution far away from the desired data distribution, OGD starts from an optimal Gaussian distribution, which minimizes the distance to intermediate data distribution at a specific noise level, cutting down the diffusion steps before that. We show that we can analytically estimate such an optimal Gaussian distribution and also, an optimal perturbation kernel distribution, at any noise level from the statistics of the data distribution. It allows flexible tuning of the diffusion steps at the inference stage, without the need to train any additional models <cit.>. We further derive a practical implementation of OGD for joint trajectory prediction and generation, where the optimal Gaussian prior is computed using the mean and variance of marginal trajectory distributions estimated with a pre-trained marginal trajectory prediction <cit.> model.

Second, we propose Estimated Clean Manifold Guidance (ECM) to accelerate guided sampling for controllable generation. To save the computational cost due to estimating the guided gradient on the noisy data, we aim to directly inject the gradient into the clean data manifold without lengthy backpropagation. ECM is motivated by the insight that guided sampling can be regarded as a multi-objective optimization problem on the clean data manifold: The first objective is to maximize the likelihood of the samples on the estimated real data distribution; the second objective is to achieve low guidance cost. ECM hierarchically solves this multi-objective problem without backpropagation throughout the entire diffusion model. We show that it leads to faster inference time and much better performance than existing approaches.  Also, to tackle the challenges imposed by the multi-modal nature of vehicle interactions, we propose to warm-start the multi-objective optimization problem with reference joint trajectory points estimated using a marginal trajectory predictor. We refer to the complete algorithm as Estimated Clean Manifold with Reference Joint Trajectory (ECMR). 

To evaluate the proposed OGD and ECM methods on real-world tasks, we implement the OGD model leveraging a pre-trained marginal prediction model, QCNet <cit.>, and conduct extensive experiments on the Argoverse 2 dataset. We show that OGD can achieve better joint trajectory prediction performance than vanilla diffusion with significantly fewer diffusion steps—it only takes 1/12 of the diffusion steps used by the vanilla diffusion model. OGD also achieves outstanding prediction accuracy compared to non-diffusion joint prediction models. In addition, ECMR, coupled with OGD, can generate samples with significantly lower guidance costs with the same level of realism scores compared to conducting controllable generation on the vanilla diffusion model using existing guided sampling approaches used in autonomous driving <cit.>, but using around 1/5 of their inference step. 






§ RELATED WORK


Diffusion models. Diffusion models have demonstrated their ability to produce high-quality, diverse samples in a variety of applications, such as image, video, and 3D generation <cit.>. Recently, diffusion models have been applied to trajectory prediction in autonomous driving. It shows great performance on representing future trajectory distribution <cit.>. However, diffusion models need to run lengthy reverse diffusion processes to generate high-quality samples <cit.>. This makes it hard to apply diffusion models to trajectory prediction in autonomous driving since it requires in-time prediction for the downstream planning module to promptly respond to dynamically changing traffic scenarios. Previous works <cit.> mostly focused on mitigating this issue by investigating how to solve reverse SDE in a faster manner. In addition, similar to ours, some works sought alternative initialization of the reverse diffusion process to achieve faster inference. For example, <cit.> initialized the reverse diffusion process with samples generated by another generative backbone network. However, the backbone network is not deliberately trained for the reverse diffusion process. <cit.> proposed to learn the initial diffusion with a generative adversarial network (GAN) <cit.>. However, it requires training a large additional model with a complex training procedure. Also, it requires specifying the reverse diffusion time as a hyperparameter prior to GAN training, which is hard to tune. 

Guided sampling. 
Diffusion models have been successfully used to tackle controllable tasks through guided sampling, such as image inpainting <cit.> and motion planning <cit.>. A notable feature of diffusion models with guided sampling is their ability to condition the generation process on the user's preference, which was not available during the training phase. In the driving domain, recent works used guided sampling to generate controllable traffic <cit.>, and controllable pedestrian animation <cit.>. Our work belongs to the prior category, where user-specified guidance cost is used to guide generation in the trajectory space. In this case, the guidance cost function encodes certain desired properties of the generated trajectories. Thus, it is normally defined on the realistic trajectory samples, which are on the so-called clean manifold, instead of the noisy manifold containing the intermediate noisy data. Some works attempted to learn the guidance cost function on the noisy manifold <cit.>. Otherwise, it will lead to numerical instability when evaluating guidance cost at intermediate noisy data <cit.>. To avoid the additional computational costs introduced by a learned guidance cost function, <cit.> proposed to project the intermediate noisy data into the clean manifold through the diffusion model, and evaluate the guidance cost on the projected point. This approach requires back-propagating throughout the entire diffusion model, which is also computationally intensive. 

Trajectory prediction.
In autonomous driving <cit.>, it is vital to precisely forecast how other participants in traffic will move in the future so that ego vehicle can plan a safe and efficient trajectory to execute in the future. Marginal trajectory prediction is used to predict the trajectory distribution for single vehicle and recent works involve kinematic constraints of the vehicles, restrictions of complex topology of roads, and interaction from the surrounding vehicles <cit.>. Recently, joint trajectory prediction has attracted the attention of researchers. It consists in predicting the joint future trajectories for all agents so that these trajectories are consistent with one another <cit.>, an aspect which marginal trajectory prediction does not consider. Scene-Transformer <cit.> uses a fixed set of learnable scene embeddings to generate corresponding joint trajectories for all the vehicles in the given scene. Models like M2I <cit.>, and FJMP <cit.> adopt a conditional approach, predicting the motions of other agents based on the movements of controlled agents. Diffusion model <cit.> has also been used to predict the joint trajectory. However, joint trajectory prediction is still a challenging problem since the complexity increases exponentially with the number of vehicles in the scene. The efficiency problem becomes more serious when using diffusion model to predict the joint trajectory distribution <cit.>, and largely limits the application of diffusion models in autonomous driving. 



§ PRELIMINARIES



 §.§ Diffusion Models


The diffusion process continuously perturbs the unknown data distribution p_data with a perturbation kernel and generates intermediate data with a given diffusion time T. Denote the distribution of the time-dependent intermediate nosiy data _t as p_t(_t), t ∈ [0,T], where p_0(_0)=p_data is the clean data distribution and _0 is the clean data. The series of intermediate data _t are generated through the Stochastic Differential Equation (SDE) <cit.>:

    d_t=f(_t,t)d t+g(t) d , _0 ∼ p_data = p_0(_0),

where f(·, t) is the drift coefficient, g(t) is the diffusion coefficient, and  is the Wiener process. We can recover p_data from reverse-time SDE

    d_t=[f(_t,t)-g(t)^2 ∇__tlog p_t(_t) ]dt+g(t)d, _T ∼ p_T(_T),

where  is another Wiener process, d t is negative timestep. 

In order to solve <ref>, we first need to approximate p_T(_T). In previous works, p_T(_T) is typically approximated by some prior distributions p_prior which contain no information of p_data. In this paper, we adopt the setting of Variance Preserving (VP) SDE <cit.>. In VP-SDE, p_T(_T) ≈ p_prior when T →∞. The perturbation kernels are in the form of p_t(_t|_0) = 𝒩(_t;√(α_t)_0, (1-α_t)_p) where scalar α_t is diffusion schedule parameter, |_p|=1. A common choice for _p is the identity matrix 𝐈 <cit.>. 

Second, we need to approximate ∇__tlog p_t(_t) for all t∈(0, T]. Some works solve score-matching problem <cit.> and learn a score function _θ(_t,t) to approximate ∇__tlog p_t(_t). In this paper, we follow the practice in DDPM <cit.> 
to learn the noise  using a network _θ(_t,t):

    _θ𝔼__0 ∼ p_data, ∼𝒩(0,_p)||-_θ(_t,t)||_2^2

where _t = √(α_t)_0+√(1-α_t), _θ(_t,t) =-√(1-α_t)_θ(_t,t). With these two approximation, we can learn q_θ(_0) to estimate unknown data distribution p_0(_0) solving <ref> from t = T to t=0.



 §.§ Guided Sampling


In prior diffusion-based controllable generation frameworks <cit.>, controllable generation is achieved via biasing of the score function for sampling:

    ∇__tlog[p_t(_t)exp(-𝒞(_t))]=∇__tlog p_t(_t) - ∇__t𝒞(_t)

where 𝒞(·) is the guidance function. It requires estimating the guidance gradient with respect to the noisy data _t. Some approaches introduce an additional neural network to approximate 𝒞(·) at different noisy levels <cit.>. The additional neural network imposes additional training costs and heavier computational burden at the inference stage. To this end, some works define an analytical guidance function (·) on the clean data _0. They first estimate _0=f_θ(_t) based on _t with the diffusion model, then calculate 𝒞(_t) as (_0). However, when taking the gradient of (_0) with respect to _t, we get 

    ∇__t(_0)=∇__0(_0) ·∇__tf_θ(_t).

Estimating ∇__tf_θ(_t) requires backpropagating throughout the entire diffusion model, i.e., f_θ(_t). It requires heavy computing resources and GPU memory.



 §.§ Trajectory Prediction and Controllable Generation

Joint trajectory prediction aims to predict the future joint trajectories _0 for all the vehicles in the scene, conditioned on context information . It can be regarded as a conditional generation task where the goal is to train a generative model to approximate the distribution p_0(_0|). For simplicity, we omit  and represent p_0(_0|) as p_0(_0). We denote n as the number of vehicles in the same scene and _0,i as the future trajectory for vehicle i, i∈{1,2,...,n}, so _0=[_0,1,_0,2,...,_0,n]. Joint trajectory prediction can be very challenging. The complex interactions among vehicles, especially in highly interactive and dense traffic, result in a complicated high-dimensional p_0(_0), which is difficult to accurately model with lightweight and computationally efficient models. A simplified solution is to approximately decompose the joint trajectory distribution into marginal ones, i.e., p_0(_0)≈∏_i^n p_0(_0,i). It leads to the marginal trajectory prediction task, which has been extensively studied with mature solutions <cit.>. One drawback is that it omits the interactions among vehicles in the predicted horizon, which leads to large errors in highly interactive scenes.

Controllable trajectory generation is closely related to trajectory prediction. In addition to generating realistic trajectory samples resembling the ground-truth p_0(_0), controllable generation imposes an additional objective—the generated trajectories should comply with specified guidance cost function (·). We term the former objective as realism and the latter as guidance effectiveness. The guidance cost function (·) can be some goal points of the vehicles, kinematic constraints, etc, which are mostly defined on the clean manifold rather than on the noisy data when it comes to diffusion-based controllable generation.  




§ METHODOLOGY




 §.§ Optimal Gaussian Diffusion


As discussed in <ref>, diffusion models typically select a non-informative prior distribution p_prior, such as a standard Gaussian, as the initial data distribution for the reverse diffusion process. Such a non-informative p_prior is reasonable since p_T(_T) converges to it when T →∞. However, it also means that a large T is required at inference time to yield good performance, which undermines computational efficiency and limits its wide real-time applications in autonomous driving. In this section, we aim to investigate a practical solution to tackle this challenge for joint trajectory prediction. Given the inherent limitation imposed by a non-informative prior, the key question we look into is: can we 
instead adopt an informative prior so that we can obtain the same level of performance with much smaller reverse diffusion steps?

First, we still consider a Gaussian prior, but with learnable parameters, i.e., we parameterize p_
prior as q_ϕ(T)=𝒩(,), where  and  are learnable. We aim to optimize  and  to enhance the generation performance for small T. We got inspiration from <cit.>, where an upper bound of the Kullback–Leibler divergence of the clean data distribution p_0(_0) and learned distribution q_θ(_0) was derived as a function of the diffusion time T: 

    KL[p_0(_0)||q_θ(_0)]≤𝒢(_θ, T)+KL[p_T(_T)||p_prior]

where 𝒢(_θ, T) is the positive accumulated error between ∇_log p_t(_t) and _θ(_t,t) from 0 to T <cit.>. Note that 𝒢(_θ, T) is an accumulated error so 𝒢(_θ, T_1) ≤𝒢(_θ, T_2), T_1 ≤ T_2. If we can achieve lower KL[p_T_1(_T_1)||q_ϕ(T_1)], then a tighter upper bound can be obtained. This opens up the possibility to achieve better performance with less diffusion time. Thus, we propose to optimize the prior distribution by minimizing KL[p_T(_T)||q_ϕ(T)]. As shown in <ref> (See proof in <ref>), it turns out that the optimal μ and Σ can be expressed analytically as functions of the ground-truth data statistics. In addition, we find that we can further minimize the target KL divergence if we set a learnable _p in the perturbation kernel p_t(_t|_0) = 𝒩(_t;√(α_t)_0, (1-α_t)_p), whose optimal value can also be expressed as a function of the data statistics. 



 
Denote _d and _d as the mean and variance of p_data. Denote ^*(i,j) and _p^*(i,j) as the element at ith row and jth column of matrix ^* and _p^*. The optimal solution to minKL[p_T(_T)||q_ϕ(_T,T)] is 

    ^* ≈√(α_T)_d 
    _p^*(i,j) ≈1/|_d|_d(i,j)
    ^*(i,j) ≈α_T_d+(1-α_T)^2_p^* = (α_T^2+(1-α_T)^2/|_d|)_d(i,j)




Thus, if we are able to estimate the mean and variance of the ground-truth data distribution, we can then analytically determine the optimal prior distribution 𝒩(^*,^*) and the optimal perturbation kernel variance ^*_p at any noise level T ∈ [0,∞). It leads to a crucial advantage against prior efforts on prior learning <cit.>, where a target noise level has to be determined at training time in order to train an additional neural network to represent the prior distribution at the pre-selected noise level. In contrast, our method enables flexible tuning of the number of diffusion steps at inference time, without additional training costs. Since the learnable q_ϕ(T) and perturbation kernel are both Gaussian, we refer to our proposed diffusion framework as Optimal Gaussian Diffusion (OGD).


For joint trajectory prediction and generation, we need to estimate the mean and variance of the joint trajectory distribution p_0(_0). It is not straightforward as only a limited number of trajectory samples exist in the dataset under the same context. Considering there exists mature and accurate marginal trajectory prediction models <cit.>, we can conveniently extract statistics of the marginal trajectory distributions, i.e., p_0(_0, i), i∈{1,2,...,n} from a pre-trained marginal trajectory predictor. Since <ref> provides element-wise optimal value, we can easily get:

 
 
Denote _d(_0,i) and _d(_0,i) as the mean and variance of marginal distribution for vehicle i and set both  and _p are block-diagonal matrices where each block represents the marginal characteristics of one single vehicle. The optimal solution to minKL[p_T(_T)||q_ϕ(_T,T)] is


    ^*=[√(α_T)_d(_0,1),...,√(α_T)_d(_0,n)]^T 
    _p^* =1/∑_i=1^n|_d(_0,i)|diag[_d(_0,1),...,_d(_0,n)]
    ^*=α_Tdiag[_d(_0,1),...,_d(_0,n)] + (1-α_T)^2 _p^*



<ref> implies that, if we further confine  and _p to be block-diagonal without covariance between the states of different vehicles, then we can determine their optimal values purely from the estimated marginal statistics _d(_0,i) and _d(_0,i), i∈{1,2,...,n}, which enables a practical implementation of the proposed OGD model for joint trajectory prediction and generation tasks. Specifically, for vehicle i, we leverage a pre-trained marginal trajectory predictor <cit.>, predict diverse marginal trajectory sample set ℛ_i={_i^l}_l=1^L and corresponding likelihood set {p(_i^l)}_l=1^L, and estimate _d(_0,i) and _d(_0,i). For example, _d(_0,i) can be estimated as 1/L∑_l=Lp(_i^l)_i^l. 



 §.§ Estimated Clean Manifold Guidance with Reference

As discussed in <ref>, the intensive computation required for guided sampling comes from the calculation of ∇__tf_θ(_t). Previous guided sampling approaches bias the score function defined on the intermediate noisy data _t. In this section, we aim to investigate whether we can inject the gradient directly into _0 to avoid the gradient propagation process. 
We first reformulate the controllable generation as a multi-objective optimization problem directly over _0 and propose an iterative algorithm to solve the formulated problem. In addition, we use reference trajectory points to create the region of interest, which helps solve the local optimal problem caused by multi-modal joint trajectory distribution and accelerate the guided sampling process.



  §.§.§ Estimated Clean Manifold Guidance.


The objective of controllable generation of sample _0 includes two different objectives. The most important is the negative likelihood, ensuring the sample lies in the clean manifold. The second important is the guidance cost representing the user preference on the generated sample _0. This multi-objective optimization problem can be represented as

    min__0   [-log q_θ(_0),(_0)]^T


Inspired by lexicographic optimization <cit.>, we solve this multi-objective optimization problem hierarchically. The main idea is to optimize each objective in the order of importance regardless of the degradation of the other less significant objectives. We first optimize the most important objective, -log q_θ(_0), to generate realistic and high-likelihood samples. The diffusion model achieves this goal effectively by reversing the diffusion process from noisy samples at specific noise level. However, exact noise level of current sample _0 is unknown. To address this, we inject noise at level t into _0, and then denoise it from t with learned diffusion model. This approach, similar to noise injection and denoising <cit.>, improves the desired sample quality.


We iteratively repeat this process K times, injecting guidance at each iteration to strengthen user preference. 

Specifically, denote _0(k) as the sample at iteration k. We first regenerate high-likelihood sample _0(k) by diffusion model:

    _0(k) ←𝔼[q_θ(_0(k)|_t_k(k))]=1/√(α_t_k)(_t_k-√(1-α_t_k)_θ(_t_k(k),t_k)),

where _t_k(k) = √(α_t_k)_0(k)+√(1-α_t_k), ∼𝒩(0,_p). t_k ∈ [0,T] is a tunable parameter. Then we minimize the guidance cost function with a small degradation of the most important objective -log q_θ(_0), 

    _0(k-1) ←_0(k)-ζ∇__0(k)(_0(k))

where ζ is the step size. Small degradation of -log q_θ(_0) is realized by one-step gradient update and proper step size ζ. See the derivation of the optimization process and the parameter tuning in <ref>.

During the iterations, _0(k), ∀ k ∈{0,1,...,K-1} are not exactly on the clean manifold. We minimize its negative log-likelihood through diffusion model, resulting in it lying on an estimated clean manifold. Thus, we call our method Estimated Clean Manifold (ECM) Guidance.



  §.§.§ Reference Joint Trajectories.
 

To generate trajectories with low guidance cost, we are essentially searching low-cost trajectories within the high-likelihood region. At the same time, joint trajectory distribution is a multi-modal distribution resulting from road topologies and different decision variables, meaning the likelihood has multiple peaks. This leads to the optimal solution of Problem <ref> having multiple local optimals, and each optimal is far away from the other. Guided sampling methods <cit.>, including our method ECM, suffer from two challenges (See <ref>): 1) it can be trapped at the local optimal around the initial position; 2) it takes massive efforts to drag the sample from one peak to another and transferring from one modal to another will need to pass through the region of low-likelihood (off clean-manifold), leading to numerical instability. 

[14]r0.6






To overcome this, we generate high-likelihood reference joint trajectories, choose the best one as the initialization. Note that combinations of samples with high marginal likelihood tend to exhibit high joint likelihood. Therefore, we can utilize the marginal sample set ℛ={ℛ_i}_i=1^n obtained from pre-trained marginal models to generate the references. Specifically, for iteration k, we construct candidate joint trajectory set ℛ⊗_0(k) = {[_1,_2,...,_n]|_i ∈ℛ_i ∪{_0,i},i=1,2,...,n}. We calculate the guidance cost of all possible combinations, (w), w ∈ℛ⊗_0(k), and choose the minimal-cost one as the reference. 
The guided sampling algorithm, ECM with reference joint trajectories (ECMR), is introduced in <ref>. 





§ EXPERIMENTS





 §.§ Experimental setup

Dataset. We use Argoverse 2 <cit.>, a widely used and large-scale trajectory prediction dataset, to test the effectiveness of our approaches for joint trajectory prediction and controllable generation. It has a large observation window of 5s and a long prediction horizon of 6s.

Implementation Details.

We use the fixed scene context encoder of pre-trained QCNet <cit.> to extract compact and representative context features from context information . Then, we utilize a cross-attention layer to update the intermediate noisy data _t with multiple contexts, including the history encodings of the target agent, the map encodings, the neighboring agents’ encodings. Inspired by <cit.>, we also add a cross-attention layer to update _t with the diverse marginal trajectory samples _i^l and its corresponding likelihood. In addition, we use self-attention to allow the interaction between _t,i and _t,j. Then the model predicts the noise _θ(_t,t). According to <cit.>, compact trajectory representation helps the diffusion model to generate high-quality trajectories efficiently. Inspired by this, we also learn a linear mapping between the 10-dimensional latent and 120-dimensional trajectories. Similar to <cit.>, we design a rapid sample clustering algorithm so that we can generate a representative joint trajectory set. To increase the efficiency of sampling, we use DDIM <cit.> to accelerate the inference, and the DDIM step stride is 10. See <ref> for details and analysis.



 §.§ Joint trajectory prediction



We now evaluate OGD for joint trajectory prediction. Given K joint trajectories, the evaluation metrics are 1) avgMinFDE_K/avgMinADE_K: the average of lowest final/average displacement error (FDE/ADE) of joint trajectory samples; 2) actorMR_K: the rate of trajectory predictions that are considered to be “missed” (>2m FDE) in the lowest minFDE joint trajectory samples; 3) actorCR_K: the rate of collisions across “best” (lowest avgMinFDE) joint trajectory samples; 4) avgBrierMinFDE_K: calculated similarly to avgMinFDE_K but scaled by the probability score of joint trajectory samples. We denote metrics with superscript “*” as those after sample clustering (see <ref>). 

As a baseline, we train a vanilla diffusion (VD) baseline that shares the same neural network architecture as OGD. Specifically, we train an Optimal Gaussian Diffusion model with diffusion time T_train=100 (OGD), vanilla diffusion with two different diffusion times T_train=100,500 (VD_100, VD_500). Note that T is denoted as the diffusion time from which the reverse diffusion process starts. First, during the inference, we change T and evaluate OGD and VD_100 who have the same T_train=100. <Ref> shows that, with the decrease of the reverse steps, avgMinFDE_128 of OGD keeps lower than VD_100 and it is more stable with the change of T. It is also interesting to note that the performance of OGD even becomes better in the early stage. We hypothesize that it is because that KL[p_T(_T)||p_prior] of OGD is small and the accumulated error of 𝒢(_θ, T) is sufficiently reduced by a small T. It also shows an advantage of OGD, which is that it is easy for OGD to tune a suitable T for reverse diffusion without training an additional model for every T <cit.>. <Ref> shows that OGD outperforms VD_500 with only 
 40 diffusion steps. 

We also compare our OGD with the other state-of-the-art methods on the Argoverse multi-world leaderboard; our approach OGD ranks 4^th on the leaderboard ranked by avgBrierMinFDE_K, which demonstrates the effectiveness of our OGD framework. Note that we only list the entries with publications or technical reports in <ref>. Please refer to the official website for the full leaderboard. 








 §.§ Controllable Generation


Tasks. Future behaviors of vehicles can be effectively can be effectively represented by a set of goal points <cit.> such as acceleration, braking, and right or left turn. Generating such diverse modes of joint trajectories is a typical motivation for using controllable generation. Thus, we study the controllable generation task where the goal is to reach diverse goal points at a specific time to fully test our guided sampling method. Considering that goal points should lie in realistic routes, and the trajectories to reach such goal points should also lie in such routes, we generate such target goal points in some realistic routes Route(τ_g). The guidance cost function can be expressed as

    (_0)=1/n||Position(_0,τ_d)-Route(τ_g)||_2^2,

where Position(_0,τ_d) is the positions of _0 at time τ_d. We choose ground-truth trajectories and a random combination set of diverse marginal samples, i.e., 𝒰 = {[_1,_2,...,_n]|_i ∈ℛ_i,i=1,2,...,n} as the realistic routes. We denote the former as GT set and the latter as U set. The underlying assumption is that diverse samples from a good marginal trajectory predictor are realistic. We design different velocity settings to cover the diverse controllable generation tasks in autonomous driving: first is Normal Speed (N), τ_d=τ_g=6s; second is Acceleration (A), τ_d=5s<τ_g=6s; third is Deceleration (D), τ_d=6s<τ_g=5s.


Metrics. We use the following metrics to evaluate controllable generation performance: Joint Route Deviation Error (JRDE), which measures the displacements to the routes to evaluate the realism, and Joint Final Displacement Error (JFDE), which evaluates the guidance effectiveness. We also evaluate from the “min” and “mean” perspectives: The “min” metric considers the best sample's performance, while the “mean” metric assesses the ratio of valid samples.

Baselines. 
Guided sampling in controllable generation is mainly divided into two approaches: the first is to directly calculate ∇__t(_t) <cit.>; the second is to calculate ∇__t(_0) <cit.>. We denote the former as Next Noisy Mean Guidance (NNM) and the latter as Score Function Guidance (SF). For a fair comparison, we use one guidance step followed by one DDIM step. We also tune the gradient step size for different guided sampling with Optimal Gaussian Diffusion and vanilla diffusion and report the results with the optimal step size. See <ref> for the details of baseline derivation and step size tuning. We evaluate the following experiments with 128 joint trajectory samples.






Evaluation. First, we evaluate the performance and efficiency with our diffusion model (OGD) and guided sampling method (ECM and ECMR) in <ref>, which demonstrates our methods can generate more realistic and effective samples with 5 times less DDIM steps. In addition, with reference joint trajectories, ECM significantly improves 'mean' metrics, indicating it addresses the issues discussed in <ref> to a certain extent. Second, we compare solely on different guided sampling methods using the same OGD model shown in <ref>. Our ECM achieves better performance both in guidance effectiveness and realism. In <ref>, our ECM can generate trajectories that reach goal points more closely than NNM and SF. And with reference joint trajectory, ECMR can easily move samples from one modal to another . Third, we evaluate the average inference time and average GPU memory usage in <ref>. Our methods can generate realistic trajectories satisfying guidance quite well with low inference time and GPU memory usage. More results on controllable generation can be found in <ref>.





§ CONCLUSION

In this work, we introduce Optimal Gaussian Diffusion (OGD) and Estimated Clean Manifold (ECM) Guidance to significantly improve the computational efficiency and performance of diffusion models in autonomous driving. These methodologies enable a substantial reduction in inference steps and computational demands while ensuring enhanced joint trajectory prediction and controllable generation capabilities. Our approaches and experimental results underscore the potential of diffusion models for real-time applications in dynamic environments, marking a pivotal advancement in the deployment of diffusion models for autonomous driving. One limitation of the current implementation is that the performance is affected by the accuracy of marginal trajectory predictors. Enhancements could be achieved with superior marginal models or by directly learning joint predictions' mean and variance, areas for future work.









§ ACKNOWLEDGEMENT

This work was supported by Berkeley DeepDrive.
[https://deepdrive.berkeley.edu]


splncs04




§ PROOF OF <REF>


In this section, we prove the <ref>.



We would like to minimize KL[p_T(_T)||q_ϕ(T)] in the VP-SDE  <cit.> setting. We expand VP-SDE setting into the setting when the perturbation kernel is in the format of p_t(_t|_0) = 𝒩(_t;a_t _0+b_t, c_t^2_p). For VP-SDE, a_t=√(α_t), b_t=0, c_t=1-α_t. For Variance Exploding (VE) SDE <cit.>, a_t=1, b_t=0, c_t=σ_t^2. In the following content, we are going to minimize KL[p_T(_T)||q_ϕ(T)] in a more general form when _T = a_T_0+b_T+_T and Var[_T]=c_T^2_p. Then, we use the results on VP-SDE.


Minimizing KL[p_T(_T)||q_ϕ(T)] is equivalent to maximizing the log likelihood max_, , _p𝔼 [log q_ϕ(_T,T)], where q_ϕ(_T,T) is denoted as the probability density of q_ϕ(T) = 𝒩(,) at _T. Assume the dataset contains N data points, which are _0^i, i=1,2,...,N. And _T^i=a_T_0^i+b_T+_T^i, i=1,2,...,N. Thus, 𝔼 [log q_ϕ(_T,T)] can be approximated as 1/N∑_i=1^N log q_ϕ(_T^i,T), denoted as l(, , _p). Our optimization problem is formulated as 


    max_, , _p      l(, , _p)
    
            s.t.      |_p|=1


We formulate the Lagrange function 

    h(, , _p,λ)    = l(, , _p) + λ (log |_p| - log 1)
       =l(, , _p) + λlog |_p|


First, we express explicitly l(, , _p) as 

    l(, , _p)=   1/N∑_i=1^N log q_ϕ(_T^i,T)
    
            =   -1/2log ||-1/2N∑_i=1^N(_T^i-)^T^-1(_T^i-)
    
            =    -1/2log || - 1/2N∑_i=1^N(a_T_0^i+b_T-)^T^-1(a_T_0^i+b_T-)
       -1/N∑_i=1^N(a_T_0^i+b_T-)^T^-1_T^i-1/2N∑_i=1^N_T^i^T^-1_T^i



Take derivative with respect to , we get

    ∂ h(, , _p,λ)/∂=   1/N∑_i=1^N(a_T_0^i+b_T-)^T^-1+1/N∑_i=1^N_T^i^T^-1
    
            =   1/N∑_i=1^N(a_T_0^i+b_T-)^T^-1 = 0


In order to take derivative with respect to , we reformulate l(, , _p) as

    l(, , _p)=   1/2log |^-1| - 1/2N∑_i=1^Ntr[(a_T_0^i+b_T-)(a_T_0^i+b_T-)^T^-1]
       -1/N∑_i=1^Ntr[_T^i(a_T_0^i+b_T-)^T^-1]-1/2N∑_i=1^Ntr[_T^i_T^i^T^-1 ]
    
            =   1/2log |^-1| - 1/2Ntr[(∑_i=1^N(a_T_0^i+b_T-)(a_T_0^i+b_T-)^T)^-1]
       -1/Ntr[(∑_i=1^N_T^i(a_T_0^i+b_T-)^T)^-1]-1/2Ntr[∑_i=1^N(_T^i_T^i^T)^-1 ]
    ≈   1/2log |^-1| - 1/2Ntr[(∑_i=1^N(a_T_0^i+b_T-)(a_T_0^i+b_T-)^T)^-1]
       -tr[Cov[, _T-]^-1]-1/2tr[c_T^2_p^-1 ]
    ≈   1/2log |^-1| - 1/2Ntr[(∑_i=1^N(a_T_0^i+b_T-)(a_T_0^i+b_T-)^T)^-1]
       -c_T^2/2tr[_p^-1 ]

where Cov[·, ·] is the covariance matrix of two random variables and Cov[_T-, _T] ≈ 0 because _T is a independent random variable.

Then take derivative of ^-1, we get:

    ∂ h(, , _p,λ)/∂^-1≈   1/2-1/2N∑_i=1^N(a_T_0^i+b_T-)(a_T_0^i+b_T-)^T-c_T^2/2_p = 0



Take derivative of _p

    ∂ h(, , _p,λ)/∂_p≈   ∂[-c_T^2/2tr[_p^-1]+λlog |_p|]/∂_p
    
            =   ∂[-c_T^2/2tr[^-1_p]+λlog |_p|]/∂_p
    
            =   -c_T^2/2^-1+λ_p^-1=0


Take derivative of λ

    ∂ h(, , _p,λ)/∂λ =    log |_p|=0


In this way, we get 

    ^* = b_T + a_T1/N∑_i=1^N_0^i 
    ^* ≈1/N∑_i=1^N(a_T_0^i+b_T-)(a_T_0^i+b_T-)^T+c_T^2_p^*
    _p^* ≈2λ/c_T^2^*
    
            |_p^*|=1


Given that _d ≈1/N∑_i=1^N_0^i and _d ≈1/N∑_i=1^N(_0^i-_d)(_0^i-_d)^T, we have ^* ≈ a_T^2_d+c_T^2_p^* and 


    ^* ≈ b_T + a_T _d 
    _p^* ≈1/|_d|_d
    ^* ≈ a_T^2_d+c_T^2_p^* = (a_T^2+c_T^2/|_d|)_d

where the approximation becomes increasingly close to an equality when N is very large.


Denote ^*(i,j) and _p^*(i,j) as the element at ith row and jth column of matrix ^* and _p^*, and we can have element-wise optimal values for  and _p,


    ^* ≈ b_T + a_T _d 
    _p^*(i,j) ≈1/|_d|_d(i,j)
    ^*(i,j) ≈ a_T^2_d+c_T^2_p^* = (a_T^2+c_T^2/|_d|)_d(i,j)


The element-wise version is important because perturbation models typically use a diagonal matrix as the diffusion kernel, implying that _p is a diagonal matrix. The element-wise version can handle not only diagonal matrix diffusion kernels but also any other variance matrix diffusion kernels. In a multi-agent setting, if we disentangle the relationships between different agents in the latent space and allow the model to learn these relationships, the variance matrix can be a block diagonal matrix. This means that there are correlations within each agent, but no correlation exists between agents. 

Apply this to VP-SDE and we get <ref> and <ref>.




§ DERIVATION OF ECM AND PARAMETER TUNING




 §.§ Derivation of ECM

As discussed in <ref>, we solve the multi-objective problem <ref> by K iterations. In this section, we derive the optimization process for each iteration.


At iteration k ∈{0,1,...,K-1}, we first minimize the most important objective -log q_θ(_0(k)) where _0(k) is the intermediate sample at iteration k. Since our ultimate goal in optimizing -log q_θ(_0(k)) is to ensure that the sample lies on the clean manifold, we can employ a diffusion model to regenerate the high-likelihood sample based on the current sample _0(k). We add the noise ∼𝒩(0,_p) to the current sample _0(k) and get _t_k(k) = √(α_t_k)_0(k)+√(1-α_t_k) where t_k ∈ [0,T] is a tunable parameter. In order to obtain q_θ(_0(k)|_t_k(k)), we can we can iteratively run the reverse diffusion process until t=0. However, this approach is time-consuming. Therefore, we use Tweedie's formula to perform a one-step estimation. Specifically, q_θ(_0(k)|_t_k(k)) can be approximated as a Gaussian distribution and its mean has the highest likelihood. Thus, given _t_k(k), we can update _0(k) to the highest likelihood position, i.e., 𝔼[q_θ(_0(k)|_t_k(k))], 


    _0(k) ←𝔼[q_θ(_0(k)|_t_k(k))]=1/√(α_t_k)(_t_k-√(1-α_t_k)_θ(_t_k(k),t_k))


Then, inspired by ϵ-Constraint method <cit.>, we minimize the guidance cost under the constraint of small degradation of the primary objective value,

    min__0(k-1)     (_0(k-1))
    
            s.t.      -log q_θ(_0(k-1)) < -log q_θ(_0(k))+γ

where γ is a small positive scalar. We assume that if the deviation of _0(k-1) from _0(k) is small enough, the constrain in Problem <ref> can be satisfied. Thus, we propose a simple approach to solve this problem by one gradient update 

    _0(k-1) ←_0(k)-ζ∇__0(k)(_0(k))

where ζ is the step size. 





 §.§ Parameter tuning and analysis

In this section, we analyze the impact of t_k, where t_k ∈ [0,T], and injected noise ∼𝒩(0,_p) on primary objective: the negative log-likelihood.

First, we examine the impact of t_k where t_k ∈ [0,T]. We first inject noise at level t_k and denoise it from t_k using learned diffusion model. Generally, a larger t_k but highly distinguished samples will negate the guidance efforts made in previous steps. Moreover, a larger t_k reduces sample quality because we use a one-step estimation, and the estimation error increases with a larger t_k. Conversely, a smaller t_k introduces minimal noise, facilitating denoising to a high-likelihood sample with only a few denoising steps. Thus, updating with mean of Tweedie’s formula is more accurate. Therefore, we design t_k to decrease as k decrease, enhancing diversity in the initial stage and refining the sample in the final stage.

Second, we analyze the impact of injected noise ∼𝒩(0,_p) in _t_k(k) = √(α_t_k)_0(k)+√(1-α_t_k). Note that  at different iterations are independent with each other, which injects excessive stochasticity. More importantly, directly injecting stochastic noise at each iteration also diffuse the guidance from previous iterations, resulting in lower guidance effectiveness. Inspired by DDIM <cit.>, we can choose to use _θ(_t_k+1(k+1), t_k+1) as the deterministic noise  at iteration k. We use Estimated Clean Manifold (ECM) Guidance with the optimal step size tuned in <ref>, and compare the performance differences when injecting stochastic versus deterministic noise. The results are shown in <ref>. We find that ECM with stochastic noise can generate samples as realistic as that with deterministic noise, but the guidance is excessively diffused. We use the deterministic injected noise in this paper.






§ ADDITIONAL IMPLEMENTATION DETAILS AND ANALYSIS


In this section, we begin by detailing the network structure and parameters such as the diffusion schedule parameters. Subsequently, we describe the process of learning compact trajectory representations through Linear Mapping (LM). Next, we introduce the sample clustering algorithm utilized in this study. Finally, we compare the influence of different pre-trained marginal trajectory predictors.



 §.§ Network Structure and parameters


We use the fixed scene context encoder of pre-trained QCNet <cit.> to extract compact and representative context features from context information . Then, we utilize a cross-attention layer to update the intermediate noisy data _t with multiple contexts, including the history encodings of the target agent, the map encodings, the neighboring agents’ encodings. Inspired by <cit.>, we also add a cross-attention layer to update _t with the diverse marginal trajectory samples _i^l and its corresponding likelihood (generated by pre-trained QCNet <cit.>). In addition, we use self-attention to allow the interaction between _t,i and _t,j. Then the model predicts the noise _θ(_t,t).

For the diffusion parameters, we set β_0=0.0001 and β_T=0.05 for all the models we trained. And then we calculate α_t=1-β_t, α_t=∏ _t=0^Tα_t. We train the models for 64 epochs.

For all experiments, we generate 128 samples by default. To facilitate comparison with other state-of-the-art methods on the Argoverse 2 multi-world leaderboard, we generate 2048 joint trajectory samples.



 §.§ Compact Latent Representation

According to <cit.>, compact and expressive latent representation helps the diffusion model to generate high-quality trajectories efficiently. In this paper, we learn a low-dimensional latent of the trajectory then apply diffusion model in the latent space. For clearer presentation, we use the notation  in this section to represent the trajectory. Denote the latent as ∈ℝ^Z × 1 and the mapping functions between  and  are = ℱ() and =𝒢(). 

First, we minimize the reconstruction loss, ℒ_rec:

    ℒ_rec=||ℱ(𝒢())-  ||_2^2


Second, we constrain mapping between between  and  should be distance-preserving transformations. The intuition behind is that the latent diffusion model aims to minimize expected error in latent space, though the ultimate objective is reducing error in trajectory space. Consequently, it disproportionately prioritizes samples with significant latent space errors, which may not correspond to those with substantial errors in trajectory space. Thus, we minimize this regularization term ℒ_reg:

    ℒ_reg=(^T - 𝒢()^T𝒢())^2


In addition, extreme variance in different dimension of the latent will cause instability of the training. Thus, we minimize the variance term ℒ_var

    ℒ_var=||std(𝒢()) - 𝐯||_2^2

where 𝐯∈ℝ^Z × 1 whose elements are all η, a learnable variable.

To further improve the efficiency during the inference, we need to find a mapping 𝒢(·) as simple as possible. <cit.> shows that PCA can have relatively low reconstruction error, and it is only a linear mapping which is computational efficient. Inspired by this, we use two linear mapping U ∈ℝ^X × Z and V ∈ℝ^Z × X to formulate the mapping function, 𝒢()= U and ℱ()= V. Since it is linear mapping, we call this method as Linear Mapping (LM). We set Z=10 and it works well both in reconstruction and performance of diffusion model. To satisfy distance-preserving requirements, columns in V should be orthogonal to each other so each dimension in  is independent with each other. Thus, if this mapping is well-trained, variance of  is diagonal matrix. In our implementation, we assume we can learn such a good linear mapping and marginal variances in <ref> are all diagonal matrices. We visualize the Top-10 components of PCA <cit.> and 10 components in V in <ref>. The modes learned from the LM exhibit greater diversity and expressiveness. In addition, latent variances derived from the LM across different dimensions are consistent, whereas PCA-derived latent variances show extreme variability, with a large variance in the main component and significantly lower variance in the remaining components. We also compare the performance of latent diffusion model with PCA and LM in <ref>, using Optimal Gaussian Diffusion. Our linear mapping approach can improve the performance.









 §.§ Sample Clustering Algorithm
 
We design a sample clustering algorithm to to generate a representative set from a batch of joint trajectory samples. Specifically, we first designate 𝒰 as the central point for the groups. Then for each vehicle i, we map the trajectory samples _0,i, i=1,2,...,N into closest _i^l, l=1,2,...,L. Thus, we assign each joint trajectory into 𝒰. Subsequently, we sort the 𝒰 by the size of its group memberships. Third, we prune the group based on the center of the groups represented by the combination of reference trajectories _i^l. Specifically, if the centers of two groups are close to each other, the group with the larger number of members will absorb the other. We define 'close enough' as a scenario where the maximum endpoint deviation between these two joint trajectories is less than 2.5 meters, aligning with the threshold used in NMS <cit.>. After pruning, we calculate the probability of each group by its size. Since our clustering algorithm does not need to calculate the exact log probability <cit.>, it is less computationally demanding compared to more intensive techniques such as Non-Maximum Suppression (NMS) <cit.> or Expectation-Maximization (EM) <cit.>. However, since we do not explicitly to calculate the likelihood of a sample, the likelihood estimation is not very accurate. This has resulted in our outperforming the rank-3rd benchmark on the Argoverse 2 Multi-world Forecasting leaderboard across all metrics, with the exception of avgBrierMinFDE_K, which is calculated based on the likelihood of the samples. Since we are focusing on optimizing diffusion model, we leave fast and accurate likelihood estimation for the future research. <Ref> demonstrates the influence of the number of samples. With more samples, all metrics improve. The performance saturates when the number of samples exceeds 1024.






 §.§ Ablation Study on Marginal Predictors

We froze the trained Optimal Gaussian Diffusion (OGD) model used in <ref> and replaced QCNet<cit.> with Forecast-MAE <cit.> as the marginal predictor. We downloaded their pre-trained weights from their official website [https://github.com/jchengai/forecast-mae] and fine-tuned them for multiple epochs. Fine-tuning is necessary because the pre-trained weights were trained on a subset of target agents for the joint prediction task <cit.>. We compare the marginal and joint metrics in <ref>. The Forecast-MAE version of our model also yielded good performance, showing the flexibility of our approach. Note that the joint metrics of OGD are correlated with the marginal metrics. It indicates that we could improve OGD's performance without re-training if a better marginal predictor is available.







§ GUIDED SAMPLING BASELINE AND STEP SIZE TUNING


In this section, we first derive two guided sampling baselines: Next Noisy Mean (NNM) Guidance and Score Function (SF) Guidance. We also introduce the comparison settings for controllable generation tasks. Secondly, we present the step size tuning experiments.



 §.§ Guided Sampling Baseline

Previous guided sampling approaches for controllable generation in autonomous driving are mainly focusing on guide the sample generation with an analytical guidance cost function (·) defined on clean data _0, such as goal point guidance and target speed guidance <cit.>. In this paper, we discuss guided sampling approach under the human-defined guidance cost function (·) with no need to train additional guidance function defined on noisy data _t, t>0 <cit.>. 

The first baseline is to directly calculate ∇__t(_t). Specifically, it first calculate the mean of _t-1 conditioned on _t, 

    _t-1=1/√(α_t)(_t-β_t/√(1-α_t)_θ(_t,t))

Then, add the guidance into _t-1

    _t-1=_t-1-clip(ζ∇__t-1(_t-1),±β_t σ_p)

where ζ is the step size, clip is elementwise clipping function, σ_p is the positive squared root of diag[_p]. Note that _t-1 is also noisy so (_t-1) suffers from numerical instability. Since this method directly inject the guidance into the mean of next noisy data distribution _t-1, we call it as Next Noisy Mean (NNM) Guidance.

The second baseline is to first project _t into _0 on the clean manifold and then calculate the guidance ∇__t(_0) <cit.>. Note that <cit.> directly train a denoiser to predict _0. In our DDPM <cit.> formulation, _0 is estimated through Tweedie’s formula,

    _0 = 1/√(α_t)(_t-√(1-α_t)_θ(_t,t))

Then, add the guidance ∇__t(_0) to the score function,

    _θ(_t,t) = _θ(_t,t) + clip(ζ√(1-α_t)∇__t(_0),±σ_p)/√(1-α_t)

where ζ is the step size. Since _θ(_t,t) =-√(1-α_t)_θ(_t,t), we have

    _θ(_t,t) = _θ(_t,t) - clip(ζ√(1-α_t)∇__t(_0),±σ_p)


We use DDIM <cit.> to accelerate the inference for controllable generation. Note that we replace β_t in <ref> with √(1-α_t/α_t') for time index t', t' < t.

To fairly compare the performance and efficiency of different guided sampling approaches, we set one network inference step followed by one gradient guidance step. For our ECM and ECMR, we set t_k to be the same as the DDIM time step t. Specifically, we set DDIM step stride to 10 and T to 100. Then t_k=10(k+1), where k=0,1,...,9.








 §.§ Step Size Tuning



We conduct a grid search to identify the optimal gradient step size for NNM, SF, ECM, and ECMR. For NNM, ECM and ECMR, we use the settings ζ∈{1,5,7,10,15,30,60,100}. For SF, ζ∈{10,500,1000,2000,3000,5000}. We find that both excessively small and large gradient step sizes result in low guidance effectiveness, similar to the inefficiency or instability seen in gradient-based optimization algorithms with overly small or large step sizes. The optimal step size lies in between. Based on <ref>, we select the optimal step size that yields the lowest minJFDE. We apply these optimal step sizes across all guided sampling methods and controllable generation tasks, and report the quantitative results using these sizes throughout the remainder of the paper.







§ EVALUATION ON CONTROLLABLE GENERATION TASKS


<Ref> showcases the performance of various guided sampling methods across different tasks. ECM and ECMR consistently outperform others in minJFDE across all tasks and in minJRDE in nearly all tasks, indicating their superior guidance effectiveness and realism. Notably, ECMR, in particular, excels in meanJFDE/meanJRDE, suggesting the generated joint trajectories are of higher average quality. This implies our methods' ability to produce valid and high-quality joint trajectories with fewer samples. The experiment results are shown in <ref>. Our ECM and ECMR approaches can generate a better joint trajectory with a small number of samples.

<Ref> and <ref> demonstrate examples in different controllable generation tasks. Our guided sampling approach, ECM, can better satisfy the guidance while simultaneously maintaining realism. With reference joint trajectory, ECM can further enhance its performance.












 Paper Title: Automated Sperm Morphology Analysis Based on Instance-Aware Part Segmentation.


	
	
	
	Distributionally Robust Optimization as a Scalable 

           Framework to Characterize Extreme Value Distributions 
 (Supplementary Material)
    Alex Zunger
    August 5, 2024
================================================================================================================================================

empty
	empty
	
	
	
	§ ABSTRACT 
		
		Traditional sperm morphology analysis is based on tedious manual annotation. Automated morphology analysis of a high number of sperm requires accurate segmentation of each sperm part and quantitative morphology evaluation. State-of-the-art instance-aware part segmentation networks follow a “detect-then-segment" paradigm. However, due to sperm's slim shape, their segmentation suffers from large context loss and feature distortion due to  bounding box cropping and resizing during ROI Align. Moreover, morphology measurement of sperm tail is demanding because of the long and curved shape and its uneven width. This paper presents automated techniques to measure sperm morphology parameters automatically and quantitatively. A novel attention-based instance-aware part segmentation network is designed to reconstruct lost contexts outside bounding boxes and to fix distorted features, by refining preliminary segmented masks through merging features extracted by feature pyramid network. An automated centerline-based tail morphology measurement method is also proposed, in which an outlier filtering method and endpoint detection algorithm are designed to accurately reconstruct tail endpoints. Experimental results demonstrate that the proposed network outperformed the state-of-the-art top-down RP-R-CNN by 9.2% AP^p_vol, and the proposed automated tail morphology measurement method achieved high measurement accuracies of 95.34%, 96.39%, 91.20% for length, width and curvature, respectively.
		
	
	
	
	
	

§ INTRODUCTION

	
	Quantitative analysis of sperm morphology is vital for diagnosing male infertility <cit.>. Traditionally, sperm morphology analysis is performed manually by clinical staff who empirically follow the World Health Organization (WHO) guidelines <cit.> to analyze each sperm. Specifically, an experienced operator carefully observes zoomed-in sperm images to performs pixel-by-pixel annotation to label 5 contours for each part of the sperm (i.e., acrosome, vacuole, nucleus, midpiece and tail). Based on the annotated parts/contours, sperm morphology parameters are then manually calculated. Manual analysis of sperm morphology is a highly tedious process. As required by WHO, the morphology of at least 200 sperm should be analyzed within each semen sample, leading to manual annotation of >1,000 contours per patient sample, which calls for techniques to automate the analysis task.
	
	To achieve automated sperm morphology measurement, firstly, an instance-level sperm parsing is needed that not only correctly detects sperm but also accurately segments the parts for each sperm (see Fig. <ref>(a)&(b)). Secondly, an automated morphology measurement method is needed for quantitatively calculating morphology parameters for each sperm part based on the segmented mask.
	
	
	
	For the first step of instance-aware part segmentation (see Fig. <ref>(b)), state-of-the-art instance-aware part segmentation networks from the human parsing field employs a “detect-then-segment" paradigm <cit.>. Firstly, various sperm are detected by bounding boxes. Then each sperm within a bounding box is cropped and resized to have a uniform shape through ROI Align <cit.>. Finally, the sperm is segmented into individual parts. This paradigm is a top-down method <cit.>. However, the top-down method is sensitive to detection errors since contexts outside a bounding box are cropped out (see Fig. <ref>(a)). Also, the resizing step in ROI Align leads to feature distortion and inaccurate contour segmentation (see Fig. <ref>(b)). Moreover, unlike traditional segmentation targets such as human, sperm is long and slim (high length-to-width ratio); thus, its segmentation suffers from higher context loss resulting from more detection errors and larger feature distortion because of greater aspect ratio resizing. Therefore, the top-down method produces larger segmentation errors for sperm analysis.		
	
	After segmentation, an automated sperm tail morphology measurement method is required. Sperm head parameters (e.g., length, width, and ellipticity) and midpiece parameters (e.g., length, width, and rotated angle) can be calculated through ellipse <cit.> and rectangle <cit.> fitting (see Fig. <ref>(c)). In comparison, morphology measurement of sperm tail is more challenging because the tail is long, curved, with uneven in width and cannot be easily fitted. Methods have been proposed for measuring curvilinear structures <cit.>, among which Steger-based methods <cit.> are advantageous due to their sub-pixel accuracy. In Steger-based methods, points whose first derivative reaches zero along the normal are detected as centerline points. Then, width can be calculated according to distances between centerline points and edge points along the normal, and curvature can be calculated as the change of the normal of the centerline points (see Fig. <ref>(d)). However, endpoints detected by Steger-based methods tend to mislocate (see Fig. <ref>(c)) because endpoints' normal is affected by gradient from the intersecting edge and not perpendicular to the line direction (see Fig. <ref>(d)).
	
	
	
	The problems tackled in this work include: 1) how to avoid context loss and feature distortion during the instance-aware part segmentation for sperm; 2) how to accurately reconstruct endpoints for measuring sperm tail morphology parameters. To address the first challenge, a novel attention-based instance-aware part segmentation network is proposed to reconstruct lost contexts outside a bounding box and fix distorted features by refining masks generated following the “detect-then-segment" paradigm. Specifically, the preliminary segmented masks are used to provide spatial cues for each located sperm, then merged with features extracted by a feature pyramid network (FPN) through the attention mechanism, finally refined by CNN to produce improved results. For the second challenge, an automated centerline-based tail morphology measurement method is proposed, in which an outlier filtering method and endpoint detection algorithm are designed to accurately reconstruct endpoints. 
	
	Experimental results demonstrate that the proposed network achieved 57.2% AP^p_vol (Average Precision based on part) on the collected dataset, outperforming the state-of-the-art top-down RP-R-CNN by 9.20%; and the proposed tail morphology measurement method substantially decreased measurement errors by 4.21%, 3.02%, 13.10% for length, width and curvature, respectively compared with Steger-based methods.
	
	
	

§ RELATED WORKS

	
	This section reviews instance-aware part segmentation works from the human parsing field, then discusses existing works on the measurement of sperm morphology parameters.
	
	

 §.§ Instance-Aware Part Segmentation

	
	The instance-aware part segmentation network was originally proposed in the human parsing field, with the aim of segmenting various human parts (e.g., arm, leg and hair) and associating each segmented part with the corresponding instance. State-of-the-art methods can be categorized into bottom-up methods and top-down methods. The bottom-up methods <cit.> first segment each part of the target as in semantic segmentation <cit.>, then group segmented parts into corresponding instances. For example, the part grouping network (PGN) <cit.> utilizes line information to separate adjacent instances. Gray-ML <cit.> and Graphonomy <cit.> employ graph neural networks to explore underlying label semantic relations. Although the bottom-up methods are suitable for global semantic segmentation, their instance distinction ability is poor since they lack the bounding box locating procedure.
	
	To accurately separate adjacent instances, the top-down methods first locate each instance within the image using bounding box detection, then segment parts for each located instance individually. Based on Mask R-CNN <cit.>, the classical instance segmentation network, Parsing R-CNN <cit.> utilizes a geometric and context encoding module for improving part segmentation accuracy. RP-R-CNN <cit.> further boosts the parsing ability by introducing a global semantic network. AIparsing <cit.> utilizes FCOS <cit.>, an anchor-free detector instead of the anchor-based detector such as in Mask R-CNN to improve detection performance. Although the top-down methods have strong instance distinction ability, they suffer from context loss and feature distortion due to bounding box cropping and resizing in ROI Align.
	
	

 §.§ Measurement of Sperm Morphology Parameters

	
	Existing works mostly focused on sperm head and midpiece. In <cit.>, head morphology parameters and midpiece morphology parameters were calculated by ellipse and rectangle fitting. In <cit.>, the head-midpiece angle was calculated as the angle between the major axis of head and the major axis of midpiece. However, morphology measurement of sperm tail is challenging and mostly ignored because sperm tail is long, curved, with uneven width and cannot be easily fitted by ellipse or rectangle. The methods proposed for measuring curvilinear structures can be categorized into pixel level methods and sub-pixel level methods. Pixel level methods include skeleton extraction methods <cit.> and direction template methods <cit.>. To improve measurement accuracy, sub-pixel level methods were developed, including grey centroid methods <cit.> and Steger-based methods <cit.>. The grey centroid methods first extract centroid points through vertically or horizontally line-by-line scanning, then link all centroid points to form the centerline. However, these methods perform poorly if the curvilinear structure is not directed either vertically nor horizontally <cit.>. In comparison, Steger-based methods calculate center points along the normal direction of centerlines, which are not affected by the direction of the curvilinear structure <cit.>.
	
	
	

§ METHODS

	
	To achieve quantitative morphology measurement for sperm parts including head, midpiece and tail, two key techniques are developed in this work. A novel attention-based instance-aware part segmentation network is designed to accurately segment sperm parts; and an automated centerline-based tail morphology measurement method is developed to quantitatively calculate tail morphology parameters based on the segmented masks.
	
	

 §.§ Attention-Based Instance-Aware Part Segmentation Network

	
	The overall structure of the proposed network is shown in Fig. <ref>. The input image is first fed into the convolutional backbone to extract features, then the extracted features are rescaled by the Feature Pyramid Network (FPN) <cit.> to acquire multi-scale features. Next, the preliminary segmentation module generates instance-level parsing masks following the top-down methods <cit.>. Firstly, sperm within the image are located by bounding boxes using candidate ROIs (region of interests) from the Region Proposal Network (RPN) <cit.> in the detection branch. Secondly, each sperm within a bounding box is cropped and resized by ROI Align to have a uniform shape, finally segmented into part masks separately. It is worth noting that segmentation is performed on P2 features generated by FPN because P2 features in Fig. <ref> have the highest resolution and contain more details. However, similar to top-down methods, the parsing masks produced by the preliminary segmentation module suffer from context loss and feature distortion (see Fig. <ref>(a)&(b)).
	
		
	
	
	To solve this issue, an attention-based refinement module is proposed to refine preliminary segmented masks. In the proposed refinement module (see Fig. <ref>), parsing masks from the preliminary segmentation module are used to provide spatial cues of each located sperm instance, and features extracted by FPN are used to compensate for lost details. Then parsing masks and extracted features are merged through the attention mechanism and later refined by CNN to produce improved instance-aware results. Moreover, edge information is incorporated to better separate the boundary between adjacent sperm.

	
	Attention mechanism <cit.> has proven to be an effective tool to selectively emphasize specific regions of an image. Therefore, the preliminary segmented masks can be used to indicate the location for each sperm during segmentation using the attention mechanism. Specifically, parsing masks {M^i}_i=1^n (for n sperm) are first reshaped and re-projected back to their original position in the image using the bounding box information. Then, the re-projected masks {M^i_r}_i=1^n go through a 1 × 1 convolution and sigmoid activation to generate attention maps according to
	
    A^i=Sigmoid(Conv_1×1(M^i_r))

	where A^i denotes the attention map encoding spatial information for the i-th sperm. As for extracted features from FPN, multi-scale features (i.e., P2, P3, P4, P5) are merged so that targets covered by small to large receptive fields are incorporated. Specifically, P3, P4, P5 are first upsampled to the same size as P2, then P2 and upsampled P3, P4, P5 are concatenated and go through a 1 × 1 convolution to produce fused features according to
	
    F_fuse = Conv_1×1(Concat[F_P2, F_P3, F_P4, F_P5])

	where F_fuse is the fused multi-scale features. Attention maps {A^i}_i=1^n are then multiplied with fused features individually to produce instance-aware part features for each sperm. Finally, the produced instance-aware part features are refined by cascaded dilated convolution layers <cit.> with dilation rates of 1 and 2 to generate accurate parsing masks for each sperm by
	
    {S^i_j}_j=1:m=Conv_1×1(DilatedConvs(A^i⊙ F_fuse))

	where {S^i_j}_j=1:m is the segmentation result for the i-th sperm with m parts, and the final output for n sperm are {S^i_j}^i=1:n_j=1:m. In addition, the edge-guided branch is added in parallel to the parsing branch to provide boundary information.
	
	As for the loss, weighted cross entropy loss <cit.> and dice loss <cit.> are employed since sperm have a low area ratio compared with the background. The loss for the refinement module is
	
    ℒ_refinement=ℒ_parsing^WCE + ℒ_edge^WCE + λℒ_parsing^Dice + λℒ_edge^Dice

	where ℒ_parsing^WCE, ℒ_edge^WCE,  ℒ_parsing^Dice, ℒ_edge^Dice are weighted cross entropy loss and dice loss for the parsing branch and the edge branch respectively; and λ is a manually selected weight.
	
	

 §.§ Automated Centerline-Based Tail Morphology Measurement

	
	In Steger-based methods <cit.>, partial derivatives r_x, r_y, r_xx, r_xy, r_yy of each pixel are first obtained by convolving the image with a discrete 2-D Gaussian partial derivative kernels. Then the normal direction 𝐧(t) = [n_x, n_y]^T for each pixel is determined by calculating the eigenvalues and eigenvectors of the Hessian matrix H(x,y) = [[ r_xx r_xy; r_xy r_yy;      ] ]. According to <cit.>, the normal direction is perpendicular to the direction of curve-line, and centerline points are selected as points whose first derivative reaches zero (i.e., grey value reaches maximum) along normal (perpendicular to the line). Finally, points with minimum distance and minimum orientation change of normal are linked to form the centerline. However, endpoints detected by Steger-based methods tend to mislocate (see Fig. <ref>(c)) because their normal is affected by gradient from the intersecting edge and not perpendicular to the direction of line (see Fig. <ref>(d)).
	
	To solve this issue, an outlier filtering method and an endpoint detection algorithm are proposed. The outlier filtering method is used to filter mislocated endpoints produced by Steger-based methods, and the endpoint detection algorithm is employed to reconstruct correct endpoints.
	
	As shown in Fig. <ref>(a), the correctly detected point lies in the middle of two opposite edges while the mislocated point lies on the diagonal of two intersecting edges. Therefore, the normal lines that cross the correctly detected and mislocated point intersect with opposite edges and intersecting edges correspondingly. Suppose P_1, P_2 are correctly detected and mislocated point, respectively, N_1, N_2 are the corresponding normal, and E_11, E_12, E_21, E_22 are corresponding intersecting points with edges. Since normal N_1 is perpendicular to the direction of line <cit.>, the gradient for E_11, E_12 (on the opposite edges) are close to parallel. On the other hand, since E_21, E_22 are on intersecting edges, the gradient for E_21, E_22 are not possible to be parallel. Thus, a constraint can be formed to discriminate correctly detected and mislocated points, i.e., 
	
	
	
	
    {   cosα = rx_1 × rx_2 + ry_1 × ry_2/√(rx_1^2 + ry_1^2)×√(rx_2^2 + ry_2^2)
        |cosα| ≥ 0.9
    .

	where rx_1, ry_1 and rx_2, ry_2 are first partial derivatives for edge points E_1 and E_2, receptively.
	
	After filtering mislocated points, the next step is to reconstruct missing endpoints. Without loss of generality, for any point P on the centerline, its gradient G can be decomposed into two directions G^'_x and G^'_y, along the direction of the line and the direction perpendicular to the line (see Fig. <ref>(b)). According to <cit.>, the first derivative (gradient) for center points along the normal direction (perpendicular to the line) is zero (i.e., G^'_y = 0), which means G = G^'_x and gradient direction for point P is along the direction of line. Thus, an endpoint detection algorithm is proposed by interpolating points along the direction of center point's gradient. As shown in Fig. <ref>(b), for a point P_A(c_x, c_y), if the angle of gradient is within [0^∘, 45^∘], then point P_B(c_x + 1, c_y + 1) and P_C(c_x + 1, c_y) along gradient direction of P_A are selected as candidate points. Then, the candidate point that minimizes w_1 × d + w_2 ×β is selected as the next point, where d is distance between candidate point to gradient of current point, β is the angle difference between the gradient of candidate point and current point, and w_1, w_2 are weights. Moreover, to avoid local minima and smooth the line, gradient for the selected next point is updated using the momentum technique <cit.> as 
	
    g_t+1^' = α· g_t + (1 - α) · g_t+1

	where g_t, g_t+1 is the gradient for current point and next point, α is the momentum hyper-parameter, and g_t+1^' is the updated gradient. Meanwhile, the normal for reconstructed endpoints is corrected as the direction perpendicular to gradient. This algorithm is continued until the interpolated point is out of the tail area.
	
	
	
	

§ EXPERIMENTS AND RESULTS

	
	

 §.§ Datasets and Evaluation Metric

	
	In this work, we collected a new dataset[The dataset is available at https://github.com/Chenwy10/Sperm-Dataset] for the sperm part segmentation task to distinguish unique sperm instances and also segment parts (i.e., acrosome, vacuole, nucleus, midpiece and tail) for each sperm. All images were manually labelled with each part of the sperm assigned both an instance ID and a part label. Each image has the resolution of 1280 × 1024 and includes 5-6 sperm averagely. The dataset contains 320 images in total, with 80% of them randomly selected as training samples and 20% randomly selected as testing samples.
	
	
	
	
	
	To evaluate the performance of our proposed network, the standard mean intersection over union (mIoU) <cit.> was adopted for evaluating global semantic segmentation; and the Average Precision based on part (AP^p) <cit.> and Percentage of Correctly parsed semantic Parts (PCP) <cit.> were used to measure instance-level segmentation accuracy. Specifically, the AP^p_50, AP^p_vol and PCP_50  values were reported, where the first and third metric has an IoU threshold of 0.5, and the secondly metric is the mean of a series of IoU thresholds ranging from 0.1 to 0.9, with the increment of 0.1.
	
	
	

 §.§ Comparisons with State-of-the-art Models

	
	To evaluate the performance of sperm part segmentation, the proposed network was compared with several state-of-the-art instance-aware part segmentation networks, including PGN <cit.>, GPM <cit.>, Deeplab V3+ <cit.> for bottom-up methods; and Parsing R-CNN <cit.>, RP-R-CNN <cit.>, AIParsing <cit.> for top-down methods, all using our collected dataset. Qualitative and quantitative comparison results are summarized in Fig. <ref> and Table <ref>, respectively. As can be seen, bottom-up methods gave low AP^p_vol, AP^p_50 and PCP_50 for instance-level segmentation (see Table <ref>) and cannot separate intersecting sperm parts (see Fig. <ref>). This is because they have low instance distinction due to the lack of the bounding box locating procedure. On the other hand, top-down methods achieved significantly higher AP^p_vol, AP^p_50 and PCP_50 in instance-level segmentation compared with bottom-up methods (e.g., 22.1%, 28.5% and 23.9% gains in AP^p_vol, AP^p_50 and PCP_50 for RP-R-CNN vs. PGN). This is because top-down methods firstly locate each sperm within an image through detection and can better separate adjacent sperm.
	
	In comparison, the proposed network further outperformed top-down methods by refining preliminarily generated masks to produce improved results. Specifically, our proposed network achieved 9.7%, 9.2%, 14.4% and 15.6% gain in terms of mIoU, AP^p_vol, AP^p_50 and PCP_50, respectively compared with the state-of-the-art RP-R-CNN. Moreover, as shown in Fig. <ref>, the proposed network can more effectively retrieve lost contexts outside a bounding box and fix distorted features compared with RP-R-CNN thanks to the refining procedure. Therefore, the proposed network yielded segmentation results that were closer to the ground truth with more details and less distortion.
	
	0.07in
	
	

 §.§ Quantification of Morphology Parameters

	
	Sperm morphology measurement was performed on the segmented masks generated from the proposed network. Quantified values for each part are summarized in Table <ref>, where Sperm 1,2,3,4 correspond to the sperm shown in the third row of Fig. <ref>. The head and midpiece parameters were calculated through ellipse and rectangle fitting <cit.>, while the tail parameters were calculated using the proposed automated centerline-based tail morphology measurement method (see demo in Video).
	
	To evaluate the performance of tail morphology measurement, an additional experiment was conducted on 50 sperm using classical Steger-based methods and our proposed method. In this experiment, the length, width and curvature of sperm tail were used for quantitative comparison, with the benchmark data obtained by manually benchmarking zoomed-in masks in ImageJ with best care <cit.>. Qualitative and quantitative comparison results are summarized in Fig. <ref>(a)(b)(c). The errors of tail length, width and curvature measurements with the Steger-based method were 8.87%, 6.63%, 21.90%, respectively. These error were mostly caused by mislocated endpoints and had significant impact on curvature measurement (see Fig. <ref>(a)). In comparison, our proposed method successfully reconstructed mislocated endpoints using the proposed outlier filtering method and endpoint detection algorithm (see Fig. <ref>(b)), and substantially decreased measurement errors by 4.21%, 3.02%, 13.10% for tail length, width and curvature, respectively (see Fig. <ref>(c)). The errors for tail measurement with our proposed method were <10% for length, width and curvature, well meeting the requirement for determining normal/abnormal  sperm <cit.>.
	
	
	
	

§ CONCLUSION

	
	This paper presented automated techniques for quantitatively measuring morphology parameters for each part of a sperm. An attention-based instance-aware part segmentation network was proposed to accurately segment each part of the sperm by avoiding context loss and feature distortion. An automated centerline-based tail morphology measurement method was designed to quantitatively calculate tail parameters based on the segmented masks. Experiments showed  that the proposed network outperformed existing state-of-the-art networks, and the proposed automated centerline-based tail morphology measurement method was capable of accurately quantifying tail morphology parameters. 
	
	-11cm   
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	IEEEtran
	
	
	


 Paper Title: MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection.





 



Marrying Object Recognition and Vision-Language Models






K. Wang et al.




School of Computer Science and Engineering, Sun Yat-sen UniversityHefei University of Technology Tencent America Dalian University of Technology Peng Cheng Laboratory School of Computer Science and Engineering, National Engineering Research Center of Digital Life, Sun Yat-sen University


wangk229@mail2.sysu.edu.cn, chenglc@hfut.edu.cn, chenwk891@gmail.com, zhpp@dlut.edu.cn, linliang@ieee.org, {isszf, liguanbin}@mail.sysu.edu.cn

MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection
    Kuo Wang10000-0003-2522-4134 Lechao Cheng2^⋆ Weikai Chen3 Pingping Zhang4 Liang Lin1,5 Fan Zhou6 Guanbin Li1,5Corresponding authors.

    August 5, 2024
=========================================================================================================================================





§ ABSTRACT 
Learning from pseudo-labels that generated with VLMs (Vision Language Models) has been shown as a promising solution to assist open vocabulary detection (OVD) in recent studies. However, due to the domain gap between VLM and vision-detection tasks, pseudo-labels produced by the VLMs are prone to be noisy, while the training design of the detector further amplifies the bias. In this work, we investigate the root cause of VLMs' biased prediction under the OVD context. Our observations lead to a simple yet effective paradigm, coded , that generates significantly better training targets and optimizes the learning procedure in an online manner by marrying the capability of the detector with the vision-language model. Our key insight is that the detector itself can act as a strong auxiliary guidance to accommodate VLM's inability of understanding both the “background” and the context of a proposal within the image. Based on it, we greatly purify the noisy pseudo-labels via Online Mining and propose Adaptive Reweighting to effectively suppress the biased training boxes that are not well aligned with the target object. In addition, we also identify a neglected “base-novel-conflict” problem and introduce stratified label assignments to prevent it. Extensive experiments on COCO and LVIS datasets demonstrate that our method outperforms the other state-of-the-arts by significant margins. Codes are available at <https://github.com/wkfdb/MarvelOVD>.









§ INTRODUCTION



Open-vocabulary object detection (OVD) <cit.> is receiving increasing attention due to its capability of detecting novel objects at test time. In a typical OVD setting, only a fraction of the target categories is annotated (referred as the base categories) while the goal of OVD is to recognize a set of novel classes at inference time. The objects of novel categories can appear in the training images but do not receive any annotations.
To enhance the generalizability of the OVD detector, recent works have proposed to incorporate the vision-language models (VLMs) <cit.>, which have been verified with excellent zero-shot recognization capacity, to improve the existing OVD pipeline. 




A common practice for the OVD task with known novel concepts is to generate pseudo-labels using the VLMs (e.g. CLIP <cit.>) in an offline manner <cit.>. However, because of the domain shift between the contrastive language-image pretraining and object detection tasks, VLMs trained with image-level data inevitably introduce noisy annotations when applied to the cropped partial images. We demonstrate an in-depth analysis of the pseudo-labels generated by the CLIP model in Figure <ref>(a). We denote the valid novel object proposal mistakenly classified as other categories as “Mis-class” and boxes that should not be considered to contain a novel object as “Noise”. In fact, the mis-classification rate of the VLM-based method is rather low (only 3.3%). The main source of error stems from its incapability of distinguishing “noisy” boxes (error rate 76.6%), e.g., the dog leg in Figure <ref>(a), that should not be considered as a valid object of interest.

The key reasons for the difficulty of the VLMs to recognize noisy proposals are two-fold. 
1) The lack of contextual information to understand the locally cropped images. 
Instead of trained with image patches, the CLIP model is fed with complete images with paired texts.
Therefore, it is not able to leverage the image context outside the input proposal, which may be crucial to interpret the semantics of the candidate box. For instance, in Figure <ref>(a), the man's arm is falsely classified by the CLIP model as “tie”, as it fails to recognize that the seemingly “tie” object is in fact connected to a human body. 
2) The unawareness of the “background” elements. The CLIP model generates the category prediction by computing the similarity between the query image feature and the text embedding of the candidate categories. Since “background” is relatively defined according to interested foreground categories, there is no pre-defined text embedding to represent the concept of “background” during the inference. However, the CLIP model still has to provide a prediction even when the input content is not related to any of the target categories.
In Figure <ref>(a), the dog leg falls in this case – it is identified as “cow” only because it appears more like cow than any other category, leading to noisy boxes.

Unlike the CLIP model, the RoI align technique in detectors naturally provides rich contextual information for local regions. Moreover, the detector is aware of the  concept of “background” during inference. As a result, the noisy boxes that confuse VLMs can be clarified by the detector as “background” with high confidence. Inspired by this key observation, we propose , a dedicated framework for open-vocabulary detection that can yield high-quality pseudo-labels and marvelous performance by combining the merits of the object detector and the vision-language models. In particular, we leverage the “context and background” awareness of the detector as strong auxiliary guidance to comprehensively improve the pseudo-label generation pipeline and the training procedure.

For pseudo-label generation, the predicted category confidence is based on a weighted sum of the outputs from the detector and the VLM, favoring the reliable classification of the VLM models while ruling out noisy boxes using the detector. To accelerate the training, we pre-generate the VLM predictions on all the candidate boxes and dynamically mine credible pseudo-labels under the guidance of the detector at each training iteration. The complementary capabilities of the detector and the VLM significantly improve the accuracy of pseudo-labels, even at the early training stage. Moreover, as the detector improves during training, the quality of the generated pseudo-labels increases as well, which eventually boosts the final performance as shown in Figure <ref>(a).


Conventional training design of object detectors <cit.> equally treats each proposal that matches with one training target. Such a design is not suitable for learning from pseudo-labels. Specifically, the generated pseudo-box may deviate a lot from the bounding box of the real novel object. Therefore, as shown in Figure <ref>(b), the overlaps between the training boxes and the actual novel object usually present a large variance. 
This means that these training boxes should not make equal contributions to the final loss, even if they match the same pseudo-label. To this end, instead of weighting the pseudo-labels <cit.>, we adaptively compute individual weights for each training box that is matched with a pseudo-label. As shown in Figure <ref>(b), training boxes with inaccurate positions will receive smaller weights and vice versa. Note that the training boxes are produced with a stratified label assignment strategy, which eliminates the conflicts between the pseudo-labels and base annotations and thus prevents the negative influence of noisy pseudo-label on the performance of base category detection. We conduct extensive experiments on COCO and LVIS datasets. consistently outperforms the state-of-the-art methods by a great margin. In summary, our contributions are as follows.

     
  * We identify the fundamental causes of the VLM's biased prediction and provide in-depth analysis.
     
  * We propose , a novel OVD framework that generates high-quality pseudo-labels by leveraging the detector as auxiliary guidance to mitigate the domain gap of VLMs on image patch prediction.
     
  * A novel proposal re-weighting mechanism and the stratified label assignment method to improve the OVD training performance.
    
  * We set the new state of arts in the MS-COCO and LVIS benchmarks.
 






§ RELATED WORK



  
Vision-Language Pre-training 
Vision-Language pre-training, aiming to align visual and textual representations, employs contrastive learning on large-scale image-caption pairs <cit.>. Significant research has enhanced various downstream tasks using Vision-Language models <cit.>. Notably, models like CLIP <cit.> and ALIGN <cit.> have leveraged billion-scale image-text pairs for vision-language representation learning, achieving remarkable success in zero-shot image classification and image-text retrieval. This success has inspired the application of Vision-Language Models (VLMs) in enhancing the range and accuracy of dense recognition tasks, such as object detection <cit.> and semantic segmentation <cit.>. Nonetheless, VLMs, typically pre-trained by considering the entire image, face domain gaps in dense prediction tasks. Our research investigates the application of VLMs for object detection in localized regions, aiming to broaden the detector's cognitive scope without relying on manual annotations.


  
Open Vocabulary Object Detection Open Vocabulary Object Detection (OVD) aims to extend the detector's recognition capability to classes not present in the training data using auxiliary data or models. The concept was initially introduced by OVR-CNN <cit.>, which empowered the detector to recognize diverse object concepts by leveraging vision and text encoders pre-trained on image-text pairs. Subsequently, several OVD methods have been developed. Recent research in OVD has explored various forms of auxiliary data, including transfer learning with image-text pairs <cit.>, knowledge distillation from pre-trained Vision-Language Models <cit.>, pseudo-label generation from image classification data <cit.>, and pretraining with grounding data <cit.>. Except for CNN-based detectors, transformer-based open vocabulary detectors <cit.> have also been widely exploited. In addition to the standard OVD scenario, which assumes that novel categories are unknown during training, some existing works <cit.> have also explored open vocabulary detection with prior knowledge about potential novel concepts. Typically, in such cases, pseudo-labels for novel categories are generated using Vision-Language Models (VLMs) before training. However, the domain gaps between vision-language pre-training and object detection introduce noisy pseudo-labels, significantly constraining the performance of existing methods. To address this challenge, we identify a critical limitation in applying VLMs to localized regions and propose a solution that involves integrating the detector's capabilities to effectively mitigate this noise, leading to substantially improved performance.




§ PRELIMINARIES


Open-vocabulary object detection aims to train detection models by leveraging a dataset 𝒟={x_i, y_i}_i=1^n and auxiliary weakly supervised data (e.g. image-text pairs, VLMs, etc.), where x_i represents the image and y_i includes the location and category of the objects contained in the image. Different from conventional detection tasks, the annotation of images only covers the base categories 𝒞^B, while the OVD task requires the detector to additionally detect novel categories 𝒞^N at the test time. Note that 𝒞^B∩𝒞^N=∅, and the label space 𝒞^N is already known during the training.

To achieve the detection in an open-vocabulary label space, the classification head in the detector is designed to compare the similarity between the region-based visual embeddings and the text embeddings <cit.>. In particular, the region-embeddings R={r_i}_i=1^N_r are obtained through RoI Align and the following feature extractor, where N_r is the number of regional boxes in the images. The text embeddings are composed of C={c_bg}∪{c_i}_i=1^N_c, where c_i is obtained by feeding the category name with template prompts (e.g., “a photo of {category name} in the scene”) to the pre-trained text encoder and N_c is number of categories. The c_bg is initialized as a learnable embedding. Based on the regional and text embeddings, the probability of the region r_i being classified as category c_j is defined as  

    p_i,j = exp(r_i·c_j)/exp(r_i·c_bg)+∑_k=1^N_cexp(r_i·c_k).

Comparing the similarity between the region and text embedding enables the detector to recognize objects in an unlimited label space. 



§ MARVELOVD

To facilitate the learning of semantics associated with open categories without manual annotation, existing approaches often employ pre-trained Vision-Language Models (VLMs) to discover potential novel objects <cit.> and generate pseudo-labels for subsequent training. The typical procedure involves training a proposal generator using base annotations to identify localized regions that may contain novel objects, followed by the generation of pseudo-labels based on VLM inference results within these cropped regions. However, as VLMs are pre-trained on entire images, their application to localized regions inevitably introduces noisy pseudo-labels, leading to disruptions in the learning process for novel categories.
To enhance the learning of novel concepts, we present MarvelOVD, which dynamically integrates the detector's capabilities into the pseudo-label generation process while optimizing the subsequent learning stages. Figure <ref> provides an overview of our framework, with detailed explanations in the following sections.





 §.§ Candidate Pseudo-Label Assignment


Our proposed MarvelOVD dynamically generates better pseudo-labels at each training iteration under the guidance of both the detector and pre-trained VLMs. Since predicting the cropped regions with VLMs during training requires unaffordable time overhead, we alternatively assign candidate pseudo-labels to each image before the training and then select precise ones to train the detector at each iteration.

Following the existing method <cit.>, we first train a class-agnostic proposal generator with the base annotation to produce regional boxes for each image. Image patches are then cropped according to the regions and fed into the CLIP image-encoder to obtain the regional-visual-embeddings. At the same time, we utilize the corresponding CLIP text-encoder and template text prompts to encode each novel category. After that, a similarity matrix is computed via dot product to describe the similarity between each visual-embedding and text-embedding. Finally, softmax is applied to obtain the distribution over novel categories for each region. Based on it, the conventional method <cit.> post-processes the boxes and selects high-confidence pseudo-labels to train the detector. The problem is that the domain gap easily leads CLIP to produce high confidence predictions on noisy regions, which greatly limits the performance of the existing methods. In contrast, we record the CLIP predictions and assign candidate boxes to the image using a low threshold (e.g. 0.5), and then dynamically select precise pseudo-labels from the candidates under the guidance of the detector, which is introduced in detail in the following section.



 §.§ Online Pseudo-label Mining


The candidate pseudo-labels can be categorized into two groups: ground-truth boxes (those tightly enclosing actual novel objects) and noisy boxes (those that should not be designated as novel objects). The primary objective is to eliminate noisy candidates while retaining ground-truth boxes as reliable pseudo-labels for training. The key distinction between the detector and CLIP in their inference of localized regions lies in the contextual information and the “background” concept. The RoI Align mechanism employed by the detector excels at extracting contextual features from the boxes, a capability that is absent when cropping images based on coordinates alone. Additionally, the detection task incorporates a specific task-specific category known as “background”, a concept that CLIP remains unaware of when inferring localized regions. 
Failing to recognize the “background” objects and the lack of contextual features are the root causes of CLIP model generating wrong predictions for noisy boxes.
Though the CLIP model struggles with the noisy boxes, 
its predictions for the ground-truth boxes are highly accurate. 
Leveraging this insight, we propose to utilize the detector's predictions for these candidates to estimate whether a box  encloses real novel objects. 
Subsequently, we combine CLIP's classification results to select high-quality pseudo-labels.



  
Burn-in. To estimate the novelty of the candidate boxes, the detector first needs to learn “what is novel”. To achieve this, we first utilize the top-1 CLIP score (top score of the distribution predicted with CLIP's image-encoder and text-encoder) and a fixed threshold 0.8 (best threshold in previous work <cit.>) to initially select pseudo-labels to burn-in the detector for ω steps. After the burn-in phase, the model will initially gain the ability to distinguish between base objects, novel objects, and the background.



  
Online Object Mining. Online object mining officially begins after the burn-in phase. We draw ideas from semi-supervised learning <cit.> to derive weak-strong image pairs for the training, which enhances the learning for pseudo-labels. In particular, we first predict the candidate boxes with the detector on weakly augmented features. Based on it, we compute a novelty score for each candidate as follows:

    z_i = ∑_k∈𝒞^Nexp(r_i·c_k)/∑_j∈𝒞^B∪𝒞^N∪{c_bg}exp(r_i·c_j)

where r is the vision-embedding calculated by the detector and c is the text-embedding of categories. 𝒞^B and  𝒞^N are the sets of base and novel categories, respectively. The novelty score z_i relatively estimates the novelty of candidate boxes with respect to the base category and background. However, its value varies drastically with different degrees of convergence. To tackle this problem, we further apply max-norm to the novelty scores of candidates to obtain stable estimations:

    s_i^det = z_i/max{z_1, z_2, ⋯, z_N_r}

where N_r represents the number of the candidates. Benefiting from the contextual reasoning capacity of the detector and the awareness of background, the novelty estimation s^det computed by the detector can more precisely distinguish the ground-truth/noise candidates. Combined with the accurate classification prediction generated by the CLIP model, we finally calculate the confidence score for each candidate box as follows:

    s_i = λ s_i^CLIP + (1-λ) s_i^det

In the above equation, s_i^CLIP means the top-1 CLIP score and λ∈ [0,1] is a scalar that controls the dependency of two different models. We utilize a fixed threshold δ to select high-quality pseudo-labels. The training is then derived on both weakly and strongly augmented images. 

The incorporation of the detector significantly reduces the confidence of the noisy candidates, which greatly improves the accuracy of the selected pseudo-labels, even at the initial training phase. Moreover, as the model converges with the training, the novelty estimation s^det would be more accurate, which results in pseudo-labels of higher quality and ultimately boosts the model's detection performance on novel categories.



 §.§ Training

In this section we describe our improvements to the conventional training design of the detector <cit.>. All the proposed methods in this section are applied both in the burn-in stage and the following online-object-mining stage.


  
Stratified Label Assignment. 
The learning of novel concepts should not affect the model's performance in recognizing base objects. However, an easily overlooked phenomenon is that the mAP for base categories drops when novel pseudo-labels are applied for the training. The reason is that the novel pseudo-labels may overlap with the base annotation, resulting in 
“base-novel-conflicts” in the IoU-based label assignment. To tackle this problem we propose stratified label assignment, which first assigns proposals with base annotations by IoU-matching, and boxes that are marked as background in the first step are secondly matched with the pseudo-labels. Experiments demonstrate that stratified label assignment helps achieve 
high accuracy of detecting novel objects without compromising the performance on estimating base categories.





  
Adaptive Proposal Reweighting.
Since the localization quality of the pseudo-label is limited, the box center may be far away from the ground-truth object center. As a result, training boxes that matched with the mislocalized pseudo-label share extremely unbalanced overlaps with the ground-truth object. However, the conventional training design of detectors <cit.> equally derives training losses on those unbalanced boxes, which hampers the learning process. 
To resolve this issue, we propose adaptive proposal reweighting to assign independent loss weights to each training box that matches with pseudo-label.

The loss function to train the detector with adaptive proposal reweighting is computed as:

    ℒ = 1/N(∑_i=1^n^basel(b_i^base, 𝒢^base)+γ∑_i=1^n^novelw_i· l(b_i^novel, 𝒢^novel))

where N=n^base+n^novel (n^base includes background box) is the total number of training boxes, γ is the overall weight for novel concept learning and w_i represents the independent weights for each novel training box. In particular, we follow the design of Eq. <ref> to define the individual weight w_i as:

    w_i = λ^' s_i + (1-λ^')r_i


In Eq. <ref>, s_i indicates the confidence of the corresponding pseudo-label and r_i is a reliability score estimated for each matched training box. Estimating the reliability score r_i is crucial and challenging. We empirically find that background score predicted on the weakly augmented images keeps close negative correlation to the overlaps with actual object and define r_i = 1-b_i, where b_i is the background score predicted according Eq <ref>. We also examine other indicators for comparison, more details are shown in supplementary materials. With adaptive reweighting, training boxes with higher overlap to real novel objects will be given greater weights and vice versa, thus de-biasing the learning procedure for novel concepts and further improving the performance.




§ EXPERIMENTS



In this section, we evaluate our MarvelOVD framework against standard benchmarks, comparing it with current state-of-the-art approaches. Additionally, our ablation studies provide in-depth analyses of the primary issues leading to noises in traditional CLIP-based pseudo-label generation and detail how our framework effectively addresses these challenges.



 §.§ Datasets
 
Our primary experiments utilize the COCO-2017 dataset <cit.> in an open vocabulary setting <cit.>, dividing 48 base and 17 novel categories for evaluation. Annotations for base categories are provided, while only category names are available for novel classes. We calculate AP_50^Novel, AP_50^Base, AP_50^All, representing the mean Average Precision at an IoU of 0.5 for novel, base, and all categories, respectively. Additionally, the LVIS-v1 dataset <cit.> is employed in standard Open Vocabulary Detection (OVD) settings <cit.>, treating 337 rare categories as novel and the rest as base. For LVIS, we report box Average Precision (AP) averaged over IoUs from 0.5 to 0.95 for rare (novel), common, frequent, and all categories, denoted as AP_r, AP_c, AP_f, and AP.



 §.§ Implementation Details
 
We utilize ViT-B/32 CLIP as the pre-trained Vision-Language Model (VLM) and its text-encoder for encoding category concepts. Consistent with existing approaches <cit.>, our experiments on the COCO dataset employ Mask-RCNN <cit.> with ResNet50-FPN <cit.> as the base detector. For training, we initially select noise pseudo labels with CLIP scores above 0.8 and use this setup for the burn-in phase for ω=0.5k iterations. Subsequently, we set λ,λ^'=0.5 to integrate the detector with CLIP and δ=0.9 for generating precise pseudo-labels. We set γ=2 as the overall weights for novel concept learning. Training is conducted on 4 GPUs, with a total batch size of 16 across 90k iterations (including 0.5k for burn-in), using a starting learning rate of 0.02, reduced by a factor of 10 at 60k and 80k iterations. The image input size adheres to the standard configuration, with the short side ranging from [640, 800] and the long side under 1333. Additionally, we apply common weak-strong augmentations from semi-supervised object detection iterature <cit.> in pseudo-label learning. For the LVIS dataset, we replicate Detic's experimental setup <cit.> and apply our method to the CenterNet2 <cit.> baseline. The model is trained on 4 GPUs, while maintaining the total batch size unchanged. All experiments are conducted in Detectron2 <cit.>, with further details provided in the supplementary materials.









 §.§ Main Results


Our approach integrates the detector to refine pseudo-label generation and the subsequent learning phase, significantly reducing noises in pseudo-labels and training boxes. As shown in Table <ref>, our method outperforms the baseline method <cit.> substantially in inferring both base and novel categories. The improvement in base categories stems from stratified label assignment, ensuring undisturbed learning of base categories despite pseudo-labels. The detector's integration effectively mitigates CLIP's inability in distinguishing noise in localized regions, enhancing pseudo-label quality and, in turn, the detector's ability to identify novel objects. Our method is also compared with other state-of-the-art open-vocabulary detection techniques utilizing pseudo-labeling in Table <ref>. While existing methods often rely on auxiliary data or supervision, like internet-sourced image-text pairs <cit.>, pseudo-region-text pair pre-training <cit.>, or auxiliary image-level labels <cit.>, our approach addresses and resolves fundamental issues in pseudo-label generation and conventional training designs <cit.>, achieving significant gains without extra data or supervision. Results on the LVIS dataset (Table <ref>) 
compares our method against the common CenterNet2 baseline <cit.>. Contrary to existing methods <cit.> that use additional classification data with image-level labels for enhanced novel object detection, our method exploits potential novel objects from original training data. The results in Table <ref> indicate that our method also excels in large-scale label spaces.







§ ABLATION STUDY


We conduct experiments on the COCO dataset to assess the effectiveness of our method's key components, with additional ablations detailed in supplementary materials.



 §.§ Effects of Each Component

Table <ref> demonstrates the contribution of each proposed algorithmic component to the final performance. 
Initially, we change the global training setting of VL-PLM <cit.>, including overall novel loss weight γ and the data augmentations. A larger novel loss weight doesn't affect the performance while Weak-Strong augmentations enhance the learning of pseudo-labels. We then implement stratified label assignment to resolve conflicts between novel pseudo-labels and base annotations. This adjustment restores base category detection to supervised performance levels without impacting novel category detection. After that, we introduce online object mining to purify the pseudo labels, which brings a significant improvement in detection accuracy on the novel categories, indicating the effectiveness of our method in offsetting CLIP's localized limitations by leveraging the detector's capabilities. Based on it, applying adaptive proposal reweighting further enhances the average precision for novel categories. The promotion comes from the independent weights computed by adaptive reweighting, which enforces the model to focus on boxes with larger overlaps with actual novel objects. In summary, our method offers a less biased pipeline for pseudo-label-based novel concept learning, which not only effectively purifies the training targets but also optimizes the learning procedure, and significantly enhances performance in both base and novel categories without requiring extra data or pretraining.





 §.§ Analysis of Pseudo Labels





We conduct an in-depth analysis of pseudo-labeling quality across different training stages using the COCO evaluation set, which comprises 2064 images featuring novel objects. Pseudo-labels with an IoU score above 0.5 relative to the ground truth (GT) novel object and correctly categorized are classified as True Positives (TPs). The findings are depicted in Figure <ref>. For equitable comparison, we align the threshold of CLIP-based pseudo-labels (δ=0.95) with our dynamic pseudo-labels (δ=0.8) to maintain comparable recall rates. Initially, the burn-in stage imparts novel object discrimination ability, leading to a stable improvement in pseudo-label quality right after this phase. Subsequently, the enhanced pseudo-labels refine the detector's ability to recognize novel objects, manifesting as a large increase in pseudo-label precision in the early stages of training. As illustrated in Figure <ref>, as training progresses, the detector increasingly differentiates novel boxes from background and base objects, dynamically enhancing pseudo-label quality. 






























 §.§ Effects of different thresholds and burn-in steps





We evaluate the performance impact of various thresholds and burn-in steps, detailed in Table <ref>. Our base setting uses a threshold δ=0.9 and ω=0.5k burn-in steps. Notably, the threshold for pseudo-label selection markedly affects model performance; while 0.8 is optimal for our baseline, a threshold of about 0.9 proves more effective for our method due to less biased pseudo-labeling confidence. Regarding burn-in steps, which guide initial learning from CLIP-generated pseudo-labels, their effect on final performance is minimal. As the model converges, the quality of these pseudo-labels improves, indicating that different initial settings eventually yield similar performance outcomes.




 §.§ Dependency analysis of λ and λ^'




We calculate the confidence of each candidate pseudo-label using Eq. <ref> and the training weights for each novel box via Eq. <ref>, where λ and λ^' determines the reliance on different models or measurements. 
The effects of varying λ and λ^' are documented in Figure <ref>. Optimal performance is observed both at 0.5, and a range of [0.3,0.7] yields comparable outcomes. 
Specifically, extreme values were also tested: λ=0 implies reliance solely on the detector post burn-in, which results in poor performance, underscoring the importance of CLIP's role in distinguishing novel categories. Conversely, at λ=1, pseudo-label generation reverts to conventional methods <cit.> that rely entirely on the CLIP model. While performance decreases in this setting, it still surpasses the baseline, suggesting that adaptive proposal reweighting effectively counters the impact of noisy boxes. By setting λ^'=1, adaptive reweighting changes to the original training design <cit.> with weighted pseudo-labels <cit.>, and it limits the model's learning of novel concepts, resulting in significant performance degradation.






§ CONCLUSION


In this paper, we address the limitations of pre-trained Vision-Language Models (VLMs) in generating accurate pseudo-labels for localized regions by integrating object detectors' capabilities. The key issue with VLMs is their lack of contextual awareness and inability to differentiate “background”, leading to biased pseudo-labels. By leveraging the detector's contextual feature extraction and background discrimination abilities, we significantly improve pseudo-label quality through online object mining and optimize the learning process with adaptive proposal reweighting. Our extensive experiments show that this approach not only enhances the detector's novel object recognition but also outperforms state-of-the-art methods without additional data or supervision, offering an efficient and effective solution for learning open vocabulary concepts by pseudo-labels.



  
Acknowledgments. This work was supported in part by the Key-Area Research and Development Program of Guangdong Province (NO. 2021B0101420004), in part by the National Natural Science Foundation of China (NO. 62322608, NO. 62106235, NO. 62325605), in part by the Guangxi Science and Technology Plan Project (NO. GuikeAD23026034), in part by the Shenzhen Science and Technology Program (NO. JCYJ20220530141211024), and in part by the Open Project Program of the Key Laboratory of Artificial Intelligence for Perception and Understanding, Liaoning Province (AIPU, No. 20230003).







§ FURTHER ABLATION STUDIES





 §.§ Different indicators for adaptive reweighting




We examine several other measurements to estimate the reliability score r_i for adaptive reweighting, including:


  * Confidence of pseudo label (r_i=s_i): For comparsion. By setting r_i=s_i, the adaptive reweighting degenerates to conventional training design <cit.> with weighted pseudo-labels <cit.>.

  * Intersection-over-Union (r_i=iou_i): Boxes with bigger overlaps from pseudo-label get larger weights and vice versa.

  * Novelty estimation (r_i=s_i^det): s_i^det defined in Eq.3 in main paper estimates the probability of containing real novel objects, we repeatedly use it for both pseudo-label mining and adaptive reweighting in this setting.

As demonstrated in Table <ref>, background score predicted with weakly augmented images achieves the best performance, and it significantly outperforms the conventional training design <cit.> with weighted pseudo-labels <cit.>.



 §.§ Ablation on global novel loss weight γ


Conventional methods <cit.> typically treat noisy pseudo-labels as ground truth and combine it with the base annotations to train the detector. In contrast, our proposed MarvelOVD strives to reduce the noises in both pseudo-labels and training boxes before utilizing them in training, which achieves more promising results. We examine different global novel loss weights and report the performance in Table <ref>. The results show that setting the global novel weight as γ=2 further improves the performance of our MarvelOVD while slightly degrading the baseline method. The reason is that the massive noise contained in the baseline method prevents further improvements. On the contrary, our approach effectively purifies the pseudo-labels and de-bias the following training designs by online mining and adaptive reweighting, allowing better performance with larger novel loss weights. Higher novel weights are also tested but do not contribute to better results.



 §.§ Qualitative Results


In Figure <ref>, we visualize the effects of our proposed stratified label assignment and online object mining. A notable observation is that base boxes might be incorrectly labeled as novel objects due to overlaps between the pseudo-label and base annotation, potentially impairing the detector's performance on base categories. Our stratified matching strategy rectifies these mislabeled base boxes, enabling the model to assimilate novel concepts without diminishing its base detection capabilities. Additionally, the bottom two rows of Figure <ref> demonstrate the efficacy of online object mining, where our approach, aided by the detector, effectively filters noise from CLIP-generated pseudo-labels.




 §.§ Effects of the weak-strong augmentation



We apply the weak-strong augmentations that are not adopted by other OVD works. The motivation comes from recent semi-supervised learning methods (e.g. fixmatch <cit.>, unbiased-teacher <cit.>), which demonstrate that enforcing the same supervision between weak-strong augmented features leads to better performance for learning on pseudo-labels. We apply weak-strong augmentations on the conventional OVD method <cit.> and MarvelOVD to evaluate its effect and the results are shown in Table <ref>. The weak-strong augmentation barely influences the average precision on base categories while equally improving the performance on novel categories in both VL-PLM and our MarvelOVD. Without the augmentation, our framework still outperforms the base method by a significant margin.





§ COMPARISONS AND COMPATIBILITY WITH EXISTING OVD WORKS





 §.§ Performance Comparison

We mainly compare our method with other pseudo-label-based OVD works in the main paper. Table <ref> demonstrates a more complete comparison between our MarvelOVD and other existing OVD works, including both transfer learning and knowledge distillation methods. Among them, our method still performs favorably against the state-of-the-art methods. Pseudo-label plays an important role in recent OVD works, where most advanced methods learn novel concepts from pseudo-labels generated by pretrained VLMs. With respect to VLM-generated pseudo labels, our MarvelOVD identifies the root causes of its noises and proposes the dedicated noise-removal strategy by integrating the context-sensing capability of the detector, which consistently improves the recent advanced method by significant margins. 

In particular, extracting CLIP embedding by roi-align from CNN-based CLIP backbones has been exploited in recent studies <cit.>, which provides a substituting context-aware operation for pseudo-label generation. Even though, our approach still stably outperforms methods that exploit such operation (e.g. SAS-Det <cit.> that recently published on arxiv). The result further indicates the effectiveness of our MarvelOVD in de-noising the pseudo-label-based learning paradigms.






§ MORE IMPLEMENTATION DETAILS







 §.§ Weak-Strong augmentations

The detailed weak-strong augmentations adopted in our method are illustrated in Table <ref>, which is identical with the semi-supervised object detection work unbiased teacher <cit.>. We only apply it to the COCO dataset. The augmentation on the LVIS dataset follows the common CenterNet2 benchmark <cit.>.



 §.§ Candidate pseudo-label assignment


We follow the pseudo-label-generation pipeline in VL-PLM to assign candidate pseudo-labels to each image before the training. In particular, the class-agnostic proposal generator is actually a detector trained with the base annotation (regarding all the base annotations as one class). Then we infer the train image with the proposal generator and recursively refine the predicted boxes with the RoI head by 10 times. The recursive refinement improves the localization quality of the boxes. After gathering the candidate regions, the prediction probability distribution p_i for each box is encoded by CLIP ViT-B/32 as follows:

    r_i    = ϕ(E_img(R_i^1×) + E_img(R_i^1.5×))
    p_i = so   ftmax{r_i · E_txt(Novel Categories)^T}

E_img,E_txt is the image-encoder and text-encoder of CLIP, R_i^1× is the box produced by proposal generator and R_i^1.5× is a region cropped by 1.5× the size of R_i^1×. 
After getting the probability distribution for each box, we filter them with a threshold 0.5 and post-process the remaining with NMS to obtain the candidate pseudo-labels. We record the TOP-1 CLIP score and the predicted category of the candidates and assign them to the image, and then we dynamically select reliable ones for training under the guidance of detector. The threshold 0.5 is not a special hyperparameter that influences the performance, it's used to remove the redundant boxes that would never be selected as pseudo-labels, which accelerates the training speeds. Refining the localization with the RoI head and extracting the region-embedding with a larger area are existing techniques that adopted by the base method VL-PLM. We also maintain them in our candidate pseudo-label assignment process.



§ LIMITATIONS



Our proposed MarvelOVD provides a better measurement to purify pseudo-labels from the fixed candidate boxes. It can not promote the localization quality of the pseudo-label. Since the candidate boxes are produced by a proposal generator trained with only base annotations, the localization quality for novel objects is limited. As the detector gains more knowledge of the novel object through pseudo-labels during training, its ability to localize the novel object should also be enhanced. How to rationally utilize the detector to dynamically optimize the localization quality of pseudo-labels is worthwhile exploring in the future.








splncs04



 Paper Title: MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition.










Journal of  Class Files, Vol. 14, No. 8, August 2021
Shell et al.: MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition

0000–0000/00$00.00  2021 IEEE



MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition
    Wenqing Gan, Yan Sun, Feiran Liu, Xiangfeng Luo
        
This work was supported in part by the National Natural Science Foundation of China under Grant 62002215, and in part by the Shanghai Pujiang Program under Grant 20PJ1404400.
Wenqing Gan, Yan Sun, Feiran Liu and Xiangfeng Luo are with the School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China (e-mail: ganwenqing@shu.edu.cn; yansun@shu.edu.cn; shirleyllfr@shu.edu.cn; luoxf@shu.edu.cn).
    August, 2024
=================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
The objective of the panoramic activity recognition task is to identify behaviors at various granularities within crowded and complex environments, encompassing individual actions, social group activities, and global activities. Existing methods generally use either parameter-independent modules to capture task-specific features or parameter-sharing modules to obtain common features across all tasks. However, there is often a strong interrelatedness and complementary effect between tasks of different granularities that previous methods have yet to notice. In this paper, we propose a model called MPT-PAR that considers both the unique characteristics of each task and the synergies between different tasks simultaneously, thereby maximizing the utilization of features across multi-granularity activity recognition. Furthermore, we emphasize the significance of temporal and spatial  information by introducing a spatio-temporal relation-enhanced module and a scene representation learning module, which integrate the the spatio-temporal context of action and global scene into the feature map of each granularity. Our method achieved an overall F_1 score of 47.5% on the JRDB-PAR dataset, significantly outperforming all the state-of-the-art methods.



Human Activity Understanding, Group Activity, Panoramic Activity Recognition.




§ INTRODUCTION

Due to the extensive application of human activity recognition tasks in various fields such as video surveillance <cit.>, healthcare <cit.> and sports analysis <cit.>, this task has become crucial in video understanding. Previously, human activity recognition primarily focused on single-granularity tasks, such as recognizing individual actions <cit.> or group activities <cit.>. These tasks require a fixed number of people in the scene and assume that, in multi-person scenes, all individuals belong to a single group without any subdividable subgroups. These assumptions are often unrealistic in real-world scenarios, thereby limiting the utility of these methods. Moreover, focusing on single-granularity tasks makes it difficult to understand the entire scene, while tasks of different granularities can mutually reinforce each other. For instance, accurate individual actions recognition can help the model learn the commonalities of individual behaviors, thereby aiding in more accurate group activity recognition. To address these issues, the panoramic activity recognition task, as proposed by <cit.>, aims to establish a unified framework capable of recognizing behaviors at different granularities simultaneously in crowded and complex scenes, including individual actions, social group activities, and global activities. This task does not impose restrictions on the number of people in the scene and allows for the subdividing groups. Recognizing behaviors of different granularities within a unified framework leverages the complementary relationships between tasks, as different granularities introduce diverse inductive biases that broaden the model’s perspective, enhancing its generalization ability to unknown data distributions.

To establish a unified framework for simultaneous multi-task learning, the most critical factor is the structure design—specifically, determining which parts are task-specific and which are shared across tasks. This significantly impacts performance, training efficiency, and generalization. Current methods utilize parameter-sharing backbones and individual feature relationship learning modules to learn the foundational features common to multi-granularity activity recognition tasks, and then use multiple task-specific parameter-independent classification heads for multitask recognition. For cross-granularity aggregation modules, JRDB-PAR <cit.> uses parameter-independent modules to better capture specific features and nuances of each task, avoiding optimization conflicts of different task supervision signals; on the other hand, MUP <cit.> uses parameter-sharing modules to extract common relationships and motion features, promoting information sharing and enhancing overall performance. However, these methods do not effectively to simultaneously leverage the advantages of both aggregation strategies and overlook their complementary effects. Besides, since social groups and global activities are aggregated from individual actions, leveraging temporal information of individuals can lead to a better understanding of the scenarios. However, this requires multi-object tracking and group evolution detection, and they are very difficult to implement in challenging panoramic scenes. Therefore, previous methods ignored temporal information and relied on single-frame recognition, resulting in suboptimal performance. Furthermore, previous methods performed panoramic activity recognition using only cropped individual features, limiting the features to a local receptive field. As a result, they inevitably neglect the advantages of the global contextual information.

In this work, we propose a novel network called Mix-Parameters Transformer for Panoramic Activity Recognition (MPT-PAR) to address those challenges. A cross-granularity aggregation module is introduced to integrate features from individuals to social groups and from individuals to the global level. This module, comprising structurally identical parameter-independent and parameter-sharing encoders, models relationships between individuals through multi-head self-attention and aggregates individual behavior information via a learnable cls token. The fused features from both encoders are combined, to capture common relationships and motion features while retaining specific task features and nuances. To exploit temporal information, we follow the conventional approach for multi-object tracking and group evolution detection in group activity recognition, which is closely related to panoramic activity recognition. We also propose spatio-temporal relation-enhanced module, which model spatial relationships between different individuals in the same frame and temporal relationships of the same individual across frames. To provide comprehensive global context, we expand the receptive field by modeling scene information of the entire video frame, capturing behavior-related elements through spatial attention, flexibly integrating scene representations and activity features through cross-attention. Extensive experiments are conducted on the JRDB-PAR dataset to validate the effectiveness of the proposed network. Our method achieved an overall F_1 score of 47.5%, significantly improving by 6.0% compared to state-of-the-art methods. Additionally, it shows the greatest improvement in the sub-task of global activity recognition compared to previous methods, with an F_1 score of 61.1%. We also compared our approach with SOTA methods that perform single-granularity tasks, the significantly improvement demonstrating the mutual reinforcement between tasks of different granularities.

The main contributions of this paper are:


    
  * We propose a novel network MPT-PAR, it uses both parameter-sharing and parameter-independent aggregation encoders for cross-granularity feature integration, leveraging the complementary advantages to improve panoramic activity recognition.
    
  * We introduce spatio-temporal relation-enhanced module to utilize temporal information by enhancing spatial and temporal relationships for individual features.
    
  * We explicitly model scene information to provide comprehensive global context, captures behavior-related elements through spatial attention, and integrate scene representations with activity features through cross-attention.
    
  * Conducting extensive experiments on the JRDB-PAR dataset, we demonstrate the effectiveness of proposed method with an overall F_1 score of 47.5%, showing a significant improvement of 6.0% over state-of-the-art methods.




§ RELATED WORK




 §.§ Single Granularity Activity Recognition



  §.§.§ Human Action Recognition

Human action recognition (HAR) is a critical research area in computer vision that aims to automatically identify various human actions from video data. The advancement of deep learning has led to the development of numerous models based on Convolutional Neural Networks (CNN) <cit.> and Recurrent Neural Networks (RNN) <cit.> for this purpose. A milestone in this field is the Two-Stream Convolutional Network <cit.>, which utilizes static frame images and optical flow images to separately capture spatial and temporal features, resulting in significant performance gains. Additionally, RNN architectures like Long Short-Term Memory (LSTM) <cit.> and Gated Recurrent Units (GRU) <cit.> have been extensively applied to video sequence data, further improving action recognition accuracy. In recent years, Transformer-based frameworks <cit.> have been incorporated to enhance the model's ability to focus on key frames and regions, thereby boosting recognition performance. These techniques capture intricate spatio-temporal relationships and feature interactions, achieving state-of-the-art performance on several public datasets. Moreover, multimodal fusion approaches, such as networks combining depth information <cit.> and skeletal data <cit.>, have shown superior performance in complex scenarios.



  §.§.§ Social Group Detection and Activity Recognition

The task of social group detection aims to divide people into subgroups based on their social activities or relationships. Building on this concept, social activity recognition aims to simultaneously identify the social activities of each subgroup. Early methods focused primarily on social group detection due to hardware limitations and small datasets. Solera et al. <cit.> manually designed features representing body and social identity to understand relationships among group members and used trajectory clustering to detect social groups. Wang et al. <cit.> introduced a benchmark for social group detection with a “global to local magnification” framework that encodes global trajectories and local interactions, yielding promising results. Recently, the focus has shifted towards social activity recognition. Ehsanpour et al. <cit.> developed an end-to-end trainable network using integrated self-attention and graph attention modules to recognize social subgroups and their behaviors. However, this method oversimplifies social activities by treating them as the dominant individual actions within each group, which is not practical in many real-world scenarios. To address this, Kim et al. <cit.> created a large-scale high-definition dataset specifically for social behavior detection and proposed a new Transformer-based model capable of handling an unknown number of groups and potential group members without relying on clustering algorithms.



  §.§.§ Group Activity Recognition

It is a subfield of human activity recognition, focuses on identifying activities involving multiple individuals. The early work in this field primarily relied on manually designed features to extract information from video frames. Hierarchical graphical models <cit.> and dynamic Bayesian networks <cit.> were commonly used to interpret group activities in video environments. With the advent of deep learning, recent approaches have favored using deep neural networks for group activity recognition tasks. Yuan et al. <cit.> proposed a GCN-based method through a Dynamic Relation module and Dynamic Walk module for spatio-temporal individual inference. Xie et al. <cit.> introduced an actor-centric causality graph that focuses on analyzing the impact of two actors on asynchronous causality detection, complementing the learned asynchronous relationships with the synchronous ones derived from a Transformer model. GroupFormer <cit.> employs a clustering attention mechanism, leveraging a clustering space-time Transformer to enhance individual and group representations by integrating spatial and temporal dependencies. Addressing the reliance on bounding box annotations for group activity recognition, a detector-free Transformer-based model called DFWSGAR <cit.> was proposed, which locates and encodes parts of the group activity context through attention mechanisms, aggregating them into a single group representation without requiring bounding box labels or object detectors.

Overall, these methods focus solely on single-granularity tasks and they impose some unrealistic limitations. For instance, human action recognition often requires videos to contain only one or a few individuals, with the actors occupying the main part of the frame in each scene. In group activity recognition, the number of people in the video is usually fixed, and all individuals are assumed to belong to a single group without subdivisions. Those unrealistic assumptions significantly limit the application of these methods. Additionally, single-granularity task models did not realize the mutual reinforcement between tasks at different levels. In contrast, the objective of this work is to simultaneously achieve individual action recognition, social group activity recognition, and global activity recognition in crowded scenes, making it more practical for many real-world applications.





 §.§ Panoramic Activity Recognition

Panoramic activity recognition aims to identify individual actions, social group activities, and global activities concurrently. As a newly proposed and challenging problem, this field remains underexplored with limited current work. Han et al. <cit.> proposed a novel hierarchical graph neural network to progressively represent and model multi-granularity activities and mutual social relationships within crowds. By modeling each individual as a graph node, a graph network is constructed. Multiple parameter-independent GCN-based cross-granularity aggregation modules aggregate individual feature nodes into social group nodes bottom-up. Similarly, social group nodes are further aggregated into global nodes. This parameter-independent aggregation approach better captures the specific features and nuances of each task, avoiding optimization conflicts in supervising signals on the same parameters and improving individual task performance. In contrast, MUP <cit.> uses a unified cross-granularity aggregation module to encode behaviors at different granularity levels with the same parameters in an end-to-end manner. This approach learns potentially generic motion patterns across multi-granularity human behaviors, facilitating information sharing among tasks and enhancing the performance of panoramic activity recognition sub-tasks.

However, these methods only use either parameter-independent or parameter-sharing modules to integrate cross-granularity activity features, overlooking the complementary strengths of these two methods. Besides, these methods perform activity recognition at each granularity based solely on individual features extracted from a single key frame, without considering temporal information and the overall scene context. 




§ PROPOSED APPROACH



 §.§ Overview


The architecture of our network is illustrated in Fig. <ref>. The backbone is a ResNet-18 network <cit.> pre-trained on ImageNet. For the input video frame sequence X_img∈ R^T×3× H× W (T indicates the number of frames), the backbone extracts feature maps resulting in scene features  X_scene∈ R^T× C× H'× W' (C means the number of channels). Each individual's features X ∈ R^T × N× D (N, D represent the number of individuals and the dimension of individual's features respectively) are then cropped using RoIAlign <cit.>. The scene features X_scene are passed through a scene representation learning module to obtain the scene representation. The cropped individual features X are enhanced by spatio-temporal relation-enhanced module (STRE), which enrich individual features with spatio-temporal relational context. Subsequently, parameter-sharing and independent cross-granularity aggregation module (PSICGA) is employed to integrate individual features into social group and global representations. Before classifying the individual, social group, and global features, they are fused with scene representations through cross-attention to fully exploit the scene context.





 §.§ Spatio-Temporal Relation-Enhanced Module (STRE)


 Previous methods ignored temporal information and relied solely on single frames for panoramic human activity recognition. However, actions are typically executed over a sequence of continuous frames. Ignoring temporal information can result in the loss of action details, leading to inaccurate understanding of actions by the model, which in turn affects the recognition of social group activity and global activity. Moreover, in real-world scenarios, interpersonal relationships are dynamically changing, making it unreliable to infer groupings of people based on a single image.
 
 To address this issue, we introduce temporal information and use spatio-temporal relation-enhanced module to obtain features enriched with temporal and spatial context. Our STRE module consist of a transformer-based <cit.> temporal encoder and a spatial encoder, which learn temporal and spatial individual contextual features, respectively. Inspired by TimeSformer <cit.>, we connect the temporal and spatial encoders serially to reduce computational complexity. Experiments demonstrated that the serial structure is simpler and more effective (details in Section <ref>).





  §.§.§ Spatial Encoder

In crowded multi-person scenes, individual actions are not entirely independent but are significantly influenced by interactions with others and the behaviors of those present in the scene. Therefore, considering spatial context is crucial for panoramic human activity recognition. We employ a transformer-based spatial encoder to learn spatial contextual information of individuals. The spatial encoder captures interaction relationships between the target individual and surrounding individuals by assigning relative importance weights.

Given the input individual representation X ∈  R^T × N× D, we treat the temporal dimension as the batch dimension and use different learnable projection matrix to map X to Q, K and V. The spatial relationship attention weights of neighboring individuals are then computed using softmax, and the weights are multiplied by V to obtain the spatial context-enhanced feature X_S∈  R^T × N× D. For the t-th frame, this process can be expressed as:

    Q^t=X^tW_tq,K^t=X^tW_tk,V^t=X^tW_tv



    X_s^t = FFN(softmax(Q^tK^t^T/√(D))V^t+V^t)
where W_tq, W_tk and W_tv are learnable projection matrices, and FFN is a feedforward network.



  §.§.§ Temporal Encoder


Considering the temporal correlation of panoramic activities, the temporal encoder captures temporal contextual information by computing the adaptive importance of individuals across consecutive frames, enriching the temporal features of individuals. The structure of the temporal encoder is identical to that of the  spatial encoder. The difference lies in treating the spatial dimension as the batch dimension for the temporal encoder. We transpose the temporal and spatial dimensions of X_s, reshaping it to X'_s ∈  R^N× T × D, and feed it into the temporal encoder, resulting in the time-context-enhanced individual representation X_t∈  R^N× T × D. As temporal information is only considered at the individual level, we average the time dimension of the enhanced individual features to obtain X_st∈  R^N× D, serving as the foundation for constructing social group and global features.



 §.§ Parameter-Sharing and Independent Cross-Granularity Aggregation Module (PSICGA)


As a multi-granularity and multi-task learning task, panoramic activity recognition requires simultaneously addressing individual action recognition, social group activity recognition, and global activity recognition. Using parameter-independent aggregation modules for bottom-up cross-granularity feature integration helps capture task-specific features relevant to each task and avoids optimization conflicts in shared parameters, thereby enhancing multi-task learning efficiency. However, as indicated by MUP <cit.>, different parameter modules model activities at various granularities separately, making it impossible for the model to learn potential general motion patterns across different granularities of human behavior. Therefore, we design a module that integrates parameter-sharing and parameter-independent mechanisms to learn task-specific features independently and alleviate optimization conflicts between tasks, while capturing common behavioral patterns during cross-granularity aggregation through shared parameters.

As shown in Fig. <ref>, the proposed PSICGA module is constructed from two encoders: a parameter-sharing aggregation encoder (PSA) and a parameter-independent aggregation encoder (PIA). Although these two encoders share the same structure, which incorporates layer normalization, multi-headed self-attention and FFN, the parameters in them are different. The PSICGA module employs a parallel structure, where the input features are fed into both the PSA and PIA encoder. The network then combines the cls tokens from the outputs of the two modules to obtain the fused features.



For individual to global aggregation, we first add a learnable cls token X_cls∈  R^1× D to the individual feature sequence X_st∈  R^N× D. It summarizes the individual behavior context and the interactions learned by the multi-headed self-attention, representing the aggregated global activity features. To better model the relationships within the aggregation module, we add learnable spatial position embedding to provide positional information. The individual feature sequence X_st'∈  R^(N+1)× D with the added cls token and spatial position encoding  is fed into the PIA encoder PIA_global() and PSA encoder PSA(). The cls tokens representing global activity features from the two modules are combined to obtain the cross-granularity aggregated global activity feature X_global. The process is defined as:


    X_st'=[X_cls,x_st^1,x_st^2...x_st^n]+P



    X_cls^global = PIA_global(X_st'),X_cls^share = PSA(X_st')



    X_global=X_cls^global+λ X_cls^share

where P represents spatial position embedding, and λ is a weight parameter controlling the addition. 

For individual to social group integration, we first use clustering to obtain social group detection results, then perform cross-granularity aggregation for each social group. The PSICGA module used in this aggregation is similar to the one used in individual to social group integration, with the only difference being that the PIA encoder with different parameters PIA_social() is used to obtain the corresponding X_cls^social, while PSA() encoder is reused for both integrations.



 §.§ Scene Representation Learning


The global scene contextual information is crucial for panoramic activity recognition. Our scene representation learning module consists of scene representation generation and scene representation fusion. First, visual scene tokens are generated from the scene features extracted by the backbone, and they are aggregated to form the scene representation. Then, cross-attention is employed to fuse the scene representation with activity features at different granularities.


  §.§.§ Scene Representation Generation

Inspired by Visual Transformers <cit.>, the global scene information in video frames can be summarized using a compact set of visual tokens. We use a set of visual scene tokens to summarize the global scene information of the current video frame and aggregate them into a scene representation. For the scene feature map X_scene∈  R^T× D× H'× W' extracted by the backbone, we first add learnable positional encodings to retain positional information. Next, pointwise convolution is applied to assign each pixel to K visual scene tokens X_scene' ∈  R^T× K× (H'·W'). A spatial attention matrix A∈ R^T ×  K ×  ( H'· W') is generated using softmax, and the spatial attention weights are used to compute the weighted sum of each pixel. 2D convolution is applied to aggregate the scene features along the spatial dimension ( H'· W') to channel D, generating K visual scene tokens Z∈ R^T ×  K ×  D. This process can be described as:

    Z=conv(softmax(conv(X_scene))X_scene)

Through the spatial attention mechanism, each visual scene token adaptively focuses on notable parts of the entire scene. Finally, we perform AvgPooling on the first two dimensions of the visual scene tokens, merging the K visual scene tokens into the scene representation X_S∈  R^1× D.



  §.§.§ Scene Representation Fusion

To fully integrate the global context information contained in the scene representation with features at various granularities, we use a multi-head cross-attention and a feedforward network for fusion. For individual-level features, the inputs to the multi-head cross-attention are the enhanced individual representation X_t∈  R^N× 1 × D and the scene representation X_S∈  R^1× D. The individual representation is used as the query, and the scene representation  as the key in the multi-head cross-attention to fuse individual and scene representations, yielding the global scene context enhanced individual representation. This operation is similarly performed at the social group level. In the global activities recognition task, we directly concatenate the global-level features with the scene representation and use a learnable MLP to fuse them.



 §.§ Multi-Task Loss Function
 

Following JRDB-PAR <cit.>, we sum the losses of multiple tasks to serve as supervision for panoramic activity recognition. The total loss is defined as:

    ℒ   = ℒ_i + ℒ_s + ℒ_g + ℒ_d 
       = ∑_u L(𝐚^I_u, 𝐚̃^I_u) + ∑_k L(𝐚^S_k, 𝐚̃^S_k) + L(𝐚^G, 𝐚̃^G) + L(𝐑, 𝐑̃)

where ℒ_i, ℒ_s, ℒ_g, and ℒ_d represent the losses for individual, social group, global activity recognition, and social group detection, respectively. u and k denote each individual's features and social group features in the scene. L(·) represents binary cross-entropy loss, and ·̃ denotes the ground truth labels. R is the interpersonal relationship matrix with values of 0 or 1, where 1 indicates that the corresponding two individuals belong to the same group.




§ EXPERIMENT






 §.§ Datasets and Metrics

 The JRDB-PAR <cit.> dataset is a novel benchmark specifically created for the panoramic human activity recognition task. This dataset extends the JRDB-Act <cit.> dataset by manually annotating social group activities and global activities, and adding annotations for social group detection. The JRDB-PAR dataset comprises 360-degree RGB videos captured by a mobile robot in various densely populated areas such as campuses, cafeterias, and classrooms, providing 628,000 annotated personal bounding boxes with IDs. The dataset employs multi-class labels for activity annotation, meaning each individual, group, or frame can have one or multiple activity labels. It includes 27 individual action classes (e.g., walking, talking), 11 social group activities (e.g., chatting, working together), and 7 global activities (e.g., commuting, conversing). The dataset consists of 27 video segments, totaling 27,920 frames, with 20 segments used for training and 7 for testing. Key frames are selected every 15 frames for annotation and evaluation.
 
 For individual actions recognition evaluation, we utilize common metrics in multi-label classification tasks: precision, recall, and F_1 score, to measure the accuracy of activity classification for each instance in the test dataset. Precision, recall, and F_1 score are denoted as P_i, R_i, and F_i respectively. Social group activity recognition involves both group detection and activity category recognition. Group detection uses the classical Half metric <cit.> (IoU > 0.5), where a detected group is considered correct if the intersection-over-union between predicted and actual group members exceeds 0.5. If the activity category of a detected group is correctly predicted, it is regarded as a correct social group activity prediction. Precision, recall, and F_1 score for social group activity recognition are denoted as P_p, R_p, and F_p respectively. For global activity recognition, we also evaluate using precision, recall, and F_1 score, denoted as P_g, R_g, and F_g. The overall evaluation metric for panoramic activity detection is the average of these three F_1 scores, denoted as the comprehensive F_1 score F_a=(F_i+F_p+F_g)/3. This approach allows for a comprehensive assessment of individual actions detection, group activity recognition, and global activity recognition.



 §.§ Implementation Details


 

  §.§.§ Model Details

    We use a ResNet-18 <cit.> pretrained on the ImageNet dataset as the network backbone. In STRE, both the spatial and temporal encoders share the same structure, consisting of two layers with eight heads each. In PSICGA, the parameter-independent aggregation encoder has four layers with 12 heads, while the parameter-sharing aggregation encoder has two layers with 12 heads. For individual-to-social group aggregation, the weight parameter in Equation (<ref>) is λ = 0.8. For individual-to-global aggregation, λ = 1. The number of visual tokens for scene representation generation is set to 16.

 

  §.§.§ Training
    
We implemented our network using the PyTorch framework and trained it on two GTX 3090 GPUs. Consistent with JRDB-PAR <cit.>, we use ground truth social group detection labels during training and spectral clustering <cit.> for social group detection during evaluation. We set the batch size to 2 and sample three frames. We use the Adam optimizer with an initial learning rate of 7 × 10^-6 and a weight decay of 10^-2, training for 30 epochs.



 §.§ Comparison with State-of-the-Art Methods
 


To demonstrate the superiority of our method, we compared it with state-of-the-art methods. Given the scarcity of existing panoramic activity recognition methods, we adapted the leading group activity recognition methods ARG <cit.>, SA-GAT <cit.>, JRDB-Base <cit.>, DFWSGAR <cit.>, and DECOMPL <cit.> for comparison. Since DFWSGAR <cit.> and DECOMPL <cit.> do not employ individual action recognition as an auxiliary task, their results for individual action recognition and social group activity recognition cannot be compared. Additionally, we applied the state-of-the-art individual activity recognition methods such as UniFormer <cit.> and Vid-TLDR <cit.> to the JRDB-PAR dataset for comparison.

As shown in Table <ref>, our method outperforms the state-of-the-art panoramic activity recognition methods in all three sub-tasks and overall scores. Furthermore, compared to methods that focus on single-granularity tasks, our method exhibits superior performance across all levels of tasks, highlighting the mutual reinforcement among multiple tasks in panoramic activity recognition.







 §.§ Ablation Studies



  §.§.§ Effectiveness of Individual Modules

We conducted experiments with different combinations of modules to verify their effectiveness. The experiment uses temporal average pooling on the collected individual features instead of STRE to evaluate the effectiveness of the module. Similarly for PSICGA, we replaced it with max pooling for cross-granularity aggregation. Additionally, we removed the scene representation learning (SRL) module for comparison. As shown in Table <ref>, when all the modules were removed, our baseline method still achieved a score of 39.0%. This result surpasses some methods that only use single-frame information, indicating that even simple temporal average pooling can significantly enhance model performance. This highlights the importance of temporal information in panoramic activity recognition tasks. When the STRE was added, the model's performance further improved significantly, reaching a score of 44.3%. Adding the PSICGA module and the SRL module individually also resulted in substantial improvements, demonstrating the effectiveness of our proposed modules. Subsequently, we combined the modules in pairs, and the synergy between the modules further improves the performance compared to using each module individually.





  §.§.§ STRE Module Structure


 As shown in Fig. <ref>, we evaluated different structures of STRE, including serial structure (a), parallel structure (b), parallel-then-serial structure (c), and using one (d) and two (e) cross-attention mechanisms to fuse the temporal encoder and spatial encoder. Table <ref> shows that compared to the serial structure, the parallel structure's performance decreased significantly. The decline in performance may be due to the interference between the individual features caused by simultaneous temporal and spatial encoding, reducing overall performance.  Additionally, incorporating additional cross-attention to fuse parallel computation results was less effective than the serial structure. Our experimental results show that the serial structure used in STRE, despite its simplicity, better captures the spatio-temporal relationships of individual features, demonstrates its outstanding effectiveness.









  §.§.§ Effect of PSICGA Module

To verify the effectiveness of this module, the network either uses parameter-sharing aggregation encoder or parameter-independent aggregation encoder individually for cross-granularity aggregation separately. As shown in Table <ref>, using the parameter-sharing aggregation encoder alone outperformed using the parameter-independent aggregation encoder alone. This might be because parameter sharing can better capture latent common behaviors across different levels, enhancing the model's generalization capability. Furthermore, when both parameter-sharing and parameter-independent aggregation encoders were used for cross-granularity aggregation, the network's performance significantly improved. This indicates that our proposed cross-granularity aggregation module can fully utilize the complementary effects of parameter sharing and independence. It can effectively learn task-specific features, alleviate parameter optimization conflicts, and capture latent common behaviors across different levels.









  §.§.§ Investigation of the number of visual scene tokens

We evaluated the impact of the number of visual scene tokens on experimental results in scene representation learning. The global scene information in video frames can be summarized using a small and compact set of visual scene tokens, with each token adaptively focusing on the important parts of the scene. Table <ref> shows that the model’s performance is gradually improved with the increasing  number of tokens from 4 to 16. More tokens help the model capture global scene information more effectively, thereby enhancing recognition accuracy. However, when the number of tokens exceeds 16, the model's performance starts to decline. This decline is likely due to information redundancy, which makes it harder for the model to extract useful information, thus negatively affecting performance.






 §.§ Qualitative Analysis




  §.§.§ Visualization of Results


We visualized the prediction results of the MPT-PAR model and MUP <cit.> at different activity granularities. Besides, we compared our method with the vid-TLDR <cit.> and DECOMPL <cit.> at individual and global activity granularities. Fig. <ref> shows the results for individual action predictions. Since vid-TLDR <cit.> relies only on cropped individual features, it failed to correctly identify the “interaction with door” label. Additionally, MUP <cit.>, which overlooks temporal information, might struggle with occlusions, leading to incorrect identification of the label as “standing”. The comparison indicates that our model, utilizing temporal and scene information, accurately recognizes individual action even in crowded scenes with occlusions. The incorporation of global scene context provides additional scene information, enhancing recognition accuracy. As illustrated in Fig. <ref>, our model perceives scene semantics, such as computers, desks, and chairs, enabling more precise global behavior recognition.
However, the proposed model exhibits certain limitations. The direct application of spectral clustering <cit.> for social group detection diminishes its accuracy. In Fig. <ref>, three individual pedestrians, who are in close proximity and performing the same “walking” action, are considered to belong to the same social group, therefore mistakenly thinking them “walking together”.




  §.§.§ Visualization of Scene Attention Maps

We visualized the spatial attention matrices computed in the scene representation learning module to better observe the distribution of attention within scenes. Fig. <ref> demonstrates that attention predominantly focuses on individuals within the scene, aligning with our expectations as individuals are the primary subjects of panoramic activity recognition. Additionally, significant attention is observed on other scene elements such as stairs, trees, railings, and chairs. These scene elements influence human behavior within the environment. For example, the presence of stairs correlates with actions of ascending or descending. The visualization results indicate that our scene representation module effectively captures contextual information relevant to behaviors, providing additional cues to enhance activity recognition in complex environments.







§ CONCLUSION

This paper introduces a novel network architecture, named MPT-PAR, for panoramic activity recognition. The network leverages complementary advantages of parameter sharing and independent cross-granularity aggregation modules, enhancing the effectiveness of activity recognition at all levels. Furthermore, temporal relationships enhanced by transformers and explicit modeling of scene information further help the model better understand the video content. Experiments on the JRDB-PAR dataset validate the efficacy of the proposed method, significantly improving performance across all granularity tasks. In future work, we will improve the reliability of social group aggregation and further enhance the practicality of the method.






1
IEEEtran


sun2022human
Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang, and J. Liu, “Human action recognition from various data modalities: A review,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 3, pp. 3200–3225, 2022.

yu2022argus++
L. Yu, Y. Qian, W. Liu, and A. G. Hauptmann, “Argus++: Robust real-time activity detection for unconstrained video streams with overlapping cube proposals,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 112–121.

li2023dual
Y. Li, Y. Liu, R. Yu, H. Zong, and W. Xie, “Dual attention based spatial-temporal inference network for volleyball group activity recognition,” Multimedia Tools and Applications, vol. 82, no. 10, pp. 15 515–15 533, 2023.

askari2022interaction
F. Askari, R. Ramaprasad, J. J. Clark, and M. D. Levine, “Interaction classification with key actor detection in multi-person sports videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3580–3588.

han2022panoramic
R. Han, H. Yan, J. Li, S. Wang, W. Feng, and S. Wang, “Panoramic human activity recognition,” in European Conference on Computer Vision.1em plus 0.5em minus 0.4emSpringer, 2022, pp. 244–261.

cao2023mup
M. Cao, R. Yan, X. Shu, J. Zhang, J. Wang, and G.-S. Xie, “Mup: Multi-granularity unified perception for panoramic activity recognition,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 7666–7675.

zouba2009assessing
N. Zouba, F. Bremond, M. Thonnat, A. Anfosso, É. Pascual, P. Mallea, V. Mailland, and O. Guerin, “Assessing computer systems for monitoring elderly people living at home,” in Proceedings of the 19th IAGG World Congress of Gerontology and Geriatrics, Paris, France, 2009, pp. 5–9.

amer2014hirf
M. R. Amer, P. Lei, and S. Todorovic, “Hirf: Hierarchical random field for collective activity recognition in videos,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13.1em plus 0.5em minus 0.4emSpringer, 2014, pp. 572–585.

zhu2013context
Y. Zhu, N. M. Nayak, and A. K. Roy-Chowdhury, “Context-aware modeling and recognition of activities in video,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2013, pp. 2491–2498.

DBLP:journals/corr/abs-2108-11743
H. Yuan, D. Ni, and M. Wang, “Spatio-temporal dynamic inference network for group activity recognition,” CoRR, vol. abs/2108.11743, 2021. [Online]. Available: <https://arxiv.org/abs/2108.11743>


Causality_Graph
Z. Xie, T. Gao, K. Wu, and J. Chang, “An actor-centric causality graph for asynchronous temporal inference in group activity,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 6652–6661.

li2021groupformer
S. Li, Q. Cao, L. Liu, K. Yang, S. Liu, J. Hou, and S. Yi, “Groupformer: Group activity recognition with clustered spatial-temporal transformer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 13 668–13 677.

kim2022detector
D. Kim, J. Lee, M. Cho, and S. Kwak, “Detector-free weakly supervised group activity recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20 083–20 093.

he2016deep
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

he2017mask
K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969.

bertasius2021space
G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?” in ICML, vol. 2, no. 3, 2021, p. 4.

dosovitskiy2020image
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.

wu2020visual
B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda, “Visual transformers: Token-based image representation and processing for computer vision,” arXiv preprint arXiv:2006.03677, 2020.

ehsanpour2022jrdb
M. Ehsanpour, F. Saleh, S. Savarese, I. Reid, and H. Rezatofighi, “Jrdb-act: A large-scale dataset for spatio-temporal action, social group and activity detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20 983–20 992.

wang2020panda
X. Wang, X. Zhang, Y. Zhu, Y. Guo, X. Yuan, L. Xiang, Z. Wang, G. Ding, D. Brady, Q. Dai et al., “Panda: A gigapixel-level human-centric video dataset,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 3268–3278.

zelnik2004self
L. Zelnik-Manor and P. Perona, “Self-tuning spectral clustering,” Advances in neural information processing systems, vol. 17, 2004.

wu2019learning
J. Wu, L. Wang, L. Wang, J. Guo, and G. Wu, “Learning actor relation graphs for group activity recognition,” in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2019, pp. 9964–9974.

ehsanpour2020joint
M. Ehsanpour, A. Abedin, F. Saleh, J. Shi, I. Reid, and H. Rezatofighi, “Joint learning of social groups, individuals action and sub-group activities in videos,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16.1em plus 0.5em minus 0.4emSpringer, 2020, pp. 177–195.

demirel2023decompl
B. Demirel and H. Ozkan, “Decompl: Decompositional learning with attention pooling for group activity recognition from a single volleyball image,” arXiv preprint arXiv:2303.06439, 2023.

li2022uniformer
K. Li, Y. Wang, P. Gao, G. Song, Y. Liu, H. Li, and Y. Qiao, “Uniformer: Unified transformer for efficient spatiotemporal representation learning,” arXiv preprint arXiv:2201.04676, 2022.

choi2024vid
J. Choi, S. Lee, J. Chu, M. Choi, and H. J. Kim, “vid-tldr: Training free token merging for light-weight video transformer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 771–18 781.

xing2023svformer
Z. Xing, Q. Dai, H. Hu, J. Chen, Z. Wu, and Y.-G. Jiang, “Svformer: Semi-supervised video transformer for action recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 18 816–18 826.

wu2023can
W. Wu, Y. Song, Z. Sun, J. Wang, C. Xu, and W. Ouyang, “What can simple arithmetic operations do for temporal modeling?” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 13 712–13 722.

pei2023key
D. Pei, D. Huang, L. Kong, and Y. Wang, “Key role guided transformer for group activity recognition,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 12, pp. 7803–7818, 2023.

zhou2022composer
H. Zhou, A. Kadav, A. Shamsian, S. Geng, F. Lai, L. Zhao, T. Liu, M. Kapadia, and H. P. Graf, “Composer: compositional reasoning of group activity in videos with keypoint-only modality,” in European Conference on Computer Vision.1em plus 0.5em minus 0.4emSpringer, 2022, pp. 249–266.

chappa2023spartan
N. V. Chappa, P. Nguyen, A. H. Nelson, H.-S. Seo, X. Li, P. D. Dobbs, and K. Luu, “Spartan: Self-supervised spatiotemporal transformers approach to group activity recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5158–5168.

haque2017towards
A. Haque, M. Guo, A. Alahi, S. Yeung, Z. Luo, A. Rege, J. Jopling, L. Downing, W. Beninati, A. Singh et al., “Towards vision-based smart hospitals: a system for tracking and monitoring hand hygiene compliance,” in Machine Learning for Healthcare Conference.1em plus 0.5em minus 0.4emPMLR, 2017, pp. 75–87.

bates2021potential
D. W. Bates, D. Levine, A. Syrowatka, M. Kuznetsova, K. J. T. Craig, A. Rui, G. P. Jackson, and K. Rhee, “The potential of artificial intelligence to improve patient safety: a scoping review,” NPJ digital medicine, vol. 4, no. 1, p. 54, 2021.

simonyan2014two
K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” Advances in neural information processing systems, vol. 27, 2014.

tran2018closer
D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018, pp. 6450–6459.

kalfaoglu2020late
M. E. Kalfaoglu, S. Kalkan, and A. A. Alatan, “Late temporal modeling in 3d cnn architectures with bert for action recognition,” in Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16.1em plus 0.5em minus 0.4emSpringer, 2020, pp. 731–747.

liu2016spatio
J. Liu, A. Shahroudy, D. Xu, and G. Wang, “Spatio-temporal lstm with trust gates for 3d human action recognition,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14.1em plus 0.5em minus 0.4emSpringer, 2016, pp. 816–833.

majd2020correlational
M. Majd and R. Safabakhsh, “Correlational convolutional lstm for human action recognition,” Neurocomputing, vol. 396, pp. 224–229, 2020.

Baradel_2017_ICCV
F. Baradel, C. Wolf, and J. Mille, “Human action recognition: Pose-based attention draws focus to hands,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, Oct 2017.

arnab2021vivit
A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid, “Vivit: A video vision transformer,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 6836–6846.

fan2021multiscale
H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer, “Multiscale vision transformers,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 6824–6835.

bruce2022mmnet
X. Bruce, Y. Liu, X. Zhang, S.-h. Zhong, and K. C. Chan, “Mmnet: A model-based multimodal network for human action recognition in rgb-d videos,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3522–3538, 2022.

7214317
F. Solera, S. Calderara, and R. Cucchiara, “Socially constrained structural learning for groups detection in crowd,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 5, pp. 995–1008, 2016.

kim2023practicalgroupactivitydetection
D. Kim, Y. Song, M. Cho, and S. Kwak, “Towards more practical group activity detection: A new benchmark and model,” 2023. [Online]. Available: <https://arxiv.org/abs/2312.02878>


7514916
Y. Xian, X. Rong, X. Yang, and Y. Tian, “Evaluation of low-level features for real-world surveillance event detection,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 27, no. 3, pp. 624–634, 2017.

10159436
R. Li and B. Bhanu, “Energy-motion features aggregation network for players’ fine-grained action analysis in soccer videos,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 2, pp. 955–972, 2024.

ji2020arbitrary
Y. Ji, Y. Yang, F. Shen, H. T. Shen, and W.-S. Zheng, “Arbitrary-view human action recognition: A varying-view rgb-d action dataset,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 1, pp. 289–300, 2020.

chen2023agpn
Y. Chen, H. Ge, Y. Liu, X. Cai, and L. Sun, “Agpn: Action granularity pyramid network for video action recognition,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 8, pp. 3912–3923, 2023.

10262342
Y. Mou, X. Jiang, K. Xu, T. Sun, and Z. Wang, “Compressed video action recognition with dual-stream and dual-modal transformer,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 5, pp. 3299–3312, 2024.








 Paper Title: Empowering Snapshot Compressive Imaging: Spatial-Spectral State Space Model with Across-Scanning and Local Enhancement.


Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision
    Chenxiang Jin, Jiajun Zhou^∗, Chenxuan Xie, Shanqing Yu, Qi Xuan, Senior Member, IEEE, Xiaoniu Yang


This work was supported in part by the Key R&D Program of Zhejiang under Grants 2022C01018 and 2024C01025, by the National Natural Science Foundation of China under Grants 62103374 and U21B2001. (Corresponding author: Jiajun Zhou.)
C. Jin, C. Xie, S. Yu and Q. Xuan are with the Institute of Cyberspace Security, College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China, with the Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou 310056, China. E-mail:{jincxiang, 221122030330, yushanqing, xuanqi}@zjut.edu.cn.
J. Zhou are with the Institute of Cyberspace Security, College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310023, China, with the Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou 310056, China. E-mail: jjzhou@zjut.edu.cn.


X. Yang is with the Institute of Cyberspace Security, Zhejiang University of Technology, Hangzhou 310023, China, and also with Science and Technology on Communication Information Security Control Laboratory, the 36th Research Institute of China Electronics Technology Group Corporation, Jiaxing 314033, China. Email: yxn2117@126.com.










    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
Snapshot Compressive Imaging (SCI) relies on decoding algorithms such as CNN or Transformer to reconstruct the hyperspectral image (HSI) from its compressed measurement. Although existing CNN and Transformer-based methods have proven effective, CNNs are limited by their inadequate modeling of long-range dependencies, while Transformer ones face high computational costs due to quadratic complexity.
Recent Mamba models have demonstrated superior performance over CNN and Transformer-based architectures in some visual tasks, but these models have not fully utilized the local similarities in both spatial and spectral dimensions.
Moreover, the long-sequence modeling capability of SSM may offer an advantage in processing the numerous spectral bands for HSI reconstruction, which has not yet been explored.
In this paper, we introduce a State Space Model with Across-Scanning and Local Enhancement, named ASLE-SSM, that employs a Spatial-Spectral SSM for global-local balanced context encoding and cross-channel interaction promoting. Specifically, we introduce local scanning in the spatial dimension to balance the global and local receptive fields, and then propose our across-scanning method based on spatial-spectral local cubes to leverage local similarities between adjacent spectral bands and pixels to guide the reconstruction process. These two scanning mechanisms extract the HSI's local features while balancing the global perspective without any additional costs. 
Experimental results illustrate ASLE-SSM's superiority over existing state-of-the-art methods, with an inference speed 2.4× faster than Transformer-based MST and saving 0.12 (M) of parameters, achieving the lowest computational cost and parameter count.







§ INTRODUCTION






r0.37
	

	
 
		
    < g r a p h i c s >

	

	

	PSNR-FLOPs comparisons of ASLE-SSM and previous End-to-End Transformer-based SOTA methods.
	

	
 


Hyperspectral images (HSIs) contain a large number of spectral bands to store rich detail information and thus are widely used in various fields such as object tracking <cit.>, remote sensing <cit.>,  super-resolution <cit.>, and medical image processing <cit.>.
Traditional imaging systems used for collecting HSIs involve scanning along the spatial or spectral dimensions but this imaging process is time-consuming and can not effectively capture dynamic objects. 
Recently, snapshot compressive imaging (SCI) systems have been developed <cit.> as an alternative method for obtaining HSIs at video rate. 
SCI systems capture multiple image bands in a single snapshot, compressing the information of HSIs into a 2D measurement. <cit.>. 
Among these systems, Coded Aperture Snapshot Spectral Imaging (CASSI) has emerged and become mainstream <cit.>. 
CASSI utilizes a physical mask to modulate signals from HSIs, shifting these modulated images to different spatial locations on the detector plane via a disperser <cit.>. 
However, the performance of the CASSI system heavily depends on the efficiency of the reconstruction algorithm.


In recent years, numerous reconstruction algorithms have been proposed, including model-based methods and deep learning-based methods.
Traditional model-based methods typically attempt to model the inverse process of image reconstruction, using manually designed priors to derive the solution to an optimization problem.
Although these methods are highly interpretable, they often require manual parameter tuning and tend to be slow. 
With the success of deep learning methods in various visual tasks, CNN and Transformer-based methods have been developed and applied to HSI reconstruction, achieving higher image reconstruction quality and speed than traditional algorithms. 
The CNN-based methods <cit.> have an advantage in extracting local features from images, but they lack the ability to model long-range dependencies effectively. 
Transformer-based methods <cit.> realize a global receptive field, resulting in superior performance compared to CNNs but higher computational costs. 
Although some mechanisms can mitigate this issue, they do not fundamentally change the quadratic complexity problem caused by the attention mechanism in the Transformer architecture. 



The limitation of Transformers in spectral SCI remains unresolved. However, in the realm of large language model and RGB image vision, an advanced solution for this problem exists, known as Mamba <cit.>.
Through data-dependent selective SSMs with parallel scanning and hardware-wise strategies, Mamba establishes a global receptive field and achieves near-linear complexity.
Mamba-based methods have been shown to outperform previous methods in various tasks due to their excellent long sequence modeling capabilities and lower computational costs <cit.>. 

Subsequently, many works have begun to explore the application and performance of the Mamba model in various visual tasks. Representative works, such as the visual state space model <cit.> and vision Mamba <cit.>, have demonstrated Mamba's superiority for image processing, particularly in obtaining fewer parameters and faster speed. However, most Mamba-based models only conduct the scanning process along the spatial dimension, lacking cross-channel interaction, despite the importance of spectral information for HSI reconstruction. 
Recently, some Mamba-based methods have made progress in developing an SSM mechanism along the channel (spectral) dimension for certain visual tasks <cit.>, achieving better results compared to scanning only along the spatial dimension. However, this scanning method does not consider the inherently local features of HSIs in both spatial and spectral dimensions.


In this paper, we introduce ASLE-SSM, a spatial-spectral SSM with across-scanning and local enhancement, to improve HSI reconstruction in SCI. 
Our ASLE-SSM aims to strike a balance between global and local information utilization in HSIs. 
ASLE-SSM comprises two main components: spatial SSM and spectral-spatial cross SSM. 
The spatial SSM employs a combination of global and local scanning methods, enhancing row-by-row scanning with both global and local modeling. 
Meanwhile, the spectral-spatial cross SSM incorporates an innovative across-scanning mechanism, leveraging spatial-spectral local cubes to enhance the module's ability to capture local details.
Overall, our contributions are summarized as follows:


    
  * We present a novel spectral-spatial state space model for HSI reconstruction, which can simultaneously integrate information from both spectral and spatial dimensions to adequately model spatial context and spectral features. 
   
    
  * We propose an across-scanning method that integrates local features from both spatial and spectral dimensions within a single scan. Additionally, we introduce a local scanning approach within the spatial SSM to further enhance local feature extraction.
   
    
  * Our network outperforms state-of-the-art (SOTA) methods on simulation and real datasets, delivering visually superior results. Importantly, it boasts the minimum parameter count, lowest computational cost, and fastest inference speed among current SOTA methods.
    








 
 


§ PRELIMINARIES AND RELATED WORKS

 
 


 §.§ CASSI System

 
 
CASSI consists of a hardware encoder and a software decoder. The encoder compresses a 3D data cube into a 2D snapshot measurement and the decoder reconstructs HSIs from the measurement. 
An imaging process CASSI system is illustrated in Fig. <ref>. The 3D HSI data cube is denoted as 𝐅∈ℝ^H × W × N_λ where H, W, and N_λ represent the height, width, and number of wavelengths of the 3D HSI, respectively. 
A physical mask 𝐌∈ℝ^H × W is used to modulate the 3D HSI 𝐅 as:
    
    𝐅^'(:,:,n_λ) = 𝐅^'(:,:,n_λ) ⊙𝐌,

where ⊙ represents the element-wise product and 𝐅^'∈ℝ^H × W × N_λ is the modulated 3D HSIs. 
After the modulated cube passes the disperser, 𝐅^' is tilted and considered to be sheared along the y-axis.
We use 𝐅^”∈ℝ^H × (W+d(N_λ-1)) × N_λ to denote the tilted cube and assume n_λ to be the reference wavelength. That is, image 𝐅^'(:,:,n_λ_c) is not sheared along the y-axis. Then we have: 
    
    𝐅^”(u,v,n_λ) = 𝐅^' (x,y + d(λ_n - λ_c),n_λ),

where (u, v) indicates the coordinate system on the detector plane, and λ_n is the wavelength of channel n_λ. Here,
d(λ_n - λ_c) signifies the spatial shifting for channel n_λ. Finally, the 2D compressed measurement 𝐘∈ℝ^H × [W+d(N_λ-1)] can be obtained by
    
    𝐘 = ∑_n_λ=1^N_λ𝐅^”(:,:,n_λ) + 𝐍,

where 𝐍∈ℝ^H × [W + d(N_λ-1)] represents the noise of CASSI caused by the sensing detector.

 
 


 §.§ State Space Models

 
 
    Recently, researchers have made significant progress in state space models (SSMs), leading to the emergence of various methods and architectures based on SSMs.
    Among them, the structured state space model (S4) <cit.> becomes a pioneer work for its advantage in modeling long-range dependencies. 
    S4 maps a 1D sequence x(t) ∈ℝ→ y(t) ∈ℝ by an implicit latent state h(t)∈ℝ^N and decodes it into an output sequence, which can be formulated as:
        
    h^'(t)=𝐀 h(t)+𝐁 x(t), 
        y(t)=𝐂 h(t)+𝐃 x(t),

    where N is the state size, A∈ℝ^N× N, B∈ℝ^N × 1, C∈ℝ^1× N, and D∈ℝ.
    After that, a discretization process is adopted to integrate Eq. (<ref>) into the deep learning algorithm, using the  zero-order hold (ZOH) rule, which is defined as follows:
        
    A   =  exp(ΔA),
    B   =(ΔA)^-1( exp(A)-I)·ΔB,

    where A and 𝐁 represent the discretized forms of the parameters 𝐀 and 𝐁, respectively. Δ is denoted as the timescale parameter to transform the continuous parameters 𝐀 and 𝐁 into discrete ones. 
    
     However, the constant dynamics in S4 prevent it from correctly selecting the context information. By introducing a selective mechanism, selective structured state space models (S6), also called Mamba <cit.> have been proposed to realize the data-dependent dynamics, thereby effectively filtering the context information and achieving a better performance in language and visual tasks.

 
 


 §.§ Mamba in Vision

 
 
    Motivated by Mamba's excellent performance in language tasks, recent research has begun exploring its application in vision tasks and has achieved promising results. 
    To address the unidirectional modeling and lack of positional awareness in the original Mamba model, 
    Vision Mamba (Vim) <cit.> incorporates the bidirectional SSMs for data-dependent global visual context modeling and position embeddings for location-aware visual recognition. 
    VMamba <cit.> introduces the Cross-Scan Module (CSM) to bridge the gap between 1-D array scanning and 2-D plain traversing. 
    However, Vim and VMamba have not explored the role of SSM in the channel dimension.
    
    More recently, some works have begun to apply the SSM mechanism to the channel dimension.
    CU-Mamba <cit.> employs a Channel SSM component to preserve channel correlation features. 
    S^2Mamba <cit.> designs an extra scanning module on the spectral dimension for retrieving semantic properties in HSI data by a bi-directional interaction between each band.
    However, these methods directly conduct the scanning process along the spectral dimension and do not utilize the local similarity of HSIs in both spatial and spectral dimensions.

 
 


§ THE PROPOSED ASLE-SSM METHOD

 
 


 §.§ Architecture of ASLE-SSM

 
 
The overall architecture of our ASLE-SSM is depicted in Fig. <ref>. 
ASLE-SSM adopts a U-shaped network with the spatial-spectral SSM block as the basic unit. 
The structure consists of the encoder, bottleneck, and decoder. Given the vectorized measurement 𝐲∈ℝ^H(W + d(N_λ - 1)), we reverse the dispersion and shift back 𝐘 to initialize the input signal 𝐗∈ℝ^H × W × N_λ:
    
    𝐗(x,y,n_λ) = 𝐘 (x,y - d(λ_n - λ_c)).

We first concatenate the input 𝐗_0 with the physical mask 𝐘∈ℝ^H × W × N_λ and then employ a 1×1 convolution layer to integrate information from the original signal and the physical mask. Moreover, the integrated input 𝐗_0 undergoes a 3×3 convolution embedding into a feature 𝐗_0^'∈ℝ^H × W × C. Subsequently, 𝐗_0^' passes through the downsample and upsample process. After passing through the decoder, we employ a 3×3 convolution mapping to obtain the residual information. By computing the sum of the integrated input and residual output, we finally obtain the reconstructed HSIs. In our implementation, we change the number of spatial-spectral SSM blocks in the encoder, bottleneck, and decoder to realize the small, middle, and large sizes of ASLE-SSM models. 



 
 


 §.§ Local Enhancement SSM (LE-SSM) Mechanism

 
 
    Although Mamba is effective in modeling long-range dependencies, directly using the SSM mechanism often leads to less effective in extracting local features. 
    Recent models tend to incorporate CNNs into the network structure, utilizing their local feature extraction capabilities to improve the performance of original SSMs. 
    However, such methods would increase the parameter count, diminishing the lightweight advantage of the original SSM mechanism. 
    While directly adding convolutional layers can enhance the model's performance, it does not fundamentally solve the problem.

    Inspired by LocalMamba <cit.>, we introduce a simplified local mechanism in the spectral dimension. 
    We use two scanning directions as the global branch, scanning the image row by row in both forward and backward directions while using the other two scanning directions as local branches to unfold and scan local patches. 
    The specific details of the scanning process are illustrated in Fig. <ref>(d). 
    Given the input 𝐗∈ℝ^H × W × C, we first generate four copies of input data with different arrangements.
    For the global branch, we directly flatten the feature map into 𝐗_1∈ℝ^L × C, where L =  H × W, and then reverse it to generate 𝐗_2. For the local branch, we first set the size of our patch flattening each patch into a 1D sequence, and then linking each patch together to generate 𝐗_3. 
    Similarly, we reverse 𝐗_3 to obtain 𝐗_4. 
    After the generation of input data in four directions, we conduct the process as follows:
        
    𝐗^' = Concat(𝐗_1,𝐗_2,𝐗_3,𝐗_4),
       𝐗̂ = LE-Spatial-SSM(𝐗^').

    The structure of LE-Spatial-SSM is shown in Fig. <ref>(b). 
    It is worth noting that our method does not employ any CNNs or spatial and channel attention mechanisms based on MLPs, ensuring that our method does not add any extra parameters or time consumption compared to the global cross-scan mechanism <cit.>.
    However, it also means we need to carefully choose the size of local patches, or else the reconstruction result will even become worse. 
    In our implementation, we use patches of size 4×4 to achieve a relatively better performance.

 
   


 §.§ Across-Scanning SSM (AS-SSM) Mechanism

 
 
Our LE-SSM mechanism establishes a local-enhanced global receptive field but does not fully exploit the rich spectral information inherent in HSIs, leading to suboptimal reconstruction.
Recently, some methods <cit.> have utilized and integrated channel information by directly scanning along the channel dimension (spectral dimension), but this scanning method does not consider the local features in both channel and spatial dimensions. 
In other words, neighboring pixels and bands often have certain similarities, but scanning the spectral dimension directly may lead to a distance increase between data points that are highly correlated. 

To address the above issues, we propose a local scanning method that integrates local features from both spatial and spectral dimensions. This mechanism attempts to exploit the local properties of HSIs in both spatial and spectral dimensions while leveraging the rich spectral information to guide the reconstruction process. 
First, we rearrange the input 𝐗∈ℝ^H × W × C into patches along the spatial context by dividing the input HSIs into a series of smaller data cubes 𝐒∈ℝ^P × P × C, which will be flattened into sequences for scanning. 
After that, we organize the scanning order within each sequence based on our spatial-spectral local cubes 𝐂∈ℝ^h × w × c.
As shown in Fig. <ref>(e), different colors indicate each spatial-spectral local cube. The scanning order of each data point is also depicted. The process can be formulated as:
    
    𝐂 = Divide(𝐗),
       𝐂^' = AS-Spectral-Spatial-SSM(𝐂),

where 𝐂^' denotes the output patch of AS-SSM. After the SSM mechanism, we reorganize the patches back into the same order as the input 𝐗. 
By scanning the data points within each data cube, we can leverage the local similarity between adjacent spectral bands and pixels, thus achieving better results than previous methods that directly scan along the spectral dimension.



 
 


§ EXPERIMENT





 §.§ Experimental Settings



The effectiveness of our ASLE-SSM network is verified on both simulation and real datasets, by using 28 wavelengths ranging from 450nm to 650nm, acquired via spectral interpolation manipulation.

Simulation HSI Data.
Two simulated HSI datasets CAVE <cit.> and KAIST <cit.> are utilized to conduct the simulation experiment. 
The CAVE dataset contains 32 HSIs with a spatial size of 512×512, while the KAIST dataset includes 30 HSIs with a spatial size of 2704× 3376. 
Following the experimental settings of TSA-Net <cit.>, we designate CAVE as the training set and select 10 scenes from the KAIST as the testing set.

Real HSI Data. 
We adopt the real HSIs obtained by the CASSI system, as developed in TSA-Net <cit.> for testing.

Evaluation Metrics. 
We select peak signal-to-noise ratio (PSNR) <cit.> and structural similarity (SSIM) <cit.> as the evaluation metrics to assess the quality of HSI reconstruction.

Implementation Details. 
Our ASLE-SSM is implemented using PyTorch and we conduct model training with Adam optimizer (β_1 = 0.9 and β_2 = 0.999) for 300 epochs.
The learning rate is set to 4×10^-4 at the beginning and is halved every 50 epochs during the training procedure. 
Image patches with size 256×256 are randomly cropped from the 3D HSI cubes with 28 channels for training in the simulation and real experiments. 
The shifting step d in the dispersion process is set to 2 and the batch size is 2. 
We apply the data augmentation techniques including random flipping and rotation. The models are trained on an RTX 4090 GPU.






 §.§ Simulation Results




r0.58


Comparisons of different frameworks on PSNR, SSIM, FLOPs, Params, and Inference Time on the test dataset.



 1.1

	0.55!
	
	
				color3Framework     MST-S <cit.>      CST-S <cit.>      ASLE-SSM-S    

				PSNR    34.14      34.53      34.74  

				SSIM     0.938       0.945      0.946  

				Params (M)     0.93       1.20     0.81   

				FLOPS (G)    12.96       11.67    7.01    

                    Inference Time(s)    0.022       0.041    0.009    

				
	
 



We conduct a comprehensive comparative analysis of our ASLE-SSM method and SOTA HSI reconstruction algorithms in Params, FLOPs, PSNR, and SSIM. 
We compared the results of ASLE-SSM with 12 SOTA methods, including two model-based methods (GAP-TV <cit.>, and DeSCI <cit.>), two CNN-based methods (λ-net <cit.>, TSA-Net <cit.>), three Transformer-based methods (HDNet <cit.>, MST-S, MST-M, MST-L <cit.>, and CST-S, CST-M, CST-L, CST-L+ <cit.>), two Plug-and-Play (PnP) methods (DIP-HSI <cit.>, DiffSCI <cit.>) and three deep unfolding methods (ADMM-Net <cit.>, GAP-Net <cit.>, DGSMP <cit.>) across 10 simulation scenes. The quantitative results of different models are listed in Table <ref>.

As shown in Table <ref>, our ASLE-SSM-L obtains 35.74dB of PSNR and 0.958 of SSIM, achieving the best reconstruction performance among SOTAs.
When compared to CNN-based and Transformer-based methods such as TSA-net <cit.>, HDNet <cit.>, MST-L <cit.>, and CST-L+ <cit.>, our ASLE-SSM-L demonstrates notable improvements over these methods, which are 4.39dB, 1.08dB, 0.67dB, and 0.03dB on average, respectively. The spectral density curves in Fig. <ref> also demonstrate our superiority.

We also conduct comparisons of our ASLE-SSM-S with previous Transformer-based models MST <cit.> and CST <cit.> in the small size.
As shown in Table <ref>, we not only achieve better performance in PSNR and SSIM compared to MST and CST but also obtain a more lightweight model with faster speed.
Specifically, the ASLE-SSM-S is 2.4 times faster than MST-S and 4.5 times faster than CST-S, while having 0.12 (M) fewer parameters than the MST-S, which means the smallest parameter count. From Fig. <ref>, we can also observe that our ASLE-SSM not only achieves better results, but the value of FLOPs is also significantly lower than that of the corresponding MST and CST models.


Except for quantitative comparisons, we also visualize the results of simulation experiments. 
We choose one of 28 channels from Scene 1 and Scene 2, against the simulation results obtained from twelve SOTA approaches. 
Visually, as illustrated in Fig. <ref>, our method yields smoother and cleaner details. 
The experimental results indicate the capability of our method to generate high-quality HSIs. 
Specifically, in Scene 1, our reconstruction results better generate the curvature of the chin compared to other models. 
In Scene 2, our model clearly reconstructs the shapes of the blocks and ropes. 
The heart-like pattern on the blocks is also well-preserved.





 §.§ Real Data Results



r0.6
    

    

        
    < g r a p h i c s >

         
        
    < g r a p h i c s >

    

    

    Spectral Density Curves.
    

    




We further apply our method to reconstruct real HSIs. We retrain our model on the CAVE <cit.> and KAIST <cit.> datasets, conducting tests on real scenes for experiments. Following the configurations established in MST <cit.>, We introduce 11-bit shot noise during training to simulate real measurement conditions. As shown in Fig. <ref>, we compare the reconstructed images from two real scenes (Scene 3 and Scene 4) using our ASLE-SSM and nine SOTA methods. Our model achieved superior results when compared to other models. Specifically, In Scene 3, our image preserves the most information compared to the results of other models. A noticeable protrusion consistent with the original image can be observed in the center of our result and other methods fail to reproduce. 

Additionally, our images do not exhibit significant artifacts or ghosting. In Scene 4, We can observe that the face shape in the original image is relatively rounded. However, the reconstruction results of previous methods generate an almond-shaped face similar to the compressed measurements, indicating that these methods did not learn this information. In contrast, our results clearly show a fuller face on both sides. Additionally, previous reconstruction methods often only reconstruct one side of the mouth, while our image noticeably reconstructs the whole mouth, though there is a slight misalignment between the two sides. Furthermore, our images do not exhibit nonexistent features that appear in previous methods' results, such as noses, wrinkles, or gaps that are not present in the original RGB image. These results demonstrate that our ASLE-SSM surpasses other methods in reconstructing image details and suppressing noise in real scenes.













 §.§ Ablation Study



Break-down Ablation. We conduct an ablation study by removing the LE-Spatial Mechanism and AS-Spectral-Spatial Mechanism from the ASLE-SSM network. The experiment results are shown in Table <ref>. Our goal is to explore the contributions of each mechanism to the results. The baseline barely adopts the U-shaped framework without any SSM modules. 
r0.45

Ablation study of ASLE-SSM's components



 1.1

	0.43!
	
	 
                    color3 Architecture     Params (M)     PSNR     SSIM  

                    Baseline w/o SSM 
                        0.18             27.41            0.678  

                    with LE-SSM                                     0.62     34.48     0.942   

                with AS-SSM                                        0.37            31.16          0.863    

                    with LE + AS SSM                            0.81             34.74         0.946    

                    
 


When we respectively employ LE-SSM and AS-SSM blocks, our model achieves 7.07 dB and 3.75 dB improvements in PSNR over the baseline, respectively. Employing both mechanisms simultaneously can further enhance the model's performance. Here we need to point out that, in consideration of maintaining lightweight, we only utilized a single-direction scan in AS-SSM. In fact, we could also deploy a four-direction scanning similar to LE-SSM. However, each scan direction implies additional computational costs so we decided not to adopt this way.

r0.45


Comparisons of AS mechanism and common scanning.



 1.1

	0.42!
	
	 
                    color3 Architecture                     PSNR     SSIM  

                    w/o Spectral SSM 
                     34.48            0.942  

                    with AS Mechanism                                         34.74         0.946    

                with Common Scanning                                 34.55         0.943    

                    
 


Across-Scanning Mechanism. To demonstrate the advantages of our across-scanning mechanism compared to common spectral scanning, we conduct the experiment on model performance under both methods. As illustrated in Table <ref>, when we adopt common scanning, it yields a 0.07 dB improvement. In contrast, when we apply the proposed across-scanning mechanism, our method achieves a significant increase of 0.26 dB compared to not using spectral SSM. Both methods were experimented with under the condition of a single scan. The superiority of our method mainly comes from paying more attention to adjacent bands and pixels, thus better extracting the local features of HSIs along both spatial and spectral dimensions. The results verify the role of our AS mechanism.


r0.45


Comparisons of LE mechanism and global scanning.



 1.1

	0.38!
	
	 
                    color3 Architecture                     PSNR     SSIM  

                    w/o Spatial SSM 
                     31.16           0.863  

                    with LE Machenism                                         34.48         0.942    

                with Global Scanning                                 34.40         0.940    

                    
 



Local-Enhancement Mechanism. Additionally, we conduct an ablation study of our LE mechanism on the CAVE and KAIST datasets to assess its contributions. As shown in Table <ref>, The evaluations begin with the model without SSM in the spatial dimension, which served as our initial baseline with a PSNR of 31.16 dB. Specifically, the addition of the global four-direction scanning along the spatial context yields a significant increase of 3.32 dB in PSNR and 0.077 in SSIM, demonstrating its effectiveness in modeling the global spatial context. 

We further apply our LE Machenism along the spatial dimension and achieve a 0.08 dB increase compared to the global scanning. our method only changes the scanning order and does not introduce any additional parameter consumption or computational cost to achieve a relatively better performance. In other words, we do not use CNN layers or spatial attention mechanisms as in other methods <cit.>. In fact, Without increasing any additional cost and with the reconstructed results already achieving a relatively good level, 0.08 dB in PSNR and 0.002 in SSIM are significant improvements. The ablation experiment demonstrates the importance of local features in HSIs and verifies the effectiveness of our LE Mechanism.






§ CONCLUSION



In this paper, we address two limitations of previous SSM-based models by our proposed mechanisms: insufficient utilization of channel (spectral) information, and limited local detail extraction capability. By designing a novel across-scanning mechanism, we fully utilize the rich spectral information in HSIs to guide the image reconstruction process. At the same time, our scanning method based on patched data cubes pays more attention to the local similarity of HSIs between adjacent bands and pixels, achieving better results compared to simply scanning along the channel dimension. By introducing local enhancement in the Spatial SSM, we integrate spatial context from both global and local perspectives, achieving better results than using a global SSM alone.

Despite the significant progress of our method, we also acknowledge that as the first framework to adopt the SSM mechanism in HSI reconstruction, our network still has room for further improvement. For example, we could design a lightweight and effective way to fuse local and global branch information in the spatial dimension, rather than simply adding them together. However, the key challenge lies not in improving the effect itself, but in controlling computational costs. We will explore this aspect in the future with the aim of designing a more efficient network architecture.



10

ot_1
M. H. Kim, T. A. Harvey, D. S. Kittle, H. Rushmeier, R. O. Prum J. Dorsey, and D. J. Brady.
3d imaging spectroscopy for measuring hyperspectral patterns on solid objects.
ACM Transactions on on Graphics, 2012.

ot_2
Zhihong Pan, Glenn Healey, Manish Prasad, and Bruce Tromberg.
Face recognition in hyperspectral images.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12):1552–1560, 2003.

ot_3
Hien Van Nguyen, Amit Banerjee, and Rama Chellappa.
Tracking via object reflectance using a hyperspectral video camera.
In IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pages 44–51. IEEE, 2010.

chen2017denoising
Yongyong Chen, Yanwen Guo, Yongli Wang, Dong Wang, Chong Peng, and Guoping He.
Denoising of hyperspectral images using nonconvex low rank matrix approximation.
IEEE Transactions on Geoscience and Remote Sensing, 55(9):5366–5380, 2017.

he2020non
Wei He, Quanming Yao, Chao Li, Naoto Yokoya, Qibin Zhao, Hongyan Zhang, and Liangpei Zhang.
Non-local meets global: An iterative paradigm for hyperspectral image restoration.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4):2089–2107, 2020.

yuan2018hyperspectral
Qiangqiang Yuan, Qiang Zhang, Jie Li, Huanfeng Shen, and Liangpei Zhang.
Hyperspectral image denoising employing a spatial–spectral deep residual convolutional neural network.
IEEE Transactions on Geoscience and Remote Sensing, 57(2):1205–1218, 2018.

dong2021model
Weisheng Dong, Chen Zhou, Fangfang Wu, Jinjian Wu, Guangming Shi, and Xin Li.
Model-guided deep hyperspectral image super-resolution.
IEEE Transactions on Image Processing, 30:5754–5768, 2021.

sr_2
Hongyuan Wang, Lizhi Wang, Chang Chen, Xue Hu, Fenglong Song, and Hua Huang.
Learning spectral-wise correlation for spectral super-resolution: Where similarity meets particularity.
In Proceedings of the 31st ACM International Conference on Multimedia, pages 7676–7685, 2023.

sr_3
Xian-Hua Han and Yen-Wei Chen.
Deep residual network of spectral and spatial fusion for hyperspectral image super-resolution.
In 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), pages 266–270. IEEE, 2019.

mi_1
V Backman, Michael B Wallace, LT Perelman, JT Arendt, R Gurjar, MG Müller, Q Zhang, G Zonios, E Kline, T McGillican, et al.
Detection of preinvasive cancer cells.
Nature, 406(6791):35–36, 2000.

mi_2
Guolan Lu and Baowei Fei.
Medical hyperspectral imaging: a review.
Journal of Biomedical Optics, 2014.

mi_3
Ziyi Meng, Mu Qiao, Jiawei Ma, Zhenming Yu, Kun Xu, and Xin Yuan.
Snapshot multispectral endomicroscopy.
Optics Letters, 2020.

sci_1
Patrick Llull, Xuejun Liao, Xin Yuan, Jianbo Yang, David Kittle, Lawrence Carin, Guillermo Sapiro, and David J Brady.
Coded aperture compressive temporal imaging.
Optics Express, 2013.

sci_2
Ashwin Wagadarikar, Renu John, Rebecca Willett, and David Brady.
Single disperser design for coded aperture snapshot spectral imaging.
Applied Optics, 2008.

sci_3
Ashwin A Wagadarikar, Nikos P Pitsianis, Xiaobai Sun, and David J Brady.
Video rate spectral imaging using a coded aperture snapshot spectral imager.
Optics Express, 2009.

sci_4
Xin Yuan, Tsung-Han Tsai, Ruoyu Zhu, Patrick Llull, David Brady, and Lawrence Carin.
Compressive hyperspectral imaging with side information.
IEEE Journal of selected topics in Signal Processing, 9(6):964–976, 2015.

sci_5
Xun Cao, Tao Yue, Xing Lin, Stephen Lin, Xin Yuan, Qionghai Dai, Lawrence Carin, and David J Brady.
Computational snapshot multispectral cameras: Toward dynamic capture of the spectral world.
IEEE Signal Processing Magazine, 33(5):95–108, 2016.

sci_6
Hao Du, Xin Tong, Xun Cao, and Stephen Lin.
A prism-based system for multispectral video acquisition.
In 2009 IEEE 12th International Conference on Computer Vision, pages 175–182. IEEE, 2009.

gap_tv
Xin Yuan.
Generalized alternating projection based total variation minimization for compressive sensing.
In 2016 IEEE International conference on image processing (ICIP), pages 2539–2543. IEEE, 2016.

icme_sci_1
Yangke Ying, Jin Wang, Yunhui Shi, and Baocai Yin.
Dual-domain feature learning and memory-enhanced unfolding network for spectral compressive imaging.
In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 1589–1594. IEEE, 2023.

mst
Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool.
Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17502–17511, 2022.

icme_sci_2
Shumian Yang, Xinxin Xiang, Fenghua Tong, Dawei Zhao, and Xin Li.
Image compressed sensing using multi-scale characteristic residual learning.
In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 1595–1600. IEEE, 2023.

lambda
Xin Miao, Xin Yuan, Yunchen Pu, and Vassilis Athitsos.
λ-net: Reconstruct hyperspectral images from a snapshot measurement.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4059–4069, 2019.

tsa_net
Ziyi Meng, Jiawei Ma, and Xin Yuan.
End-to-end low cost compressive spectral imaging with spatial-spectral self-attention.
In Proceedings of the European Conference on Computer Vision, pages 187–204. Springer, 2020.

mst_pp
Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, Radu Timofte, and Luc Van Gool.
Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 745–755, 2022.

dauhst
Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc V Gool.
Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging.
In Proceedings of the Advances in Neural Information Processing Systems, volume 35, pages 37749–37761, 2022.

cst
Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool.
Coarse-to-fine sparse transformer for hyperspectral image reconstruction.
In Proceedings of the European Conference on Computer Vision, pages 686–704. Springer, 2022.

gu2023mamba
Albert Gu and Tri Dao.
Mamba: Linear-time sequence modeling with selective state spaces.
arXiv preprint arXiv:2312.00752, 2023.

mamba_survey
Xiao Wang, Shiao Wang, Yuhe Ding, Yuehang Li, Wentao Wu, Yao Rong, Weizhe Kong, Ju Huang, Shihao Li, Haoxiang Yang, et al.
State space model for new-generation network alternative to transformers: A survey.
arXiv preprint arXiv:2404.09516, 2024.

vmamba2
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang.
Vision mamba: Efficient visual representation learning with bidirectional state space model.
arXiv preprint arXiv:2401.09417, 2024.

mambair
Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia.
Mambair: A simple baseline for image restoration with state-space model.
arXiv preprint arXiv:2402.15648, 2024.

lightm-unet
Weibin Liao, Yinghao Zhu, Xinyuan Wang, Cehngwei Pan, Yasha Wang, and Liantao Ma.
Lightm-unet: Mamba assists in lightweight unet for medical image segmentation.
arXiv preprint arXiv:2403.05246, 2024.

shi2024vmambair
Yuan Shi, Bin Xia, Xiaoyu Jin, Xing Wang, Tianyu Zhao, Xin Xia, Xuefeng Xiao, and Wenming Yang.
Vmambair: Visual state space model for image restoration.
arXiv preprint arXiv:2403.11423, 2024.

vision_mamba
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang.
Vision mamba: Efficient visual representation learning with bidirectional state space model.
arXiv preprint arXiv:2401.09417, 2024.

s2mamba
Guanchun Wang, Xiangrong Zhang, Zelin Peng, Tianyang Zhang, Xiuping Jia, and Licheng Jiao.
S2mamba: A spatial-spectral state space model for hyperspectral image classification.
arXiv preprint arXiv:2404.18213, 2024.

huang2024spectral
Lingbo Huang, Yushi Chen, and Xin He.
Spectral-spatial mamba for hyperspectral image classification.
arXiv preprint arXiv:2404.18401, 2024.

s4gu2021
Albert Gu, Karan Goel, and Christopher Re.
Efficiently modeling long sequences with structured state spaces.
In ICLR, 2021.

mambagu2023mamba
Albert Gu and Tri Dao.
Mamba: Linear-time sequence modeling with selective state spaces.
arXiv preprint arXiv:2312.00752, 2023.

liu2024vmamba
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu.
Vmamba: Visual state space model.
arXiv preprint arXiv:2401.10166, 2024.

cumamba
Gu T. Deng R.
Cu-mamba: Selective state space models with channel learning for image restoration.
arXiv preprint arXiv:2404.11778, 2024.

desci
Yang Liu, Xin Yuan, Jinli Suo, David J Brady, and Qionghai Dai.
Rank minimization for snapshot compressive imaging.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(12):2990–3006, 2018.

hdnet
Xiaowan Hu, Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool.
HDNet: High-resolution dual-domain learning for spectral compressive imaging.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17542–17551, 2022.

self
Ziyi Meng, Zhenming Yu, Kun Xu, and Xin Yuan.
Self-supervised neural networks for spectral snapshot compressive imaging.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2622–2631, 2021.

diffsci
Zhenghao Pan, Haijin Zeng, Jiezhang Cao, Kai Zhang, and Yongyong Chen.
Diffsci: Zero-shot snapshot compressive imaging via iterative spectral diffusion model.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

admm-net
Jiawei Ma, Xiao-Yang Liu, Zheng Shou, and Xin Yuan.
Deep tensor admm-net for snapshot compressive imaging.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10223–10232, 2019.

gsm
Tao Huang, Weisheng Dong, Xin Yuan, Jinjian Wu, and Guangming Shi.
Deep gaussian scale mixture prior for spectral compressive imaging.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16216–16225, 2021.

gapnet
Ziyi Meng, Xin Yuan, and Shirin Jalali.
Deep unfolding for snapshot compressive imaging.
International Journal of Computer Vision, 131(11):2933–2958, 2023.

localmamba
Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu.
Localmamba: Visual state space model with windowed selective scan.
arXiv preprint arXiv:2403.09338, 2024.

cave
Jong-Il Park, Moon-Hyun Lee, Michael D Grossberg, and Shree K Nayar.
Multispectral imaging using multiplexed illumination.
In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.

kaist
Inchang Choi, MH Kim, D Gutierrez, DS Jeon, and G Nam.
High-quality hyperspectral reconstruction using a spectral prior.
In Technical report, 2017.

PSNR
Luen C. Chan and Peter Whiteman.
Hardware-constrained hybrid coding of video imagery.
IEEE Transactions on Aerospace and Electronic Systems, AES-19(1):71–84, 1983.

ssim
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.
Image quality assessment: from error visibility to structural similarity.
IEEE Transactions on Image Processing, 13(4):600–612, 2004.






