Paper Title: Tora: Trajectory-oriented Diffusion Transformer for Video Generation.




oldmaketitlemaketitle

Tree-Cotree-Based Tearing and Interconnecting for 3D Magnetostatics:
 A Dual-Primal Approach
    [
    August 5, 2024
============================================================================================




§ ABSTRACT 
Recent advancements in Diffusion Transformer (DiT) have demonstrated remarkable proficiency in producing high-quality video content. Nonetheless, the potential of transformer-based diffusion models for effectively generating videos with controllable motion remains an area of limited exploration. This paper introduces Tora, the first trajectory-oriented DiT framework that integrates textual, visual, and trajectory conditions concurrently for video generation.
Specifically, Tora consists of a Trajectory Extractor (TE), a Spatial-Temporal DiT, and a Motion-guidance Fuser (MGF). The TE encodes arbitrary trajectories into hierarchical spacetime motion patches with a 3D video compression network. The MGF integrates the motion patches into the DiT blocks to generate consistent videos following trajectories.  Our design aligns seamlessly with DiT's scalability, allowing precise control of video content's dynamics with diverse durations, aspect ratios, and resolutions. Extensive experiments demonstrate Tora's excellence in achieving high motion fidelity, while also meticulously simulating the movement of the physical world. 














§ INTRODUCTION

Diffusion models <cit.> have demonstrated their capability to generate diverse and high-quality images or videos. Previously, video diffusion models <cit.> predominantly employed U-Net architectures <cit.>, focusing primarily on synthesizing videos of limited duration, typically around two seconds, and were constrained to fixed resolutions and aspect ratios. Recently, Sora <cit.>, a text-to-video generation model leveraging Diffusion Transformer (DiT) <cit.>, has showcased video generation capabilities that significantly outstrip current state-of-the-art methods. Sora excels not only in the production of high-quality videos ranging from 10 to 60 seconds, but also distinguishes itself through its capacity to handle diverse resolutions, various aspect ratios, adherence to the laws of actual physics.

Video generation entails the creation of consistent motion within a sequence of generated images, thereby emphasizing the importance of motion control. Prior research endeavors, such as VideoComposer <cit.> and DragNUWA <cit.>, have successfully implemented generalized motion manipulation techniques. These methods typically involve extracting motion vectors and optical flow data to direct the video generation models. Building upon these advancements, MotionCtrl <cit.> innovates by separately managing camera movements and object motions, thereby expanding the achievable motion ranges and patterns. Despite promising controllable motion quality, these methods are limited to generating videos of up to 16 frames in length and at a lower, fixed resolution. Moreover, limitations in the number of frames can hinder the seamless portrayal of motion, particularly in scenarios involving significant positional shifts. This often leads to issues such as motion blur, appearance distortions, and unnatural movements, such as parallel drifting, which deviate from real-world physical dynamics.

To address these challenges, we introduce Tora—a pioneering DiT model that simultaneously integrates text, image, and trajectory data, ensuring meticulous and versatile control over video content. Tora innovatively embeds the provided trajectory into multiple patches, enhancing motion fluidity and more closely adhering to the natural principles of object kinetics, leveraging its remarkable scaling properties. Notably, our work adopts OpenSora <cit.>, an open-source version of Sora, as the foundational DiT framework. To facilitate motion control, we design two innovative modules: the Trajectory Extractor (TE), which transforms arbitrary trajectories into hierarchical spacetime motion patches, and the Motion-guidance Fuser (MGF), meticulously crafted to integrate these motion patches within the stacked architecture of DiT blocks.
More specifically, TE initially converts positional displacements along trajectories between successive frames into the RGB domain via flow visualization techniques. These visualized displacements undergo Gaussian filtering to mitigate scattered issues. Subsequently, a 3D Variational Autoencoder (VAE) <cit.> encodes trajectory visualizations into spacetime motion latents, which share the same latent space with video patches. The motion latents are then decomposed into multiple levels of motion conditions via a series of stacked lightweight modules. Our VAE architecture is inspired by MAGVIT-v2 <cit.> but simplified by foregoing codebook dependencies and utilizing solely VAE reconstruction loss for training. The MGF integrates adaptive normalization layers <cit.> to infuse multi-level motion conditions into the corresponding DiT blocks. We explored various adaptations of transformer blocks including adaptive layer normalization, cross-attention, and extra channel connections to inject the motion conditions. Among these, adaptive layer normalization emerged as the most effective to generate consistent videos following trajectories.

To train Tora, annotated videos with captions and movement trajectories are essential. We adapt OpenSora's workflow to transform raw videos into high-quality video-text pairs and leverage an optical flow estimator <cit.> for trajectory extraction. Significantly, we combine motion segmentation results <cit.> with flow scores to filter out instances that primarily contain camera movement. This strategic approach improves our ability to accurately follow the trajectory of specific objects within videos. As a result, this careful selection process leads to the creation of a dataset containing 630k high-quality video clips with consistent motion. With an adapter-like strategy <cit.>, we solely train the temporal blocks, together with the TE and MGF. This strategy seamlessly integrates DiT's inherent generative knowledge with external motion signals.

The main contributions of our work are as follows:



  * We introduce Tora, the first trajectory-oriented DiT for video generation. As illustrated in Figure <ref>, Tora seamlessly integrates a broad range of visual and trajectory instructions, thereby proficiently enabling the creation of motion-manipulable videos.

  * To align with the scalability of DiT, we design a novel trajectory extractor and a motion-guidance fusion mechanism to obtain spacetime motion patches, subsequently injecting these patches into DiT blocks. We ablate several architecture choices and offer empirical baselines for future research in DiT-based motion control.

  * Experiments demonstrate that Tora is capable of generating 720p resolution videos with varying aspect ratios, extending up to 204 frames, all guided by the specified trajectories. Furthermore, it demonstrates superiority in simulating movements within the physical world.





§ RELATED WORK



 §.§ Diffusion models for Video Generation

Diffusion models have demonstrated an impressive capability to generate high-quality video samples. Previous research <cit.> commonly used video diffusion models (VDMs) that incorporated temporal convolutional and attention layers into the pre-trained image diffusion models. Subsequently, VideoCrafter <cit.> and SVD <cit.> expand the application of video diffusion models to larger datasets, while TF-T2V <cit.> directly learn from extensive text-free videos. Lumiere <cit.> introduces the concept of temporal downsampling within the 3D U-Net framework. Nonetheless, these methods encounter limitations in generating long videos, owing to the inherent constraints on capacity and scalability within the UNet design. Conversely, DiT-based models <cit.> can directly generate videos extending up to tens of seconds. Sora <cit.> converts visual data into a unified representation, facilitating large-scale training and enabling the generation of 1-minute high-definition video. Vidu <cit.> is capable of generating both realistic and imaginative videos in various aspect ratios and resolutions. For our study, we adopt OpenSora <cit.> as the foundational model, which is an open-source alternative to Sora.






 §.§ Motion control in Video Generation

To better control motion in generated video,  a multitude of studies have endeavored to introduce diverse motion signals in VDMs.
Pioneering works like MotionDirector <cit.> and VMC <cit.> have utilized reference videos to extract motion patterns applicable to diverse video generations. VideoComposer <cit.> expands upon this by adopting depth maps, sketches, or motion vectors from references as conditional inputs for motion control. Nonetheless, these methodologies are limited to reproducing existing motion patterns. Conversely, approaches that leverage trajectories or bounding boxes <cit.> in video generation promise greater adaptability and user accessibility.
DragNUWA <cit.> breaks new ground by integrating trajectory-based conditioning into VDMs, facilitating complex camera and object movements. AnimateAnything <cit.> employs motion masks for precise control over the moving regions. TrailBlazer <cit.>, employs explicit attention mechanisms to maneuver generated objects along precise trajectories. MotionCtrl <cit.> facilitates more flexible control, allowing separate adjustment of both camera movements and individual object motions.
However, all of them often yield noticeable artifacts in both motion consistency and visual presentation when applied to longer video sequences. In contrast, our method first integrates trajectories into DiT architecture, specifically designed to accommodate scaling properties. This enables motion fluidity and closer adherence to the physical world.




§ METHODOLOGY



 §.§ Preliminary

Latent Video Diffusion Model (LVDM). The LVDM enhances the stable diffusion model <cit.> by integrating a 3D U-Net, thereby empowering efficient video data processing. This 3D U-Net design augments each spatial convolution with an additional temporal convolution and follows each spatial attention block with a corresponding temporal attention block. It is optimized employing a noise-prediction objective function:

    l_ϵ = ||ϵ - ϵ_θ(z_t, t, c)||^2_2,

Here, ϵ_θ(·) signifies the 3D U-Net's noise prediction function. The condition 
c is guided into the U-Net using cross-attention for adjustment. Meanwhile, z_t denotes the noisy hidden state, evolving like a Markov chain that progressively adds Gaussian noise to the initial latent state z_0:

    z_t=√(α̅_t)z_0 + √(1 - α̅_t)ϵ,   ϵ∼𝒩(0, I),

where α̅_t = ∏_i=1^t(1-β_t) and β_t is a coefficient that controls the noise strength in step t.


Diffusion Transformer (DiT). The DiT <cit.> constitutes a seminal architectural innovation that synergistically combines the advantages of diffusion models with the transformer architectures <cit.>. This integration is purposed to transcend the limitations of traditional U-Net-based LDMs, augmenting their performance, versatility, and scalability. While maintaining the overarching formulation congruent with established LDM frameworks, the paradigm shift resides in the substitution of the U-Net with a transformer architecture for the denoising function ϵ_θ(·) learning, thereby marking a pivotal advance in the realm of generative modeling.



 §.§ Tora

Tora adopts OpenSora as the foundational model for its DiT architecture. To facilitate precise and user-friendly motion control across diverse durations, Tora introduces two novel motion-processing components: the Trajectory Extractor (TE) and the Motion-guidance Fuser (MGF). These modules serve to encode provided trajectories into multi-level spacetime motion patches, meticulously integrating these patches within the stacked structure of DiT blocks. An overview of Tora's workflow is depicted in Figure <ref>.

Spatial-Temporal DiT. The ST-DiT architecture incorporates two distinct block types: the Spatial DiT Block (S-DiT-B) and the Temporal DiT Block (T-DiT-B), arranged in an alternating sequence. The S-DiT-B comprises two attention layers, each performing Spatial Self-Attention (SSA) and Cross-Attention sequentially, succeeded by a point-wise feed-forward layer that serves to connect adjacent T-DiT-B block. Notably, the T-DiT-B modifies this schema solely by substituting SSA with Temporal Self-Attention (TSA), preserving architectural coherence. Within each block, the input, upon undergoing normalization, is concatenated back to the block's output via skip-connections. By leveraging the ability to process variable-length sequences, the denoising ST-DiT can handle videos of variable durations.

During processing, a video autoencoder <cit.> is first employed to diminish both spatial and temporal dimensions of videos. To elaborate, it encodes the input video X ∈ℝ^L × H × W × 3 into video latent z_0∈ℝ^l × h × w × 4, where L denotes the video length and l = L / 4, h = H / 8, w = W / 8.  z_0 is next “patchified", resulting in a sequence of input tokens I ∈ℝ^l × s × d. Here, s = hw/p^2 and p denotes the patch size. I is then forwarded to the ST-DiT, which models these compressed representations. In both SSA and TSA, standard Attention is performed using Query (Q), Key (K), and Value (V) matrices:

    Q = W_Q· I_norm; K = W_K· I_norm; V = W_V· I_norm

Here, I_norm represents the normalized I, W_Q,W_K, W_V are learnable projection matrices.
The textual prompt is first embedded utilizing a T5 text encoder. The cross-attention mechanism is employed between the embedded prompt and the intermediate features from either the SSA or the TSA.

Trajectory Extractor. Trajectories have proven to be a more user-friendly method for controlling the motion of generated videos. Specifically, given a trajectory traj={ (x_i, y_i) } _i=0^L-1, where (x_i, y_i) denotes the spatial position (x, y) at the i-th frame the trajectory passes through. Previous studies primarily encode the horizontal offset u(x_i,y_i) and the vertical offset v(x_i,y_i) as the motion condition for (x_i,y_i): 

    u(x_i,y_i)= x_i+1 - x_i;   v(x_i,y_i)= y_i+1 - y_i.

However, the DiT model employs a video autoencoder and a patchification process to convert the video into video patches. Here, each patch is derived across multiple frames, rendering it inappropriate to straightforwardly employ frame-to-frame offsets. To address this, our TE converts the trajectory into motion patches, which inhabit the same latent space as the video patches. Particularly, we begin by transforming the traj into a trajectory map g ∈ℝ^L × H × W × 2, enhanced with a Gaussian Filter to mitigate scatter. Notably, the first frame employs a fully-zero map. Afterward, the trajectory map g is translated into the RGB color space, producing g_vis∈ℝ^L × H × W × 3 through a flow visualization technique. We use a 3D VAE to compress trajectory maps, achieving an 8x spatial and 4x temporal reduction, aligning with OpenSora framework. Our 3D VAE is based on the Magvit-v2 architecture, with spatial compression initialized using the VAE of SDXL <cit.> to accelerate convergence. We train the model using only reconstruction loss to obtain the compact motion latent representation g_m∈ℝ^l × h × w × 4 from the g_vis.

To match the size of the video patches, we use the same patch size on 
g_m and encode it using a series of convolutional layers, resulting in spacetime motion patches f ∈ℝ^l × s× d_v. Here d_v is the dimension of motion patches. The output of each convolutional layer is skip-connected to the input of the next layer to extract multi-level motion features:

    f_i= Conv^i (f_i - 1) + f_i - 1.

where f_i is the motion condition for i-th ST-DiT block. 




Motion-guidance Fuser. To incorporate DiT-based video generation with the trajectory, we explore three variants of fusion architectures that inject motion patches into each ST-DiT block. These designs are illustrated in Figure <ref>.


  * Extra channel connections. Denote h_i ∈ℝ^l × s × d as the resultant output from the i-th block of the ST-DiT. Following the widespread use of concatenation in GAN-based LVDM, the motion patches are simply concatenated with the previous hidden state h_i-1 along the channel dimension. An additional convolution layer is then added to maintain the same latent size:

    h_i = Conv ([h_i-1, f_i]) + h_i-1.


  * Adaptive Norm layer. Inspired by the adaptive normalization layers employed in GANs, we initially convert f_i into scale γ_i and shift β_i by adding two zero-initialized convolution layers into the ST-DiT block. Subsequently, γ_i and β_i are integrated into h_i through a straightforward linear projection:

    h_i = γ_i· h_i-1 + β_i + h_i-1.



  * Cross-Attention layer. The ST-DiT block has been modified to include an additional Cross-Attention layer following the SSA or TSA, with the motion patches serving as the key and value to integrate with the hidden state h: 

    h_i = CrossAttn ([h_i-1, f_i]) + h_i-1.



We experiment with three types of fusion architectures, finding that the adaptive norm demonstrates the best generation performance and compute efficiency. For the remainder of the paper, MGF employs the adaptive norm layer unless otherwise specified.







 §.§ Training Strategy and Data Processing

To achieve fine-grained control while generating a video using arbitrary trajectories, as well as text, images, or their composites, we introduce several training strategies for different condition injections.

Motion condition training. Inspired by DragNUWA and MotionCtrl, we use a two-stage training approach for trajectory learning. In the first stage, we extract the dense optical flow <cit.> from the training video as the trajectory, offering richer information to accelerate motion learning. During the second stage, to adapt the model from complete optical flow to more user-friendly trajectories, we select a random sample of 1 to N object trajectories from the optical flow according to the motion segmentaton results and flow scores. To address the scattered issues of sparse trajectories, we apply a Gaussian filter for refinement. Upon completion of the two-stage training phase, Tora enables flexible motion control using arbitrary trajectories.

Image condition training. We follow the mask strategy employed by OpenSora to support visual conditioning. Specifically, we randomly unmask frames during training, and the video patches of the unmasked frames are not subjected to any noise. This enables our Tora model to seamlessly integrate text, images, and trajectories into a unified model.

Data Processing. For Tora, it is crucial that the video clips in the training dataset are annotated with both captions and movement trajectories (optical flow). To fulfill this requirement, we employ a structured data processing method. Initially, raw videos are segmented into shorter clips based on scene detection. Subsequently, these clips are evaluated using aesthetic scores. Given our criteria, static videos are excluded by retaining only those with optical flow scores exceeding 3.  To better obtain object trajectories,  We employ the results of motion segmentation and camera detection to remove instances that primarily contain camera trajectories in dynamic scenes. Additionally, dramatic camera or object motions in some videos can cause significant optical flow deviations, interfering with trajectory training. Consequently, we retain these videos with a probability of (1 - score / 100). 
For eligible videos, we generate captions using the PLLaVA model <cit.>. To balance speed and performance, we utilize the 13B version.




§ EXPERIMENTS






 §.§ Experimental Setup

Implementation Details. We train Tora based on OpenSora v1.2 weights. The resolutions of the training videos vary from 144p to 720p. The numbers of frames range from 51 to 204. To balance the training FLOP and required memory for different resolutions and numbers of frames per iteration, we adjust the batch size accordingly from 1 to 25. We use Adam Optimizer <cit.> with a learning rate of 2×10^-5 on 4 NVIDIA A100. We train Tora with the dense optical flow for 2 epochs and fine-tune it with the sparse optical flow for 1 epoch. The maximum number of sampling trajectories N is set to 16. The inference step and the guidance scale are set to 30 and 7.0, respectively.

Dataset. Our training videos are sourced from three datasets: 1) Panda-70M <cit.>, from which we use the training-10M subset containing high-quality videos; 2) Mixkit <cit.>; and 3) Interval Video. After applying our data processing pipeline, we select approximately 630k eligible videos for the training dataset. For inference, we curate 185 clips with diverse motion trajectories and scenes, to serve as a new benchmark for evaluating the motion controllability. Additional information about the construction of the evaluation datasets is available in our supplementary materials.

Metrics. We leverage standard metrics such as Fréchet Video Distance (FVD) <cit.>, and CLIP Similarity (CLIPSIM) <cit.> to quantitatively evaluate video quality. For assessing motion controllability, we leverage Trajectory Error (TrajError), which computes the average distance between the generated and pre-defined trajectories. Human evaluation is also introduced.






 §.§ Quantitative and Qualitative Results

We compare our method against prevalent motion-guided video generation approaches. The evaluation is conducted under three settings: 16, 64, and 128 frames, all with a resolution of 512 × 512 for fair comparison. The provided trajectories are clipped to accommodate different evaluated video lengths. For most U-Net based methods, we employ sequenced inference, wherein the last frame generated by the previous batch serves as the visual condition for the current batch, aligning with their inference settings. As illustrated in Table <ref>, under the 16-frame setting commonly employed by U-Net based methods, MotionCtrl and DragNUWA exhibit better alignment with the provided trajectories but are still weaker compared to our proposed Tora. As the frame count increases, the U-Net based methods show significant deviations in certain frames, with misalignment errors propagating and leading to deformation, motion blur, or object disappearance in subsequent sequences. In contrast, Tora demonstrates high robustness to varying frame numbers due to the integration of the transformer's scaling ability. The movement generated by Tora is smoother and more aligned with the physical world. When evaluated under the 128-frame test setting, Tora's trajectory accuracy surpasses other methods by a factor of 3 to 5, demonstrating its exceptional motion control capabilities. In Figure <ref>, we provide an analysis of the Trajectory Error across varying resolutions and durations. Unlike U-Net based models, which exhibit significant trajectory errors over time, Tora shows only a gradual increase in Trajectory Error as duration increases. This gradual increase in error aligns with the decrease in video quality observed in the DiT model as the duration extends. The results clearly indicate that our method maintains effective trajectory control over longer durations.








Figure <ref> presents a comparative analysis of our proposed method against mainstream motion control techniques. In the first scenario, which involves the co-movement of two men, all methods are able to generate relatively accurate motion trajectories. However, our approach demonstrates superior visual quality. This advantage is primarily due to the utilization of longer sequence frames, which facilitate smoother motion trajectories and more realistic background rendering. For instance, in our generated bicycle scenario, the human legs exhibit realistic pedaling motions, whereas the output from DragNUWA features legs that appear almost horizontally floating, violating physical realism. Additionally, both DragNUWA and MotionCtrl suffer from significant motion blur towards the end of their videos. Furthermore, MotionCtrl introduces unintended camera movements during the riding sequence, despite the absence of camera movement conditions. In another case, DragNUWA shows severe deformation of the lantern as the provided trajectory rises and falls continuously. Although MotionCtrl’s trajectory is relatively accurate, the resulting video fails to match the expected depiction of two lanterns. Overall, our method not only adheres closely to the provided trajectories but also minimizes object deformation, thereby ensuring higher fidelity in motion representation.























































 §.§ Ablation study

In this section, we conduct serval ablation studies to analyze the effects of our design choices. All models are evaluated using 480p resolution, a 16:9 aspect ratio, and 204 frames.



Trajectory Compression. To incorporate the trajectory vector within the same latent space as video patches, we investigate three distinct methods for trajectory compression, as summarized in Table <ref>. The first method samples the mid-frame as the keyframe for successive 4-frame intervals and employs patch-unshuffle for spatial compression. Despite its simplicity, this method proves suboptimal for motion control, primarily due to the potential flow estimation errors when encountering rapid motions or occlusions. Additionally, the magnified dissimilarity between patches induced by the chosen frame interval exacerbates learning challenges. The second method utilizes average pooling to gather successive frames. While this captures a general sense of movement, it inadvertently sacrifices precision by homogenizing the trajectory's direction and magnitude, thereby diluting critical motion details. To preserve the trajectory information between consecutive frames as much as possible, we further employ a 3D VAE to extract the global context of successive trajectory intervals. The trajectory data is visually translated into an RGB image format to leverage existing 3D VAE weights.  Extensive training on a large scale of trajectory videos with this setup yields the most favorable outcomes, underscoring the efficacy of our tailored 3D VAE approach in trajectory compression.

Block design and integrated position of MGF. We train the three variant MFG blocks as previously described, with the results presented in Table <ref>. Notably, the adaptive norm block achieves lower FVD and Trajectory Error compared to both cross-attention and extra channel conditioning methods, while also exhibiting the highest computational efficiency. This superiority is attributed to its capability for dynamic feature adaptation based on diverse conditions without requiring strict alignment, a limitation often encountered with cross-attention. Moreover, it ensures temporal consistency by modulating the conditional information over time, which is crucial for injecting motion cues. In contrast, channel concatenation may cause information congestion, rendering motion signals less effective. During training, we observe that initializing the normalization layer as the identity function is important for achieving optimal performance. Additionally, we evaluate the integration position of the MGF module within the Spatial DiT and Temporal DiT blocks. Our findings reveal that embedding the MGF within the Temporal DiT block notably enhances trajectory motion control, as evidenced by a substantial decrease in Trajectory Error (from 23.39 to 14.25). This approach enhances the MGF's efficiency in interacting with temporal dynamics, thereby significantly improving the fidelity of motion synthesis.

Training Strategies. We evaluate the effectiveness of the two-stage training approach, with the results summarized in Table <ref>. Training exclusively with dense optical flow proves ineffective as it fails to accurately capture the intricate details of the provided sparse trajectories. On the other hand, training solely with sparse trajectories offers limited information, making the learning process more challenging. By initially training with dense flows and subsequently fine-tuning with sparse trajectories, our model demonstrates improved adaptability to various types of trajectory data. This method not only enhances overall performance but also increases the model's versatility in handling diverse motion patterns.


























§ CONCLUSION

This paper introduces Tora, the first trajectory-oriented Diffusion Transformer framework for video generation that integrates text, image, and trajectory conditions. Tora effectively encodes arbitrary trajectories into spacetime motion patches, which align with the scaling properties of DiT, thereby enabling more realistic simulations of physical world movements. By employing a two-stage training process, Tora achieves motion-controllable video generation across a wide range of durations, aspect ratios, and resolutions. Remarkably, it can generate high-quality videos that adhere to specified trajectories, producing up to 204 frames at 720p resolution. This capability underscores Tora's versatility and robustness in handling diverse motion patterns while maintaining high visual fidelity. We hope our work provides a strong baseline for future research in motion-guided Diffusion Transformer methods.







 Paper Title: AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning.
























  Baylor University
  One Bear Place
  Waco
  Texas
maisha_rashid1@baylor.edu



  Baylor University
  One Bear Place
  Waco
  Texas
pablo_rivas@baylor.edu













§ ABSTRACT 
  Multimodal machine learning models that combine visual and textual data are increasingly being deployed in critical applications, raising significant safety and security concerns due to their vulnerability to adversarial attacks. This paper presents an effective strategy to enhance the robustness of multimodal image captioning models against such attacks. By leveraging the Fast Gradient Sign Method (FGSM) to generate adversarial examples and incorporating adversarial training techniques, we demonstrate improved model robustness on two benchmark datasets: Flickr8k and COCO. Our findings indicate that selectively training only the text decoder of the multimodal architecture shows performance comparable to full adversarial training while offering increased computational efficiency. This targeted approach suggests a balance between robustness and training costs, facilitating the ethical deployment of multimodal AI systems across various domains. 







<ccs2012>
 <concept>
  <concept_id>10010147.10010178.10010219.10010220</concept_id>
  <concept_desc>Computing methodologies Image representations</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>10010147.10010257.10010321</concept_id>
  <concept_desc>Computing methodologies Language resources</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>10002978.10003022.10003023</concept_id>
  <concept_desc>Security and privacy Adversarial learning</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
 <concept>
  <concept_id>10010147.10010257.10010293.10010294</concept_id>
  <concept_desc>Computing methodologies Neural networks</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
 <concept>
  <concept_id>10010147.10010257.10010282</concept_id>
  <concept_desc>Computing methodologies Cross-validation</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
</ccs2012>


[500]Computing methodologies Image representations
[500]Computing methodologies Language resources
[300]Security and privacy Adversarial learning
[300]Computing methodologies Neural networks
[100]Computing methodologies Cross-validation

























AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning
    Pablo Rivas
    
======================================================================================





§ INTRODUCTION

Multimodal machine learning is an advanced area of artificial intelligence that integrates and processes multiple types of data inputs, such as text, images, and audio, to perform tasks that mimic human cognitive abilities <cit.>. This integration allows the model to leverage the strengths of each modality, leading to richer interpretations and more accurate predictions that could be achieved by models processing the data in isolation. One of the most compelling applications of multimodal learning involves the combination of visual and textual data, commonly referred to as image-text pairs. This pairing is particularly significant as it mirrors the way humans often receive and interpret information, making the study of these models not only interesting but also aligned with natural human communication patterns.

The increasing deployment of multimodal models in critical applications also raises significant safety and security concerns. These models, like all machine learning systems, are susceptible to adversarial attacks which means intentional inputs designed to confuse the model and provoke incorrect outputs <cit.>. The robustness of multimodal models, therefore, becomes a critical area of focus. Ensuring these models can withstand adversarial attacks is not just a technical necessity but a safety imperative, particularly when these models are employed in sensitive contexts such as autonomous driving, healthcare, and content moderation. 

Making multimodal machine learning models more robust against adversarial attacks is a key challenge <cit.>. These models, which combine different types of data like text and images, can be especially vulnerable because of the way these data types interact. Traditional methods that focus on making each type of data robust on its own can be very demanding and might not address all the issues.

In this paper, we look at a simpler and more efficient approach. Instead of trying to make the entire model robust, we focus on improving just one part of it. We apply adversarial training techniques specifically to the text decoder part of our image captioning model. Our experiments with the Flickr8k and COCO datasets show that this method works well. Training only the text decoder, while keeping the image encoder fixed, gives us results almost as good as training the whole model. However, when we fix the text decoder and train only the image encoder, the performance drops significantly. This shows that the text decoder plays a crucial role in making the model robust against attacks.

This paper helps improve ethical AI by showing a focused way to make multimodal models stronger. This method not only improves the technical side of AI but also helps build public trust in AI systems used in different areas.

This paper is organized as follows: Section 2 reviews related work on adversarial attacks and defenses in multimodal machine learning. Section 3 describes our model architecture and the adversarial training method. Section 4 presents the experimental setup and results on the Flickr8k and COCO datasets. Section 5 discusses what our findings mean for developing robust and ethical AI systems. Finally, Section 6 concludes the paper and suggests future research directions.



§ BACKGROUND

In <cit.>, the authors investigated the vulnerability of multimodal deep learning models to adversarial attacks, finding that even if only one input modality is attacked, the overall performance degrades significantly. The authors highlight the need to explore ways to obtain robust features from multimodal data to achieve useful information from each modality and improve adversarial defense mechanism. To develop adversarial defense mechanisms we explored different adversarial attacks in machine learning models. 
One of the first and best-known adversarial attack techniques is the Fast Gradient Sign Method (FGSM), which involves applying perturbations to the input data in the direction of the gradient of the loss function with respect to the input data to produce adversarial examples <cit.>. Another adversarial attack similar to FGSM, which creates the attack iteratively, is Projected Gradient Descent (PGD) <cit.>. PGD generates adversarial examples by iteratively taking small steps toward the direction of maximum loss function and projecting the outcome back onto the allowed data range at each step. Other well-known adversarial attack strategies are Jacobian Saliency Map Adversary (JSMA) and C&W Attack.
JSMA is an iterative adversarial attack technique that computes a saliency map based on the Jacobian matrix to identify the most significant pixels to perturb in order to mislead neural network classifiers <cit.>. The C&W attack, proposed by Carlini and Wagner, is an powerful and efficient optimization-based adversarial attack that finds quasi-imperceptible perturbations to input examples that reliably cause misclassification in neural networks <cit.>.

The authors in <cit.> discussed about adversarial attack in multimodal machine learning model like CLIP in their paper. They presented new types of adversarial attacks against the multi-modal neural network model CLIP, which integrates the detection of visual objects with text reading. Basic typographical manipulations like misspellings and font changes are included in the attacks, along with conceptual ones that use contradicting text and image inputs to trick the model. Adversarial training is one of the most common defense mechanism for adversarial attack in machine learning models. Adversarial training is using adversarial examples as samples in the training phase of the model. The authors in <cit.> proposed a fast adversarial training algorithm that produces robust models at a low computational cost by recycling the gradient information computed during training. The method is able to train robust image classifiers for the ImageNet dataset in a short time on a modest hardware setup.


§ METHODOLOGY



 §.§ Model Architecture

This work uses an architecture for the image captioning task that combines GPT-2 and Vision Transformer (ViT) models <cit.>, with the GPT-2 <cit.> acting as a decoder and the ViT as an encoder. The ViT model divides an input image of 224 x 224 pixels into 16 x 16 pixel patches. Then, these patches are flattened and projected linearly into a space with dimensions of 768. The transformer encoder receives the image patch sequence and a [CLS] token. The encoder consists of multiple layers of position-wise feedforward and multi-headed self-attention networks. The  token, which compiles data from all patches, represents the whole image. This token's final hidden state at the ViT encoder's output captures the contextualized global aspects of the image, which are essential for generating a caption.  

The decoder obtains the encoded image features from the ViT's [CLS] token, which uses them as the starting point for caption generation. Through many layers of masked multi-headed self-attention, the GPT-2 model processes this input, allowing each point in the output sequence to attend only to earlier positions in the sequence. This arrangement preserves the autoregressive characteristic by ensuring that each word in the caption is generated solely based on the words that came before it. The GPT-2 model produces a sequence of tokens that comprise the image's caption. Every token is created one after the other, and the model predicts the subsequent token by analyzing the preceding tokens and the contextual data extracted from the image.  



 §.§ Adversarial Attack

In our work, we used the adversarial example technique to create adversarial attacks. Specifically, we employed the Fast Gradient Sign Method (FGSM) to generate these examples. FGSM is a method for crafting adversarial examples, which are inputs to machine learning models that have been intentionally modified to cause the model to make incorrect predictions <cit.>.

Given an input 𝐱 and its true label y, we define the loss function J(θ, 𝐱, y), where θ represents the parameters of the model. The goal is to find a small perturbation η such that the perturbed input 𝐱' = 𝐱 + η leads to a misclassification. FGSM achieves this by using the gradient of the loss function with respect to the input 𝐱.

First, we compute the gradient of the loss function:

    𝐠 = ∇_𝐱 J(θ, 𝐱, y).


Then, the adversarial example 𝐱' is generated by adding a perturbation in the direction of the sign of this gradient:

    𝐱' = 𝐱 + ϵ·sign(𝐠).


Here, ϵ is a small constant that controls the magnitude of the perturbation. The sign function, sign(𝐠), is applied element-wise to the gradient, ensuring that each element of the perturbation has the same magnitude, ϵ.




Table <ref> shows examples of the adversarial perturbations generated using FGSM on the Flickr8k and COCO datasets. Despite its simplicity, FGSM is effective in generating adversarial examples that are close to the original inputs but cause the model to make incorrect predictions. This method is computationally efficient and serves as a useful tool for testing and improving the robustness of machine learning models. This table shows how adversarial attacks are imperceptible to human eyes, as shown in the second column. The third column demonstrates the perturbations by showing the difference between the original and perturbed images. In the 'Difference' column, we can see square or patch-like patterns. This happens because of ViT's architecture, which processes input images by dividing them into patches. FGSM leverages the gradients computed for each patch, generating perturbations that exploit the vulnerabilities within these regions. Table <ref> underscores the necessity for robust models capable of defying such attacks. By incorporating adversarial examples into the training process, we can enhance the model's ability to defend against such attacks and improve its overall robustness.



 


 §.§ Dataset

In this study, we leverage two widely-used and benchmark datasets for the image captioning task: Flickr8k and COCO (Common Objects in Context).


  §.§.§ Flickr8k Dataset

The Flickr8k dataset comprises 8,000 images, each accompanied by five human-annotated captions. The images are sourced from Flickr, a popular online photo-sharing platform, and are categorized into 20 distinct classes.


  §.§.§ COCO Dataset

The COCO dataset is a larger and more challenging resource for image captioning, containing 123,287 images with five captions per image. Unlike the Flickr8k dataset, the images in COCO are sourced from various online platforms, including Flickr, and are categorized into 80 classes. 


 §.§ Evaluation Metric

To evaluate the performance for image captioning task, we used Bilingual Evaluation Understudy (BLEU) score <cit.>. Comparing the generated captions to the reference captions in the dataset generates a similarity score known as the BLEU score. Higher scores correspond to a closer match. The BLEU score computes the n-gram overlap between the model's output and the ground truth. There is a perfect match between the generated and reference captions when the BLEU score is 1, from 0 to 1. 


 §.§ Experiment

We used a multi-phased approach incorporating adversarial training to increase the robustness of our multimodal image captioning model, which combines a Vision Transformer (ViT) encoder and a GPT-2 decoder. We conducted our experiment in five phases as follows:

    
  * We trained the ViT-GPT-2 architecture on the Flickr8k dataset without considering adversarial factors, creating a baseline model that served as a benchmark for assessing the following experiments.
    
  * We applied the FGSM to create adversarial examples for the training and test sets. We used these samples to train the model and evaluated the performance of the model. 
    
  * To improve robustness, we performed adversarial training on the model using both the original data and adversarial examples.
    
  * We investigated a more focused strategy to further improve the model's robustness. Initially, we used only the GPT-2 decoder for adversarial training while freezing the ViT encoder.
    
  * We reversed the process by freezing the GPT-2 decoder and using the ViT encoder alone for adversarial training.



We conducted steps four and five three times each and calculated the average of these trials.  
Focusing on a single modality at a time allowed us to implement more effective adversarial defense mechanisms without having to scale them across the entire model. We repeated these five steps of experiment with COCO dataset as well to access the robustness of our model. For both the datasets we have used the same perturbation magnitude, ϵ = 0.1.



§ RESULTS

After conducting experiments on both the Flickr8k and COCO datasets, we evaluated the performance of our multimodal machine learning models using the BLEU score metric. The goal of these experiments was to find an efficient way to increase the robustness of these models against adversarial attacks. As shown in Table <ref>, our baseline model, trained on the original data, initially achieved the best performance. However, when we generated adversarial examples using the FGSM and trained the model with these samples, its performance decreased significantly. This suggests that our multimodal model is not robust against adversarial attacks, as generating a single adversarial example for each image in the dataset was sufficient to degrade its performance.

To enhance the model's robustness, we trained it with both the original data and the adversarial examples. This adversarial training approach improved the model's performance, indicating that incorporating adversarial examples during training can help mitigate the impact of such attacks.

Aiming to find an efficient way to increase the robustness of our multimodal model, we investigated whether training only one of its components (either the image model or the text model) would be sufficient. First, we froze the image model (ViT) and trained the text model with both the original data and the adversarial examples. As shown in Table <ref>, the model's performance in this case was lower than with full adversarial training, although the difference was not significant. Alternatively, we froze the text model (GPT) and trained the image model; in this case, we can see the performance is much lower than the full adversarial training's performance. 


We conducted similar experiments using the COCO dataset, and the results are presented in Table <ref>. The baseline performance on the COCO dataset was higher than on Flickr8k. However, when the model was trained solely on adversarial examples generated from the COCO dataset, the performance decreased significantly, as expected. In the third phase, where we trained the model on both the original data and the adversarial samples, the result was comparable to the baseline model's performance.

When we froze the image model (ViT) and trained only the text model, the difference in BLEU score compared to the full adversarial training approach was relatively small. This observation aligns with our findings from the Flickr8k dataset.





§ DISCUSSION

Through experiments on both the Flickr8k and COCO datasets, we can conclude that while the adversarial training technique improves the robustness of our multimodal models against adversarial attacks, their performance does not fully match the baseline levels achieved with clean data. However, the adversarially trained models showed performance close to the baseline, suggesting that this approach can effectively mitigate the impact of adversarial attacks.

Our findings suggest that performing adversarial training only in the text model can achieve comparable performance to full adversarial training. However, freezing the text model or the GPT-2 model during training significantly degrades performance. When the ViT encoder is frozen and adversarial training is performed solely on the GPT-2 decoder, the model still benefits from the ViT encoder's robust feature extraction capabilities. The ViT encoder continues to deliver consistent and reliable image features, enabling the GPT-2 decoder to concentrate on learning to generate text robustly. In contrast, when the GPT-2 decoder is frozen and adversarial training is applied solely to the ViT encoder, the model's capacity to generate coherent and contextually relevant text is hampered. Since the parameters of the GPT-2 decoder remain fixed, it cannot adjust to the alterations in the image features produced by the ViT encoder during adversarial training. Consequently, this lack of adaptability in the text generation process results in a decline in performance. In conclusion, this finding presents an opportunity to increase computational efficiency by focusing the training efforts on a text modality, with a modest trade-off in terms of performance compared to training the entire multimodal model.



§ CONCLUSION

Our research has shown important steps forward in making multimodal machine learning models more robust, especially for tasks like image captioning. We combined Vision Transformer (ViT) and GPT-2 architectures and used adversarial training methods to focus on specific parts of the model. This focused approach helps protect against adversarial attacks without needing to train the entire model, which saves time and resources. Through our experiments with the Flickr8k and COCO datasets, we found that applying adversarial training to just the text decoder can make the models much more robust. While the performance of these models with adversarial training is slightly lower than those trained only on clean data, the trade-off is minimal. This means we can improve model safety without losing much accuracy. Freezing the image encoder and training the text decoder helps balance performance and robustness efficiently. On the other hand, training the image encoder alone does not provide the same benefits, highlighting the text decoder's key role in maintaining robustness.

Future work will explore changes in activation functions for additional gains in adversarial robustness <cit.>. Current and future work supports the goals of Ethical AI by making AI systems safer and reliable. By focusing on specific parts of the model, we can make the adversarial defense process more efficient. This targeted approach can lead to the development of AI systems that are both resource-efficient and secure for public use. Enhancing the robustness of AI models against adversarial attacks builds public trust and confidence in AI systems, especially in critical applications like healthcare, autonomous driving, and content moderation. 




§ ACKNOWLEDGMENT

This research work was part of a project funded by the National Science Foundation under grants CNS-2210091, CHE-1905043, and CNS-2136961.

ACM-Reference-Format








 Paper Title: MUFASA: Multi-View Fusion and Adaptation Network with Spatial Awareness for Radar Object Detection.




MUFASA





X. Peng et al.



Technical University of Munich, Chair for Design Automation, Munich, Germany Infineon Technical University of Munich, Neubiberg, Germany
Xiangyuan.Peng@infineon.com China University of Geosciences (Wuhan), China






MUFASA: Multi-View Fusion and Adaptation Network with Spatial Awareness for Radar Object Detection
    Xiangyuan Peng1,2 Miao Tang3 Huawei Sun1,2Kay Bierzynski2Lorenzo Servadei1 Robert Wille1

    August 5, 2024
==================================================================================================


§ ABSTRACT 
In recent years, approaches based on radar object detection have made significant progress in autonomous driving systems due to their robustness under adverse weather compared to LiDAR. However, the sparsity of radar point clouds poses challenges in achieving precise object detection, highlighting the importance of effective and comprehensive feature extraction technologies.
To address this challenge, this paper introduces a comprehensive feature extraction method for radar point clouds. 
This study first enhances the capability of detection networks by using a plug-and-play module, GeoSPA. It leverages the Lalonde features to explore local geometric patterns. Additionally, a distributed multi-view attention mechanism, DEMVA, is designed to integrate the shared information across the entire dataset with the global information of each individual frame.
By employing the two modules, we present our method, MUFASA, which enhances object detection performance through improved feature extraction. The approach is evaluated on the VoD and TJ4DRaDSet datasets to demonstrate its effectiveness. In particular, we achieve state-of-the-art results among radar-based methods on the VoD dataset with the mAP of 50.24%.







§ INTRODUCTION

Object detection plays an important role in autonomous driving systems, providing essential perception capabilities. Initially, object detection relied heavily on camera-based methods, leveraging high-resolution RGB images to accomplish 2D detection <cit.>. However, cameras lack 3D geometric clues, which is crucial for accurate perception. Hence, Light Detection and Ranging (LiDAR) sensors are integrated into systems to generate 3D point clouds, which are capable of offering geometric details and lead to more accurate object detection <cit.>. Nevertheless, LiDAR's performance diminishes under adverse weather conditions, including fog, rain, and snow <cit.>. The high cost also limits its widespread application <cit.>.

As an alternative, radar has gained sufficient interest due to its robust performance under adverse conditions and capability to provide 3D spatial and velocity information <cit.>. Yet, radar point clouds pose a unique challenge: sparsity. Numerous methods have been proposed to extract rich features from the sparse points, including sparse theory and advanced signal processing techniques <cit.>.
However, these methods are still insufficient to fully explore the information in radar point clouds. 

Currently, most feature extraction technologies for sparse radar point clouds primarily focus on extracting local and global features within individual frames <cit.>. 
While understanding the global semantic information within each frame is essential for scene comprehension, the hidden shared information across the entire dataset is often neglected. For instance, the locations of cars and pedestrians are usually in the center and at the edges of different scenes, respectively. These dataset-wide features can enhance detection confidence by incorporating shared information across different frames.
Moreover, local feature extraction often overlooks the significance of geometric spatial patterns. Accurately distinguishing between similar objects, such as pedestrians and signposts, or cars and square trash bins, heavily relies on precise analysis of local geometric patterns. However, during the learning process in feature extraction, the details of these patterns can be missed.
These considerations underscore the need for more comprehensive and sophisticated feature extraction strategies.

Therefore, an advanced feature extraction network, MUFASA, is proposed to leverage complex local and global information within each radar point cloud frame, along with shared dataset-wide features. 

Contribution This study focuses on enhancing radar point cloud object detection capabilities through comprehensive feature extractions. The main contributions are as follows:




  ∙ A flexible plug-and-play Geometric Spatial Pattern Analysis module, GeoSPA, is designed to explore local geometric features within various detection networks. It enhances the extraction of local features and provides a deep understanding of spatial information in the feature extraction backbone.

  ∙ A Distributed External Multi-View Attention module, DEMVA, is designed to integrate dataset-wide features across different frames. It merges the dataset-wide information with the global features of each frame from the cylinder and Bird's Eye View (BEV) projections.

  ∙ Extensive experiments on the View-of-Delft (VoD) <cit.> and TJ4DRadSet <cit.> datasets
demonstrate the superior efficiency and accurate detection performance of the proposed method.




§ RELATED WORK




 §.§ Radar-based 3D Object Detection

Radar point cloud detection methods, often inspired by LiDAR techniques, are classified into voxel-based, pillar-based, point-based, and fusion-based networks <cit.>.
Voxel-based strategies <cit.> convert point clouds into 3D voxel grids, sacrificing some information due to discretization.
Pillar-based approaches, such as FastPillars <cit.>, transform point clouds into 2D pseudo-images for BEV analysis, effectively utilizing 2D convolutional neural networks (CNNs) for feature extraction.
Point-based methods, highlighted by PointNet <cit.>, directly extract features from raw points, preserving fine-grained details. However, they may suffer higher computational costs.
Fusion-based strategies, like PV-RCNN <cit.>, integrate multiple approaches to improve detection performance while considering computational demands.

Despite advancements, extracting precise features from radar point clouds with a single method remains a significant challenge because of the sparsity. Therefore, our network integrates pillar and point-based approaches to ensure comprehensive feature extraction while balancing computational efficiency and detection accuracy.




 §.§ Descriptors for Feature Extraction

The inherent sparsity and irregular distribution of radar point clouds necessitate efficient feature extraction in 3D spaces. An effective point cloud descriptor is capable of capturing spatial geometric characteristics while being invariant to translation, scaling, and rotation. Optimizing these descriptors can improve object detection accuracy while reducing the computational cost.

Conventional descriptors combine spatial statistical information of feature points and their neighbors to extract geometric features <cit.>. With the advancement of deep learning, integrating hand-crafted and deep learning-derived features has marked a significant evolution. For instance, <cit.> employs spherical histograms for spatial mapping around points, with neural networks transforming these representations into Euclidean space. PPFNet <cit.> and PPF-FoldNet <cit.> leverage Fisher Vectors in CNNs to navigate the unstructured characteristic of point clouds. 
The fusion of traditional 3D feature descriptors with deep learning approaches represents a noteworthy advancement in point cloud processing, resulting in more robust and invariant feature representations.

However, a major drawback of most feature extraction methods is the descriptors are tightly coupled with specific network structures, limiting their applicability across different detection systems. To address this limitation, we introduce our GeoSPA, which can integrate into various detection networks, enhancing the generalization of feature extraction.



 §.§ Attention Mechanism in 3D Object Detection

The attention mechanism achieves permutation invariance, dynamically focuses on relevant features, and adapts to sparsity and density variations. Also, the ability of attention mechanisms to stably extract information makes them particularly suitable for the complex point cloud analysis <cit.>.

For instance, Box attention <cit.> introduces a grid-based attention weighting for 3D object detection in outdoor scenes. TANet <cit.> enhances point cloud discrimination by applying attention at multiple levels to refine the feature representation. CSANet <cit.> employs self-attention to fuse point cloud features with coordinate information. Additionally, Point-BERT <cit.> designs a transformer-based pre-training for point clouds.

These advancements, mainly based on the self-attention mechanism, highlight the flexibility and effectiveness of the attention mechanism in 3D object detection. However, external attention, which has a strong learning ability to integrate the information across the entire dataset, is not commonly employed in point cloud detection. Therefore, we designed DEMVA to capture the shared dataset-wide information.



§ PROPOSED METHOD

In this section, we outline the overall architecture of MUFASA. Subsequently, the GeoSPA module, designed to extract local geometric patterns within the point clouds, is presented. Later, the DEMVA module is introduced, which utilizes shared information across the entire dataset and the global features in each frame, thereby enhancing the model's overall performance and robustness.



 §.§ Overall Structure

The structure of MUFASA is illustrated in Fig. <ref>.
Initially, the radar point clouds undergo transformation into multi-view representations, including BEV and cylindrical projections, inspired by the MVFAN <cit.>. On the one hand, extracted features through the cylindrical branch maintain the integrity of 3D information. On the other hand, the BEV branch captures a comprehensive scene perspective, effectively mitigating obstruction by obstacles.
Subsequently, the features from both representations are fed into the DEMVA. The distributed attention mechanisms enhance feature comprehension by exploring relationships across the entire dataset, addressing and mitigating potential information loss.


In parallel, our network incorporates a point-wised branch with Farthest Point Sampling (FPS) and our GeoSPA module, designed to capture point-wise features. The point-wise features are then fused with the multi-view features derived from the DEMVA branch.
Enriched point-wise features are subsequently reprojected back into the BEV. The final features are treated as input for the detection heads to generate proposals. 

The GeoSPA is designed for integration with various detection frameworks, acting as a flexible plug-and-play module integrated into point-wise feature extraction. In single-stage detectors like PointPillars <cit.>, with a detection head SSD <cit.>, GeoSPA enriches point-wise feature extraction without the need for proposal refinement. In two-stage detectors like PV-RCNN <cit.>, where the proposal generation head is RFN <cit.>,  GeoSPA can be utilized in two aspects. It is applied not only in the point-wise feature extraction but also within the Region of Interest (RoI) to refine spatial features, aiding in the proposal refinement. The implementation of GeoSPA in different locations demonstrates its adaptability across different detection architectures as a plug-and-play module.



 §.§ Geometric Spatial Pattern Analysis


In road object detection, cars often exhibit distinct rectangular shapes, particularly for stationary vehicles, where Doppler velocity provides less information. Meanwhile, cyclists differ from pedestrians in both width and height. To enhance the features obtained from deep learning alone, MUFASA integrates the GeoSPA based on the Lalonde descriptor, which is able to capture spatial features.




  
Definition of Lalonde Feature: The spatial descriptors are derived through two processes: Signature and Histogram. Following the methodology outlined in <cit.>, we analyze the distribution of points within a local spatial domain by conducting Principal Component Analysis (PCA) on the covariance matrix of the 3D points' locations. The covariance matrix M is expressed as Equation <ref>



    M = 1/N∑_i=1^N (X_i - X̅)(X_i - X̅)^T,

where {X_i} = {(x_i, y_i, z_i)^T} is the i^th point in the point clouds and X̅ = 1/N∑_i=1^N X_i is the mean value of all the points. The eigenvectors obtained from PCA are e_1, e_2 and e_3 with eigenvalus x_1, x_2 and x_3 while x_1 ≥ x_2 ≥ x_3. For different geometric representations, the eigenvalues showcase different features:



    x_1 ≈ x_2 ≈ x_3      for scattered points, 
    
    x_1 ≫ x_2 ≈ x_3      for a linear structure, 
    
    x_1 ≈ x_2 ≫ x_3      for a solid surface.

Then, l_scatter, l_linear, and l_surface are defined as the scatter-ness, linear-ness, and surface-ness. They can be calculated as Equation <ref>



    l_scatter = x_1,    l_linear = x_1 - x_2,    l_surface = x_2 - x_3.

By taking the histogram of l_scatter, l_linear, and l_surface for all points within the point clouds, we obtain the three Lalonde features L_1, L_2, and L_3 of the object.


The Lalonde features L_1, L_2, and L_3 for cars, cyclists, and pedestrians across different orientations in the VoD dataset are depicted in Fig. <ref>. Here, cars exhibit more pronounced planarity features in L_1 and L_2 compared to other objects, while cyclists can be more effectively detected from the side due to prominent point and planar features. Pedestrians consistently exhibit similar geometric characteristics across different orientations.



  
Implementation of GeoSPA: We introduced GeoSPA based on Lalonde features to capture the detailed spatial pattern. 
Its structure is shown in Fig. <ref>.


On one hand, a PointNet <cit.> architecture is used to extract point-wised feature as Equation <ref>

 

    F_pw = PointNet(S) = max(MLP(S)),

where F_pw is the point-wised features within the subset S through the application of MLP layers and max pooling in PointNet. 
On the other hand, for each point p_i ∈ S, we compute the Lalonde descriptors L(p_i) within its fixed neighborhood N(p_i) and transform it into high-dimensional Lalonde features F_Lalonde(p_i) using an MLP layer. This process is captured by Equation <ref>



    F_Lalonde(p_i) = MLP(L(N(p_i))).

The final spatial feature, F_sp(p_i), is calculated by concatenating the point-wise features and the high-dimensional Lalonde features as Equation <ref>



    F_sp(p_i) = F_pw(p_i) 
     c  F_Lalonde(p_i),

where c denotes the concatenation operation.

We designed GeoSPA with plug-and-play functionality, allowing it to be embedded into any point-wise feature extraction process. In single-stage networks, GeoSPA is incorporated into the initial feature extraction branch. For two-stage networks with proposal refinement, GeoSPA can also operate within the Region of Interest (ROI) to enhance local spatial features except in a point-wise feature extraction branch, thereby aiding in the refinement of proposals.




 §.§ Distributed External Multi-View Attention

Our DEMVA module leverages dual-projection techniques: cylindrical and BEV views with external attention mechanisms. The multi-view extraction captures global features from multiple perspectives, while the distributed attention mechanism incrementally learns shared information across the entire dataset.

For BEV projection, each point p_n (x_n, y_n, z_n) is systematically allocated to a defined pillar as defined in Equation <ref>



    ℳ_BEV(p^n_BEV) → q^m_BEV,

where ℳ_BEV symbolizes the mapping operation that assigns points to pillars in the BEV projection, p^n_BEV denotes the n^th point in BEV coordinates, and q^m_BEV denotes the m^th pillar.
In the cylindrical projection, a transformation of points into cylindrical coordinates (ρ_n, θ_n, z'_n) is applied to maintain the fidelity of vertical dimensions as below:



    ρ_n = √(x_n^2 + y_n^2),   θ_n = arctan(y_n/x_n),    z'_n = z_n,

where ρ_n is the radial distance from the origin to the n^th point, θ_n specifies the angular orientation around the z-axis, and z'_n upholds the original elevation data. Following this transformation, a similar pillar structuring is implemented in the cylindrical space as Equation <ref>



    ℳ_cyl(p^n_cyl) → q^m_cyl,

where ℳ_cyl means the function that maps the n^th point to the m^th pillar. 

Later, q^m_BEV and q^m_cyl are mapped onto pseudo images. Utilizing two 2D CNNs, we further extract global features I_BEV and I_cyl with dimension (B, C × Z, H, W) from the BEV and cylindrical pseudo images. Here, B denotes the batch size, C indicates the number of channels, Z is the feature depth, H and W refer to the height and width of the feature maps.

Strong dataset-wide correlations, such as vehicles predominantly occupying central positions while pedestrians and cyclists are often at the edges, help to enhance the understanding of the road environment. Furthermore, objects within the same category always exhibit similar shapes and motion patterns. Therefore, we employ external attention mechanisms, effectively capturing and encoding shared information across the entire dataset.



Fig. <ref> shows the process of extracting dataset-wide features by DEMVA. Initially, the inputs undergo a preliminary transformation by a matrix M_k, leading to an attention map via a Softmax and Normalization layer. This attention map is then multiplied with a second external matrix, M_v, leading to the feature graph. The resultant feature graph is subsequently fused with the original cylindrical and BEV features.
The M_k and M_v are the key and value components in the attention mechanism. 
Both matrices, as the weight of one-dimensional convolution, are dynamically updated throughout the learning phase.
Therefore, the matrices effectively function as memory units, continuously integrating information from the new frame into the matrices, thus enabling the extraction of dataset-wide features.
As a result, the DEMVA applies this dataset-wide knowledge to each individual frame and facilitates the fusion of dataset-wide features with single-frame global features.

Given the inherent sparsity of radar point clouds, our DEMVA plays a crucial role in enhancing the model's performance by extracting global information through multiple perspectives and further enriching it with dataset-wide features via external attention mechanisms.



§ EXPERIMENT




 §.§ Dataset and Metrics

The MUFASA network is evaluated on the VoD [https://github.com/tudelft-iv/view-of-delft-dataset] and TJ4DRadSet [https://github.com/TJRadarLab/TJ4DRadSet] dataset to validate its efficacy. The VoD dataset comprises 8,693 frames of synchronized and calibrated data from LiDAR, cameras, and 4D radar. The data is mainly acquired in busy urban environments, including diverse traffic components. We evaluated our methods on the validation set, as the test server is unavailable. The TJ4DRadSet dataset includes 40,000 frames of synchronized LiDAR, camera, and radar data, with 7,757 frames annotated. This dataset spans various scenarios, including elevated roads, complex crossroads,  single-direction lanes, and urban streets.

To assess our method, we employed Average Precision (AP) for each class and calculated the mean Average Precision (mAP) across all the classes. To calculate AP, we determined the Intersection over Union (IoU) between the ground truth and predicted 3D bounding boxes. A minimum IoU threshold of 50% was set for the car and truck categories, while a threshold of 25% was applied for pedestrian and cyclist categories. 
In the VoD dataset, the detection performance was analyzed in two distinct regions: the entire scene and the driving corridor, aligned with the original paper <cit.>. 
For the TJ4DRadSet dataset, evaluations are conducted in both BEV and 3D perspectives as in <cit.>.




 §.§ Implementation Details


The models are trained for 80 epochs with a batch size of 8, utilizing 4 NVIDIA Tesla P40 GPUs. For training, we used the Adam optimizer <cit.>, with an initial learning rate of 0.03 and a weight decay rate of 0.01.
Our implementation is based on OpenPCDet <cit.>, a comprehensive library designed for point cloud tasks. 
Data augmentation is implemented, including rotation, flipping, and scaling, to improve the models' robustness and generalization capabilities.



 §.§ Quantitaive Results


We performed a comprehensive comparison of our proposed method with existing point cloud object detection techniques on the VoD and TJ4DRaDSet datasets. The results are detailed in Table <ref> and <ref>.




Table <ref> demonstrates the remarkable performance of our method, surpassing state-of-the-art radar-based object detection methods on the VoD dataset.
Leveraging the flexibility of key modules, the MUFASA can be embedded into different detection heads. The results also show a consistent enhancement in performance across different detection architectures. 
Notably, when incorporated with the PV-RCNN detection head, our approach achieves the highest mAP of 50.24% over the entire area and 70.43% within the driving corridor. This marks a significant improvement of 3.96% and 5.81%, respectively, over the PV-RCNN performance metrics. 

Furthermore, compared to radar and camera fusion methodologies such as BEVFusion <cit.> and RCFusion <cit.>, our approach maintains competitive performance with the highest mAP, underscoring its efficacy and robustness in capturing comprehensive features.





From Table <ref>, MUFASA demonstrates robust performance in radar-based object detection, achieving the highest mAP among radar-only methods, with 30.23% for 3D detection and 39.10% for BEV detection. Notably, MUFASA excels in detecting trucks with PV-RCNN head, highlighting its effectiveness in large object detection. Furthermore, MUFASA with the PV-RCNN head (pv) significantly outperforms the PointPillars head (pp) in detecting cyclists and trucks on the TJ4DRaDSet dataset, showing improvements of 5.81% and 5.64% in 3D detection, respectively. This enhanced performance is due to the GeoSPA module in the ROI stage, which is particularly effective at refining bounding boxes with an obvious length-width ratio.
However, in comparison to radar and camera fusion methods, such as MVX-Net <cit.> and RCFusion <cit.>, MUFASA still exhibits a minor performance gap.



 §.§ Qualitative Results

We visualize the bounding boxes generated by PointPillars<cit.>, PV-RCNN<cit.>, and MUFASA on the VoD dataset. Ground truth boxes are marked in green, while predictions are in red. From Fig. <ref>, it is evident that MUFASA performs better detection with fewer false negative bounding boxes.





 §.§ Ablation Study

The effectiveness of the critical modules are illustrated through an ablation study. Additionally, the generalization of our methods is demonstrated by incorporating key models with different settings.



  §.§.§ Analysis of key modules

We first removed the DEMVA and GeoSPA from the MUFASA architecture. The remaining network was retrained with the same parameters as a baseline. Subsequently, we incrementally reintegrated each module into the baseline to observe their individual contributions. 



The analysis from Table <ref> reveals that the DEMVA module enhances the detection performance for all the classes by integrating the dataset-wide features into each frame. Meanwhile, the GeoSPA module is particularly effective in improving the detection of pedestrians, which have fewer points. For instance, incorporating GeoSPA into our framework results in a substantial increase in pedestrian AP by 6.14% and 7.63% across all areas and the driving corridor, respectively, compared to the baseline. The main reason is our GeoSPA effectively captures the local features. Pedestrians, despite their varied movement patterns, maintain relatively consistent shapes. Learning the detailed geometric information in the neighborhood helps to identify the features that are uniquely from one category.



  §.§.§ Analysis of Different Settings


To validate the rational layout of key modules, we strategically deployed the DEMVA and GeoSPA modules at different locations within the network architecture.



  
GeoSPA on different locations:
Table <ref> analyzes the impact of the GeoSPA module when applied at various locations within the network. Initially, we establish a baseline scenario where GeoSPA is not employed. Then, we apply GeoSPA in the early stage for the point-wise feature extraction and in the later stage to refine proposals with the PV-RCNN detection head.



The outcomes indicate that integrating GeoSPA at both stages yields improvements in mAP for the entire area as well as the driving corridor.




  
DEMVA on different branches:
We implement external attention to different projection branches. 


The results in Table <ref> highlight the impact of external attention mechanisms within both the cylinder and BEV branches. Therefore, extracting hidden dataset-wide features with external attention mechanisms consistently enhances object detection accuracy. 



§ CONCLUSION AND DISCUSSION


Traditional radar object detection methods often struggle to fully exploit the detailed local and global features in a single frame, as well as the dataset-wide features across the entire dataset. 
In our method, MUFASA, the GeoSPA is designed as a flexible plug-and-play module, enhancing the model's capability to capture critical geometric details and recognize local spatial patterns across various detection networks.
Additionally, the DEMVA module synthesizes shared information across different frames and integrates it into the global features of each individual frame. 
Empirical evaluations on the VoD and TJ4DRaDSet datasets demonstrate the superiority of our approaches over existing radar point cloud-based object detection techniques. Specifically, our method achieved an mAP of 50.24% and 70.43% on the entire area and driving corridor of the VoD dataset, representing a notable improvement of 3.96% and 5.81%.

However, when compared to sensor fusion-based detection methods, our approaches may still exhibit a lower mAP. This observation underscores the need for future research focusing on integrating camera, LiDAR, and radar data. Such integration aims to improve detection accuracy and robustness under various environments, thereby expanding the practical use of our methods in real-world scenarios.



§ ACKNOWLEDGEMENT

This research has been conducted as part of the DELPHI project, which is funded by the European Union, under grant agreement No 101104263. However, views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor the granting authority can be held responsible for them.













10

kaur2022tools
J. Kaur and W. Singh, “Tools, techniques, datasets and application areas for object detection in an image: a review,” Multimedia Tools and Applications, vol. 81, no. 27, pp. 38297–38351, 2022.

wu2020deep
Y. Wu, Y. Wang, S. Zhang, and H. Ogai, “Deep 3d object detection networks using lidar data: A review,” IEEE Sensors Journal, vol. 21, no. 2, pp. 1152–1171, 2020.

tang2020performance
L. Tang, Y. Shi, Q. He, A. W. Sadek, and C. Qiao, “Performance test of autonomous vehicle lidar sensors under different weather conditions,” Transportation research record, vol. 2674, no. 1, pp. 319–329, 2020.

raj2020survey
T. Raj, F. Hanim Hashim, A. Baseri Huddin, M. F. Ibrahim, and A. Hussain, “A survey on lidar scanning mechanisms,” Electronics, vol. 9, no. 5, p. 741, 2020.

jiang2022t
T. Jiang, L. Zhuang, Q. An, J. Wang, K. Xiao, and A. Wang, “T-rodnet: Transformer for vehicular millimeter-wave radar object detection,” IEEE Transactions on Instrumentation and Measurement, vol. 72, pp. 1–12, 2022.

ding2023sparsity
C. Ding, L. Zhang, H. Chen, H. Hong, X. Zhu, and F. Fioranelli, “Sparsity-based human activity recognition with pointnet using a portable fmcw radar,” IEEE Internet of Things Journal, 2023.

sun20214d
S. Sun and Y. D. Zhang, “4d automotive radar sensing for autonomous vehicles: A sparsity-oriented approach,” IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 4, pp. 879–891, 2021.

meyer2021graph
M. Meyer, G. Kuschk, and S. Tomforde, “Graph convolutional networks for 3d object detection on radar data,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3060–3069, 2021.

yang2020radarnet
B. Yang, R. Guo, M. Liang, S. Casas, and R. Urtasun, “Radarnet: Exploiting radar for robust perception of dynamic objects,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16, pp. 496–512, Springer, 2020.

palffy2022multi
A. Palffy, E. Pool, S. Baratam, J. F. Kooij, and D. M. Gavrila, “Multi-class road user detection with 3+ 1d radar in the view-of-delft dataset,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4961–4968, 2022.

zheng2022tj4dradset
L. Zheng, Z. Ma, X. Zhu, B. Tan, S. Li, K. Long, W. Sun, S. Chen, L. Zhang, M. Wan, et al., “Tj4dradset: A 4d radar dataset for autonomous driving,” in 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp. 493–498, IEEE, 2022.

mao20223d
J. Mao, S. Shi, X. Wang, and H. Li, “3d object detection for autonomous driving: A review and new outlooks,” arXiv preprint arXiv:2206.09474, 2022.

shi2019part
S. Shi, Z. Wang, X. Wang, and H. Li, “Part-aˆ 2 net: 3d part-aware and aggregation neural network for object detection from point cloud,” arXiv preprint arXiv:1907.03670, vol. 2, no. 3, 2019.

sindagi2019mvx
V. A. Sindagi, Y. Zhou, and O. Tuzel, “Mvx-net: Multimodal voxelnet for 3d object detection,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 7276–7282, IEEE, 2019.

zhou2023fastpillars
S. Zhou, Z. Tian, X. Chu, X. Zhang, B. Zhang, X. Lu, C. Feng, Z. Jie, P. Y. Chiang, and L. Ma, “Fastpillars: A deployment-friendly pillar-based 3d detector,” arXiv preprint arXiv:2302.02367, 2023.

qi2017pointnet
C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652–660, 2017.

shi2020pv
S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, “Pv-rcnn: Point-voxel feature set abstraction for 3d object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10529–10538, 2020.

lalonde2006natural
J.-F. Lalonde, N. Vandapel, D. F. Huber, and M. Hebert, “Natural terrain classification using three-dimensional ladar data for ground robot mobility,” Journal of field robotics, vol. 23, no. 10, pp. 839–861, 2006.

ghorbani2022novel
F. Ghorbani, H. Ebadi, A. Sedaghat, and N. Pfeifer, “A novel 3-d local daisy-style descriptor to reduce the effect of point displacement error in point cloud registration,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 2254–2273, 2022.

khoury2017learning
M. Khoury, Q.-Y. Zhou, and V. Koltun, “Learning compact geometric features,” in Proceedings of the IEEE international conference on computer vision, pp. 153–161, 2017.

deng2018ppfnet
H. Deng, T. Birdal, and S. Ilic, “Ppfnet: Global context aware local features for robust 3d point matching,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 195–205, 2018.

deng2018ppf
H. Deng, T. Birdal, and S. Ilic, “Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors,” in Proceedings of the European conference on computer vision (ECCV), pp. 602–618, 2018.

qiu2021investigating
S. Qiu, Y. Wu, S. Anwar, and C. Li, “Investigating attention mechanism in 3d point cloud object detection,” in 2021 International Conference on 3D Vision (3DV), pp. 403–412, IEEE, 2021.

nguyen2022boxer
D.-K. Nguyen, J. Ju, O. Booij, M. R. Oswald, and C. G. Snoek, “Boxer: Box-attention for 2d and 3d transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4773–4782, 2022.

liu2020tanet
Z. Liu, X. Zhao, T. Huang, R. Hu, Y. Zhou, and X. Bai, “Tanet: Robust 3d object detection from point clouds with triple attention,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, pp. 11677–11684, 2020.

wang2022cross
G. Wang, Q. Zhai, and H. Liu, “Cross self-attention network for 3d point cloud,” Knowledge-Based Systems, vol. 247, p. 108769, 2022.

yu2022point
X. Yu, L. Tang, Y. Rao, T. Huang, J. Zhou, and J. Lu, “Point-bert: Pre-training 3d point cloud transformers with masked point modeling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19313–19322, 2022.

yan2023mvfan
Q. Yan and Y. Wang, “Mvfan: Multi-view feature assisted network for 4d radar object detection,” in International Conference on Neural Information Processing, pp. 493–511, Springer, 2023.

lang2019pointpillars
A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12697–12705, 2019.

liu2016ssd
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pp. 21–37, Springer, 2016.

kingma2014adam
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.

od2020openpcdet
O. Team et al., “Openpcdet: An open-source toolbox for 3d object detection from point clouds,” OD Team, 2020.

yan2018second
Y. Yan, Y. Mao, and B. Li, “Second: Sparsely embedded convolutional detection,” Sensors, vol. 18, no. 10, p. 3337, 2018.

xu2021rpfa
B. Xu, X. Zhang, L. Wang, X. Hu, Z. Li, S. Pan, J. Li, and Y. Deng, “Rpfa-net: A 4d radar pillar feature attention network for 3d object detection,” in 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3061–3066, IEEE, 2021.

yin2021center
T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detection and tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11784–11793, 2021.

liu2023bevfusion
Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. L. Rus, and S. Han, “Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation,” in 2023 IEEE international conference on robotics and automation (ICRA), pp. 2774–2781, IEEE, 2023.

zheng2023rcfusion
L. Zheng, S. Li, B. Tan, L. Yang, S. Chen, L. Huang, J. Bai, X. Zhu, and Z. Ma, “Rcfusion: Fusing 4d radar and camera with bird’s-eye view features for 3d object detection,” IEEE Transactions on Instrumentation and Measurement, 2023.

xiong2023lxl
W. Xiong, J. Liu, T. Huang, Q.-L. Han, Y. Xia, and B. Zhu, “Lxl: Lidar excluded lean 3d object detection with 4d imaging radar and camera fusion,” IEEE Transactions on Intelligent Vehicles, 2023.









 Paper Title: Image Super-Resolution with Taylor Expansion Approximation and Large Field Reception.


























Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals
Image Super-Resolution with Taylor Expansion Approximation and Large Field Reception
    Jiancong Feng, Yuan-Gen Wang, Mingjie Li, and Fengchuang Xing




Feng, J., Wang, Y.-G., Li, M., and Xing F. are with the School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou 510006, China (e-mail: fengjiancong@e.gzhu.edu.cn; wangyg@gzhu.edu.cn; limingjie@gzhu.edu.cn; xfchuang@e.gzhu.edu.cn). 




    August 5, 2024
===========================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Self-similarity techniques are booming in blind super-resolution (SR) due to accurate estimation of the degradation types involved in low-resolution images. However, high-dimensional matrix multiplication within self-similarity computation prohibitively consumes massive computational costs. We find that the high-dimensional attention map is derived from the matrix multiplication between Query and Key, followed by a softmax function. This softmax makes the matrix multiplication between Query and Key inseparable, posing a great challenge in simplifying computational complexity. To address this issue, we first propose a second-order Taylor expansion approximation (STEA)  to separate the matrix multiplication of Query and Key, resulting in the complexity reduction from 𝒪(N^2) to 𝒪(N). Then, we design a multi-scale large field reception (MLFR) to compensate for the performance degradation caused by STEA. Finally, we apply these two core designs to laboratory and real-world scenarios by constructing LabNet and RealNet, respectively. Extensive experimental results tested on five synthetic datasets demonstrate that our LabNet sets a new benchmark in qualitative and quantitative evaluations. Tested on the RealWorld38 dataset, our RealNet achieves superior visual quality over existing methods. Ablation studies further verify the contributions of STEA and MLFR towards both LabNet and RealNet frameworks.





Blind Super-Resolution, Self-similarity, Taylor Expansion, High-dimensional matrix multiplication 










§ INTRODUCTION



Single-Image Super-Resolution (SISR) <cit.> has obtained widespread application in various fields, such as remote sensing imagery, security surveillance, and medical imaging. SISR methods are generally divided into two categories: non-blind SR and blind SR. The non-blind SR <cit.> assumes that the degradation type is known, which degradation function is typically represented as:

    𝐱_lr = 𝐱_hr↓_s,

where 𝐱_hr and 𝐱_lr denote the HR and LR images respectively, and ↓_s denotes the bicubic downsampling operation with a scale factor of s. However, the non-blind SR represented by Eq. (<ref>) fails to accurately approximate the real-world degradation due to the inaccessibility of the reference image.



Subsequently, researchers have developed blind SR methods, which assume that the LR image is obtained from the HR image through a degradation function with a complex and unknown degradation type, and this degradation function is typically assumed to follow the equation:

    𝐱_lr = (𝐱_hr⊗𝐊𝐞𝐫𝐧𝐞𝐥) ↓_s + N,

where ⊗ denotes the convolutional operation, 𝐊𝐞𝐫𝐧𝐞𝐥 represents the unknown blur kernel, ↓_s signifies the downsampling operation with a scale factor of s, and N accounts for sophisticated noise. According to Eq. (<ref>), we can clearly see that a single HR image corresponds to infinitely many possible LR images, which is typically an ill-posed problem. Nonetheless, a multitude of blind SR methods has been proposed to overcome this problem. They can be broadly divided into two main types: methods without blur kernel estimation <cit.> and methods with blur kernel estimation <cit.>.

As for methods without blur kernel estimation, SRMD <cit.> is the first to combine the preset blur kernel, noise, and LR image as the input to the SR network. After that, <cit.> incorporated Maximum A Posteriori Probability into iterative optimization. For these methods, the selection of parameters for the preset blur kernel is crucial for achieving effective results in practical applications. However, it is challenging to manually choose appropriate preset blur kernel parameters in the real world. To overcome this challenge, researchers have developed some methods to estimate the blur kernel. IKC <cit.> proposed to iteratively estimate the blur kernel, which divides the SR reconstruction process into two steps: 1) estimating the blur kernel from the LR image and 2) combining the estimated blur kernel and the LR image to reconstruct the SR image. However, the independence between the training of the blur kernel estimation model and the SR model leads to incompatibility issues. To address this issue, DANv1 <cit.> proposed an alternating optimization strategy between the blur kernel estimation and SR models, which makes the two models form a closed loop that can be trained in an end-to-end manner. Although DANv1 has achieved impressive performance in blind SR tasks, NLCUNet <cit.> has made significant advancements by incorporating the Non-Local Sparse Attention (NLSA) <cit.> module. It screens global features in the feature map by calculating self-similarities, enabling implicit estimation of the blur kernel. Therefore, the self-similarity technique has received the widest application in blind SR tasks. However, blind SR models that incorporate the calculation of self-similarity suffer from high computational complexity, making them unsuitable for resource-constrained devices. It is worth noting that the commonly used module for calculating self-similarity is Non-Local Attention (NLA) <cit.>.

In this paper, we go deep into the execution process of NLA. We find that during the self-similarity calculation, a matrix multiplication operation between the high-dimensional attention map and the Value is required, leading to high computational complexity. The attention map is obtained by multiplying the Query matrix with the Key matrix. Then the softmax function is further performed, which hinders the separation of Query and Key. Based on this finding, we propose a second-order Taylor expansion approximation (STEA) to significantly reduce the computational complexity. Compared to self-similarity calculations, STEA loses the ability to filter global features. Hence, we design a multi-scale large field reception (MLFR) to compensate for the performance loss caused by STEA. Finally, we apply our two core designs to laboratory and real-world scenarios by constructing LabNet and RealNet, respectively. Extensive experiments validate the effectiveness of the proposed method. The major contributions of this paper are summarized as follows.

    
  * We propose the second-order Taylor expansion approximation (STEA) strategy, which significantly reduces the computational complexity in self-similarity-based blind SR tasks.
    
  * We design a multi-scale large field reception (MLFR) to compensate for the performance loss caused by STEA, showcasing competitive performance over the state-of-the-art.
    
  * We construct the LabNet and the RealNet by integrating our two core designs (STEA and MLFR) to respectively verify the effectiveness of our designs in the laboratory and real-world scenarios. Extensive experimental results demonstrate that our LabNet achieves state-of-the-art performance while maintaining linear computational complexity, and our RealNet effectively handles various real-world degradations and achieves superior performance in diverse blind SR scenarios.






The structure of this paper is organized as follows: Section II provides a comprehensive overview of the proposed methodology. Specifically, Section II-A presents a detailed explanation of the intricate aspects of STEA. Section II-B elucidates the specific details of MLFR. Section II-C outlines the implementation specifics of LabNet, while Section II-D delves into the implementation details of RealNet. Subsequently, Section III encompasses the dataset and evaluation metrics (Section III-A), implementation details of LabNet (Section III-B), implementation details of RealNet (Section III-C), comparison with state-of-the-art techniques (Section III-D), ablation studies and analysis of LabNet (Section III-E), ablation study and analysis of RealNet (Section III-F) and adjustable parameter details in RealNet (Section III-G). Section IV concludes the paper.




§ METHODOLOGY

In this section, we provide a detailed explanation of the STEA and MLFR algorithms, as well as their deployment to the LabNet (testing for laboratory settings) and RealNet (testing for real-world scenarios).











 §.§ Second-Order Taylor Expansion Approximation (STEA)

The self-similarity has been widely applied to the blind SR task, which is commonly implemented by a Non-Local Attention (NLA) <cit.>. However, we found that the NLA brings extremely high computational complexity. This mainly arises from the matrix multiplication between the high-dimensional Attention Map and the V matrix, as highlighted by the red dashed box of Fig. <ref>. Without loss of generality, Q, K, V ∈ℛ^N× d denote query, key, and value matrix, respectively. Then, we know the computational complexity of NLA is 𝒪(N^2× d), where N and d represent the dimension of the feature map and the number of channels, respectively. In this part, we propose an STEA algorithm to significantly reduce the computational complexity of NLA. To this end, we first analyze the execution process of the NLA module. As shown in Fig. <ref>, the Attention Map is derived from the matrix multiplication of Q and K^T. Due to the limitation imposed by the softmax operation, the matrix multiplication between Q and K^T cannot be separated, resulting in a 𝒪(N^2) complexity. To address this challenge, we propose a Second-Order Taylor Expansion to approximate the NLA module. Here, we mathematically rewrite the execution process of the NLA as follows:

    Q = XW_q, K = XW_k, V = XW_v,

where W_q, W_k and W_v stand for the parameters of three 1× 1 convolutions, respectively. Q, K and V represent the Query, Key and Value, respectively. X denotes the input feature map. Next, we perform the matrix multiplication between Q and K^T, followed by the softmax operation, to obtain the Attention Map:

    AttMap = SoftMax(QK^T),

where AttMap represents the Attention Map, and SoftMax(·) denotes the softmax function. By substituting Eq. (<ref>) to Eq. (<ref>), we have

    AttMap = SoftMax(XW_qW_k^TX^T).

We denote W_qk = W_qW_k^T, resulting in the below expression:

    AttMap = SoftMax(XW_qkX^T).

In the original NLA module, the multiplication between Q and K^T followed by the softmax operation gives rise to softmax attention. Although softmax attention can effectively enhance the performance of the SR network by making the matrix full-rank, it introduces high computational complexity since such a high-dimensional Attention Map (AttMap) must be multiplied with V in the subsequent computation. For that, we propose to simplify the softmax function as an exponential form:

    AttMap = SoftMax(QK^T) = 1/k× e^QK^T = 1/k× e^XW_qkX^T,

where k is a learnable parameter used to approximate the denominator of softmax. By now, we can see that the operation between QK^T and V is still nonlinear due to the presence of an exponential function, which disables us from separating Q and K from QK^T. In subsequent multiplication by V, the basic operation of the matrix cannot be flexibly used, failing to reduce the high computational complexity. For this purpose, we exploit a second-order Taylor expansion as a substitute for the exponential function, achieving a good trade-off between



efficiency and accuracy. 

We know that in principle, the more remainder terms kept in the Taylor expansion of the exponential function, the more the expansion approximates the raw function. According to our empirical results and considering a good trade-off between accuracy and complexity, we adopt the second-order Taylor expansion on Eq. (<ref>) as follows:

    AttMap = 1/k× e^XW_qkX^T
     = 1/k× (I_n + XW_qkX^T + (XW_qkX^T)^2/2! + O(XW_qkX^T)),

where I_n stands for identity matrix and O(·) stands for the high-order remainder of the Taylor expansion. By ignoring the constant and remainder terms of Eq.(<ref>), we achieve:

    AttMap= 1/k× (I_n + XW_qkX^T + 
     XW_qkX^TXW_qkX^T).

Then, we multiply AttMap by V and refer to Eq.(<ref>) to obtain:

    Output = AttMap× V = 1/k× (XW_v + 
     XW_qkX^TXW_v + XW_qkX^TXW_qkX^TXW_v).

Eq. (<ref>) clearly indicates that the constraint of the softmax function has been removed, and therefore we can apply the associative property of the matrix operation in Eq. (<ref>), which allows us to calculate X^TX first. Considering that X^TX is a symmetric matrix, we can transform X^TX into a diagonalization expression as follows:  

    X^TX = ZBZ^T,

where B is a diagonal matrix with its diagonal elements being the eigenvalues of X^TX, and Z is an orthogonal matrix whose columns are the eigenvectors of X^TX corresponding to the eigenvalues. By substituting Eq. (<ref>) into Eq. (<ref>), we obtain:

    Output = 1/k× (XW_v + XW_qkZBZ^TW_v + 
     XW_qkZBZ^TW_qkZBZ^TW_v).

According to Eq. (<ref>), we can see that obtaining B through the matrix eigen-decomposition operation is computationally expensive. If we randomly initialize a learnable diagonal matrix B to avoid the high computational complexity from the eigen-decomposition operation of X^TX. Unfortunately, this strategy makes the matrix B independent of the input feature maps (X). Besides, in the random initialization, the matrix B is required to be a fixed size, making X have to be the fixed size. This disables the input LR image to be an arbitrary size. To reduce the complexity of the matrix eigen-decomposition and implement the SR task on the LR images with an arbitrary size, we design two separative eigenvalue extraction modules to respectively obtain two diagonal matrices B_1 and B_2, as shown in Fig. <ref>. Our design avoids randomly initializing a single B, and meanwhile enhances the representation ability of the model. Therefore, we can obtain a learning-based approximation expression as follows: 

    Output = 1/k× (XW_v + XW_qkZB_1Z^TW_v + 
     XW_qkZB_1Z^TW_qkZB_2Z^TW_v).

It is observed from Fig. <ref> that our design modules can dynamically extract global information from the matrices, which can well match the matrix eigen-decomposition where global information is exploited dynamically. 

Similarly, we adopt learnable parameters to substitute Z, solving the high computational complexity of the eigen-decomposition of X^TX. Next, we let W_1 = W_qkZ, W_2 = Z^TW_qkZ, and W_3 = Z^TW_v. Hence, we obtain:

    Output = 1/k× (XW_v + XW_1B_1W_3 + XW_1B_1W_2B_2W_3).

By now, we equate the softmax function to an exponential function and perform a Taylor expansion, making the operations of Q, K, and V more flexible. It is worth noting that the softmax function can transform the resulting matrix derived from Q and K^T into a full-rank matrix, while the matrix obtained by direct multiplication of Q and K^T is not a full-rank matrix, which significantly reduces the feature diversity, in turn, leads to a degradation of the model's performance <cit.>. Therefore, as done in <cit.>, we perform a depthwise convolution (DWC) operation on XW_v of Eq. (<ref>) to compensate for the performance degradation. Finally, we obtain:

    FinalOutput = 1/k× [DWC(XW_v) + 
     XW_1B_1W_3 + XW_1B_1W_2B_2W_3].

As shown in Fig. <ref>, we design the same network block to learn W_1, W_2, and W_3, which first reduces the number of channels to a quarter and then restores it to the original channel number. Meanwhile, an efficient Conv_1×1 operation is adopted to approximate W_v. 

As shown in Fig. <ref>, N and d are a multiplication of the width and height of a feature map and the number of channels. In general, N is far larger than d. According to Eq. (<ref>), our STEA strategy reduces the complexity of the original NLA from 𝒪(N^2× d) to 𝒪(N× d^2). The subsequent experimental results validate that our STEA is more efficient compared to the NLA module, however, exhibits a slight performance degradation due to the ignoring of remaining terms of Taylor extension and the use of the learning-based matrix. To compensate for such performance degradation, we design a Multi-Scale Large Field Reception (MLFR), which will be detailedly described in the following.





 §.§ Multi-Scale Large Field Reception (MLFR)

As mentioned in Section II-A, compared to the NLA module, our STEA achieves higher efficiency but sacrifices



the ability to extract global features from the feature maps. To this end, we design a Multi-Scale Large Field Reception (MLFR) strategy to obtain a large receptive field, as shown in Fig. <ref>. For the unfilled parts of the dilated convolution, we use normal convolution to fill in the dilation. Specifically, we process the input feature map X through three separate branches. In the first branch, the input is processed by a dilated convolution with an expansion rate of 6 and a depth-wise convolution with a kernel size of 7×7, respectively. The output results are concatenated along the channel dimension. Simultaneously, in the second branch, the input is processed by a dilated convolution with an expansion rate of 4 and a depth-wise convolution with a kernel size of 5×5, respectively. Similarly, the output results are concatenated along the channel dimension. In the last branch, the input is processed by a dilated convolution with an expansion rate of 2 and a depth-wise convolution with a kernel size of 3×3, respectively. Again, the output results are concatenated along the channel dimension. We represent the above execution process using the following equation: 

    Output_(i,j)=Concat(DDWC(X)_dr=i, 
    DWC(X)_ks=j)_i=2,4,6
    j=3,5,7,

where Concat(·,·) stands for concatenate operation, DDWC(·) stands for dilation convolution operation on the input, dr is the dilation rate, and DWC(·) stands for depth-wise convolution operation on the input, ks is the convolution kernel size. Furthermore, each value of i corresponds uniquely to one value of j. Subsequently, we employ 1×1 convolutions to perform channel-wise feature fusion on the outputs of the three branches, effectively reducing the dimension of the channels:

    Fusion_(i,j) = Conv_1×1(Output_(i,j))_i=2,4,6
    j=3,5,7,

where Fusion_(i,j) represents the output obtained after channel-wise feature fusion of the input, and Conv_1×1(·) denotes the 1×1 operation performed on the input. Similarly, each value of i corresponds uniquely to one value of j. In addition, the input X undergoes a 1×1 bypass convolution in addition to the aforementioned three branches:

    Bypass = Conv_1×1(X).

The purpose of the 1×1 convolution in Eq. (<ref>) is to perform subtle adjustments to the channels of the input X. Finally, we concatenate the outputs of the three branches with the output of the bypass 1×1 convolution along the channel dimension and perform channel-wise feature fusion using a 1×1 convolution to reduce the channel dimension:

    FinalOutput=Conv_1×1(Concat(Fusion_(i,j), 
     Bypass)_i=2,4,6
    j=3,5,7).

In summary, we utilize multiple dilated convolutions with different scales to extract features with various receptive fields. For the unpadded regions in dilated convolutions, we employ depth-wise convolutions to fill in the dilation. Additionally, we utilize 1×1 convolutions to perform subtle adjustments on the input as bypass convolutions. In the final step of feature fusion, we concatenate all the branches along the channel dimension and then employ a 1×1 convolution for channel-wise feature fusion, resulting in the final output as shown in Eq. (<ref>).







 §.§ Application to Laboratory Scenario (LabNet)

To show the effectiveness of our STEA and MLFR designs in a laboratory scenario, we construct a novel network termed LabNet for testing. As shown in Fig. <ref>, our LabNet is an end-to-end blind SR framework composed of three parts: shallow feature extraction, deep feature extraction, and HR image reconstruction. For the shallow feature extraction part, we employ a 3×3 convolution to extract shallow features from the input LR image. Subsequently, the extracted shallow features are fed into the UNet for deep feature extraction. Each LSTEA module in the UNet consists of multiple LSTEA blocks. When the input features pass through an LSTEA block, they undergo a channel-wise LayerNorm operation, followed by two branches. One branch utilizes multiple depth-wise convolutions for local feature extraction, while the other branch employs the STEA and MLFR modules for large-scale feature extraction. The channel features are then fused using a 1×1 convolution and filtered using the channel attention module. Subsequently, the features pass through the GDFN <cit.> module, which performs a nonlinear mapping operation on the input to enhance network performance. Finally, for the HR image reconstruction part, the channel dimension is adjusted using a 3×3 convolution. The H and W dimensions are then expanded to the target size using a pixel shuffle operation, and a final refinement is performed using a 3×3 convolution to obtain the ultimate output results.



 §.§ Application to Real-world Scenario (RealNet)

To show the effectiveness of our STEA and MLFR designs in real-world scenarios, we construct a novel network termed RealNet for testing. As illustrated in Fig. <ref>, our RealNet comprises three key components: shallow feature extraction, deep feature extraction, and HR image reconstruction. Firstly, we perform shallow feature extraction on the input LR image using two separate 3×3 convolutions. Subsequently, the extracted features are separately subjected to deep feature extraction in the denoising branch and the deblurring branch. Both the Denoising and Deblurring modules are composed of multiple LSTEA blocks. The features extracted from the Denoising module and the Deblurring module are then fused in proportion using Adapter modules. Finally, we employ a Fusion module for HR image reconstruction.

Adapter Module. For images contaminated by noise, there are numerous detailed features within the image that can be learned by the deblurring branch. Similarly, for blurry images, there are many low-frequency features within the image that can be learned by the denoising branch. Therefore, as depicted in Fig. <ref>, we propose the Adapter module. The parameters α and β control the proportions of denoising and deblurring, respectively. Analyzing the denoising branch, we take the features extracted by the deblurring branch in the previous step, apply a 3×3 convolution, and perform a channel-wise concatenation with the features extracted by the denoising branch. Subsequently, a 1×1 convolution is employed for channel-wise feature fusion, followed by a channel attention module for channel-wise feature selection. Similarly, a similar processing approach is utilized for the deblurring branch.

Fusion Module. To adjust the proportions of the outputs from the final denoising module and deblurring module, we also utilize two adjustable parameters (α and β). After adjusting the two sets of output feature maps, we perform a channel-wise concatenation and then apply a 1×1 convolution for channel information fusion and adjustment of channel dimensions. Finally, the output feature maps are resized to match the target size using a pixel shuffle operation and fine-tuned with a 3×3 convolution.

RealNet-GAN. To achieve better visual results, we employ a GAN training strategy for the proposed RealNet. Specifically, we use RealNet as the generator and utilize a U-Net as the discriminator <cit.>. The training process consists of two parts. Firstly, we train a PSNR-oriented model using the ℓ_1 loss, which is then used as the generator in the GAN training strategy. We then train a GAN-oriented model using a combination of ℓ_1 loss <cit.>, perceptual loss <cit.>, and adversarial loss <cit.>. The overall loss function of our network is

    ℒ_entire = ℒ_1 + ℒ_perc + 0.1 ×ℒ_adv,

where ℒ_1, ℒ_perc, and ℒ_adv represent the ℓ_1 loss, perceptual loss, and adversarial loss, respectively. According to our empirical study, their weights are set to be 1, 1, and 0.1, respectively. 
We utilize the {conv1,..., conv5} feature maps before activation in the pre-trained VGG19 <cit.> model (with weights {0.1, 0.1, 1, 1, 1}) as the perceptual loss.







§ EXPERIMENTS





 §.§ Datasets and Evaluation Metrics

We train all our proposed SR models using the DF2K dataset (DIV2K <cit.> + Flickr2K <cit.>). There are 3450 2K-resolution images in the dataset. For the quantitative evaluation, we conducted tests on five standard super-resolution image benchmark datasets: Set5 <cit.>, Set14 <cit.>, BSD100 <cit.>, Urban100 <cit.>, and Manga109 <cit.>. To conduct a qualitative evaluation, we compared the visual results of our approach with previous methods on the RealWorld38 <cit.> dataset. The quantitative performance metrics we use for comparison are the Peak Signal-to-Noise Ratio (PSNR) <cit.> and Structural Similarity (SSIM) <cit.>.





































 §.§ Implementation Details of LabNet

Our LabNet is built on the foundation of the LSTEA module, forming a U-Net. Each LSTEA module consists of multiple LSTEA blocks, as shown in Fig. <ref>, the number of LSTEA blocks in each LSTEA module from left to right is: {6, 8, 10, 6, 8, 8}. During the training process, we set the size of the input image to 64 × 64, and we use the Mean Absolute Error (MAE) as the loss function. The total iterations are 1.2×10^6. The initial learning rate is set to 4×10^-4 and is halved every 3×10^5 iterations. We use Adam <cit.> as the optimizer with β_1 = 0.9 and β_2 = 0.99. All models are trained on Tesla T4 GPUs.

Data Preprocessing of LabNet. For the training dataset, a blur operation with a blur kernel size of 21 × 21 is applied. For different scaling factors (×2, ×3, and ×4), the variance of the blur kernel is uniformly sampled during the training process from the ranges [0.2, 2.0], [0.2, 3.0], and [0.2, 4.0], respectively. For the test dataset, we uniformly select 8 blur kernels from the ranges [0.80, 1.60], [1.35, 2.40], and [1.8, 3.2] for different scaling factors (×2, ×3, and ×4), respectively.



 §.§ Implementation Details of RealNet

Our RealNet is constructed by two branches: denoising and deblurring. As illustrated in Fig. <ref>, both the denoising branch and the deblurring branch consist of 4 modules, each composed of 5 LSTEA modules. The entire network is trained for 30 epochs. The initial learning rate is set to 1× 10^-3 and is halved every 15 epochs. We use Adam as the optimizer with β_1 = 0.9 and β_2 = 0.99. All models are trained on Tesla T4 GPUs.

Data Preprocessing of RealNet. As shown in Fig. <ref>, we first independently and in parallel apply second-order noise and second-order blur processing to the HR image, resulting in two corresponding LR images. Then, these two LR images are passed through the denoising and deblurring branches of RealNet, respectively. Finally, a fusion operation is performed to obtain the restored SR image.






 §.§ Comparisons with State-of-the-art

Quantitative Evaluations in Lab Scenarios. From Table <ref>, it can be observed that our LabNet significantly outperforms other state-of-the-art deep SR models on nearly all benchmarks and scaling factors. For instance, compared to NLCUNet <cit.> with a scale factor of ×2, our LabNet achieves performance improvements of 0.03 dB, 0.05 dB, 0.11 dB, and 0.02 dB on the Set14, BSD100, Urban100, and Manga109 datasets, respectively. Similarly, compared to NLCUNet with a scale factor of ×4, our LabNet achieves performance improvements of 0.04 dB, 0.02 dB, and 0.11 dB on the Set14, BSD100, and Manga109 datasets, respectively. Our LabNet consistently outperforms DCLS <cit.> across all scale factors. Additionally, as shown in Table <ref>, when the input image size is (1,3,64,64) and the scaling factor is ×4, our LabNet achieves an inference time of 1.18 seconds on a GPU, which is 1.05 seconds faster than NLCUNet but slightly slower than DCLS. Furthermore, our LabNet has significantly fewer GFLOPs compared to NLCUNet and is comparable to DCLS. In summary, our LabNet not only achieves comparable performance to NLCUNet but also exhibits higher efficiency. This can be attributed to the utilization of LSTEA in our LabNet, which combines STEA and MLFR to capture multi-scale features with a large receptive field while maintaining linear complexity.


Qualitative Evaluations in Lab Scenarios. As shown in Fig. <ref>, by comparing the reconstruction result of the 'zebra' in Fig. <ref>, we can observe that the black and white stripes on the legs of the zebra generated by our LabNet are close to the GT image. Furthermore, by comparing the reconstruction result of 'YamatoNoHane', we can observe that our LabNet performs better in recovering the textual content in the image compared to other methods.


Quantitative Evaluation in Real-World Scenarios. As commonly observed, GAN-based SR methods often generate lower values in terms of PSNR and SSIM metrics while producing visually captivating results. As shown in Table <ref>, Real-ESRGAN and BSRGAN perform significantly lower than non-GAN-based SR methods on the five benchmark test datasets. However, Fig. <ref> reveals the impressive visual outcomes achieved by Real-ESRGAN and BSRGAN. Additionally, due to the absence of corresponding GT images for the degraded images in the RealWorld38 dataset, we only conducted visual comparisons between our RealNet and other blind SR methods.


Qualitative Evaluation in Real-World Scenarios. The experimental results are presented in Fig. <ref>. We evaluated the generalization capability of our model using the RealWorld38 dataset, which consists of real-world images. Our RealNet outperforms previous methods for reconstructing LR images in real-world scenarios. Among all the methods, the non-deep learning bicubic interpolation method performs the worst, followed by DANv2, DCLS, and NLCUNet. These methods only employ Gaussian blur and downsampling during training, resulting in visually unsatisfactory outcomes. BSRGAN <cit.> and Real-ESRGAN achieve favorable visual results due to their training on datasets with complex degradations. However, upon comparing the 'OST_009' image in Fig. <ref>, we observe that the result obtained by BSRGAN exhibits a significant color deviation from the LR input, while the result produced by Real-ESRGAN evokes a strong sense of discordance. In contrast, our RealNet reconstruction result closely resembles the color of the input LR image and demonstrates improved overall coherence.





 §.§ Ablation Study and Analysis of LabNet



  §.§.§ Impact of STEA

Table <ref> showcases the effect of employing first-order Taylor expansion approximation (FTEA) and second-order Taylor expansion approximation (STEA) on the performance of blind SR tasks.
By comparing the PSNR and SSIM values of the first and second rows in Table <ref>, we observed an improvement in performance across all test datasets when DWC(·) was applied to the Value. Furthermore, comparing the PSNR and SSIM values of the second and third rows in Table <ref>, we found that utilizing STEA for the exponential function outperformed FTEA across all test datasets. Particularly on the Urban100 dataset, the PSNR was higher by 0.11 dB when using a STEA compared to a FTEA. Furthermore, we found that when the order of Taylor expansion is greater than 2, the performance does not show a significant improvement. 

Efficiency Analysis. As shown in Table <ref>, when applying DWC(·) to Value, there is a slight increase in the number of parameters and a slight rise in MFLOPs. By comparing rows 2 to 4 in Table <ref>, as the order of Taylor expansion increases, the third-order Taylor expansion approximation (TTEA) parameter count and MFLOPs are greater than STEA, which are greater than FTEA. Ultimately, we used STEA to build our LabNet and RealNet.



  §.§.§ Impact of MLFR

Table <ref> highlights the influence of incorporating multi-scale dilated convolutions in MLFR on blind SR performance. By comparing the PSNR and SSIM values of the first to third rows in Table <ref>, we found that in MLFR, when utilizing dilated convolutions with dilations of 2, 4, and 6 simultaneously, the performance was the best across all test datasets. Particularly on the Urban100 dataset, the PSNR was 0.12 dB higher when using dilated convolutions with dilations of 2, 4, and 6 compared to dilated convolutions with dilations of 4 and 6. 

Efficiency Analysis. As shown in Table <ref>, in MLFR, the maximum number of parameters and MFLOPs are achieved when utilizing dilated convolutions with Dilated = 6, 4, and 2 simultaneously.




  §.§.§ Impact of STEA and MLFR

As shown in Table <ref>, LabNet achieves the best performance on all five benchmarks when both STEA and MLFR are used simultaneously.



 §.§ Ablation Study and Analysis of RealNet

The experimental results from Table <ref> highlight the following insights: Our RealNet without NLA has the fastest inference speed, processing a 64 × 64 input image in just 0.40 seconds. In contrast, Our RealNet with NLA and NLSA have much higher computational costs, with inference times of 4.21 seconds and 5.15 seconds, respectively. Our RealNet with the STEA, achieves high efficiency by processing an input image in only 0.48 seconds. Our RealNet with the STEA and the MLFR strikes a favorable balance between efficiency and visual result, processing an input image in 0.53 seconds.



 §.§ Details of adjustable parameters in RealNet

Adjusting a single parameter: Users can adjust the values of the denoising and deblurring parameters in RealNet to achieve the desired effects. In the visualization shown in Fig. <ref>, Increasing α_1 slightly enhances image details, but excessive increase may lead to distorted edges. Gradually increasing α_3 improves the distinction between objects and the background, but setting it too high can negatively impact the image. α_2 progressively enhances image visualization as it increases, but setting it to extremely large values can damage the image. Careful adjustment of α_4 is necessary to prevent color distortion, as it is related to the deep layer. It is advisable to avoid setting β_1 to β_4 to excessively large values to prevent color distortion in the resulting image. 

Joint modification of multiple parameters: Users can still fine-tune the denoising and deblurring ratios (i.e., α_i and β_i) according to their preferences, even after the training process. In the visualization shown in Fig. <ref>, the second and third rows display the restored results using different ratios for denoising and deblurring. This demonstrates that RealNet is capable of restoring rich texture information from LR images, catering to different user preferences.



§ CONCLUSION

In this paper, we presented a second-order Taylor expansion approximation (STEA) to handle the high computational complexity of self-similarity techniques. Then, we designed a multi-scale large-field reception (MLFR) to compensate for the super-resolution (SR) performance loss caused by STEA. To confirm the superiority of our designed STEA and MLFR, we respectively constructed the LabNet for testing in laboratory scenarios and the RealNet for testing in real-world scenarios. Extensive experimental results demonstrated the effectiveness of our method in blind SR tasks. The ablation study further validates their contribution to the overall framework.
IEEEbib



























 Paper Title: From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification.








Journal of  Class Files, Vol. 14, No. 8, December 2023
Shell et al.: A Sample Article Using IEEEtran.cls for IEEE Journals




From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification
    

        
        Fanzhi Jiang, 
        Su Yang1, 
        Mark W. Jones, 
        Liumei Zhang1
        
        
        1Corresponding author
        
This paper was produced by the IEEE Publication Technology Group. They are in Piscataway, NJ.
Manuscript received April 20, 2024; revised August 16, 2024.
        
    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 

  Text-based person re-identification (Re-ID) is a challenging topic in the field of complex multimodal analysis, its ultimate aim is to recognize specific pedestrians by scrutinizing attributes/natural language descriptions. Despite the wide range of applicable areas such as security surveillance, video retrieval, person tracking, and social media analytics, there is a notable absence of comprehensive reviews dedicated to summarizing the text-based person Re-ID from a technical perspective. To address this gap, we propose to introduce a taxonomy spanning Evaluation, Strategy, Architecture, and Optimization dimensions, providing a comprehensive survey of the text-based person Re-ID task. We start by laying the groundwork for text-based person Re-ID, elucidating fundamental concepts related to attribute/natural language-based identification. Then a thorough examination of existing benchmark datasets and metrics is presented. Subsequently, we further delve into prevalent feature extraction strategies employed in text-based person Re-ID research, followed by a concise summary of common network architectures within the domain. Prevalent loss functions utilized for model optimization and modality alignment in text-based person Re-ID are also scrutinized. To conclude, we offer a concise summary of our findings, pinpointing challenges in text-based person Re-ID. In response to these challenges, we outline potential avenues for future open-set text-based person Re-ID and present a baseline architecture for text-based pedestrian image generation-guided re-identification (TBPGR).



Person Re-Identification, Text, Natural Language, Attributes, Diffusion Model.





§ INTRODUCTION

THE demands in public safety and the subsequently installed surveillance networks make the manual tracking and identifying individuals increasingly challenging. 
Automatic person re-identification (Re-ID) has emerged as a popular research area in computer vision to address this issue. Also known as person search  <cit.>, person Re-ID specifically refers to recognizing and tracking specific individuals using non-overlapping images and videos captured by cameras, determining whether a specific query person appears persistently or momentarily across different times and locations  <cit.>. 
It finds important applications in scenarios like searching for characters in movies, locating missing children, and tracking criminals  <cit.>. 
The source data collection for these applications is usually from CCTV footage on the street. Given the constraints of the equipment and the environment, a number of challenges arise. 
These challenges include lighting variations, occlusions, background changes, pose variations, and differences in camera resolutions. 
A typical person re-identification system involves conducting image queries and searching for corresponding individuals in a gallery of images or a video pool. 
It involves extracting features directly from pedestrian appearance images and performing direct matching, then ranking against existing appearance images in the gallery. 
These scenarios assume that visual examples of individual identities are always available as queries. However, some special cases where the visualization samples of personal identity are not available, we perform retrieval based on textual descriptions only, which is called text-based person re-identification, as shown in Figure <ref>.






 §.§ Attribute-based Person Re-ID

Text-based person re-identification is a special form of person re-identification that, instead of using image data, relies on descriptions to reflect a person's appearance  <cit.>. In the task of text-based person Re-ID, a natural language description and an image are provided as input, and the output is the identification of the person matching the description  <cit.>. There has been a desire to perform high-level semantic person search using free-form natural language descriptions.  However, early research initially started with the attribute-based person re-identification <cit.>. 
Attribute-based methods (Pedestrian Attributes Recognition, PAR) stem from the high correlation between attributes and pedestrian images. Pedestrian attributes such as gender, age, clothing type, clothing color, etc. are arranged and combined to make the pedestrian identity appear distinguishable <cit.>.
Extracting attribute information from pedestrian images can be achieved by using pre-trained classifiers or attribute detectors. These classifiers or detectors can recognize and locate different attributes(body part) in the images. These extracted pedestrian attributes information needs to be encoded into a feature vectors pattern matching. This often involves transforming discrete attributes into continuous feature representations, common methods include one-hot encoding and embedding encoding <cit.>. During the recognition process, matching is conducted by comparing the attribute feature vectors of query images and gallery images. Common matching methods include computing distances or similarities between feature vectors (such as Euclidean distance, cosine similarity, etc.). Finally, based on the results of attribute matching, the final recognition result is determined through fusion of matching scores from different attributes or by adopting decision strategies.
The attribute-based person Re-ID approach improves recognition accuracy and retrieval interpretability. This is because these attributes can be considered as high-level semantic information that is robust to viewpoint changes and different viewing conditions <cit.>.

Previous work on attribute-based person retrieval has attempted to reduce modality gaps by aligning each person category and its corresponding images in a joint embedding space through modal adversarial training <cit.>, or by enhancing the expressive power of embedding vectors for person categories and images hierarchically <cit.>. While these pioneering studies reveal important but less explored methods for person retrieval, there remains significant room for further improvement. Firstly, due to their adversarial learning strategy, they exhibit instability and high computational cost during training <cit.>. For other, they incur high inference costs due to the need for additional networks to match high-dimensional embedding vectors <cit.>. More importantly, these methods treat person categories as independent labels of person images and overlook their relationships, such as how many attributes differ between them, despite the potential rich supervisory signals for learning better representations of person categories and images. Dong et al.  <cit.> address this issue by capturing rich information of two modalities through hierarchical embedding. However, their model entails high computational complexity as it computes high-dimensional embeddings and deploys additional networks for matching. Yin et al.  <cit.> and Cai et al.  <cit.> learn a joint embedding space where person categories and images are directly matched. To bridge the modality gap, their embedding space is trained in a modal adversarial manner; however, due to the nature of min-max optimization, this often leads to unstable and slow convergence. Furthermore, all these methods suffer from a limitation where person categories are treated as separate labels and the important relationships among them are overlooked. In the joint embedding space of the two modalities, the loss pulls images of the same person category closer to achieve modal alignment. <cit.> introduces a novel loss, Adaptive Semantic Margin Regularizer (ASMR), for learning cross-modal embeddings in the context of attribute-based person retrieval.





 §.§ Natural Language-based Person Re-ID

Attribute-based person retrieval <cit.> provides a method for person re-identification based on free-form descriptions. 
However, there is an awareness that the leverage of attributes to describe a person's appearance could be limited in some circumstances. For instance, the PETA dataset <cit.> defines 61 binary and 4 multi-class person attributes, along with hundreds of phrases used to describe a person's appearance.  In addition, even with a constrained set of attributes, labeling them for large-scale person image datasets is challenging and expensive. Using free-form natural language (NL) descriptions may offer more flexibility and robustness compared to these predefined attributes. 

Attribute-based person retrieval also has limitations in terms of the results robustness. For instance, shuffling and flipping some attributes while using a basic fact set for retrieval can lead to significant performance degradation of the results. Compared to manually assigned sets of attributes, textual descriptions can be seen as an unstructured form of annotation. In general, most attributes fall into the category of noun phrases, and natural language includes higher-level semantic words and complex events than the attributes. In principle, natural language processing (NLP) techniques can facilitate all attributes with any possible words and lengths, capturing unique details through natural language descriptions. Additionally, descriptions can include indicators of certainty or fuzziness to retain as much information as possible, potentially allow sets of multi-class attribute assignments. Both attribute-based and NLP-based person re-identification are referred to as text-based person re-identification in this paper.

Li et al. <cit.> introduced the first dataset for NL-based person Re-ID, evaluated and compared various possible models, and proposed an RNN with a gated neural attention mechanism. Jing et al.  <cit.> proposed to focus on mining fine-grained local parts and performing fine-grained visual-textual matching. These methods can learn distinctive part representations and achieve 81.23% accuracy, but they are troubled by the problem of ambiguous embeddings. Image regions and words are correlated at different semantic granularities <cit.>. The discriminative parts of a person usually appear with different granularities, which was overlooked in the previous methods and ambiguated embeddings. To alleviate this issue, words (based on NL) as guidance have been integrated into the related image regions. However, irrelevant words may mislead the model and even exacerbate the ambiguous embeddings problem. 

Cross-domain embeddings technique is one of the main trends in the Re-ID community. Work proposed in <cit.> added supplementary annotations to extract meaningful image regions independently, these regions, however, are not always available in the real-world scenes. Work in <cit.> proposed to learn cross-modal embeddings to improve matching results using a shared attention mechanism. <cit.> designed a cross-modal projection matching loss and a cross-modal projection classification loss for learning discriminative image-text embeddings. <cit.> proposed a matrix factorization-based dictionary learning algorithm to eliminate the impact of style and pedestrian pose information on cross-domain Re-ID. <cit.> proposed a patch-word matching model and designed an adaptive threshold mechanism in the model, it attempted to establish global and local image-language associations to enhance the semantic consistency between local visual and language features. 

Deep adversarial graph attention convolutional network was also proposed to address the problem of text-based person re-identification<cit.>. Niu et al. <cit.>proposed a Multi-level Image-Text Alignment (MIA) model, which introduced cross-modal adaptive attention in three different granularities, integrating these three similarities to comprehensively judge the affinity of image-sentence pairs and better distinguish interested pedestrians. Zhu et al. <cit.>achieved an environment-person separation (DSSL) method under the mutual exclusion constraint, and also annotated another dataset for person search using language called Real Scene Text-based Person Re-ID (RSTPReID). Wu et al. <cit.> designed a twin task, image coloring and text completion, and explicitly built fine-grained cross-modal associations bidirectionally based on color reasoning, their LapsCore method achieved 63.4% rank@1 retrieval accuracy on the CUHK-PEDES dataset. <cit.> proposed a novel cross-modal matching method named Cross-modal Co-occurrent Attribute Alignment (C2A2), which can better handle noise and significantly improve retrieval performance in person search by language. Relying on matrix factorization to construct visual and textual attribute dictionaries and cross-modal alignment using denoising reconstruction features addresses noise generated by extraneous elements of pedestrians. Zheng et al. <cit.> made the first attempt to perform NL-based person search within unsegmented complete images.



 §.§ Person Re-Identification previous survey

In order to provide researchers with a comprehensive understanding of the current status and research directions in the domain of text-based person re-identification, we conducted an in-depth investigation of text-based person re-identification methodologies and synthesized recent research achievements. Prior to this effort, several researchers have conducted reviews in the field of person re-identification <cit.> and we summarize the primary contributions of these studies in the Table <ref>. 
Among them, certain surveys <cit.> introduced previous challenges in the field of person Re-ID. <cit.> summarized deep learning based pedestrian re-recognition methods. <cit.> investigated the available image/video-based person Re-ID datasets. <cit.> surveyed image-based or video-based pedestrian re-recognition systems. <cit.> proposed metric learning based classification method for pedestrian re-recognition. 
<cit.> presented a pedestrian re-recognition taxonomy based on representation learning. Huang et al.<cit.> provided a visible-infrared cross-modal categorization of pedestrian re-identification studies. Wang et al.<cit.> reviewed cross-modality based person Re-ID research from four aspects, including sketch, text, low resolution (LR) and infrared (IR), as Figure <ref> is introduced later.

However, these surveys exhibit areas for improvement, such as lacking a systematic categorization and analysis of text-based deep learning methods in person re-identification, and omitting a comprehensive discussion of text-based person re-identification. Therefore, in comparison to the aforementioned reviews, this paper delves into an in-depth analysis of recent research in text-based person re-identification from dimensions including datasets, strategies, architectures, and optimizations. We comprehensively review existing deep learning-based approaches and discuss their strengths and limitations.








§ EVALUATION

The evaluation of text-based person Re-ID tasks, typically involves publicly available benchmark datasets and common performance metrics. This provides a standardized testing platform, enabling a quantified and objective comparative assessment of the effectiveness of different experimental algorithms.



 §.§ Datasets

























  §.§.§ attribute-based Person Re-ID Datasets

Attribute-based Person Re-identification (Re-ID) often involves the utilization of semantic attributes that describe appearance characteristics of individuals, such as gender, age, clothing type, clothing texture, and clothing color <cit.>. However, these attributes frequently encompass low-level semantic attributes that can be directly associated with image regions, such as clothing type, clothing texture, and clothing color, as well as high-level semantic attributes that directly correspond to people, such as gender and age. By leveraging these attributes, the negative impact of variations in viewpoint and appearance on person re-identification can be mitigated, resulting in more reliable recognition and tracking performance <cit.>. To facilitate research in Attribute-based Person Re-ID, researchers have initiated the construction of several datasets specifically tailored for this purpose. Notable publicly available datasets in this domain include PETA <cit.>, UAV-Human <cit.>, RAP  <cit.>, RAP2.0 <cit.>, PA-100K <cit.>, Market1501-Attribute <cit.>, and DukeMTMC-Attribute <cit.>. 

These datasets typically consist of a substantial collection of images capturing individuals from surveillance cameras, each accompanied by attribute labels. The dataset creation process generally involves selecting a subset of images from existing Person Re-ID datasets. Subsequently, attribute labels are assigned to each image either manually or through automated annotation. Ultimately, the annotated images along with their corresponding attribute labels are organized into a unified dataset, which serves as a resource for researchers in the field, as shown in Table dataset.



  §.§.§ Natural Language-based Person Re-ID Datasets

Publicly available NL-based person re-identification datasets have significant practical implications for benchmarking, algorithm improvement, security and surveillance applications, privacy protection, and advancing multimodal research. Currently, there are three large publicly available datasets, CUHK-PEDES, RSTPReid, and ICFG-PEDES. The details of these datasets will be described. 



  
CUHK-PEDES
CUHK-PEDES is the first description dataset introduced by The Chinese University of Hong Kong <cit.>, designed for training and evaluating person description and image retrieval tasks in computer vision. The dataset was collected by aggregating person images from five existing and well-known pedestrian re-identification evaluation datasets, namely CUHK03 <cit.>, Market-1501 <cit.>, SSM <cit.>, VIPER <cit.>, CUHK01 <cit.>, and subsequently annotated with descriptive text by crowdsourced workers from Amazon Mechanical Turk (AMT). It consists of over 40,206 images with 80,412 person descriptions, each containing at least 23 words, across 13,003 unique person identities. The training set includes 34,054 images, 11,003 unique persons, and 68,108 textual descriptions. The validation set contains 3,078 images, 1,000 unique persons, and 6,158 textual descriptions, while the test set comprises 3,074 images, 1,000 unique persons, and 6,156 textual descriptions. The individuals in the dataset were captured in street and public places in the Hong Kong area. CUHK-PEDES provides diverse person descriptions encompassing appearance, actions, postures, and interactions, with each person description linked to the corresponding image ID. The release of the CUHK-PEDES dataset promotes interdisciplinary research between computer vision and natural language processing, providing researchers in the fields of person description and image retrieval with the first benchmark dataset.



  
RSTPReid
The RSTPReid dataset is a pedestrian re-identification dataset introduced by Nanjing University of Science and Technology <cit.>, designed for training and evaluating pedestrian re-identification tasks in computer vision. To address the issue in the CUHK-PEDES dataset where each specific pedestrian is captured by the same camera under the same time-space conditions, which does not reflect real-world scenarios, the authors constructed the Real Scenarios Text-based Person Reidentification (RSTPReid) dataset based on MSMT17 <cit.>. The RSTPReid dataset contains 20,505 images of 4,101 individuals captured by 15 independent cameras from different viewpoints, lighting conditions, locations, and weather conditions. Each individual has 5 corresponding images taken by different cameras, and each image is accompanied by 2 text descriptions with no fewer than 23 words. The training set includes 3,701 identities, the validation set has 200 identities, and the test set also consists of 200 identities. The dataset includes a relatively small number of pedestrian identities but covers both indoor and outdoor scenarios. Each pedestrian appears in multiple images with different shooting angles and lighting conditions, making this dataset more challenging and realistic.



  
ICFG-PEDES
The Identity-Centric and Fine-Grained Person Description Dataset (ICFG-PEDES) is a person description dataset introduced by South China University of Technology <cit.>, which also serves as a new benchmark dataset for research in person description and image retrieval domains. Similar to CUHK-PEDES, this dataset contains a large number of person descriptions paired with corresponding image IDs. ICFG-PEDES consists of 54,522 pedestrian images of 4,102 distinct identities, all collected from the MSMT17 database <cit.>. The training set comprises 34,674 image-text pairs of 3,102 pedestrians, while the test set contains 19,848 image-text pairs of 1,000 pedestrians. Each image is associated with only one text description, with an average of 37.2 words per description. Compared to the CUHK-PEDES dataset, ICFG-PEDES emphasizes more on fine-grained person descriptions and reduces some irrelevant action and background information. Additionally, it addresses the issue of consistent backgrounds in the former dataset by emphasizing more on appearance variations. Consequently, the ICFG-PEDES dataset can be employed for more challenging image retrieval and person description tasks.



 §.§ Metrics




  §.§.§ Mean Average Precision(mAP)


For each query i, we define a precision P_i(j) as the proportion of correct matches in the top j matches, and then for each positive instance, we calculate the average of the proportions of all positive instances before this one, which is the Average Precision AP_i:


    AP_i = 1/M_i∑_j=1^M_i P_i(j) · I(rank_i = j).


Where M_i is the number of positive instances for query i, and ∑ is the summation over all positive instances. The symbol I(rank_i = j) usually denotes an indicator function. The indicator function takes the value of 1 under certain conditions, otherwise it takes the value of 0. Specifically:

    I(rank_i = j)
 indicates that the indicator function I(rank_i = j) takes the value of 1 when the first i query's first j result is a correct match, otherwise it takes the value of 0. In other words, it determines whether or not there is a correct match in the ranked position of j.
Then, mAP is the average of the Average Precision of all queries:


    mAP = 1/N∑_i=1^N AP_i.


Where N is the total number of queries, and ∑ is the summation over all queries.



  §.§.§ Rank-N Accuracy

For each query i, we define an indicator function I(rank_i ≤ n), which is 1 if the correct match is within the top n matches, and 0 otherwise. Then, Rank-n accuracy is the average of this function:


    Rank-n = 1/N∑_i=1^N I(rank_i ≤ n).


Where N is the total number of queries, and ∑ is the summation over all queries.







§ STRATEGY

In the task of text-based person re-identification, image data is typically captured through CCTV in open environments [45]. This data not only contains key descriptive features of individuals but also includes a significant amount of background noise. Early works <cit.> typically directly extract global image text features for simple cross-modal matching. However, due to the uniqueness of the person re-identification task, which differs from existing image-text retrieval <cit.>, the categories to be retrieved all belong to highly similar individuals. Identification based solely on distinguishing features such as clothing and posture increases the challenges of the text-based person re-identification task.



 §.§ Stripe Segmentation

Some work suggests that various parts of the human body are evenly arranged in images, and Stripe Segmentation can be employed as a guiding model to achieve multimodal local alignment supervision <cit.>. Stripe Segmentation is a commonly used technique in pedestrian re-identification, aiming to mitigate the impact of pose, occlusion, and lighting variations on re-identification performance within pedestrian images <cit.>, <cit.>. The fundamental idea is to partition pedestrian images into multiple vertical stripes, also referred to as person parts. Each stripe can be considered a subregion of the pedestrian image (head, upper body, lower body, foot). The advantage of this approach lies in dividing pedestrian images into multiple local regions, effectively capturing local features of pedestrians to enhance recognition accuracy <cit.>. For instance, local texture, color, and shape features can be extracted within each stripe, facilitating better differentiation between different parts of pedestrians. Stripes of different heights can correspond to different scales of pedestrian parts, providing richer scale information to address scale variation issues in pedestrian images. Ding et al. <cit.> proposed a Semantic Self-Alignment Network (SSAN) capable of automatically extracting part-level textual features for corresponding visual regions. They designed a multi-view non-local network to capture relationships between body parts, thereby establishing better correspondences between body parts and noun phrases. Li et al <cit.> proposed to vertically divide a character image into multiple regions using two different approaches, including overlapping slices and key-point-based slices. Niu et al. <cit.> also emphasized local-local alignment, employing Stripe Segmentation in the proposed Bidirectional Fine-Grained Matching (BFM) module to match visual human body parts with noun phrases. However, this strategy is fragile and sensitive to conditional changes. For example, the regions of image samples do not always contain a complete human body, and at times, the human head does not consistently appear in the first stripe, significantly impacting the robustness of existing methods.



 §.§ Multi-scale Fusion

Due to the small inter-class variance in both pedestrian images and descriptions, comprehensive information is required for pedestrian retrieval to coordinate visual and textual clues across all scales. Subsequent to the introduction of local feature extraction strategies other than stripe segmentation, the utilization of fused features is referred to as multi-scale feature fusion <cit.>, as illustrated in Figure  <ref>. Image Structure Graph Network (A-GANet) <cit.> employs residual modules for visual representation extraction and utilizes designed graph attention convolutional layers for structured high-level visual semantic feature extraction. Specifically, it captures relationship features of different pedestrian parts through graph convolutional networks. Wang et al. <cit.> proposed the Visual Text Attribute Alignment model (ViTAA), which learns to decompose the person's feature space into subspaces corresponding to attributes using an optically assisted attribute segmentation layer. On the image side, a segmentation layer is employed to divide pedestrian images into full body, head, upper clothes, Lower clothes, shoes, and bag parts. On the text side, corresponding phrases of different segmented parts are extracted. 

Ji et al. <cit.> introduced an Asymmetric Cross-Scale Alignment (ACSA) method. Global text representation and local phrase representation are employed for text extraction. The global representation is divided into four regions as local visual representations, namely Head, Upper body, Lower body, and foot. These features are used afterwards for feature concatenation and computation. The partition strategy does not involve additional computational costs but can better retain prominent body parts for fine-grained matching. Wang et al.<cit.> proposed a novel Multi-Granularity Embedding Learning (MGEL) model. It generates multi-granularity embeddings of partial human bodies from coarse to fine by revisiting person images at different spatial scales. Chen et al.<cit.> adopted a multi-branch representation in the learning path, enabling text-adaptive matching of corresponding visual local representations, thereby aligning text and visual local representations. Additionally, a multi-stage cross-modal matching strategy was proposed to eliminate modality gaps from low-level, local, and global features, gradually narrowing the feature gap between image and text domains. 

Niu et al.<cit.> presented a Multi-Granularity Image-Text Alignment (MIA) framework to address cross-modal fine-grained issues. It consists of three modules corresponding to three granularities. The Global Contrast (GC) module is used for global-global alignment. The Relation-Guided Global-Local Alignment (RGA) module filters global-local relationships. The Bidirectional Fine-Grained Matching (BFM) module aligns local to local based on trained fine-grained local components. Zheng et al. <cit.> proposed a Hierarchical Adaptive Matching model to learn subtle feature representations from three different granularities (i.e., word-level, phrase-level, and sentence-level) for fine-grained image-text retrieval tasks. Wang et al. <cit.> addressed the issue that existing methods typically learn similar mappings between local parts of images and text or embed entire images and text into a unified embedding space, resulting in a problem of local-global correlation. They designed a Divide and Merge Embedding (DME) learning framework for text-based person search. It models the relationship between local parts and global embedding, merging local details into the global embedding. Li et al.<cit.> constructed features for text and person image matching, different from other methods. The latter acquires global features by aggregating aligned local features, leading to misalignment between text and visual features. They proposed placing features with the same semantics in the same spatial position to construct the final features for person-text-image matching, achieving semantic consistency and interpretability of global features. 

Yan et al.<cit.> proposed an Implicit Local Alignment module, adaptively aggregating image and text features into a set of modality-shared semantic topic centers, implicitly learning local fine-grained correspondence between images and text without additional supervision information and complex cross-modal interaction. Additionally, global alignment is introduced as a supplement to the local perspective. Gao et al. <cit.> introduced a method called Non-Local Alignment on Full Scale (NAFS), capable of adaptively aligning image and text features at all scales. First, a novel ladder network structure is proposed to extract full-size image features with better locality. Secondly, a language model, Bidirectional Encoder Representations from Transformers (BERT), with local constraint attention is proposed to obtain description representations at different scales. Then, instead of aligning features separately at each scale, a novel context non-local attention mechanism is applied to simultaneously discover potential alignment at all scales. 

In summary, applying target detection or additional branch networks to detect significant regions and then extract local features for obtaining multi-scale features has certain accuracy advantages. However, due to the increased external network overhead, these methods often result in higher computational costs.



 §.§ Attention Mechanism

Cross-modal alignment is inherently challenging in achieving fine-grained matching between text and images. In addition to implicit alignment strategies achieved through the automatic fusion of multi-scale features, another key approach involves the use of explicit alignment strategies. For instance, employing attention mechanisms enhances the capability of feature extraction by capturing discriminative feature representations related to both language descriptions and visual appearances. This aids in intuitively understanding and controlling the alignment process. Attention models can establish correspondences between body parts and words <cit.>, offering the advantage of not relying on external cues for aligning feature embeddings. These attention strategies can be categorized into spatial attention, channel attention, mixed attention, non-local attention, and positional attention.

Liu et al. <cit.> proposed an attention-based deep neural network capable of capturing multiple attention features from low-level to semantic layers to learn comprehensive features for fine-grained pedestrian representation. Wang et al. <cit.> utilized a multi-head self-attention module to extract embeddings for different granularity portions of the text stream, employing adaptive filtering to remove interference and obtain fine-grained features. Lee et al. <cit.> introduced a Stacked Cross Attention Mechanism (SCAN) that compares attention information for text and image in two stages, determining the importance of image regions and words to achieve potential alignment between the two modalities. Spatial attention-based methods often focus solely on local discriminative features of pedestrians, neglecting the impact of feature diversity on pedestrian retrieval.

Li et al. <cit.> proposed a cubic attention convolutional neural network that combines spatial and channel attention to maximize complementary information from different scale attention features, addressing the challenges of cross-modal alignment in pedestrian re-identification. Wang et al. <cit.> designed a Feature Division Network (FDN) that embeds inputs into K locally guided semantic representations using self-attention, each representing different parts of a person. They then introduced a Correlation-based Subspace Projection (RSP) method to merge different local representations into a compact global embedding. Yan et al. <cit.> presented an Efficient Joint Information and Semantic Alignment Network (ISANet) for text-based person search. Specifically, they designed an image-specific information suppression module that suppresses image background and environmental factors through relationship-guided localization and channel attention filtering, effectively alleviating information inequality and achieving information alignment between text and image.

Considering the indistinguishability of features extracted from first-order attention (e.g., spatial and channel attention) in complex camera views and pose-changing scenarios, Farooq et al. <cit.> applied non-local attention directly after calculating interactions between text features to model long-term dependencies between different phrases in the text. Inspired by self-attention, Gao et al. <cit.> proposed a Context Non-Local Attention to enable cross-modal features to align with each other in a coarse-to-fine manner based on their semantics, rather than relying solely on predefined and fixed rules (e.g., local-local, global-global). Wang et al. <cit.> introduced a Part-based Multi-scale Attention Network (PMAN), consisting of a dual-path feature extraction framework with attention-based branches for a multi-scale cross-modal matching module. This module extracts visual semantic features from different scales and matches them with text features.

Addressing concerns that utilizing soft attention mechanisms to infer semantic alignment between image regions and corresponding words in a sentence may lead to the fusion of unrelated multimodal features and result in redundant matching, Wang et al. <cit.> proposed an IMG-Net model. This model combines intra-modal self-attention and cross-modal hard region attention with a fine-grained model to extract multi-granularity semantic information. Zheng et al. <cit.> introduced a Hierarchical Gumbel Hard Attention Module, using the Gumbel top-k reparameterization algorithm as a low-variance, unbiased gradient estimator to select strongly semantically related regions for all regions of the image and corresponding words or phrases, enhancing precise alignment and similarity calculation results.

While these techniques often bring about noticeably superior retrieval performance, they frequently come at the cost of efficiency. Specifically, for M galleries and N queries, their complexity is O(M+N). In contrast, the complexity of attention-based cross-modal approaches rises to O(MN) due to pairwise inputs. To address this issue, Yang et al. <cit.> initially employ cross-modal non-attention features to quickly identify candidates and then deploy attention-based modules to refine the final ranking scores.



 §.§ External Auxiliary

For Text-based Person Re-Identification (Re-ID) tasks, several auxiliary strategies such as image segmentation, random mask, human body keypoints, attribute prediction, clustering analysis, and color extraction can be employed to extract advanced semantics, thereby improving cross-modal retrieval performance. <cit.> proposed a dual-part simultaneous alignment representation scheme using attention mechanisms to capture distinguishing information beyond body parts by leveraging complementary information from accurate human parts and noisy parts to update representations.

To leverage multi-level visual content, Jing et al. <cit.> introduced a Pose-guided Multi-granularity Attention Network (PMA). They initially proposed a Coarse Alignment network (CA) that uses similarity-based attention to select relevant image regions for global description. Pose information can be utilized to learn potential semantic alignments between visual body parts and textual noun phrases. Shu et al. <cit.> introduced Bidirectional Mask Modeling (BMM) without the need for additional manual labeling. They applied random masks to images and corresponding text keywords, forcing the model to explore more useful matching clues. This approach increases data diversity and enhances model generalization. 

Wang et al. <cit.> proposed a novel Visual-Textual Attribute Alignment model (ViTAA). For semantic feature extraction, segmentation labels are used to drive the learning of attribute-aware features from input images. Aggarwal et al. <cit.> introduced a method for text-based person search by learning attribute-driven spatial and class-driven spatial information. The goal is to create semantically retained embeddings through an additional task of attribute prediction. Suo et al. <cit.> proposed a novel Simple Robust Correlated Filtering (SRCF) framework, which effectively extracts key clues and adaptively aligns multimodal features by constructing a set of universal semantic templates (filters). This framework differs from previous attention-based methods, focusing on computing similarities between templates and inputs. Two types of filtering modules (denoising filter and dictionary filter) were designed to extract key features and establish multimodal mappings.

Zhao et al. <cit.> introduced a Cross-Modal Mutual Training (CMMT) framework. Specifically, to alleviate intra-class variation, clustering methods generate pseudo-labels for visual and text instances. To further refine clustering results, CMMT provides a mutual pseudo-label refinement module, which utilizes clustering results in one modality to refine clustering results in another modality constrained by text-image pairwise relationships. Chen et al. <cit.> integrated information theory and adversarial learning into an end-to-end framework, exploring information theory to reduce heterogeneity gaps in cross-modal retrieval. This method is beneficial for building a shared space to further learn commonalities between cross-modal features, applicable to video-text matching. Additionally, regularization terms based on KL-divergence and temperature scaling were introduced to address the issue of data imbalance. 

Li et al. <cit.> proposed a novel Joint Label and Feature Alignment Framework (TFAF) to gradually reduce inter-modal and intra-class gaps. Firstly, a dual-path feature learning network is constructed to extract features and align them to reduce inter-modal gaps. 
Secondly, a text generation module is designed to generate label sequences using visual features, followed by label alignment to reduce intra-class gaps. Finally, a fusion interaction module is introduced, utilizing a multi-stage feature fusion strategy to further eliminate modality heterogeneity. Chen et al. <cit.> introduced the concept of knowledge distillation to image and text networks, aiming to improve the matching capabilities of two networks with different modalities by balancing information from both networks. 
Wang et al. <cit.> addressed the color-dependency issue in cross-modal pedestrian retrieval by proposing a Color-Aware Information Beyond Color(CAIBC) architecture through joint optimization of a multi-branch structure. 
CAIBC includes three branches(RGB, Grayscale, and Color) employing mutual learning mechanisms for knowledge exchange across branches. 
It achieved an accuracy of 88.37% in top-10 on the CUHK-PEDES dataset for weakly supervised text-to-person retrieval.








§ ARCHITECTURE

When mapping features from a specific modality to a common manifold, the feature distribution of other modalities remains imperceptible. This implies that embedding and aligning multimodal features in the common manifold entirely depend on the model's own experience rather than the actual data distribution. In other words, a major challenge in cross-modal Person Re-ID is ensuring that the feature distribution in the common manifold accurately reflects the feature distribution in the original modalities. This necessitates the model to possess sufficient capability to capture and understand the relationships between different modalities, which typically requires abundant data and carefully designed model structures.



 §.§ Convolutional Neural Network

Text-based person Re-ID can be viewed as a multimodal pedestrian retrieval problem, with the primary challenge lying in the extraction of features from both visual and textual data. Due to the capability of neural networks to automatically learn and extract features from pedestrian images, the cumbersome process of manual feature design can be avoided. Most existing efforts consider employing network architectures designed for image classification as the backbone of visual networks. Consequently, convolutional neural networks (CNNs) have been utilized extensively in text-based pedestrian re-identification, primarily leveraging CNNs <cit.>, and their major variants such as VGG-16 <cit.>, MobileNet <cit.>, ResNet50 <cit.>, ResNet101 <cit.>, ResNet-152 <cit.>, etc., for extracting crucial information in visual features. These networks are sometimes also employed for text feature extraction <cit.>. The combination of pedestrian images extracted by CNNs and textual description information is achieved, aligning low and high-level semantic information across modalities in an implicit space using matching methods. This facilitates the recognition and matching of pedestrian identities.

Two independently pre-trained ResNet-50 models on ImageNet serve as the visual backbone in study <cit.>. Addressing the challenge of manual annotation for existing cross-modal data, Jing et al. propose a Matrix Alignment Network (MAN) to tackle cross-modal cross-domain person search tasks, applying the single-modal cross-domain adaptive idea to the problem of image-text-based cross-modal person re-identification <cit.>. Farooq et al. introduce the Deep Vision Net model based on ResNet-50 to construct the visual feature extraction backbone <cit.>. Deviating from the original architecture, the proposed network includes two fully connected layers before the classifier layers, applies batch normalization after the two FC layers, and employs dropout before the final layer. They further present a two-stream deep convolutional neural network framework supervised by cross-entropy loss <cit.>. The language network is modified to resemble the visual branch in terms of the number of layers in deep residual networks, connecting the class probability weights between the second-to-last layer and the final layer, i.e., sharing logits of the softmax layer across both networks. AXM-Net, proposed by <cit.>, is a novel architecture based on convolutional neural networks (CNN) that does not rely on external cues for explicit alignment of feature embeddings and is capable of learning semantically aligned cross-modal feature representations. Most of the aforementioned works initialize pre-trained ResNet-50 <cit.> and ResNet-101 <cit.> backbones on large datasets to better map modal-corresponding information. Additionally, the application of large models such as CLIP in some fine-grained visual tasks has demonstrated effective paradigm transfer, fully migrating visual-text large models based on contrastive learning to text-based person re-identification tasks <cit.>. Presently, many studies commonly regard representation learning as a classification method for person re-identification, believing that feature embedding is a core module for text-based person re-identification tasks, crucial for narrowing the semantic gap between modalities. CNNs exhibit significant advantages in perceptual localization of local features in pedestrian vision and end-to-end learning. However, they still have certain limitations in handling long-term dependent data relationships.



 §.§ Recurrent Neural Network

Beyond visual features, textual descriptions directly impact the accuracy of text-based Person Re-ID. Efficient extraction of textual features is crucial under existing textual data descriptions <cit.>. Recurrent Neural Networks (RNNs) and their variants, as neural networks for processing sequential events, including text, have been widely accepted in the research community <cit.>. Early on, in this field, word embedding techniques were initially employed to map words to vector spaces. Subsequently, variants such as LSTM <cit.>, Bi-LSTM <cit.> attempted to capture long dependencies of sentences for semantic feature extraction from text sequences. Later, with the emergence of large language models like Bidirectional Encoder Representations from Transformers (BERT)<cit.>, the paradigm shift of "pre-training + fine-tuning" has been triggered in the Text-based Person Re-ID domain. Utilizing pre-trained models to learn general features of natural language and obtaining high-quality word embeddings, followed by inputting into RNNs variants to capture contextual dependencies.

Niu et al. <cit.> first proposed a concise and effective framework for image-text alignment to address visual-semantic differences. Innovatively integrating two opposing directions, source to target and target to source, for cross-domain adaptation. Wang et al. <cit.> introduced a novel Adversarial Multi-space Embedding Network (AMEN) to learn and match embeddings in multiple spaces. Bi-GRU is used for text feature extraction, and the paradigms of inter-modal and intra-modal reconstruction collaborate to embed features correctly into the opposite modal space while learning a robust common space. <cit.> introduces a Text-Graph Attention Network consisting of a Bi-LSTM layer, a text scene graph module, and a joint embedding layer. Bi-LSTM units gradually take the embeddings of each word in the text as input, capturing the historical and future context information of processed words. Yan et al. <cit.> employed an LSTM-based Recurrent Feature Aggregation Network to accumulate discriminative features from the first LSTM node to the deepest LSTM node, effectively alleviating interference caused by occlusion, background noise, and detection failures. Chen et al. <cit.> decomposed video sequences into multiple segments and used LSTM to learn to detect the segments where the images are located in both time and space features. This approach reduces variations in the same person across samples, favoring the learning of similarity features. Both of the above methods independently process each video frame. Ding et al. <cit.> employed Bidirectional Long Short-Term Memory Network (Bi-LSTM) to process each text description, capturing relationships between words. With contextual cues, the Word Attention Module (WAM) based on word representations infers word part correspondences, referring to word part correspondences to obtain original part-level texture features. Wang et al. <cit.> proposed a new algorithm called LBUL to learn a Consistent Cross-Modal Common Manifold (C3M) for text-based person retrieval. For textual samples, the proposed method embeds the text modality into C3M after considering the distribution features of cross-modalities comprehensively to achieve more reliable cross-modal distribution consensus, rather than blind predictions. However, RNN and its variants also have some issues in the Text-based Person Re-ID task. For instance, the embedding quality of pedestrian text inputs is often very sensitive, features extracted are typically influenced by the length of data sequences, and RNNs only establish temporal correlations on high-level features, thus failing to capture the temporal clues describing the local details of images.



 §.§ Autoencoder

Sometimes, text-based Person Re-ID tasks also involve the utilization of Autoencoder network architectures. Commonly, these models map images or textual descriptions into a low-dimensional embedding space and employ this embedding space for person re-identification tasks <cit.>. In this manner, pedestrian images and their corresponding textual descriptions are mapped to a low-dimensional vector space, which encapsulates crucial semantic information, as both image and text provide pivotal details about the identity of the pedestrian. Autoencoders, particularly those with atypical encoder components, exhibit several advantages in Text-based Person Re-ID tasks. Firstly, they can overcome issues prevalent in traditional approaches arising from the lack of semantic information in image-text pairs. Secondly, Autoencoder models proficiently capture essential information from textual descriptions and compress it, facilitating effective data compression. Additionally, this approach boasts good interpretability, as relationships between pedestrian images and textual descriptions can be visualized based on vectors within the embedding space. Wang et al. <cit.> introduced an innovative Adversarial Multi-space Embedding Network (AMEN) for learning and matching embeddings across multiple spaces. Following an encoder-decoder paradigm, the inter-modal reconstruction paradigm collaborates with the intra-modal reconstruction paradigm to embed features accurately into the opposite modal space while learning a robust common space. <cit.> proposed an Improved Embedding Learning through Virtual Attribute Decoupling (iVAD) model. This method enhances embedding learning by using an encoder-decoder structure to decompose attribute information in images and text. Pedestrian attributes are treated as hidden vectors, and attribute-related embeddings are obtained. Diverging from previous works that separate attribute learning from image-text embedding learning, a hierarchical feature embedding framework is introduced. The Attribute-Enhanced Feature Embedding (AEFE) module merges attribute-related embeddings into the learned image-text embeddings, leveraging attribute information to enhance feature discriminability. However, a drawback of this approach is its reliance on a substantial amount of training data to train the Autoencoder model, and meticulous tuning of model parameters and embedding space dimensions is necessary to achieve optimal performance.



 §.§ Graph Neural Network

Utilizing Graph Neural Networks (GNN) for text-based pedestrian re-identification has become a noteworthy research direction in recent years <cit.>. GNN is a type of deep learning method based on graph structures, suitable for handling unstructured data and relationships, and applicable to various data types such as images, text, and structured data. In text-based Person Re-ID tasks, GNN is often employed to construct a graph structure, where each node represents a pedestrian body part along with its corresponding phrase description. Edges between nodes represent semantic relationships between different body parts and descriptions, such as verb (wear), noun (women), adjective (yellow), and others. The methods utilizing GNN for text-based Person Re-ID typically involve the following steps. First, images and corresponding text descriptions for each pedestrian are extracted from the dataset. Second, two graph convolution structures are established based on auxiliary strategies, such as attention mechanisms, image segmentation, etc. Afterwards, Graph Convolutional Networks (GCN) are employed to learn the relationships between body nodes in the graph structure and transform these relationships into feature vectors. Finally, feature vectors of body structures undergo cross-modal asymmetric alignment, and the text modality is input into a classifier for Person Re-ID.

The methods employing GNN for text-based Person Re-ID offer several advantages. Firstly, they can capture semantic relationships between different pedestrian images and text descriptions, facilitating future inference-based retrieval, i.e., recognizing a pedestrian engaged in complex events. Secondly, GNN can handle text descriptions of varying lengths and structures, providing the advantage of data flexibility. Numerous studies have explored GNN-based Person Re-ID methods, including those using Graph Convolutional Networks (GCN) and methods based on graph self-attention mechanisms. Liu et al. <cit.> proposed a novel deep adversarial graph attention convolutional network (A-GANet) for text-based person search. A-GANet utilizes text and visual scene graphs, including object properties and relationships, from pedestrian text queries and gallery images to learn information-rich text and visual representations. It learns an effective joint text-visual latent feature space through adversarial learning, bridging the modality gap and promoting person matching. Li et al. <cit.> presented a fundamental framework using graph convolution as a feature representation. They simulate adversarial attacks caused by text and image diversity on feature extraction by embedding additional attack nodes in graph convolution layers to enhance the model's robustness to textual and visual diversity. However, when graph neural networks compute edges for pedestrian body nodes and entity relationships, it may result in high computational complexity and memory requirements. Moreover, GNNs are more susceptible to overfitting compared to general network structures.



 §.§ Transformer

Due to the success of Transformers in multimodal tasks, it has become a popular architecture widely employed in text-based Person Re-ID tasks <cit.>. Currently, a commonly used approach involves a dual-stream network architecture, where one stream is dedicated to processing pedestrian images, and the other handles the corresponding text descriptions. In the image stream, convolutional neural networks <cit.> or vision transformers <cit.> are utilized to extract visual features from pedestrian images. Simultaneously, in the text stream, Long Short-Term Memory (LSTM) <cit.> or transformers <cit.> are applied to extract feature vectors from the corresponding text descriptions. Li et al. <cit.> introduced a transformer-based Person Re-ID architecture, evaluating the shared attention between language reference terms and visual features through transformer blocks. Apart from achieving outstanding search performance, the proposed method also provides interpretability by visualizing the attention between image parts in person pictures and their corresponding referential terms. Existing works often overlook the granularity differences between features of the two modalities, where visual features tend to be fine-grained while text features are coarse-grained, leading to significant modality gaps. Shao et al. <cit.> proposed an end-to-end framework named Learning Granularity-Unified Representations (LGUR), based on the transformer, aiming to map visual and text features to a unified granularity space. Addressing alignment deficiencies between image region features and text features in weakly supervised learning methods, resulting in poor feature distribution, Gao et al. <cit.> presented a novel method called Conditional Feature Learning based Transformer (CFLT). CFLT maps sub-regions and phrases to a unified latent space and dynamically adjusts features from one modality based on data from another modality by constructing explicit conditional embeddings.

With the significant progress achieved by large models, Sarafianos et al. <cit.> demonstrated for the first time the effectiveness of BERT as an openly accessible word embedding extraction language model in the domain of image-text matching. Subsequently, they combined two feature vectors using a cross-attention mechanism and applied them to pedestrian re-identification tasks. Li et al. [70] proposed a semantically aligned feature aggregation network that adaptively aggregates unit features with the same semantics into different part-perceptive features. Initially introducing Transformer-based backbones ViT <cit.> and BERT into visual and text modalities, Ji et al. <cit.> extracted robust feature representations for text-based person search. They proposed an Asymmetric Cross-Scale Alignment (ACSA) method, employing Swin Transformer for visual representation extraction and BERT for text and phrase representation in the text domain. In contrast to previous methods that use two separate networks for feature extraction, Wang et al. <cit.> introduced an Implicit Visual-Text (IVT) framework, a single-stream network architecture using only a single transformer. Text and image feature extraction utilizes a backbone network, optimizing network shared parameters with cross-modal data to facilitate learning a common space mapping. Recently, Vision-Language Pre-training (VLP) models in visual language pre-training have demonstrated the transferability of knowledge for downstream Text-Picture Search (TPS) tasks, enhancing performance effectively. An approach called CLIP-driven Fine-grained Information Mining (CFine) <cit.> was proposed to leverage CLIP's visual knowledge for person re-identification, incorporating fine-grained information mining to explore intra-modal discriminative clues and inter-modal correspondences. However, the lack of VLP text representations may compromise the valuable cross-modal alignment of the two encoders. Hence, Wang et al. <cit.> introduced a model with two pre-training modes, TP-TPS, where both text and image encoders are based on transformers, and all backbones are initialized with pre-training weights from CLIP. They introduced Multi-Integrity Description Constraints (MIDC) to enhance the robustness of the text modality by combining different components of a fine-grained corpus during training. The advantage of the Transformer model lies in its ability to effectively learn sequence information, making it capable of capturing information in text descriptions efficiently. Additionally, this approach exhibits good interpretability in the field of multi-scale learning, as it allows the visualization of regions of interest in pedestrian images and keywords in text descriptions based on attention weights. Finally, it plays a crucial role in unified modality architectures.








§ OPTIMIZATION

Deep metric learning plays a paramount optimization role in cross-modal person re-identification <cit.>. 
For instance, it enhances the discriminative power of feature representation, addresses a multitude of noise and variations such as diverse lighting conditions, poses, and occlusions, adapts the distance metric accordingly, embeds more semantically meaningful spatial structures, and leverages domain adaptation to tackle the challenges posed by multiple viewing angles. Of particular significance is the optimization of loss functions, thereby improving the matching performance in cross-modal person re-identification tasks <cit.>. This approach effectively addresses a spectrum of challenges, elevating the network's robustness and generalization capacity, consequently achieving more accurate person matching across distinct camera scenes and conditions. 
This study places emphasis on prevalent loss functions used in text-based person re-identification tasks, including identity loss, verification loss, contrastive loss, triplet loss, quadruplet loss, and adversarial loss, as shown in Figure <ref>.





 §.§ Identity Loss

Zheng et al. <cit.> introduced the ID Discriminative Embedding (IDE) network, which treats each pedestrian as a separate category and employs the pedestrian's ID as a classification label to train a deep neural network. This allows the network to learn the capability of predicting the identity given a pedestrian image. The classification loss is typically computed using the cross-entropy loss function. During training, the classifier optimizes its weight parameters by minimizing the classification loss. A fully connected layer (FC) for classification is appended to the end of the network, and the softmax activation function maps the features to a probability space representing identity labels <cit.>. The cross-entropy loss for the multi-class pedestrian Re-ID task can be expressed as:


    L_ID = -1/N∑_i=1^Ny_ilog(p_i).


Here, x_i denotes the input image, y_i represents the corresponding label, p_i is the probability that x_i is recognized as y_i after the softmax function, and N indicates the batch size. Identity loss is widely adopted in pedestrian re-identification <cit.>, due to its automatic hard sample mining and ease of training. However, as the labeled data increases significantly, issues related to decreased efficiency in classifier training might arise, especially when dealing with datasets containing a large number of similar pedestrians. In cross-modal pedestrian re-identification, the classification loss is often used in conjunction with other loss functions <cit.>, such as contrastive loss and triplet loss <cit.>, to enhance performance of the model.



 §.§ Verification Loss

The verification loss is employed to quantify the similarity between two pedestrian images <cit.>. In pedestrian re-identification, it is often necessary to compare two images to determine if they belong to the same individual. The verification loss aims to achieve this objective by comparing the feature vectors of two images to compute their similarity <cit.>. Specifically, the verification loss is calculated by inputting the feature vectors of two images into a similarity metric function. This similarity metric function can be Euclidean distance, cosine similarity, or other similar measures. If two images belong to the same person, their feature vectors should exhibit high similarity, resulting in a high similarity score <cit.>. Thus, the verification loss is minimized to enhance the similarity between two images of the same person. The specific formula is as follows:


    L_veri = -1/N∑_i=1^Ny_ilog(p_i) + (1-y_i)log(1-p_i).


Here, N is the number of sample pairs, y_i is a label indicating whether the sample pair x_i and x_j belongs to the same category (1 denotes the same category, 0 denotes different categories), and the softmax function is utilized to calculate the probability p_i of having the same label for both samples. Due to the necessity of pairwise computation, the verification loss tends to have lower efficiency in re-identification tasks. Therefore, it is often used in combination with other losses, such as identity loss, to leverage the advantages of both methods, thereby enhancing recognition speed and accuracy.



 §.§ Contrastive Loss

Contrastive Loss was originally introduced by Sumit Chopra et al. in the context of face verification tasks <cit.>, aiming to train neural networks to determine whether two inputs belong to the same facial identity. In the context of person re-identification, the principle of encouraging similar identity feature representations to be closer in the feature space while dispersing feature representations of different identities is leveraged for model training within Siamese networks <cit.>. The Contrastive Loss not only considers the distances between images of the same identity but also takes into account the distances between images of different identities. The function can be expressed as follows:


    L_con = 1/2N∑_n=1^Ny_nD_n^2 + (1-y_n)max(margin - D_n, 0)^2.


Here, N is the number of sample pairs, y_n is a label indicating whether sample pairs x_i and x_j belong to the same class (1 for the same class, 0 for different classes), D_n represents the distance between feature representations (Euclidean distance, cosine distance, etc.), and margin is a threshold indicating that when sample pairs belong to different classes, their distance should be greater than this value. For sample pairs belonging to the same class, the loss is the square of the distance D_n, while for sample pairs belonging to different classes, the loss is the square of the difference between the distance and the threshold value margin. This framework aims to bring feature representations of sample pairs from the same class closer together and push feature representations of sample pairs from different classes further apart. By adjusting the value of margin and the weight of the loss, the distance range between sample pairs can be controlled, influencing the learning of feature representations <cit.>.



 §.§ Triplet Loss

The Triplet Loss is a widely utilized loss function in pedestrian re-identification, aiming to learn discriminative feature representations by comparing the distances between three images <cit.>. Its fundamental concept involves learning feature representations with discrimination, such that the distance between images of the same person is minimized, while the distance between images of different individuals is maximized <cit.>, <cit.>, achieved by minimizing the Triplet Loss during training. Specifically, given an anchor image, a positive sample image, and a negative sample image, the objective of Triplet Loss is to minimize the distance between the anchor and the positive sample while simultaneously maximizing the distance between the anchor and the negative sample <cit.>, <cit.>. The formal definition of Triplet Loss is as follows:


    L_triplet =    ∑_n=1^Nmax(||f(x_n^a) - f(x_n^p)||_2^2
          - ||f(x_n^a) - f(x_n^n)||_2^2 + margin, 0).


Here, f is a function that maps an image x to a feature vector, x_n^a, x_n^p and x_n^n are the anchor, positive, and negative sample images of the n-th instance, respectively. ||·||_2 represents the Euclidean distance between two embedding vectors, max(·) represents taking the maximum value between 0 and ·, margin is a hyperparameter controlling the difference in distances between images of the same person and images of different people. It is worth noting that in practice, Triplet Loss has encountered certain issues, such as sample selection and slow learning speed. Consequently, improved versions have been proposed <cit.>. Wang et al. <cit.> proposed a novel mutually connected classification loss (MCCL) based on triplet loss to fully exploit identity-level information. It not only incorporates identification information into both images and language descriptions but also encourages similar cross-modal classification probabilities for instances belonging to the same identity. These improved versions prove to address certain existing issues more effectively and have demonstrated excellent performance in person re-identification tasks.



 §.§ Quadruplet Loss

The Quadruplet Loss is an extension of the Triplet Loss <cit.>. The Triplet Loss is commonly employed to train a neural network to embed images of the same person into close proximity within the embedding space, while images of different individuals are embedded far from the space occupied by the same person. However, the Triplet Loss may suffer from issues such as gradient vanishing or exploding, which could render it less stable in practical usage. The Quadruplet Loss enhances the stability of the Triplet Loss by introducing a fourth sample.

In specific terms, the Quadruplet Loss computes pairs of embedding distances for two images of the same person and two images of different individuals, and employs these four distances to formulate the loss function. In this regard, the two images of the same person should be closer, while the two images of different individuals should be farther apart. By separately using the two images of the same person and the two images of different individuals to calculate distances, the Quadruplet Loss can yield more stable training, thus enhancing the performance of person re-identification. The formula for the Quadruplet Loss is as follows:


    L_quad = ∑_n=1^Nmax(   ||f(x_n^a) - f(x_n^p)||_2^2 - ||f(x_n^a) - f(x_n^n1)||_2^2
        + α, 0)+ max(||f(x_n^a) - f(x_n^n1)||_2^2
       - ||f(x_n^n1) - f(x_n^n2)||_2^2 + β, 0).


Here, f denotes the embedding function of the neural network, x_n^a, x_n^p, x_n^n1 and x_n^n2 respectively represent the anchor image, positive sample image, first negative sample image, and second negative sample image for the n-th sample. ||·||_2 represents the Euclidean distance between two embedding vectors. max(·) represents taking the maximum value between 0 and ·, The hyperparameters α and β control the difference in distances between the two images of the same person and the two images of different individuals.

The Quadruplet Loss effectively reduces the within-class variance and improves the discriminative performance of the model by introducing an additional negative sample. However, the increase in parameters also requires the computation of more pairs of samples, leading to a significant increase in computational complexity and longer training time. Therefore, suitable strategies are required to select effective quadruples in practical applications.



 §.§ Adversarial Loss

The adversarial loss primarily leverages adversarial training to learn more robust pedestrian feature representations <cit.>. Its fundamental concept involves mapping pedestrian feature embeddings from the training data into a similarity space, where the distribution of generated image features is made closer to that of real image features, enhancing the robustness of the network in handling multi-modal information <cit.>. Specifically, adversarial loss employs adversarial samples during training, where the model seeks to minimize the adversarial loss. The computation of the adversarial loss involves introducing perturbations to the model input, making the input pedestrian image appear visually similar to the original image while introducing significant disparities in the embedding space. These perturbations are generated by a generator network, which attempts to produce images similar in appearance to the input image but with substantial deviations in the embedding space. The discriminator seeks to maximize the adversarial loss to effectively discriminate between these two types of features. The general formula for adversarial loss is as follows:


    L_adv =    𝔼_x∼ p_data(x)[log D(x)] 
       + 𝔼_z∼ p_z(z)[log(1 - D(G(z)))].


Here, G is the generator, D is the discriminator, x represents real image features, z represents generated image features, log D(x) denotes the logarithm of the discriminator's predicted probability for real image features, and log(1 - D(G(z))) denotes the logarithm of the discriminator's predicted probability for generated image features. Liu et al. <cit.> proposed the application of adversarial principles in pedestrian re-identification to acquire more robust feature representations. They contend that the distributions of text and image features should be modality-invariant and semantically discriminative, while maintaining latent cross-modal semantic similarities. These features exhibit enhanced resistance to adversarial attacks. Subsequently, an adversarial learning module was devised to optimize the representations extracted from image graph attention networks and text graph attention networks. The objective of the modality discriminator is to reliably distinguish between modalities of samples as much as possible, given a well-trained feature transformer, until the point where discrimination becomes infeasible. Adversarial loss is employed to classify the modality labels (belonging to either image or text) of input samples. Despite its relatively recent application in pedestrian re-identification, adversarial loss has demonstrated significant efficacy in some recent studies <cit.>.








§ CHALLENGE


In this paper, we first give an overview of text-based person Re-ID in two parts: attribute-based and natural language-based person Re-ID. Secondly, the datasets based on the two parts are summarized and described respectively. Moreover, the commonly used technical strategies in text-based person Re-ID are analyzed. In addition, an overview of the deep learning architectures that usually appear in text-based person Re-ID is made. Further, the loss functions (alignment optimization methods) commonly used in text-based person Re-ID are summarized and described in detail. Finally, we analyze the challenges and possible future directions of text-based person Re-ID from the perspectives of feature extraction, data scarcity, pre-trained models, and generative models, respectively.

Although text-based person Re-ID technology offers numerous advantages in fields such as video surveillance, security, social media, and virtual reality<cit.>, it still faces various challenges. The most critical issues include the ambiguity and granularity of natural language. Identical descriptions may be incomplete, ambiguous, or contain ambiguities, potentially leading to incorrect identity matches. On the other hand, photographs available in real-world galleries pertaining to pedestrians often lack high resolution, ample lighting, and freedom from obstruction<cit.>. 
Zhang et al. <cit.> demonstrated recently that gait information may be the basis for general person Re-ID, and that its features may be more robust compared to the highly homogeneous pedestrian appearance feature signature. However, existing gait recognition methods are not able to meet the requirements of less controlled scenarios, e.g., most of the studies are based on well controlled scenarios such as narrow corridors. If one wants to improve the efficiency of person Re-ID by combining gait features with appearance features, it may be necessary to obtain more coherent and high-quality gait data, while the appearance is recognizable. 
Therefore, integrating text descriptions with image/video information requires more efficient and robust cross-modal fusion techniques for the system to better understand and match entities. Pre-trained large-scale multimodal models may represent a reasonable direction for achieving improved encoding effects. Furthermore, text-based person re-identification necessitates more high-quality data to enable the model to learn relationships between various descriptions and identities <cit.>. It is noteworthy that the recent popularity of text-to-image generation models, as a novel approach in pedestrian re-identification, holds significant potential, as these models generate graphics from textual descriptions for retrieval. Given that cameras in the real world do not always capture images from an available query library, customizing the generation of corresponding target entities based on eyewitness text is meaningful. However, due to the insufficient maturity of text-to-image generation model controllability, we remain optimistic about this direction.








§ FORESIGHT




 §.§ A Novel Baseline

Since the current text-based person Re-ID tasks being based on disparate datasets, specialized models are separately trained on large samples. However, in the real world, the lack of a uniform style (resolution, size, angle, brightness) in pedestrian Re-ID data is often due to the constraints imposed by different specifications of data collection devices (cameras). Retrieving arbitrary pedestrian targets in an open-world scenario becomes quite challenging for dedicated models. There is an urgent need for a new approach that enables zero-shot retrieval of targets from any gallery. On the other hand, text-to-image retrieval fundamentally relies on comparing cross-modal data distributions from small datasets through contrastive learning. If the gallery itself lacks target pedestrians, this results in ineffective retrieval, contradicting the concept of open-world pedestrian recognition. Inspired by the controllable generation retrieval in text-to-image for pedestrians, we propose a novel baseline model for text-based person re-identification, which involves re-retrieving target images from text to image.

As illustrated in Figure <ref>, the entire text-based pedestrian image generation guided Re-ID baseline architecture is demonstrated. The initial step involves encoding natural language descriptions of pedestrians and utilizing a diffusion model to generate corresponding pedestrian images. Subsequently, these generated pedestrian images are employed in the task of pedestrian re-identification. Given that current text-to-image generation models primarily focus on coarse-grained overall image generation tasks, emphasizing inter-class coordination among different objects rather than concentrating on intra-class distinctiveness in human images, and while simultaneously considering authenticity, a strategy based on human body key-point assistance is adopted. This strategy involves extracting features from human body key-points to assist in controlling the pedestrian generation process, ensuring compliance with the requirement for the presence of the majority of the human body in the pedestrian re-identification task. The generated pedestrian images serve as queries for retrieval and are input into the backbone network for visual feature extraction. To enhance retrieval accuracy, emphasis is placed on restoring features with the highest relevance to the original image. For instance, in comparison to generated images, sensitivity to channel attention may be more pronounced in original images. Various attention mechanisms are employed to assess their impact on retrieval effectiveness. However, due to the coarser granularity of linguistic descriptions compared to images, the generation of pedestrian images may not be sufficiently detailed in terms of the style and material of the pedestrian's clothing. For example, the textual description may be as broad as "black coat", but the image may be as specific as "short black leather jacket", which results in feature learning inaccuracy. In order to improve the accuracy of localized feature generation for pedestrians in a realistic open set, an approach using cross-attention based image editing is recommended. After using natural language instructions, the local features of pedestrians are controlled to change, and the witness's memory of the local features for the target person is refined and specialized. Finally, the results of pedestrian retrieval are obtained through different optimization strategies.







 §.§ Experiment & Results

For A) Text-based pedestrian generation module, We employed the Frozen CLIP Text Encoder <cit.> to encode pedestrian descriptions, followed by inputting them into stable diffusion <cit.> to generate pedestrian images. The denoising module utilized the cross-attention U-Net <cit.> backbone, which proves beneficial for the generation of high-quality pedestrian images. For pedestrian pose assistance, we adopt ControlNet <cit.> for Pedestrian integrity control, which ensures that the generated pedestrian image has full body. Specifically, OpenPose <cit.> was employed for detecting key points on the pedestrian body. Subsequently, pose features were extracted through skip connections and zero convolutions, facilitating their incorporation into pedestrian generation. 

In the part of B) pedestrian local feature refinement, we input the local image editing instructions and then use GPT-3 <cit.> to rewrite and replace the pedestrian image generation instructions. Re-inputting the rewritten text cue and the pedestrian image before rewriting into the stable diffusion model, the visual and textual features of pedestrians are fused using the cross-attention layer, and the image can be edited by editing the text only to realize the mapping from the source image to the target image <cit.>. As a comparative analysis, we also attempted fine-tuning the baseline model using a subset of text-image pairs data based on text-driven pedestrian re-identification. In the retrieval module, the generated pedestrian images were employed as queries. 

In the C) pedestrian Re-ID system, ResNet-50 <cit.> served as the visual backbone, incorporating self-attention mechanisms. The Part-based Convolutional Baseline (PCB)<cit.> strategy was employed for segmenting human bodies to extract feature maps, and random horizontal image flipping was applied for data augmentation. The training process involved the use of Instance Batch Normalization (IBN) <cit.>strategy, with a batch size set to 32, droprate at 0.5, erasing probability at 0.2, ins-gamma set to 32, learning rate at 0.05, and weight decay at 0.0005. Furthermore, the training amalgamated ID loss, Contrastive loss, and Triplet loss as the loss functions. The optimization utilized SGD optimizera with momentum, and the model was trained for 60 epochs.


As depicted in the Table  experiment results, we evaluated the proposed Text-Based Pedestrian Image Generation Guided Re-ID (TBPGR) baseline architecture on three widely used text-based person Re-ID datasets (CUHK-PEDES, RSTPReid, ICFG-PEDES). Specifically, retrieval of pedestrian images was generated using the generation modules with and without fine-tuned Stable Diffusion, respectively. We report the performance of visual representation modules DenseNet, EfficientNet Swin-Transformer and ResNet on the task for each dataset and each condition of with and without fine tuning. 
ResNet generally shows better performance. After the image generation in the intermediate step, we qualitatively analyzed the content of the images. It was found that the quality of the pedestrian images generated before fine-tuning was higher, and the quality of the images generated after fine-tuning became lower, but the retrieval efficiency was slightly improved, conjecturing that the quality of the pedestrian image text data pairs had some effect on the data distribution. 
Besides, since the CUHK-PEDES dataset retrieval effect is better than other datasets, it is hypothesized that it is because the generation of pedestrian images seldom involves the factors such as lens occlusion and illumination changes, which have some implications on the pedestrian retrieval performance, as shown in Figure  <ref>.

Although our baseline model(TBPGR) is still not powerful enough in comparison to traditional training approaches based on implicit spatial alignment of text and images, we believe that such an open-retrieval paradigm is far more valuable in future Open Set text-based person Re-ID scenarios than just pursuing the accuracy of the present moment.









§ ACKNOWLEDGMENTS

The authors would like to thank all the anonymous reviewers and associate editors for their valuable comments to improve the paper. The authors would also like to thank the authors of the three publicly available datasets.

















 







1
IEEEtran

aggarwalTextbasedPersonSearch2020a
S. Aggarwal, R. V. Babu, and A. Chakraborty, “Text-based Person Search via
  Attribute-aided Matching,” in 2020 IEEE Winter Conference on
  Applications of Computer Vision (WACV), pp. 2606–2614, Mar.
  2020.

ALIGNScalingVisual
“ALIGN: Scaling Up Visual and Vision-Language Representation
  Learning With Noisy Text Supervision – Google AI Blog.”
  https://ai.googleblog.com/2021/05/align-scaling-up-visual-and-vision.html.

baiRaSaRelationSensitivity2023
Y. Bai, M. Cao, D. Gao, Z. Cao, C. Chen, Z. Fan, L. Nie, and M. Zhang,
  “RaSa: Relation and Sensitivity Aware Representation Learning
  for Text-based Person Search,” May 2023.

bedagkar-galaSurveyApproachesTrends2014b
A. Bedagkar-Gala and S. K. Shah, “A survey of approaches and trends in
  person re-identification,” Image and Vision Computing, vol. 32,
  pp. 270–286, Apr. 2014.

birdNLTKNaturalLanguage2006
S. Bird, “NLTK: The natural language toolkit,” in Proceedings of the
  COLING/ACL on Interactive Presentation Sessions, COLING-ACL
  '06, (USA), pp. 69–72, Association for Computational Linguistics, July
  2006.

caiParallelDataAugmentation2022a
H.-Q. Cai, X. Li, Y. Ji, Y. Li, and C.-P. Liu, “Parallel Data Augmentation
  for Text-based Person Re-identification,” in 2022 International
  Joint Conference on Neural Networks (IJCNN), pp. 1–8, July 2022.

caoSymbioticAdversarialLearning2020
Y.-T. Cao, J. Wang, and D. Tao, “Symbiotic Adversarial Learning for
  Attribute-Based Person Search,” in Computer Vision
  – ECCV 2020: 16th European Conference, Glasgow,
  UK, August 23–28, 2020, Proceedings, Part XIV,
  (Berlin, Heidelberg), pp. 230–247, Springer-Verlag, Aug. 2020.

CascadeAttentionNetwork2018
“Cascade Attention Network for Person Search: Both Image and
  Text-Image Similarity Selection.”
  https://deepai.org/publication/cascade-attention-network-for-person-search-both-image-and-text-image-similarity-selection,
  Sept. 2018.

chaharStudyDeepConvolutional2017
H. Chahar and N. Nain, “A Study on Deep Convolutional Neural Network
  Based Approaches for Person Re-identification,” in Pattern
  Recognition and Machine Intelligence (B. U. Shankar, K. Ghosh, D. P.
  Mandal, S. S. Ray, D. Zhang, and S. K. Pal, eds.), Lecture Notes in
  Computer Science, (Cham), pp. 543–548, Springer International
  Publishing, 2017.

changpinyoConceptual12MPushing2021
S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, “Conceptual 12M:
  Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual
  Concepts,” in Proceedings of the IEEE/CVF Conference on
  Computer Vision and Pattern Recognition, pp. 3558–3568, 2021.

chechikLargeScaleOnline2010
G. Chechik, V. Sharma, U. Shalit, and S. Bengio, “Large Scale Online
  Learning of Image Similarity Through Ranking,” The Journal of
  Machine Learning Research, vol. 11, pp. 1109–1135, Mar. 2010.

chenCrossModalKnowledgeAdaptation2021
Y. Chen, R. Huang, H. Chang, C. Tan, T. Xue, and B. Ma, “Cross-Modal
  Knowledge Adaptation for Language-Based Person Search,” IEEE
  Transactions on Image Processing, vol. 30, pp. 4057–4069, 2021.

chenCrossModalRetrievalHeterogeneous2022
D. Chen, M. Wang, H. Chen, L. Wu, J. Qin, and W. Peng, “Cross-Modal
  Retrieval with Heterogeneous Graph Embedding,” in Proceedings of
  the 30th ACM International Conference on Multimedia, MM '22,
  (New York, NY, USA), pp. 3291–3300, Association for Computing Machinery,
  Oct. 2022.

chenEndtoEndUnsupervisedVisionandLanguage2022
C. Chen, P. Li, M. Sun, and Y. Liu, “End-to-End Unsupervised
  Vision-and-Language Pre-training with Referring Expression Matching,”
  in Conference on Empirical Methods in Natural Language
  Processing, 2022.

chengPersonReidentificationMultiChannel2016
D. Cheng, Y. Gong, S. Zhou, J. Wang, and N. Zheng, “Person
  Re-identification by Multi-Channel Parts-Based CNN with Improved
  Triplet Loss Function,” in 2016 IEEE Conference on Computer
  Vision and Pattern Recognition (CVPR), pp. 1335–1344, June 2016.

chenImprovingDeepVisual2018
D. Chen, H. Li, X. Liu, Y. Shen, J. Shao, Z. Yuan, and X. Wang, “Improving
  Deep Visual Representation for Person Re-identification by Global
  and Local Image-language Association,” in Proceedings of the
  European Conference on Computer Vision (ECCV), pp. 54–70, 2018.

chenImprovingTextBasedPerson2018
T. Chen, C. Xu, and J. Luo, “Improving Text-Based Person Search by
  Spatial Matching and Adaptive Threshold,” in 2018 IEEE Winter
  Conference on Applications of Computer Vision (WACV),
  pp. 1879–1887, Mar. 2018.

chenIntegratingInformationTheory2021
W. Chen, Y. Liu, E. M. Bakker, and M. S. Lew, “Integrating information theory
  and adversarial learning for cross-modal retrieval,” Pattern
  Recognition, vol. 117, p. 107983, Sept. 2021.

chenKeypointMessagePassing2022
D. Chen, A. Doering, S. Zhang, J. Yang, J. Gall, and B. Schiele, “Keypoint
  Message Passing for Video-Based Person Re-identification,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36,
  pp. 239–247, June 2022.

chenTIPCBSimpleEffective2022
Y. Chen, G. Zhang, Y. Lu, Z. Wang, and Y. Zheng, “TIPCB: A simple but
  effective part-based convolutional baseline for text-based person search,”
  Neurocomputing, vol. 494, pp. 171–181, July 2022.

chenTripletLossDeep2017
W. Chen, X. Chen, J. Zhang, and K. Huang, “Beyond triplet loss: A deep
  quadruplet network for person re-identification,” Apr. 2017.

chenUNITERUNiversalImageTExt2020
Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu,
  “UNITER: UNiversal Image-TExt Representation Learning,” in Computer Vision – ECCV 2020 (A. Vedaldi, H. Bischof,
  T. Brox, and J.-M. Frahm, eds.), Lecture Notes in Computer Science,
  (Cham), pp. 104–120, Springer International Publishing, 2020.

chenyangTexttoImagePersonSearch2022a
Z. Chenyang, F. Jun, and W. Jiaqing, “Text-to-Image Person Search Based on
  SSAN Model and Re-rank Post-Processing,” in 2022
  International Conference on Computer Engineering and Artificial
  Intelligence (ICCEAI), pp. 409–413, July 2022.

chopraLearningSimilarityMetric2005
S. Chopra, R. Hadsell, and Y. Lecun, Learning a Similarity Metric
  Discriminatively, with Application to Face Verification, vol. 1.
July 2005.

cormierUPARChallengePedestrian2023
M. Cormier, A. Specker, J. C. S. Jacques, L. Florin, J. Metzler, T. B.
  Moeslund, K. Nasrollahi, S. Escalera, and J. Beyerer, “UPAR Challenge:
  Pedestrian Attribute Recognition and Attribute-based Person Retrieval
  - Dataset, Design, and Results,” 2023 IEEE/CVF Winter
  Conference on Applications of Computer Vision Workshops (WACVW),
  pp. 166–175, Jan. 2023.

dengImageNetLargescaleHierarchical2009b
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet:
  A large-scale hierarchical image database,” in 2009 IEEE
  Conference on Computer Vision and Pattern Recognition,
  pp. 248–255, June 2009.

dengPedestrianAttributeRecognition2014a
Y. DENG, P. Luo, C. C. Loy, and X. Tang, “Pedestrian Attribute Recognition
  At Far Distance,” in Proceedings of the 22nd ACM International
  Conference on Multimedia, MM '14, (New York, NY, USA),
  pp. 789–792, Association for Computing Machinery, Nov. 2014.

devlinBERTPretrainingDeep2019
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training
  of Deep Bidirectional Transformers for Language Understanding,” May
  2019.

dingSemanticallySelfAlignedNetwork2021
Z. Ding, C. Ding, Z. Shao, and D. Tao, “Semantically Self-Aligned Network
  for Text-to-Image Part-aware Person Re-identification,” ArXiv,
  July 2021.

dongPersonSearchText2019
Q. Dong, X. Zhu, and S. Gong, “Person Search by Text Attribute Query As
  Zero-Shot Learning,” 2019 IEEE/CVF International Conference on
  Computer Vision (ICCV), pp. 3651–3660, Oct. 2019.

dosovitskiyImageWorth16x162023
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai,
  T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit,
  and N. Houlsby, “An Image is Worth 16x16 Words: Transformers
  for Image Recognition at Scale,” in International
  Conference on Learning Representations, Mar. 2023.

duanDeepLocalizedMetric2018
Y. Duan, J. Lu, J. Feng, and J. Zhou, “Deep Localized Metric Learning,”
  IEEE Transactions on Circuits and Systems for Video Technology,
  vol. 28, pp. 2644–2656, Oct. 2018.

engilbergeFindingBeansBurgers2018
M. Engilberge, L. Chevallier, P. Perez, and M. Cord, “Finding Beans in
  Burgers: Deep Semantic-Visual Embedding with Localization,” in
  2018 IEEE/CVF Conference on Computer Vision and Pattern
  Recognition, (Salt Lake City, UT), pp. 3984–3993, IEEE, June 2018.

faghriVSEImprovingVisualSemantic2017
F. Faghri, D. J. Fleet, J. Kiros, and S. Fidler, “VSE++: Improving
  Visual-Semantic Embeddings with Hard Negatives,” in British
  Machine Vision Conference, July 2017.

farooqAXMNetImplicitCrossModal2022
A. Farooq, M. Awais, J. Kittler, and S. S. Khalid, “AXM-Net: Implicit
  Cross-Modal Feature Alignment for Person Re-identification,” in Proceedings of the AAAI Conference on Artificial Intelligence,
  vol. 36, pp. 4477–4485, June 2022.

farooqConvolutionalBaselinePerson2020
A. Farooq, M. Awais, F. Yan, J. Kittler, A. Akbari, and S. S. Khalid, “A
  Convolutional Baseline for Person Re-Identification Using Vision and
  Language Descriptions,” Feb. 2020.

farooqCrossModalPerson2020
A. Farooq, M. Awais, J. Kittler, A. Akbari, and S. S. Khalid, “Cross Modal
  Person Re-identification with Visual-Textual Queries,” in 2020
  IEEE International Joint Conference on Biometrics (IJCB),
  pp. 1–8, Sept. 2020.

frolovAdversarialTexttoImageSynthesis2021
S. Frolov, T. Hinz, F. Raue, J. Hees, and A. Dengel, “Adversarial
  Text-to-Image Synthesis: A Review,” Neural Networks, vol. 144,
  pp. 187–209, Dec. 2021.

galiyawalaPersonRetrievalSurveillance2021
H. Galiyawala and M. S. Raval, “Person retrieval in surveillance using textual
  query: A review,” Multimedia Tools and Applications, vol. 80,
  pp. 27343–27383, July 2021.

gaoConditionalFeatureLearning2022
C. Gao, G. Cai, X. Jiang, F. Zheng, J. Zhang, Y. Gong, F. Lin, X. Sun, and
  X. Bai, “Conditional Feature Learning Based Transformer for Text-Based
  Person Search,” IEEE Transactions on Image Processing, vol. 31,
  pp. 6097–6108, 2022.

gaoContextualNonLocalAlignment2021
C. Gao, G. Cai, X. Jiang, F. Zheng, J. Zhang, Y. Gong, P. Peng, X.-W. Guo, and
  X. Sun, “Contextual Non-Local Alignment over Full-Scale
  Representation for Text-Based Person Search,” ArXiv, Jan. 2021.

geVisualTextualAssociationHardest2019a
J. Ge, G. Gao, and Z. Liu, “Visual-Textual Association with Hardest
  and Semi-Hard Negative Pairs Mining for Person Search,” Dec. 2019.

gomezChapterSelfSupervisedLearning2019
R. Gomez, L. Gomez, J. Gibert, and D. Karatzas, “Chapter 9 - Self-Supervised
  Learning from Web Data for Multimodal Retrieval,” in Multimodal Scene Understanding (M. Y. Yang, B. Rosenhahn, and V. Murino,
  eds.), pp. 279–306, Academic Press, Jan. 2019.

grayEvaluatingAppearanceModels2007
D. Gray, S. Brennan, and H. Tao, “Evaluating appearance models for
  recognition, reacquisition, and tracking,” Proc. IEEE Int. Workshop
  Vis. Surveill. Perform. Eval. Tracking Surveill., Oct., Jan. 2007.

guLookImagineMatch2018
J. Gu, J. Cai, S. Joty, L. Niu, and G. Wang, “Look, Imagine and Match:
  Improving Textual-Visual Cross-Modal Retrieval with Generative
  Models,” June 2018.

hanFashionViLFashionFocusedVisionandLanguage2022
X. Han, L. Yu, X. Zhu, L. Zhang, Y.-Z. Song, and T. Xiang, “FashionViL:
  Fashion-Focused Vision-and-Language Representation Learning,” arXiv,
  2022.

hanTextBasedPersonSearch2021a
X. Han, S. He, L. Zhang, and T. Xiang, “Text-Based Person Search with
  Limited Data,” in BMVC 2021, arXiv, Oct. 2021.

heDeepResidualLearning2016
K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image
  Recognition,” in 2016 IEEE Conference on Computer Vision and
  Pattern Recognition (CVPR), pp. 770–778, June 2016.

hermansDefenseTripletLoss2017
A. Hermans, L. Beyer, and B. Leibe, “In Defense of the Triplet Loss
  for Person Re-Identification,” Nov. 2017.

huangDeepLearningVisibleinfrared2023
N. Huang, J. Liu, Y. Miao, Q. Zhang, and J. Han, “Deep learning for
  visible-infrared cross-modality person re-identification: A comprehensive
  review,” Information Fusion, vol. 91, pp. 396–411, Mar. 2023.

huangSeeingOutBOx2021
Z. Huang, Z. Zeng, Y. Huang, B. Liu, D. Fu, and J. Fu, “Seeing Out of
  tHe bOx: End-to-End Pre-training for Vision-Language Representation
  Learning,” 2021 IEEE/CVF Conference on Computer Vision and Pattern
  Recognition (CVPR), pp. 12971–12980, June 2021.

iodiceTextAttributeAggregation2020
S. Iodice and K. Mikolajczyk, “Text Attribute Aggregation and Visual
  Feature Decomposition for Person Search,” in 2020 British
  Machine Vision Conference, 2020.

islamPersonSearchNew2020
K. Islam, “Person search: New paradigm of person re-identification: A
  survey and outlook of recent works,” Image and Vision Computing,
  vol. 101, p. 103970, Sept. 2020.

jeongASMRLearningAttributeBased2021
B. Jeong, J. Park, and S. Kwak, “ASMR: Learning Attribute-Based Person
  Search with Adaptive Semantic Margin Regularizer,” 2021 IEEE/CVF
  International Conference on Computer Vision (ICCV), pp. 11996–12005, Oct.
  2021.

jiangCrossModalImplicitRelation2023b
D. Jiang and M. Ye, “Cross-Modal Implicit Relation Reasoning and
  Aligning for Text-to-Image Person Retrieval,” in Proceedings of
  the IEEE/CVF Conference on Computer Vision and Pattern
  Recognition, pp. 2787–2797, 2023.

jiaScalingVisualVisionLanguage2021
C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung,
  Z. Li, and T. Duerig, “Scaling Up Visual and Vision-Language
  Representation Learning With Noisy Text Supervision,” in Proceedings
  of the 38th International Conference on Machine Learning,
  pp. 4904–4916, PMLR, July 2021.

jiAsymmetricCrossScaleAlignment2022
Z. Ji, J. Hu, D. Liu, L. Y. Wu, and Y. Zhao, “Asymmetric Cross-Scale
  Alignment for Text-Based Person Search,” IEEE Transactions on
  Multimedia, pp. 1–11, 2022.

jingCrossModalCrossDomainMoment2020
Y. Jing, W. Wang, L. Wang, and T. Tan, “Cross-Modal Cross-Domain Moment
  Alignment Network for Person Search,” 2020 IEEE/CVF Conference on
  Computer Vision and Pattern Recognition (CVPR), pp. 10675–10683, June 2020.

jingPoseGuidedJointGlobal2018
Y. Jing, C. Si, J. Wang, W. Wang, L. Wang, and T. Tan, “Pose-Guided Joint
  Global and Attentive Local Matching Network for Text-Based Person
  Search,” Sept. 2018.

jingPoseGuidedMultiGranularityAttention2020a
Y. Jing, C. Si, J. Wang, W. Wang, L. Wang, and T. Tan, “Pose-Guided
  Multi-Granularity Attention Network for Text-Based Person Search,”
  Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34,
  pp. 11189–11196, Apr. 2020.

karanamSystematicEvaluationBenchmark2019
S. karanam, M. Gou, Z. Wu, A. Rates-Borras, O. Camps, and R. J. Radke, “A
  Systematic Evaluation and Benchmark for Person Re-Identification:
  Features, Metrics, and Datasets,” IEEE Transactions on
  Pattern Analysis and Machine Intelligence, vol. 41, pp. 523–536, Mar. 2019.

laviSurveyReliableDeep2020
B. Lavi, I. Ullah, M. Fatan, and A. Rocha, “Survey on Reliable Deep
  Learning-Based Person Re-Identification Models: Are We There Yet?,”
  Apr. 2020.

leeStackedCrossAttention2018
K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He, “Stacked Cross Attention for
  Image-Text Matching,” July 2018.

liAlignFuseVision2021
J. Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, and S. Hoi,
  “Align before Fuse: Vision and Language Representation Learning
  with Momentum Distillation,” in Neural Information Processing
  Systems, July 2021.

liaoPersonReidentificationLocal2015
S. Liao, Y. Hu, Xiangyu Zhu, and S. Z. Li, “Person re-identification by
  Local Maximal Occurrence representation and metric learning,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR), (Boston, MA, USA), pp. 2197–2206, IEEE, June 2015.

liDiversityRegularizedSpatiotemporal2018
S. Li, S. Bak, P. Carr, and X. Wang, “Diversity Regularized Spatiotemporal
  Attention for Video-based Person Re-identification,” Mar. 2018.

liHumanReidentificationTransferred2013
W. Li, R. Zhao, and X. Wang, “Human Reidentification with Transferred
  Metric Learning,” in Computer Vision – ACCV 2012
  (K. M. Lee, Y. Matsushita, J. M. Rehg, and Z. Hu, eds.), Lecture Notes in
  Computer Science, (Berlin, Heidelberg), pp. 31–44, Springer, 2013.

liHybridAttentionNetwork2020
Y. Li, H. Xu, and J. Xiao, “Hybrid Attention Network for Language-Based
  Person Search,” Sensors, vol. 20, p. 5279, Jan. 2020.

liIdentityAwareTextualVisualMatching2017
S. Li, T. Xiao, H. Li, W. Yang, and X. Wang, “Identity-Aware Textual-Visual
  Matching With Latent Co-Attention,” in Proceedings of the IEEE
  International Conference on Computer Vision, pp. 1890–1899, 2017.

liJointTokenFeature2022a
S. Li, A. Lu, Y. Huang, C. Li, and L. Wang, “Joint Token and Feature
  Alignment Framework for Text-Based Person Search,” IEEE Signal
  Processing Letters, vol. 29, pp. 2238–2242, 2022.

liLearningSemanticAlignedFeature2021
S. Li, M. Cao, and M. Zhang, “Learning Semantic-Aligned Feature
  Representation for Text-based Person Search,” Dec. 2021.

linImprovingPersonReidentification2019
Y. Lin, L. Zheng, Z. Zheng, Y. Wu, Z. Hu, C. Yan, and Y. Yang, “Improving
  person re-identification by attribute and identity learning,” Pattern
  Recognition, vol. 95, pp. 151–161, Nov. 2019.

linPersonSearchChallenges2021a
X. Lin, P. Ren, Y. Xiao, X. Chang, and A. Hauptmann, “Person search challenges
  and solutions: A survey,” in Proceedings of the Thirtieth
  International Joint Conference on Artificial Intelligence,
  pp. 4500–4507, Association for the Advancement of Artificial Intelligence
  (AAAI), 2021.

liPedestrianReidentificationBased2021
C. Li, X. Yang, K. Yin, Y. Chang, Z. Wang, and G. Yin, “Pedestrian
  re-identification based on attribute mining and reasoning,” IET Image
  Processing, vol. 15, no. 11, pp. 2399–2411, 2021.

liPersonSearchNatural2017
S. Li, T. Xiao, H. Li, B. Zhou, D. Yue, and X. Wang, “Person Search With
  Natural Language Description,” in Proceedings of the IEEE
  Conference on Computer Vision and Pattern Recognition,
  pp. 1970–1979, 2017.

liPersonTextImageMatching2022
F. Li, H. Zhou, H. Li, Y. Zhang, and Z. Yu, “Person Text-Image Matching
  via Text-Feature Interpretability Embedding and External Attack Node
  Implantation,” 2022.

liRichlyAnnotatedDataset2016a
D. Li, Z. Zhang, X. Chen, H. Ling, and K. Huang, “A Richly Annotated
  Dataset for Pedestrian Attribute Recognition,” Apr. 2016.

liRichlyAnnotatedPedestrian2019
D. Li, Z. Zhang, X. Chen, and K. Huang, “A Richly Annotated Pedestrian
  Dataset for Person Retrieval in Real Surveillance Scenarios,” IEEE Transactions on Image Processing, vol. 28, pp. 1575–1590, Apr. 2019.

liTransformerBasedLanguagePersonSearch2021
H. Li, J. Xiao, M. Sun, E. Lim, and Y. Zhao, “Transformer-Based
  Language-Person Search With Multiple Region Slicing,” IEEE
  Transactions on Circuits and Systems for Video Technology, Apr. 2021.

liUAVHumanLargeBenchmark2021
T. Li, J. Liu, W. Zhang, Y. Ni, W. Wang, and Z. Li, “UAV-Human: A Large
  Benchmark for Human Behavior Understanding with Unmanned Aerial
  Vehicles,” Aug. 2021.

liuDeepAdversarialGraph2019
J. Liu, Z.-J. Zha, R. Hong, M. Wang, and Y. Zhang, “Deep Adversarial Graph
  Attention Convolution Network for Text-Based Person Search,” in Proceedings of the 27th ACM International Conference on Multimedia,
  MM '19, (New York, NY, USA), pp. 665–673, Association for Computing
  Machinery, Oct. 2019.

liuEfficientTextbasedPerson2022
T.-Y. Liu, C. Zhu, and L. Yang, “Efficient Text-based Person Search via
  Single-stage Identity-guided Attribute Parsing and Alignment,” 2022 26th International Conference on Pattern Recognition (ICPR),
  pp. 4111–4117, Aug. 2022.

liuGraphStructuredNetwork2020a
C. Liu, Z. Mao, T. Zhang, H. Xie, B. Wang, and Y. Zhang, “Graph Structured
  Network for Image-Text Matching,” in 2020 IEEE/CVF
  Conference on Computer Vision and Pattern Recognition (CVPR),
  pp. 10918–10927, June 2020.

liuHydraPlusNetAttentiveDeep2017
X. Liu, H. Zhao, M. Tian, L. Sheng, J. Shao, S. Yi, J. Yan, and X. Wang,
  “HydraPlus-Net: Attentive Deep Features for Pedestrian
  Analysis,” Sept. 2017.

liuSwinTransformerHierarchical2021
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin
  Transformer: Hierarchical Vision Transformer using Shifted
  Windows,” Aug. 2021.

liuTS2NetTokenShift2022
Y. Liu, P. Xiong, L. Xu, S. Cao, and Q. Jin, “TS2-Net: Token Shift and
  Selection Transformer for Text-Video Retrieval,” in Computer
  Vision – ECCV 2022: 17th European Conference, Tel
  Aviv, Israel, October 23–27, 2022, Proceedings,
  Part XIV, (Berlin, Heidelberg), pp. 319–335, Springer-Verlag, Oct.
  2022.

luoSpectralFeatureTransformation2018
C. Luo, Y. Chen, N. Wang, and Z. Zhang, “Spectral Feature Transformation
  for Person Re-identification,” Nov. 2018.

manningStanfordCoreNLPNatural2014
C. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. Bethard, and D. McClosky,
  “The Stanford CoreNLP Natural Language Processing Toolkit,” in Proceedings of 52nd Annual Meeting of the Association for
  Computational Linguistics: System Demonstrations, (Baltimore,
  Maryland), pp. 55–60, Association for Computational Linguistics, June
  2014.

mathurBriefSurveyDeep2020
N. Mathur, S. Mathur, D. Mathur, and P. Dadheech, “A Brief Survey of
  Deep Learning Techniques for Person Re-identification,” in 2020
  3rd International Conference on Emerging Technologies in Computer
  Engineering: Machine Learning and Internet of Things
  (ICETCE), pp. 129–138, Feb. 2020.

mingDeepLearningbasedPerson2022
Z. Ming, M. Zhu, X. Wang, J. Zhu, J. Cheng, C. Gao, Y. Yang, and X. Wei, “Deep
  learning-based person re-identification methods: A survey and outlook of
  recent works,” Image and Vision Computing, vol. 119, p. 104394, Mar.
  2022.

niExpandingLanguageImagePretrained2022
B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, and H. Ling,
  “Expanding Language-Image Pretrained Models for General Video
  Recognition,” in Computer Vision – ECCV 2022
  (S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, eds.),
  Lecture Notes in Computer Science, (Cham), pp. 1–18, Springer
  Nature Switzerland, 2022.

niuCrossmodalCooccurrenceAttributes2022
K. Niu, L. Huang, Y. Huang, P. Wang, L. Wang, and Y. Zhang, “Cross-modal
  Co-occurrence Attributes Alignments for Person Search by
  Language,” in Proceedings of the 30th ACM International
  Conference on Multimedia, MM '22, (New York, NY, USA),
  pp. 4426–4434, Association for Computing Machinery, Oct. 2022.

niuFusingTwoDirections2019a
K. Niu, Y. Huang, and L. Wang, “Fusing Two Directions in Cross-Domain
  Adaption for Real Life Person Search by Language,” in 2019
  IEEE/CVF International Conference on Computer Vision Workshop
  (ICCVW), pp. 1815–1818, Oct. 2019.

niuImprovingDescriptionBasedPerson2020
K. Niu, Y. Huang, W. Ouyang, and L. Wang, “Improving Description-Based
  Person Re-Identification by Multi-Granularity Image-Text Alignments,”
  IEEE Transactions on Image Processing, vol. 29, pp. 5542–5556, 2020.

niuTextualDependencyEmbedding2020
K. Niu, Y. Huang, and L. Wang, “Textual Dependency Embedding for Person
  Search by Language,” Proceedings of the 28th ACM International
  Conference on Multimedia, pp. 4032–4040, Oct. 2020.

PapersCodeUAVHuman
“Papers with Code - UAV-Human: A Large Benchmark for Human
  Behavior Understanding with Unmanned Aerial Vehicles.”
  https://paperswithcode.com/paper/uav-human-a-large-benchmark-for-human.

parkLearningDiscriminativePart2020a
J. Park, B. Jeong, J. Shin, J. Lee, and S. Kwak, “Learning Discriminative
  Part Features Through Attentions For Effective And Scalable Person
  Search,” in 2020 IEEE International Conference on Image
  Processing (ICIP), pp. 2376–2380, Oct. 2020.

pengDeepLearningBasedOccluded2023
Y. Peng, J. Wu, B. Xu, C. Cao, X. Liu, Z. Sun, and Z. He, “Deep
  Learning-Based Occluded Person Re-Identification: A Survey,” ACM Transactions on Multimedia Computing, Communications, and Applications,
  July 2023.

perwaizUbiquitousVisionTransformers2023
N. Perwaiz, M. Shahzad, and M. M. Fraz, “Ubiquitous vision of transformers for
  person re-identification,” Machine Vision and Applications, vol. 34,
  p. 27, Feb. 2023.

radfordLearningTransferableVisual2021
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry,
  A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning
  Transferable Visual Models From Natural Language Supervision,” in Proceedings of the 38th International Conference on Machine
  Learning, pp. 8748–8763, PMLR, July 2021.

reedLearningDeepRepresentations2016
S. Reed, Z. Akata, B. Schiele, and H. Lee, “Learning Deep Representations
  of Fine-grained Visual Descriptions,” May 2016.

sarafianosAdversarialRepresentationLearning2019
N. Sarafianos, X. Xu, and I. A. Kakadiaris, “Adversarial Representation
  Learning for Text-to-Image Matching,” in Proceedings of the
  IEEE/CVF International Conference on Computer Vision,
  pp. 5814–5824, 2019.

scheirerMultiattributeSpacesCalibration2012
W. J. Scheirer, N. Kumar, P. N. Belhumeur, and T. E. Boult, “Multi-attribute
  spaces: Calibration for attribute fusion and similarity search,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition,
  pp. 2933–2940, June 2012.

schusterBidirectionalRecurrentNeural1997
M. Schuster and K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Transactions on Signal Processing, vol. 45, pp. 2673–2681, Nov. 1997.

shahNaturalLanguagePerson2018
A. Shah and T. Vuong, “Natural Language Person Search Using Deep
  Reinforcement Learning,” Sept. 2018.

shaoLearningGranularityUnifiedRepresentations2022
Z. Shao, X. Zhang, M. Fang, Z. Lin, J. Wang, and C. Ding, “Learning
  Granularity-Unified Representations for Text-to-Image Person
  Re-identification,” in Proceedings of the 30th ACM International
  Conference on Multimedia, (Lisboa Portugal), pp. 5566–5574, ACM,
  Oct. 2022.

shuSeeFinerSee2023
X. Shu, W. Wen, H. Wu, K. Chen, Y. Song, R. Qiao, B. Ren, and X. Wang, “See
  Finer, See More: Implicit Modality Alignment for Text-Based
  Person Retrieval,” in Computer Vision – ECCV 2022
  Workshops (L. Karlinsky, T. Michaeli, and K. Nishino, eds.), Lecture
  Notes in Computer Science, (Cham), pp. 624–641, Springer Nature
  Switzerland, 2023.

singhComprehensiveSurveyPerson2022
N. K. Singh, M. Khare, and H. B. Jethva, “A comprehensive survey on person
  re-identification approaches: Various aspects,” Multimedia Tools and
  Applications, vol. 81, pp. 15747–15791, May 2022.

songDeepMetricLearning2015
H. O. Song, Y. Xiang, S. Jegelka, and S. Savarese, “Deep Metric Learning
  via Lifted Structured Feature Embedding,” Nov. 2015.

sunPartModelsPerson2018
Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, “Beyond Part Models:
  Person Retrieval with Refined Part Pooling (and A Strong
  Convolutional Baseline),” in Proceedings of the European
  Conference on Computer Vision (ECCV), pp. 480–496, 2018.

sunSVDNetPedestrianRetrieval2017
Y. Sun, L. Zheng, W. Deng, and S. Wang, “SVDNet for Pedestrian
  Retrieval,” Aug. 2017.

suoSimpleRobustCorrelation2022a
W. Suo, M. Sun, K. Niu, Y. Gao, P. Wang, Y. Zhang, and Q. Wu, “A Simple
  and Robust Correlation Filtering Method for Text-Based Person
  Search,” in Computer Vision - ECCV 2022 - 17th European
  Conference, Tel Aviv, Israel, October 23-27, 2022,
  Proceedings, Part XXXV (S. Avidan, G. J. Brostow, M. Cissé,
  G. M. Farinella, and T. Hassner, eds.), vol. 13695 of Lecture Notes
  in Computer Science, pp. 726–742, Springer, 2022.

tangImprovingPedestrianAttribute2019
C. Tang, L. Sheng, Z. Zhang, and X. Hu, “Improving Pedestrian Attribute
  Recognition With Weakly-Supervised Multi-Scale Attribute-Specific
  Localization,” Oct. 2019.

vaqueroAttributebasedPeopleSearch2009
D. A. Vaquero, R. S. Feris, D. Tran, L. Brown, A. Hampapur, and M. Turk,
  “Attribute-based people search in surveillance environments,” in 2009
  Workshop on Applications of Computer Vision (WACV),
  pp. 1–8, Dec. 2009.

variorSiameseLongShortTerm2016
R. R. Varior, B. Shuai, J. Lu, D. Xu, and G. Wang, “A Siamese Long
  Short-Term Memory Architecture for Human Re-Identification,” July
  2016.

vinyalsShowTellNeural2015
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and Tell: A Neural
  Image Caption Generator,” in Proceedings of the IEEE Conference
  on Computer Vision and Pattern Recognition, pp. 3156–3164, 2015.

wangAMENAdversarialMultispace2021
Z. Wang, J. Xue, A. Zhu, Y. Li, M. Zhang, and C. Zhong, “AMEN:
  Adversarial Multi-space Embedding Network for Text-Based Person
  Re-identification,” in Pattern Recognition and Computer
  Vision: 4th Chinese Conference, PRCV 2021, Beijing, China,
  October 29 – November 1, 2021, Proceedings, Part
  II, (Berlin, Heidelberg), pp. 462–473, Springer-Verlag, Oct. 2021.

wangCAIBCCapturingAllround2022
Z. Wang, A. Zhu, J. Xue, X. Wan, C. Liu, T. Wang, and Y. Li, “CAIBC:
  Capturing All-round Information Beyond Color for Text-based Person
  Retrieval,” in Proceedings of the 30th ACM International
  Conference on Multimedia, MM '22, (New York, NY, USA),
  pp. 5314–5322, Association for Computing Machinery, Oct. 2022.

wangCrossDomainPersonReidentification2021
Y. Wang, S. Yang, S. Liu, and Z. Zhang, “Cross-Domain Person
  Re-identification: A Review,” in Artificial Intelligence in
  China (Q. Liang, W. Wang, J. Mu, X. Liu, Z. Na, and X. Cai, eds.),
  Lecture Notes in Electrical Engineering, (Singapore), pp. 153–160,
  Springer, 2021.

wangDivideandMergeEmbeddingSpace2021
C. Wang, Z. Luo, Z. Zhong, and S. Li, “Divide-and-Merge the embedding
  space for cross-modality person search,” Neurocomputing, vol. 463,
  pp. 388–399, Nov. 2021.

wangExploitingTextualPotential2023
G. Wang, F. Yu, J. Li, Q. Jia, and S. Ding, “Exploiting the Textual
  Potential from Vision-Language Pre-training for Text-based Person
  Search,” Mar. 2023.

wangImageForeignLanguage2022
W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K.
  Mohammed, S. Singhal, S. Som, and F. Wei, “Image as a Foreign Language:
  BEiT Pretraining for All Vision and Vision-Language Tasks,” Aug.
  2022.

wangIMGNetInnercrossmodalAttentional2020e
Z. Wang, A. Zhu, Z. Zheng, J. Jin, Z. Xue, and G. Hua, “IMG-Net:
  Inner-cross-modal attentional multigranular network for description-based
  person re-identification,” Journal of Electronic Imaging, vol. 29,
  Aug. 2020.

wangImprovingEmbeddingLearning2022
C. Wang, Z. Luo, Y. Lin, and S. Li, “Improving embedding learning by virtual
  attribute decoupling for text-based person search,” Neural Computing
  and Applications, vol. 34, pp. 5625–5647, Apr. 2022.

wangInteractEmbedEnlargE2022
Z. Wang, C. Li, A. Zheng, R. He, and J. Tang, “Interact, Embed, and
  EnlargE: Boosting Modality-Specific Representations for Multi-Modal
  Person Re-identification,” Proceedings of the AAAI Conference on
  Artificial Intelligence, vol. 36, pp. 2633–2641, June 2022.

wangIntramodalitySurveyHeterogeneous2020
Z. Wang, Z. Wang, Y. Zheng, Y. Wu, W. Zeng, and S. Satoh, “Beyond
  Intra-modality: A Survey of Heterogeneous Person
  Re-identification,” Apr. 2020.

wangLanguagePersonSearch2019
Y. Wang, C. Bo, D. Wang, S. Wang, Y. Qi, and H. Lu, “Language Person
  Search with Mutually Connected Classification Loss,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics,
  Speech and Signal Processing (ICASSP), pp. 2057–2061, May 2019.

wangLearningFineGrainedImage2014
J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and
  Y. Wu, “Learning Fine-Grained Image Similarity with Deep Ranking,”
  in 2014 IEEE Conference on Computer Vision and Pattern
  Recognition, (Columbus, OH, USA), pp. 1386–1393, IEEE, June 2014.

wangLookYouLeap2022
Z. Wang, A. Zhu, J. Xue, X. Wan, C. Liu, T. Wang, and Y. Li, “Look Before
  You Leap: Improving Text-based Person Retrieval by Learning A
  Consistent Cross-modal Common Manifold,” Proceedings of the 30th ACM
  International Conference on Multimedia, pp. 1984–1992, Oct. 2022.

wangPartBasedMultiScaleAttention2022a
Y. Wang, D. Qi, and C. Zhao, “Part-Based Multi-Scale Attention Network
  for Text-Based Person Search,” in Pattern Recognition and
  Computer Vision (S. Yu, Z. Zhang, P. C. Yuen, J. Han, T. Tan, Y. Guo,
  J. Lai, and J. Zhang, eds.), Lecture Notes in Computer Science,
  (Cham), pp. 462–474, Springer International Publishing, 2022.

wangPersonReIdentificationCascaded2018
Y. Wang, Z. Chen, F. Wu, and G. Wang, “Person Re-Identification With
  Cascaded Pairwise Convolutions,” in Proceedings of the IEEE
  Conference on Computer Vision and Pattern Recognition,
  pp. 1470–1478, 2018.

wangPoseGuidedFeatureDisentangling2022
T. Wang, H. Liu, P. Song, T. Guo, and W. Shi, “Pose-Guided Feature
  Disentangling for Occluded Person Re-identification Based on
  Transformer,” Proceedings of the AAAI Conference on Artificial
  Intelligence, vol. 36, pp. 2540–2549, June 2022.

wangSUMSerializedUpdating2022
Z. Wang, A. Zhu, J. Xue, D. Jiang, C. Liu, Y. Li, and F. Hu, “SUM:
  Serialized Updating and Matching for text-based person retrieval,”
  Knowledge-Based Systems, vol. 248, p. 108891, July 2022.

wangSurveyPersonReidentification2018
K. Wang, H. Wang, M. Liu, X. Xing, and T. Han, “Survey on person
  re-identification based on deep learning,” CAAI Transactions on
  Intelligence Technology, vol. 3, no. 4, pp. 219–227, 2018.

wangTextbasedPersonSearch2021
C. Wang, Z. Luo, Y. Lin, and S. Li, “Text-based Person Search via
  Multi-Granularity Embedding Learning,” in Proceedings of the
  Thirtieth International Joint Conference on Artificial Intelligence,
  (Montreal, Canada), pp. 1068–1074, International Joint Conferences on
  Artificial Intelligence Organization, Aug. 2021.

wangViTAAVisualTextualAttributes2020a
Z. Wang, Z. Fang, J. Wang, and Y. Yang, “ViTAA: Visual-Textual
  Attributes Alignment in Person Search by Natural Language,” in
  Computer Vision – ECCV 2020 (A. Vedaldi,
  H. Bischof, T. Brox, and J.-M. Frahm, eds.), Lecture Notes in Computer
  Science, (Cham), pp. 402–420, Springer International Publishing, 2020.

weiFineGrainedImageAnalysis2021
X.-S. Wei, Y.-Z. Song, O. Mac Aodha, J. Wu, Y. Peng, J. Tang, J. Yang, and
  S. Belongie, “Fine-Grained Image Analysis with Deep Learning: A
  Survey,” Nov. 2021.

weiPersonTransferGAN2018a
L. Wei, S. Zhang, W. Gao, and Q. Tian, “Person Transfer GAN to Bridge
  Domain Gap for Person Re-identification,” in 2018 IEEE/CVF
  Conference on Computer Vision and Pattern Recognition, pp. 79–88,
  June 2018.

wuDeepLearningbasedMethods2019
D. Wu, S.-J. Zheng, X.-P. Zhang, C.-A. Yuan, F. Cheng, Y. Zhao, Y.-J. Lin,
  Z.-Q. Zhao, Y.-L. Jiang, and D.-S. Huang, “Deep learning-based methods for
  person re-identification: A comprehensive review,” Neurocomputing,
  vol. 337, pp. 354–371, Apr. 2019.

wuLapsCoreLanguageGuidedPerson2021
Y. Wu, Z. Yan, X. Han, G. Li, C. Zou, and S. Cui, “LapsCore:
  Language-Guided Person Search via Color Reasoning,” in Proceedings of the IEEE/CVF International Conference on Computer
  Vision, pp. 1624–1633, 2021.

wuOverviewDeepLearning2022
D. Wu, H. Huang, Q. Zhao, S. Zhang, J. Qi, and J. Hu, “Overview of deep
  learning based pedestrian attribute recognition and re-identification,” Heliyon, vol. 8, p. e12086, Dec. 2022.

xiaoJointDetectionIdentification2016b
T. Xiao, S. Li, B. Wang, L. Lin, and X. Wang, “Joint Detection and
  Identification Feature Learning for Person Search,” Apr. 2016.

xiaoJointDetectionIdentification2017a
T. Xiao, S. Li, B. Wang, L. Lin, and X. Wang, “Joint Detection and
  Identification Feature Learning for Person Search,” in 2017
  IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR), pp. 3376–3385, July 2017.

xuE2EVLPEndtoEndVisionLanguage2021
H. Xu, M. Yan, C. Li, B. Bi, S. Huang, W. Xiao, and F. Huang, “E2E-VLP:
  End-to-End Vision-Language Pre-training Enhanced by Visual
  Learning,” Proceedings of the 59th Annual Meeting of the Association
  for Computational Linguistics and the 11th International Joint Conference on
  Natural Language Processing (Volume 1: Long Papers), pp. 503–513, 2021.

xuMultilevelLanguageVision2019
H. Xu, K. He, B. A. Plummer, L. Sigal, S. Sclaroff, and K. Saenko, “Multilevel
  Language and Vision Integration for Text-to-Clip Retrieval,”
  Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33,
  pp. 9062–9069, July 2019.

yaghoubiSSSPRShortSurvey2021
E. Yaghoubi, A. Kumar, and H. Proença, “SSS-PR: A short survey of
  surveys in person re-identification,” Pattern Recognition Letters,
  vol. 143, pp. 50–57, Mar. 2021.

yanCLIPDrivenFinegrainedTextImage2022
S. Yan, N. Dong, L. Zhang, and J. Tang, “CLIP-Driven Fine-grained Text-Image
  Person Re-identification,” 2022.

yanCrossdomainPersonReidentification2022b
S. Yan, Y. Zhang, M. Xie, D. Zhang, and Z. Yu, “Cross-domain person
  re-identification with pose-invariant feature decomposition and hypergraph
  structure alignment,” Neurocomputing, vol. 467, pp. 229–241, Jan.
  2022.

yangSurveyUnsupervisedTechniques2021a
C. Yang, F. Qi, and H. Jia, “Survey on Unsupervised Techniques for
  Person Re-Identification,” in 2021 2nd International Conference
  on Computing and Data Science (CDS), pp. 161–164, Jan. 2021.

yangUnifiedTextbasedPerson2023a
S. Yang, Y. Zhou, Y. Wang, Y. Wu, L. Zhu, and Z. Zheng, “Towards Unified
  Text-based Person Retrieval: A Large-scale Multi-Attribute and
  Language Search Benchmark,” Aug. 2023.

yanImageSpecificInformationSuppression2022
S. Yan, H. Tang, L. Zhang, and J. Tang, “Image-Specific Information
  Suppression and Implicit Local Alignment for Text-based Person
  Search,” 2022.

yanImageSpecificInformationSuppression2023
S. Yan, H. Tang, L. Zhang, and J. Tang, “Image-Specific Information
  Suppression and Implicit Local Alignment for Text-based Person
  Search,” July 2023.

yanPersonReIdentificationVision2018
F. Yan, J. Kittler, and K. Mikolajczyk, “Person Re-Identification with
  Vision and Language,” in 2018 24th International Conference
  on Pattern Recognition (ICPR), pp. 2136–2141, Aug. 2018.

yeDeepLearningPerson2022
M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S. C. H. Hoi, “Deep
  Learning for Person Re-Identification: A Survey and
  Outlook,” IEEE Transactions on Pattern Analysis and Machine
  Intelligence, vol. 44, pp. 2872–2893, June 2022.

yeSpecificPersonRetrieval2015
M. Ye, C. Liang, Z. Wang, Q. Leng, J. Chen, and J. Liu, “Specific Person
  Retrieval via Incomplete Text Description,” in Proceedings of the
  5th ACM on International Conference on Multimedia Retrieval,
  ICMR '15, (New York, NY, USA), pp. 547–550, Association for Computing
  Machinery, June 2015.

yeVisibleThermalPerson2018
M. Ye, Z. Wang, X. Lan, and P. C. Yuen, “Visible Thermal Person
  Re-Identification via Dual-Constrained Top-Ranking,” in Proceedings of the Twenty-Seventh International Joint Conference on
  Artificial Intelligence, (Stockholm, Sweden), pp. 1092–1099,
  International Joint Conferences on Artificial Intelligence Organization,
  July 2018.

yinAdversarialAttributeimagePerson2018
Z. Yin, W.-S. Zheng, A. Wu, H.-X. Yu, H. Wan, X. Guo, F. Huang, and J. Lai,
  “Adversarial attribute-image person re-identification,” in Proceedings
  of the 27th International Joint Conference on Artificial
  Intelligence, IJCAI'18, (Stockholm, Sweden), pp. 1100–1106, AAAI
  Press, July 2018.

yuCascadeTransformersEndtoEnd2022
R. Yu, D. Du, R. LaLonde, D. Davila, C. Funk, A. Hoogs, and B. Clipp, “Cascade
  Transformers for End-to-End Person Search,” Mar. 2022.

zahraPersonReidentificationRetrospective2023
A. Zahra, N. Perwaiz, M. Shahzad, and M. M. Fraz, “Person re-identification:
  A retrospective on domain specific open challenges and future trends,”
  Pattern Recognition, vol. 142, p. 109669, Oct. 2023.

zellersRecognitionCognitionVisual2019
R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi, “From Recognition to
  Cognition: Visual Commonsense Reasoning,” in Proceedings of the
  IEEE/CVF Conference on Computer Vision and Pattern
  Recognition, pp. 6720–6731, 2019.

zhangDeepCrossModalProjection2018a
Y. Zhang and H. Lu, “Deep Cross-Modal Projection Learning for Image-Text
  Matching,” in Computer Vision – ECCV 2018
  (V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, eds.), Lecture
  Notes in Computer Science, (Cham), pp. 707–723, Springer
  International Publishing, 2018.

zhangGLIPv2UnifyingLocalization2022
H. Zhang, P. Zhang, X. Hu, Y.-C. Chen, L. H. Li, X. Dai, L. Wang, L. Yuan,
  J.-N. Hwang, and J. Gao, “GLIPv2: Unifying Localization and
  Vision-Language Understanding,” in Advances in Neural Information
  Processing Systems, Oct. 2022.

zhangGlobalLocalMultipleGranularity2021
L. Zhang, G. Du, F. Liu, H. Tu, and X. Shu, “Global-Local Multiple
  Granularity Learning for Cross-Modality Visible-Infrared Person
  Reidentification,” IEEE Transactions on Neural Networks and Learning
  Systems, pp. 1–11, 2021.

zhangTextbasedPersonSearch2021
S. Zhang, D. Long, Y. Gao, L. Gao, Q. Zhang, K. Niu, and Y. Zhang, “Text-based
  Person Search in Full Images via Semantic-Driven Proposal
  Generation,” ArXiv, Sept. 2021.

zhaoWeaklySupervisedTextBased2021
S. Zhao, C. Gao, Y. Shao, W.-S. Zheng, and N. Sang, “Weakly Supervised
  Text-Based Person Re-Identification,” in Proceedings of the
  IEEE/CVF International Conference on Computer Vision,
  pp. 11395–11404, 2021.

zhengDiscriminativelyLearnedCNN2018
Z. Zheng, L. Zheng, and Y. Yang, “A Discriminatively Learned CNN Embedding
  for Person Re-identification,” ACM Transactions on Multimedia
  Computing, Communications, and Applications, vol. 14, pp. 1–20, Feb. 2018.

zhengDualpathConvolutionalImageText2020
Z. Zheng, L. Zheng, M. Garrett, Y. Yang, M. Xu, and Y.-D. Shen, “Dual-path
  Convolutional Image-Text Embeddings with Instance Loss,” ACM
  Transactions on Multimedia Computing, Communications, and Applications,
  vol. 16, pp. 51:1–51:23, May 2020.

zhengHierarchicalGumbelAttention2020
K. Zheng, W. Liu, J. Liu, Z.-J. Zha, and T. Mei, “Hierarchical Gumbel
  Attention Network for Text-based Person Search,” Proceedings of
  the 28th ACM International Conference on Multimedia, pp. 3441–3449, Oct.
  2020.

zhengPersonReidentificationMeets2015
L. Zheng, L. Shen, L. Tian, S. Wang, J. Bu, and Q. Tian, “Person
  Re-identification Meets Image Search,” Feb. 2015.

zhengPersonReidentificationPresent2016
L. Zheng, Y. Yang, and A. G. Hauptmann, “Person Re-identification:
  Past, Present and Future,” Oct. 2016.

zhengReIdentificationConsistentAttentive2019
M. Zheng, S. Karanam, Z. Wu, and R. J. Radke, “Re-Identification With
  Consistent Attentive Siamese Networks,” in Proceedings of the
  IEEE/CVF Conference on Computer Vision and Pattern
  Recognition, pp. 5735–5744, 2019.

zhouAttentionBasedNaturalLanguage2017
T. Zhou, M. Chen, J. Yu, and D. Terzopoulos, “Attention-Based Natural
  Language Person Retrieval,” in 2017 IEEE Conference on Computer
  Vision and Pattern Recognition Workshops (CVPRW), pp. 27–34, July
  2017.

zhuDSSLDeepSurroundingsperson2021
A. Zhu, Z. Wang, Y. Li, X. Wan, J. Jin, T. Wang, F. Hu, and G. Hua, “DSSL:
  Deep Surroundings-person Separation Learning for Text-based Person
  Retrieval,” in Proceedings of the 29th ACM International
  Conference on Multimedia, MM '21, (New York, NY, USA),
  pp. 209–217, Association for Computing Machinery, Oct. 2021.

zhuDualCrossAttentionLearning2022
H. Zhu, W. Ke, D. Li, J. Liu, L. Tian, and Y. Shan, “Dual Cross-Attention
  Learning for Fine-Grained Visual Categorization and Object
  Re-Identification,” in 2022 IEEE/CVF Conference on Computer
  Vision and Pattern Recognition (CVPR), pp. 4682–4692, June 2022.

zhuIdentityGuidedHumanSemantic2020
K. Zhu, H. Guo, Z. Liu, M. Tang, and J. Wang, “Identity-Guided Human
  Semantic Parsing for Person Re-identification,” Computer Vision
  – ECCV 2020, vol. 12348, pp. 346–363, 2020.

zouPersonReidentificationBased2021
G. Zou, G. Fu, X. Peng, Y. Liu, M. Gao, and Z. Liu, “Person re-identification
  based on metric learning: A survey,” Multimedia Tools and
  Applications, vol. 80, pp. 26855–26888, July 2021.


dengPedestrianAttributeRecognition2014
Y. DENG, P. Luo, C. C. Loy, and X. Tang, “Pedestrian Attribute Recognition
  At Far Distance,” in Proceedings of the 22nd ACM International
  Conference on Multimedia, MM '14, (New York, NY, USA),
  pp. 789–792, Association for Computing Machinery, Nov. 2014.

hirzerPersonReidentificationDescriptive2011
M. Hirzer, C. Beleznai, P. M. Roth, and H. Bischof, “Person re-identification
  by descriptive and discriminative classification,” in Proceedings of
  the 17th Scandinavian Conference on Image Analysis, SCIA'11,
  (Berlin, Heidelberg), pp. 91–102, Springer-Verlag, May 2011.

laynePersonReidentificationAttributes2012
R. Layne, T. M. Hospedales, and S. Gong, “Person Re-identification by
  Attributes,” in British Machine Vision Conference, BMVC
  2012, Surrey, UK, September 3-7, 2012, pp. 1–11, BMVA Press,
  2012.

liDeepReIDDeepFilter2014
W. Li, R. Zhao, T. Xiao, and X. Wang, “DeepReID: Deep Filter Pairing
  Neural Network for Person Re-identification,” in Proceedings of
  the 2014 IEEE Conference on Computer Vision and Pattern
  Recognition, CVPR '14, (USA), pp. 152–159, IEEE Computer Society,
  June 2014.

linImprovingPersonReidentification2019
Y. Lin, L. Zheng, Z. Zheng, Y. Wu, Z. Hu, C. Yan, and Y. Yang, “Improving
  person re-identification by attribute and identity learning,” Pattern
  Recognition, vol. 95, pp. 151–161, Nov. 2019.

wangPedestrianAttributeRecognition2019
X. Wang, S. Zheng, R. Yang, B. Luo, and J. Tang, “Pedestrian Attribute
  Recognition: A Survey,” Jan. 2019.

xiongPersonReIdentificationUsing2014
F. Xiong, M. Gou, O. Camps, and M. Sznaier, “Person Re-Identification Using
  Kernel-Based Metric Learning Methods,” in Computer Vision
  – ECCV 2014 (D. Fleet, T. Pajdla, B. Schiele, and
  T. Tuytelaars, eds.), (Cham), pp. 1–16, Springer International
  Publishing, 2014.

  caoOpenPoseRealtimeMultiPerson2019
  Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, “OpenPose:
    Realtime Multi-Person 2D Pose Estimation using Part Affinity
    Fields,” May 2019.
  
  panTwoOnceEnhancing2020
  X. Pan, P. Luo, J. Shi, and X. Tang, “Two at Once: Enhancing Learning
    and Generalization Capacities via IBN-Net,” Mar. 2020.
  
  radfordLearningTransferableVisual2021a
  A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry,
    A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning
    Transferable Visual Models From Natural Language Supervision,” Feb.
    2021.
  
  rombachHighResolutionImageSynthesis2022
  R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer,
    “High-Resolution Image Synthesis with Latent Diffusion Models,”
    Apr. 2022.
  
  ronnebergerUNetConvolutionalNetworks2015
  O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional
    Networks for Biomedical Image Segmentation,” May 2015.

  zhangadding2023
  L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to
      text-to-image diffusion models,” arXiv preprint arXiv:2302.05543,
      2023.
    
  brown2020language
    T. B. Brown, “Language models are few-shot learners,” arXiv preprint
      arXiv:2005.14165, Jul 2020.
    
  hertz2022prompt
    A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or,
      “Prompt-to-prompt image editing with cross attention control,” arXiv
      preprint arXiv:2208.01626, Aug 2022.
    
  zhang2022realgait
    S. Zhang, Y. Wang, T. Chai, A. Li, and A. K. Jain, “Realgait: Gait recognition
      for person re-identification,” arXiv preprint arXiv:2201.04806, 2022.






§ APPENDIX

This appendix provides tables and figures that supplement the data and analyses presented in the main body of the paper. 
These tables and figures provide more detailed information and further support the findings discussed. 
They are referenced throughout the paper where relevant.


  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
      

 
  
  
  


  









 






















 Paper Title: IN-Sight: Interactive Navigation through Sight.



[
    R. Sasportes^†
    August 5, 2024
==================

empty
empty








§ ABSTRACT 





Current visual navigation systems often treat the environment as static, lacking the ability to adaptively interact with obstacles. This limitation leads to navigation failure when encountering unavoidable obstructions. In response, we introduce IN-Sight, a novel approach to self-supervised path planning, enabling more effective navigation strategies through interaction with obstacles. Utilizing RGB-D observations, IN-Sight calculates traversability scores and incorporates them into a semantic map, facilitating long-range path planning in complex, maze-like environments. To precisely navigate around obstacles, IN-Sight employs a local planner, trained imperatively on a differentiable costmap using representation learning techniques. The entire framework undergoes end-to-end training within the state-of-the-art photorealistic Intel SPEAR Simulator. We validate the effectiveness of IN-Sight through extensive benchmarking in a variety of simulated scenarios and ablation studies. Moreover, we demonstrate the system's real-world applicability with zero-shot sim-to-real transfer, deploying our planner on the legged robot platform ANYmal, showcasing its practical potential for interactive navigation in real environments. 

Supplementary Video: <https://youtu.be/ja0Vjm72ZDw>






§ INTRODUCTION


In robotics, navigating real-world environments from vision sensors has been a long studied subject <cit.>. Yet, most traditional navigation approaches treat the environment as a passive entity, not capturing the dynamic nature of the real world where obstacles can be moved deliberately. Visual Interactive Navigation (VIN) takes this into account by engaging with the surrounding, actively moving obstacles out of the way to enable more effective traversal strategies (cf. Figure <ref>).



Engaging with the environment introduces several challenges. Firstly, an agent must discern whether an obstacle is movable and whether moving it is beneficial for reaching the goal. This involves predicting how obstacles will respond to interaction, a task complicated by the intricate nature of such dynamics. As a result, the field gravitates towards learning-based approaches. Reinforcement Learning (RL) has shown promise in enabling navigation in interactive indoor environments using vision sensors, but it can suffer from poor sample efficiency, especially with high-resolution visual inputs. Therefore, many studies resort to fast non-photorealistic physics simulators <cit.><cit.><cit.> or vector observations rather than raw visual inputs <cit.>. While successful in simulation, these approaches often fall short in real-world application.

Real-world environments, characterized by maze-like structures such as buildings or public spaces, pose a significant challenge to agents that rely on purely reactive policies without utilizing past observations, leading them to easily get lost. <cit.> attempts to mitigate this by integrating a transformer-based memory into the RL policy, however causing stability complications during training and limited sim2real transfer. In contrast, the supervised learning approach <cit.> incorporates memory using explicit mapping. The authors fine-tune a pretrained object pose estimation module to localize paper boxes, combining it with explicit path planning on a pre-computed map. This approach alleviates the memory problem and avoids training on non-photorealistic data, improving applicability to real-world environments. However, this work does neither address the complexities related to reasoning whether an obstacle can be moved, nor how generalization to unseen obstacles with similar properties can be achieved. Relying on a trial-and-error approach, the robot has to actively push the obstacle to determine whether its movable, resulting in many undesired interactions. Given that the visual appearance of real-world environments varies drastically and physical sensors are affected by sensing artifacts, deploying VIN systems to robotic hardware remains a major complication.

In this work we introduce a hierarchical planner, continuously integrating traversability estimates into a global map, which is then used to perform high-level path planning. From the high-level path plan we extract subgoals, which are then tracked by a local planner. This allows us to navigate cluttered and maze-like environments at scale. Contrary to previous RL-based approaches, we train our system in a self-supervised fashion, achieving fast convergence to a robust policy. We generate a diverse photorealistic training set consisting of sequential data using the Intel SPEAR simulator. We further employ various robustification techniques in order to mitigate the effects of sensing artifacts and imperfect photorealism.























Our main contributions are as follows:



    
  * We propose a new training paradigm by extending the established imperative approach <cit.> to accommodate interactive environments, leveraging sequential observations to support interactive navigation.
    
  * We introduce a scalable approach to generate self-supervision for path prediction and traversability estimation, based on different heuristics such as obstacle mass, friction, and terrain inclination.
    
  * We employ a blend of robustification strategies and accelerate the ICL-NIUM depth noise algorithm <cit.> by two orders of magnitude, allowing to simulate truly random depth sensor artifacts without slowing down training.
    
  * We demonstrate successful zero-shot transfer of the learned policy to the ANYmal legged robot.




§ RELATED WORK

Visual navigation is a widely explored research field with a substantial amount of influential contributions over the past two decades. A variety of relevant concepts, developed to address the different challenges of visual navigation, can be directly applied to visual interactive navigation. 
As mentioned previously, memory allows the agent to utilize past observations in decision processes. Not only limited to classical planning pipelines, memory also becomes an essential building block for successful learning-based architectures <cit.>. Often encountered in hierarchical planners, mapping allows to leverage explicit path planning methods such as A-star, Dijkstra's algorithm, or fast marching methods <cit.>. Beyond simple occupancy maps used in classical pipelines, learning-based approaches enable to encode more complex characteristics such as goal proximity <cit.>, semantic classes <cit.>, or goal reachability costs <cit.>. Mostly these systems encode heuristics which align sub-optimally with the local planners objective, as they are optimized using different labels. The labels we use to train our global mapping module are generated on the fly from the same cost-map the local planner is optimized on. This way we achieve full consistency between global- and local path planning. For the mapping, we rely on traversability estimation, determining which parts of the visible terrain can be safely traversed. Geometry-based methods such as <cit.> using Lidar or depth cameras suffer from poor terrain estimates as deformable instances such as grass, shrubs, or snow cannot be reliably classified. Semantic-based methods such as <cit.> successfully circumvent this problem by incorporating RGB information, but often suffer from a lack of robustness against appearance changes in real-world environments. In <cit.>, the authors utilize a visual transformer as a strong feature extractor and train a small network online for traversability adaption. We adopt this idea and fuse our representation learner with RGB information encoded by the pretrained DinoViT-V2 <cit.>. 

Various techniques in representation learning aim to extract the most relevant features from the input data. <cit.> explores this challenge by using a Variational Auto-Encoder (VAE) <cit.> in conjunction with an RNN to encourage the learning of a Latent Dynamics Model (LDM), showing that the RNN is able to learn a compact representation of the world that it can propagate forward in time. Similarly, <cit.> applies this concept for dimensionality reduction, pre-training a world model to then train a policy network on the outputs of the world model using RL. Both works however rely on complicated training schedules with the different modules trained disjointly, leading to sub-optimal representations potentially containing information not relevant for the downstream task. We address this shortcoming by training our entire representation learner end-to-end. Using variational inference, we learn regularized representations of the environment, implicitly encoding information about the dynamic obstacles such as shape, position, or velocity.




§ PLANNER ARCHITECTURE






 §.§ System Overview

The global planner utilizes an online-generated global map to search for a potentially feasible path, which is then tracked by the local planner. The workflow of our planning algorithm is illustrated in Figure <ref>. Specifically, the perception module encodes the RGB observation R_k and the depth observation D_k at timestep k into a compact representation z_k and predicts a pixel-wise traversability mask T_k. The traversability mask is then fed to the mapping module where it is integrated into a 2D grid map. The grid map is used to compute a minimum-cost path to the goal by executing A* path search. From the discrete path, a subgoal is selected which is then fed to the local planner together with the embedding z_k. The local planner then predicts a path to the subgoal, consisting of N intermediate waypoints. The idea is that the global planner uses explicit memory (grid-map) to capture the macroscopic structure of the environment and is therefore well suited to provide a long-term planning strategy. However, the semantic map has limited spatial resolution and contains temporary artifacts (decaying over time) in case of non-stationary obstacles. Therefore, we introduce a local planner that takes the responsibility of locally navigating around obstacles and deciding if an interaction is desirable. The local planner comprises a Recurrent Neural Network (RNN) which allows the planner to grasp short-term context from previous time steps. Combining both concepts enables the system to efficiently explore large environments, while locally considering interactions with obstacles to reduce the traversal effort. 




 §.§ Long-horizon planning

The traversability mask T encodes for each pixel whether it belongs to free space, a movable obstacle or non-traversable space. In order to integrate the mask into the map, we first compute the point cloud from the depth image using the pinhole projection model, then transform the points onto the world frame. We then project the points on the 2D plane and compute the respective grid cell index. Finally, we sample for each active grid cell the number of previous observations m from the weight map to compute the update weight α (cf. Eq. <ref>) and then update the grid map (cf. Eq. <ref>) in a weighted average fashion

    α   =    2 / (1 + m) , 
    
        c    =   α· c + (1 - α) ·c,

where c denotes the observed traversability class of the point and c denotes the average traversability class of the grid cell. The number of observations is clipped between [0,100] such that the influence of new measurements does not eventually converge to zero. This leads to sufficiently fast decay of map-artifacts due to moving obstacles. For each updated grid cell the associated cell in the weight map is incremented. 

For global path planning, we resort to the A* algorithm with L2 heuristic. Due to the weighted averaging of the measurements during the mapping, our map no longer contains discrete area classes (0:free-space, 1:obstacle, 2:non-traversable, 3:unexplored) but a continuous belief value over the area classes. To obtain the discrete area classes back, we threshold the continuous values with the following ranges: [0, 0.5)→ free space, [0.5, 1.5)→ obstacles, [1.5, 3.0)→ non-traversable, [3]→ unexplored. We assign different penalties for planning through grid cells belonging to movable obstacles and non-traversable space. After the global path is found, we extract the element from the path in direct line of sight with the robot
we determine from the global path a set of points, in which each two consecutive points can be connected by a straight line without touching neither obstacle space nor non-traversable space. From this set, we use the closest point as the next subgoal g_k for the local planner.



 §.§ Short-horizon planning





Figure <ref> illustrates our planner on an implementation level. The components can be categorized into perception (left) and planning (right).  We aim to learn a compact representation of the world from a sequence of visual observations, encoding relevant information about the environment such as position, velocity, and semantic information of surrounding obstacles. For this we let the perception network model the depth inputs D by a small set of latent factors z, using a Latent Variable Model (LVM) of the form

    log p_θ(D)    = log∫ p_θ(D|z)p_θ(z)dz
       = [t]
            𝔼_z∼ q_ϕ(z)[logp_θ(D|z)p_θ(z)/q_ϕ(z)]    
    
            + 𝕂𝕃(q_ϕ(z)||p_θ(z|x))_≥ 0.

Reformulating the LVM using parameterized families of distributions we can resort to parameter optimization instead of variational calculus. Following <cit.> we choose the following parametrization

    p_θ(z)    →𝒩(0,I) , 
        
        p_θ(D|z)    = 𝒩(D| f_ψ(z), I) = p_ψ(D|z) , 
        
        q_ϕ(z)    = 𝒩(z|ϕ=(μ, Σ))  with ϕ = g_λ([D,R]]),


where the parameters θ and ϕ are computed using armortized inference. For this the functions g_λ and f_ψ are learned from a dataset consisting of synthetic image sequences mixed with real-world images. To inject semantic information into the LVM, RGB information R_k is used as an additional input for estimating the latent parameters ϕ. We encode the RGB images using the DinoViT V2 <cit.> with frozen parameters, serving as a robust feature extractor. The depth input D_k is encoded using a residual CNN. The inputs are then concatenated and fed through another residual CNN to estimate the latent parameters ϕ_k. Both μ and Σ are of shape [23,40,8], thus we model the latent space with 7360 latent variables. We sample a latent tensor z_k∼ q_ϕ_k(z) and decode it using the shared-parameter transposed CNN f_ψ to get a reconstruction D̂_k of the depth input D_k. Tracing the information flow from D_k to D̂_k it becomes visible that the parameterized models involved in the computation form a Variational Auto-Encoder (VAE).
We also decode the mean tensor μ_k of the latent distribution to get the traversability mask T_k, using a transposed CNN. The traversability mask is used for the mapping as described earlier. The parameters of the latent distribution ϕ_k = [μ, Σ]_k are passed to the convolutional GRU which additionally takes the subgoal G_k as an input. The GRU propagates its hidden state h_k→ h_k+1. The mean of the latent μ_k, the subgoal G_k, and the old hidden-state h_k are fed to the Path-CNN which predicts N waypoints wp_k, forming the local path to the subgoal. The new hidden-state h_k+1 is fed through the CNN g_η(h_k+1) to predict the parameters χ of the distribution over the next latent state q_χ(z_k+1). We again sample a latent vector z_k+1∼ q_χ(z_k+1) and pass it through the shared-parameter transposed CNN f_ψ. Tracing the information flow from D_k to D̂_k+1 it becomes visible that parameterized models involved in the computation form a predictive VAE.
Intuitively the GRU in the planner maintains a (world) belief state h and updates it based on the current (world) observation ϕ_k, resulting in the updated prediction of the belief state h_k+1. We then use this prediction to infer the distribution over the next observation z_k+1. With the structural bias of the RNN, our system is encouraged to learn the dynamics of the latent variables. As commonly done in LVMs, optimizing the Evidence Lower-Bound (ELBO, cf. Equation <ref>) introduces latent-regularization by minimization of the KL Divergence. Using an uncorrelated Unit-Normal prior encourages more disentangled representations eventually promoting decoupling in the learned system dynamics <cit.><cit.>. With this in mind, our side-objective can be understood as optimizing an LDM, which empirically has been shown to improve the expressiveness and robustness of the learned representations <cit.>.
As a second side-objective, we use the tensors h_k and h_k+1 to predict the change in pose (position, heading) between the current and next time step, P_k. 



§ TRAINING




Our training set consists of sequential data, synthetically generated using Intel SPEAR. During data collection, the agent has access to the navigation mesh <cit.> encoding all traversal- and obstacle-interaction costs in the scene. This privileged information allows the agent to traverse along the minimum-cost path to the goal. Once the goal is reached, the environment resets. To demonstrate domain randomization, the positions and orientations of the agent and all obstacles are randomized on reset.
To account for sensor imperfections, we add white noise to the RGB images and simulate common depth sensor artifacts <cit.> (cf. Figure <ref>). We accelerate the sampling procedure of <cit.> using GPU vectorization (x500 speedup on i7-8750H w. RTX2070 mobile) allowing us to sample unique artifacts on a per-sample basis. This way, all noise artifacts are truly random between epochs. We also inject independent real-world images of multiple depth cameras from the SUN-RGBD dataset to robustify against unmodeled sensor intrinsics.


The training loss follows Figure <ref>

    ℒ   =    c_nav· (c_path·ℒ_path + ℒ_odom) + 
           c_trav·ℒ_trav + c_repr·ℒ_repr

with

    ℒ_repr = ℒ_reco,1 + ℒ_KLD,1 + ℒ_reco,2 + ℒ_KLD,2.

where the coefficients c are hand-tuned parameters. The path loss follows

    ℒ_PATH = α·ℒ_o + (1-α) · (ℒ_m + ℒ_g),


where ℒ_o penalizes proximity towards obstacles, ℒ_m penalizes irregular distances between consecutive waypoints and ℒ_g penalizes a high L2 distance between last waypoint and goal. ℒ_o is computed by projecting the waypoints onto the 2D costmap and sampling the cost values <cit.>. The differentiable cost map (cf. Figure <ref>, Training) is derived from the navigation mesh, which we modify to encode obstacle locations and semantic classes (free space, light-weight / medium-weight / heavy-weight obstacle, non-traversable space) during simulation. The cost value of a grid-cell depends on the distance to the next obstacle and the obstacles semantic class. Using the tuning value α we manually balance the preference of the agent to be more goal-directed or more obstacle avoidant. 
ℒ_trav is computed using the Multi-class Cross-Entropy. We compute the traversability labels by unprojecting the true depth image into a point-cloud, transforming it into world frame, and then projecting it onto the 2D grid map. We can then sample the cost values from the costmap and associate each cost value to each pixel from the depth image, such that we obtain the 2D ego-perspective traversability mask. 






For the auxiliary loss ℒ_odom we take sum of the L2 norm of the translational error and the
L2 norm of the rotational error between the prediction of the change in pose and the actual change in pose.
We train the system on an NVIDIA RTX3090 for 10 epochs using PyTorch's AdamW optimizer with a constant learning rate of 1e-4 and weight decay of 1e-8. We also apply gradient scaling and train in half-precision (16 bit). We resort to a sequence length of 5 time instances where each time instance has a batch of size 10. Therefore, we use 50 samples to compute the gradients. The DinoViT parameters are frozen. The dataset is shuffled after each epoch.



§ EXPERIMENTS











We evaluate our planner across three distinct simulated environments: The maze-like House 1, featuring manipulable obstacles that either appear in the dataset or are entirely new (cf. <ref>); House 2, populated exclusively with new obstacles that have textures different from those in the training set; and the cluttered Forest, which contains no movable obstacles.



 §.§ Simulation Benchmarks


We conduct each experiment over 100 epochs. At the start of each epoch, the agent is respawned, and we sample five random start/goal pairs, choosing the one with the greatest integral path length. This method eliminates overly simplistic start/goal pairs that could be solved with a direct path. An epoch is considered successful if the agent reaches its goal within 500 iterations (equivalent to 50 seconds at a rate of 10Hz). In our simulations, the agent navigates straight between local waypoints, executing in-place rotations upon reaching a target waypoint. An episode is deemed successful if the agent comes within 1 meter of the goal. Every 10 epochs, we reposition all non-stationary obstacles to a new location within a small area around their original, manually selected positions. For benchmarking, we employ the following metrics: the Success Rate (SR) (Equation <ref>) and Success Weighted by Path Length (SPL) (Equation <ref>)

    SR   =   num_success/num_success + num_failure
    SPL   =   1/N∑_i=0^N-1 S_i L_i/max(L_i, L_i).


where L_i is the length of the traversed path and L_i is the length of the least-cost path computed with respect to the navigation mesh (only available in simulation, cf. <ref>). We also capture the number of collisions with static elements η_static and movable obstacles η_obstacle.
The performance of the proposed planner, along with its ablated versions, is indicated in Table <ref>. We observe that the agent manages to reach the goal in time with consistently high SR and SPL across all three test environments. We further observe that the agent successfully inherits the navigation preferences encoded into the costmap, as a result of optimizing the policy with respect to the ℒ_o loss (cf <ref>).
An example of interactive navigation is provided in Figure <ref>, demonstrating the planner's ability to manage obstacles blocking the shortest path to the goal. In this scenario, the agent encounters a wooden crate on its direct route. It correctly identifies the crate as movable and determines that displacing the crate to proceed along the obstructed pathway is more resource-efficient than detouring through an alternative room. It then tips the crate over and proceeds with the traversal. This behavior demonstrates the planner's ability to intelligently negotiate obstacles, enhancing navigation strategies and, crucially, averting navigation failures by addressing obstructions directly when no unobstructed alternative paths are available.



 §.§ Ablation Study

We ablate the planner to examine the impact of each design choice on the final planning performance. We also measure the performance of a simplistic baseline configuration inspired by <cit.>, where we take the encoder architecture from the proposed planner but instead predict the waypoints directly from the embeddings Φ_k using a convolutional residual network. All ablated configurations are trained under the same conditions as the proposed planner, as outlined in Section <ref>.


The Deterministic configuration has the stochastic blocks removed (blue blocks, Figure <ref>) and is not optimized for the KL-Divergence. While having comparable SR and SPL to the proposed planner in the House 1 environment, it shows inferior performance on House 2 and Forest. We also observe that the number of collisions η_static increases for the scenes comprising a high obstacle diversity. This confirms the hypothesis that the latent regularization improves generalization by allowing for better interpolation in the latent space (cf. Section <ref>).

In the noRNN configuration the local planner lacks memory. Thus, the network is not able to maintain a temporally consistent representation of the nearby obstacles. If an obstruction drops out of sight when circumventing, the agent turns prematurely towards the obstacle again. This reduces the path margin and increases the risk of a collision. Consequently, η_static increases and the agent times out more often (reduced SR). 

In the onlyVAE configuration, having both the RNN and predictive VAE branch removed, the SR significantly deteriorates in both the Forest and House 2 environments, together with an increase in η_static in all test environments. Similar observations can be made when removing the odometry network (cf. noAUX). This confirms the hypothesis that optimizing an LDM or odometry estimator as a co-objective benefits the learned representations and leads to improved generalization (cf. Section <ref>). 

The minimal Baseline configuration performs equally good to the proposed planner in the less complex, non-interactive Forest environment. However, in the interactive, labyrinth-like environments the performance drops significantly, underlining the efficacy of our architectural design choices.







 §.§ Real-World Experiments






We test our planner at 5 Hz on the quadrupedal ANYbotics ANYmal D robot equipped with a NVidia Jetson AGX PC and an Intel Realsense D435 camera. The robot autonomously navigates from a start point to a predetermined goal without human intervention. We intentionally obstruct the direct route with unfamiliar obstacles not encountered during training (cf. Figure <ref> [A,E]), including both static (tables, heavy chairs, walls) and movable objects (paper boxes, lightweight office chairs, foam bricks). Our observations reveal that the planner effectively distinguishes between traversable and non-traversable areas (cf. Figure <ref> [B]) and consistently generates paths within the navigable space (cf. Figure <ref> [D]). It identifies movable obstacles (cf. Figure <ref> [A1,B1]), pushing them aside when bypassing is not feasible (cf. Figure <ref> [A2:3,B2:3]). Using the depth reconstructions (cf. Figure <ref> [C]) we can confirm that the obstacle shapes are successfully captured by the latent. From the local path visualization (cf. Figure <ref> [D]) we can observe that the local path correctly bends around static obstacles, confirming that the learned planner correctly reasons about obstacles. Moreover, the stability of local path predictions over time, even when obstacles are no longer visible, underscores the planner's temporal consistency.



§ CONCLUSION AND FUTURE WORK

We presented IN-Sight: a visual path planner for interactive navigation in cluttered and maze-like environments, demonstrating successful transfer to robotic hardware. We introduced a self-supervised training approach, leveraging the navigation mesh to auto-generate consistent path- and traversability supervision. This significantly reduces the manual labor required to create large photorealistic datasets suitable for planner and traversability estimator training, opening new opportunities for interactive navigation research. For future work, we aim to enhance our current setup in terms of sim-to-real transfer. By using low Level of Detail (LOD) collision meshes for the obstacles, we can generate depth reconstruction labels D̂_k with low LOD. These labels are expected to facilitate the learning of more basic structural representations, thereby improving the system's ability to generalize. Different from that, we seek to explore the application of Large Vision-Language Models for reasoning about the movability of obstacles. Utilization of the extensive amount of data encapsulated within these foundation models combined with their ability to reason, could potentially allow for the consideration of minute details in the environment, leading to more accurate predictions of whether an obstacle can be displaced.



-12cm  
                                  
                                  
                                  
                                  
                                  













§ ACKNOWLEDGMENT

This work was supported by Intel Labs Munich, in a collaboration between the Robotic Systems Lab at ETH Zurich, and the Smart Robotics Lab at TUM. Moreover, this work has been conducted as part of ANYmal Research, a community to advance legged robotics.



bibliography/IEEEtranN

tocchapterBibliography



 Paper Title: MonoMM: A Multi-scale Mamba-Enhanced Network for Real-time Monocular 3D Object Detection.



Article Title]MonoMM: A Multi-scale Mamba-Enhanced Network for Real-time Monocular 3D Object Detection










1]Youjia Fuyoujia_fu@cqut.edu.cn
These authors contributed equally to this work.

1]Zihao XuXu139@stu.cqut.edu.cn
These authors contributed equally to this work.

1]Junsong Fujuns_fu@stu.cqut.edu.cn

1]Huixia Xuexue_xue@stu.cqut.edu.cn

1]Shuqiu Tantsq@cqut.edu.cn

[2,3]Lei Lililei@di.ku.dk







[1]Chongqing University of Technology
*[2] University of Washington
*[3]University of Copenhagen








Recent advancements in transformer-based monocular 3D object detection techniques have exhibited exceptional performance in inferring 3D attributes from single 2D images. By incorporating depth information and visual features from images, these methods enhance spatial awareness, thereby playing a crucial role in applications such as autonomous driving and augmented reality. However, most existing methods rely on resource-intensive transformer architectures, which often lead to significant drops in computational efficiency and performance when handling long sequence data. To address these challenges and advance monocular 3D object detection technology, we propose an innovative network architecture, MonoMM, a Multi-scale Mamba-Enhanced network for real-time Monocular 3D object detection. This well-designed architecture primarily includes the following two core modules: Focused Multi-Scale Fusion (FMF) Module , which focuses on effectively preserving and fusing image information from different scales with lower computational resource consumption. By precisely regulating the information flow, the FMF module enhances the model’s adaptability and robustness to scale variations while maintaining image details. Depth-Aware Feature Enhancement Mamba (DMB) Module : It utilizes the fused features from image characteristics as input and employs a novel adaptive strategy to globally integrate depth information and visual information. This depth fusion strategy not only improves the accuracy of depth estimation but also enhances the model’s performance under different viewing angles and environmental conditions. Moreover, the modular design of MonoMM provides high flexibility and scalability, facilitating adjustments and optimizations according to specific application needs. Extensive experiments conducted on the KITTI dataset show that our method outperforms previous monocular methods and achieves real-time detection.








[
    *
    August 5, 2024
==================





§ INTRODUCTION


In 2D object detection technologies based on convolutional neural networks (CNNs) <cit.> have developed rapidly. They are widely used in various fields, such as license plate recognition and defect detection. However, in areas such as robot navigation, autonomous driving, and multi-object tracking, obtaining detection results that are closer to the real world is difficult to achieve with 2D object detection. Therefore, to obtain detection results that are closer to reality, many researchers use distance sensors, e.g. LiDAR <cit.> or binocular cameras <cit.> as input devices to acquire accurate depth information. Although high performance is achieved, these methods face high hardware costs. In contrast, obtaining high-precision 3D detection results from a single image can greatly reduce computation and equipment costs. Currently, some methods using only monocular cameras <cit.> have been proposed, achieving promising results by utilizing the geometric constraints between 2D and 3D. However, since a single image cannot directly obtain depth information, the performance is far from satisfactory, thus attracting the attention of many researchers.

Depth estimation has witnessed substantial advancements, culminating in enhanced precision for monocular depth estimation models. Consequently, a plethora of researchers have leveraged sophisticated monocular depth estimation models to generate depth information, thereby augmenting monocular 3D object detection tasks. Pseudo-LiDAR-based methodologies <cit.> transmute estimated depth maps into point clouds, simulating authentic LiDAR signals, and subsequently employ advanced LiDAR detectors for 3D object detection. Furthermore, fusion-based approaches <cit.> discretely extract and amalgamate image features and depth features for 3D object detection. While these methodologies exhibit proficiency in object localization, they are susceptible to the potential loss of salient features during the feature extraction process and are encumbered by inaccuracies inherent in the estimated depth images.


Pioneering research has expanded the application of Mamba to visual tasks, exemplified by VMamba <cit.>, which enhances global effective receptive fields in both horizontal and vertical directions through selective scanning. Nonetheless, the utilization of Mamba in 3D monocular object detection remains relatively unexplored. This paper investigates the application of Mamba in monocular 3D object detection, optimizing its design to address the specific requirements of this task. Through targeted adjustments, we aim to establish a robust foundation for future research endeavors in this domain.

To address the issues identified, we propose a network for monocular 3D object detection named MonoMM. This network introduces a Focused Multi-Scale Fusion (FMF) module, which captures rich information across multiple scales using a specialized feature-focusing module with parallel depth convolutions. This approach facilitates the improved fusion of detailed contextual information from each scale, thereby enhancing 3D object detection performance. Furthermore, the FMF module is lightweight, reducing computation time compared to other multi-scale fusion methods <cit.>.

Unlike MonoDTR <cit.>, this paper introduces the Depth-Aware Feature Enhancement Mamba (DMB) module, which is based on Mamba and models the image sequentially. This method linearly projects feature information into vectors, treating them as sequential data similar to text. By utilizing a bidirectional selective state space, the visual information is efficiently compressed, promoting complementarity between deep and shallow features, and enabling comprehensive image content capture. Moreover, this approach incorporates positional embeddings into the feature information, enhancing spatial perception and increasing the model's robustness in dense prediction scenarios.


Our contributions are summarized as follows:

    
  * This paper introduces a novel Focused Multi-Scale Fusion (FMF) module that employs feature focusing and diffusion mechanisms. These mechanisms distribute features with rich contextual information across different detection scales, effectively reducing noise and interference.
    
  * We propose an innovative Depth-Aware Feature Enhancement Mamba (DMB) module, which is the first work to efficiently integrate image contextual information with Mamba. By using an adaptive strategy, it combines depth and visual information, thereby enhancing the performance of the detection head.
    
  * Experimental results on the KITTI dataset demonstrate that our method outperforms most monocular approaches and achieves real-time detection. Furthermore, the FMF and DMB modules can be easily integrated into existing image-based frameworks, enhancing overall performance.





§ RELATED WORK



 §.§ Monocular 3D Target Detection

Several existing monocular 3D object detection methods build on 2D object detectors <cit.>. MonoRCNN <cit.> enhances 3D object detection by decomposing object distance into physical height and projected 2D height, modeling their joint probability distribution. MonoDLE <cit.> emphasizes the importance of accurate 2D bounding box estimation for predicting 3D attributes, identifying depth error as a critical limitation. M3D-RPN <cit.> introduces depth-aware convolutions to generate 3D object proposals constrained by 2D bounding boxes. MonoCon <cit.> incorporates auxiliary learning tasks to improve generalization performance. Monopair <cit.> leverages spatial relationships between object pairs to enhance 3D positional information. MonoJSG <cit.> refines depth estimation using pixel-level geometric constraints. MonoFlex <cit.> addresses long-tail object prediction issues with multiple depth estimators. PDR <cit.> simplifies this approach with a lighter architecture using a single perspective-based estimator. MonoGround <cit.> incorporates a local ground plane prior and enriches depth supervision by sampling points around the object's bottom plane. MonoDDE <cit.> extends depth prediction branches based on keypoint information, emphasizing depth diversity. Despite these advances, purely monocular methods face challenges in precise object localization due to the lack of depth cues and the accumulation of errors in geometric constraints.



 §.§ Depth-assisted Monocular 3D Object Detection

To achieve higher performance, many approaches leverage depth information to assist in 3D object detection <cit.>. Some methods <cit.> employ advanced depth estimators and camera parameters to map images into 3D space, transforming them into pseudo-LiDAR data representations. These representations are then used with LiDAR-based 3D object detectors to enhance detection. Pseudo-LiDAR++ <cit.> optimizes stereo depth estimation, leveraging sparse LiDAR sensors to improve 3D object detection in autonomous driving. D4LCN <cit.> and DDMP-3D <cit.> develop fusion-based methods that combine images and estimated depth with specialized convolutional networks. CaDDN <cit.> learns class-specific depth distributions per pixel to construct Bird's Eye View (BEV) representations and recover bounding boxes from BEV projections. MonoDTR <cit.> uses LiDAR point clouds as auxiliary supervision for its Transformer and employs learned depth features as input queries for the decoder. MonoDETR <cit.> predicts foreground depth maps using object labels for depth guidance. To enhance inference efficiency, MonoATT <cit.> introduces an adaptive tokens Transformer that assigns finer tokens to more critical regions in the image.




 §.§ State Space Models

State space models (SSMs), originating from classical control theory <cit.>, have recently gained prominence in deep learning due to their capability to manage long-term dependencies and sequential data. Innovations such as the Hippo model <cit.>, which utilized polynomial highest power operator initialization, have enhanced SSMs' ability to capture long-term dependencies. The LSSL model <cit.> demonstrated the effectiveness of SSMs in handling these dependencies, though it faced challenges related to computation and memory efficiency. To address these issues, Gu et al. introduced the Structured State Space Sequence Model (S4) <cit.>, which reduced computational overhead through a normalized parameterization strategy, thus making SSMs more practical. S4 advanced deep-state space modeling for long-term dependencies. The S5 model <cit.> incorporated MIMO SSM and efficient parallel scanning, while H3 <cit.> narrowed the performance gap between SSMs and Transformer models in NLP. Recently, Mamba <cit.> has shown superior performance in NLP with its advanced design, including selective mechanisms and optimized hardware usage, surpassing some Transformer models. Mamba efficiently handles long sequences, maintaining linear computational costs with sequence length, unlike Transformers, which require exponentially increasing resources. This efficiency makes Mamba a more effective solution for large-scale sequence data.





§ METHODS




 §.§ Overview


<ref> illustrates the MonoMM framework, which consists of five main components: a Backbone Network (Backbone), a Focused Multi-Scale Fusion (FMF) module, a Depth-Assisted Perception (DAP) module, a Depth-Aware Feature Enhancement Mamba (DMB) module, and a 2D-3D detection head. Following the approach in <cit.>, we adopt DLA-102 <cit.> as the Backbone Network. Given an RGB image input of size H_in× W_in, the Backbone Network outputs features at different layers. The FMF module integrates these feature maps to produce a fused feature map F ∈ℝ^C × H × W, where H = H_in/8, W = W_in/8, and C = 256. Convolutional layers enhance feature representation by integrating spatial positional information across the image, thereby extracting higher-level and more abstract visual features. Additionally, we introduce the Depth-Assisted Perception (DAP) mechanism <cit.>, which employs multiple convolutional layers to specifically capture depth-related features. Subsequently, the DMB module efficiently integrates the visual and depth-aware features, further enhancing feature comprehensiveness and discriminability. To fully leverage these fused features, we utilize an anchor-based detection head architecture and select appropriate loss functions, aiming to achieve precise 2D and 3D object detection tasks simultaneously.




 §.§ Focused Multi-Scale Fusion Model


Existing multi-scale feature fusion methods primarily focus on integrating contextual information across different layers. However, due to limitations in receptive field sizes, it is challenging to effectively fuse contextual information from different layers together. Moreover, incorporating attention mechanisms poses risks of increasing additional computational burden. To address these issues, inspired by the work of CFP <cit.> and SMT <cit.> in dense prediction tasks, we propose a Global Focused Fusion (FMF) module for monocular image-based 3D object detection. In contrast to existing multi-scale fusion methods <cit.>, our proposed FMF module not only captures long-range dependencies between different scales to comprehensively focus on integrating contextual information across layers but also avoids attention mechanisms, significantly reducing computational costs.

The initial fusion stage, depicted in  <ref>(a), consists primarily of three components: AConv <cit.>, a 1×1 convolutional layer, and an upsampling layer. Inside the AConv unit, two mechanisms are integrated: Average Convolution (AC) and Maximum Convolution (MC). The AC submodule effectively aggregates and smooths features by combining average pooling with convolution operations, reducing computational complexity and enabling efficient extraction of crucial features. Conversely, the MC submodule enhances feature capture of prominent features in the image by leveraging maximum pooling coupled with convolution, thereby improving the model's robustness to small-scale displacements. Specifically, different scale features extracted from the backbone are processed in parallel with the above operations to obtain rich multi-scale feature representations. This process enhances feature expressiveness, and by adjusting the size of feature maps, it effectively captures and retains rich information and details across various scales during the initial fusion stage. This comprehensive approach ensures a thorough grasp of both global and local features in the image. The resulting feature is denoted as F ∈ R^3C × H × W, where C_1=256,H_1=H/16,W_1=W/16. The process is illustrated in Equation <ref>.
    
    X_i=Conv_i(F_i),i ∈1,2,3, 
       F=Cat(X_i), i ∈1,2,3


In the detail fusion stage, the network focuses on deepening and refining the features generated in the initial fusion stage to produce more precise and detailed outputs. As shown in <ref>(b), this stage begins by employing Depthwise Separable Convolution (DWConv) to parallelly process the initial fused features. This strategy delves deeper into feature extraction and effectively manages computational resource consumption, achieving deep-level feature fusion. Next, feature maps from various branch paths are stacked along the 0th dimension to ensure the complete preservation of multi-path information. To fuse this multi-path information into a unified feature map, the module adopts summation along the 0th dimension, significantly consolidating diverse information and subtle textures from different path transmissions, thereby constructing a rich and comprehensive feature representation for subsequent detection tasks. Subsequently, convolution and skip residual connection mechanisms are utilized to integrate the initial fused features, which have not undergone refinement, with the current features optimized through Depthwise Separable Convolution. This approach maintains the consistency and continuity of both global and local image features. Finally, the feature map dimensions are doubled by applying Transposed Convolution (TConv), effectively compensating for potential information loss introduced by previous downsampling steps. This step enhances resolution while ensuring detailed information's thorough restoration and transmission. The process is illustrated in Equation <ref>.


    
    F_feature   =Reduction(Stack(DW_i(F),F),F) ⊕ F, 
    
                F_out   =TConv(F_[feature]).




 §.§ Depth-Aware Feature Enhancement Mamba Model



To obtain depth-aware features for subsequent fusion with visual features, we drew inspiration from MonoDTR <cit.> and introduced the DAF module based on this approach. The DAF module effectively learns and refines depth-related feature information by incorporating precise depth maps as auxiliary supervision signals. This ensures that the DMB module comprehensively and deeply integrates depth-aware features with visual features, capturing and optimizing subtle information fused from both domains. As a result, it enhances the overall detection performance and accuracy.

The architecture of the DMB module we proposed is depicted in  <ref>(a). To handle the fused feature F^d derived from the fusion of visual and depth-aware features, we first transform F^d ∈ R^C × H × W into a flattened 2D sequence x^p ∈ R^p × C, where p=H/psize^h×W/psize^w denotes the total number of feature patches, and C represents the number of channels. This transformation is formulated as shown in equation <ref>:
    
    T_0=[F_1^dW,...,F_p^dW]

Where F_p^d represents the p-th patch of depth-aware features, W ∈ R^p × C is a projection matrix with learnable characteristics. It maps these patches to a higher-dimensional feature space to capture more complex visual patterns. This approach draws inspiration from cutting-edge research such as ViT <cit.> and Vim <cit.>, which innovatively apply patch-based methods by dividing images into continuous token sequences for global analysis. Following this concept, the patches segmented from F^d are transformed into token sequences and fed into the DMB module, producing outputs T_l after l layers. To further refine and stabilize these feature representations, T_l undergoes normalization to ensure consistency in the distribution of features across dimensions. This process is represented by Equation <ref>:
    
    T_l   =DMB(T_l-1)+T_l-1
    
           f   =SiLU(Norm(_L^0))


Specifically, after normalizing the input tokens, they are processed through two parallel branches. In the first branch, the sequence undergoes linear projection to transform it into a vector of dimension E, followed by a deformable convolution (DCN) to better capture and express the complexity of original information while reducing parameter count. Subsequently, it passes through a SiLU activation function, a DSSM layer, and normalization (Norm). In the second branch, the sequence also undergoes linear projection to transform it into a vector of dimension E, followed by a SiLU activation function. Afterward, features from both branches are aggregated through element-wise multiplication. E represents the dimensionality of the hidden state. Finally, the features are projected back to the original size, generating output with the same shape as the input. The specific process is detailed in Algorithm.1:



In this context, the DSSM in this paper is inspired by the bidirectional information flow concept from Vim <cit.>, which captures dependencies between local regions in a top-to-bottom order and captures global context and details in a bottom-to-top order, forming more comprehensive information transmission to improve performance further. As shown in  <ref>(b), in the forward branch of the first branch, features transform a linear layer to map them to a new feature space to enhance their expressive power. Subsequently, a 1D convolution layer with a SiLU activation function introduces non-linear characteristics, enhancing feature learning capability. This process is combined with the SSM layer to promote effective feature refinement jointly. The reverse branch executes the same processing sequence as the forward branch in the opposite direction, ensuring comprehensive communication and refinement of feature information, deepening the model's understanding of details. The second branch also begins with a linear mapping step to preserve more global coarse-grained information, which is crucial for avoiding potential gradient explosion during training. A SiLU activation function follows this to maintain the non-linear propagation of features. Features from both branches are then fused through Hadamard multiplication, an aggregation method that effectively combines feature strengths from different perspectives. Finally, the fused features are reshaped to their original dimensions, preparing them for processing in the next network layer. 



 §.§ Train loss and 2D-3D Detection


We employed a single-stage detection method <cit.>, which uses predefined 2D-3D anchor points to determine bounding boxes. Each anchor point is defined with both 2D bounding box parameters [x_2d,y_2d,w_2d,h_2d] and 3D bounding box parameters [x_p,y_p,z,w_3d,h_3d,l_3d,θ]. Here, [x_2d,y_2d] represents the center point of the 2D box, while [x_p,y_p] denotes the projection of the 3D object center onto the image. [w_2d,h_2d] is the size of the 2D box, and [x_3d,y_3d,l_3d] is the size of the 3D box. The parameter z refers to the distance from the 3D object center to the camera, and θ represents the viewing angle. During training, all ground truth labels are mapped to the 2D space to compute Intersection over Union (IoU) with each 2D anchor. When IoU exceeds 0.5, the anchor is selected for optimizing and assigning to the corresponding 3D box.

Our output transformation method draws inspiration from Yolov5's<cit.> strategy, predicting a set of 2D parameters [t_x,t_y,t_w,t_h] and a set of 3D parameters [t_x,t_y,t_w,t_h,t_l,t_z,t_θ] for each anchor point. These parameters quantify the offsets for 2D and 3D bounding boxes and output a classification score cls for each category. By combining the anchor points' baseline positions with the network's predictions, we can accurately recover the bounding boxes of the targets.

Loss function. In this paper, the classification loss L_cls and regression loss L_reg are adopted using focal loss <cit.> and smooth L1 loss <cit.>, respectively. Specifically:

    
    L_cls(y,y)   =-α(1-y_x)^γlog(y_x),
    
            L_reg(y,y)   =
            
            0.5(y-y)               | y-y| < 0,
    δ(| y-y| - 0.5δ)                   other.


Where α is the weight coefficient to balance positive and negative samples, adjusting the importance of different classes; γ is the focusing parameter to reduce the weight of easily classified samples, allowing the model to focus more on hard-to-classify samples. δ is a threshold parameter: when the absolute error is less than δ, the loss function switches to squared loss to ensure smooth gradient output and avoid significant disturbance to the model in case of small errors. When the absolute error is greater than or equal to δ, absolute loss is used instead, which helps handle larger errors and mitigate the impact of outliers on model training.

For the depth loss L_dep, we follow the approach set by MonoDTR <cit.>, treating depth estimation as a classification task. We utilize the ground truth values of depth bins D̂ generated from LiDAR. We employ the focal loss for this purpose, formulated as follows:
        
    L_dep=1/| P |p∈ P∑FL(D(p),D(p).


wherein P is a pixel region on the image with a valid depth label.




§ EXPERIMENTS



 §.§ Experimental setup


To validate the effectiveness of our proposed method, we selected the widely used KITTI 3D object detection dataset as our evaluation platform. This dataset comprises a total of 7,481 training images and 7,518 testing images. Adhering to the methodology outlined in <cit.>, we partitioned the training set into two subsets: a training subset containing 3,712 images and a validation subset consisting of 3,769 images. Within this partitioning framework, we conducted ablation experiments to systematically analyze the contribution of each component to the enhancement of model performance. This rigorous experimental design ensures the reliability of the results and validates the soundness of our proposed methodology.


Regarding evaluation methods, whether for 3D object detection or bird's-eye-view (BEV) detection tasks, Average Precision (AP) is the key performance assessment metric. This study employs the AP at 40 recall positions metric to mitigate potential biases. In benchmark testing, detection difficulty is categorized into "easy," "moderate," and "hard" based on object size, occlusion level, and truncation. The methods are primarily ranked based on their 3D AP values under the "moderate" difficulty setting (Mod.), consistent with the KITTI benchmark. Additionally, following official guidelines, separate Intersection over Union (IoU) thresholds of 0.7, 0.5, and 0.5 are set for detecting car, cyclist, and pedestrian categories, respectively.

The specific operations during implementation are as follows: Following the setup in <cit.>, the Adam optimizer is employed to train the network model. The entire training process spans 100 epochs, with each batch containing 12 samples. The initial learning rate is set to 0.0001 and dynamically adjusted using cosine annealing. At each pixel position of the feature map, 48 anchor boxes are designed, covering 3 different aspect ratios (0.5, 1.0, 1.5). Additionally, 12 anchor boxes of varying height sizes are generated based on an exponential function, indexed from 0 to 15. For the parameters of the 3D anchor boxes, the mean and variance of the 3D ground truth bounding boxes in the training dataset are computed as reference statistics for the anchor boxes. To reduce inference time, the top 100 pixels of each image are cropped during preprocessing, and all images are uniformly resized to 288 × 1280 dimensions. During training, random horizontal flipping is applied for data augmentation. Predictions with confidence scores below 0.75 are discarded during inference, and a Non-Maximum Suppression (NMS) algorithm with an IoU threshold of 0.4 is used to eliminate redundant predictions.





 §.§ Main Results

In the results for the car category on the KITTI test set, as presented in Table <ref>, the method proposed in this paper (MonoMM) is evaluated against other recent state-of-the-art monocular 3D detection methods. The findings indicate that MonoMM achieves superior performance in AP_3D for moderate objects, which is the most critical metric in KITTI evaluations. This is particularly noteworthy as moderate samples generally exhibit smaller sizes, and prior algorithms <cit.> have a propensity to incorporate excessive coarse-grained information during feature fusion, leading to detection inaccuracies. Conversely, MonoMM alleviates the issue of excessive coarse-grained information integration through the implementation of the FMF module, thereby focusing more on detailed information. This refinement results in significant performance improvements in the detection of moderate objects.



In this paper, experiments were conducted on the car category of the KITTI validation set under different IoU thresholds and task conditions, as shown in Table <ref>. The proposed method outperformed several image-only methods. Specifically, compared to MonoDTR <cit.>, our approach demonstrated advantages across easy, moderate, and hard settings at an IoU threshold of 0.5. The model significantly improved 3D Average Precision (AP_3D) by 1.22, 2.21, and 1.00 percentage points, respectively. Additionally, Average Precision in Bird's-Eye-View (AP_BEV) increased by 1.92, 1.43, and 2.19 percentage points, respectively. These enhancements underscore the model's superior accuracy and validate its robust performance and broad applicability in complex environments.



Table <ref> presents the results for the pedestrian and cyclist categories on the KITTI test set, further illustrating the performance of our model in these categories. Detecting pedestrians and cyclists is more challenging than the car category, primarily due to their smaller size and non-rigid body structure, making accurate localization more difficult. Overall, the model significantly outperforms other state-of-the-art methods in the pedestrian category, achieving approximately a 15% improvement. Regarding 3D detection for cyclists, we also achieved results comparable to CaDDN <cit.> in the easy difficulty and surpassed other methods in the moderate and hard difficulties. The results shown in Table 3 validate our model's exceptional versatility and scalability. Therefore, the method proposed in this paper enables accurate detection of objects with diverse appearances.





 §.§ Runtime Analysis


On a single Nvidia 4090 GPU, the entire validation set was processed with a batch size of 1, yielding an average runtime of 40 frames per second. This performance highlights the efficiency of our proposed approach. In comparison to the current state-of-the-art methods, our MonoMM model demonstrates a substantial speed enhancement, being 15 times faster than the CaDDN method <cit.>. Several key factors contribute to this significant improvement in speed.

Firstly, CaDDN generates bird's-eye-view representations for 3D detection by predicting depth maps, which necessitates a more complex architectural design to achieve precise depth prediction, thereby resulting in increased processing times. Secondly, fusion methods typically deploy two separate backbone networks to extract both image and depth features, a process that inherently consumes more computational resources and time. Moreover, the inference time required for the depth estimator is an additional factor, not accounted for in Table <ref>.

Conversely, the model proposed in this paper employs a lightweight FMF module, which is specifically designed to enhance detailed feature representation without incurring significant computational overhead. Additionally, the DMB module effectively integrates depth-aware features with visual features, substantially improving the model's feature processing capabilities while markedly reducing runtime. This strategic integration and efficient module design underscore the superior performance of our model.



 §.§ Ablation Study


To validate the effectiveness of the proposed components in this paper's model, We conducted ablation experiments as shown in Table <ref>, where the model of [34] is taken as the baseline model. (a) Baseline.(b) Based on the baseline model, this paper replaces the feature fusion module with FMF to observe the impact of this feature fusion approach on the 3D target detection performance. (c)Based on the baseline model, the image features are sensed using the DMB module, and its impact on the results is observed using only image-sensing features. (d) Integration of contextual features using convolutional operations, summing them with depth-aware feature elements, and observing the results again. (e) MonoMM integrates all the above-mentioned modules.


As illustrated in Table <ref>, the experimental results delineate the contributions and effects of each module on overall performance. According to the experiments, introducing the FMF  module into the baseline significantly improves the Average Precision of 3D detection (AP_3D) across different difficulty levels from 19.35/15.47/12.83 to 20.71/16.95/13.26, underscoring the positive impact of the FMF module in enhancing feature details and promoting detection performance. 

Integrating the DMB module into the baseline further elevates AP_3D to 24.32/18.89/14.12. This substantial improvement indicates that the DMB module is markedly more efficient in integrating depth and visual information compared to the original configuration, thereby significantly boosting 3D object detection accuracy.

Moreover, combining the DMB module with convolutional layers in the baseline configuration results in an increase in AP_3D to 25.03/19.62/14.14. This outcome highlights the effectiveness of convolutional operations in integrating contextual features and validates the DMB module as a robust strategy for the integration of visual and depth features.

Ultimately, the MonoMM model, which incorporates all proposed optimization modules, achieves substantial gains of 6.39/5.31/3.89 in AP_3D over the baseline, thereby confirming the significant improvements in detection accuracy, model stability, and robustness as posited in this study. The aforementioned ablation experiments collectively demonstrate the rationality of the module designs and the efficacy of the proposed approach in addressing monocular 3D object detection tasks.






To validate the ease of integrating the modules into existing monocular 3D object detection models and to improve their performance, we selected two state-of-the-art models: M3D-RPN <cit.> and MonoDLE <cit.>. We integrated the proposed modules (FMF and DMB) into these models, resulting in the fused M3D-RPN model depicted in  <ref>. The practical steps involved the following operations: initializing with the features of these models (before the detection heads), processing these features using FMF and DMB to generate integrated features, and finally using the original model's detection heads to detect 3D objects using these integrated features. The core idea of this process is that our approach can be easily integrated ("plug-and-play") directly into existing models without requiring extensive modifications to their original structures. This flexibility is a significant advantage of our modules. As shown in Table <ref>, with the assistance of FMF and DMB modules, these detectors achieved notable performance improvements on the KITTI validation set. This outcome demonstrates our approach's flexibility and underscores its applicability to various 3D object detection models. Furthermore, these enhancements illustrate that our method can improve performance without altering the original model architecture.







 §.§ Qualitative Results


 <ref> shows a qualitative example on the KITTI validation set that visually validates our approach. As can be seen from the figure, the predicted bounding boxes show a high degree of consistency and close fit between the predicted and actual labels, demonstrating a significant improvement in the accuracy of our model predictions. Finally, the model in this paper allows real-time inference. Our framework achieves a state-of-the-art trade-off between performance and latency.








§ CONCLUSION


We introduce a novel algorithm for monocular 3D object detection, incorporating the Focused Multi-Scale Fusion Model (FMF). The FMF employs advanced feature focusing and diffusion mechanisms to disseminate context-rich features across multiple detection scales, thereby effectively mitigating noise interference. Additionally, this paper presents an innovative integration of the Depth-Aware Feature Enhancement Mamba (DMB) module into monocular 3D object detection, efficiently amalgamating contextual information from images. A pioneering adaptive strategy that combines depth and visual information significantly enhances the performance of subsequent depth prediction heads. This approach not only improves the accuracy of depth estimation but also optimizes model performance across various viewpoints and environmental conditions. Comprehensive experiments conducted on the KITTI dataset demonstrate that our model achieves real-time detection and surpasses the performance of existing monocular detection methods.

Acknowledgements
the Chongqing Basic Research and Frontier Exploration Project (Chongqing Natural Science Foundation) under Grant CSTB2022NSCQ-MSX0918

the Science and Technology Research Project of Chongqing Education Commission (Youth) under Grant KJQN202101149



§ DECLARATIONS

Competing interestsThe authors declare that there is no conflict of interest in this study.

Ethics approval and consent to participateThere were no ethical issues in this study.

Data availabilitySelected data supporting the results of this study are available from the authors upon request.
















 Paper Title: Text-Guided Video Masked Autoencoder.





 



Text-Guided Video Masked Autoencoder






Fan et al.




Amazon Prime Video Amazon Fulfillment Technology







Text-Guided Video Masked Autoencoder
    David Fan1 Jue Wang1 Shuai Liao2 Zhikang Zhang1 

Vimal Bhat1 Xinyu Li1

    August 5, 2024
============================================================================



§ ABSTRACT 
Recent video masked autoencoder (MAE) works have designed improved masking algorithms focused on saliency. These works leverage visual cues such as motion to mask the most salient regions. However, the robustness of such visual cues depends on how often input videos match underlying assumptions. On the other hand, natural language description is an information dense representation of video that implicitly captures saliency without requiring modality-specific assumptions, and has not been explored yet for video MAE. To this end, we introduce a novel text-guided masking algorithm (TGM) that masks the video regions with highest correspondence to paired captions. Without leveraging any explicit visual cues for saliency, our TGM is competitive with state-of-the-art masking algorithms such as motion-guided masking. To further benefit from the semantics of natural language for masked reconstruction, we next introduce a unified framework for joint MAE and masked video-text contrastive learning. We show that across existing masking algorithms, unifying MAE and masked video-text contrastive learning improves downstream performance compared to pure MAE on a variety of video recognition tasks, especially for linear probe. Within this unified framework, our TGM achieves the best relative performance on five action recognition and one egocentric datasets, highlighting the complementary nature of natural language for masked video modeling.










    


§ INTRODUCTION



The success of masked language modeling <cit.> has recently inspired the adoption of the masked autoencoder (MAE) for masked image and video modeling. Masking out random image patches <cit.> and reconstructing the missing image patches via an asymmetric encoder-decoder architecture achieves promising results in image recognition. In a similar fashion, works such as VideoMAE <cit.> and ST-MAE <cit.> achieve promising results in video recognition by extending random masking from 2D image patches to 3D video cubes.

These initial works demonstrate the strong potential of masked visual modeling. Subsequent works have further explored the question of “where to mask?” While simple and effective, random masking assumes that input information density is uniformly distributed. Several new masking strategies have been proposed which challenge this assumption and attempt to directly mask visual saliency. For instance, in image domain, SemMAE <cit.> decomposes foreground into coarse semantic parts and then masks visual patches from each semantic segment according to a defined sampling probability. AutoMAE <cit.> utilizes adversarial training with bounding boxes to learn an object-centric mask. Both of these works intend to mask the foreground more often than background. These masking algorithms achieve better performance on image recognition benchmarks than the random masking baseline. However, there exists a trade-off point: masking information dense regions too aggressively degrades performance on downstream tasks <cit.>.

In video domain, MGM <cit.> and MGMAE <cit.> leverage motion as a video-specific prior for saliency. Specifically, they mask the patches with highest motion over time, where motion is obtained either from motion vectors in the video codec <cit.> or optical flow <cit.>. Motion-guided masking achieves better performance than random masking in video domain, suggesting that masking saliency is also important for masked video modeling.

These previous works exploit various visual priors with the goal of masking objects and motion. However, their robustness depends on how often input videos match the underlying statistical assumptions. For instance, not all videos have higher foreground motion than background motion. On the other hand, natural language captions are an information dense representation of video that describe both  “nouns” (e.g. humans and objects) and “verbs” (e.g. actions), without the need to make any prior assumptions. Provided a well-aligned vision-language model, it is possible to directly mask salient regions according to the input text. This makes natural language a promising source of saliency for masked video modeling that has not yet been explored. Thus, this work explores a new direction of improving masked video modeling with natural language, and presents a strong baseline composed a novel masking algorithm and additional loss.


 

First, we discard visual priors used in previous works and ask whether the content within captions can already capture the most salient regions of video. To that end, we first introduce a novel text-guided masking () algorithm, which masks the video regions with highest correspondence to a given caption that is either machine-generated or human-annotated. We present the interesting insight that text-guided masking is competitive with state-of-the-art motion-guided masking, despite using no explicit motion guidance. This confirms the intuition that captions can capture video saliency without prior assumptions.



Second, to further leverage the semantics of natural language for masked video modeling, we introduce a general unified framework for MAE and masked video-text contrastive learning. For any given masking algorithm, introducing an optional contrastive loss aligns the masked encoder representation with the text. To our knowledge, we are the first to unify the generative nature of MAE pretraining and discriminative nature of masked contrastive learning for video and obtain benefits from both pretraining paradigms.





Training from scratch without any bells and whistles, our text-guided masking outperforms MGM by up to 1.3% on Kinetics-400 (K400) and 0.5% on Something-Something V2 (SSv2) in finetuning performance, and by up to 1.7% in linear evaluation. We show that text-guided masking generalizes to smaller action recognition datasets as well as egocentric action recognition. Lastly, we demonstrate that the synergistic nature of masked contrastive learning also applies to other masking algorithms such as random and motion-guided masking.

In summary, our contributions are:


    
  * Text-guided masking () – a simple yet effective masking algorithm.
    
  * Unifying masked video modeling and video-text contrastive learning.
    
  * Empirical evidence for the synergistic nature of masked contrastive learning and masked video modeling on five action recognition datasets and one egocentric understanding dataset.
    
  * Introducing a new area of research into language-guided masked video modeling and detailed insights that will help inspire future work.



§ RELATED WORK









 §.§ Masked Image Modeling

Early work such as iGPT<cit.> performs masked image modeling at the pixel-level. BEIT <cit.> elevates the reconstruction target from individual pixels to pretrained dVAE tokens. MAE <cit.> reconstructs normalized pixel patches instead and demonstrates the efficacy of an asymmetric encoder-decoder design with high masking ratio. Recent work addresses the question of whether random masking is ideal. SemMAE <cit.> uses attention maps to obtain a coarse segmentation and masks patches based on a sampling distribution defined over each semantic class. AutoMAE <cit.> utilizes adversarial training with bounding boxes to learn an object-centric mask. Both SemMAE and AutoMAE find that masking higher proportion of foreground tokens improves image representations, up until a certain point where masking too aggressively degrades performance. 

In contrast to these works, our work is designed for video and leverages textual information as the primary proxy for video saliency.



 §.§ Masked Video Modeling

Some works use tokenization-based reconstruction targets. VIMPAC and BEVT use pretrained VQ-VAE <cit.> tokens. However, these works require extra pretraining. MaskFeat instead uses HoG features. VideoMAE <cit.> and ST-MAE <cit.> directly reconstruct randomly masked 3D video patches, achieving promising results on video benchmarks. Recent work explores whether motion-based priors can lead to improved masking algorithms for video. MGM <cit.> and MGMAE <cit.> mask the video patches with highest motion, under the assumption that higher motion co-occurs with higher saliency. These motion-guided masks enhance video representations compared to random masking. In contrast to these works, our work leverages textual information as a guide for where to mask, and also introduces a masked video-text contrastive loss that has not been explored yet by video MAE works to the best of our knowledge.



 §.§ Contrastive Visual Pretraining

MOCO <cit.> and SimCLR <cit.> introduce contrastive learning as an image representation learning paradigm. The encoder is trained by forcing invariance between the encoded representation for two views of the same image, which are typically generated through image augmentations. In video domain, CVRL <cit.> and ρ-MoCo <cit.> extend contrastive learning to video domain by sampling two subclips from the same video and applying video augmentation. BRAVE <cit.> and LSTCL <cit.> sample overlapping short and long clips to enforce temporal correspondence. These works all use intra-instance positive samples which are generated from the same image/video either through augmentation or re-sampling.

Other works go beyond intra-instance positives to explore inter-instance positive pair sampling. NNCLR <cit.> uses nearest-neighbor images as positive samples to the anchor. Similarly, IIVCL <cit.> uses multiple nearest-neighbor videos as the positive samples to the anchor.

In contrast to these works, we do video-text contrastive learning rather than visual contrastive learning, and we apply masking on top of the video.



 §.§ Vision-Language Pretraining

CLIP <cit.> and related works such as ALIGN <cit.> popularized image-to-text contrastive learning on hundreds of millions of image-text pairs. FLIP <cit.> scales CLIP to higher throughput by introducing image masking. A second line of works such as CoCa <cit.> and Florence <cit.> have explored captioning as a vision-language pretraining task. CoCa additionally introduces an optional image-text contrastive loss. Other works apply masked modeling on both image and text. For example, M3AE <cit.> combines image patches and text tokens as the input to a unified masked autoencoder. In contrast to these works, we unify video MAE and masked video-text contrastive learning, do not use captioning as a task, and do not apply masking to text. We also pretrain from scratch on only ∼200K videos. We additionally propose our novel text-guided masking strategy.

Another line of work attempts to recreate the success of CLIP for video-text pretraining. CLIP4CLIP <cit.> takes pretrained CLIP and applies it frame-wise to video and explores different temporal aggregation strategies such as mean pooling and Transformer encoder to achieve a video-level embedding. CLIP4CLIP only explores retrieval tasks. ViCLIP <cit.> upgrades the ViT image encoder from CLIP with spatiotemporal attention to make it a video encoder, and trains on a self-curated high-quality dataset with 200M video-text pairs. InternVideo <cit.> alternates between video MAE pretraining and video-text contrastive learning using a different visual backbone for the contrastive learning. In contrast, our approach jointly optimizes the video MAE and video-text contrastive loss with the same visual backbone and the contrastive loss operates on the masked MAE encoder output. We do not use any cross-attention layers nor multimodal fusion. We additionally introduce our novel text-guided masking algorithm.



§ METHOD



 §.§ Revisiting Video Masked Autoencoders

Given a video V ∈ℝ^T × H × W × C, where T,H,W,C denote the number of frames, height, width, and RGB-channels, the video is typically first split into cubes of size t × h × w × C and processed with a patch embedding layer 𝒫 to obtain a sequence of cube embeddings V_p. Typically t = 2 and h = w = 16.


    V_p = 𝒫(V); V_p ∈ℝ^T/t×H/h×W/w× D


Next, a masking function η (e.g. random <cit.>, tube <cit.>, or motion-guided <cit.>) generates a binary mask M to select a set of visible patches with mask ratio γ.


    M    = η(V_p,γ)
    
        V_p_visible   = V_p ⊙ (∼ M) 
    
        V_p_masked   = V_p ⊙ M


The encoder ϕ then processes only the visible patches V_p_visible while the decoder ξ processes the full set of encoded patches and masked tokens ϕ(V_p_visible) ∪ V_p_masked to reconstruct the video. This work uses the same asymmetric encoder-decoder design as <cit.>.


    E=ϕ(V_p_visible),   
        V'=ξ(E ∪ V_p_masked)


Finally, the model is trained with the MSE reconstruction loss ℒ_MSE, which is computed between V and V'. In this work, we propose a novel mask generator η(γ) which is guided by text.



 §.§ Caption Generation

Because Kinetics-400 and Something-Something v2 do not have human annotated captions, we utilize BLIP-2 <cit.> offline to generate video-text pairs for pretraining. For each pretraining video, we uniformly sample 3 keyframes and inference a caption per frame, for a total of 3 captions per video. During pretraining, we randomly sample from these 3 captions per video to form the text-video pair. Note that the captioning model is only used offline to obtain captions and does not receive any gradients during training. Note that using off-shelf captioning models for video is convenient but leads to noisier captions than human annotation. Other caption sources are ablated in <Ref>.



 §.§ Text-Guided Masking

We leverage the aligned representation space of CLIP <cit.> to compute a text-guided mask. First, we compute the visual features. For each frame f_t, we compute the feature map V_t ∈ℝ^H/h×W/w× D using ViT-B/32. This is done by patchifying each frame and resizing each patch to full input resolution. We then take the CLS token as the embedding per patch. To get the text embedding w ∈ℝ^D, we follow CLIP and take the activation map from the last layer of the transformer at the [] token. We then compute the cosine similarity between V_t and w and take the top k patches by text-video cosine similarity to form the binary mask ℳ_t, where k = H/h·W/w·γ and ∑ℳ_t / ℓ(ℳ_t) = γ to satisfy the masking ratio γ. We then apply ℳ_t per frame f_t to obtain the visible V_p_visible and masked V_p_masked.



 §.§ Video-Text Alignment

The video-text contrastive loss is a standalone module that can be optionally applied on top of the MAE pipeline for improved performance. The MAE encoder ϕ already processes only the visible patches V_p_visible, so no additional computation from the encoder is required for the video-text contrastive loss, as shown in <Ref>. Let i index the mini-batch. The global video embedding v_i for video i is computed by mean pooling ϕ(V_p_visible) across all patches. We then compute (ℒ^NCE(v_i, t_i, t_j ≠ i) + ℒ^NCE(t_i, v_i, v_j ≠ i)) / 2 over the mini-batch of size N, where the negative samples are all other text embeddings t_j ≠ i and video embeddings v_j ≠ i respectively. Similar to SimCLR <cit.>, we use a prediction head with global batch norm. We note that one convenience of masking is that we get large batch size by design, which is beneficial for contrastive learning.

The InfoNCE loss <cit.> ℒ^NCE maximizes the similarity of a given sample q with its positive key k^+, while minimizing similarity to negative samples 𝒩^-:


    ℒ^NCE(q, k^+, 𝒩^-) = -logexp(sim(q, k^+) / τ)/∑_k ∈{k^+}∪𝒩^-exp(sim(q, k) / τ)

where τ > 0 is a temperature hyper-parameter and sim(·) denotes the similarity function — which in this work is the dot product (cosine) similarity between two ℓ_2 normalized vectors: sim(q, k) = q · k = q^Tk / ( ‖ q ‖‖ k ‖ ). 



 §.§ Text-Guided MAE

The final loss is either ℒ_MSE in the case of pure MAE, or ℒ_MSE + ℒ^NCE when MAE and video-text contrastive loss are combined. After this self-supervised pretraining, the model is then transferred to downstream tasks such as classification via finetuning and linear probe with cross-entropy loss.



§ RESULTS





 §.§ Datasets

We conduct experiments on six commonly used datasets:

Something-Something V2 (SSv2) <cit.> contains 220K videos with 174 action classes. SSv2 is considered a motion heavy dataset, as most of the labels are defined by the motion and directionality of the actual action. Kinetics-400 (K400) <cit.> is the de-facto standard dataset used to evaluate video recognition. It contains 240K Internet videos with 400 action classes. UCF101 <cit.> is a dataset containing 13K Internet short videos with 101 action classes. HMDB51 <cit.> is a dataset containing 5K short movie clips from 51 action classes. Diving48 <cit.> contains 18K untrimmed video clips from 48 action classes, all of which are types of dives. We report the top-1 accuracy on the evaluation set for all datasets following standard practices <cit.>. Only UCF101 and HMDB51 have multiple split versions; we use split 1. Epic-Kitchens 55 <cit.> contains around 30K egocentric first-person video clips from nearly 3K action classes. Egocentric videos feature heavy occlusion, camera motion, and jitter.



 §.§ Implementation Details

Model Configuration:
The default backbone is ViT-Base <cit.> with global joint space-time attention. For fair comparison, we use the same input patch size of 2 × 16 × 16 for all models following <cit.>.

Pre-Processing: 
We pretrain with clips of 16 frames sampled at a temporal stride of 4 for K400 and stride of 2 for SSv2 respectively following <cit.>. We use a fixed spatial resolution of 224 × 224 for all experiments. We apply multi-scale-crop and horizontal flip augmentation by default (flip is not applied to SSv2). We follow <cit.> to use AdamW <cit.> optimizer with a base learning rate 1.5e-4, weight decay of 0.05, β=[0.9, 0.95], and cosine learning rate decay. 


Finetuning and Linear Probe:
We use the same 16-frame clip for finetuning and multi-view evaluation protocol following standard practice <cit.>. We use TSN-style sampling <cit.> on SSv2 dataset with 2 temporal × 3 spatial views during test-time following <cit.> for fair comparison. For Kinetics-400, UCF101, HMDB51, Diving48, and Epic-Kitchens 55, we use 5 temporal × 3 spatial views during test-time following <cit.> for fair comparison. See the supplementary material for  hyperparameter details which are mostly the same as <cit.>.


 



 §.§ Text-Guided Masking

We first apply our  to pure MAE (no contrastive learning) on SSv2 and K400 in <Ref> and compare to random tube masking <cit.> and motion-guided masking <cit.>. We chose MGM <cit.> over MGMAE <cit.> due to the higher scalability of motion vectors than optical flow.

On SSv2, our  achieves better finetune and linear probe performance than tube masking. On K400, our  achieves better finetune performance than both motion-guided masking and tube masking, and better linear probe performance than motion-guided masking. We do not claim state-of-the-art results, but instead emphasize the surprising and useful insight that text-guided masking is already competitive with other state-of-the-art masking algorithms — without leveraging any explicit visual cues for saliency such as motion vectors.




 §.§ MAE with Masked Video-Text Contrastive Learning


Next, we introduce the masked video-text contrastive loss as a mask-agnostic module that can be used both by itself, or optionally combined with MAE. We evaluate downstream finetune and linear probe performance for three masking algorithms on SSv2 in <Ref>. Recall that previous video MAE works do not explore masked video-text contrastive learning, so our results and insights for random tube masking and motion-guided masking are new.



Comparing within same masking algorithm.
In this section, we focus on each masking algorithm individually for an apple-to-apple comparison. First, note that pure masked video-text contrastive learning does not achieve competitive results for any masking algorithm. For instance, with random tube masking, pure masked video-text contrastive learning trails pure MAE with random tube masking by over 20% in finetune and 8% in linear probe performance. Second, we note that when MAE and contrastive loss are combined, linear probe performance improves notably compared to pure MAE with the same masking algorithm. For example, this boost is 12.5% for tube masking, 4.1% for motion-guided masking, and 7.2% for our . Finetune performance also improves by 0.6% for tube masking and by 0.4% for . In the case of motion-guided masking, a small drop of 0.3% in finetune performance is counterbalanced by a 4.1% improvement in linear probe performance. Overall, the results indicate that the masked video-text contrastive algorithm is synergistic with MAE pretraining, and that this benefit is general across multiple masking algorithms.

Comparing across different masking algorithms.
With the combined MAE and contrastive loss, we see that  achieves the highest finetune performance (+0.5% over MGM and +2.0% over tube masking), while MGM achieves the highest linear probe performance. However,  still achieves a reasonable trade-off which is competitive with both tube and motion-guided masking. Thus, in subsequent experiments, we primarily focus on our  to evaluate its generalizability to other downstream tasks and yield additional insights.






 §.§ Transfer Learning



  §.§.§ Small Action Recognition Datasets.

We next evaluate  pretrained on K400 when transferred to smaller action recognition datasets: UCF101 <cit.> (13K videos), HDMB51 <cit.> (5K videos), and Diving48 <cit.> (18K videos). These datasets range greatly in content diversity; UCF101 contains Internet videos, HDMB51 contains cinematic clips, and Diving48 contains sports videos. Motivated by the poor linear probe performance of previous MAE works in both image and video domain <cit.>, we focus on evaluation methods that do not finetune the backbone. Specifically, we use both linear evaluation and zero-shot retrieval to see whether  has learned sufficient semantics to generalize even when limited labeled data is available. First, we compare  without contrastive learning to tube masking and motion-guided masking, and see that  is competitive in both the linear probe and zero-shot retrieval settings. When combined with contrastive learning,  achieves a notable performance boost of up to 22.7% in linear probe over  without contrastive learning, as well as up to a 18% boost in recall@1. Thus, we see that the findings from <Ref> and <Ref> still hold true in this small dataset setting.




  §.§.§ Egocentric Action Recognition.

To further challenge , we shift to a new task of egocentric action recognition which is uniquely challenging due to featuring first-person perspectives. Egocentric video contains high occlusion, camera motion, and jitter, so it is an interesting setting for evaluating whether our text-guided masking can still be competitive with motion-guided masking — despite not leveraging any visual cues. In <Ref>, we see that without contrastive learning, our  is competitive with both tube and motion-guided masking and only suffers a minor performance drop. With contrastive learning,  improves by 2.5% in finetune and 5.7% in linear probe performance, and also outperforms MGM with contrastive learning. This is surprising since MGM explicitly models motion while our  does not leverage explicit visual cues for where to mask. These results indicate that the semantic alignment from video-text contrastive learning is still synergistic with MAE in this high-motion setting.





 §.§ Ablations



Mask Ratio.   In <Ref>, we ablate the mask ratio used for  with contrastive learning when pretrained on SSv2 for 200 epochs. The optimal masking ratio for our text-guided mask is 0.6, which is significantly lower than other video MAE works. For example, the optimal mask ratio in VideoMAE <cit.>, ST-MAE <cit.>, and MGMAE <cit.> is 0.9, while the optimal mask ratio in MGM <cit.> is 0.75. This suggests that  masks more information dense regions.

Top vs. Bottom-K Textual Similarity.   In <Ref>, we ablate the choice to mask the top patches by visual-to-text similarity. When we instead mask the bottom patches with lowest visual-to-text similarity, there is a slight drop in performance in both finetune and linear evaluation. However, even with bottom-K sampling, finetune performance is still better than random masking, while linear probe performance is slightly better. We hypothesis that textual guidance is more structured than random masking, but sampling patches with lower CLIP similarity makes the text-to-video alignment noisier.

Number of Captions.   In <Ref>, we ablate the number of captions per video. We sample either the center frame or 3 uniformly spaced frames and observe that both finetune and linear probe performance is better with more diverse captions, however the center frame caption already provides strong performance.



 §.§ Visualizations

We offer visualizations from three perspectives. First, we visualize the masking algorithm and see that the text-guided mask masks the most salient regions of the video matching the natural language description generated by BLIP for the video. Second, we see that the reconstruction quality is decent, meaning the model has learned to solve the reconstruction task. Third, we plot the encoder attention map using the center patch of the center frame as the query. The encoder roughly attends the salient regions of the video. Overall, this suggests that our model has achieved good alignment with the text while solving the MAE reconstruction task. We emphasize that visualizations are not intended to provide a formal explanation for model behavior. Our intention is to provide additional insights into the model to complement our quantitative results.





§ DISCUSSION AND LIMITATIONS

Unifying MAE and Contrastive Learning.   The performance of previous contrastive learning works such as CLIP <cit.> may make the boost from contrastive learning observed in <Ref>, <ref>, <ref> seem obvious. We offer two perspectives for why these results are insightful.

First, previous contrastive learning works leverage pretrained weights and/or are trained on hundreds of millions of image/video-text pairs – which is several orders of magnitude more data than what we use in this paper (roughly 200K video-text pairs). For instance, CLIP <cit.> and FLIP <cit.> train on 400 million image-text pairs and ViCLIP <cit.> is trained on 200 million video-text pairs. We showed in <Ref> that on Kinetics-400 and Something-Something, pure video-text contrastive learning achieves much lower performance than pure MAE. The benefits of video-text contrastive learning when training from scratch on our scale of data are only realized when combined with MAE.

Second, careful design is required to unify the generative nature of MAE and discriminative nature of contrastive learning. Previous work even suggests that MAE and video-text contrastive learning are antagonistic, which conflicts with our findings. For instance, FLIP <cit.> reports degraded finetuning performance when combining image MAE with masked contrastive image-text learning. It is not obvious that MAE and video-text contrastive learning are synergistic for video. Our empirical results contribute the useful insight that this is not only the case, but also that this benefit can be realized across multiple masking algorithms – even with noisy machine-generated captions on “regular” sized datasets.




r0.45
    

    
        
    < g r a p h i c s >

    
    

    Contrastive loss for each mask alg. with and without optimization.
    
    




We leave better theoretical understanding of why these paradigms are complementary to future work. To provide an additional perspective, we plot the contrastive loss with and without optimization of the contrastive loss for all three mask algorithms in <Ref>. We observe that even for pure MAE, the contrastive loss naturally decreases. This suggests that the MAE encoder already learns semantics that somewhat align with text, even when there is no textual supervision. This further motivates the benefit of unifying MAE and contrastive learning to facilitate semantic-aware reconstruction.

Choice of Captions.   Our work utilizes frame-wise captions from BLIP-2 <cit.> which is an off-shelf image captioning model. An interesting question is whether our performance is dependent on this particular captioning model, and whether higher model capacity helps. In <Ref>, we first test an oracle captioner that directly outputs the action label. We then use GPT3.5 <cit.> which is a large language model with several orders of magnitude more parameters than BLIP. GPT3.5 is a pure language model so it requires a textual prompt. Previous works <cit.> demonstrate that GPT-3 is capable of generating detailed descriptions about specific object and action categories with only textual inputs. Following CuPL <cit.>, we utilize three prompt templates listed in <Ref> to generate “vision-free” captions per action class. We then utilize these vision-free captions for pretraining to assess the effect of large-language model generated captions in our combined framework.

We see that linear probe performance for both the oracle and GPT3.5 is much higher than BLIP, while finetune performance drops. We posit that this is primarily because the oracle and GPT3.5 are vision-free, so the generated caption is not guaranteed to capture the granular visual details of any video. However, the textual caption generated by GPT3.5 may abstractly match the action class better since it is asked to describe each action class in general, and GPT3.5 has several orders of magnitude more parameters and training data. In summary, more model capacity and external domain knowledge does not necessarily translate to better performance in our framework, however vision-free captions are still surprisingly useful. Combining BLIP and GPT3.5 captions degrades performance probably due to the conflicting nature of vision-free captions.




Computational Efficiency.   Although mask generation requires CLIP inference, the wall clock time for training is not significantly higher. This is because the high mask ratio and asymmetric encoder-decoder design of the MAE pipeline contribute to dataloading itself being the primary bottleneck <cit.>. BLIP and CLIP both are frozen.

Limitations.   One limitation is the reliance upon captions. However, the captions used in our work are far from perfect. For instance, BLIP is an image captioning model and frame-wise captioning may not capture temporal details of video. Despite this limitation, we showed the efficacy of these noisy captions within our framework. Video captioning is a difficult problem and we expect that as state-of-the-art improves, our performance would also improve. Another limitation is that strong vision-text pretraining is needed to leverage captions for mask generation. However, the growing availability of CLIP-like models makes research into how to leverage these resources all the more timely, especially for designing real-world systems. We have proposed both a new direction of research and a simple yet effective baseline that offers room for further research.



§ CONCLUSION

We motivated a new research direction into leveraging natural language to improve masked video representation learning. We first presented , a novel masking algorithm that masks the regions of video with highest alignment to text captions. Next, we introduced masked video-to-text contrastive learning as an optional module that can be combined with video MAE to enrich semantic learning across a host of masking algorithms. When combined with specifically our , we observe improved performance in finetuning, linear probe, and even zero-shot retrieval across six different downstream datasets. Our approach is simple yet effective and sets a baseline for future research in this new direction of language-guided MAE.

Acknowledgements.   We thank Linda Liu and Pichao Wang for their valuable feedback on this work.










splncs04








 Paper Title: Gradient Harmonization in Unsupervised Domain Adaptation.


























































Journal of  Class Files, Vol. 14, No. 8, August 2015
Shell et al.: Bare Demo of IEEEtran.cls for Computer Society Journals





































§ ABSTRACT 
Unsupervised domain adaptation (UDA) intends to transfer knowledge from a labeled source domain to an unlabeled target domain. Many current methods focus on learning feature representations that are both discriminative for classification and invariant across domains by simultaneously optimizing domain alignment and classification tasks. However, these methods often overlook a crucial challenge: the inherent conflict between these two tasks during gradient-based optimization.
In this paper, we delve into this issue and introduce two effective solutions known as Gradient Harmonization, including GH and GH++, to mitigate the conflict between domain alignment and classification tasks. GH operates by altering the gradient angle between different tasks from an obtuse angle to an acute angle, thus resolving the conflict and trade-offing the two tasks in a coordinated manner. Yet, this would cause both tasks to deviate from their original optimization directions. We thus further propose an improved version, GH++, which adjusts the gradient angle between tasks from an obtuse angle to a vertical angle. This not only eliminates the conflict but also minimizes deviation from the original gradient directions.
Finally, for optimization convenience and efficiency, we evolve the gradient harmonization strategies into a dynamically weighted loss function using an integral operator on the harmonized gradient.
Notably, GH/GH++ are orthogonal to UDA and can be seamlessly integrated into most existing UDA models. Theoretical insights and experimental analyses demonstrate that the proposed approaches not only enhance popular UDA baselines but also improve recent state-of-the-art models.





Unsupervised Domain Adaptation, Gradient Harmonization, Transfer Learning, Image Classification.




Gradient Harmonization in Unsupervised Domain Adaptation
    Fuxiang Huang, 
         Suqi Song,
         Lei Zhang

        
This work was partially supported by National Key R&D Program of China (2021YFB3100800), National Natural Science Fund of China (62271090), Chongqing Natural Science Fund (cstc2021jcyj-jqX0023). This work is also supported by Huawei computational power of Chongqing Artificial Intelligence Innovation Center. (Corresponding author: Lei Zhang)

Fuxiang Huang, Suqi Song and Lei Zhang are with the School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China. (E-mail: huangfuxiang@cqu.edu.cn, songsuqi@stu.cqu.edu.cn, leizhang@cqu.edu.cn).








Manuscript received April 19, 2005; revised August 26, 2015.

    
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================






























§ INTRODUCTION




























Deep convolutional neural networks and transformers, driven by extensive labeled samples, have achieved remarkable success in various computer vision tasks such as classification, semantic segmentation and object detection.  However, these models often demonstrate high vulnerability when deployed in novel application scenarios due to data distribution discrepancies. The process of collecting and annotating data across various domains is expensive, labor-intensive and time-consuming. Consequently, Unsupervised Domain Adaptation (UDA) arises to transfer the knowledge from a labeled source domain to an unlabeled target domain <cit.>. 



In recent years, UDA algorithms have made significant progress in enhancing classification performance <cit.>. The primary focus of these approaches is to acquire domain-invariant feature representations, thereby achieving domain alignment and narrowing the probability distributions across domains. Currently, domain alignment methods fall into two main categories: distance metrics-based methods <cit.> and adversarial learning-based methods <cit.>. The distance metrics-based approaches align the source and target domains by mapping them into a shared feature space while minimizing the distribution disparities between them. Inspired by generative adversarial networks <cit.>, adversarial learning techniques have been introduced to tackle domain adaptation challenges <cit.>. For instance,  <cit.> is among the pioneering methods to achieve domain alignment by introducing a game between the feature extractor and domain classifier. <cit.> adjusts the adversarial domain adaptation models on discriminative information conveyed in the classifier predictions. These methods usually jointly optimize domain alignment and classification tasks in the course of training.

However, due to their objective differences between the two tasks, the optimal gradient descent directions from the two tasks may be uncoordinated or imbalanced. Therefore, directly sharing network parameters (i.e., feature generator) may result in optimization conflict between tasks and affect domain-invariant feature learning. During optimization, the angle between the gradients of the two tasks is sometimes an obtuse angle, indicating an obvious gradient conflict. As shown in Fig. <ref>, the black arrow and red arrow point to the optimal gradient descent direction of the domain alignment task and classification task, respectively. In the joint optimization process, obtuse angle formed by the two gradients of both tasks leads to optimization conflict, which reveals the sub-optimality of the two tasks.

To further verify the above observations, we conduct some experiments on real cross-domain classification scenarios. Fig. <ref> displays inner product distributions of two gradients between two tasks (alignment vs. classification) throughout the training process on task of MNIST → USPS. From Fig. <ref>, we can obtain the following two observations. First, it implies that the inner product between two gradients is approximately normally distributed with a mean slightly around above zero. Second, both acute and obtuse angles exist on the two models, where obtuse angles account for about 42% of the total number in MCD <cit.> and 37% in DWL <cit.> (DWL relieves the imbalance by trade-offing the transferability and discriminability to some extent but ignores optimization conflicts, whereas MCD does not take into account the conflict problem). In the joint optimization process, the aggregated gradient direction generated by two obtuse gradients will be seriously deviated from their respective optimal gradient descent directions, thereby resulting in sub-optimal alignment and classification. Further, we empirically observe that the obtuse angle is always present during training over time and cannot be eliminated automatically. Therefore, how to reasonably eliminate optimization conflicts while maintaining the task-specific optimality during training is a challenging and practical work. 

Multi-task learning (MTL), as explored in studies such as <cit.>, focuses on simultaneously optimizing multiple conflicting criteria. These approaches typically seek one or more Pareto optimal solutions that offer different trade-offs. Inspired by MTL, ParetoDA <cit.> introduces a target-classification-mimicking (TCM) loss based on held-out target data and dynamically seeks a desirable Pareto optimal solution for the target domain. However, Pareto optimization relies on additional information to make decisions and requires significant computational resources and time in high-dimensional or complex systems.

In this paper, to address the aforementioned challenges, we propose an idea of de-conflict on the gradients. Technically, we introduce two intuitive yet highly effective approaches called Gradient Harmonization, involving GH and GH++, which aim to alleviate optimization conflicts that emerge between the domain alignment and classification tasks by leveraging insights from geometric awareness in model optimization. Specifically, we first calculate the original gradients of two tasks to find out if there is a conflict, i.e., the angle between the gradients of the tasks is obtuse (negatively correlated). Subsequently, we employ GH to turn the angle between the original gradients of two tasks from an obtuse angle to an acute angle, i.e., positively correlated. This harmonization of gradients enables both tasks to progress in a coordinated manner. However, this process may cause both tasks to deviate from their original directions to compromise performance.
To further mitigate gradient deviation while maintaining the task-specific optimality, we introduce an improved version, GH++, which turns the angle between the original gradients of two tasks from an obtuse angle to a vertical angle, i.e., orthogonal. Then both tasks can evolve harmoniously during joint training while preserving their individual task-specific optimality.


Detailed theoretical support is also provided for the proposed approaches.

Furthermore, in order to facilitate optimization and improve computation efficiency, we derive an equivalent but more efficient model of GH/GH++ for unsupervised domain adaptation. The equivalent model is derived as a dynamic objective function, namely UDA+GH/GH++, which can achieve fast elimination of optimization conflicts without sacrificing the task-specific objective. The proposed GH/GH++ is a universal plug-and-play approach for between-task balance learning and can be easily embedded in most alignment-based unsupervised domain adaptation methods. This work is a new upgrade and more general form of our previous CVPR paper <cit.> from the perspective of gradient harmonization.

The main contributions and novelties of this paper are summarized as follows:


  * We verify and visualize the gradient conflict in UDA methods and develop two novel Gradient Harmonization approaches, including  GH and GH++. The proposed approaches alleviate the gradient conflict problem between the domain alignment task and classification task by adjusting their gradient angle from obtuse to acute/vertical angle adaptively. Fig. <ref> is a representative and illustrative example to depict the UDA model based on GH/GH++.

  * The proposed approaches are orthogonal to most existing UDA methods. To facilitate optimization and improve computation efficiency, we further derive the equivalent models, i.e., UDA with GH/GH++. Specifically, the proposed approaches can be evolved into dynamically weighted loss functions via an integral operation, and promote optimization.

  * Extensive experiments validate the effectiveness and universality of our approaches on various benchmarks in several mainstream UDA models. More insights and analyses justify the reasonability of the proposed approaches.





§ RELATED WORK




 §.§ Unsupervised Domain Adaptation (UDA)




Unsupervised Domain Adaptation (UDA) aims to leverage the knowledge learned from a labeled source dataset to solve similar tasks in a new unlabeled domain. Recent works <cit.> have focused on UDA based on domain alignment and discriminative feature learning methods.



Domain Alignment. The mainstream approach for UDA aims to realize domain alignment by learning domain-invariant feature representations across domains, which can be mainly summarized into two categories: distance metric based methods and adversarial learning based methods. First, the methods based on distance metrics mostly use several existing distance measurement indicators to assess the alignment degree across domains, such as MMD <cit.>, MDD <cit.> and their variants. With the popularity of deep learning methods, more and more researchers apply deep neural networks to domain adaptation. Compared with traditional non-deep domain adaptation methods, deep methods directly improve the learning effect on different classification tasks. Second, some domain adaptation methods based on adversarial training can learn the domain-invariant feature representations well and better implement knowledge transfer. For example, CDAN <cit.> proposes an adversarial adaptation model for the discriminative information transmitted in the prediction of the classifier. GVB <cit.> learns the domain-invariant feature representations by applying the gradually vanishing bridge mechanism on the feature generator.
Besides, considering the equilibrium problem of adversarial learning, FGDA <cit.> reduces the distribution discrepancy by constraining feature gradient. CGDM <cit.> explicitly minimizes the discrepancy of gradients generated by source samples and target samples to improve the accuracy of target samples. FixBi <cit.> introduces a fixed ratio-based mixup to augment multiple intermediate domains between the source and target domain.

Classification. Other approaches focus on improving the performance of the classification task by enhancing class discriminability feature learning. ETD <cit.> puts forward an attention-aware optimal transport distance to measure the domain discrepancy under the guidance of the prediction feedback and enables the model to learn distinguished feature representations. MCD <cit.> proposes to maximize the prediction discrepancy of two classifiers to obtain strong discriminant feature representations. BSP <cit.> penalizes the largest singular values so that other eigenvectors can be relatively strengthened to boost the feature discriminability. CAN <cit.> optimizes the metric for minimizing the domain discrepancy, which explicitly models the intra-class domain discrepancy and the inter-class domain discrepancy. RSDA <cit.> introduces a spherical classifier for label prediction and a spherical domain discriminator for discriminating domain labels and utilizes robust pseudo-label loss in the spherical feature space. DMRL <cit.> focuses on guiding the classifier to enhance consistent predictions between samples and enriches the intrinsic structures of the latent space. To combine the source and target domains, DMRL introduces two mixup regularizations based on randomness. Recently, SSRT <cit.> and TVT <cit.> learn both transferable and discriminative features by applying Vision Transformer (ViT).

Balancing Domain Alignment and Classification.
Previous approaches concentrate on learning domain-invariant and class-discriminative feature representations by jointly optimizing domain alignment and classification task but ignore imbalance or uncoordinated optimization problem between tasks.

Recently, some methods have started to pay attention to this problem and adopt strategies to alleviate it. To adapt to different cross-domain classification scenarios, DWL <cit.> proposes dynamic weighted learning to avoid the discriminability vanishing problem caused by excessive alignment and domain misalignment problem caused by excessive discriminant learning from a macro perspective, which ignores gradient conflict between two tasks during optimization.
Meta <cit.> maximizes inner product of the gradients of the two tasks during training. However, it cannot guarantee the adjusted gradient direction is close to the original optimal gradient descent direction, and even appears serious deviation. Since Meta processes the gradients of the two tasks in an extreme way, it may produce some negative feedback, such as sacrificing partial alignment or classification performance.
Recently, in order to mitigate the effects of conflicting gradients, ParetoDA <cit.> adopts Pareto optimization <cit.> to cooperatively optimize all training objectives, which searches for the desirable Pareto optimal solution on the entire Pareto front. However, it is computationally inefficient. In this paper, we propose two intuitive yet efficient approaches, including GH and GH++, to handle the gradient conflict.

Differences. The proposed approaches represent a fundamental departure from previous methods like DWL <cit.>, Meta <cit.>, and ParetoDA <cit.>. Our primary goal is to implicitly address the optimization conflict throughout the entire training process by active gradient harmonization, thereby ensuring a balanced learning between tasks. However, as depicted in Figure <ref>, the optimization conflict still persists in DWL. Additionally, ParetoDA introduces Pareto optimization and an additional target-classification-mimicking (TCM) loss, which necessitates the involvement of all class-wise discriminators. In contrast, the proposed GH/GH++ is evolved into a dynamically weighted loss function via an integral operator on the harmonized gradient. 






 §.§ Multi Task Learning

Multi-task learning (MTL) <cit.> aims at learning multiple tasks in a unified model to achieve mutual improvement among tasks considering their shared knowledge. By sharing parameters across tasks, MTL methods learn more efficiently with an overall smaller model size compared to learning with separate models.



Prior MTL approaches formulate the total objective as a weighted sum of task-specific objectives, such as DWA <cit.> and GradNorm <cit.>, which optimize weights based on task-specific learning rates or by random weighting. However, these weighted optimization based MLT may not achieve satisfactory performance due to the presence of gradient conflicts among different tasks. Gradient optimization based MLT <cit.> overcome this limitation, mitigating effects of conflicting or dominating gradients. MGDA <cit.> proposes to simply update the shared network parameters along a descent direction which leads to solutions that dominate the previous one. PCGrad <cit.> proposes a “gradient surgery” to avoid gradient conflicts. CAGrad <cit.> seeks the gradient direction within the neighborhood of the average gradient and maximizes the worst local improvement of any objective. <cit.> merge the gradients of auxiliary tasks and applied a scaling factor to adaptively adjust their impact on the main tasks, followed by applying gradient sharing between the main tasks and the merged auxiliary task. GradDrop <cit.> proposes a probabilistic masking procedure, which samples gradients at an activation layer based on their level of consistency. Aligned-MTL <cit.> eliminates instability in the training process by aligning the orthogonal components of the linear system of gradients.



In this study, we introduce two simple yet highly efficient approaches, GH and GH++,  to effectively address gradient conflicts between any two distinct tasks. Detailed theoretical derivations and fundamental insights regarding the proposed approaches are elaborated in Sections <ref>. Additionally, under thorough theoretical analysis and a wealth of experiments, we have substantiated the soundness and effectiveness of the proposed methodologies.







§ PROPOSED APPROACH





 §.§ Problem Definition

Given a labeled source domain 𝒟_s={x_i^s,y_i^s}_i=1^n_s with n_s samples and an unlabeled target domain 𝒟_t={x_j^t}_j=1^n_t with n_t samples, where y_i^s is the class label of the i^th source sample x_i^s. 𝒟_s and 𝒟_t share the same feature space and category space, but have different data distributions. Our purpose is to utilize the labeled data 𝒟_s and unlabeled data 𝒟_t to learn a deep model, which can accurately predict the class label of samples in the target domain.




 §.§ A General Framework of UDA


Adversarial learning has proven to be an effective method for domain alignment, starting from Domain Adversarial Neural Network (DANN) <cit.>. The basic idea is to trick the domain discriminator D by generating features via a feature generator G. Then the domain discriminator predicts whether the generated feature by G is from the source domain or the target domain. The training of domain alignment is achieved through the game between generator and domain discriminator. The parameter θ_g of generator G and the parameter θ_d of domain discriminator D are optimized by the following domain alignment objective function.

    ℒ_dom(θ_g,θ_d)=    𝔼_x_i^s∼𝒟_slog[D(G(x_i^s))]+
            𝔼_x_j^t∼𝒟_tlog[1-D(G(x_j^t))].


In order to improve the classification performance of the target domain samples, we must first ensure that the classifier C can correctly classify the samples from the source domain. Thus, the supervised classification loss can be described as

    ℒ_cls(θ_g,θ_c)=1/N_s∑_i=1^N_sℒ_ce(C(G(x_i^s;θ_g);θ_c),y_i^s),

where ℒ_ce is the standard cross-entropy loss function.

During training stage, the existing methods usually jointly optimize the two objective functions (ℒ_dom and ℒ_cls) to obtain domain-invariant and class-discriminant feature representation. The overall minimax objective function is

    θ_g,θ_cmin  θ_dmax  ℒ_dom+ℒ_cls,

where θ_g, θ_d, θ_c denote the parameters of feature generator, domain discriminator and classifier, respectively.






 §.§ Gradient Harmonization (GH)


Domain alignment and classification are two different tasks. Their optimal gradient descent directions may not be coordinated, which results in the optimization conflict of the two loss functions in the training process and deteriorates the final domain adaptation performance.

In order to ensure that the two target tasks can be optimized in a coordinated manner, we propose an idea of de-conflict on the gradients, which then formulates the proposed GH technique, i.e., altering the gradient angle between different tasks from an obtuse angle to an acute angle. The de-conflict process for the gradients (i.e., g_1 and g_2) is schematically shown in Fig. <ref>. Specifically, we first provide three Lemmas to support our idea. Then the proposed GH is summarized as Theorem 1. For convenience, we define ℒ_1(Θ) and ℒ_2(Θ) as the general form of any two conflicted loss functions. In UDA, ℒ_1(Θ) and ℒ_2(Θ) represent the domain alignment loss ℒ_dom(θ_g,θ_d) and the classification loss ℒ_cls(θ_g,θ_c), respectively. Θ={θ_g, θ_d,θ_c} indicates model parameters of generator, discriminator and classifier. In fact, the three Lemmas aim to deduce the gradient harmonization formula during model optimization (training) phase, by exhausted mathematical solving process.


Given two objective functions ℒ_1(Θ) and ℒ_2(Θ), we define g_1 and g_2 as their gradient, respectively, and g̃_1 is the result of harmonizing the gradient g_1. For minimizing the objective ℒ_1(Θ-g̃_1)+ℒ_2(Θ-g̃_1), we consider to first minimize the objective ℒ_1(Θ-g̃_1), and then there is,

    g̃_1=g_1-δ(g_1^Tg_2<0)g_1^Tg_2/‖ g_2 ‖^2g_2,

where δ(·) represents the indicator function whose value is 0 or 1 and the mathematical expression is:

    δ(A) =
    
    1,      if A is true 
    
    0,      if A is false







In order to minimize the loss function ℒ_1(Θ-g̃_1), the problem is transformed into solving the following model:

    g̃_1min  ℒ_1(Θ-g̃_1)  
    
    s.t.    ℒ_1(Θ-g̃_1)≤ℒ_1(Θ),  ℒ_2(Θ-g̃_1)≤ℒ_2(Θ).

Due to the gradient g_1=∇_Θℒ_1(Θ) is the fastest descent direction of the objective function ℒ_1(Θ), if the harmonic gradient g̃_1 is the same as g_1 as much as possible, the loss function ℒ_1(Θ) will be optimized along the gradient descent direction. If the angle between the harmonic gradient g̃_1 and g_1 is an acute angle, then g̃_1 points to the direction of gradient descent. Naturally, the angle between g̃_1 and g_2=∇_Θℒ_2(Θ) is also expected to be an acute angle. Therefore, the above model is transformed into:

    g̃_1min  1/2‖ g_1-g̃_1‖^2  
       s.t. g̃_1^Tg_1≥0, g̃_1^Tg_2≥0.

The problem (<ref>) is a convex optimization problem. To solve the problem (<ref>), we define the Lagrangian function










    ℒ(g̃_1,α_1,α_2)=1/2‖ g_1-g̃_1‖^2
        -α_1g̃_1^Tg_1-α_2g̃_1^Tg_2,

where α_1, α_2 are the Lagrangian multipliers associated with the inequality constraint of problem (<ref>). To simplify the solution, first note that

    P=    max_α_1≥0,α_2≥0ℒ(g̃_1,α_1,α_2) 
    
    =   {[        1/2‖ g_1-g̃_1‖^2, g̃_1^Tg_1≥0, g̃_1^Tg_2≥0;                      +∞,                otherwise;                          ].

This means that we can express the optimal value of the primal problem (<ref>) as:

    p^∗ = min_g̃_1  max_α_1≥0,α_2≥0  ℒ(g̃_1,α_1,α_2).

By the definition of the dual function, we can express the optimal value of the dual problem:


    d^∗=α_1≥0,α_2≥0max  g̃_1min  ℒ(g̃_1,α_1,α_2),

where α_1≥0,α_2≥0 is the dual feasibility condition. It is easy to get the weak duality d^∗≤ p^∗. When the strong duality holds, i.e., d^∗ = p^∗, we have

    α_1g̃_1^Tg_1=0, α_2g̃_1^Tg_2=0.

Eq. (<ref>) is known as complementary slackness condition. To summarize, for convex  optimization problem (<ref>), we have











    {[        ∇_g̃_1ℒ(g̃_1,α_1,α_2)=0,                               1;       g̃_1^Tg_1≥0, g̃_1^Tg_2≥0,                               2;                    α_1≥0,α_2≥0,                               3; α_1g̃_1^Tg_1=0, α_2g̃_1^Tg_2=0.                               4;                                 ].

Conditions (<ref>) are called the Karush-Kuhn-Tucker (KKT) conditions<cit.>, which is the the necessary condition for pair of primal and dual optimal points. 

We can solve the dual problem Eq. (<ref>). As for the inner optimization problem, i.e., D=g̃_1min  ℒ(g̃_1,α_1,α_2), we then obtain its optimal solution by setting its derivative with respect to g̃_1 as zero. Then,

    g̃_1=g_1+α_1g_1+α_2g_2.


We can calculate the extremum of D by substituting Eq. (<ref>) into Eq. (<ref>) :

    D   =g̃_1min  ℒ(g̃_1,α_1,α_2)
    
           =1/2g̃_1^Tg̃_1 -g̃_1^Tg_1-α_1g̃_1^Tg_1-α_2g̃_1^Tg_2+const
       =1/2g̃_1^Tg̃_1- g̃_1^T(g_1+α_1g_1+α_2g_2)+const
       = -1/2g̃_1^Tg̃_1+const
       =-1/2‖ g_1+α_1g_1+α_2g_2‖^2+const.

We then get the outer optimization problem by substituting Eq. (<ref>) into Eq. (<ref>) as follows:

    α_1≥0,α_2≥0max  -1/2‖ g_1+α_1g_1+α_2g_2‖^2.

For the problem (<ref>), we can take the derivative of α_1 and α_2 respectively, and set the derivative to 0. Then we can obtain the optimal solution: α_1^*=-1, α_2^*=0. Obviously, the obtained α_1^* does not satisfy the dual feasibility condition. Therefore, it is necessary to find the boundary solution that satisfies the constraints, which is established under two possible cases:



Case 1: Suppose α_1^*=0, the Eq. (<ref>) is transformed into:

    α_2min  1/2‖ g_1+α_2g_2‖^2.

We can set the derivative of α_2 to 0, i.e., (g_1+α_2 g_2)^T g_2 = 0. Then we can obtain the optimal solution:




    α_2^*=-g_1^Tg_2/‖ g_2 ‖^2.

Obviously, Eq. (<ref>) satisfies the KKT conditions 124 in Eq. (<ref>). According to dual feasibility, i.e., 3 in Eq. (<ref>), α_2^*=-g_1^Tg_2/‖ g_2 ‖^2≥0, then we have g_1^Tg_2≤0. Therefore, we can obtain a boundary solution

    α_1^*=0,
    α_2^*=-g_1^Tg_2/‖ g_2 ‖^2,    g_1^Tg_2≤0


Case 2: Suppose α_2^*=0, the Eq. (<ref>) is transformed into:

    α_1min  1/2‖ g_1+α_1g_1‖^2.

Then we can obtain α_1^*=0 in nature. Obviously, it satisfies the KKT conditions 134 in Eq. (<ref>). According to primal feasibility, i.e., 2 in Eq. (<ref>), we have g_1^Tg_2≥0. Therefore, the boundary solution is

    α_1^*=0,
    α_2^*=0,     g_1^Tg_2≥0






In summary, the optimal solution of problem (<ref>) is:

    α_1^*=0
    α_2^*=-δ(g_1^Tg_2<0)g_1^Tg_2/‖ g_2 ‖^2.

Then the optimal value of the primal problem (<ref>) g̃ can be obtained by substituting Eq. (<ref>) into Eq. (<ref>).


    g̃_1=g_1-δ(g_1^Tg_2<0)g_1^Tg_2/‖ g_2 ‖^2g_2,

where g̃_1 is the result of harmonizing the gradient g_1 of ℒ_1. Then the proof of Eq. (<ref>) in Lemma 1 is completed.

The above process not only solves the problem (<ref>) but also harmonizes the gradient g_1 of ℒ_1(Θ), i.e., Fig. <ref> (b1).



Define g̃_2 as the result of harmonizing the gradient g_2, we consider to further minimize the optimization objective ℒ_2(Θ-g̃_2), then there is,

    g̃_2=g_2-δ(g_1^Tg_2<0)g_2^Tg_1/‖ g_1 ‖^2g_1.







In order to minimize the loss function ℒ_2(Θ-g̃_2), similar to Lemma 1, we need to solve the following minimization problem.

    g̃min  ℒ_2(Θ-g̃_2)  
    
    s.t.    ℒ_1(Θ-g̃_2)≤ℒ_1(Θ), ℒ_2(Θ-g̃_2)≤ℒ_2(Θ).

The above model can be transformed into

    g̃_2min  1/2‖ g_2-g̃_2‖^2  
       s.t. g̃_2^Tg_1≥0, g̃_2^Tg_2≥0.

Similar to problem (<ref>), by solving (<ref>), we can obtain

    g̃_2=g_2-δ(g_1^Tg_2<0)g_2^Tg_1/‖ g_1 ‖^2g_1,

where g̃_2 is the result of harmonizing the gradient g_2 of ℒ_2(Θ), i.e., Fig. <ref> (b2). 





Define the overall loss function ℒ=ℒ_1(Θ)+ℒ_2(Θ) of two tasks, and g̃ denotes the overall harmonized gradient. We consider the whole optimization objective ℒ_1(Θ-g̃)+ℒ_2(Θ-g̃), then there is,

    g̃=   g̃_1+g̃_2
    
        =   g_1+g_2-δ(g_1^Tg_2<0)g_1^Tg_2/‖ g_2 ‖^2g_2-δ(g_1^Tg_2<0)g_2^Tg_1/‖ g_1 ‖^2g_1.












According to Lemma 1 and Lemma 2, the results of harmonizing the gradients g_1 and g_2, i.e., g̃_1 and g̃_2, can be obtained. By aggregating Eq. (<ref>) in Lemma 1 and Eq. (<ref>) in Lemma 2, the whole gradient after harmonization can be obtained as Eq. (<ref>), and proof of Lemma 3 is completed.



For brevity, we summarize the above process of GH in Theorem 1.


For any two different tasks, optimization conflicts can be eliminated by Gradient Harmonization (GH). Define the overall loss function ℒ (Θ)=ℒ_1 (Θ)+ℒ_2 (Θ), composed of two sub-objectives (refer to domain alignment and classification in this paper), and g_1 and g_2 as the gradient of ℒ_1 (Θ) and ℒ_2 (Θ). Then the overall harmonized gradient g̃ of the whole loss ℒ (Θ) with GH can be formulated as:

    g̃=g_1+g_2-δ(g_1^Tg_2<0)g_1^Tg_2/‖ g_2 ‖^2g_2-δ(g_1^Tg_2<0)g_2^Tg_1/‖ g_1 ‖^2g_1.


















Obviously, in the optimization process of standard SGD algorithm, the overall gradient of the loss ℒ is g=g_1+g_2. In this paper, two cases 
are considered based on the correlation between gradients g_1 and g_2.

Case 1: When g_1 and g_2 are positively correlated or unrelated, i.e., cos(g_1,g_2)≥0 or g_1^Tg_2≥0, the angle between the two gradients is an acute or vertical angle. Then, we believe that there is no conflict between the two objectives in training, that is, two gradients are coordinated or balanced. Thus g_1 and g_2 do not need to be harmonized.



Case 2: When g_1 and g_2 are negatively correlated, i.e., cos(g_1,g_2)<0 or g_1^Tg_2<0, the angle between the two gradients is an obtuse angle. Then, we believe that there is a gradient optimization conflict between the two objectives. In this case, g_1 and g_2 need to be harmonized. For the overall loss function ℒ (Θ)=ℒ_1(Θ)+ℒ_2(Θ) of an arbitrary model, the proposed UDA+GH is to find a gradient g̃ that satisfies the following problem:



















    g̃min  ℒ_1(Θ-g̃)+ℒ_2(Θ-g̃),

where Eq. (<ref>) can be solved by aforementioned three lemmas. Firstly, the harmonic gradient g̃_1 of g_1 can be obtained by Lemma 1. Then the harmonic gradient g̃_2 of g_2 can be obtained by Lemma 2.  Finally, as can be seen from Lemma 3, it is easy to acquire the whole aggregated gradient g in Eq. (<ref>) after harmonization and the gradient aggregation process is visualized in Fig. <ref>.











 §.§ Essence and Insights of GH


To understand the essence of GH more intuitively, we demonstrate the proposed gradient harmonization by backward inference in this section. Fig. <ref> (a) and (b) show the essential analysis by performing gradient harmonization on original gradients g_1 and g_2, respectively. Note that we mainly consider the case where gradient conflict exists, that is, the angle between g_1 and g_2 is an obtuse angle.

First, we analyze the essence of gradient harmonization for g_1 (i.e., Fig. <ref> (a)). θ denotes the angle between original gradients g_1 and g_2. n is the projection of g_1 on the reversely extended line of g_2. m is perpendicular to g_2. Note that each letter except θ on the figure represents a vector. Naturally, m can be derived as Eq. (<ref>), from which we can see that the expression of the vector m is the same as Eq. (<ref>) when g_1^Tg_2<0. That is, the vector m is the result of gradient harmonization of g_1 (i.e., g̃_1).

    m=g_1-n=   g_1-|g_1|· cos(π-θ)· (-g_2/|g_2|)
    
        =   g_1-|g_1|· cosθ·g_2/|g_2|
    
        =   g_1-|g_1|·⟨ g_1,g_2⟩/|g_1|· |g_2|·g_2/|g_2|
    
        =   g_1-⟨ g_1,g_2⟩/g_2^2g_2.

where ⟨·⟩ is the inner product operator.



Second, we analyze the essence of gradient harmonization for g_2 (i.e., Fig. <ref> (b)). Similar to (a), the vector e is the projection of g_2 on the reversely extended line of g_1. The vector f is perpendicular to g_1. Through the derivation of Eq. (<ref>), it can be found that the expression of the vector f is the same as Eq. (<ref>) when g_1^Tg_2<0. That is, the vector f is the result of harmonizing the gradient g_2 (i.e., g̃_2).

    f=g_2-e=   g_2-|g_2|· cos(π-θ)· (-g_1/|g_1|)
    
        =   g_2-|g_2|· cosθ·g_1/|g_1|
    
        =   g_2-|g_2|·⟨ g_1,g_2⟩/|g_1|· |g_2|·g_1/|g_1|
    
        =   g_2-⟨ g_1,g_2⟩/g_1^2g_1.




From the above derivations, the nature of GH is summarized as follows. 1) The harmonic gradient g̃_1 is essentially the projection of the raw gradient g_1 on the vertical line of g_2. 2) The harmonic gradient g̃_2 is essentially the projection of the raw gradient g_2 on the vertical line of g_1.


Therefore, with the above nature, the following three observations can be obtained. 1) The proposed gradient harmonization method not only ensures that the harmonic gradient (g̃_1/g̃_2) is close to the original gradient (g_1/g_2), but also ensures that the aggregated gradients before and after applying GH are close to each other. 2) If the angle between the original gradients g_1 and g_2 is θ, then the angle between harmonized gradient g̃_1 and g̃_2 becomes π-θ after applying GH. That is, GH really realizes the transformation from obtuse angle to acute angle. 3) The proposed GH aims to move the harmonic gradients towards the direction favorable for optimization, rather than those arbitrary gradient directions with acute angles.







 §.§ Improved Version: GH++

In this section, we propose an improved version called GH++, which aims to adjust the gradient angle between the two tasks from an obtuse angle to a vertical angle, i.e., making them orthogonal. This is to eliminate the conflict but simultaneously relieve the gradient deviation. Fig. <ref> visually illustrates the concepts of the proposed GH and GH++, with a primary focus on scenarios where gradient conflict exists, i.e., when the angle between g_1 and g_2 is obtuse.


Gradient deviation. For clarity, we designate the original gradients as g_1=OA and g_2=OB. Let θ represent the angle between g_1 and g_2. As illustrated in Fig. <ref> (a), we denote the gradients after applying GH as g̃_1=OC and g̃_2=OD, where OC⊥ OB and OD⊥ OA. Apparently, the sum of the angles deviated from the original directions is 2(θ - π/2), i.e., ∠ AOC+∠ DOB. In other words, although GH can promote the positive correlation between the two gradients, its optimization gradient is seriously deviated from the original gradient. Therefore, we propose an improved version, GH++, to eliminate the conflict and minimize the sum of the gradient deviations. As shown in Fig. <ref> (b), GH++ adjusts the gradient angle between the two tasks from an obtuse angle to a vertical angle, i.e., ∠ EOF. The sum of the gradient deviation angles is (θ - π/2) i.e., ∠ AOE+∠ FOB, which is half of the sum of the gradient deviations of GH.





Specifically, as shown in Fig. <ref> (b), let denote the harmonized gradients of GH++ as g̃_1=OE and g̃_2=OF.  In order to resolve the conflict and relieve the deviation from the original gradient directions, we designate OE⊥ OF,  i.e., ∠ EOF = π/2, where points E and F  move along arcs AC and BD, respectively. Intuitively, △ EOF forms a rotatable right triangle. We define the direction of rotation from g_1 to g̃_1 as positive. β represents the rotating angles from g_1 to g̃_1, which is a positive angle and β̅ represents the rotating angles from g_2 to g̃_2, which is a negative angle. According to Fig. <ref> (b), the harmonization gradients of GH++, i.e., g̃_1 and g̃_2, can be represented as



    g̃_1 = OE = OA+AE= g_1+ AE,
    g̃_2 = OF = OB+BF= g_2+ BF.

Since the △ AOE and △ BOF are isosceles triangles, we can obtain AE and BF according to the trigonometric formula, whose expression can be written as

    AE = 2 ·OA·sin∠ AOE/2= 2 · g_1 ·sinβ/2, 
    BF = 2 ·OB·sin∠ BOF/2= 2 · g_2 ·sinβ̅/2.

Substituting Eq. (<ref>) into Eq. (<ref>), we can derive the harmonized gradients of GH++.

    g̃_1 = (1+2 ·sinβ/2)· g_1,
    g̃_2 = (1+2 ·sinβ̅/2)· g_2.


Referring to Figure <ref> (b), we have a variable angle β = ∠ AOE ∈ [0, θ - π/2], where θ = ∠ AOB = arccosg_1^Tg_2/|g_1|·|g_2|∈ (π/2, π]. To simplify, we can express β as follows:

    β = λ∠ AOC = λ (θ-π/2) = λ(arccosg_1^Tg_2/|g_1|·|g_2|-π/2),

where λ∈ [0, 1] serves as a trade-off parameter, controlling the magnitude of deviation from g̃_1 to g_1. When λ = 0, g̃_1 = g_1= OA remains unchanged, and g_2 = OD. When λ = 1, g_1= OC, and g̃_1 = g_2 = OB remains constant. Since OE⊥ OF,  i.e., ∠ EOF = π/2, β̅ can be represented as

    β̅    =∠ BOF = - ∠ FOB = -(θ-π/2 - β) = β-(θ-π/2).

Substituting Eq. (<ref>) into Eq. (<ref>), we have

    β̅    =  λ (θ-π/2) + (θ - π/2)= (λ-1)(θ - π/2)
       =(λ-1)(arccosg_1^Tg_2/|g_1|·|g_2|-π/2).

Finally, substituting Eq. (<ref>) and Eq. (<ref>) into Eq. (<ref>), we can obtain the harmonization gradients of GH++ as follows:

    g̃_1 = (1+2 ·sinλ(arccosg_1^Tg_2/|g_1|·|g_2|-π/2)/2)· g_1,
       g̃_2 = (1+2 ·sin(λ-1)(arccosg_1^Tg_2/|g_1|·|g_2|-π/2)/2)· g_2.


Similar to GH, when there is no conflict, i.e., g_1^Tg_2≥ 0, no action. The proposed GH++ is indicated as Theorem 2.





For any two different tasks, optimization conflicts can be eliminated by GH++. Define the overall loss function ℒ (Θ)=ℒ_1 (Θ)+ℒ_2 (Θ), composed of two sub-objectives (refer to domain alignment and classification in this paper). Define g_1 and g_2 as the gradient of ℒ_1 (Θ) and ℒ_2 (Θ), respectively, then the general expression of the overall harmonized gradient g̃ of the whole loss ℒ (Θ) with GH++ can be formulated as:

    g̃   =g̃_1 + g̃_2 
       =(1+2 δ(g_1^Tg_2<0) sinλ(arccosg_1^Tg_2/|g_1|·|g_2|-π/2)/2) g_1
       +(1+2 δ(g_1^Tg_2<0) sin(λ-1)(arccosg_1^Tg_2/|g_1|·|g_2|-π/2)/2) g_2.




It is worth noting that GH++ presents a more favorable optimization scenario for the original task compared to GH. This advantage arises from the fact that the sum of the gradient deviation from the original direction of GH++ is only half of GH.



































 §.§ Equivalent Model of UDA with GH/GH++



By combining the general UDA model (Eq. (<ref>), described in Section <ref>) with gradient harmonization strategies (Eq. (<ref>)/Eq. (<ref>), a well-balanced UDA model can be trained and implemented. However, the gradient aggregation operator in Eq. (<ref>)/Eq. (<ref>) is intricate and has an impact on optimization efficiency. Therefore, we introduce a computation-efficient alternative model that is functionally equivalent to UDA with GH/GH++. For convenience, we can express the proposed gradient harmonization approaches, i.e., Eq. (<ref>)/Eq. (<ref>), as follows:

    g̃=τ_1g_1+τ_2g_2,

where τ_1 and τ_2 are constants that can be calculated by using the original gradients g_1 and g_2. Notably, the gradient g_1 of the original loss ℒ_1(Θ) and the gradient g_2 of the original loss ℒ_2(Θ) can be easily computed, only if the UDA model is fixed.  Then, if GH is chosen to resolve the conflict, τ_1 and τ_2 can be calculated as follows.

    τ_1=1-δ(g_1^Tg_2<0)g_2^Tg_1/‖ g_1 ‖^2,  
       τ_2=1-δ(g_1^Tg_2<0)g_1^Tg_2/‖ g_2 ‖^2.

If GH++ is used to eliminate the conflict, τ_1 and τ_2 can be calculated as follows.

    τ_1=(1+2 δ(g_1^Tg_2<0) sinλ(arccosg_1^Tg_2/|g_1|·|g_2|-π/2)/2),
       τ_1=(1+2 δ(g_1^Tg_2<0) sin(λ-1)(arccosg_1^Tg_2/|g_1|·|g_2|-π/2)/2).

Ultimately, we can derive the equivalent UDA model embedded GH/GH++ by conducting the integral operation on the gradient in Eq. (<ref>). The overall loss of UDA with GH/GH++ model can be represented as



    ℒ̃=∫(τ_1g_1+τ_2g_2)d Θ=τ_1ℒ_1(Θ)+τ_2ℒ_2(Θ),

where ℒ_1(Θ) and ℒ_2(Θ) can be the domain alignment loss and classification loss described in Eq. (<ref>) and Eq. (<ref>), respectively, for unsupervised domain adaptation.
Finally, the objective function of the UDA with GH/GH++ is:

    θ_g,θ_cmin  θ_dmax  ℒ̃=τ_1 ℒ_dom(θ_g,θ_d)+τ_2 ℒ_cls(θ_g,θ_c).



It can be seen that the proposed approaches essentially reweight the original loss function ℒ_1(Θ) and ℒ_2(Θ). The computation of the weights τ_1 and τ_2 needs to use the gradient of the original loss function, as presented in  Eq. (<ref>)/Eq. (<ref>). Finally, a balanced and efficient UDA model is formulated. Without loss of generality, when g_1^Tg_2>0, we can observe τ_1 =1 and τ_2 =1, and the overall loss function is degenerated into ℒ̃=ℒ_1(Θ)+ℒ_2(Θ), i.e., the general UDA model. We describe the training procedure in Algorithm <ref>.

Notably, the proposed approaches are  plug-and-play modules and can be deployed in almost all models with different losses ℒ_1(Θ) and ℒ_2(Θ). In order to clearly observe the computational cost of the equivalent model, we discuss the training speed and recognition accuracy before and after applying GH in Table <ref>.





§ EXPERIMENTS




 §.§ Datasets






Office-31 <cit.> is a mainstream benchmark dataset for visual domain adaptation, which consists of three distinct domains: Amazon (A), DSLR (D), Webcam (W). It totally contains 4,652 images from 31 categories. We evaluate our method in all 6 different transfer tasks across domains.


Office-Home <cit.> is a more challenging and harder benchmark than Office-31. It contains 15.5K images across 65 object categories from 4 different domains: Artistic images (Ar), Clip Art (Cl), Product images (Pr), and Real-World images (Rw). We evaluate our method in all 12 different transfer tasks across domains.

Digits Datasets. We mainly study three datasets: MNIST (M) <cit.>, USPS (U) <cit.> and SVHN (S) <cit.>. MNIST and USPS are two general handwriting recognition datasets involving 10 categories. SVHN is obtained from house numbers in Google Street View images. We conduct experiments in 3 universal tasks, including M→U, U→M and S→M.

VisDA-2017 <cit.> is a simulation-to-real dataset for domain adaptation with two distinct domains: synthetic object images rendered from 3D models and real object images. It contains over 280K images across 12 categories.



DomainNet <cit.> is the largest domain adaptation dataset consisting of about 600K images from six distinct domains, including Clipart (clp), Infograph (inf), Painting (pnt), Quickdraw (qdr), Real (rel), Sketch (skt). There are 48K-172K images categorized into 345 classes per domain. We evaluate our method in all 30 cross-domain tasks.








 §.§ Implementation Details

We compare our method with the following state-of-the-art unsupervised domain adaptation methods: ResNet <cit.>, DAN (Deep Adaptation Networks) <cit.>, DANN (Domain-adversarial Neural Networks) <cit.>, JAN (Joint Adaptation Networks) <cit.>, DRCN (Deep Reconstruction-Classification Networks) <cit.>, CoGAN (Coupled Generative Adversarial Networks) <cit.>, ADDA (Adversarial Discriminative Domain Adaptation) <cit.>, CyCADA ( Cycle-consistent Adversarial Domain Adaptation) <cit.>, CAT (Cluster Alignment with a Teacher) <cit.>, TPN (Transferrable prototypical networks) <cit.>, LWC (Light-weight Calibrator) <cit.>, ETD (Enhanced Transport Distance) <cit.>, TAT (Transferable Adversarial Training) <cit.>, TADA (Transferable Attention for Domain Adaptation) <cit.>, SymNets (Symmetric Networks) <cit.>, BNM (Batch Nuclear-norm Maximization) <cit.>, ALDA (Adversarial-learned Loss for Domain Adaptation) <cit.>, MIMTFL (Mutual Information Maximisation
and Transferable Feature Learning) <cit.>, Meta (Metaalign) <cit.>, MDD (Margin Disparity Discrepancy) <cit.>, CDAN <cit.>, MCD (Maximum Classification Discrepancy) <cit.>, GVB (Gradually Vanishing Bridge) <cit.> and DWL (Dynamic Weighted Learning) <cit.>, SAFN (Stepwise Adaptive Feature Norm) <cit.>, SWD (Sliced Wasserstein Discrepancy) <cit.>, DMRL (Dual Mixup Regularized Learning) <cit.>, CAN (Contrastive Adaptation Network) <cit.>, FixBi <cit.>, FGDA (Feature Gradient Distribution Alignment) <cit.> and ParetoDA (Pareto Domain Adaptation) <cit.>, CGDM (Cross-domain Gradient Discrepancy Minimization) <cit.>, SSRT (Safe Self-Refinement for Transformer-based domain adaptation) <cit.> and TVT (Transferable Vision Transformer) <cit.>.  SSRT and TVT are transformer-based approaches, while other approaches are CNN-based. We use the results in their original papers for fair comparison. When there is no relevant experimental result in the original papers, we implement them according to their released source codes.




 
Furthermore, we investigate the effect of GH/GH++ on five popular UDA approaches, including four CNN-based approaches (i.e., CDAN <cit.>, MCD <cit.>, GVB <cit.> and DWL <cit.>) and one transformer-based approach (i.e., SSRT <cit.>). For CNN-based approaches, we use pre-trained ResNet-50 <cit.> on ImageNet-1K <cit.> as backbone network for Office-31 and Office-Home, and adopt the pre-trained ResNet-101 <cit.> as backbone network for VisDA-2017 and DomainNet. We employ LeNet <cit.> as backbone for Digits datasets. For transformer-based approach, SSRT uses the ViT-base with 16 × 16 patch size pre-trained on ImageNet-21K and ImageNet-1K (denoted by SSRT^∗) as the transformer backbones, respectively. In terms of optimization, we follow the original protocol of the baselines. Note that, in this paper, CDAN means CDAN+E in <cit.>) and GVB refers to GVB-GD in <cit.>. For GH++, we empirically set λ to 0.5.
MCD achieves domain alignment by the game between two classifiers and the generator. 
Therefore, the gradient of the discrepancy loss between the two classifiers is just the gradient of domain alignment loss. We implement our method in PyTorch by following UDA protocol <cit.> that source domain is labeled and target domain is unlabeled. 

For fair comparisons, we independently run all experiments five times and report the average target accuracy. 



 §.§ Results on UDA


























































Tables <ref>, <ref>, <ref>, <ref> and <ref> present evaluation results on Office-31, Office-Home, VisDA-2017, Digits and DomainNet, respectively. Generally, the transformer-based results, such as TVT and SSRT, are much better than CNN-based results, which has been validated in previous work. For TVT and SSRT, the ViT backbone pre-trained on ImageNet-1K is slightly inferior than ImageNet-21K (i.e., TVT^∗&TVT, SSRT^∗&SSRT). Considering that the CNNs are pre-trained on ImageNet-1K, we take SSRT^∗ for fair analysis as default in the following.  

In order to demonstrate the effectiveness and universality of the proposed approaches in UDA, we investigate the effect of GH/GH++ on five popular UDA approaches, including CNN-based approaches (i.e., CDAN, MCD, GVB and DWL) and transformer-based approach (i.e., SSRT). We can observe that the proposed approaches can further improve the baseline, whether based on transformer or CNN. This is attributed to the proposed approaches in alleviating gradient conflict between domain alignment task and classification task in optimization. Meanwhile, it verifies that the proposed approaches can be easily plugged and played in the existing UDA methods. In addition, GH++ usually performs better than GH. The reason is that during removing the gradient conflict between the two tasks, GH++ minimizes the gradient deflection, such that the two tasks can evolve harmoniously during joint training while preserving their individual task-specific optimality.

Specifically, as shown in Table <ref>, when GH/GH++ is applied, CDAN, MCD, GVB, DWL and SSRT outperform the original models by 1.5%/2.2%, 1.0%/1.4%,  0.2%/0.7%, 0.7%/0.8% and 0.7%/0.9% on Avg^ of Office-31, respectively.
As shown in Table <ref>, CDAN, MCD, GVB, DWL and SSRT after applying GH/GH++ outperform the original models by 2.6%/3.2%, 4.5%/5.1%, 0.8%/1.2%, 0.4%/0.6% and 0.6%/0.8% on Avg^ of Office-Home, respectively. As shown in Table <ref>, we can observe that CDAN, MCD, GVB, DWL and SSRT after applying GH/GH++ outperform the original models by 0.9%/0.9%, 2.2%/2.4%,  1.1%/1.3%, 0.6%/0.9% and 0.3%/0.3% on Avg of Digits, respectively.
As shown in Table <ref>, CDAN, MCD, GVB, DWL and SSRT after applying GH/GH++ outperform the original models by 1.4%/3.9%, 4.1%/4.5%, 0.5%/0.8%, 0.7%/0.3% and 0.7%/0.6% on VisDA-2017, respectively.
From the results in Table <ref>, we can observe that CDAN, MCD, GVB, DWL and SSRT after applying GH/GH++ outperform the original models by 3.2%/3.7%, 7.1%/7.3%, 0.7%/1.0%, 1.0%/1.3% and 1.1%/1.4% on Avg of DomainNet, respectively. The proposed approaches generally improve CDAN and MCD more than other baselines because the original models of CDAN and MCD have more obvious optimization conflicts than other baselines. Fig. <ref> also confirms this perspective. Therefore, the proposed approaches have better performance when the inherent gradient conflict is serious. Besides, it is worth noting that the proposed approaches usually achieve greater performance gains on large-scale datasets, i.e., VisDA-2017 and DomainNet, which further indicate the effectiveness and versatility of our method.

























































§ MODEL ANALYSIS AND DISCUSSION



 §.§ Feature Visualization
  Fig. <ref> describes the t-SNE <cit.> visualizations of features learned by MCD (baseline) and MCD+GH on the tasks of U → M and M → U. Fig. <ref> (a) and (c) are visualization features generated by MCD. Fig. <ref> (b) and (d) are visualization features generated by MCD+GH. It can be observed that both features learned by MCD and MCD+GH achieve well-performed global alignment effect with 10 clusters under two tasks. Further, the visualization feature distributions with GH deployed have better clustering effect and have fewer samples distributed across class boundaries, which intuitively boosts the feature discriminability. In addition, visualization results further validate the balance learning ability of our GH approach.



 §.§ Convergence Analysis
  We present the convergence curves of test error with respect to the number of iterations on tasks of U → M and Synthetic → Real as shown in Fig. <ref>. For each subfigure, the blue line represents the test error of different baselines, and the red line represents the test error for baseline+GH (e.g., CDAN+GH). Obviously, compared with baselines, the introduction of GH can further improve the test performance and convergence. This fully indicates that GH plays an active coordination role in practical optimization process, which promotes the cooperation of domain alignment task and classification task towards a benign direction as expected. 

















 §.§ Confusion Matrix Visualization


Fig. <ref> displays the visualizations of confusion matrix for the classifier trained by DWL and DWL+GH. DWL obtains several uncertain predictions with small values while DWL+GH obtains more confident predictions. Comparing with Fig. <ref> (a) and (b), the confusing “Class 1, 2, 7, 8, and 9” are correctly recognized in DWL+GH. From Fig. <ref> (c) and (d), the confusing “Class 8” is corrected in DWL+GH. It implies that optimization conflict during training deteriorates the discriminability in the target domain. Our method preserves the feature discriminability of target samples in the course of harmonic training. DWL+GH improves the between-task balance by coordinating the explicit external task conflict and the implicit internal optimization conflict.





 §.§ Balance Analysis of GH-based UDA
  Fig. <ref> shows MMD distance <cit.> and the max J(W) <cit.> values based on the feature representation learned by MCD+GH. The left vertical axis corresponding to the red curve represents the MMD distance used to measure the alignment degree across domains. The smaller value means the better domain alignment. The right vertical axis corresponding to the blue curve represents the degree of discriminability based on Linear Discriminant Analysis (LDA) <cit.>. The larger max J(W) implies better discriminability, which directly affects the classification result of samples. Fig. <ref>(a) and (b) reflect the degree of domain alignment and discriminability on task of M→U and U→M, respectively. From Fig. <ref>, MCD+GH has a smaller MMD distance and a larger max J(W) value during training. This shows that GH facilitates the coordinated optimality of alignment and classification tasks while maintaining their task-specific optimality. Enhanced and balanced domain-invariant and class-discriminative feature representations can be obtained.



 §.§ Training Speed and Accuracy
 In order to observe the efficiency of the proposed equivalent model more clearly, we present the training speed and classification accuracy before and after applying GH. As shown in Table <ref>, for tasks M→U and U→M on digits datasets, the training speed of MCD+GH is 0.14254 s/epoch and 0.09809 s/epoch longer than MCD, respectively. In other words, MCD+GH takes less training time than MCD in training process, but can get 2.5%/2.7% classification accuracy gain. The computational cost of employing GH is quite low, and thus GH is a powerful and efficient auxiliary tool to facilitate those popular domain adaptation baselines towards more outstanding classification performance.







 §.§ Gradient Inner Product Visualization
  Fig. <ref> presents inner product distributions of two gradients between domain alignment and classification tasks before and after applying Gradient Harmonization for MCD. From Fig. <ref> (a), we observe the acute and obtuse angles between gradients of the two tasks before coordination. The obtuse angles account for about 40% of the total number, which exhibits the identical property as Fig. <ref>. In other words, there exists between-task conflict in the model optimization process but paid less attention. After Gradient Harmonization, as shown in Fig. <ref> (b), the inner products of the two gradients are all positive. That is, the gradient angles between the two tasks are coordinated into acute angles. The proposed GH avoids the optimization conflict by separately adjusting the gradients of the two tasks to achieve the purpose of optimal coordination. Experimental results fully illustrate the effectiveness of the proposed GH.







 §.§ Rationality and Comparison to Other Alternatives



The proposed GH/GH++ aims at altering the gradient angle between two different tasks from an obtuse angle to an acute/vertical angle. 
Whether the same effect can be achieved with other intuitive gradient correction alternatives or not? For instance, one might consider convert an obtuse angle into an acute angle by simply applying a negative sign to one of the two gradients, i.e., transforming either g_1 into -g_1 or g_2 into -g_2. As depicted in Fig. <ref>, it is apparent that the results reveal a significant decline in classification accuracy when using the -g_1&g_2 and g_1&-g_2 methods in contrast to the proposed approaches. While applying a negative sign can indeed transit from an obtuse angle to an acute angle and force the two gradients to be positively correlated, it cannot replicate the excellent classification performance achieved by the proposed GH/GH++.

The rationale behind this observation is that directly applying a negative sign to one of the gradients alters the gradient towards the opposite direction of the original optimal gradient descent direction. This significantly undermines the primary objective of the original task. In other words, the performance deteriorates sharply when the gradient harmonization direction significantly deviates from the original direction. GH/GH++ reasonably adjusts the two gradients to achieve the purpose of optimization coordination on the premise of keeping the harmonic gradient as close as possible to the original gradient descent direction. Therefore, the proposed approaches do not break the task-specific optimality, but pursues a better cooperation, which verifies the rationality of the proposed GH/GH++.








 §.§ Parameter Sensitivity Analysis

To investigate the effect of the parameter λ in GH++, we conduct experiments on three tasks (i.e., M→U, U→M and S→M) based on MCD and GVB by varying λ∈{0, 0.1, 0.3, 0.5,0.7, 0.9, 1}. The results are presented in Fig. <ref>. We can observe GH++ is little sensitive to the scale variety of λ, which indicates that GH++ is robust across different baselines and tasks. Besides, we observe that when λ=0.5, models generally achieve the best performance. In other words, when the gradient deviation of the two tasks is relieved, the performance can be largely improved.
























 §.§ Scalability to Other Multi-task Problems

To further demonstrate the universality and scalability of the proposed approaches, we evaluate GH/GH++ in the object detection and multi-modal interactive retrieval fields, which also involves optimization of multiple objectives.



Dataset. We select widely used benchmarks, i.e., PASCAL VOC 2007 <cit.> and CSS <cit.> for object detection and multi-modal interactive retrieval, respectively.
PASCAL VOC 2007 consists of about 5K trainval images and 5K test images over 20 object categories.
CSS consists of 38K synthesized images with different colors, shapes and sizes. It contains about 19K training image-text pairs and 19K testing image-text pairs, respectively. 



Implementation Details. For object detection, we use the DSSD <cit.> model with 321× 321 inputs as the baseline. We follow the same experimental settings and protocol as <cit.> and adopt Average Precision (averaged AP at IoUs from 0.5 to 0.9) to measure performance. For multi-modal interactive retrieval, we use TIRG <cit.> and the adversarial training version(AT-TIRG) follows the paper <cit.> as the baseline. For evaluation, we follow the same protocols as <cit.> and use retrieval accuracy R@N as our evaluation metric, which computes the percentage of test queries where at least one target or correctly matched image is within the top N retrieved images. Note that we exactly follow the original experimental setups of the baselines. In other words, DSSD+GH/GH++ and AT-TIRG+GH/GH++ indicate that GH/GH++ are inserted directly into the DSSD and AT-TIRG, respectively, without any change in either the backbone or hyper-parameters.





Results on Object Detection. Object detection models <cit.> usually consider two tasks: classification and regression. In this section, we use the proposed GH/GH++ to mitigate the conflict between these two tasks. Table <ref> presents the Average Precision (AP) on the PASCAL VOC 2007 test set. We can observe that GH/GH++ can improve the AP with different bounding boxes, which verifies the effectiveness of the proposed approaches in object detection.








Results on Multi-modal Interactive Retrieval. Recently, <cit.> introduces adversarial training <cit.> into multi-modal interactive retrieval model (abbreviated as AT-TIRG) to improve model robustness, which aims to train a generalized and robust model suitable for both clean samples and adversarial attack samples. During adversarial training, there is conflict in training the clean samples and adversarial attack samples, which results in the limitation of the final performance. In this section, we try to adopt the proposed approaches to solve this problem. The experimental results are provided in Table <ref>. Comparing AT-TIRG+GH with AT-TIRG, we can see the proposed GH can enhance the AT. For the adversarial attack samples, AT-TIRG+GH outperforms the AT-TIRG by 3.4%, 2.9%, and 2.2% on R@1, R@5, and R@10, respectively. AT-TIRG+GH++ outperforms the AT-TIRG by 5.2%, 4.9%, and 3.9% on R@1, R@5, and R@10, respectively. The performance of the clean samples is also improved in most cases. Note that the performance of clean samples is usually close to 100%, so there is little space for improvement.





































§ CONCLUSION

In this paper, we pay attention to the optimization conflict (i.e., imbalance or incoordination) problem between different tasks (i.e., the alignment task and the classification task) in alignment-based unsupervised domain adaptation models. To mitigate this problem, we propose two simple yet efficient Gradient Harmonization approaches, including GH and GH++, which take measures to de-conflict between the gradients of both tasks in optimization. Besides, to facilitate the harmonization during adaptation, we derive the equivalent but more efficient model of UDA with GH/GH++, which becomes a dynamically reweighted loss function of most existing unsupervised domain adaptation models. Further, the essence and insights of the proposed approaches are provided to indicate its rationality. Exhaustive experiments and model analyses demonstrate that the proposed approaches significantly improve the existing UDA models and contribute to achieving state-of-the-art results.
In addition, we have verified that the proposed approaches can be adapted to other problems and areas, such as object detection and multi-modal retrieval, to de-conflict between the gradients of any two tasks in optimization and improve model performance.




























 

  


































IEEEtrans






























































 Paper Title: Point-supervised Brain Tumor Segmentation with Box-prompted MedSAM.






Point-supervised Brain Tumor Segmentation with Box-prompted MedSAM
    Xiaofeng Liu, Jonghye Woo, Chao Ma, Jinsong Ouyang, and Georges El Fakhri
X. Liu, C. Ma, J. Ouyang and G. El Fakhri are with the Department of Radiology and Biomedical Imaging, Yale University, New Haven, CT 06519.
J. Woo is with the Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114.

    August 5, 2024
===========================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Delineating lesions and anatomical structure is important for image-guided interventions. Point-supervised medical image segmentation (PSS) has great potential to alleviate costly expert delineation labeling. However, due to the lack of precise size and boundary guidance, the effectiveness of PSS often falls short of expectations. Although recent vision foundational models, such as the medical segment anything model (MedSAM), have made significant advancements in bounding-box-prompted segmentation, it is not straightforward to utilize point annotation, and is prone to semantic ambiguity. In this preliminary study, we introduce an iterative framework to facilitate semantic-aware point-supervised MedSAM. Specifically, the semantic box-prompt generator (SBPG) module has the capacity to convert the point input into potential pseudo bounding box suggestions, which are explicitly refined by the prototype-based semantic similarity. This is then succeeded by a prompt-guided spatial refinement (PGSR) module that harnesses the exceptional generalizability of MedSAM to infer the segmentation mask, which also updates the box proposal seed in SBPG. Performance can be progressively improved with adequate iterations. We conducted an evaluation on BraTS2018 for the segmentation of whole brain tumors and demonstrated its superior performance compared to traditional PSS methods and on par with box-supervised methods.


 




§ INTRODUCTION
 



The costly labeling effort in medical image delineation significantly hinder the development of data-driven AI models. There are increasing interests on weakly supervised segmentation to utilize bounding box or even a single point as label for training supervision <cit.>. However, due to the absence of accurate size and boundary guidance, there is a large performance gap between point-supervised medical image segmentation (PSS) and mask/box-supervised counterparts <cit.>. 

The recent progress of vision foundational models has achieved breakthroughs in several weakly supervised tasks, benefited by their strong zero-shot generalizability. For instance, the point-prompt natural image segment anything model (SAM) has been applied to enhance pseudo labels for point-supervised segmentation <cit.>. Recently, the medical image version of SAM (MedSAM) <cit.> has been trained with 1.5 million segmented medical images to achieve generalizable box-prompt segmentation. MedSAM takes both an image slice and a bounding box, i.e., prompt, to predict the possible segmentation mask within the box. Notably, the natural image SAM itself follows point-prompt design. Therefore, it is not straight forward to integrate box-prompt MedSAM to PSS task as <cit.>. In addition, a significant limitation of SAM/MedSAM is the lack of classification ability, resulting in class / structure-agnostic segmentation results <cit.>. They are designed for general use, while fails to accurately delineate the specific lesion or structure as desired <cit.>.  



 

To our knowledge, this is the first attempt to integrate MedSAM to facilitate semantic-aware PSS. We adopt an iterative framework to achieve coarse-to-fine progression of the bounding box. The point prompt is first converted to a box-proposal seed. Then, we configure a semantic box-prompt generator (SBPG) to propose and pick the reasonable box according to semantic similarity as <cit.>. It is followed by a box prompt guided spatial refinement (PGSR) to utilize the generalizable MedSAM to predict the segmentation mask. In addition, the smallest box that covers the mask is further used as the box seed in the next round of SBPG. We do not need to fine-tune the MedSAM to be semantic aware, which is prone to catastrophic forgetting. Notably, only the prompt refiner θ with the encoder part of the segmentor model will be trained, which involves about half the parameters of mask-supervised UNet training. 

We demonstrated its effectiveness in BraTS2018 for PSS of the whole brain tumor segmentation from T2-weighted MRI slices. We show that in testing, with 3 to 5 rounds of iteration, the point-prompt can achieve superior performance to approximate the box-supervised MedSAM <cit.>.








§ METHODOLOGY

In PSS, we are given tuplet {x^i,ρ_c^i,s^i}, in which x^i can be an MR slice, while the point prompt ρ^i_c={ρ^i_x,ρ^i_y,c} indicates the 2D spatial coordinates and the interested class c, e.g., tumor or normal tissue. We would expect the segmentation mask prediction ŝ^i_c to approximate the expert annotation s^i_c. 

To catering the box-prompt MedSAM <cit.>, an initial box proposal seed of class c, i.e., b_c:0^i, is generated from ρ^i_c, which is centered on point location (ρ^i_x,ρ^i_y) with the size of X× Y. We do not expect b_c:0^i fit the region of interests very well as there is no prior information about the size in the initial step. Then, the proposal bag ℬ^i_c={b_c:n^i}_n=1^N is created by scaling b_c:0^i with N different scales. Therefore, the boxes in group ℬ^i_c has strong spatial correlation with ρ_c^i, which avoid the inefficient random proposal in whole image <cit.>. 

To enable the model be aware of the specific semantic class, for example tumor in our task, a parameterized prompt refiner θ with a segmentor encoder and two fully connected layers is applied. Specifically, for the training sample with segmentation mask label s^i_c, we convert s^i_c to the corresponding smallest rectangle bounding box label b^i_c. Then, we store a set of feature f^i_c:n=θ(x^i,b_c:n^i) in the most recent M batches to a memory buffer, and calculate the mean of {f^i_c:nm}_n,m=1^N,M as the prototype 𝒱_c. As conventional multiple instance learning (MIL) based methods <cit.>,
the instance level probability indicating the likelihood of b^i_c:n matches 𝒱_c can be approximated with 
    p(b^i_c:n,𝒱_c)=e^cos(f^i_c:n,𝒱_c)/∑_c=1^C e^cos(f^i_c:n,𝒱_c),
where cos(·) indicates the cosine similarity, and C is the number of segmentation category. With the ground truth b^i_c, θ is trained with the binary cross entropy loss of 

    ℒ=-∑_c=1^C {c^i_1log∑_n^N p(b^i_c:n,𝒱_c) + (1-c^i_1)log(1-∑_n^N p(b^i_c:n,𝒱_c))},
where c^i_1 is the one-hot class label. Therefore, the prompt refiner is optimized to be semantic aware for the specific anatomical structure. The prototype in buffer is also updated with the new model θ. With p(b^i_c:n,𝒱_c) for each b^i_c:n, we pick the highest probability one  in ℬ^i_c as our optimal proposal b^i_c:* in current stage, which is also our bounding box inference.


Then, in the PGSR module, the off-the-shelf MedSAM <cit.> takes both x^i and b^i_c:* to predict the possible segmentation mask ŝ^i_c. Notably, no information about the interested class can be directly informed in MedSAM. Although the MedSAM has strong ability of zero-shot segmentation to delineate the relatively accurate mask within the current box prompt b^i_c:*, the alignment of ŝ_c^i is highly dependent on the precise b^i_c:*. This motivated us to refine b^i_c:* with a more suitable proposal seed b_c:0^i in SBPG. 

In practice, we can simply apply the smallest bounding box b̂^i_c:0 for ŝ^i_c:0, and use b̂^i_c:0 as our box proposal seed in the next round of SBPG. With T rounds of iteration, we expect that the box-prompt can be progressively refined to inform the MedSAM to predict an accurate segmentation mask. 

In particular, in the last round of testing, we do not need to generate b̂^i_c:0. Instead, the T-th round ŝ^i_c:0 is used as our final prediction.









§ EXPERIMENTS AND RESULTS

We evaluated on BraTS dataset for whole tumor segmentation as <cit.>, and use T2-weighted slices as our input. We used the 80%/20% split for training or testing. We generate the smallest bounding box for the mask and use its center as ρ^i_c. Our prompt refiner adopted the encoder part of the segmentor in <cit.>, which is followed by two fully connected layers of 1024 and 256 dimensions, respectively. We empirically set X× Y=21×21 and search proper T∈{1,2,3,5,10}. We followed <cit.> to compare with the conventional point-supervised method of WISE-Net without the use of the vision fundation model. Also, we directly input the ground truth of the bounding box into MedSAM, which can be “upper bound" of the performance of pointwise method. In Tab.<ref>, we can see that our model with T≥5 outperforms WISE-Net by a large margin. The Dice score is close to the box-supervised MedSAM.   





§ CONCLUSION

In this work, we proposed an efficient iterative framework to enable box-prompt MedSAM to point-supervised medical image segmentation. A lightweight prompt refiner with only encoder is specifically trained for the interested structural class. We show that the fixed off-the-shelf large model can flexibly target the specific downstream task without the need of large scale fine-tuning. The medical PSS performance can be largely improved by the advance of vision foundation model, e.g., MedSAM, which show great promise for facilitating the image-guided interventions. 
 


 


§ ACKNOWLEDGMENT

This work is supported by R21EB034911, R01CA165221, R01DC018511, P41EB022544, and NAIRR240016. 

ieee





 Paper Title: TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models.






 Tel-Aviv University
   
 



 NVIDIA, Tel-Aviv University
   
 



 Tel-Aviv University
   
 



 Tel-Aviv University
   
 



 Tel-Aviv University
   
  

§ ABSTRACT 
    
Diffusion models have opened the path to a wide range of text-based image editing frameworks. However, these typically build on the multi-step nature of the diffusion backwards process, and adapting them to distilled, fast-sampling methods has proven surprisingly challenging. Here, we focus on a popular line of text-based editing frameworks - the “edit-friendly” DDPM-noise inversion approach. We analyze its application to fast sampling methods and categorize its failures into two classes: the appearance of visual artifacts, and insufficient editing strength. We trace the artifacts to mismatched noise statistics between inverted noises and the expected noise schedule, and suggest a shifted noise schedule which corrects for this offset. To increase editing strength, we propose a pseudo-guidance approach that efficiently increases the magnitude of edits without introducing new artifacts. All in all, our method enables text-based image editing with as few as three diffusion steps, while providing novel insights into the mechanisms behind popular text-based editing approaches. 

Project page: <https://turboedit-paper.github.io/>





    
    < g r a p h i c s >


    Our method enables text-based editing of real images with as few as 3 diffusion steps (0.321 seconds on an A5000 GPU).








TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models
    Daniel Cohen-Or
    August 5, 2024
===================================================================



§ INTRODUCTION


The unprecedented expressive power of large-scale text-to-image diffusion models <cit.> has contributed to a rise of text-based editing frameworks. Through these frameworks, users are empowered to modify existing, real images using natural language instructions. However, these methods commonly rely on the multi-step nature of the diffusion process. In such many-step scenarios, any deviation from train-time statistics can be lessened by injecting smaller changes across many steps, or by allowing the final steps of the diffusion process to draw the result back to the prior distribution, thereby correcting for any artifacts that may arise along the way.

Recently, model distillation methods <cit.> have enabled the creation of “fast” diffusion models, from which novel images can be sampled in few (1-8) steps. Ideally, we would like to use such models as a backbone for image editing, thereby accelerating the editing process. However, absent sufficient steps, existing text-based editing approaches tend to create noticeable artifacts, or show poor performance.

Here, we propose to address these limitations by analyzing the behaviour of one such family of editing methods — the DDPM noise-inversion framework <cit.>.
There, inversion takes the form of a series of per-timestep noise maps. These are pre-calculated, such that using them in place of the random noise samples in the DDPM backwards process, will lead to a re-construction of the original image under a given prompt (see <ref> for more details). Importantly, editing through this approach is simple, requiring only a change of the text describing the image, while still using the same pre-calculated noise samples during the DDPM backwards process. However, attempting to apply this approach to image editing with fast-sampling methods (e.g., SDXL Turbo <cit.>) leads to the creation of severe visual artifacts, and significantly diminished adherence to the novel prompts (insufficient editing strength). 

To tackle the visual artifacts, we analyze the statistics of the inverted noise maps, and observe that they behave more closely to the noise expected from earlier (more noisy) diffusion steps. We hypothesize that in the case of few-step models, there is no time to correct for the noise-distribution shift induced by these corrections, and hence artifacts arise. To overcome them, we propose a shifted denoising schedule, where the denoising sampler is instructed to remove noise as if it was observing an earlier, more noisy step. Additionally, in typical noise-inversion schemes, there is little impact to the noise used at the last denoising step, and indeed it is often simply discarded. In the case of few-step models, the last step's impact is non-trivial, and we find it helpful to inject noise also in this step, while explicitly normalizing it towards the correct statistics.

To overcome the insufficient editing strength, we first rephrase the noise-inversion approach, demonstrate that it bears similarities to Delta-Denoising methods <cit.>, and that under certain conditions the DDPM-inversion and Delta-Denoising approaches are exactly equivalent. This re-phrasing not only provides additional insight as to why DDPM noise-inversion approaches are successful, but it reveals that the inversion process itself can be skipped and replaced with an evaluation of a single correction term at each denoising step. Hence, it can be combined into the same batch as the backwards denoising steps themselves, further decreasing editing times.
Finally, we identify specific terms in the edit-friendly denoising process that are responsible for the impact of the prompt, and strengthen them in a similar approach to classifier-free guidance (CFG, <cit.>). We show that under commonly occurring assumptions, this pseudo-guidance approach is equivalent to re-introducing CFG into the fast-sampling method, but using fewer network evaluation steps.

By combining these components, we enable real-image editing with as few as 3 diffusion steps, achieving ×5-×500 speedup compared with existing editing methods, while preserving and even improving their output quality. Our code will be made public.





§ RELATED WORK




  
Fast Diffusion Sampling.
Early diffusion models <cit.> were notoriously slow to sample from, requiring hundreds of expensive neural evaluations (and several minutes) to produce a single image. To overcome this hurdle, a range of advanced sampling methods were proposed <cit.>. These are typically grounded in ordinary differential equation solvers, and can successfully reduce the sampling process to several dozen steps without modifying the underlying denoising model.

To further reduce sampling times, a recent line of work proposes to employ a distillation process <cit.>, where a pre-trained diffusion model is fine-tuned using objectives that promote sampling in few (1-8) steps. These can range from the use of adversarial <cit.> losses <cit.> to distribution matching <cit.> and consistency objectives <cit.>. 

The emergence of these fast-sampling methods offers an opportunity to speed up existing workflows. However, existing control and editing methods often struggle in this few-step regime. Hence, several approaches were proposed to enable additional controls <cit.> or improve personalization <cit.> using few-step models. Our work seeks to bring another crucial component into this fast-sampling realm - the text-based image editing workflow.



  
Text-based image editing.
The unprecedented semantic control offered by large scale text-to-image diffusion models <cit.> has inspired a large volume of work that leverages them as a prior for image editing.
Early works proposed simple approaches, such as adding noise to an image and removing it conditioned on a novel prompt <cit.>. However, such methods often lead to significant changes in the image shape and layout. Hence, more advanced methods proposed to use in-painting and other localization approaches <cit.> to modify a single image region.

Others works align attention maps <cit.> or other internal feature representations <cit.> to better preserve the original image structure. The outputs of such methods can also be distilled into another diffusion model, trained to modify a conditioning image based on an instruction prompt <cit.>.
In another approach, the diffusion model itself is fine-tuned on the source image <cit.> to better align it to the original content. However, such approaches are often costly in both time and compute.
Finally, a recent self-attention sharing approach <cit.> has proven effective in maintaining image content across prompts <cit.>, and in transferring styles <cit.> or appearances <cit.> across images.

Common to all these approaches is that they operate in a many-step regime, they typically require dozens of diffusion steps, and struggle when applied to fast sampling models <cit.>. We aim to enable text-based editing in the fast sampling regime.



  
Diffusion inversion.
Applying editing techniques to a real image commonly requires one to first find some latent representation of the image, which can be fed into the model in order to reconstruct the image. This latent can then be perturbed, directly or through modifications of the generative pass, to affect a change in the image.

Initial inversion efforts focused on GANs <cit.>, opting for either direct optimization <cit.> or encoder-based approaches <cit.>. With the rise of diffusion-based editing, several works sought to invert real images into the diffusion space, often by determining some initial noise that will be cleaned into a specific image. Initial inversion approaches overwrote the low-frequency content of a generated image with the low-pass filtered content from a source image <cit.>, enabling scribble-based modifications or texture changes. Others relied on inverting the deterministic DDIM <cit.> process <cit.>, but these commonly require many steps to be accurate, and struggle to modify the image through inference-time prompt changes <cit.>. To overcome this issue, several works intervene in the CFG process and replace the null text condition with a learned embedding that represents the original image <cit.>. However, these commonly require lengthy optimization. As an alternative to optimization, DDIM-based inversion can also be improved by leveraging a fixed-point iterative scheme <cit.>, 
but such solutions require dozens of inversion steps even when attached to a fast-sampling backwards process <cit.>.

Rather than focusing on predicting an initial noise that will re-create the image with deterministic sampling, an alternative approach is to consider the DDPM generative process, and invert the image into the intermediate noise maps that are added to a generated image <cit.>. This approach serves as our backbone for few-step editing, and we expand on it in greater detail in <ref>.

Finally, some methods invert sets of images into the text conditioning space of the model  <cit.>. However, these are typically used for personalization, where the goal is not to preserve the structure of an image, but to learn a global representation that allows to re-create a concept in novel scenes.


§ PRELIMINARIES


We begin with a high level overview of the DDPM noise-inversion approach of <cit.>. There, the goal is to find a meaningful latent representation that can be used to reconstruct an image, such that this latent can be manipulated in more intuitive ways than with existing approaches like DDIM inversion. To this end, the authors propose to use the DDPM-noise space (the noise maps added to the image at each step of the DDPM denoising process), and term it an “edit-friendly” noise space.

More concretely, recall the DDPM denoising equation:

    x_t-1 = μ_t(x_t, c) + σ_t z_t,   t=T,…,1,

where z_t∼𝒩(0,I), c is a conditioning prompt, μ_t(x_t, c) is derived from the denoising network's output through:

    μ_t(x_t, c) = 1/√(α_t)(x_t - 1-α_t/√(1-α̅_t)ϵ_θ(x_t, t, c)),

where ϵ_θ(x_t, t, c) is the noise prediction and α_t,σ_t are derived from the noising schedule.

In the noise-inversion process, given an image x_0, one first calculates its noisy representations through the standard forward diffusion process equation, using a different independently-sampled noise for each time-step:

    x_t = √(α̅_t) x_0 + √(1-α̅_t) ϵ̃_t,   1,…,T,

where ϵ̃_t∼𝒩(0,I) are statistically independent. This independence is a core difference between the “edit-friendly” approach and prior noise inversion approaches (CycleDiffusion,  <cit.>), and the authors show that it is crucial for achieving pleasing results.

Given two such noisy images, x_t and x_t-1, one then calculates the noise that would be needed for <ref> to clean x_t into x_t-1:

    σ_t z_t = x_t-1 - μ_t(x_t, c)


Finally, to perform text-based editing, one can simply denoise an image from the pre-calculated x_T under a novel prompt ĉ, while applying the inverted noises at each step in place of the typical DDPM noise samples, i.e.:

    x̂_t-1 = μ_t(x̂_t, ĉ) + σ_t z_t = μ_t(x̂_t, ĉ) + x_t-1 - μ_t(x_t, c) .


Throughout the rest of the paper, we make use of this “edit-friendly” DDPM noise-inversion technique. We demonstrate how it can be adapted to work with a few-step SDXL-Turbo model, and provide additional insights on the reason behind its success.



§ METHOD


In the following section, we outline a series of modifications that can enable the DDPM noise-inversion editing flow to operate in the few-step diffusion domain, and specifically with SDXL-Turbo. We analyze the two primary issues with the baseline results - the appearance of visual artifacts, and insufficient editing strength, and suggest a fix for each. Throughout the process, we shed some additional light on the DDPM-inversion process itself. 




 §.§ Treating the visual artifacts

As noted above, directly applying the DDPM noise-inversion approach to SDXL-Turbo leads to considerable artifacts in the generated images (see <ref>). 
In their original work, <cit.> observe that their inverted noises follow different dynamics than the standard Gaussian noise used in DDPM generation. Drawing on their observations, we hypothesize that the emergence of visual artifacts can be traced back to such deviations. To verify this hypothesis, we analyze the statistics of the noises derived from a noise-inversion process, and compare them with the scale of noises injected during a standard DDPM generation flow. Specifically, we apply standard edit-friendly inversion using an SDXL model with a few-step schedule, and compare the standard deviation of 
x_t-1 - μ_t(x_t) the DDPM-inversion corrections,
with N(0, σ_t ^2), the standard DDPM noise. Results are in <ref>. 

An immediate observation is that the per-pixel standard deviations of the noise-inversion corrections are higher than those of the DDPM-noise schedule, throughout the entire process. Importantly, this contrasts with the dynamics observed in many-step scenarios <cit.>, where the statistics eventually converge towards the end of the generation process. A second observation, is that the noise statistics seem to deviate by an approximately constant margin. In other words, we notice that throughout much of the process, the inverted noises behave like a standard noise originating from a time-step shifted by roughly 200 steps to the past (see also <ref> in the supplementary). All-in-all, this disagreement between the noise statistics and the timestep used to condition the model and the denoising step, leads to a train-test mismatch and results in the creation of artifacts.




Following these observations, we propose to alleviate the artifacts by re-aligning the denoising timestep schedule to the scale of the noises. Specifically, during the inversion process, for each timestep t we first add noise to x_0 according to the standard DDPM noising schedule, and create x_t. However, we inject a larger timestep t+Δ into the denoising model and the scheduler's denoising step:

    σ_t · z_t = x_t-1 - μ_t +Δ(x_t, c),

where the subscript t +Δ denotes the timestep input of both the denoising network and the scheduler. By applying this correction, we can once again compare the statistics of the inverted noises with standard noise schedule (<ref>, blue) and see that they are in much better agreement across all timesteps.

Note that for the generative pass, we must also employ a similar shift to keep the two network prediction components synchronized: 

    x̂_t-1 = μ_t +Δ(x̂_t, ĉ) + (x_t-1 - μ_t +Δ(x_t, c)) .

This synchronization is crucial for our analysis in <ref>. 







As an additional correction, we find it beneficial to clip the norm of the last noise-inversion correction (<ref>, purple). The motivation here arises from the contrast with multi-step approaches, where the last correction is small and can be skipped. Here, the last step is large, and its associated correction still captures many of the details of the original image. Hence, we do not want to discard it. Instead, we simply clip it to avoid the introduction of novel artifacts.
This is reminiscent of other editing methods, e.g. <cit.> where the control is decreased towards the end of the generation process.




 §.§ Improving prompt alignment

Having treated the visual artifacts, we must now deal with the second limitation: insufficient editing strength. The issue here is that changing the prompt between the noise-inversion step and the generation pass, often leads to little or no change in the final image. 

To investigate this limitation, we once again consider the edit-friendly inference formula (<ref>), which we can re-write as:

    x̂_t-1 = x_t-1 + (μ_t(x̂_t, ĉ) - μ_t(x_t, c)) .

Under this framing, we can already see a hint to an underlying reason behind the noise-inversion approach's success. Specifically, the second term captures the difference between the model's prediction on the edited image, using the novel prompt, and the same prediction on the original image using the original prompt that describes it. This is analogous to the correction term found in Delta Denoising Score (DDS, <cit.>) where the authors show that the performance of score distillation sampling methods (SDS, <cit.>) on image editing can be significantly improved if one computes a loss using a similar difference between the network's prediction on an edited image with the new prompt, and the prediction on the original image with the original prompt that describes it. The intuition provided for such a term in DDS (and later expanded on in NFSD <cit.>), is that this difference helps cancel out components of the denoising network's prediction that are unrelated to the prompt (e.g., the direction towards a cleaner image, or any general network bias). The existence of a similar term in the edit-friendly process implies that the same mechanism contributes to its improved performance in image editing tasks. Indeed, we will later show that the connection between EF and DDS is much deeper, with the two methods being functionally equivalent.

We can further expand this analysis of the edit-friendly process by adding and subtracting a cross-term:

    x̂_t-1 = x_t-1 + (μ_t(x̂_t, ĉ) - μ_t(x̂_t, c)^cross-prompt  + μ_t(x̂_t, c) - μ_t(x_t, c)^cross-trajectory) .

Now, we can identify two different directions of change to the original image. The first term is the difference between predictions for the generation trajectory under the novel and original prompts. Hence, one can regard this term as one that represents the direction that takes an image on the novel trajectory from the old prompt to the new one. The second term is the difference between predictions for the new trajectory and the old one, under the same prompt. Hence, it can be regarded as the direction that takes an image from the old trajectory, to the new one. Intuitively, applying both of them to x_t-1, itself a point on the original trajectory, will first shift it to the new trajectory and then pull it further in the direction of the difference between prompts.

Recall that our goal was to strengthen the effect of the prompt. Hence, we can simply draw inspiration from CFG and extrapolate along this cross-prompt direction. Importantly, we do not want to increase the weight of the cross-trajectory term, because we do not want to overshoot the new trajectory. Indeed, in <ref> we investigate the behaviour of these two terms and demonstrate that scaling the cross-trajectory term leads to the creation of novel visual artifacts and increased saturation, while scaling the cross-prompt term leads to stronger editing effects. As such, we propose to convert the edit-friendly inference equation as follows:

    x̂_t-1 = x_t-1 + μ_t(x̂_t, c) - μ_t(x_t, c) + w ·(μ_t(x̂_t, ĉ) - μ_t(x̂_t, c)) ,

where w is the pseudo-guidance scale.








In the supplementary, we further analyze the connection between this guidance formula and a re-introduction of CFG into SDXL-Turbo. Specifically, we show that our approach is equivalent to applying CFG during both inversion and inference, so long as:

    μ_t(x̂_t, c) - μ_t(x_t, c)≈μ_t(x̂_t, ϕ) - μ_t(x_t, ϕ),

where ϕ indicates the null prompt. Experimentally, we find that this condition commonly holds for SDXL-Turbo predictions. Specifically, we find an average cosine similarity of 0.93 between the two sides of <ref>, compared to an averaged cosine similarity of -0.04 between these two sides and the cross-prompt term of <ref>.

Hence our approach roughly aligns with a re-introduction of CFG into SDXL-Turbo, but requires fewer neural function evaluations (3 instead of 4), or equivalently a smaller batch size. In both cases, this allows us to gain further speed improvements over existing approaches. Moreover, this result sheds further light on the observations of <cit.>, which report better performance when applying CFG during both the inversion and the generation process with multi-step models. If CFG is only applied to one pass and not the other, then CFG also increases the weight of the cross-trajectory term which may harm the results.






Finally, while the derivation of  <ref> was rooted in the “edit-friendly” DDPM-inversion process, we note that its evaluation does not require any pre-computation of a multi-step noise map. Indeed, since neither x_t-1 nor μ_t(x_t, c) depend on a multi-step denoising pass (they depend only on a closed-form projection of x_0 via <ref>), they can simply be calculated at inference-time, with μ_t(x_t, c) being predicted as part of the same batch. This approach is more closely related to CycleDiffusion's <cit.> denoising pass, and can effectively cut the number of editing steps by half.









 §.§ Connecting EF and DDS

In <ref> we observed edit-friendly inversion and DDS share a similar structure. In both cases, the image is being edited by shifting it along the vector between the diffusion predictions on the new trajectory with the new prompt, and the diffusion predictions on the original image with a prompt that describes it. In <ref> we demonstrate that this connection runs deeper, and that under an appropriate choice of timesteps and learning rates during the DDS optimization process, both methods become functionally equivalent, with the DDS corrections terms collapsing to <ref>. Please see the supplementary for the proof, an empirical demonstration, and a deeper discussion of this equivalency.









 §.§ Implementation details

We implemented our method on top of SDXL-Turbo <cit.>. Unless otherwise noted, all results use a pseudo-guidance scale w=1.5 and 4 denoising steps (starting at t=599 with Δ=200). We clip the norm of the final step corrections to a maximum of 15.5. All experiments are performed on a single NVIDIA A5000 GPU.




§ RESULTS


We structure our experimental verification in two parts. In the first part, we demonstrate our methods ability to edit real images in few steps. There, we conduct a series of evaluations, where we contrast our method against a range of baselines. These include both existing multi-step baselines, as well as several few-step alternatives.
We demonstrate that our approach can match, or even exceed the quality of current multi-step editing methods, while being significantly faster.
Then, we conduct an ablation study where we demonstrate the effect of individual components in our proposed solution. 




 §.§ Evaluation


We being by evaluating our method, starting with a qualitative evaluation. In <ref> we show a range of editing results achieved with our method using 4 diffusion steps. These include object-level modifications, style changes, or object replacement. Additional results are shown in <ref>.

Next, we compare our method against a series of baselines, including both many-step approaches as well as few-step sampling alternatives. In <ref> we compare our method to multi-step methods. We consider a large range of approaches, including both optimization based methods (Null-text, <cit.>), feature-preserving ones (PnP, <cit.>), the baseline edit-friendly editing approach and the recently introduced ReNoise <cit.> which performs multi-step inversion (∼ 40 steps) but re-generates the image via a fast sampling method (∼ 4 steps). Our method achieves comparable or better results than most multi-step baselines, while being significantly faster.

In <ref> we compare our method to SDEdit <cit.> and vanilla applications of edit-friendly inversion with SDXL-Turbo using 4 steps. Our method can better maintain the details of the original image, while also achieving better prompt alignment. Notably, the edit-friendly approach creates significant artifacts. Please zoom in to better see the results.








For a quantitative evaluation, we employ 4 commonly used metrics. The first two are the CLIP-space <cit.> similarity between the input image and the editing result (CLIP-I), and the CLIP-space similarity between the target text and the editing result (CLIP-T). We further evaluate the CLIP-directional similarity <cit.>, the cosine-similarity between the CLIP-space direction that connects the pre- and post-editing image, and the direction that points from a description of the original image to a description of the target edit (CLIP-Dir). Intuitively, this metric measures how well the image-space change aligns with the prompt difference, without changing content that is unrelated to the prompt. Finally, we also compare the LPIPS <cit.> score between the pre- and post-edited images. 


The results are shown in <ref>. Notably, our method achieves favorable results on both prompt-alignment metrics, and indeed outperforms all other methods which do not also suffer from significant deviation from the original image. On the image preservation front, our approach does not outperform the state-of-the-art, but it achieves comparable results to many multi-step methods, despite being quicker by a factor of ×5-×500.

To further validate our results, we also conduct a user preference study. We used a two-alternative forced choice setup, where each user was shown the source image, the target prompt, and two editing results including our method and one baseline. Users were asked to select the image that better aligns with the prompt, while preserving the content of the original image. We focused on the 5 most performant baselines, including both multi- and few-step approaches. In total, we collected 202 responses from 25 users. Results are shown in <ref>. Overall, our approach is strongly preferred over competing few-step methods, while being competitive with multi-step results. Plug & Play achieve higher user preference, but they are slower by a factor of ×500. All in all, these results demonstrate that our approach can successfully edit real images in as few as 3 diffusion steps, without sacrificing quality.









 §.§ Ablation study





Next, we turn to an ablation study where we analyze the effect of the different components inherent in our approach. Specifically, we consider: (1) the effect of disabling the time-shift applied to the network predictions, (2) the effect of removing the noise-clipping at the final denoising step, (3) the effects of other noise normalization schemes, such as shifting only the last step or directly re-normalizing the noise statistics (4) the effect of disabling the pseudo-guidance, and (5) the performance of the naive edit-friendly DDPM inversion in this few-step setup. Visual results are provided in <ref> while quantitative results are found in <ref>.

Removing the time-shift maintains the text-alignment, but harms the image-to-image similarity because of the introduction of visual artifacts. A similar effect can be observed to a lesser extent when removing the noise clipping or when shifting only the last step. Explicit re-normalization of noise maps leads to overly-smoothed results and loss of detail. Without pseudo-guidance, we observe significantly diminished prompt-alignment. Finally, with all methods disabled, the results show significantly diminished scores on all fronts, and have considerable visual artifacts.























        

            
























§ LIMITATIONS





Our method shares many of the limitations of prior text-based editing work. In particular, it can struggle with changing geometries through the prompt (asking for a person to cross his arms). Similarly, it may struggle to introduce new objects into the scene (adding a hat to a horse). Another limitation can be found in prompt leakage to undesired areas of the scene (turning a car red may lead to a shift in background color). In addition, it can struggle with stylization when converting real imagery to painting or vice versa. We show examples of limitations and failure cases in <ref>.



§ CONCLUSIONS

We presented TurboEdit, a fast text-based editing method that leverages newly introduced fast-sampling methods to significantly reduce the time it takes to manipulate an image. Notably, our method achieves sub-second editing times, enabling an interactive editing experience.

Our approach is derived from a deeper analysis of the dynamics of existing noise-inversion approaches, leading to a series of technically simple fixes. Hence, we hope that our work can provide not only a useful application for creative workflows, but also shed a light on components and ideas that should be kept in mind when adapting other tools to the few-step regime.

In the future, we hope to investigate further improvements for editing flows in the few-step regime, such as a more principled alignment of noise schedules to the expected statistics, or exploring ways to improve geometric modifications and object insertion.






 §.§ Acknowledgements

This work was partially supported by ISF (grants 2492/20 and 3441/21). 




ACM-Reference-Format









 Supplementary Materials





§ CONNECTION BETWEEN PSEUDO-GUIDANCE AND CLASSIFIER-FREE GUIDANCE


Let us now provide proof, that under the assumption that 

    μ_t(x̂_t, c) - μ_t(x_t, c)≈μ_t(x̂_t, ϕ) - μ_t(x_t, ϕ)

Our proposed guidance is equivalent to using CFG but with fewer network evaluations. We consider the case where CFG is applied both during inversion and during generation. Let x, ϕ + λ_s · (c - ϕ) be the notation for a U-net prediction followed by a scheduler step with CFG strength λ_s. Then, the following holds:

    x, ϕ + λ_s · (c - ϕ) = x, ϕ + λ_s · [x, c - x, ϕ] ,

, one can first perform a scheduler step after summing the U-net predictions, or first perform two scheduler steps and then sum the results.
We can now write the inference equation as:

    t-1 = x_t-1 + [t, ϕ + λ_s( - ϕ) - x_t, ϕ + λ_s(c - ϕ)] .

Adding and subtracting the cross term μ_t(x̂_t, ϕ + λ_s(c - ϕ)), we get:

    t-1    = x_t-1       + [ t, ϕ+ λ_s(- ϕ) 

                      - μ_t(x̂_t, ϕ+ λ_s(c - ϕ)) 

                      + μ_t(x̂_t, ϕ+ λ_s(c - ϕ)) 

                      - x_t, ϕ+ λ_s(c - ϕ)] 

               = x_t-1       + [t, ϕ + λ_s[t,  - t, ϕ] 

                      - t, ϕ - λ_s[t, c - t, ϕ] 
 
                      + t, ϕ + λ_s[t, c - t, ϕ] 
 
                      - x_t, ϕ - λ_s[x_t, c - x_t, ϕ]] 

               = x_t-1       + t, ϕ - x_t, ϕ + λ_s [
 
                       t,  - t, ϕ - t, c + t, ϕ 
 
                      + t, c - t, ϕ - x_t, c + x_t, ϕ ] 

               = x_t-1       + t, ϕ - x_t, ϕ + λ_s [ 
 
                      t,  - t, c 

                      + (t, c - x_t, c) - (t, ϕ - x_t, ϕ)] 

If μ_t(x̂_t, c) - μ_t(x_t, c)≈μ_t(x̂_t, ϕ) - μ_t(x_t, ϕ) 
then the term in line (15) is approximately zero and line (14) can be replaced: 
t, ϕ - x_t, ϕ→t, c - x_t, c 
and we get that DDPM inversion with CFG is equivalent to our proposed guidance. Empirically, we find that relation of <ref> often holds for 3 and 4 steps with SDXL-Turbo, but we offer no guarantees for the general case.



§ FURTHER ANALYSIS ON TIME SHIFTS


Here we provide further analysis on the behaviour of the inversion noise scales, and demonstrate the importance of applying the shifts throughout the entire generation process. In <ref>(blue) we plot the distribution of offsets between the DDPM-inversion corrections and the denoising step for which the true noise schedule is closest to this step. These shifts are approximately distributed around the 200-step shift, which inspires our choice to correct for them through a 200 step offset. We further plot the same distribution of offsets after applying our fix. Now, most predicted noises match the step in which they are used, and the overall shift is diminished.

In <ref> we ask whether it is important to shift the schedule throughout the entire generative process, or if it is sufficient to shift only the last step. The intuition behind this experiment is that applying only a final-step shift can lead to increased smoothing of the image (due to the removal of extra noise), and possibly account for the artifact removal on its own. However, we find experimentally that this is not the case. As can be seen, the unshifted steps accumulate strong artifacts, and the final step fails to overcome them. Moreover, we observe that this approach leads to worse preservation of background objects. All in all, we find that shifting the entire schedule is beneficial.









§ EDIT-FRIENDLY AND DELTA-DENOISING EQUIVALENCE


Here, we provide proof that Delta Denoising Score (DDS) and Edit-Friendly DDPM Inversion (EF) are equivalent, under an appropriate choice of learning rate and timestep sampling for the DDS method.

First, recall the DDPM sampler's definition of μ_t(x_t, c):


    μ_t(x_t, c) = 1/√(α_t)(𝐱_t-1-α_t/√(1-α̅_t)ϵ_t^ϕ(𝐱_t, c)) ,

and the noising equation:

    x_t = √(α̅_t) x_0 + √(1-α̅_t) ϵ̃_t ,

where α_t is derived from the diffusion noising schedule and α̅_t = ∏_s=1^t α_s and ϵ_t^ϕ denotes the diffusion model's output.

For simplicity, we denote a_t = 1/√(α_t), b_t = 1-α_t/√(1-α̅_t), c_t = √(α̅_t) and d_t = √(1-α̅_t), and re-write these equations as:

    μ_t(x_t, c) = a_t(x_t-b_tϵ_t^ϕ(x_t, c)) ,


    x_t = c_t x_0 + d_t ϵ̃_t


Next, recall that DDS operates through an iterative optimization scheme. Let x be the original image, x̂^i the optimized image at iteration i, and let x_t and x̂^i_t be their respective representations after noising to time t via <ref>. The DDS gradient is then:

    ∇_x̂ℒ_DDS = ϵ_t^ϕ(x̂^i_t, ĉ) - ϵ_t^ϕ(x_t, c) ,

where c and ĉ are the prompts describing the original image and the edit target respectively and t is sampled randomly for each optimization iteration.
Since this gradient is applied directly to x̂^i, we thus have:

    x̂^i + 1 = x̂^i - γ (ϵ_t^ϕ(x̂^i_t, ĉ) - ϵ_t^ϕ(x_t, c))


Next, we consider DDS under the following two conditions: (1) Rather than sampling diffusion timesteps randomly, we do so sequentially following a standard denoising schedule (we start with t=T and move towards t=0). We will thus use x̂^t_0 to denote the un-noised, DDS-optimized image at the iteration corresponding to timestep t, and x̂^t_t to denote its noised version at this timestep.
(2) We select a time-dependent learning rate given by:


    γ_t = 1 - α_t/√(α̅_t)√(1-α̅_t) = b_t/c_t


Plugging these conditions into <ref> we have:


    x̂^t-1_0    = x̂^t_0 - b_t/c_t (ϵ_t^ϕ(x̂^t_t, ĉ) - ϵ_t^ϕ(x_t, c)) 
       = = x̂^t_0 - b_t/c_t (ϵ_t^ϕ(x̂^t_t, ĉ) - ϵ_t^ϕ(x_t, c)) + 1/c_t x_t - 1/c_t x_t ,

and by repeatedly applying <ref> we have:

    x̂^t-1_0    =  x̂^t_0 - b_t/c_t (ϵ_t^ϕ(x̂^t_t, ĉ) - ϵ_t^ϕ(x_t, c)) + 1/c_t x_t - 1/c_t x_t 
       = x̂^t_0 + b_t/c_tϵ_t^ϕ(x_t, c) - 1/c_t x_t - b_t/c_tϵ_t^ϕ(x̂^t_t, ĉ) + x_0 + d_t/c_tϵ̃_t 
       = 1/c_t (c_t x̂^t_0 + d_t ϵ̃_t ) - 1/c_t (x_t - b_t ϵ_t^ϕ(x_t, c)) - b_t/c_tϵ_t^ϕ(x̂_t, ĉ) + x_0 
       = 1/c_tx̂_t - 1/c_t a_tμ(x_t, c) - b_t/c_tϵ_t^ϕ(x̂_t, ĉ) + x_0 
       = x_0 + 1/c_t (x̂_t  - b_t ϵ_t^ϕ(x̂_t, ĉ)) - 1/c_t a_tμ(x_t, c) 
       = x_0 + 1/c_t a_t (μ(x̂_t, ĉ) - μ(x_t, c)) .



From the definitions of c_t, a_t and α̅_t we further have:

    1/c_t a_t = (√(∏_s=1^t α_s)1/√(α_t))^-1 = (√(∏_s=1^t-1α_s))^-1 = 1/c_t-1


Adding noise (<ref>) to <ref> and plugging in this constant relation gives:

    x̂_t-1   = c_t-1 (x_0 + 1/c_t-1 (μ(x̂_t, ĉ)) -  μ(x_t, c)) + d_t-1ϵ̃_t-1
       = (c_t-1 x_0 + d_t-1ϵ̃_t-1) + (μ(x̂_t, ĉ) -  μ(x_t, c))  
       = x_t-1 + (μ(x̂_t, ĉ) -  μ(x_t, c))

which is exactly equal to the edit friendly step (<ref>).

This surprising result confirms that DDS and Edit-friendly share not only a similar correction term, but indeed under the specific choice of consecutive timesteps samples and a specific learning rate, the two methods converge to the exact same update rule. We further verify this empirically by editing a set of images using both methods, where DDS uses the modified schedule and learning rate. The results are shown in <ref>. As can be observed, the results are visually identical, further confirming our theoretical observation.

Finally, we note that a similar analysis can be applied to methods that combine DDS and EF (e.g., Posterior Distillation Sampling <cit.>). Indeed, through an appropriate choice of learning rates and sampling steps, such methods can also be made to follow <ref> for the specific case of image editing.








