Paper Title: Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolution.





 



Hierarchically Conditioned Diffusion Model







M. Khurana et al.





 Virginia Tech, Blacksburg VA, USA 

            {mridul, karpatne}@vt.edu Oak Ridge National Laboratory, Oak Ridge TN, USA University of California, Irvine CA, USA Tulane University, New Orleans LA, USA Battelle, Columbus OH, USA Duke University, Durham NC, USA University of North Carolina at Chapel Hill, Chapel Hill NC, USA The Ohio State University, Columbus OH, USA Rensselaer Polytechnic Institute, Troy NY, USA 


Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolution
    Mridul Khurana10009-0003-9346-3206 Arka Daw20009-0006-3319-1271 M. Maruf 1 0000-0003-0637-1753 Josef C. Uyeda 1 0000-0003-4624-9680 Wasila Dahdul 30000-0003-3162-7490 Caleb Charpentier 1 0000-0002-9787-7081 Yasin Bakış 40000-0001-6144-9440 Henry L. Bart Jr. 40000-0002-5662-9444 Paula M. Mabee 50000-0002-8455-3213 Hilmar Lapp 60000-0001-9107-0714 James P. Balhoff 70000-0002-8688-6599 Wei-Lun Chao 80000-0003-1269-7231 Charles Stewart 90000-0001-6532-6675 Tanya Berger-Wolf80000-0001-7610-1412 Anuj Karpatne 10000-0003-1647-3534

    August 5, 2024
======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



fancy
 
[L]Published as a conference paper at ECCV 2024
[C]




§ ABSTRACT 
A central problem in biology is to understand how organisms evolve and adapt to their environment by acquiring variations in the observable characteristics or traits of species across the tree of life. 
With the growing availability of large-scale image repositories in biology and recent advances in generative modeling, there is an opportunity to accelerate the discovery of evolutionary traits automatically from images. 
Toward this goal, we introduce Phylo-Diffusion, a novel framework for conditioning diffusion models with phylogenetic knowledge represented in the form of HIERarchical Embeddings (HIER-Embeds). 
We also propose two new experiments for perturbing the embedding space of Phylo-Diffusion:  trait masking and trait swapping, inspired by counterpart experiments of gene knockout and gene editing/swapping.
Our work represents a novel methodological advance in generative modeling to structure the embedding space of diffusion models using tree-based knowledge. Our work also opens a new chapter of research in evolutionary biology by using generative models to visualize evolutionary changes directly from images.
We empirically demonstrate the usefulness of Phylo-Diffusion in capturing meaningful trait variations for fishes and birds, revealing novel insights about the biological mechanisms of their evolution. 
[Model and code can be found at https://imageomics.github.io/phylo-diffusion/]
  
  





§ INTRODUCTION









Given the astonishing diversity of life forms on the planet, an important end goal in biology is to understand how organisms evolve and adapt to their environment by acquiring variations in their observable characteristics or traits (, beak color, stripe pattern, and fin curvature) over millions of years in the process of evolution. Our knowledge of species evolution is commonly represented in a graphical form as the “tree of life” (also referred to as the phylogenetic tree <cit.>, see Figure <ref>), illustrating the evolutionary history of species (leaf nodes) and their common ancestors (internal nodes). Discovering traits that are heritable across the tree of life, termed evolutionary traits, is important for a variety of biological tasks such as tracing the evolutionary timing of trait variations common to a group of species and analyzing their genetic underpinnings through gene-knockout or gene-editing/swapping (, CRISPR <cit.>) experiments. However, quantifying trait variations across large groups of species is labor-intensive and time-consuming, as it relies on expert visual attention and subjective definitions <cit.>, hindering rapid scientific advancement <cit.>. 

The growing deluge of large-scale image repositories in biology <cit.> presents a unique opportunity for machine learning (ML) methods to accelerate the discovery of evolutionary traits automatically from images. In particular,  with recent developments in generative modeling such as latent diffusion models (LDMs) <cit.>, we are witnessing rapid improvements in our ability to control the generation of high-quality images based on input conditioning of text or image prompts. This is facilitating breakthroughs in a variety of commercial use-cases of computer vision where we can analyze how changes in the input prompts affect variations in the generated images <cit.>.
We ask the question: can we leverage LDMs to control the generation of biological images of organisms conditioned on the position of a species in the tree of life? In other words, can we encode the structure of evolutionary relationships among species and their ancestors as input conditions in LDMs?
This can help us analyze trait variations in generated images across different branches in the phylogenetic tree, revealing novel insights into the biological mechanisms of species evolution. 


Toward this goal, we introduce Phylo-Diffusion, a novel framework for discovering evolutionary traits of species from images by conditioning diffusion models with phylogenetic knowledge (see Figure <ref>). One of the core innovations of Phylo-Diffusion is a novel HIERarchical Embedding (HIER-Embed) strategy that encodes evolutionary information of every species as a sequence of four vectors, one for each discretized level of ancestry in the tree of life (covering different evolutionary periods). 
We also propose two novel experiments for analyzing evolutionary traits by perturbing the embedding space of Phylo-Diffusion and observing changes in the features of generated images, akin to biological experiments involving genetic perturbations.
First, we introduce Trait Masking, where one or more levels of information in HIER-Embed are masked out with noise to study the disappearance of traits inherited by species at those levels. This is inspired by  gene knockout experiments <cit.>, wherein one or more genes are deactivated or “knocked out” to investigate the gene's function, particularly its impact on the traits of the organism. Second, we introduce Trait Swapping, where a certain level of HIER-Embed in a reference species is swapped with the embedding of a sibling node at the same level, similar in spirit to gene editing/swapping experiments made possible by the CRISPR technology <cit.>. The goal of trait swapping is to visualize trait differences at every branching point in the tree of life that results in the diversification of species during evolution.


Here are the main contributions of this paper. Our work represents a novel methodological advance in the emerging field of knowledge-guided machine learning (KGML) <cit.> to structure the embedding space of generative models using tree-based knowledge. Our work also opens a new chapter of research in evolutionary biology by using generative models to visualize evolutionary changes directly from images, which can serve a variety of biological use-cases. For example, Phylo-Diffusion can help biologists automate the discovery of synapomorphies, which are distinctive traits that emerge on specific evolutionary branches and are crucial for systematics and classification <cit.>. Our proposed experiments of trait masking and swapping can also be viewed as novel image-based counterparts to genetic experiments, which traditionally take years. Our work thus enables biologists to rapidly analyze the impacts of genetic perturbations on particular branches of the phylogenetic tree–a grand challenge in developmental biology <cit.>.
We empirically demonstrate the usefulness of Phylo-Diffusion in capturing meaningful trait changes upon perturbing its embedding for fishes and birds, generating novel hypotheses of their evolution.


§ RELATED WORKS AND BACKGROUND


 

  
Interpretable ML:
Discovering evolutionary traits from images requires the identification and interpretation of fine-grained features in images that define and differentiate species. Several methodologies have recently been developed in the field of interpretable ML for localizing image regions that contain discriminatory information of classes, including ProtoPNet <cit.>, PIP-Net <cit.> and INTR <cit.>. 
Despite their effectiveness and applicability across a wide range of applications, these methods are not directly suited for our target application of discovering evolutionary traits for two primary reasons. First, they are not designed to incorporate structured biological knowledge (, knowledge of the tree of life) in the learning of interpretable features, and thus are unable to provide biologically meaningful explanations of feature differences across groups of species in the phylogenetic tree, which is key to discovering evolutionary traits. Second, since most methods in interpretable ML are designed for the task of classification, it is non-trivial to integrate them in generative modeling frameworks to produce synthetic images with controlled perturbations in the embedding space similar to gene knockout and gene editing/swapping experiments, in contrast to our proposed experiments in Phylo-Diffusion.



  
Phylogeny-guided Neural Networks (Phylo-NN):

A recent work closely aligned with our goal of discovering evolutionary traits directly from images is Phylo-NN <cit.>. Phylo-NN uses an encoder-decoder architecture to represent images of organisms as structured sequences of feature vectors termed “Imageomes”, where different segments of Imageomes capture evolutionary information from varying levels of ancestry in the phylogenetic tree. While Phylo-NN shares several similarities with our proposed framework, Phylo-Diffusion, in terms of motivations and problem formulations, there are also prominent differences. The primary goal of Phylo-NN is specimen-level image reconstruction, whereas Phylo-Diffusion considers a different goal of controlling image generation at the species-level. As a result, Phylo-NN learns a unique Imageome sequence for every organism, enabling us to study the variability in individuals from the same species and the analysis of similarity in Imageome segments learned at shared ancestry levels. On the other hand, Phylo-Diffusion learns a unique embedding for every species and ancestor node in the tree of life, which serves as input conditions to generate distributions of synthetic images. Phylo-Diffusion thus uses hard constraints to ensure that all species with a common ancestor learn the exact same embeddings at their shared ancestry levels, making it easy to analyze trait commonalities and variations across groups of species, in contrast to Phylo-NN. 
Additionally, Phylo-Diffusion allows for perturbations in the embedding space of generative models in biologically meaningful ways inspired by gene knockout and gene editing/swapping experiments, going beyond the capabilities of Phylo-NN. We consider Phylo-NN as a baseline in our experiments to compare its performance with Phylo-Diffusion.



  
Background on Latent Diffusion Models (LDMs):
One of the state-of-the-art approaches in generative modeling is the framework of Diffusion Models <cit.>, which learns a target distribution p(x) by incrementally transforming a noisy sample x generated from a Gaussian distribution 𝒩(0, I) into one that is more likely to be generated from p(x) over a series of timesteps T.
While early frameworks of diffusion models (, DDPM <cit.>, DDIM <cit.> and ADM <cit.>) suffered from high computational costs and long training/inference times, Latent Diffusion Models (LDMs) <cit.> are able to address these concerns to a large extent by operating in a compressed latent space, significantly accelerating their ability to generate high-resolution images. The basic idea of LDMs is to train a separate auto-encoder to map an input image x into its latent representation z_0 = ℰ(x) using encoder ℰ, which when fed to decoder 𝒟 produces a reconstruction of the original image,  x̃ = 𝒟(z̃_0). LDMs employ diffusion models in the compressed latent space z by modeling the conditional probability of the reverse diffusion process as z̃_t-1∼ p_θ(z̃_t-1 | z̃_t, y, t), where y is the input condition. This is implemented using a conditional denoising U-Net backbone ϵ_θ(z_t, y, t) with learnable parameters θ.
LDMs also pre-process y using a domain-specific encoder 𝐄 = τ_ϕ(y) trained alongside the U-Net backbone ϵ_θ to project y into the intermediate layers of ϵ_θ using cross-attention mechanisms. The learnable parameters of LDMs are trained by minimizing the following loss function:

    ℒ(θ,ϕ) = 𝔼_z_t, y, t, ϵ∼𝒩(0, I)[ ϵ̂_θ(z_t, τ_ϕ(y), t) - ϵ^2 ]



§ PROPOSED FRAMEWORK OF PHYLO-DIFFUSION




 §.§ Hierarchical Embedding (HIER-Embed)
 


Phylo-Diffusion uses a novel hierarchical embedding (HIER-Embed) strategy to structure the embedding E of every species node using phylogenetic knowledge. 
As a first step, we consider a discretized version of the phylogenetic tree involving four ancestral levels, level-1 to level-4, where every level corresponds to a different range of time in the process of evolution. (See <Ref> for a detailed characterization of the four ancestry levels for fish species used in this study.) Given a set of n species,
𝒮 = {S_1, S_2, S_3, ..., S_n}, let us represent the position of species S_i ∈𝒮 in the phylogenetic tree at the four ancestry levels as  {S_i^1, S_i^2, S_i^3, S_i^4}, where S_i^l represents the ancestor node of S_i at level-l. Hence, if two species S_i and S_j share common ancestors till level-k, then S_i^l = S_j^l for l=1 to k.
We define the level-l embedding of species S_i as:

    𝐄_i^l = (S_i^l)   ∈ℝ^d',

where (.) is a learnable embedding layer that provides a simple way to store and look-up the trained embeddings of every node. The combined hierarchical embedding (HIER-Embed) of species S_i is obtained by concatenating its embeddings across all four levels as follows:


    𝐄_i = τ (S_i) =  𝐄_i^1,  𝐄_i^2,  𝐄_i^3,  𝐄_i^4  ∈ℝ^d,

where  denotes the concatenation operation and y=S_i is the input condition used in LDMs. Note that different segments of 𝐄_i capture information about the traits of S_i acquired at different time periods of evolution. In particular, we expect the embedding vectors learned at earlier ancestry levels of 𝐄_i to capture evolutionary traits of S_i common to a broader group of species. On the other hand, embeddings learned at later ancestry levels are expected to be more specific to S_i. 
In the following, we present two novel experiments for studying evolutionary traits by perturbing the embedding space learned by HIER-Embed.







 §.§ Proposed Experiment of Trait Masking




The goal of this experiment is to verify if HIER-Embed is indeed able to capture hierarchical information in its level-embeddings such that masking information at lower levels of the embedding only erases traits acquired at later stages of evolution while retaining trait variations learned at earlier levels. In other words, we want to verify that the embeddings learned by HIER-Embed at level-l capture information common to all descendant species that are part of the same sub-tree at level-l. 
<Ref> represents a schematic diagram of the process followed for trait masking. We start with the combined embedding containing information at all four levels, [, , , ]. To examine what is learned at the last level of this embedding, we mask it out by substituting it with Gaussian noise defined as 𝐳_𝐧𝐨𝐢𝐬𝐞∼𝒩(0, I) ∈ℝ^d'. This results in the perturbed embedding [, , , ], effectively eliminating the species-level (or ) information. This masking should prompt the model to generate images that reflect only the information learned up to the third level while obscuring species-level details. We can extend this experiment by incrementally introducing noise at later levels, , at both levels 3 & 4, and so on.



  
Expected Changes in Probability Distributions: Note that when all four level embeddings are used, [, , , ], the generated images are expected to be classified to a unique species S_i. In terms of probability distributions, the probability of predicting species S_i should be distinctly higher than the probability of predicting any other species. However, when we mask out certain level embeddings (, mask out information at level 4), we are intentionally removing information necessary to distinguish species S_i from its siblings species that are part of the same sub-tree (e.g., those that share a common ancestor at level 3). For this reason, we expect the generated images to show higher probabilities of being classified as any of the descendant species of the sub-tree, compared to the other species that are outside of the sub-tree. 
To quantify this behavior, we can measure the
change in probability distributions for species within and outside the sub-tree after masking out an internal node. Since we expect probabilities to increase only for species within the sub-tree, the mean increase in probabilities for within-subtree species should be higher than that for out-of-subtree species, as empirically demonstrated later in the Results Section.



 §.§ Proposed Experiment of Trait Swapping



In trait swapping, we substitute the level-l embedding of a source species with the level-l embedding of a sibling subtree at an equivalent level. 
<Ref> shows a schematic representation of the trait swapping experiment 
where the level-3 embedding (green) of a source species is replaced with its sibling level-3 embedding (yellow). 
Images generated for this perturbed embedding are expected to retain all of the traits of the source species except the swapped embedding, which should borrow traits from the sub-tree rooted at the sibling node. Visualizing trait differences in the generated images before and after trait swapping can help us understand the evolutionary traits that branched at a certain level (, those leading to the diversification of green and yellow sub-trees at level-3 in the example phylogeny of Figure <ref>). 
In terms of the probability distribution, similar to trait masking, we expect to see a drop in the probabilities of the source species (pink), and simultaneously, we expect an increase in probabilities for all the descendent species in subtree at node yellow, red and purple.





§ EVALUATION SETUP




  
Datasets:
We use a collection of fish images as our primary dataset for evaluation. This dataset was procured from the Great Lakes Invasives Network (GLIN) <cit.>  Project, comprising a total of 5434 images spanning 38 fish species. We obtained the phylogenetic tree of fish species from opentree <cit.> python package (see <Ref> for details on the phylogenetic tree). The raw museum images were pre-processed and resized to 256 × 256 pixels and the dataset was partitioned into training and validation sets, following a 75-25 split. We provide additional results on the CUB-200-2011 dataset <cit.> of bird species in <Ref>.



  
Baselines of Conditioning Mechanisms:
(1) Class Conditional: One of the simplest ways of encoding information about a species class is to map class labels y ∈ [1, N_c] to a fixed d-dimensional embedding vector e ∈ℝ^d using a trainable embedding layer. Note that the resulting embeddings are not designed to contain any hierarchical information in contrast to HIER-Embed. (2) Scientific Name Encoding:  The scientific name of a species contains valuable biological information typically comprising of a combination of the genus name and species name. Since species that share their genus name are likely to contain common phylogenetic traits, we use them as a baseline for conditioning LDMs for discovering evolutionary traits. Specifically, we employ a pre-trained frozen CLIP model <cit.> to encode the scientific names of species into fixed d-dimensional embeddings.



  
Training details:
We used d'=128 as the embedding dimension for each level of HIER-Embed, which when concatenated across the four levels produces the combined hierarchical embedding of d=512 dimensions. Phylo-Diffusion uses this d-dimensional embedding to condition LDMs through cross-attention in denoising the U-Net backbone and train LDMs without classifier-free guidance. We used VQGAN <cit.> as the backbone encoder-decoder to achieve the latent representations desired for LDMs with a downsampling factor of 4. All the models with different encoders are trained for 400k iterations, employing the best model checkpoint if convergence occurs early. Additional hyperparameters, such as learning rate, batch size, and U-Net architecture, are detailed in <Ref>.








§ RESULTS




 §.§ Quality of Generated Images

<Ref> compares the quality of generated images of baselines using the metrics of Fréchet Inception Distance (FID) score, Inception Score (IS), and Precision, Recall calculated in the feature space as proposed in <cit.>. Our results show that Phylo-Diffusion is at par with state-of-the-art generative models, achieving an FID of 11.38 compared to LDM's 11.46. We show a sample of generated images in <Ref>, with additional images provided in <Ref>. We also show the robustness of Phylo-Diffusion's results with varying numbers of phylogenetic levels and embedding dimensions in <Ref>.






 §.§ Classification Accuracy


We used a separate model for species classification, specifically a ResNet-18 model <cit.> trained using the same training/validation split as Phylo-Diffusion. The primary objective behind building this classifier is to verify if images generated by Phylo-Diffusion contain sufficient discriminatory information to be classified as their correct species classes. <Ref>  compares the classification F1-scores over 100 samples generated by baseline conditioning schemes. We can see that the synthetic images generated by Phylo-Diffusion achieve the highest F1 score (82.21%), which is quite close to the F1 score of the base classifier on the original test images (85%). We present additional results showing the generalizability of Phylo-Diffusion in classifying generated images to unseen species in <Ref>.



 §.§ Matching Embedding Distances with Phylogenetic Distances


We investigate the quality of embeddings produced by baseline methods by comparing distances in the embedding space with the ground-truth (GT) phylogenetic distances computed from the tree of life, as illustrated in <Ref>.
Ideally, we expect distances in the embedding space of species pairs to be reflective of their phylogenetic distances. 
For Class Conditional, we can see that the distance matrix does not show any alignment with the GT phylogenetic distance matrix.
In the case of Scientific Name Encoding, the distance matrix exhibits notable similarities to the phylogenetic distances, thanks to the hierarchical nature of information contained in scientific names (i.e., genus-name & species-name). However, one limitation of this encoding is its inability to capture inter-genus similarities or differences. In contrast, HIER-Embed shows a distance matrix that closely aligns with the GT phylogenetic distance matrix, validating its ability to preserve evolutionary distances among species in its embedding space.









 §.§ Trait Masking Results



To obtain classification probabilities or logits associated with generated images, we employ the classifier detailed in Section <ref>. For the masked embeddings of subtrees at level 3, defined as [, , ,  ], we analyze the logits of generated images and compare them with those generated without masking. <Ref> demonstrates that for a specific subtree, in this case Lepomis, logits for species within the subtree are higher compared to those for species outside it. This outcome aligns with the expectation that Phylo-Diffusion, when provided with information up to Level 3, can capture overarching characteristics of all species within the given subtree. Additionally, <Ref> presents probability distributions for species within the Lepomis subtree when the full set of hierarchical encodings [, , , ] is provided. It demonstrates that the probabilities are significantly higher for the targeted class, as intended for image generation. After masking, we observe that the generated images are very similar and capture common features of the Lepomis genus. For all our calculations and plots, we generate 100 images for each subtree and node.
<Ref> contains additional histograms that detail logit distributions across all different subtrees at each level, offering comprehensive insights into how the model discriminates and learns the hierarchical structure across different levels and nodes.







  
Quantitative Evaluation of Probability Distrubtions:

To quantitatively evaluate the ability of Phylo-Diffusion to capture hierarchical information and show desired changes in probability distributions after masking, we compute the following metrics. Let us denote the set of all species in the data as 𝒮 and for a given sub-tree at an internal node i of level l, let us denote the subset of descendant species as 𝒮_i^l = {S_1, S_2, …, S_n}. We first compute the reference probabilities P_ref of every species before masking (, by using all four level embeddings). Let us denote the probability of predicting a generated image using all four embeddings of a descendant species S_j ∈𝒮_i^l into species class S_k as P_S_j(S_k). The reference probability of a species S_k can then be given as: 

    P_ref(S_k) = 1/|𝒮_i^l|-1∑_S_j ∈𝒮_i^l ∖ S_k P_S_j(S_k),   if  S_k ∈𝒮_i^l,
    1/|𝒮_i^l|∑_S_j ∈𝒮_i^l P_S_j(S_k),                if  S_k ∉𝒮_i^l.

Note that when S_k is part of the sub-tree, , S_k ∈𝒮_i^l,  P_ref(S_k) is computed by averaging over |𝒮_i^l|-1 probability values since we exclude the case when S_k is used to generate the images. On the other hand, when S_k is outside of the sub-tree, ,  S_k ∉𝒮_i^l, we average over all  |𝒮_i^l| probability values. Given these reference probabilities values before masking, we can compute the change in probability of predicting species S_k after masking as P_diff(S_k) = P_mask(S_k) - P_ref(S_k), where P_mask(S_k) is the probability of predicting a generated image after masking to S_k. We expect P_diff to be larger for descendant species S_k∈𝒮_i^l compared to species that are outside of the sub-tree because of the dispersion of probabilities in a sub-tree as a consequence of masking.  We thus compute the average P_diff for species that belong to subtree 𝒮_i^l as  P_diff^sub(i,l) and species that are outside the subtree 𝒮_i^l as P_diff^out(i,l). 


<Ref> shows the box plot of P_diff^sub and P_diff^out across internal nodes at levels 2 and 3. We observe that species within the subtree exhibit a more pronounced increase in probabilities compared to species outside the subtree, aligning with our expectations. Notably, this trend is consistently observed across both levels 2 and 3. More details of class-wise probability distribution shifts for each level are provided in <Ref>.
The outcomes of these experiments affirm that Phylo-Diffusion effectively identifies unique features at Levels 2 and 3, and captures shared features or traits of any chosen subtree at the internal nodes of the phylogeny.






 §.§ Trait Swapping Results


Figure <ref> shows examples of trait swapping results enabled by Phylo-Diffusion. For the first example (first row of <Ref>), we swap the level-2 embedding of source species Noturus exilis with level-2 embedding of its sibling group Notropis/ Carassius. The goal here is to discover traits of Noturus exilis inherited at level-2 that differentiate it from other groups of species that branched out at this point of time in evolution. We can see that the generated images of the perturbed embedding (center) exhibit the absence of barbels (whiskers) highlighted in purple, while the caudal (or tail) fin is beginning to fork (or split), a trait adopted from Notropis (right). In contrast, other fins such as the dorsal, pelvic, and anal fins highlighted in green remain similar to those of the source species, Noturus exilis (left). This suggests that at level-2, Notropis and Noturus species diverged by developing differences in two distinct traits, barbels and forked caudal fins while keeping other traits intact.




For the second example (<Ref>, row 2), we swap level-2 information of Gambusia affinis (left) with that of Esox americanus (right). The generated images of the perturbed embedding (center) exhibit a more pointed head highlighted in purple, and a slimmer body shape resembling Esox americanus. Notably, the perturbed species retains discoloration at the bottom from the source species highlighted in green. <Ref> presents probability distributions (or logits) of Gambusia affinis before and after trait swapping using the classifier detailed in Section <ref>. We observe a slight decrease in logits for Gambusia affinis and an increase in logits for Esox americanus, consistent with our expectations. In the third row of <Ref>, we swap level-3 information of Lepomis gulosus (left) with that of Morone genus (right). The resulting images from the perturbed embedding (center) capture the horizontal line pattern characteristic of Morone genus, and the dorsal fin highlighted in purple begins to split. All these examples suggest novel scientific hypotheses about differences in evolutionary traits acquired by species at different ancestry levels, which can be validated by biologists in subsequent studies.
Note that our experiments are most effective at levels near the species nodes, specifically at levels 2 & 3, since phylogenetic signal is known to diminish as we move toward the root of the tree <cit.>.
Additional visualizations of trait swapping results are provided in <Ref>.





  
Comparisions with Phylo-NN:
<Ref> compares trait swapping results of Phylo-Diffusion and Phylo-NN for the same set of example species. <Ref> shows trait swapping at level-3 for the source species of Lepomis gulosus (top) and target sub-tree of the Morone genus (bottom). In Phylo-NN, images generated by perturbing the Imageome sequences appear blurry (red circle), while Phylo-Diffusion effectively captures the splitting of dorsal fin (purple circle) and the horizontal stripe pattern of the Morone genus, while maintaining the fin structure of Lepomis gulosus (green circle). Similarly, <Ref> compares trait swapping for Noturus miurus (top) with the target sub-tree of Notropis genus (bottom) at level-2. For Phylo-NN, the perturbed images are almost identical to the source species. However, Phylo-Diffusion shows visible trait differences such as the absence of barbels and the caudal (or tail) fin beginning to fork or split (purple circle), which are traits picked from the target sub-tree of Notropis genus. Note that we had considered the same target sub-tree in <Ref> row 1 and observed similar trait differences in the generated images after perturbation, further validating the ability of Phylo-Diffusion to discover consistent evolutionary traits. We provide additional results comparing Phylo-Diffusion and Phylo-NN trait swapping results in <Ref>.


§ CONCLUSIONS AND FUTURE WORK

In this work, we introduced Phylo-Diffusion, a novel framework for discovering evolutionary traits from images by structuring the embedding space of diffusion models using tree-based knowledge. 
In the future, our approach can be extended to work on other applications involving image data linked with phylogenies or pedigrees.
Our work also has limitations that need to be addressed in future research. For example, while our current work is limited to discretized trees with a fixed number of levels, future works can focus on discovering evolutionary traits at every internal node of the phylogenetic tree with varying levels without performing any discretization. Future works can also attempt to capture convergent changes in evolution, i.e., changes that occur repeatedly in different branches of the tree, and perform ancestral state reconstruction with uncertainty estimates.



§ ACKNOWLEDGMENTS

This research is supported by National Science Foundation (NSF) awards for the HDR
Imageomics Institute (OAC-2118240). We are thankful for the support of computational resources provided by the Advanced Research Computing (ARC) Center at Virginia Tech.

This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (
https://www.energy.gov/doe-public-access-plan).









splncs04




§ SUPPLEMENTARY MATERIALS

Here is a summary of additional details and experiments included in the Appendices.


    
  * <ref>: Hyperparameter Settings and Training Details
    
  * <ref>: Details about Phylogenetic Tree
    
  * <ref>: Additional Details about Trait Masking Experiments
    
  * <ref>: Additional Examples for Trait Swapping Experiments
    
  * <ref>: Additional Comparisons with PhyloNN
    
  * <ref>: Additional Samples of Generated Images
    
  * <ref>: Ablation Results
    
  * <ref>: CUB Dataset Results





§ HYPERPARAMETER SETTINGS AND TRAINING DETAILS
 

<Ref> lists all the hyperparameters for the models trained. We used cross-attention as the conditioning mechanism for all the models and all the models were trained from scratch. At the inference stage, we used DDIM <cit.>sampling with 200 steps. For computing metrics like FID, IS, , we use ADM's <cit.> TensorFlow evaluation script.




All diffusion models require about 7 days to train on a single A100 GPU for both bird and fish datasets. Inference throughput is 0.9 samples/sec using DDIM with 200 steps computed over generating 100 images per class. We do not have any additional overheads in training and inference time compared to LDMs.



§ DETAILS ABOUT PHYLOGENETIC TREE
 
<Ref> shows the phylogeny tree for all the species in the fish dataset along with the information of the four discrete levels used in our study (marked by different colored circles). <Ref> and <ref> list out all the groupings (subtrees) made after discretizing the tree into four levels where the fourth level is the species itself.












§ ADDITIONAL DETAILS ABOUT TRAIT MASKING EXPERIMENTS




  §.§.§ Additional Visualizations of Changes in Probability Distributions after Masking:
 <Ref>, <ref>, <ref>, <ref> and <ref> show additional examples of changes in probability distributions when level-4 information is replaced with noise. In each figure, the first two plots display probability distributions (or logits) of images generated using embeddings from all four levels, [, , , ], of two representative species sharing a common ancestry up to level-3 (highlighted in green). We show that the logits are higher for the targeted species as expected. The third plot logits after masking level-4 embeddings, leading to a dispersion of probabilities across all descendant species within the subtree up to level-3 (highlighted in green). The only exception is <Ref>, where there is some skewness in the logits of descendant species, which is likely due to the data imbalance across classes at higher levels of the tree and also due to biases in the classifier (classifier test accuracy is 85% as reported in <Ref>). In this case, the classifier sometimes misclassifies Notropis boops as Notropis blennius in the first plot and Notropis dorsails as Notropis buccatus in the second plot. Consequently, the third plot for the Notropis subtree shows a higher probability for Notropis blennius. Similarly, <Ref>, <ref> and <ref> provide examples of trait masking where both Level 3 and 4 are replaced with noise, [, , , ]. We observe a similar trend in the dispersion of probabilities across all descendant species within the same subtree at level-2. In all trait masking visualizations, we consistently observe that logits of generated images of species within the subtree (highlighted in green) are higher than for species outside the subtree. This demonstrates Phylo-Diffusion's ability to effectively capture hierarchical information at various levels of the phylogenetic tree.






















  §.§.§ Additional Quantitative Results of Trait Masking:
 Tables <ref> and <ref> show the change in probabilities for different nodes at levels 3 and 2, respectively. We can see that indeed P_diff^sub is larger than P_diff^out for all internal nodes at levels 2 and 3 (except node 5 at level 2), indicating that Phylo-Diffusion is capturing the necessary hierarchical information required for the dispersion of probabilities after masking. <Ref> in the main paper shows the box plots of Tables <ref> and <ref>. <Ref> summarizes this information by showing the average P_diff^sub and P_diff^out for all nodes at a given level. It is important to note that for this experiment, we focus on nodes that have more than one species in the defined subtree.











§ ADDITIONAL EXAMPLES FOR TRAIT SWAPPING EXPERIMENTS




  §.§.§ Additional Visualizations of Trait Swapping Experiments:
 <Ref> illustrates trait swapping for the source species Noturus exilis (left), where the information at Level-2 is swapped with that of a sibling subtree at Node B (right). The image in the center is generated using the trait swapped embedding. This visualization of the perturbed species helps us study the trait changes that would have branched out at level-2 between Node A and Node B.In the generated image (center),  we observe the absence of barbels(whiskers), and the caudal fin (tail) is getting forked (or split) highlighted in pink, which are traits adopted from species in the subtree at B (Notropis). Whereas other fins like the dorsal, pelvic, and anal fin still resemble the source species Noturus exilis highlighted in green. The same is also reflected in the change of probability distribution after perturbations; the probability distribution of source species Noturus exilis decreases and the probability of it being a Notropis increases slightly.

Similarly in <Ref>, for the studied species Lepomis gulosus (left), the information at Level 3 is swapped with subtree at Node B (Morone). The perturbed species generated (center) captures traits from both lineages. The spotted pattern in the body and fins is retained from Lepomis (Node A) but these spots now start to follow the horizontal stripes pattern observed in Morone (species at Node B). Additionally, the dorsal fin highlighted in pink starts to split into two, with the left half retaining the spiny structure from Lepomis highlighted in green. This observation suggests that the species at Node A and B possess distinct traits at this level-3 branching node. The same is reflected in the shift in probability distributions towards species in Node B after trait swapping.








§ ADDITIONAL COMPARISONS WITH PHYLONN


<Ref> compares the trait swapping experiment for Phylo-Diffusion with the PhyloNN baseline, where level-2 information of Gambusia affinis is replaced with that of Esox americanus. In the highlighted pink circle, the face of the image generated after perturbations (center) becomes more pointed, and the body shape flattens to resemble Esox americanus. This perturbed image also retains traits like the caudal (tail) fin and the black-spotted pattern towards the bottom (highlighted in green) from the source species, Gambusia affinis. The differences observed with Phylo-Diffusion are notable, whereas the PhyloNN generates a perturbed image nearly identical to the original, showing no significant changes.

Similarly, <Ref> shows a comparison after replacing level-2 information of Notropis husonius with that of Noturus. For Phylo-Diffusion, the caudal (tail) fin is vibily joining highlighted in pink, resembling the caudal fins of Noturus. This change is analogous to <Ref>, where level-2 information of Noturus was replaced with Notropis (vice-versa), resulting in the caudal (tail) fin getting forked or split. Hence, this helps us understand that at Level-2, the two species diverged to develop different caudal fins. However, for PhyloNN, the generated image after trait-swapping is blurry, and most of the traits still closely resemble close the source species, which is unlikely given that the level-2 embeddings have been replaced.









§ ADDITIONAL SAMPLES OF GENERATED IMAGES


<Ref> shows additional examples of generated images for different species using Phylo-Diffusion. Each row of the figure depicts the generated images for the same species while the different rows represent distinct species. Notably, we observe inter-class variations among species belonging to the same class, such as differences in fish orientation and size.






§ ABLATION RESULTS
 



 §.§ Generalization to Unseen Species: Leave-three-out

As an additional ablation experiment, we conduct a leave-three-out experiment by excluding three species from different subtrees during training to test the model's ability to generalize to new species and situate them in the phylogeny. This experiment involves training the model excluding three species, Notropis blennius, Noturus gyrinus, and Lepomis humilis that belong to different subtrees as seen in <ref>. The generated images from the three subtrees after trait masking closely resemble the actual images of the 3 species, with an F1 score of 95.6 on a classifier trained to discriminate the 3 species. This experiment underscores the robustness and accuracy of Phylo-Diffusion in embedding and generating phylogenetically consistent images.





 §.§ Effect of Varying the Number of Levels in Phylo-Diffusion

To demonstrate the robustness of Phylo-Diffusion, we perform ablation experiments with varying numbers of levels in the discretization of the phylogeny tree. We show that the choice of the number of levels depends on the depth of the phylogenetic tree and the internal nodes to be studied.
We train models with {2, 4, 6,8} levels on the phylogeny shown in <Ref>. <Ref> demonstrates that the model is robust to the choice of the number of levels.





 §.§ Effect of Varying Embedding Dimensions in Phylo-Diffusion

We further evaluate the effect of varying the number of embedding dimensions used in HIER-Embed on the performance of Phylo-Diffusion. In this experiment, we trained Phylo-Diffusion by varying HIER-Embed's dimension in the following range of values: {16,32,64,128,256,512,1024}. <Ref> shows that Phylo-Diffusion is quite robust to the choice of embedding dimension with minimal drop in performance as we reduce the embedding dimension even to small values.






§ CUB DATASET RESULTS
 
To show the applicability of our approach on other datasets with larger and deeper phylogenies, we perform additional experiments on 190 bird species from the CUB-200-2011 dataset (see <Ref>). We selected the set of bird species based on whether we are able to obtain their phylogenetic knowledge from https://birdtree.org/Bird Tree, which are pre-processed similar to the fishes. We removed the background of these images using segmentation masks to focus only on the body of the birds.







  §.§.§ Trait Masking:
 Similar to fishes, <Ref> shows the changes in probability distributions when Level 3 & 4 information is replaced with noise. The first two plots show the logits of images generated for Black-footed albatross and Sooty albatross using embeddings from all the four levels. The third plot shows the dispersion of logits across the three descendant species that are part of the sub-tree defined till level 2, i.e., masking level 3 & 4. We see similar results for CUB as well where the probability of classifying the generated images into any of the descendant species that share a subtree (highlighted in green) is generally greater than the species outside the subtree. 








  §.§.§ Trait Swapping:
 
<Ref> shows an example of the trait swapping experiment on the birds dataset, similar to the experiments for fishes in the main paper. We see that the image generated from the perturbed embedding (center) picks up the trait of black coloration around the eye (purple circle) that is shared by the target sub-tree (right) while traits like pointed beak (green circle) are retained from the source species (left).







 Paper Title: Enhanced Uncertainty Estimation in Ultrasound Image Segmentation with MSU-Net.





Uncertainty Estimation with MSU-Net






R. Banerjee et al.




Carnegie Mellon University, Pittsburgh PA 15213, USA {rohinib,cgmorale,awd}@andrew.cmu.edu


Enhanced Uncertainty Estimation in Ultrasound Image Segmentation with MSU-Net
    Rohini Banerjee* Cecilia G. Morales* Artur Dubrawski
    August 5, 2024
=============================================================================



*These authors contributed equally to this workfootnote

§ ABSTRACT 
Efficient intravascular access in trauma and critical care significantly impacts patient outcomes. However, the availability of skilled medical personnel in austere environments is often limited. Autonomous robotic ultrasound systems can aid in needle insertion for medication delivery and support non-experts in such tasks. Despite advances in autonomous needle insertion, inaccuracies in vessel segmentation predictions pose risks. Understanding the uncertainty of predictive models in ultrasound imaging is crucial for assessing their reliability. We introduce MSU-Net, a novel multistage approach for training an ensemble of U-Nets to yield accurate ultrasound image segmentation maps. We demonstrate substantial improvements, 18.1% over a single Monte Carlo U-Net, enhancing uncertainty evaluations, model transparency, and trustworthiness. By highlighting areas of model certainty, MSU-Net can guide safe needle insertions, empowering non-experts to accomplish such tasks.








§ INTRODUCTION


Trauma, the leading cause of death among young individuals in the U.S. <cit.>, often results in blood loss, which requires rapid fluid resuscitation for vital organ oxygenation. In austere settings, accessing timely medical care can be challenging due to limited access, dangerous conditions, time constraints, or the absence of medical infrastructure, making expertise for optimal needle insertion sites critical. Autonomous robotic systems can assist in intravenous fluid administration when medical experts are unavailable, providing support in emergencies. These systems can also guide non-experts in accurate performance of phlebotomy tasks, empowering them to contribute effectively in dire medical situations.

Ultrasound imaging is widely used for locating vessels to enable fluid resuscitation due to its affordability, speed, safety, and portability, unlike CT or MRI imaging that are not portable and use ionizing radiation <cit.>. Despite advancements in autonomous needle insertion into blood vessels <cit.>, a critical challenge persists: inaccurate predictions can lead to severe consequences. Failing to anticipate vessel structure during intravenous cannulation may result in catastrophic outcomes for critically injured individuals; incorrectly predicting two adjacent vessels as a single one could lead to laceration of the vessel wall and cause hemorrhage upon needle insertion <cit.>. Both medical personnel and automated tools need to assess the certainty of their estimates of vessel location and structure to mitigate this risk <cit.>. 
As healthcare increasingly adopts automation, reliance on Artificial Intelligence (AI) models grows. Building a  model that communicates its uncertainty becomes essential to help users focus their attention and actions on where the model is confident, such as guiding needle insertion accurately within vessel segments. In this paper, we attack the problem of uncertainty estimation of models trained to guide needle insertion tasks. Our contributions include: 

(1) We introduce MSU-Net, a novel Multistaged Monte Carlo U-Net, demonstrating significant advancements in uncertainty quantification; (2) Our model achieves state-of-the-art performance in ultrasound image segmentation, marking the first known improvement in uncertainty estimation for this application.





§ RELATED WORKS



 §.§ Uncertainty Quantification in Deep Learning

Uncertainty quantification is vital in understanding the reliability of AI model predictions. Traditionally, the frequentist approach assumes a single point estimate of network weights, using estimated class likelihoods as confidence measures for predictions. However, studies have shown that these likelihoods often overestimate accuracy <cit.>, and the popular metric used to quantify confidence, expected calibration error, has been criticized for bias and inconsistency <cit.>. This limits utility, motivating the need for alternative approaches to accurately quantify model uncertainty rather than just confidence.

Predictive uncertainty comprises of aleatoric and epistemic components <cit.>. Aleatoric uncertainty accounts for inherent noise in observations, while epistemic uncertainty arises from limited training data and model parameter uncertainty. Recent advancements in Bayesian inference and Bayesian neural networks have provided robust frameworks to quantify both forms of uncertainty by estimating posterior distributions over model weights. Gal and Ghahramani <cit.> introduced Monte Carlo (MC) dropout for Bayesian inference in deep learning, leveraging dropout in convolutional layers for stochastic forward passes to approximate Bayesian variational inference. Bayesian approximation using MC dropout has been extensively applied: Kendall and Gal <cit.> developed Bayesian SegNet for scene understanding, while Dechesne and Lassalle <cit.> used it in U-Net for high-accuracy image segmentation. Seedat <cit.> proposed a human-in-the-loop system using model uncertainty. Yet, single-model architectures are now supplanted by model ensembles due to difficulties in capturing inherent variability.




 §.§ Multistage Neural Network Ensembles

Training multiple individual models through model ensembling offers a range of tools to encourage member model diversity and improve overall accuracy. Diversity can be achieved through several techniques, the most popular being bagging <cit.>, stacking <cit.>, and boosting <cit.>. Non-Bayesian deep ensembles improve predictive uncertainty estimation compared to single model architectures <cit.>, yet remain limited by naïve aggregation strategies such as simple or weighted averaging or majority vote <cit.>. Yang et al. <cit.> addresses drawbacks of traditional ensembles by training a secondary neural network to adaptively assign weights, inspired by stacked generalization. This approach offers flexibility in candidate selection and leverages non-linear modeling capabilities of neural networks. 


Lai et al. <cit.> designed a multistage reliability-based neural network ensemble learning approach for credit card evaluation based on a decorrelation maximization procedure to select diverse models. Their approach combines outputs using different aggregation strategies, achieving superior accuracy over single and hybrid models. Multistage ensembling with U-Nets also proves beneficial; Yin and Hu <cit.> designed Paw-Net, a multistage ensemble for semantic segmentation. Paw-Net demonstrates higher final Intersection-over-Union (IoU) scores by integrating outputs of multiple U-Nets specialized in different classes. 

Our proposed MSU-Net architecture combines the benefits of these appro­aches: candidate U-Nets are first overproduced and then selected based on decorrelation to compose an ensemble. The final stage combines outputs using a Monte Carlo U-Net approach to produce accurate segmentation maps.






§ METHODS



 §.§ Monte Carlo U-Net (MCU-Net)

We introduce stochasticity into our inference process by incorporating dropout layers into our chosen U-Net architecture, thereby enabling MC dropout, as in MCU-Net  <cit.>. MCU-Net is henceforth referred to as our baseline model. Indiscriminately placing dropout after each convolutional layer can lead to reduced fit and poor test performance <cit.>. We opt to situate dropout layers in the decoder section of our U-Net instead. Each decoder block consists of two sets of convolutional layers (3x3 convolution filter, batch normalization, ReLU activation) and an attention block. This enables us to approximate Bayesian inference by conducting T forward passes, or MC samples, of the U-Net during testing. Our empirical results show no significant improvement beyond T=30. Our model outputs both logits and logit variances <cit.> to use for segmentation and epistemic uncertainty maps,



with  [p_t, σ^2_t] =f^ω_t(x) representing the t-th forward pass of MCU-Net with learned weights ω_t. Given sigmoid activation, σ_SIG, the ensemble prediction is  aggregated by averaging individual model outputs, σ_SIG(p_t),  across all T samples. Similarly, the raw epistemic uncertainty map is obtained by averaging logit variances across all MC samples.




 §.§ Multistage U-Net (MSU-Net)

The MSU-Net architecture, illustrated in Fig <ref>, is structured in three main stages: (1) ensembling, (2) candidate selection, and (3) combining. In stage 1, we train multiple models referred to as candidates. Bagging is employed to induce diversity into our training data; given the training dataset of size n_TR, we repeatedly sample with replacement to generate M training subsets, each also of size n_TR. We then proceed to train a total of 15 bootstrapped models <cit.>. Hyperparameter optimization, specifically determining the number of epochs of training for each candidate model, is performed using the VS1 validation set, held out for this purpose. Early stopping is applied to each candidate when validation loss begins to increase, to mitigate model overfitting. 

In stage 2, candidates are selected to form a diverse and efficient ensemble, aiming to reduce computational costs by minimizing correlation. We apply the decorrelation maximization method from <cit.>, using the Brier score loss matrix computed from models f^ω_1_1, f^ω_2_2, ⋯, f^ω_M_M on a separate validation set VS2 within the specified region of interest, replacing binary error with Brier score for finer error measurement. The mean, variance, and covariance of the Brier score loss matrix is then computed in order to construct the correlation matrix R, where the ith row and jth column represents the degree of correlation between f^ω_i_i and f^ω_j_j. We write R in a block form for candidate model f^ω_i_i as


    R [  R_-i   r_i; r_i^T     1 ]


where R_-i is the principle submatrix of R resulting from deleting the ith row and ith column. Subsequently, we compute the plural-correlation coefficient <cit.>

    ρ_i^2 = r_i^T R^-1_-i r_i

by which candidate f^ω_i_i is kept if ρ_i^2 ≤θ for some threshold θ, else discarded, in order to build our final ensemble. Empirical trials testing θ thresholds indicate that K=3 performs almost equally as well as K=15, suggesting that minimal additional training will suffice for our new architecture.






















In stage 3, we train a final MCU-Net combiner taking ensemble member outputs as inputs and outputting segmentation maps. We compute the segmentation and epistemic uncertainty maps of MSU-Net analogous to MCU-Net.





§ EXPERIMENTS



Dataset and Training  
We used a ultrasound scanning system described by Morales et al. <cit.>  
to scan the CAE Blue Phantom anthropomorphic gel model simulating femoral vessels. Equipped with a 5MHz linear transducer, the system can scan up to 5cm in depth, producing 2D transverse ultrasound images. Expert clinicians annotated these images using the Computer Vision Annotating Tool (CVAT) <cit.>, followed by cropping and resizing to 256 × 256 pixels. The dataset was split into training (1392 images), validation (907 images), and testing (856 images). The validation set was further randomly split into two disjoint sets, VS1 and VS2.

For image segmentation, we employed U-Net architectures with a ResNet34 backbone in Pytorch, utilizing the Segmentation Models library <cit.> pretrained on ImageNet. To integrate MC Dropout, dropout layers with optimal rates of 0.4 and 0.5 were added after each ReLU activation  in the decoder <cit.>. Training used a batch size of 8, Adam optimizer with a learning rate of 0.0001, and early stopping based on validation loss stabilization. MSU-Net incorporated bagging for ensembling and the plural-correlation coefficient as a correlation metric.







































Distribution Divergence Estimation
Model quality hinges on uncertainty distribution: low for correct, high for incorrect predictions, enhancing calibration and user trust <cit.>. We utilize the Rényi divergence (RD) statistic as our metric for comparison. It is a generalization of Kullback-Leibler (KL) divergence that measures the dissimilarity between two probability distributions p and q. van Erven and Harremoës <cit.> relay an operational characterization of RD as the number of bits by which a mixture of two codes, p and q, can be compressed.  For some measurable set ℳ_0 that p and q lie in and order α∈ℝ∖{1 }, RD of distribution p from distribution q is defined as:


    R_α(p || q) = 1/α-1ln∫_ℳ_0 p^α(x) q^1-α(x)  dx


A nonparametric estimator of RD that is conditionally L_2-consistent using only k-nearest-neighbor statistics has been proposed to considerably reduce computational effort <cit.>. Let X_1:n_0=(X_1, ⋯, X_n_0) be an i.i.d. sample from a distribution with density p and Y_1:n_1=(Y_1, ⋯, Y_n_1)  an i.i.d. sample from a distribution with density q. We denote ρ_k(i) to be the k-th nearest neighbor of observation X_i in X_1:n_0 and v_k(i) the k-th nearest neighbor of X_i in Y_1:n_1. With B_k, α = Γ(k)^2/Γ(k-α+1) Γ(k+α-1), we can estimate the RD by:


    R_α(p || q) = 1/α-1ln( n_0^-1∑_i=1^n_0((n_0-1) ρ_k(i)/n_1 v_k(i))^1-α B_k,α)


We use RD to measure the ability of our method to distinguish correct predictions (p) from incorrect predictions (q). We conduct permutation testing to assess deterministicity and bootstrapping to obtain confidence intervals for the results. Nearest neighbor estimators are sensitive to perturbations in the underlying distribution, and hence their limited variance cannot be consistently estimated by a naïve Efron-type bootstrap <cit.>. 
Since this behavior may result in a non-negligible positive bias in bootstrap estimates, we instead apply a direct M-out-of-N (MooN) type bootstrap <cit.> for this metric, shown in Algorithm <ref>.









§ RESULTS AND DISCUSSION


We evaluate MSU-Net using quantitative and qualitative metrics. To alleviate heavy class imbalance, we choose to average metrics over a predefined region of interest (ROI), shown in  Fig. <ref>. Fig. <ref> displays model uncertainty distributions on the test set. We employ kernel density estimation with a Gaussian kernel and optimal bandwidth via Silverman’s rule of thumb to visualize difference in means between correct and incorrect prediction uncertainty distributions over 100,000 samples. To quantify separation between these distributions with RD, we select k=4 for k-nearest neighbors and α=0.85 for numerical stability and interpretability <cit.>. We perform B=1000 permutations and bootstrapped samples. For MooN-type bootstrap, γ=0.8 is selected to maintain the largest proportion of original data while achieving the closest coverage probability of 0.95 for 95% intervals. Our final results are displayed in Table <ref>.

 


Fig. <ref>a shows training behavior for both models. MSU-Net shows a more stable convergence and consistently outperforms MCU-Net during validation performed after each epoch. MSU-Net achieves 18.1% better mean IoU and a significant improvement in sensitivity and false negative rate scores at alpha level 0.05, while other metrics remain similar. We additionally utilize precision-recall curves, seen in  Fig. <ref>b, which are resilient to unbalanced classes since they only focus on positive class predictions. Performance results are displayed in Table <ref>.












Permutation tests for MCU-Net and MSU-Net (p ≤ 0.003 for both) indicate that the observed separation between correct and incorrect predictions is not random in both. Yet, at the 95% confidence level, we see no overlap between their 95% confidence intervals, showing that the ability of MSU-Net to distinguish correct from incorrect predictions is significantly better than that of MCU-Net.

Qualitative uncertainty maps visually validate our findings and capture local variations in model performance <cit.>. MCU-Net exhibits indiscriminately low uncertainty in Fig. <ref>, whereas MSU-Net provides more interpretable uncertainty values. MSU-Net shows clearer vessel tops with lower uncertainty and higher uncertanties at the bottom of vessels which is consistent with our assessment. 
MSU-Net yields a higher credibility in assessing its predictive reliability. 



MSU-Net generally outperforms MCU-Net. Although it performs marginally worse at specificity and false positive rate (FPR), the precision-recall curves from Fig. <ref>b show a higher average precision score for MSU-Net at 0.945 than MCU-Net at 0.875 compared to the baseline score of 0.09 for a `no-skill' classifier. As such, at the same level of recall, MSU-Net correctly classifies a greater proportion of pixels that are actually vessels than MCU-Net. Crucially, MSU-Net achieves a considerably lower false negative rate (FNR) than MCU-Net. In this context, failing to anticipate a true vessel can have more catastrophic consequences to a critically injured individual than incorrectly anticipating a vessel. MSU-Net improves credibility not only through higher quality results, but also through more accurate results while avoiding potentially disastrous deficiencies of predictive modeling in the context of medical image segmentation.




§ CONCLUSION

This paper introduces MSU-Net, a multistaged Monte Carlo U-Net for uncertain ultrasound image segmentations. It improves model transparency and trustworthiness compared to standard U-Net by enhancing uncertainty evaluation, despite minimal additional training. Our framework sets a benchmark for future studies, offering qualitative maps for intuitive model assessment. While our preliminary results are limited to phantom data and binary image segmentation, we aim to validate these findings on live animal and human data in future work.





  §.§.§ 
 Authors thank Nico Zevallos, Dr. Michael R. Pinsky, and Dr. Hernando Gomez for gathering experiment data. This work was partially supported by the U.S. Dept. of Defense contracts W81XWH-19-C0083 and W81XWH-19-C0101.



  §.§.§ 

The authors have no competing interests to declare that are
relevant to the content of this article.








 splncs04
 























 Paper Title: Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation.








xiaoye@hust.edu.cn

  Huazhong University of Science and Technology
  China



chenqiyuan1012@gmail.com

  Zhejiang University
  China



weiw@hust.edu.cn

  Huazhong University of Science and Technology
  China



gasolsun36@gmail.com

  Xiamen University
  China





dongjf24@gmail.com

  Zhejiang Gongshang University
  China







§ ABSTRACT 
Despite the remarkable ability of large vision-language models (LVLMs) in image comprehension, these models frequently generate plausible yet factually incorrect responses, a phenomenon known as hallucination. 
Recently, in large language models (LLMs), augmenting LLMs by retrieving information from external knowledge resources has been proven as a promising solution to mitigate hallucinations. 
However, the retrieval augmentation in LVLM significantly lags behind the widespread applications of LVLM. 
Moreover, when transferred to augmenting LVLMs, sometimes the hallucination degree of the model is even exacerbated. 
Motivated by the research gap and counter-intuitive phenomenon, 
we introduce a novel framework, the Active Retrieval-Augmented large vision-language model (ARA), specifically designed to address hallucinations by incorporating three critical dimensions: 
(i) dissecting the retrieval targets based on the inherent hierarchical structures of images. (ii) pinpointing the most effective retrieval methods and filtering out the reliable retrieval results. (iii) timing the retrieval process to coincide with episodes of low certainty, while circumventing unnecessary retrieval during periods of high certainty. To assess the capability of our proposed ARA model in reducing hallucination, we employ three widely used LVLM models (LLaVA-1.5, Qwen-VL, and mPLUG-Owl2) across four benchmarks. 
Our empirical observations suggest that by utilizing fitting retrieval mechanisms and timing the retrieval judiciously, we can effectively mitigate the hallucination problem. 
We hope that this study can provide deeper insights into how to adapt the retrieval augmentation to LVLMs for reducing hallucinations with more effective retrieval and minimal retrieval occurrences.



†The first two authors contribute equally.

*Corresponding author.

































Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation
    Jianfeng Dong
    August 5, 2024
============================================================================================





§ INTRODUCTION


Recently, Large Vision-Language Models (LVLMs)  <cit.> have gained widespread application across various scenarios due to their significant capacity to generate contextually appropriate text for visual inputs. 
Benefiting from the advancements in model architecture, training methods, and data diversity, LVLMs exhibit superior performance for various tasks such as visual question answering <cit.> and image captioning <cit.>.
Despite these advancements, LVLMs still suffer from a significant challenge termed “hallucination”, whereby the models produce semantically plausible but factually inaccurate text, misaligned with the ground-truth content of the associated image  <cit.>.
This problem damages the practical employment of LVLMs, particularly in fields that require accurate content generation, such as medical <cit.> and robotics <cit.>, potentially resulting in severe consequences. 
Hence, addressing the hallucination problem is paramount for bolstering the credibility of LVLMs across real-world scenarios.

To mitigate hallucinations in LVLMs, a series of attempts have been recently proposed and can be broadly classified into two categories. 
The first class of approach retrains the LVLMs with constructed hallucination-related datasets by
supervised fine-tuning (SFT)  <cit.>, or Reinforcement Learning from Human Feedback (RLHF) <cit.>.
Although effective, these methods introduce significantly additional training costs. 
The other solutions focus on designing more robust decoding strategies <cit.>. For example, VCD  <cit.> introduces visual contrastive decoding to contrast output distributions derived from original and distorted visual inputs. 
In this way, the LVLM reduces the over-reliance on statistical bias and unimodal prior.
Although these methods are training-free, they still suffer from the limitations of LVLMs’ static parametric capacity. 
Recently, in the realm of large language models (LMs), augmenting LLMs <cit.> by retrieving information from external knowledge resources has shown promise in reducing language hallucinations. Furthermore, this retrieval augmentation serves as a flexible way to extend beyond the model's inherent knowledge without the burden of extensive training costs.
In this paper, we aim to propose a novel framework to augment LVLMs with external knowledge by introducing an Active Retrieval-Augmented large vision-language model (ARA) for mitigating hallucinations. As shown in Figure <ref>, given an input image and query, the LVLM tends to produce hallucinated answers. Instead, our model can accurately identify the most relevant pairs and output the correct answer.



Concretely, our proposed ARA is grounded in three critical dimensions for mitigating hallucinations in LVLMs. 
Firstly, given the inherently hierarchical nature of images, simple full-image retrieval may result in noise and irrelevant outcomes, thus it is imperative to decompose the target object causing hallucination from the input image for more accurate retrieval. 
Secondly, in contrast to augmenting LLMs, LVLMs offer diverse retrieval modalities due to their multimodal inputs. 
In this way, discerning the most effective retrieval technique and securing trustworthy results is crucial for ensuring retrieving performance. 
Lastly, excessive retrieval activations may lead to undue time expenditure and unnecessary retrieval.
By initiating retrieval only during periods of low LVLM certainty and knowledge deficiency, one can circumvent unnecessary or improper information retrieval.


Based on the above analysis, given the input image and query, our proposed ARA first devises a coarse-to-fine retrieving paradigm. 
The target object is first extracted from the input query and subsequently positioned within a particular region of the input image. To build a robust retrieving system, we simultaneously engage in coarse-grained and fine-grained retrieval process which retrieves the full image as well as the specific regions pertaining to the target object. Recognizing the plurality of retrieval strategies due to the multimodal input, an extensive analysis of different methods is conducted to determine the most optimal retrieving approach. 
Subsequent to acquiring pertinent text and image pairs from external databases, a reranking strategy is employed to eliminate unreliable outcomes, thereby augmenting LVLMs with the appropriate external information.
Particularly, retrieval operations are circumvented when the LVLM exhibits high certainty, to prevent the inclusion of superfluous retrieval that may introduce redundant information. 
In this context, retrieval is activated using a difficulty metric that relies on the mutual information between multimodal inputs.

To assess the efficacy of ARA, empirical evaluations are performed using three prevalent LVLMs across four benchmarks related to hallucination challenges.
The promising results demonstrate that our framework can effectively mitigate hallucinations. 
To sum up, our contributions are summarized as follows:


    
  * Our research thoroughly investigates the three critical dimensions of augmenting LVLMs with external knowledge, including accurately dissecting the retrieval targets, pinpointing the most effective retrieval methods, and the sensible triggering of the retrieval process. 
    
    
  * We introduce an Active Retrieval-Augmented large vision-language model (ARA) for mitigating hallucination. 
    Our empirical evidence indicates that, by employing appropriate retrieval mechanisms and carefully timing the retrievals, we can significantly address the issue of hallucination while maintaining a moderate frequency of retrieval.

    
  * Through comprehensive experiments, we demonstrate the superior performance of our proposed ARA in alleviating hallucinations using three well-known LVLMs across four hallucination-related benchmarks. The results in our paper also suggest that with optimal retrieval settings, the potential of augmenting LVLMs can be more effectively harnessed.




§ RELATED WORK




 §.§ Large Visual-Language Models


Recently, large-scale models have attracted significant attention and gained widespread applications <cit.>. 
LVLMs significantly enhance the interaction between humans and AI in a more natural and intuitive
ways and demonstrate remarkable capabilities in understanding and generating multi-modal content.
With the aid of advanced Large Language Models like LLaMA <cit.> and Qwen <cit.>, a batch of LVLMs such as LLaVA-1.5 <cit.>, Qwen-VL <cit.>, and mPLUG-Owl2 <cit.> have emerged, which can comprehend and generate a wide array of content by utilizing information from distinct modalities like texts and images. 
The training of LVLMs is divided into two critical phases: feature alignment and instruction fine-tuning. The former aims to model visual and textual inputs coherently, while the latter strives to enhance the ability of LVLMs to comprehend user queries. 
Despite the advancements, these LVLMs continue to face significant challenges with hallucination issues. Thus, in this paper, we focus on solving hallucination problems to promote the use of LVLMs in practical scenarios.



 §.§ Hallucination in LVLMs


In LVLMs, hallucination refers to models that generate seemingly plausible outputs inclusive of objects,
attributes, or relations that do not correspond with the images <cit.>.
Approaches to address hallucinations in LVLMs have largely fallen into two camps. The first emphasizes model refinement through supervised fine-tuning or application of reinforcement learning techniques. For instance, LRV <cit.> seeks to diminish hallucinatory outputs by utilizing broad and varied supervised fine-tuning datasets. 
For methods based on reinforcement learning, LLaVA-RLHF <cit.> is the pioneer in applying Reinforcement Learning with Human Feedback (RLHF) to mitigate hallucination in LVLMs. 
This approach is further refined by RLHF-V <cit.>, which incorporates detailed corrective human feedback.
Considering the instability and training difficulty of RLHF, Zhao et al. <cit.> employ Direct Preference Optimization (DPO) and build a hallucination-aware dataset for alleviating hallucination. 
Although 
While these methods have yielded notable advancements, they also introduce significant training overheads and are prone to overfitting the training data. 
The second category consists of training-free strategies designed to circumvent hallucination without incurring additional training expenses.
VCD <cit.> endeavors to rectify the model's over-reliance on linguistic priors and statistical leanings by comparing the output distributions from both unaltered and visually perturbed inputs.
OPERA <cit.> mitigates overconfidence in model predictions by integrating a penalty term into the model logits during beam search decoding, further enhanced with a strategic rollback feature. 
Although these methods are effective, they still suffer from the limitations of LVLMs’ static parametric capacity. 
In this paper, different from the above two paradigms, we focus on augmenting LVLMs with external knowledge to mitigate hallucination in LVLMs.  



 §.§ Retrieval-Augmented Generation


In the realm of large language models (LLMs), Retrieval-Augmented Generation (RAG) has been widely used and shown promising results in mitigating hallucinations. RAG in LLMs retrieves relevant information from an external knowledge base before LLMs respond to a query <cit.>, thereby enabling them to collaboratively generate responses by leveraging the retrieved external non-parameterized knowledge alongside their internal parameterized knowledge. 
However, its application in LVLMs has not been thoroughly explored. Since LVLMs present a multimodal nature, findings pertinent to LLMs cannot be indiscriminately extrapolated to LVLMs. 
Predominantly, existing research on retrieval augmentation in multimodal tasks has been confined to image captioning <cit.> or image generation <cit.>, with a narrow focus that overlooks the broader implications of RAG in LVLMs, especially concerning its capacity for hallucination reduction.
In this paper, our research conducts an extensive evaluation of RAG's applicability in LVLMs to mitigate hallucination by tailoring retrieval strategies and optimizing active retrieval processes. 
It is worth noting that the active retrieving strategy has also been discussed in <cit.> in LLM. However, they propose a token-level confidence for triggering the retrieval, which is not applicable in LVLMs and we will investigate it in the ablation study.





§ METHOD


In this section, we first describe the generation procedure of the LVLMs, laying the groundwork for comprehending our approach. Subsequently, we will introduce our Active Retrieval-Augmentation large vision-language model (ARA) in detail. As shown in Figure <ref>, given the input image and query, the LVLM first analyzes them based on mutual information and determines whether to trigger the retrieval process. If the retrieval is necessary, our ARA will follow a coarse-to-fine paradigm to dissect the retrieval targets and conduct jointly coarse-grained and fine-grained retrieval. Following this, a reranking of the retrieval outcomes is performed for additional refinement. Finally, the LVLM harnesses this externally sourced knowledge to generate the final response.





 §.§ LVLM Input and Decoding

 
The input of LVLMs contains both image and text. Typically, the LVLMs use a vision encoder (e.g. CLIP) to extract visual tokens from the input images, and then map them into the input space of LLMs with a linear projection module. 
Subsequently, the mapped visual tokens are used as part of the LLM input, along with the text input. 
In this paper, we denote the visual tokens as 𝐱^v = {x_0,x_1,…,x_N-1}. Here N is the length of the visual tokens and it is a fixed number in most cases.  
Correspondingly, the input text is tokenized and we denote it as 𝐱^p = {x_0,x_1,…,x_M-1}.
The image and text tokens are concatenated as the final input sequence and we denote it as  {x_i}_t=0^T-1 that T=N+M.


During decoding, LVLM gets the probabilities for the next token prediction, formally:

    p(x_t|x_<t) = SoftMax[LVLM(h_t)]_x_t,    x_t∈𝒱,

where h_t denotes the output hidden states at time step t for the input image and query, x_<t to simplify the sequence {x_i}_i=0^t-1 and 𝒱 means the whole vocabulary set. For greedy decoding, the token with the highest probability will be chosen.




 §.§ Active Triggering Retrieval


As shown in Figure <ref>, given the input image and query, our ARA model first determines whether the input pair requires retrieval. In this way, we can circumvent unnecessary retrieval during periods of high certainty. 
In this section, we propose three active retrieval methods to explore the difficulty metrics that influence when to retrieve, including Confidence-aware Active Retrieval, 
Question-aware Active Retrieval, and Image-aware Active Retrieval. The first method relies on the confidence of the model output, and the latter two matrices depend on the mutual information between the inputs.
An ideal metric should effectively reduce the frequency of retrievals and can easily apply to different LVLMs.

Confidence-aware Active Retrieval. We feed the image and query into LVLMs to generate the answer. If the LVLM is confident about the answer, we accept it without retrieving additional information. Specifically, we actively trigger retrieval if any token of the answer has a confidence lower than the threshold θ. In this method, θ=0 indicates retrieval will never be used and θ=1 means that all inputs will use the retrieval. This strategy is inspired by <cit.> in LLM.

Query-aware Active Retrieval. To better evaluate the difficulty of queries from a view of language bias, we evaluate the degree of LVLMs answering relying on the input image. To this end, we define a difficulty metric using mutual information based on the difference below:


    M_ij = logP(a_ij | V_i, Q_i)/P(a_ij) - logP(a_ij | Q_i)/P(a_ij) = logP(a_ij | V_i, Q_i)/P(a_ij | Q_i).


where P(a_ij | Q_i) indicates the output probability with only query input to the LVLMs and P(a_ij | V_i, Q_i) means the output probability with both query and image as model input. The higher M_ij indicates more visual reasoning is needed to answer. M_ij<0 illustrates that the answer excessively depends on the input query. Thus, we define a threshold θ and trigger retrieval if M_ij<θ, where insufficient visual information is provided for answering the query. Incorporating retrieved information will enhance the confidence of LVLMs to answer.  

Image-aware Active Retrieval. Similar to the above query-aware active retrieval, we use the probability difference between the original and noisy image to evaluate the image information to answer the query. During the implementation, we use the method in <cit.> to add noise.


    D_ij = logP(a_ij | V_i, Q_i)/P(a_ij) - logP(a_ij | V'_i, Q_i)/P(a_ij) = logP(a_ij | V_i, Q_i)/P(a_ij | V'_i, Q_i).


where P(a_ij | V_i, Q_i) indicates the output probability with query and input image, P(a_ij | V'_i, Q_i) means the output probability with query and noisy image. Analogously, we define a boundary θ and trigger retrieval if D_ij<θ. 





 §.§ Coarse-to-Fine Hierarchical Retrieval


Given the inherently hierarchical nature of images, simple full-image retrieval may result in noise and irrelevant outcomes. As shown in Figure <ref>, using the full image for retrieval can not obtain the most relevant information for reasoning, namely “red shirt". Thus it is imperative to decompose the target object causing hallucination from the input image for more accurate retrieval. 
In the following, we will first describe the coarse-grained retrieval and then introduce the fine-grained retrieve. 

Coarse-grained Retrieve. 
Initially, based on the input image, we use the CLIP to extract the visual embedding. Similarly, we build the embeddings for the images and corresponding texts in the knowledge base. Then, we can retrieve the relevant images from knowledge bases with the similarity between images. Finally, the top-K image and text pairs will be fetched for the following reasoning. 
Benefiting from the multimodal encoding ability of CLIP, we can also retrieve the top-K pairs by matching the input image with texts in the database. We will compare different retrieving methods in the ablation study.


Fine-grained Retrieve. While full-image retrieval can incorporate valuable information, it may overlook fine-grained details, such as diminutive objects and intricate attribute data. Therefore, it is crucial to propose a fine-grained retrieval mechanism that concentrates on the distinct objects within an image. 
To conduct fine-grained retrieval, we initially extract key entities from the input query utilizing the large language model LLaMA2-7B <cit.>. This extraction is facilitated by in-context learning <cit.>, where demonstrations are presented to the LLM, which subsequently outputs the specific entities from the input query. 
Following this, Grounding Dino <cit.> is deployed to identify the objects in the image that correspond to the extracted entities. As illustrated in Figure <ref>, we crop the targeted object and then perform retrieval in a manner akin to the previously described coarse-grained retrieval.





 §.§ Reranking Retrieving Results


Throughout the above retrieving process, the similarity is computed by the CLIP embeddings between images. In this way, the visually similar but semantically different pairs may be recalled. 
To avoid noisy retrieving results which bring different semantic, after obtaining the retrieved pairs for both coarse-grained and fine-grained retrieval, we subsequently propose a reranking strategy designed to recalibrate the retrieved outcomes. 
Specifically, we adopt the LVLM to describe the input image and generate a detailed caption. For coarse-grained retrieval, the input full image is characterized, while describing the cropped image for the fine-grained retrieval.
In this way, we are able to derive the semantics of the input image.

Then, we extract the textual embeddings for both image captions and captions of retrieved images with the CLIP. Finally, we can rerank the retrieved pairs according to the semantic similarity between these captions. In this stage, we also explore other reranking methods, such as k-reciprocal encoding <cit.> or using an external visual entailment tool <cit.>. We will compare them in the ablation study. 






 §.§ Joint Coarse-grained and Fine-grained Decoding


After retrieving and ranking the pairs, we can conduct a joint coarse-grained and fine-grained decoding to yield the final responses. According to the different ways of using the retrieved results, we have two schemes including probability-level fusion and instance-level fusion. The former integrates the results of coarse-grained and fine-grained retrieval into a single prompt for inference, while the latter conducts separate inferences on the coarse-grained and fine-grained retrieval results, followed by a probabilistic-level weighting.

Probability-level Fusion. In this paradigm, we use the prompt for both coarse-grained retrieval and fine-grained retrieval: “Here are the image-caption pairs similar to the test image: ⟨Retrieval Pairs⟩. Based on these pairs and this image: ⟨Input Image⟩. Answer this question: ⟨Input query⟩". 
Here we denote the retrieval-augmented LVLMs as below:


    p^coarse(x_t|x_<t) = SoftMax[LVLM([R_I:h_t])]_x_t,    x_t∈𝒱,


where R_I indicates the hidden states of retrieved image-level pairs. For simplicity, we directly use [:] here. We use p^coarse to represent decoding with coarse-grained retrieval. Similarly, 


    p^fine(x_t|x_<t) = SoftMax[LVLM([R_O:h_t])]_x_t,    x_t∈𝒱,



where R_O indicates the retrieved fine-grained object-level pairs. We represent the decoding with object-level fine retrieval by p^fine(x_t|x_<t). For the probability-level fusion, we can combine them with a hyperparameter α in [0,1] to control the importance of two kinds of retrieval. 


    p^coarse&fine(x_t|x_<t) = α p^coarse(x_t|x_<t) + (1-α)p^fine(x_t|x_<t),



Instance-level Fusion. 
For this kind of fusion, we can integrate both coarse-grained retrieval and fine-grained retrieval with prompts: “Here are the image-caption pairs similar to the test image: ⟨Coarse-grained Retrieval Pairs⟩. Here are the image-caption pairs: ⟨Fine-grained Retrieval Pairs⟩ similar to the ⟨entity⟩ in the input image. Based on these pairs and this input image: ⟨Input Image⟩. Answer this question: ⟨Input query⟩." Here we extract the entity from the input query and fill it in the prompt. 
 

    p^coarse+fine(x_t|x_<t) = SoftMax[LVLM([R_I:R_O:h_t])]_x_t,    x_t∈𝒱,







§ EXPERIMENT


This section will elaborate on the evaluation of our proposed Active Retrieval-Augmented large vision-language model (ARA) across different LVLMs.



 §.§ Evaluation Metrics


POPE <cit.>, the Polling-based Object Probing Evaluation proposed a simplified method for evaluating object hallucinations. In this evaluation framework, the LVLM is tasked with determining the presence of specific objects in a given image, with a balanced distribution of queries for object presence and absence. The evaluation protocol encompasses three distinct sampling settings: random, popular, and adversarial, each employing unique strategies for constructing negative samples. In the random setting, objects not present in the image are chosen randomly. The popular setting selects missing objects from a high-frequency pool, whereas the adversarial setting prioritizes coexisting objects that could be easily mistaken for existing objects in the image. 
The POPE benchmark aggregates data from three diverse sources: MSCOCO <cit.>, A-OKVQA <cit.>, and GQA <cit.>. Within each dataset, every sampling setting comprises 500 images, with 6 questions posed for each image, resulting in a total of 27,000 question-answer pairs derived from the development sets of these datasets. The evaluation predominantly assesses metrics such as accuracy, precision, recall, and F1 score.

MME  <cit.> is a comprehensive benchmark designed specifically for evaluating LVLMs on multiple dimensions. It includes ten sub-tasks related to perception and four sub-tasks emphasizing cognition. According to previous studies <cit.>, we utilize the existence and count subsets for object-level hallucination evaluation, and the position and color subsets for attribute-level hallucination evaluation. Performance is quantified by accuracy and accuracy+, which is also the official evaluation method.


MMStar  <cit.>, foundational to vision-reliant multi-modal evaluation, consists of 1,500 challenge samples meticulously chosen by human experts. It is tailored to systematically benchmark six key capabilities alongside eighteen granular facets, aiming to rigorously assess the multi-modal proficiency of LVLMs using a selection of samples that have been carefully refined and equilibrated.


MMbench  <cit.> is collected from multiple sources, including public datasets and the Internet, and currently, contains 2974 multiple-choice questions, covering 20 ability dimensions. Similar to MME, we focus on hallucination-related ability. Thus, we select 4 ability subsets including object localization, attribute recognition, spatial relationship, and action recognition as our evaluation benchmarks. These subsets form a comprehensive assessment of hallucinations, addressing issues at the object-level, attribute-level, and relation-level comprehensively.





 §.§ Implementation Details
 

In the case of coarse-grained retrieval, we use the COCO dataset <cit.> as the retrieval database. For fine-grained retrieval, we use the VisualGenome <cit.> to build the fine-grained retrieval database as it provides a detailed caption for each grounding box in the image. CLIP-Large is used to extract image embeddings for retrieval.   
To decompose target objects from the input query, we extract key entities from the input query utilizing the large language model LLaMA2-7B <cit.>. 
Grounding Dino <cit.> is deployed to identify the objects in the image that correspond to the extracted entities. If the objects can not be located in the image, we only use coarse-grained retrieval for the following reasoning process.
In this paper, we implement our ARA model on three LVLMs, including LLaVA 1.5, Qwen-VL, and mPLUG-Owl2 to evaluate the effectiveness of our method. For LLaVA 1.5 and mPLUG-Owl2, we use the retrieved texts to augment, while we apply the retrieved image and text pairs to boost Qwen-VL. For these three LVLMs, we implement augmentation using 3, 4, and 5 retrieval components respectively. For active retrieval, we use the query-aware active retrieval. For the hyperparameter α presented in Equation 6, we apply a value of 0.8 for the POPE, MMstar, and the object-level subsets within the MME benchmark, as well as for OL, SR, and ACR in the MMbench. A value of 0.4 is utilized for the remaining evaluation subsets. For active triggering retrieval, we use query-aware active retrieval for its simplicity and effectiveness. The threshold of active triggering is determined by grid search. 

In our experiments, to have a strict comparison of our RAR and vanilla LVLMs, we consistently utilize greedy decoding instead of other probability sampling methods. With this strategy, the token with the highest probability from the post-softmax distribution is directly selected as the next token. 



 §.§ Experiment Results



Results on POPE. Experimental results on POPE under the random, popular, and adversarial settings are summarized in Table <ref>. This benchmark mainly focuses on the object-level hallucination. 
Specifically, the performances of our ARA consistently surpass the baseline results on all of the LVLMs. 
This suggests its pivotal role in augmenting LVLMs with retrieval, thereby reducing instances of object-level hallucination. 
In a more detailed model-specific analysis, RAR demonstrates varied effects across different LVLMs. For all three LVLMs, the F1 score elevation is predominantly driven by a recall boost (e.g., up to 10 points for mPLUG-Owl2), showcasing retrieving to external knowledge base can effectively help detect object presences.








Results on MME Hallucination Subset. 
The MME subset evaluations extend beyond POPE’s scope, encompassing both object-level and attribute-level hallucination. Results in Table <ref> show that our ARA leads to a uniform enhancement in addressing object-level hallucination for all models, except the count score of Qwen-VL. Meanwhile, our ARA demonstrates significant improvements in attribute-level Color scores, contributing to substantial overall performance gains. These improvements are attributable to our designed coarse-to-fine retrieval paradigms, which effectively focus on the target object.

Results on MMStar. In a more difficult benchmark MMStar which rigorously assesses the multi-modal proficiency, as depicted in Table <ref>, our method ARA secures a notable enhancement across all six subsets. For instance, LLaVA-1.5 realizes an average improvement of 8.8 points. Unlike POPE, which may include statistical bias, answering questions in MMStar necessitates reliance on image information. Hence, our retrieval augmentation method introducing external image and text knowledge significantly bolsters model performance.

Results on MMBench Hallucination Subset. 
This benchmark extends our hallucination evaluation to incorporate the relation level in addition to the above object and attribute levels, providing a more holistic assessment for reducing hallucinations. As evidenced in Table <ref>, benefiting from coarse-grained retrieval, our ARA manifests discernible enhancements in the object and relation-level subsets, such as OL, SR, and ACR. For instance, our ARA model surpasses the vanilla LLaVA-1.5 with 5.08 percent points on object localization (OL).  
Concurrently, the integration of fine-grained retrieval into our model facilitates superior performance in attribute-level subsets, namely attribute recognition (ATR). 












§ ABLATION STUDY


In this section, we undertake a series of ablation studies with the aim of identifying the critical elements integral to constructing a resilient and efficient retrieval-augmented large vision-language model.



 §.§ Ablation of Coarse-to-Fine Retrieval



In this section, in order to explore the effects of different factors on coarse-to-fine retrieval in a simple and effective way, we did not use the rerank method or active trigger strategy.


Which kind of retrieval is better? 
Due to the multimodal characteristics of the input, there are various retrieval methods at our disposal. As shown in Table <ref>, we attempt to use the input query and image to search for images and descriptions in the database. For this study, we only conduct coarse-grained retrieval and we use the COCO dataset as the database which contains a caption for each image. 
As a result, there are four retrieval methods available to us. 
As reflected in the results within this table, we adopt the input image as our primary retrieval modality for images in the database throughout our series of experiments.
Surprisingly, using the input query to search the database for image leads to substantially poorer outcomes, achieving only 51.6% accuracy. 
This lower performance may stem from the fact that, despite the presence of relevant keywords, the images retrieved this way tend to be surrounded by a significant degree of noise.

Additionally, we investigate the influence of different embedding models on the efficacy of image-to-image retrieval. 
In our comparison, we examine three models: CLIP-Base/Large <cit.> and DINOV2-Base <cit.>. Among them, we notice that CLIP-base performs better than DINO (86.33 vs 86.07) and this may be related to their different training methods. Moreover, when comparing CLIP models of various sizes, we observe that the larger-scale CLIP models contribute to more pronounced improvements in performance. Therefore, in the context of this study, CLIP-Large is selected as the image embedding model for retrieval purposes.







How to augment the LVLM? After retrieving, we can obtain retrieved images and paired captions. In this section, we explore how to augment the LVLM with this external information. For a fair comparison, in this study, we fix the number of retrieved pairs to 3 for both coarse-grained and fine-grained retrieval. As demonstrated in Table 6, when using coarse-grained retrieval, we observe that the image and caption pair perform worse than the single caption (86.93 vs 87.43). 
However, the opposite results are observed in fine-grained retrieval and the Fine (I+T) achieves 86.53 compared with 86.20 of Fine (T). These results imply that
retrieval augmentation is sensitive to specific configurations. In this experiment, we do not carefully compare with Coarse (I) or Fine (I) as we have found that the LVLMs we use in our experiments are still quite weak in multi-image reasoning.

In Section 3.5, we propose a joint coarse-grained and fine-grained decoding, designed to optimally utilize the two retrieval mechanisms. 
In Table <ref>, we also investigate the effectiveness of probabilistic-level fusion denoted as Coarse & Fine and instance-level fusion labeled as Coarse+Fine. 
As shown from the results, we observe that Coarse+Fine even performs than Coarse retrieval, signifying that simply integrating the coarse-grained and fine-grained retrieved results in a prompt fails to enhance performance, given the distinct granularity of the information retrieved through these two approaches. Instead, the probabilistic-level fusion Coarse & Fine achieves better accuracy than both two kinds of retrieval. 
Thus, in our paper, we use probabilistic-level fusion for the following experiments. It is worth noting that Coarse & Fine (T) performs better than Coarse & Fine (I+T) for LLaVA-1.5 but Coarse & Fine (I+T) achieves better for Qwen-VL (we will describe in Figure 3 later). This result further validates the necessity of carefully selecting retrieval configurations to obtain the most favorable experimental results.

How much external information do we need? 
With the above retrieval, we can obtain multiple image-caption pairs from the database. This research delves into the efficacy of varying retrieved pair configurations. 
Considering that different LVLMs may be sensitive to retrieval configurations, our study incorporates both (T) and (I+T) configurations for a thorough evaluation. The analysis is constrained to examining between one to five retrieved pairs due to the input token limitations inherent in contemporary LVLMs. 
As depicted in Figure <ref>, LLaVA (T) reaches peak accuracy with three retrieved texts, whereas mPLUG (T) begins to saturate when there are four retrieval texts.
Conversely, QwenVL (I+T) demonstrates improved outcomes with a greater number of retrieval pairs; however, it still lags considerably behind the performance achieved by the other two LVLM models. 












 §.§ The effectiveness of Reranking Method


To substantiate the efficacy and indispensability of reranking, we begin by providing a case study within the POPE benchmark. 
As depicted in Figure <ref>, in instances where an image portrays people in a pool, leveraging CLIP for image retrieval may inadvertently introduce irrelevant content due to its predominant emphasis on visual congruence.
For instance, it might retrieve an incongruent image captioned “fruit and a bowl of bread sit on the table." Therefore, an adept reranking methodology can proficiently filter and organize the retrieved images in a more relevant manner. In this particular scenario, our reranking strategy aligns the input image caption with those of the retrieved images. 
By assessing these similarity metrics, we are able to reorganize the pool of retrieved images into a more accurately ranked series, thereby enhancing the precision of the retrieval process.

In order to further quantify the effect of reranking and compare different reranking methods, we conduct experiments in Table <ref>. 
The result reveals that the omission of re-ranking precipitates a decline in accuracy, dropping from 87.17% to 86.93%
This suggests that re-ranking effectively mitigates noise, particularly when employing the top three re-ordered retrieval pairs. 
For comparative purposes, we executed two alternative methodologies. The k-Reciprocal approach <cit.>, a prevalent offline person re-identification method, re-ranks images using the k-Reciprocal nearest neighbor algorithm. While this method offers a swifter execution by circumventing the captioning process of input images, its efficacy slightly underperforms our strategy. Additionally, we examined the employment of the external tool VSR <cit.>, which determines the entailment between the caption of the input image and the retrieved images. As shown in Table 7, this method gains only a slight increase in accuracy (86.93 vs 86.97). This may be due to the visual entailment model not being sufficiently versatile.









 §.§ Ablation of Active Triggering Retrieval


As excessive retrieval activations may lead to undue time expenditure and unnecessary retrieval, in this paper, retrieval is activated using a difficulty metric. In this ablation study, we compare three kinds of active retrieval methods, including confidence-aware active retrieval, query-aware active retrieval, and image-aware active retrieval. As shown in Figure <ref>, we have drawn the proportion of queries that need to be retrieved to achieve peak performance. Firstly, it is quite intuitive, the fluctuations of confidence-aware active retrieval among different models are very significant. For mPLUG-Owl2, we even need to retrieve all the queries, which indicates that confidence-aware active retrieval is not a good indicator of whether to perform a retrieval. For both query-aware active retrieval and image-aware active retrieval, the LVLMs can better determine a trigger threshold. 
Considering that query-aware active retrieval is more concise and the process of adding noise is omitted, we use query-aware active retrieval in all our experiments. 
It is worth noting that the very slight decline in the model's performance after continuing to use retrieval beyond the trigger threshold is due to the fact that the model itself has a high level of confidence, but the retrieval has introduced some redundant noise information, which we cannot filter out through reranking.



 §.§ Comparison with State-of-the-Art Method



In this paper, we augment LVLMs with external knowledge to mitigate hallucinations. 
To further demonstrate the effectiveness of our method, we compare it with VCD <cit.> and the results are demonstrated in Table <ref>. VCD is a state-of-the-art training-free method that explores intra-model knowledge with visual contrastive decoding. From the results, we can notice that our method significantly surpasses VCD in all three settings. For instance, in popular settings, our method gains a 4.06 percent improvement. It is worth noting that VCD performs comparably with the vanilla LLaVA-1.5 when using greedy decoding. This phenomenon is also observed in <cit.>. Instead, our method significantly surpasses the vanilla LLaVA-1.5 model as presented in Tabe 1. This result further verifies the robust improvement of introducing external information when applying our ARA method.







 §.§ Text Generation Quality Evaluation


In this section, we evaluated the effectiveness of our method in improving text generation quality on the MS-COCO validation set[We use the official COCO evaluation package.]. As shown in Table 9, all models LLaVA, Qwen-VL, and mPLUG exhibited significant improvements after applying our method, with an average improvement of 7.7% across the three models. Notably, the improvement was most pronounced for Qwen-VL, with an average increase of approximately 14.9%.




 §.§ Qualitative Results


To qualitatively verify the effectiveness of our ARA method on downstream tasks, we presented five examples from the POPE MSCOCO dataset and MME benchmark. As shown in Figure <ref>, we present three image-caption pairs for both coarse-grained retrieval and fine-grained retrieval. In the above three examples of determining the existence of objects, fine-grained retrieval helps to increase the model's existence information about specific objects. For the last two examples, the coarse-grained retrieval contributed effectively to capturing the overall information of the images. Thus, with both kinds of retrieval, our RAR model can effectively increase the accuracy compared with vanilla LLaVA-1.5. 








§ CONCLUSION


In this paper, we propose a novel framework to augment LVLMs with external knowledge by introducing an
Active Retrieval-Augmented large vision-language model (ARA) for mitigating hallucinations. Specifically, our proposed ARA is grounded in three critical dimensions for mitigating hallucinations in LVLMs, including coarse-to-fine retrieval for dissecting the retrieval targets, pinpointing the most effective retrieval methods, and triggering the retrieval process properly. 
Our findings indicate that by utilizing fitting retrieval mechanisms and timing the retrieval judiciously, we can effectively mitigate the hallucination problem. Our extensive experiments across four benchmarks and three widely-used LVLM confirm ARA’s efficacy in reducing hallucination.

ACM-Reference-Format







 Paper Title: A Systematic Review on Long-Tailed Learning.















§ ABSTRACT 
Long-tailed data is a special type of multi-class imbalanced data with a very large amount of minority/tail classes that have a very significant combined influence. Long-tailed learning aims to build high-performance models on datasets with long-tailed distributions, which can identify all the classes with high accuracy, in particular the minority/tail classes. It is a cutting-edge research direction that has attracted a remarkable amount of research effort in the past few years. In this paper, we present a comprehensive survey of latest advances in long-tailed visual learning. We first propose a new taxonomy for long-tailed learning, which consists of eight different dimensions, including data balancing, neural architecture, feature enrichment, logits adjustment, loss function, bells and whistles,  network optimization, and post hoc processing techniques. Based on our proposed taxonomy, we present a systematic review of long-tailed learning methods, discussing their commonalities and alignable differences. We also analyze the differences between imbalance learning and long-tailed learning approaches. Finally, we discuss prospects and future directions in this field.



Long-tailed learning, Long-tailed data, Imbalance learning,  Deep learning, Deep imbalance learning.


A Systematic Review on Long-Tailed  Learning
    Chongsheng Zhang,  Senior Member, IEEE,
        George Almpanidis,
         Gaojuan Fan1,
         Binquan Deng,
         Yanbo Zhang,
         Ji Liu,
         Aouaidjia Kamel,
         Paolo Soda,
        João Gama,  Fellow, IEEE

Chongsheng Zhang, George Almpanidis, Gaojuan Fan, Binquan Deng, Yanbo Zhang and Aouaidjia Kamel are with Henan University, Henan Key Lab of Big Data Analysis and Processing, Kaifeng, China. E-mail: cszhang@ieee.org; almpanidis@ieee.org; {fangaojuan,bqdeng,zhangyanbo,kamel}@henu.edu.cn. 
 Gaojuan Fan is the corresponding author of this work.
Ji Liu is with Baidu Inc., Beijing, China. E-mail: jiliuwork@gmail.com.
Paolo Soda is with University Campus Bio-Medico di Roma, Rome, Italy.  Email: p.soda@unicampus.it.
João Gama is with the Laboratory of Artificial Intelligence and Decision Support, Faculty of Economics, University of Porto, Porto, Portugal. Email: jgama@fep.up.pt. 

    
======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================







§ INTRODUCTION


The term “Long Tail Theory” was popularized by Chris Anderson's book  “The Long Tail: Why the Future of Business Is Selling Less of More” <cit.>, in which the shift in business from advertising and marketing a small number of bestseller products to paying attention to the vast amount of niche products was observed and analyzed, since their combined sales/revenues could be as significant as that of the head (most popular) products. It is closely related to “the Pareto Principle”, also known as the “80/20 Rule”, the Zipf's law, or the “Vital Few Rule”. Generally speaking, they are like two sides of the same coin. While the Pareto Principle focuses on the heads/hits and cuts off the rest, the Long Tail Theory aims to capture the tails and emphasizes the combined importance of the tail  classes/niche items.

Long-tailed data is essentially a special type of multi-class imbalanced data with a sufficiently large number of tail (minority) classes. Moreover, the combined importance of these tail classes is very significant, although each tail class by itself only has  a small number of samples (sales). Long-tailed distributed data is a relatively common phenomenon in real-world scenarios, which often needs to be handled by artificial intelligence (AI) systems and applications, e.g. high-speed train fault diagnosis <cit.>, escalator safety monitoring <cit.>, etc. Indeed, AI should not only satisfy the head/hit applications, but also be able to cover a large number of tail cases to ensure the robustness and generalization capability of the techniques and systems.

Long-tailed learning (hereafter referred to as LTL for short) is a sub-domain of artificial intelligence/machine learning that aims to build effective models for applications/tasks having long-tailed distributed data. Its main goal is to significantly improve the recognition accuracy on the tail (rare/minority) classes or cases while maintaining the same or similar accuracy on the head (frequent/majority) classes or cases. Specifically, 1) in object recognition/classification, researchers have proposed many LTL methods <cit.> that can substantially improve the recognition/prediction accuracy on the tail classes/cases, e.g. rare species, defective industrial products. Studies in <cit.> have shown promising results by disentangling deep feature representation learning and classifier training. There are also a large number of loss reweighting approaches that endow different weights to samples of the head and tail classes to adjust the decision boundary under the long-tailed settings <cit.>. 2) in object detection, a few LTL methods <cit.> have also been designed to automatically locate rare objects or cases from images or videos, e.g. locating cracks in industry equipment or detecting the engineering vehicles near natural parks; 3) in image segmentation, researchers have also developed methods <cit.> that can identify and segment the rare objects/cases in images, such as abnormal actions, pathology areas in medical images, etc. In addition, specific data augmentation techniques for LTL have also been devised to mitigate the data scarcity problem in the tail classes <cit.>.



However, due to the rapid development of this field, keeping pace with recent advances in LTL is becoming increasingly difficult. As such, a comprehensive survey of existing methodologies in this field is urgent and beneficial to the community. This motivates us to conduct an in-depth survey of recent advances in long-tailed visual learning to gain insights into their principles and technical aspects in a systematic way. Based on the inherent learning process, we first propose a novel taxonomy that categorizes existing long-tailed visual learning approaches into eight groups, depicted in Figure <ref>, which are data balancing, neural architecture, feature enrichment, logits adjustment, loss function, bells and whistles,  network optimization, and post hoc processing approaches. Based on this new taxonomy, we present a comprehensive survey of state-of-the-art LTL approaches and discuss their ideas and characteristics. Moreover, we make comparisons between imbalance learning and long-tailed learning, in which we elucidate their connections and differences. Finally, we discuss challenges and future research opportunities in this field.

Though a few recent papers <cit.> also provide literature reviews on long-tailed learning, we distinguish our survey from them by the following differences: 1) Existing reviews often employ the conventional taxonomy that categorizes existing LTL approaches into three types, which are the data resampling and augmentation category, the loss reweighting category, and the transfer learning category. However, such a taxonomy does not facilitate a comprehensive understanding of state-of-the-art LTL methods since it can not sufficiently cover LTL's whole learning process. In this work,  we propose a taxonomy that provides a more unified perspective on LTL based on its internal learning process, where we identify four major steps with eight main categories, as shown in Figure <ref>. Using this new taxonomy, we provide a more up-to-date and insightful review of existing LTL methods, discussing their commonalities and alignable differences. 2) We make detailed comparisons between LTL and imbalance learning. 3) We include the latest advances in LTL and point out future directions in this field.

The main contributions of this work can be summarized as follows:




  * We propose a  unified taxonomy with eight dimensions to characterize and organize existing long-tailed learning methodologies.


  * We present a comprehensive, taxonomy-guided survey of state-of-the-art long-tail visual learning approaches, focusing on recent advances and trends.



  * We compare long-tailed learning with imbalance learning, elucidate their connections and  differences.


  * We summarize and analyze the results of different LTL methods in different downstream tasks, using the corresponding benchmark datasets.


  * Finally, we discuss a number of future research directions and trends that can be interesting to researchers in this field.



The remainder of this paper is organized as follows. In Section <ref>, we present the background and related survey papers in long-tailed visual learning. In Section <ref>, based on our proposed taxonomy, we present a comprehensive overview of state-of-the-art LTL methods.  In Section <ref>, we provide detailed comparisons between imbalance learning and long-tailed learning, then present a preliminary discussion on long-tailed distribution in Section <ref>.  In Section <ref>, we summarize and analyze the experimental results of different LTL methods in different downstream tasks. We discuss future directions and trends in Section <ref>, and conclude the  paper in Section <ref>. 





§ BACKGROUND AND RELATED WORK
 



 §.§ Background
 
Data imbalance has been a long-standing research issue in both machine learning and computer vision. In the past, researchers commonly used the term “imbalance” to refer to datasets with skewed class proportions <cit.>. However, in the last few years, many deep learning-based methods have emerged for overcoming the data imbalance challenges in computer vision, making this research field even more popular. For instance, <cit.> are among the first methods that adopt deep learning to learn feature representations from imbalanced image data. In the meantime, people have been increasingly using the term “long tail” in these works. Today, these two terms are often used interchangeably.

One reason for the popularity of the “long tail” term in computer vision in recent years is that the training datasets nowadays are often of large scale and present highly imbalanced distributions, and they have a large number of tail (minority) classes. Another reason is the use of deep learning for overcoming the challenges of class-imbalanced distributions and the new problems and research opportunities brought by such deep learning methods. Specifically, the built-in data-driven feature extraction mechanism in deep learning is significantly influenced by the scale of the datasets and their distributions. For image datasets with long-tailed distributions, deep learning-based methods tend to become class-biased in the feature extraction step, hence the need for more advanced  LTL approaches that can effectively extract representative features for the tail classes.

In comparison, traditional computer vision methods use hand-crafted feature extraction methods, which are often model-driven and independent of data distributions. After the feature extraction step, if the resulting features are class-imbalanced, these approaches rely on the subsequent data classification stage to counter such imbalance issues. That is, traditional methods mainly rely on the data classification algorithms/stage for tackling the imbalance problem, where many existing imbalance learning algorithms <cit.> can be straightforwardly applied. Early approaches such as <cit.> fall into this category, which modifies the kernel features of SVM to optimize the class boundary when the extracted features are class-imbalanced.

Besides computer vision, the term “long tail” has also been used in many others fields. 1) In search engines and recommender systems, the long tail issue has been studied in depth, aiming to suggest items or results that are highly relevant to infrequent queries <cit.> or specific users <cit.>. 2) In network traffic modeling,  it is widely accepted that network traffic follows a long-tailed distribution, and different distribution models such as hyper-exponential distribution <cit.> and mixed Erlang model <cit.> have been proposed to model such  long-tailed network traffic data. 





 §.§ Related Work
 

In the literature, there are a few related survey papers on long-tailed learning <cit.>. In <cit.>, the authors propose a taxonomy of three main categories, which are information augmentation, class re-balancing and module improvement.  Their first category consists of data augmentation and transfer learning techniques; the second category contains resampling, class-sensitive learning and logit adjustment methods; the third category consists of  metric learning, classifier design, decoupled training and ensemble learning approaches. In <cit.>, the authors uses a taxonomy for the training stage of LTL that consists of data augmentation, resampling, cost-sensitive loss, multiple experts and transfer learning. In <cit.>, the authors present a taxonomy of data processing, cost-sensitive weighting, decoupled learning and other methods. 

In our taxonomy,  the LTL process is organized into four major modules, containing a total of eight parts/categories. As can be observed from Figure <ref>, the first module is the (input) data pre-processing step, which aims to balance the number of samples of different classes to be used for neural network training via data balancing. The second module is the neural network design  (modeling)  step for effective representation / embedding learning. Based on the internal processing flow of a neural network, it consists of  five parts/categories, which are neural architecture, feature enrichment, logits adjustment, loss function, and bells and whistles (i.e., various training strategies, techniques and practices for model performance improvement). The third module is the (internal) network optimization step, which  aims to effectively update the huge amount of  weights in a network via optimization algorithms and computing the gradients which are the partial derivatives of the loss function with respect to the weights for guiding the update process. The last module is the post-processing (output) step, which adjusts the confidence of the trained long-tailed classifiers or refines their predictions to better fit to the scenarios of real-world applications. 

Comparing with the above surveys <cit.>, our taxonomy is significantly different. First of all, we divide the overall LTL process into four major modules based on the internal learning process of a neural network, including the (input) data pre-processing/balancing step, the neural network design (modeling) step, the  (internal) network optimization step, and the post hoc processing step,  which is more holistic and natural, and easier to understand and accept for those who are new to the LTL field. Second, our taxonomy is more comprehensive, systematic and and in-depth. 1) Previous overview papers in LTL use the term/category “loss reweighting” or “cost-sensitive weighting/learning”, which can not cover other loss functions for the general embedding learning purposes, which we will address in subsection <ref>.  2) Previous overview papers <cit.> lack a systematic discussion on logit adjustment, which is a preliminary step ahead of loss function design. In subsection <ref>, we will specifically illustrate the logits calibration and Softmax margin learning techniques under this category. 3) We categorize weights and gradients rebalancing approaches <cit.> into network optimization <cit.>, which can not be covered by taxonomies in existing survey papers. 












§ LONG-TAILED LEARNING METHODOLOGIES
 

In Figures <ref> and <ref>, we present a new taxonomy for long-tailed learning, in which we divide existing techniques into eight categories, which are 1) data balancing; 2) neural architecture; 3) feature enrichment; 4) logits adjustment; 5) loss functions; 6) bells and whistles; 7) network optimization; and 8) post hoc processing. 

Data balancing approaches utilize data resampling or data augmentation or data synthesis techniques to build class-balanced training sets for training neural network models. Neural architecture approaches design specific network architectures to improve LTL performance. Feature enrichment methods aim to augment the features of the tail samples using a memory bank or the perturbation strategy, or utilize the additional or multi-modal features extracted from pre-trained/foundation  models. Logits adjustment approaches adjust the logit values or enlarge the classification margins to improve LTL performance. Methods in loss function design either upweight the loss values of the tail-class samples or difficult samples, or devise comprehensive loss functions as optimization objectives to enhance the embedding/representation learning effectiveness. “Bells and whistles” contains various strategies and technical details for network training and performance enhancement. Network optimization mainly involves the weights and gradients updating techniques for the internal network optimization.  Finally, post hoc processing methods calibrate the confidence of a  long-tail model to fit to real-world scenarios.  In the following, we will introduce representative techniques in each category.






 §.§ Data Balancing
 

Data balancing aims to increase the volume and diversity of the training samples of the minority (tail) classes to make the training samples class-balanced. Along this line, there are three representative subcategories of approaches, which are a) data resampling, which balances the number of samples in different classes; b) data augmentation, which expands the size of the training data via transformation operations; c) data synthesis, which generates synthetic samples using more advanced techniques such as GANs, distribution estimation/transfer methods, foundation models (such as GPT-4V, DALL-E <cit.>), etc. Table <ref> summarizes representative methods in each subcategory. 






  §.§.§ Data Resampling
 
In machine learning, oversampling/upsampling the minority classes and undersampling/downsampling the majority classes are two commonly used strategies for handling class-imbalanced distributions <cit.>. In deep learning, class-wise sampling (e.g.<cit.>) and instance-wise sampling (e.g. <cit.>) are two frequently used methods for balancing the number of per-class training samples in the mini-batches. Essentially, class-wise resampling may involve both oversampling (the samples of the minority classes) and undersampling (the samples of the majority classes), while instance-wise resampling (i.e. instance-wise resampling) adopts random/uniform sampling (undersampling) to select samples from the original datasets, regardless of their class labels. In <cit.>, the authors show that class-balanced re-sampling can learn discriminative feature representations when the training samples are highly semantically related to their target labels; otherwise, uniform sampling is even better than class-balanced re-sampling.

In <cit.>, the authors first perform clustering within each class, then select samples through a  quintuplet sampling scheme that enforces both inter-cluster and inter-class margins. Such a sampling strategy can alleviate the information loss brought by random under-sampling. 

Existing approaches often treat samples of the same class equally, without taking into account the  difficulty of individual instances. Hard example mining is a commonly adopted technique in deep learning for enhancing the learning capability of the models <cit.>. To improve the recognition performance on datasets with extremely imbalanced image attribute distributions, the authors in <cit.> propose to mine both the hard positives, i.e. samples from the minority classes that are predicted to belong to other classes or with low prediction scores on the ground-truth class, and the hard-negatives, i.e. samples from other classes that are misclassified to one of the minority classes, which can help enlarge the margins between classes.





  §.§.§ Data Augmentation

Data augmentation is a commonly used technique in deep learning for artificially expanding the training set to improve the accuracy of the models while avoiding overfitting. It includes basic image transformations such as cropping and colour space transformations, and more advanced approaches such as image mixing <cit.>.

RandAugment and its variants. AutoAugment <cit.> and RandAugment <cit.> aim to find a group of transformation operations for data augmentation such that the deep model trained by a neural network can obtain better or the best accuracy on the target dataset. AutoAugment automatically searches for the best image transformation choices/policies for a dataset. RandAugment further investigates how to reduce the search space, where straightforward grid search is found to be very effective. Unlike RandAugment, CUrriculum of Data Augmentation (CUDA) <cit.> is designed to find more fine-grained class-wise augmentations, i.e., the number of candidates for sequential augmentation operations for different classes. They reveal that strong/more data augmentations should be applied to samples of the majority (easy) classes instead of the minority (difficult) classes. Therefore, minority classes are given shallow-degree sequential augmentation operations in the training process for curriculum learning. 

Mixup and its variants. Mixup <cit.> is a data augmentation technique that generates synthetic images with soft labels by performing linear interpolations in both the raw input space and label space. The soft label describes the prior inter-class relationship between the two input images. The main difference between Mixup and conventional transformation-based augmentation techniques lies in that Mixup alters both the raw input space and the label space when creating synthetic/augmented images. The idea of Manifold Mixup <cit.> is similar to Mixup <cit.>, with the only difference being that the former interpolates two images in the latent feature space. Remix <cit.> adapts Mixup to the long-tailed setting by assigning a soft label in favour of the minority class, giving a higher label mixing coefficient to the minority classes. Similarly, UniMix <cit.> extends Mixup to the long-tailed setting by adopting a tail-favoured Mixup factor and an inverse sampling strategy to encourage more occurrences of head-tail pairs. Recently, the authors in <cit.> propose BEM which is a novel data augmentation method for long-tailed semi-supervised learning. It uses a sampling strategy based on class-wise entropy to rebalance data uncertainty in data sampling, and the class activation map to achieve more accurate localization to avoid redundant areas in data mixing. <cit.>  proposes a dynamic Mixup approach for data augmentation, which determines the number of per-class images to select in the next training epoch according to the recognition performance in the previous training epochs. <cit.> proposes the Long-Tail Mixed Reconstruction (LMR) algorithm that reconstructs the few-shot videos through weighted combinations of the head-class samples/videos in the batch using Mixup, which reduces overfitting on few-shot classes.




Cutmix and its variants. Cutmix <cit.> is another image data augmentation strategy that replaces a randomly selected region in an image with a patch from another image during image mixing. The soft label of the generated image can be obtained in the same way as Mixup, but the weights to be used in the linear interpolations should be in proportion to the areas of the patches in the generated images. Based on the idea of Cutmix, the Context-Rich Minority Oversampling (CMO) method <cit.> creates synthetic minority images by simply pasting the real minority samples (as the foreground) onto the majority samples (as the background). This way, the rich contexts can be transferred from the majority samples to the synthetic minority samples, making them more diversified for training LTL models. <cit.> proposes a new augmentation method that first extracts the rich background images from the head classes using activation maps then pastes the tail class images upon them to generate tail-class images with diverse contexts. Instead of arbitrarily pairing the foreground and background images like <cit.>, OTmix <cit.> considers the semantic distances between the foreground and background images/distributions and selects the most suitable background image to generate more meaningful samples.

Other augmentation methods. ISDA <cit.> is a data augmentation method that produces diversified augmented samples by translating features along semantically meaningful directions (e.g. colour, backgrounds, visual angles). MetaSAug <cit.> extends ISDA to the long-tailed setting by learning the semantic directions via meta-learning. Major-to-minor Translation (M2m) <cit.> iteratively modifies (translates) a majority sample until it can be confidently classified as a minority sample by a pre-trained classifier built upon the imbalanced dataset.  M2m  leverages the richer information of the majority samples and does not overly use the minority samples, which can thus learn more generalisable features for the minority classes. <cit.> reveals that existing data augmentation methods have side effects since they sacrifice the performance of certain classes. They propose DODA that dynamically maintains class-wise data augmentation preference list by comparing the number of correctly predicted samples of each class in the current and previous epochs. 

Augmentation methods for long-tailed object detection. In <cit.>, the authors present a comprehensive review on the class imbalance problems in object detection. They summarize four types of imbalance problems in object detection: class imbalance, scale imbalance, spatial imbalance, and objective imbalance. <cit.> analyzes the issue of Intersection over Union (IoU) distribution imbalance in two-stage object detection and proposes the positive Region of Interest (pRoI) generator, which increases the number of positive examples, especially for those with greater IoU values, and improves the final object detection performance.

The authors in <cit.> point out that, for long-tailed object detection, image-level resampling alone is not enough to yield a sufficiently balanced distribution at the object level. They thus propose a joint resampling strategy, termed RIO (Resampling at Image-level and Object-level), which augments the minority classes in the current batch with the object-centric RoI features and box coordinates stored in memory over the past batches. MosaicOS <cit.> is another augmentation method that generates pseudo scene-centric images by stitching object-centric images (i.e. ImageNet) for long-tailed object detection. The difference between RIO and MosaicOS is that RIO uses a memory bank to reuse the object-centric RoI features and box coordinates over the past batches; in contrast, MosaicOS straightforwardly generates synthetic scene images by stitching several object-centric images into one image.







  §.§.§ Data Synthesis

While data augmentation techniques generate synthetic images based on image transformations, data synthesis approaches utilize more complex techniques for image generation, including GANs <cit.>, distribution estimation/transfer  <cit.>, pre-trained vision/vision-language models (foundation models <cit.>), etc. 

GAN based data generation. <cit.> finds that information at the lower resolutions tends to be class-independent thus can be shared by all the classes, while class-specific features are usually unveiled at higher solutions. Inspired by such observations, in GAN-based image synthesis for long-tailed data, it proposes to calculate the unconditional (class-independent) objective for low-resolution images, while injecting conditional information at higher-resolution images. 


Distribution based data synthesis. The authors in <cit.> point out that estimating the centroids of the tail classes is much noisier because they have few samples and propose DisRobuLT, which optimizes against the worst-case centroids within a safety hyper ball around the empirical centroid. Rare-class sample generator (RSG) <cit.> is a data synthesis method which obtains the feature displacement of a real frequent-class sample by subtracting its feature vector with the closest cluster centre of the same class to remove the class-relevant information and obtain the divergence value. It next adds the above divergence value of the frequent-class samples directly to the feature vector of each rare-class sample to create synthetic minority samples. The authors in <cit.> propose LEAP, which first uses the angles of the cosine similarity between each feature vector and the centre of the corresponding class centre to model a Gaussian distribution for each class, next forces the variance of the Gaussian distribution for each tail class to be the same as the averaged variance of the head classes, then randomly selects feature vectors using the new distribution of each tail class. LEAP enlarges the variance of the feature distribution of each tail class so that they become more separated from other classes. GistNet <cit.> also proposes to share the geometry of the head classes with the tail classes. However, unlike LEAP, it uses an alternative implementation of the Softmax function by adding the weight vector with a shape parameter that contains the geometry information shared by all the classes. Since the head classes tend to dominate the optimization of the shared parameters, the geometry of the head classes will be naturally transferred to the tail classes.  Overall, RSG forces the intra-class variation information of the tail classes to be the same as the head classes to generate synthetic samples for the tail classes, while DisRobuLT estimates the centroids of the tail classes using the distributional robustness theory. LEAP uses the averaged variance value of all the head classes as the variance of the tail classes to model the distribution of each tail class, whereas GistNet proposes to learn the shared geometry parameters from the head classes and then transfer them to the tail classes. 




Foundation model based data generation. With the rise and prosperity of large pre-trained models/foundation models, e.g.,  CLIP <cit.>, GPT-4V and DALL-E <cit.>, researchers  attempt using such large pre-trained models to generate synthetic data to address the long-tail problem. In <cit.>, the authors leverage Text-to-Image (T2I) model <cit.> to generate diverse images for the tail classes, then refine them by  measuring the cosine similarity between the  image and the corresponding text features extracted via CLIP <cit.>.








 §.§ Neural Architecture

Neural architecture design, i.e., devising an effective neural network structure/model  for a specified task, long with loss function design, are the two preliminary approaches to improving the overall performance of a neural network (model). Along this line, related methods can be divided into subcategories of decoupled learning framework, early-exit framework, knowledge distillation framework, dual streams inference framework, and multi-expert (ensemble) learning framework.






  §.§.§ The Decoupled Learning Framework

Decouple <cit.> is a seminal work that reveals representation learning and classifier training should be carried out separately (i.e. decoupled) for improving long-tailed recognition. Specifically, as shown in Figure <ref>, the proposed decoupled LTL framework first adopts random sampling (instance-based sampling) and the Cross-Entropy loss for training a conventional feature extraction module in the first stage. It then freezes the backbone and uses class-balanced (class-wise) sampling to retrain the classifier in the second stage. Under the same decoupling framework as  <cit.>, <cit.> proposes to train several teacher models upon batches built via instance sampling in the first stage so that features extracted by different models can be more complementary, then distils the knowledge from the teachers to a student model. <cit.> also adopts the decoupling framework to address the class imbalance issue in medical image datasets, the difference is that it utilizes contrastive learning <cit.> to learn the feature space in the first stage. In <cit.>, the authors use class-balanced sampling and conventional random sampling simultaneously to construct mini-batches of data for the feature extraction step.  They reveal that random sampling avoids the risk of overfitting on the tail classes, while class-balanced sampling prevents the training process from being dominated by head classes. 

The above methods have a clear boundary between the two decoupled learning stages. In Bilateral-Branch Network (BBN) <cit.>, the authors propose to use an annealing factor to gradually transits between the two stages, which can adjust the model to learn the global features from the original distributions first, then gradually transit to the tail data modeling and discrimination learning. In Domain Balancing (DB) <cit.>, the authors devise a soft gate to switch between a conventional feature learning module and a tail-class favoured feature learning module. For each tail-class image, the features from the two modules are integrated to improve the LTL performance. Hybrid-SC <cit.> proposes a hybrid network structure composed of a supervised contrastive learning branch to learn better feature representations from the long-tailed data and a  conventional CNN branch with a cross-entropy loss to learn the classifiers. Like BBN, it smoothly transits between the two branches via a  cumulative learning adaptor. The main difference between Hybrid-SC and BBN is that the former uses supervised contrastive learning in the first stage, which can learn better features. In <cit.>, the authors reveal that the inference of visual relationships in a scene faces the problem of the dual long-tailed distributions of the entity (object) and predicate (relation) classes. To address this problem, they propose the DT2 model, which is based on an iterative decoupled training scheme. Like BBN, DT2 first trains (warms up) the model on the original data with uniform sampling, then fixes the backbone part and fine-tunes the classifier on the (remaining) network with the proposed Alternating Class Balanced Sampling (ACBS) method to capture the interplay between the long-tailed distributions of entities and predicates using knowledge distillation.  

<cit.> points out that the performance bottleneck of the two-stage (decoupling) learning framework seems to be the classifier retraining issue in the second stage. It thus proposes DisAlign for confidence calibration to adjust the decision boundary in the second (classifier retraining) stage. In <cit.>, the authors propose the long-tailed class incremental learning problem and present a preliminary method under the two-stage LTL framework that adds a layer of learnable scaling weights to integrate the outputs of the classifier heads for class-incremental learning.




  §.§.§ The Early-Exit Framework
 
In <cit.>, the authors point out that existing approaches to long-tailed recognition reweight samples only by class size, but without considering the fact that there might be easy examples in the minority classes that get incorrectly up-weighted and hard examples in the majority classes that get erroneously down-weighted. They thus employ the early exiting framework (ELF) to early-exit easy examples in the shallow layers of the network to focus on the hard examples in each class in the deeper layers. In implementation, ELF augments a backbone network with several auxiliary classifier branches. During training, it finds and early-exits the correctly and confidently classified examples at the earliest possible exit branch, while harder examples will exit later in the network, sustaining their impact on the overall loss. Similarly, <cit.> proposes the Hardness Aware Reweighting (HAR) framework, which also adopts the early-exiting strategy in the training process. Specifically, HAR augments a backbone network with auxiliary classifier branches, and samples that are confidently and correctly classified at a branch will early exit from that branch.




  §.§.§ The Knowledge Distillation Framework

In <cit.>, the authors propose Self-Supervised Distillation for long-tailed visual recognition (SSD-LT). Adopting the decoupled learning framework, it trains two heads for knowledge distillation, which are the self-distillation head and the classification head, respectively. It thus owns the merits of rebalanced sampling and decoupled training strategy and is reported to obtain state-of-the-art performance in long-tailed recognition. The authors in <cit.> propose the “Distill the Virtual Examples” (DiVE) pipeline for long-tailed recognition. In DiVE, a teacher model is first trained for the long-tailed task with the existing methods, and the predictions from such a teacher model are used as virtual examples. It then uses knowledge distillation to transfer the knowledge from the teacher model to the student model.

<cit.> discusses the use of ViT/Transformers for long-tailed learning. It proposes DeiT-LT which distills knowledge from a CNN teacher to a ViT backbone/student using out-of-distribution images generated via CutMix and Mixup, in which it reweights the distillation loss to enhance the focus on the tail classes.  Recently, for long-tailed object detection, RichSem <cit.> adds a new branch to the detection framework for distilling semantics from CLIP, which can extract rich semantics for the bounding box patches within the image. 




  §.§.§ The Dual Stream Inference Framework
 
<cit.> designs a neural network architecture with two branches: one branch trains a baseline model on the original long-tailed distribution, while the other trains a second model upon the artificially balanced distributions. They add a cross-branch consistency loss based on their logit difference to enforce the two branches to train the multi-label visual recognition model collaboratively. For long-tailed face recognition, Center-dispersed (CD) Loss <cit.> first splits the original data into head-class data and tail-class data, then feeds it into the “unequal-training” framework that consists of a  backbone and two learning branches for the head classes and tail classes, respectively. The first learning branch is based on the head-class data and is used to stabilise face representation, supervised by a noise-resistant loss. The second learning branch is based on the tail-class data and is used to enhance the inter-class discrimination capability. The training of the framework alternates between the two branches. One difference between <cit.>  and <cit.> lies in that the latter trains two models on the split data (head-classes data and tail-classes data) separately, whereas the former trains two models on the original/complete data.

In SimCal <cit.>, the authors investigate instance segmentation over data with long-tail distributions. Upon Mask R-CNN, their proposed method makes three main modifications: i)  it changes the inputs of the original dataset with batches of class-balanced samples, then selects class-balanced object proposals for each class in the region proposal generation (RPN) stage;  ii) in the classification/segmentation stage, it trains a calibrated classifier with Mask R-CNN on the above proposals; iii) in the inference step (after the training is over), it employs the calibrated classifier and the classifier trained on the original dataset using the vanilla Mask R-CNN algorithm to predict the tail and head classes, respectively. Such a dual-head inference framework yields improved performance on the tail classes while still maintaining similar performance on the head classes.




  §.§.§ The Multi-Expert Learning Framework
 
Ensemble learning methods aim to utilize multiple experts/models to make more accurate predictions. Representative ensemble learning methods for LTL include ACE <cit.>, RIDE <cit.>,  CBD <cit.>, LFME <cit.>, BalPoE <cit.>, SHIKE <cit.>, MDCS <cit.> and LGLA <cit.>. 

The authors in <cit.> propose ACE, a multi-expert ensemble network for long-tailed recognition. It aims to achieve biased representation learning and unbiased classifier training in a single stage. The class indices are first ordered descendingly by their number of samples, the first branch (expert) trains a classifier/model with the samples of all the classes; the next branch (expert) first eliminates the current head class, then trains a new classifier with the samples of the rest classes, and so on. Therefore, the number of classes handled by different classifiers decreases monotonically, and the medium-shot or tail classes will have chances to dominate a classifier, thus mitigating the bias towards the majority classes. Overall, ACE can simultaneously improve the recognition accuracy of both the head and tail classes. RIDE <cit.> trains multiple recognition models/experts on randomly sampled subsets and uses their logits mean to make predictions. It also devises a routing algorithm to skip the inferences of certain experts. CBD <cit.> also trains multiple recognition models/experts on randomly sampled subsets, it proposes to distil the knowledge from the multiple teachers/experts into a single student model. Similarly, LFME <cit.> also propose to distil the knowledge from multiple teachers into a unified/single student model. LFME differs from CBD in that the teacher models of the LFME are trained over non-overlapped partitions of classes and samples, while CBD trains multiple teacher models over randomly sampled subsets which may have overlapped classes/samples. Compared to CBD and LFME, ACE <cit.> trains cascaded models/classifiers by iteratively removing the dominating group of classes and the corresponding samples before training the next teacher model, while RIDE  straightforwardly utilize logits mean/average instead of knowledge distillation to ensemble/ally the teacher models. 

The authors in <cit.> propose BalPoE, which is also an ensemble learning framework composed of logit-adjusted experts <cit.>. They prove that the ensemble is Fisher-consistent for minimizing the balanced error. SHIKE <cit.> is a knowledge distillation-based multi-expert ensemble learning framework for long-tailed recognition. The authors find that shallow features of a network (or a shallow network) can better represent the tail classes. They thus propose aggregating the deep features in different experts/branches with different layers of shallow features, respectively. Next, they use knowledge distillation techniques to let the experts learn from each other. Like other ensemble learning algorithms, the averaged predictions from the multi-experts will then be used as the final prediction. Nested Collaborative Learning (NCL) <cit.> also trains multiple experts on all the categories and some of the hard categories, respectively. It then uses knowledge distillation to learn the multiple experts collaboratively. MDCS <cit.> is a multi-expert learning framework that introduces a diversity loss (which is essentially the Balanced Softmax loss <cit.>) to control experts' focus on different categories with an adjustable weight to improve the diversity of experts, and performs  self-distillation between weakly and strongly augmented logits sets for each expert to distill the richer knowledge of weakly augmented instances to reduce model variance. 

Considering the fact that the class distributions of test data are not necessarily uniform but might be long-tailed or even inversely long-tailed, the authors in  <cit.> propose SADE which is a test-agnostic long-tailed recognition approach that trains three experts/models on uniform, long-tailed, and inversely long-tailed class distributions separately, then weighted aggregates them at the inference stage. DirMixE <cit.> is also an multi-expert learning approach, which uses the Monte Carlo method to estimate the mean and semi-variance of a meta-distribution for capturing both the global and local variations to adapt to test-agnostic distributions. LGLA <cit.> trains multiple experts on the entire dataset to extract more discriminative features for different subsets, which uses logit adjustment strategy to enlarge the discrepancy among the subsets. It also learns an expert excelling at inversely long-tailed distribution, next ensembles these experts to achieve superior prediction performance.

<cit.> proposes to leverage the powerful discrimination ability of the large-scale pre-trained vision models <cit.> to tackle long-tailed classification directly. It proposes LPT which first adapts the pre-trained model to the target domain using shared prompt tuning, next adopts the divide-and-conquer strategy for long-tailed recognition by using the pre-trained model and shared prompt to generate a  group-specific prompt set for each group of classes. 

Besides, lately, the Transformer architecture  <cit.> is being increasingly utilized in LTL <cit.>. Transformer based LTL methods, such as  LTGC <cit.> and VL-LTR <cit.> reported significantly better performance than CNN based methods in long-tailed recognition, while Step-wise <cit.>  which adopts the CNN-Transformer combined architecture, also showed superior performance in long-tailed object detection. 




 §.§ Feature Enrichment




Feature enrichment methods enrich/augments features of the tail classes  using fusion-based approaches <cit.>,  perturbation based approaches <cit.>, large pre-trained models/foundation models<cit.>, etc. 




  §.§.§ Fusion Based Feature Enrichment


As depicted in Figure <ref>, the Retrieval Augmented Classification (RAC) method <cit.> uses an external memory module to store features of the tail samples explicitly. Next, for each input image, it finds in the memory bank the Top-K most similar features of the same class, which are then fed into a BERT-like encoder to be transformed into logits, then fused with the logits derived from a conventional backbone. Likewise, the authors in <cit.> propose PFENet, a prior guided feature enrichment network that enriches query features with support features and prior masks to overcome the spatial inconsistency between the query and support samples in few-shot segmentation. LOCE  <cit.> also utilizes a memory module to store features of the tail-class samples for augmenting the features of the tail classes, which works collaboratively with the proposed  Equilibrium Loss (EBL) that adjusts the margins for the tail classes in long-tailed detection. In OLTR <cit.>, the authors combine the features extracted by a conventional CNN pipeline and the features stored in an additional memory module to generate features of the tail classes, which can transfer knowledge between head and tail classes, and increase the sensitivity of the model to novel classes.

<cit.> proposes to disentangle image features into  class-agnostic features (a.k.a. category-related/class-relevant) and class-generic features (a.k.a. category-independent/class-irrelevant)  for  one-shot image generation. It uses an auxiliary memory module to store class-generic features. The class-agnostic features from the input image and the class-generic features from the memory module are then fused and sent to the generator of a vanilla GAN for image generation. Likewise, in <cit.>, the authors point out that some features are shared by the head and tail classes, while other features are class-agnostic (unique to certain classes). The authors thus propose to transfer the class-generic features from the head to the tail classes. <cit.> propose to augment tail classes by grafting the diverse semantic information from head classes, where it replaces a portion of feature maps from tail classes with those belonging to head classes. Such fused features substantially enhance the diversity of tail-class representations. To prevent the neural networks from fitting the head classes first, MFW <cit.> weakens features of the head classes by mixing them with features of other classes in the same mini-batches but without altering the corresponding labels to equalize the training progress across classes.

CoSSL <cit.> devises a tail-class feature enhancement mechanism that gives a higher probability to the tail classes when blending features of unlabeled and labelled data.




  §.§.§ Perturbation Based Feature Augmentation

The authors in <cit.> propose the FASA method for long-tailed instance segmentation that generates class-wise virtual features by random perturbation over a Gaussian distribution on the real samples of each class. 

CC-SAM <cit.> proposes to improve the generalization capability of the LTL models by robust training against class-conditioned perturbations on the features of the last convolutional layer. Its idea is similar to that of LogitAdjust <cit.>. The main difference is that LogitAdjust performs perturbations over logit values in the output layer, whereas CC-SAM adds noises to the features in the last convolutional layer (the last hidden layer).



  §.§.§ Foundation Model Based Feature Enrichment


Recently, foundation models such as CLIP <cit.> and ALIGN <cit.> have demonstrated impressive representation and generalization capabilities. In <cit.>, researchers have  explored using such foundation models for feature-level enhancements. They simultaneously utilize the visual and linguistic representations extracted by the foundation models CLIP <cit.> for long-tailed recognition. For long-tailed object detection, RichSem <cit.> adds a new branch to the detection framework for distilling knowledge from  CLIP, which can extract rich visual and linguistic semantics to enhance the representations of the objects within the image.




 §.§ Logits Adjustment
 
In the output layer, logits contain the raw prediction values of the neural network, which will be used as the input to the Softmax activation function to turn the logit values into a probability distribution over the input classes. Methods that fall into the logits adjustment category either optimize the inherent process in deriving the logit values  <cit.>, or calibrate the logit values in a post hoc manner <cit.>. It also includes softmax margin learning methods <cit.> that enforce margins into the logit values so as to influence the resulting Softmax classifier to have larger classification margins <cit.>. We note that, sometimes it is challenging to clarify the differences between logits calibration and Softmax margin learning. For instance, LogitAdjust <cit.> can be considered to be belonging to both subcategories. In Table <ref>, we summarize the formulas of representative methods in this category.






  §.§.§ Logits Calibration


Inherent Logits Calibration. When deriving logits from the last hidden layer to the output layer, IR-Softmax <cit.> sets the weights for each class as their corresponding class centres in the feature space to avoid the shift between the weights and their centres. The authors in <cit.> propose Balanced MSE that straightforwardly uses the mean square error on each class to replace the corresponding logit element in imbalance regression. 

Post hoc Logits Calibration. In <cit.>, the authors argue that the obscure regions between classes in classification are caused by Softmax saturation. To alleviate this issue, they propose the Gaussian Clouded Logit adjustment (GCL) loss that perturbs the normalized logit with an additive margin that has a random disturbing parameter. In <cit.>, the authors reveal that pseudo-labels in semi-supervised learning are naturally imbalanced. They propose a Debiased Pseudo-Labeling method (DebiasPL) that deducts the probability for each class from the logit. After that, margin learning methods can still be applied. 

Like GCL <cit.>, in BLV <cit.>, category-wise variations are introduced into network predictions (logits) for long-tailed semantic segmentation, where tail classes obtain larger variations by multiplying the random value in a Gaussian distribution with a coefficient that is inversely proportional to the per-class sample size. This simple strategy improves the segmentation performance on the tail categories.

<cit.> finds that a classifier trained on long-tailed/imbalanced data produces skewed class probabilities on a solid color image, rather than uniform probabilities, it thus proposes to adjust the classifier's predicted logits on a test sample by subtracting its predicted logits on a white image, leading to improved recognition performance.

Most LTL methods train a model on long-tailed data but evaluate their performance on uniformly distributed data. To address this issue, the authors in <cit.> propose the LAbel distribution DisEntangling (LADE) method, which regularizes the logits so that they can be explicitly disentangled from the source label distribution and adapt to arbitrary target distributions.




  §.§.§ Softmax Margin Learning

Softmax margin learning algorithms alter the logit values before sending them to the Softmax classifier (Softmax activation function) to improve the discrimination power of the classifiers. Therefore, it can also be considered belonging to the “Post hoc Logits Calibration” sub-category. In label-distribution-aware margin loss (LDAM) <cit.>, when using the logits to calculate the  Softmax output,  the corresponding logit element for the ground-truth class of the input sample will subtract a class-wise margin parameter, which is reversely proportional to the quadratic root of the number of samples in the class. Equalization Loss (EQL) <cit.> removes the logit values of other tail classes in the denominator of the Softmax activation function. AutoBalance <cit.> adopts a generalized cross entropy loss with multiplicative and additive margin parameters, and uses a bi-level optimization procedure to learn these parameters on a validation set directly. The overall ideas of BALMS <cit.> and  LogitAdjust <cit.> are very similar to AutoBalance: the main difference lies in the multiplicative and additive margin parameters used in the Softmax function, as can be seen from Table <ref>. Similarly, in <cit.>, the authors propose the Category-Aware Angular Margin (C2AM) loss for long-tailed object detection, which adds a margin for every non-ground-truth class,  based on the ratio between the weight norms for the ground-truth class and every other class. In <cit.>, the authors deduce that margin-based binary classification provides a closed-form solution for the ideal margin of each object category in long-tailed object detection and propose the Effective Class-Margin (ECM) loss for optimizing margin-based classification.

In <cit.>, the authors propose a progressive margin loss (PML) for facial age classification. They propose a framework that consists of an ordinal margin learning head and a variational margin learning head specially tailored to long-tailed data. The weighted sum of the two margins is adopted to make adjusted predictions.

As a short summary, logits adjustment either intervenes the derivation process of logits, which occurs between  the last hidden layer and the output layer, or modifies the existing logit values, primarily by adding different margin parameters. Ultimately, the calibrated logits will be fed into the Softmax Activation Function, resulting in a better-performing Softmax classifier.




 §.§ Loss Function

An embedding space, also known as the latent space or latent feature space, is the new space produced by a learning algorithm where semantically similar objects are positioned close together, and dissimilar objects are placed far apart <cit.>, as illustrated in Figure <ref>. Roughly speaking, embedding (metric) learning refers to any technique or process that converts an input image into an effective feature representation. Loss function design, along with neural architectures,  are the main approaches to embedding learning and optimization. In fact, the aforementioned feature enrichment techniques, logit adjustment techniques, neural architecture,  as well as training strategies (bells and whistles), are all integral parts of the overall embedding learning process. 






  §.§.§ Loss Function For Embedding Learning
 

In this subsection, we will  introduce various loss functions for deep embedding learning in long-tailed settings.


Long-tailed  Recognition. The authors in <cit.>  propose Affinity Loss which jointly performs max-margin classification and feature space clustering for enhancing inter-class separability and reducing intra-class variations. DASO <cit.> generates unbiased pseudo-labels by blending two complementary pseudo-labels obtained via a linear classifier and the similarity computation between the feature vector of a sample and each class centre, respectively. In DLSA <cit.>, the authors propose to tackle long-tailed recognition by adjusting the label space of the samples via deep clustering. They first cluster the samples according to their likelihood in the feature space. The well-clustered samples are sent to a dedicated Cluster-aided Classifier, which exploits the cluster prior information in classification. The remaining samples are outliers and are sent to the next adjustment module to decrease the training difficulty further. SBCL <cit.> devises a triplet loss like contrastive loss  for long-tailed recognition. It initially divides the head classes into multiple sub-classes  and enforces a sample to be closer to samples of the same sub-class than other samples in the same class, while being closer to samples of a different sub-class under the same class than samples from different classes in the loss function. GLMC <cit.> is based upon the Siamese network architecture: for each pair of samples, it respectively uses the CMO <cit.> and Mixup <cit.> methods to create two synthetic images, then uses both the cosine distances between the features of the two images and the classification loss on the two images to optimize the network.  

In <cit.>, the authors define the distribution overlap coefficient to quantify the dominance severity of the head classes in the representation space and formulate the inter-class discrepancy and class-feature consistency losses to close the gap. The authors in <cit.> find that the distribution of the super-classes in imbalanced data is more balanced than the original data. They thus propose SuperDisco to discover the super-class in a hierarchy of semantic abstraction, then refine the original features of a sample using the corresponding super-class features. 

Contrastive learning  <cit.> has recently gained popularity since it can substantially improve the capability of the neural network in representation learning. The authors in <cit.> propose the K-positive Contrastive Loss (KCL) for LTL, which draws k instances from the same class to form the positive sample set for contrastive learning. Based upon KCL, targeted supervised contrastive learning (TSC) <cit.> adds a new loss term between the samples and the predefined optimal class centres to move the samples closer to the centres of their class and away from centres of other classes.  The PaCo <cit.> loss introduces learnable class-wise centers into the supervised contrastive learning process to make the probability that two samples are a true positive pair become more balanced across different classes, which is beneficial for long-tailed recognition. Model-Aware K-center (MAK) <cit.> sorts the external samples by their empirical contrastive loss expectation value, then selects the Top-N samples to perform K-means clustering and chooses the final samples. The authors in <cit.> propose  Balanced Contrastive Learning loss (BCL) that includes cluster compactness in the contrastive learning process. 

In <cit.>, the authors design a hybrid network comprising a prototypical supervised contrastive (PSC) learning module and a classifier training module. A curriculum is designed to control the weightings of these two modules during network training.  Motivated by the observation in <cit.> that model pruning negatively influences the prediction accuracy on samples of the infrequent classes, the authors in <cit.> design the  Self-Damaging Contrastive Learning (SDCLR) framework, which adds a pruned contrastive learning module to the (original/non-pruned) unsupervised contrastive learning module, to amplify the prediction differences between the pruned and non-pruned models and increase the weights of the infrequent class samples and hard samples in the overall loss. SDCLR increases both the overall accuracy and the balanceness of the learned representations.

<cit.> reveals that existing semi-supervised learning methods obtain low recall performance in the minority classes, but the corresponding precision performance is high. It thus proposes CReST for  semi-supervised LTL that iteratively samples pseudo-labelled data from the unlabeled set to expand the labelled set, then retrains the semi-supervised learning model. SimPro <cit.> is a simple probabilistic framework for long-tailed semi-supervised learning, which explicitly separates the estimation process into conditional and marginal distributions using the Expectation-Maximization algorithm.  ABC <cit.> attaches an auxiliary classifier head to the representation layer of a semi-supervised learning framework to avoid bias towards the majority classes in LTL. ACR <cit.> uses a dual-branch network to adaptively handle various class distributions of unlabeled data, in which it derives the scaling parameter using the bidirectional KL divergence between the estimated class distribution and three anchor distributions. 

Long-tailed  Regression. When quantizing the imbalanced regression target space into interval classes for imbalance regression, HCA <cit.> proposes a coarse-to-fine hierarchical classification strategy which first uses coarse classifiers to provide the range, next selects the corresponding finer classifier in the range, and achieves improved regression performance.

Long-tailed  Object Detection and Segmentation. The authors in <cit.> propose the Balanced Group Softmax method (BAGS), which groups the classes by the number of instances, then trains a separate Softmax classifier for each group to reduce the suppression by the head classes. However, finding a proper group partition number for  BAGS could be laborious and tricky. Thus, the authors in <cit.> devise the Adaptive Class Suppression Loss (ACSL), which utilizes the output confidence to prevent the classifiers of tail categories from being over-suppressed by the samples from head categories, and only the samples from visually confusing categories are kept for discriminative learning. Learning to Segment the Tail (LST) <cit.> also adopts the “divide-and-conquer” strategy for long-tailed segmentation. It divides the original data into several balanced parts and devises a knowledge distillation loss which measures the logit difference of the same class in neighbouring parts to preserve knowledge learned in previous parts. <cit.> adds an Adaptive Hierarchical Representation (AHR) loss to the traditional object detection loss for more discriminative embedding learning. It calculates both the coarse-grained classification loss for distinguishing each cluster clearly, and the fine-grained classification loss within each cluster by adopting adaptive margins. In <cit.>, the authors point out that, in contemporary deep learning-based object detectors, classification and regression are often optimized independently, which leads to inconsistent predictions with high classification scores but low localization accuracy, and vice versa. They thus propose a harmonic mutual-supervision method which devises a regression-aware classification loss to optimize the classification branch and a classification-aware regression loss to supervise the optimization of the regression branch. <cit.> proposes an unsupervised instance segmentation method that uses a pre-trained mask generation network to generate object proposals, next samples mask triplets from the proposals to capture the hierarchical relations and learn the embeddings. 

Considering the advantages of the Gumbel distribution in modeling extreme events with very low probabilities, the authors in <cit.> propose GOL that utilizes the Gumbel activation function for long-tailed object detection and segmentation. Through large-scale experiments, they demonstrate the superiority of Gumbel against Sigmoid and Softmax in detecting/segmenting rare-class objects.




  §.§.§ Loss Reweighting
 

After obtaining the loss on each class or sample, loss reweighting approaches aim at adjusting the importance of their loss values through the weighting factors. Existing approaches along this line can be divided into class-wise reweighting and instance-wise reweighting methods. In Table <ref>, we report the formulas of representative methods in this sub-category.





Class-wise Loss Reweighting. Class-wise loss reweighting methods assign the same weighting factor to instances of the same class or group, as demonstrated in Figure <ref>. CB Loss <cit.> is a class-balanced loss function for long-tailed distributions that reweights the samples by the inverse class frequency, i.e. the weight/factor of a class is inversely proportional to the number of samples in the class. AREA <cit.> proposes to re-weight the loss of each class based on the inverse Pearson correlation of the samples in the same class.  Range Loss <cit.> is also a class-wise reweighting method that consists of an intra-class loss and an inter-class loss: the former is defined as the harmonic mean of the two largest intra-class distances, while the latter is the shortest distance between any two class centres. The Learning to Reweight method (L2RW) <cit.> is a meta-learning algorithm that assigns weights to a batch of samples based on their similarities in gradient directions.  TCR <cit.> is a two-component reweighting loss function that simultaneously computes the class-wise weights using the CB Loss <cit.> and the conditional distribution between the source and target with L2RW <cit.>. 


Instance-wise Loss Reweighting. The above methods assign the same weight to all samples belonging to the same class. In contrast, instance-wise reweighting methods assign different weights to different samples based on their learning difficulty. Focal Loss <cit.> proposes to assign relatively larger weights to samples with high prediction loss, independent of their class labels. In <cit.>, the authors propose the Equalized Focal Loss (EFL) for long-tailed object detection, which modifies the original Focal loss by supplementing the focusing parameter with a new term that is the inverted ratio between the accumulated positive and negative gradients of the input sample's ground-truth class. In <cit.>, the authors propose the Population Stability Index (PSI) to measure the overall trend that an instance experiences loss increase over time. It pays more attention to difficult instances, including those from the majority classes.  In IB Loss <cit.>,  the authors propose a method that reweights each sample differently using the designed influence function to alleviate overfitting in the decision boundary. The authors in <cit.> discover that models trained on long-tailed datasets are more miscalibrated and over-confident than those trained on balanced data. They thus propose the label-aware smoothing soft label method  (MiSLAS) that assigns different weights to the head and tail classes to handle different degrees of over-confidence. 




 §.§ Bells and Whistles

In the context of long-tailed learning, “bells and whistles” refer to various techniques, training strategies and practices that go beyond the neural architectures and loss functions but are also essential for improving model performance. Common “bells and whistles” include network training and  optimization strategies and choices, such as advanced optimizer, learning rate schedulers, regularization, and commonly adopted techniques in deep learning such as data augmentation, pre-training and transfer learning, knowledge distillation, and ensemble learning for improving the performance of deep learning models. 

There are different types of optimization algorithms used in neural networks <cit.>, among them,  Stochastic Gradient Descent (SGD) is the major algorithm for optimizing the parameters (weights and biases) in neural networks for long-tailed learning. Learning rate (LR) schedulers are methods that adjust the learning rate during network training, such as linear decay, step decay (such as CosineAnnealing), time-based decay and exponential decay to optimize model performance over time. Regularization techniques in deep learning are methods used to prevent overfitting and improve the generalization ability of neural networks.  Commonly used regularization techniques include L2 regularization (Weight Decay) <cit.>, dropout, batch normalization, label smoothing (which replaces hard labels with smoothed distributions to reduce the model's confidence and encourage it to be more robust and generalize better) <cit.>, etc. 

Since data augmentation and synthesis, knowledge distillation, ensemble (multi-expert) learning methods have already been addressed in the previous subsections,  we only present a few pre-training and transfer learning techniques, and foundation models based approaches. 

Pre-training and transfer learning for long-tailed learning. SSP <cit.> introduces self-supervised pre-training into class-imbalanced learning to yield good network initialisation, which can alleviate the label bias issue in imbalanced datasets and enhance LTL performance. FBGL <cit.> uses both contrastive pretraining and normalization for improving the feature extractor and classifier for long-tailed recognition, which consists of a new balanced contrastive loss to improve  long-tailed pretraining and a generalized normalization method for the classifier (feature and weight vectors). The authors in <cit.> propose a simple yet effective step-wise learning framework based on Deformable DETR <cit.> that combines fine-tuning and knowledge transfer for long-tailed object detection. In specific, after selecting representative examplars for the head and tails classes,  it first pre-trains a model on all the classes, next learns a head-class model via fine-tuning, then transfers knowledge  from the head-class model to the final model. 

Leveraging large foundation models and multi-modal models for long-tailed learning. Lately, there has been a popular trend in utilizing the powerful feature representation capability of foundation models <cit.> and pre-trained multi-modal models <cit.>  to improve the LTL performance, which can also be considered as a “trick” for long-tailed learning. For instance, integration of textual and visual features  enhances the model's ability to discriminate between classes with fewer examples <cit.>, or enables the model to understand and reason about the semantic relationships between concepts depicted in images and described in texts <cit.>.

Finally, we note that, these strategies and techniques are often collectively used to make LTL models more effective and efficient. 




 §.§ Network Optimization

Network optimization refers to the internal mechanism for updating and optimizing the neural networks to  minimize the loss/objective function. Besides neural architecture and loss function design, optimizing the huge amount of internal learnable parameters of a neural network, mainly the weights and biases, plays a very important role in efficiently and effectively training and optimizing a deep learning model. In particular, this process involves techniques in updating the weights, and the gradients with respect to the loss function and weights.  Existing approaches in this line of research can be divided into weights rebalancing and gradients rebalancing approaches.

We clarify that, “network optimization” refers to techniques that optimize or utilize the weights and gradients of the network for internal network optimization, while “Bells and Whistles” in the above subsection refer to different external training strategies, choices, techniques and tricks for obtaining better learning performance.




  §.§.§ Weights Rebalancing

The authors in <cit.> show an obvious correlation between sampling frequency and the norm of the weight vectors, and propose to raise the weight vector of each class by multiplying the ratio between the number of samples in the most frequent class and the current class to adjust the decision boundary in long-tailed recognition. The authors in <cit.> also reveal that, for long-tailed data, a normally trained deep learning model has imbalanced norms of classifier weights, with the majority classes having dominating weights. They propose to balance network weights using standard regularization techniques such as weight decay (L2 regularization) and notably improve the long-tailed recognition performance. <cit.> discusses why weight balancing <cit.> is effective in two-stage long-tailed learning. It reveals that  weight decay helps achieve better feature extraction, while  class-balanced loss enables implicit logit adjustment in the training. Curvature-Balanced  Regularizer (CR) <cit.> uses the sum of the logarithm of the inverse of the maximum normalized Gaussian curvature mean of each class as a regularizer to enable the model to learn curvature-balanced manifolds. 




  §.§.§ Gradients Rebalancing

<cit.> reveals that long-tailed data suffers from gradient distortion problem in which the overall gradient is shifted toward the head gradient,  it thus proposes a two-phase long-tailed recognition method which first updates model parameters  using the gradient on  the head classes only, next grows the classifier on the added tail classes. The authors in <cit.> propose BSGAL which is a gradient-based generated data contribution estimation method for long-tailed instance segmentation that calculates the loss and gradient of the model on a batch of data with and without augmentation to actively accept or reject the generated data. SeeSaw Loss <cit.> dynamically rebalances the gradients of positive and negative samples for each category with two complementary factors to reduce punishments on tail categories and increase the penalty of misclassified instances. For long-tailed class-incremental learning, <cit.> proposes to separately reweight the gradients for the new tasks and learned tasks, based on the accumulated magnitude of gradients and the volume of the lost training data of each class, respectively. 

Overall, properly tuning the above factors is crucial for the internal optimization of the neural network to achieve high model accuracy and prevent overfitting in long-tailed learning.





 §.§ Post Hoc Processing



As demonstrated in Figure <ref>, post hoc processing methods modify the outputs of the (conventionally) trained models to tackle the imbalance problem at the inference stage, instead of improving the feature representation capability or adjusting the decision boundary at the model training process. The expected calibration error (ECE) <cit.> is often used to measure how well the predicted probabilities match the ground-truth probabilities, which first partitions the confidence scores into several equal-size bins,  then computes the difference between the accuracy and the averaged confidence in each bin.




  §.§.§ Post Hoc Confidence Calibration

Temperature Scaling (TS) <cit.> uses a single scalar parameter T to rescale the logits of a trained neural network, which will then be transformed into calibrated confidence scores using the Softmax activation function. However, this approach has yet to be validated in long-tailed settings. 

For confidence calibration in long-tailed object detection, NORCAL <cit.>  multiplies the prediction (output) scores with a vector of factors that are inversely proportional to the training sample size of each class. 

For confidence calibration in long-tailed object segmentation, the Pairwise Class Balance (PCB) loss <cit.> post-calibrates a sample's prediction probability on each class by multiplying the corresponding pairwise misclassification bias obtained from the confusion matrix. 




  §.§.§ Confidence Aggregation

When a neural network has multiple heads for inference, e.g. the dual streams inference framework and the multi-expert learning framework addressed in subsections <ref> and <ref>, these methods commonly use the logits mean <cit.> or the weighted aggregation of the logits <cit.>, as well as the confidence mean to make the final prediction.





§ COMPARISONS BETWEEN LONG-TAILED LEARNING AND IMBALANCE LEARNING
 

Imbalance learning is a long-standing research direction in machine learning <cit.>.  Long-tailed learning  belongs to imbalance learning, but they also have differences in the learning approaches. While conventional imbalance learning algorithms commonly deal with rational/tabular imbalanced data, long-tailed learning often handles image data, which needs the feature representation learning step as a prerequisite. That is, for image data, LTL needs deep representation learning techniques to extract features from images, in which neural architecture, feature enrichment and logit adjustment techniques are needed. 

However, for relational/tabular data, its features are its attribute  values, therefore,  features are already available before imbalance learning, thus feature representation techniques such as neural architecture, feature enrichment and logits adjustment, are not essential in imbalance learning. But we note that, in recent years,  deep imbalance learning approaches have also emerged, which use neural networks to improve the imbalance learning performance on relational/tabular data. 

In the following, we will mainly compare the data balancing and loss functions/cost-sensitive learning methods, as well as the multi-expert/ensemble learning methods adopted in both LTL and imbalance learning. 




 §.§ Data Balancing





  §.§.§ Data Resampling


In long-tailed learning, only random sampling is utilized, either at class-level or instance-level, referred to as class-wise sampling and instance-wise sampling, respectively. However, in imbalance learning, more advanced undersampling and oversampling techniques have been proposed. 

Undersampling techniques in imbalance learning. In Condensed Nearest Neighbor Rule <cit.>, for each sample from the majority set, if it can not be correctly classified by the Nearest Neighbor (NN) rule, it is kept; otherwise, it is discarded. In <cit.>, the authors propose an intuitive geometric space partition (GSE) based method for imbalance classification which iteratively uses a new hyper-plane classifier to cut the current geometric data space into two partitions, then removes the partition that only contains the majority of samples. 

Oversampling techniques in imbalance learning. ROS  <cit.> is the most straightforward oversampling technique, which  balances the data by replicating the minority class samples. The Majority Weighted Minority Oversampling Technique (MWMOTE) <cit.> first identifies the most important and hard-to-learn minority class samples, then assigns weights to them according to their Euclidean distance to the nearest majority class samples.




  §.§.§ Data Augmentation


For image data, in long-tailed learning, Mixup <cit.>, ReMix <cit.> and UniMix <cit.> create synthetic samples with interpolated feature vectors and labels out of two original samples.

For relational/tabular data, since the features are already available before imbalance learning, it is easy and common to use linear interpolation algorithms to generate synthetic minority samples between two real minority samples of the same class. SMOTE <cit.> creates synthetic minority samples by randomly selecting a new point on the line segment between a pair of neighbouring minority samples. Borderline SMOTE <cit.> extends SMOTE by emphasizing the borderline samples that have both majority and minority class points in their neighbourhood and ignoring the rest of the minority samples. ADASYN <cit.> creates synthetic minority samples according to data density, where a comparatively larger number of synthetic samples are created (using SMOTE) in regions of low density of the minority class than in higher-density regions. SMOTE-RSB <cit.> first uses SMOTE to generate synthetic instances for the minority classes, then employs the rough set theory as a cleaning (undersampling) method to select only the synthetic minority examples that belong to the lower approximation of their class.

In general, the idea of Mixup, ReMix and UniMix is similar to SMOTE  and its variants, but the former interpolates the label space as well, while SMOTE and its variants only handle samples of the same minority classes,  but without modifying the labels of the synthetic samples. 




  §.§.§ Data Synthesis


In LTL, there are distribution based data synthesis methods, such as (RSG) <cit.>, LEAP <cit.> and GistNet <cit.>, GAN based <cit.> and foundation model based data generation <cit.> methods. In comparison, in imbalance learning,  GANs are commonly utilized to generate synthetic tabular data, such as TableGAN <cit.>, CTGAN <cit.>, medGAN <cit.> and QAST <cit.>. 

TableGAN <cit.> was the first attempt to synthesize tables/relational databases using GANs. It adopts the Deep Convolutional Generative Adversarial Network (DCGAN) architecture, but adds a new neural network module called “classifier” and a corresponding loss function to measure the discrepancy between the label of a generated record and the label predicted by the classifier for this record. CTGAN <cit.> models the distribution of the values with a few Gaussian distributions (modes) for each continuous attribute. For discrete attributes, CTGAN introduces a conditional generator and training-by-sampling method to ensure that each category of a discrete attribute gets a fair chance to be included during sampling. This way, it can simultaneously generate a mix of discrete and continuous attributes. Medical Generative Adversarial Network (medGAN) <cit.> uses Auto-Encoders to learn the salient features of discrete attributes and GAN to generate realistic samples in the embedding space. To address the usability of GAN-generated samples in imbalanced learning,  QAST <cit.> designs a semantic pseudo-labelling module to control the quality of the generated features, then calibrates their corresponding semantic labels using a classifier committee consisting of multiple pre-trained shallow classifiers. The above approaches only generate synthetic examples for the minority classes. In <cit.>, the authors propose using GAN to generate a subset of the majority of samples that follow the same distribution as the original dataset. Next, they construct a few balanced subsets, each containing all the minority samples and an equal-sized subset of GAN-generated majority samples. Then, upon each decomposed two-class subset, they use deep metric/embedding learning to build a two-class classification model. These models are then integrated to make the final predictions.  Recently, a diffusion model <cit.> based imbalance learning method, namely TabDDPM  <cit.>,  has been proposed for generating rational samples.




 §.§ Loss Function




  §.§.§ Loss Function for Embedding Learning


The idea behind embedding learning is to map complicated (often high-dimensional) metric spaces into easier (often low-dimensional) metric spaces, where inter-class and intra-class distances can be optimized simultaneously. In LTL, there are many specially designed embedding learning approaches, such as <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, etc.

In imbalance learning over relational data, many traditional (non-deep learning based) embedding learning methods have been proposed to transform the learning space. IML <cit.> is an imbalance learning method that aims to find a more stable neighbourhood space for the testing data by performing metric learning on the data and training sample selection according to the testing data iteratively until the selected training samples in two adjacent iterations are the same. MLFP <cit.>  expands the decision boundaries around the positive samples and learns better metric space using a triplet-based cost function on the false negative and false positive samples. 

In recent years, many deep imbalance learning approaches have also been proposed which use neural networks to transform the metric space and learn better embedding/feature representations. Khan et al. <cit.> propose a cost-sensitive (CoSen) deep neural network method to learn feature representations for both the majority and minority classes. <cit.> addresses the problem of imbalanced data for the multi-label classification problem using deep learning. It introduces incremental minority class discrimination learning by formulating a class rectification loss regularization, which imposes an additional batch-wise class balancing on top of the cross-entropy loss. This can rectify model learning bias due to the over-representation of the majority classes. GAMO <cit.> is an end-to-end deep oversampling model for imbalance classification, which effectively integrates sample generation with classifier training. 




  §.§.§ Loss Reweighting

In LTL, loss-level reweighting methods assign different weights to the loss values of the samples, which include class-wise reweighting algorithms such as CB Loss <cit.> and L2RW <cit.>, instance-wise reweighting algorithms such as IB Loss <cit.>, as illustrated in subsection <ref>. 

In imbalance learning, there are also a few cost-sensitive learning approaches that modify the internal cost function/objective function by assigning different weights to different samples based on their difficulty. Such methods usually assign a higher cost to minority classes to boost the importance of the minority classes during the learning process <cit.>. To adapt SVM to imbalanced data, cost-sensitive SVM (CostSVM) <cit.> adds a class-specific classification penalty in its cost function. 

Overall, the general ideas for loss-level reweighting in LTL and cost-sensitive learning in imbalance learning are very similar, i.e., they all give higher weights to samples of the minority classes or the misclassified samples. 




 §.§ Multi-Expert Learning


Multi-expert learning (ensemble learning) techniques are applicable to both LTL and imbalance learning.  For instance, the idea of ACE <cit.> to train cascaded learning models for LTL is similar to that of BalanceCascade <cit.> and AdaCost <cit.> for imbalance learning. 

There are also a few differences, for instance, i) Knowledge distillation-based ensemble techniques have been adopted in LTL <cit.>, but not in imbalance learning; ii) Dichotomy-based ensemble technique for multi-class imbalance learning <cit.> are not needed in long-tailed learning, since cross-entropy loss based LTL classifiers are naturally multi-class classifiers.




 §.§ Evaluation Metrics


In terms of evaluation metrics,  existing LTL methods commonly assume that the test sets are class-balanced, that is, all the classes have the same number of test instances in the testing/inference stage. Under such an assumption, only the Top-1 accuracy/overall accuracy metric is reported. However, we note that, in real-world applications, the test sets might also be long-tailed, or at least non-uniformly distributed.  In contrast, in imbalance learning the test sets are often considered to be imbalanced, and various evaluation metrics have been proposed <cit.>, including the overall accuracy, G-mean (the geometric mean of the per-class accuracy), Macro-F1, etc. 

In <cit.>, the authors argue that the average accuracy metric used in LTL incurs little penalty on classes with very low per-class recalls. They thus propose three new metrics for evaluating long-tailed recognition algorithms: the geometric mean and harmonic mean of the per-class recall (referred to as G-mean and H-mean for short), and the lowest per-class recall. They also propose the Geometric Mean Loss (GML), which essentially calculates the logarithm of the averaged prediction probability for samples of each class. 

As a short summary, when the test set distributions are non-uniform or skewed, major evaluation  metrics including overall accuracy, G-mean,  Macro-F1 and the lowest per-class recall, should  be adopted in LTL and imbalance learning to more comprehensively assess the performance of the proposed algorithms.





§ LONG-TAILED DISTRIBUTIONS: A DISCUSSION
 




 §.§ Comparing Long-tailed Data with Imbalanced  Data


When referring to long-tailed data, one may neglect the fact that long-tailed data is a special type of imbalanced data, and long-tailed learning (LTL)  is essentially part of imbalance learning (IL). 

The long tail phenomenon occurs in imbalanced data when a small number of classes are highly frequent and are thus easy to model. In the meantime, there are also a substantial number of tail classes that have insufficient samples and are thus difficult to model. Moreover, the combined quantity or influence of all the tail classes is comparable to that of the head classes. Specifically, there are three main differences between long-tailed distributed data and imbalanced data:

First, in long-tailed data, there should be a significantly large number of minority (tail) classes compared to the majority (head) classes. It thus has a very long tail, also named “fat tail” or “heavy tail”. However, in the case of imbalanced data, the number of classes may be as few as two (two-class imbalanced data) or only a few (multi-class imbalanced data). Therefore, long-tailed data must be class-imbalanced data, but class-imbalanced data is not necessarily long-tailed data.
  
Second, since long-tailed data has a very long tail, the overall trend/pattern of event frequency/sample size per class should decrease exponentially from the head classes to the tail classes, but the sub-pattern/trend among the tail classes should be a relatively flat curve with a slowly decreasing pattern. Ordinary class-imbalanced data does not necessarily have this property.
  
Third, the combined quantity or influence of the tail class samples in long-tailed data should be significant, e.g. 60% or 20% of the total quantity/influence of the head classes. This unique property does not hold in ordinary class-imbalanced data.




 §.§ Comparing Long-tailed Distribution with Pareto Distribution


A Pareto distribution is a power-law probability distribution with heavy, slowly decaying tails. In comparison, a “long-tailed distribution”  essentially corresponds to a set of qualitative properties/constraints added to a class-imbalanced distribution rather than being a statistical distribution that can be rigorously modelled. In the following, we empirically study the differences between a Pareto distribution and a long-tailed distribution.




In Figure <ref>, we first depict the empirical rank-size distributions of two real-world datasets, iNaturalist 2018 and LVIS, typically used in long-tailed learning settings. A rank-size distribution gives the empirical relative frequencies  of the data with the classes ordered in descending order in size across the x-axis[When calculating the relative frequencies, for normalisation reasons, we divide the sample size/frequency of each class by the maximum frequency (the frequency of the Top-1 class), instead of the total number of samples of all the classes.], which corresponds to the empirical survival distribution function (ESDF).



    S(x)=[ 1 + ( x-μ/σ) ^ 1/γ] ^ -α , x≥μ , μ∈ℝ , σ , γ > 0, α≥ 1


For comparing a long-tailed distribution with a Pareto distribution, in Figure <ref> we fit each dataset with a Generalised Pareto Type IV (DPIV) survival function (SF), using Equation <ref>, where  μ is the location, σ is the scale (which corresponds to the minimum value of x),  γ is the inequality parameter, and α is the tail index (shape). Robust estimators for the parameters of DPIV have been proposed in the literature. Here, after fixing parameters μ and σ in relation to the empirical distribution of each dataset, we exhaustively vary the values of α and γ with a step of s=0.1 to heuristically find the set of parameter values that minimises the difference between the ESDF and the fitting Pareto survival function, measured in terms of the relative errors/distances between them, which can be observed from the bottom sub-figures of Figure <ref>.

As the Pareto distribution (DP) embodies a heavy (fat) tail, it is natural to present the survival rates of both the theoretical and the empirical distribution on a logarithmic scale. In Figure <ref>, on the two real-world long-tailed datasets iNaturalist and LVIS, we see that the derived Pareto distribution is not guaranteed to consistently fit to all segments of the ESDF of the real long-tailed data.

In summary,  unlike the Pareto distribution, a long-tailed distribution is not a specific statistical distribution that can be precisely modelled, but rather an imbalanced distribution with a set of properties/constraints.




 §.§ When Using the Pareto Distribution to Model the Long Tail Phenomena



In case a Pareto distribution is used to approximately model the long tail phenomena, we need to add the following constraint to the Pareto distribution:

In long-tailed data, the combined quantity/significance of the tail class samples should exceed a user-specified ratio compared to the head classes, e.g. 60%, or 50%, or 20%.

The Cumulative Distribution Function (CDF) of the Pareto Distribution is given by:


    F(x)  = 1 - (x_m/x)^α


where x_m is the scale parameter, e.g. x_m = 10 or 3, which can be considered as the minimum number of samples that a tail class has.  α is the shape parameter, which is essentially the power law slope. 

Based on the Pareto Distribution, the constraint added by the long-tailed data depicted above can be formulated as: 


    F̅(h_l)= P(X ≥ h_l) = (x_m/h_l)^α≤ 1 - r


where h_l is the lower bound frequency of the head classes, i.e. the number of samples of the least-ranked head class, and r is a user-specified threshold which is the ratio between the combined quantity/influence of the tail classes and the total quantity/influence of all the classes.  Different r values can be set in different circumstances; for instance, r can be 60%, 50%, or 20%.  Essentially, for modeling long-tailed data using the Pareto distribution, Equation <ref> adds to the Pareto distribution a joint constraint on the parameters x_m, α, h_l and r.





§ PERFORMANCE ANALYSIS
 




 §.§ Datasets



For long-tailed recognition, benchmark datasets include CIFAR-100-LT <cit.>, ImageNet-LT <cit.>, Places-LT <cit.>, and iNaturalist 2018 <cit.>. For long-tailed object detection and instance segmentation, LVIS <cit.> is the primary real-world dataset. LVIS v0.5 consists of 1,230 categories, whereas LVIS v1.0 contains 1,203 categories, with 100K images and 19.8K images for the training and test sets, respectively. 




 §.§ Evaluation Metrics

For long-tailed recognition tasks, since existing LTL methods all assume that the test set is uniformly distributed, i.e.,  the sample size for each class in the test set is equal, therefore only the overall accuracy metric is needed for performance evaluation. Besides, these methods often separately report the specific performance on three super groups, which are “Many-shot” (if a class has > 100 instances), “Medium-shot” (20-100 instances) and “Few-shot” (< 20 instances).

For long-tailed detection and segmentation tasks,  the main evaluation metric is Average Precision (AP). For long-tailed segmentation, separate performance on three super groups are also reported, which are AP_f for frequent classes (having > 100 instances), AP_c for  common class (having 11-100 instances), and AP_r for rare classes (having < 11 instances).





 §.§ Performance Comparison





  §.§.§ Long-tailed Recognition


Table <ref> summarizes the performance of representative long-tailed classification algorithms on four benchmark datasets CIFAR-100-LT (with two different imbalance ratios (IR) which are 50 and 100), ImageNet-LT,  Places-LT, and iNaturalist 2018. It can be observed that, under the CNN backbone (ResNet), MDCS <cit.>, LGLA <cit.> and SHIKE <cit.> achieve the best overall performance on all the four benchmark datasets. Besides, NCL <cit.>, SADE <cit.>,  RIDE <cit.> and ACE <cit.> also obtain remarkable performance, all of which adopt the multi-expert learning framework, owing to its effectiveness. While under the Transformer backbone <cit.>, due to the utilization of foundation models, LTGC <cit.> and VL-LTR <cit.> achieve significantly better performance than CNN based methods, in particular, their performance improvements over MDCS <cit.> and LGLA <cit.> on ImageNet-LT  are near 20% in terms of overall accuracy, which is very substantial. 

Besides, we see that the performance of loss reweighting methods tend to saturate, since almost all the possible forms have been exhaustively examined, as can be observed from Table <ref>. 

In Table <ref>, we report the experimental settings of representative long-tailed recognition algorithms on the iNaturalist 2018 dataset. 

Overall, on ImageNet-LT and iNaturalist 2018, the state-of-the-art performance is only 82.5% in terms of overall accuracy, while the best overall performance is only 54.1% on Places-LT, which can hardly satisfy the requirements of real-world applications. Therefore, there is still a large room for performance improvement.




  §.§.§ Long-tailed Object Detection and Segmentation

Table <ref> reports the performance of representative long-tailed object detection and segmentation methods on the LVIS v1.0 benchmark dataset. We see that  BSGAL <cit.>, RichSem <cit.>, Step-wise <cit.>, PCB <cit.>, ECM <cit.>, C2AM <cit.> achieve the best long-tailed object detection performance, while BSGAL <cit.>,  GOL <cit.>, ECM <cit.>, PCB <cit.>, C2AM <cit.> and NORCAL <cit.> are among the best performers in long-tailed segmentation. We would like to note that, Step-wise <cit.> is implemented based upon Deformable DETR <cit.> which is essentially a CNN-Transformer combined detection framework, while the other methods are commonly based on Mask R-CNN with ResNet as the backbone. 

Overall, the state-of-the-art performance is below 35.4% in both  long-tailed object detection and segmentation tasks, and the segmentation performance on rare classes is even lower, which is less than 25.4%. Therefore, there is considerable room for future research.



















§ FUTURE DIRECTIONS
  
While numerous approaches have been proposed to address the long tail issue, many research opportunities remain.





 §.§ Federated Long-Tailed Learning

Federated learning enables multiple clients to learn a global model collaboratively without transmitting the local private data on each client to a centralized server <cit.>. The problem becomes more challenging when the training data across the clients is both heterogeneous and long-tailed. 

Fairness refers to whether a model can perform equally for diverse types of categories. Although there are a few preliminary studies <cit.>, ensuring fairness within a federated learning system in long-tailed settings still needs in-depth investigations.




 §.§ Long-Tailed Out-of-Distribution Detection



Out-of-Distribution (OOD) samples are samples that do not match the training data distribution, regardless of whether they belong to known (existing) categories or entirely new (open-set) categories. In <cit.>, the authors point that it is sometimes challenging to distinguish between the in-distribution tail class samples and the OOD samples that belong to entirely new (open-set) categories. To address this issue, <cit.> expands the in-distribution class space by introducing multiple abstention classes and adopts mixture of experts for classification. It also augments the context-limited tail classes using both the head and OOD classes via Cutmix. At test time, the sum of probabilities for the k abstention classes is used for OOD detection. In <cit.>, the authors also investigate the open-set long-tailed recognition problem, in which a long-tailed distribution also contains samples of new/unknown/open-set categories. They propose to simultaneously handle long-tailed classification, few-shot learning and open-set recognition (OOD detection) in a unified framework. 




 §.§ Active Learning for Long-Tailed Distributions


Active learning approaches iteratively select the most informative samples from underrepresented classes for annotation and training to optimize the use of limited labeled data to improve model performance over time. The authors in <cit.>  incorporate class imbalance into active learning by considering both class imbalance and labelling cost. <cit.> proposes a progressive sampling mechanism that explores active learning of OOD samples for enhancing the performance in both long-tailed classification and OOD detection. 




 §.§ Long-Tail Class Incremental Learning


To acquire novel knowledge from long-tailed class-incremental settings, <cit.> proposes to use sub-protypes to discover and store data distributions for sub-tasks with long-tailed distributions, and adopts the knowledge distillation technique to avoid catastrophic forgetting when training a new task. 




 §.§ Long-tailed  Domain Generalization


Domain generalization aims to learn from multiple training domains to generalize to previously unseen target domains. In <cit.>, the authors introduces the maximum square loss which has a linearly increasing gradient to address the long-tailed distribution issue in domain generalization.  While <cit.> focuses on single domain imbalance, <cit.> investigates  the multiple domains LTL problem which aims to learn from multi-domain imbalanced data and generalize to all domains. It proposes a loss for bounding the transferability statistics based on the contrastive loss over the positive and negative cross-domain pairs. 




 §.§ Adversarial Training Under Long-Tailed Distributions


Adversarial training is a defense strategy aimed at enhancing the robustness of machine learning models against adversarial attacks by incorporating adversarial examples into the training process. 
<cit.> investigates the adversarial vulnerability in long-tailed recognition and proposes RoBal, which includes a scale-invariant cosine classifier and a two-stage LTL framework. In <cit.>, the authors reveal that adversarial training under long-tailed distributions suffers from robust overfitting, i.e., the long-tailed model overfits to current adversarial samples. They show that simply combining Balanced Softmax Loss <cit.> and data augmentation can simultaneously alleviate robust overfitting and improve model robustness against adversarial attacks. 




 §.§ Diverse Applications with Long Tail Concerns

Long-tailed data widely exists in many real-world domains, e.g., defect identification <cit.>,  remote sensing <cit.>, nutrition ingredients analysis <cit.>, etc. While we can apply long-tailed learning techniques in these domains, such real-world applications often pose new challenges to be addressed,  which  will advance the long-tailed learning techniques in turn. 





§ CONCLUSIONS
  
In this paper, we propose a new taxonomy for long-tailed learning that divides existing techniques into eight categories, which are data balancing, neural architecture, feature enrichment, logits adjustment, loss function, bells and whistles, network optimization, and post hoc processing techniques. Based on this taxonomy, we systematically review long-tailed learning methods in each category, including the latest advances. We also examine the distinctions between imbalance learning and long-tailed learning approaches. We summarize the experimental results of representative methods, and show that there is still a very large room for further improvements. Finally, we discuss future directions in this field.

IEEEtran




 Paper Title: EUDA: An Efficient Unsupervised Domain Adaptation via Self-Supervised Vision Transformer.








Preprint


EUDA: An Efficient Unsupervised Domain Adaptation via Self-Supervised Vision Transformer
    Ali Abedi, Graduate Student Member, IEEE,
        Q. M. Jonathan Wu, Senior Member, IEEE,
        Ning Zhang, Senior Member, IEEE,
        Farhad Pourpanah, Senior Member, IEEE
A. Abedi, Q. M. J. Wu, and N. Zhang are with the Department of Electrical and Computer
Engineering, University of Windsor, Windsor, ON N9B 3P4, Canada (e-mail:
abedi3@uwindsor.ca, jwu@uwindsor.ca, ning.zhang@uwindsor.ca).
F. Pourpanah is with the Department of Electrical and
Computer Engineering, Queen’s University, Kingston, ON K7L 3N6, Canada (e-mail: f.pourpanahnavan@queensu.ca).
Manuscript
    August 5, 2024
=================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Unsupervised domain adaptation (UDA) aims to mitigate the domain shift issue, where the distribution of training (source) data differs from that of testing (target) data. 
Many models have been developed to tackle this problem, and recently vision transformers (ViTs) have shown promising results. However, the complexity and large number of trainable parameters of ViTs restrict their deployment in practical applications. This underscores the need for an efficient model that not only reduces trainable parameters but also allows for adjustable complexity based on specific needs while delivering comparable performance.
To achieve this, in this paper we introduce an Efficient Unsupervised Domain Adaptation (EUDA) framework.
EUDA employs the DINOv2, which is a self-supervised ViT, as a feature extractor followed by a simplified bottleneck of fully connected layers to refine features for enhanced domain adaptation. Additionally, EUDA employs the synergistic domain alignment loss (SDAL), which integrates cross-entropy (CE) and maximum mean discrepancy (MMD) losses, to balance adaptation by minimizing classification errors in the source domain while aligning the source and target domain distributions. The experimental results indicate the effectiveness of EUDA in producing comparable results as compared with other state-of-the-art methods in domain adaptation with significantly fewer trainable parameters, between 42% to 99.7% fewer. This showcases the ability to train the model in a resource-limited environment. The code of the model is available at: https://github.com/A-Abedi/EUDA.




Unsupervised domain adaptation, vision transformers, Max mean discrepancy, self-supervised Learning













§ INTRODUCTION

Unsupervised domain adaptation (UDA) has shown promising results in addressing the domain shift problem, where the distribution of the training data (source domain) differs from that of the test data (target domain), and knowledge transfer <cit.>. The domain shift problem can significantly degrade the performance of conventional deep neural networks (DNNs) when applied directly to the target domain <cit.>. UDA overcomes this issue by leveraging unlabeled data from the target domain to align feature distributions and enhance model adaptation <cit.>.
Additionally, traditional DNNs often rely on a large number of annotated samples, which limits their applicability in conditions where labeling is expensive, such as autonomous driving <cit.>. UDA solves this problem by transferring knowledge from previously labeled data (source domain) to unlabeled data in the target domain, thereby enhancing model performance without the need for extensive new annotations. 

Existing UDA methods can be classified into: (i) adversarial methods, which use generative adversarial networks to generate domain-invariant features <cit.>; (ii) reconstruction methods, which employ encoder-decoder structures to learn invariant features by reconstructing domains <cit.>; (iii) transformation methods, which optimize inputs through transformations to enhance adaptation <cit.>; and (iv) discrepancy methods, which align domain features through statistical metrics like maximum mean discrepancy (MMD) <cit.> and correlational alignment (CORAL) <cit.>. 
Regardless of which UDA method is chosen, the backbone of the network for feature extraction plays a crucial role. The backbone is responsible for extracting features from the input data, and the more generalized and domain-invariant these features are, the better the UDA results will be. Recently, vision transformers (ViTs) <cit.>  have revolutionized computer vision by adapting the transformer architecture, originally designed for natural language processing <cit.>, to process images as sequences of patches. This approach enables ViTs to capture global image dependencies, offering a more comprehensive understanding than the local feature extraction typical of convolutional neural networks (CNNs) <cit.>. This capability makes ViTs well-suited for complex tasks such as object recognition and scene understanding <cit.> and particularly effective in UDA <cit.>. 

The success of ViTs in large-scale pre-training also shows significant performance improvements on benchmarks, particularly in UDA, making the integration of self-supervised learning (SSL) strategies effective <cit.>. SSL allows ViTs to learn detailed representations from vast, unlabeled datasets, enhancing generalization and robustness across various visual domains <cit.>. Among these methods, DINOv2 <cit.> stands out for its ability to extract general-purpose features <cit.>. This positions DINOv2 as a powerful tool for advancing UDA, highlighting its transformative potential in modern visual computing.

Despite the promising performance of the state-of-the-art UDA models, they often follow complex frameworks. Particularly those utilizing ViT architectures as backbone, which involve a significant amount of learnable parameters. These parameters need powerful computational resources for training. This complexity not only increases the cost of deployment but also limits their applicability in environments with limited resources.
Therefore, there is a critical need for UDA models that are inherently simpler and have fewer trainable parameters. Such models would ensure more stable training and better generalization, and offer the versatility to adjust their complexity based on specific task demands and the complexity of various applications. By achieving this, we can facilitate more efficient domain adaptation across a wider range of practical settings, making advanced UDA techniques more accessible and sustainable.

In this paper, we introduce a novel Efficient Unsupervised Domain Adaptation framework, known as EUDA, that utilizes the capabilities of self-supervised ViTs to address the challenges of UDA. Specifically, EUDA integrates DINOv2 as a feature extractor with a simple yet effective bottleneck consisting of fully connected layers. 
Using the fully connected layers aims to refine the features extracted by DINOv2 for effective domain adaptation. Additionally, inspired by <cit.> EUDA uses the synergistic domain alignment loss (SDAL), which combines cross-entropy (CE) loss and MMD loss. This hybrid loss function effectively balances the adaptation by minimizing classification errors on the source domain while aligning the distributions between the source and target domains, as shown in Fig. <ref>. By leveraging the self-supervised pre-training of DINOv2 and the computational efficiency of the MMD, our approach offers a promising solution to UDA. It reduces the dependency on large, complex models and extensive computational resources, making it not only effective but also suitable for real-world applications where adaptability and efficiency are crucial. 
To the best of our knowledge, this is the first application of DINOv2 within a UDA framework, marking a significant advancement in leveraging self-supervised learning techniques for addressing domain adaptation challenges.



In summary, the main contributions of this paper are as follows:


    
  * To the best of our knowledge, we are the first to apply DINOv2, a self-supervised vision transformer, for UDA, utilizing its strength to generate robust, domain-invariant features.
    
  * We propose EUDA (Efficient Unsupervised Domain Adaptation), a novel framework that leverages self-supervised ViTs and MMD to tackle the challenges of UDA. EUDA features a flexible architecture with a simple bottleneck of fully connected layers, which require significantly fewer parameters to be tuned and can be adjusted according to the complexity of the problem. 
    
  * We perform comprehensive experiments and ablation studies to validate the effectiveness of our proposed method on multiple benchmarks.


The rest of this paper is organized as follows: Section <ref> reviews UDA, ViTs, and self-supervised learning approaches within ViTs. Section <ref> details the proposed EUDA model. The experimental results across various configurations are presented in Section <ref>. Finally, Section <ref> concludes the paper with insights and implications of our findings.



§ RELATED WORK




 §.§ Unsupervised Domain Adaptation


As stated in the introduction, the domain shift problem is the major concern in UDA. Over the years, numerous methods have been developed to tackle this issue, aiming to enable effective model performance on the target domain without relying on labeled data <cit.>. 
Adversarial methods use adversarial training to create domain-invariant features. For example, DANN <cit.> leverages a gradient reversal layer that inverts the gradient sign during training. ADDA <cit.> trains a source encoder on labeled images, and mixes source and target images to confuse the discriminator. Adversarial methods are computationally intensive, often exhibit unstable training processes, and do not always guarantee accurate feature mapping.

Transformation methods enhance domain adaptation by pre-processing input samples to optimize their condition for model training. DDA <cit.> preprocesses input data to align signal-to-noise ratios and reduce domain shifts, while TransPar <cit.> applies the lottery ticket hypothesis to identify and adjust transferable network parameters for better cross-domain generalization. These approaches are simple and effective but may not capture all domain differences.

Reconstruction methods utilize an encoder-decoder setup to harmonize features across domains by reconstructing target domain images from source domain data. MTAE <cit.> utilized a multitask autoencoder to reconstruct images from multiple domains. DSNs <cit.> enhance model performance by dividing image representations into domain-specific and shared subspaces. This improves generalization and surpasses other adaptation methods. They usually face challenges with high computational costs, training instability, and potential overfitting to the source domain.

Discrepancy methods have emerged as particularly effective for UDA. DAN <cit.> embeds task-specific layer representations into a reproducing kernel Hilbert space (RKHS) and uses MMD to explicitly match the mean embeddings of different domain distributions. WeightedMMD <cit.> introduces a weighted MMD that incorporates class-specific auxiliary weights to address class weight bias in domain adaptation. This approach optimizes feature alignment between source and target domains by considering class prior distributions. Joint adaptation networks <cit.> align the joint distributions of multiple domain-specific layers across domains using a Joint MMD criterion to improve domain adaptation by considering the combined shift in input features and output labels.



 §.§ Vision Transformer


Transformers, initially introduced by Vaswani et al. <cit.>, have demonstrated exceptional performance across various language tasks. The core of their success lies in the attention mechanism, which excels at capturing long-range dependencies. 
ViT <cit.> represents a groundbreaking approach to applying transformers in vision tasks. It treats images as sequences of fixed-size, non-overlapping patches. Unlike CNNs that depend on inductive biases such as locality and translation equivariance, ViT leverages the power of large-scale pre-training data and global context modeling. ViT offers a straightforward yet effective balance between accuracy and computational efficiency <cit.>.

In the context of UDA, ViTs have demonstrated remarkable potential. TVT <cit.> introduces a transferability adaptation module to guide the attention mechanism and a discriminative clustering technique to enhance feature diversity. CDTrans <cit.> consists of a triple-branch structure with weight-sharing and cross-attention to align features from source and target domains, alongside a two-way center-aware pseudo labeling strategy to improve label quality. WinTR <cit.> uses two classification tokens within a transformer model to learn distinct domain mappings with domain-specific classifiers. This enhances the cross-domain knowledge transfer through source-guided label refinement and single-sided feature alignment. PMTrans <cit.> combines patches from both domains using game-theoretical principles, mixup losses, and attention maps for effective domain alignment and feature learning.

While these methods have shown promising performance in solving UDA problems, they typically rely on complex architectures with extensive trainable parameters and sophisticated training regimes, including multi-branch transformers, cross-attention, adversarial training, game-theoretical principles, and mixup losses. Furthermore, these models generally require training the entire network, resulting in a substantial computational burden. As a result, achieving promising outcomes necessitates extensive training on large-scale models, limiting their practical applicability in resource-constrained environments <cit.>.



 §.§ Self-supervised Vision Transformer

SSL has revolutionized the field of computer vision by enabling models to learn effective representations from unlabeled data, eliminating the dependency on large annotated datasets. These models learn representations by performing pre-text tasks, such as rotation prediction <cit.> and image colorization <cit.>, and then apply the learned representations to downstream tasks.
In the context of ViTs, SSL has been pivotal in enhancing their ability to extract robust, domain-invariant features. Jigsaw-ViT <cit.> integrates the jigsaw puzzle-solving problem into vision transformer architectures for improving image classification. EsViT <cit.> utilizes a multi-stage Transformer architecture to reduce computational complexity and introduces a novel region-matching pre-training task.


Among recent advancements in SSL for ViTs, DINO <cit.> and DINOv2 <cit.> have notably enhanced SSL by scaling up ViTs to effectively match representations across different views of the same image. With improvements like automatic data curation and innovative loss functions, DINOv2 excels in stability and efficiency. It can learn domain-invariant features essential for image classification and other vision tasks. Its capacity to generate robust feature maps makes DINOv2 an excellent choice for a feature extractor and representation generator. Because of these properties, we use DINOv2 in this research to significantly boost performance and generalization across various domains.




§ METHODOLOGY








 §.§ Problem Formulation

UDA aims to learn a function f: 𝒳→𝒴 that performs well on an unlabeled target domain by leveraging information from a labeled source domain. Let 𝒳 and 𝒴 denote the input and label spaces, respectively, 𝒟_s = {(x_i^s, y_i^s)}_i=1^n_s indicates the source domain data, where x_i^s ∈𝒳 and y_i^s ∈𝒴 represent the i^th input-output pairs, and 𝒟_t = {x_j^t}_j=1^n_t indicates the target domain data, where x_j^t ∈𝒳 is the j^th input samples without labels, n_s and n_t are the number of samples in the source and target domains, respectively. The goal of UDA is to train a model on labeled source data 𝒟_s and unlabeled target data 𝒟_t in such a way that it performs well on the predicting target data labels y_i^t.





 §.§ Model Overview

Fig. <ref> shows an overview of our framework for addressing UDA problems. 
The model receives images from labeled source and unlabeled target domains. It then extracts features using a pre-trained self-supervised ViT, while its weights are frozen to ensure stability and efficiency. The extracted features from both domains are fed into a bottleneck composed of multiple fully connected layers to refine and condense the information.
The bottleneck's output serves a dual purpose. Firstly, it inputs into the classification head to compute the CE loss on source domain.
Secondly, it contributes to the computation of the MMD loss to minimize the distance between source and target domain distributions in RKHS to align their feature representations. The pseudocode for our training procedure of EUDA is outlined in Algorithm <ref>. In the following subsections, we present different components of our model in detail.





 §.§ Feature Extractor

In our effort to leverage the capabilities of ViTs for domain adaptation, we adopt DINOv2 <cit.> as a self-supervised pre-trained model for feature extraction. 
It utilizes self-distillation to derive insights from unlabeled data autonomously. Central to DINOv2's design is its twin-network structure, which includes a student and a teacher network. Both networks employ the same underlying architecture based on ViTs. During training, these networks process 
different augmentations of the same image. They aim to extract consistent features regardless of the input variations.

During the training phase (see Fig. <ref>), the student network's parameters are continually updated, while the teacher network's parameters are progressively updated through an exponential moving average of the student's parameters. This ensures that the teacher model remains robust and generalizable. Moreover, DINOv2 uses registers <cit.> to improve the performance and interpretability of ViTs by addressing the problem of artifacts in feature maps, commonly observed in both supervised and self-supervised ViT networks. Registers are additional tokens added to the input sequence of ViTs to absorb high-norm token computations that typically occur in low-information areas of images. 

In our study, we leverage DINOv2 as the primary feature extractor due to its robust training on a large-scale, diverse dataset through self-supervised learning. To enhance efficiency, we freeze the model's parameters. This approach reduces the computational burden and significantly decreases the number of trainable parameters. This makes our method notably more efficient compared to other UDA techniques that require extensive training. Consequently, our streamlined model is well-suited for deployment in real-world scenarios and on-edge devices, where computational resources are often limited. Fig. <ref> shows the attention map of the pre-trained DINOv2 base model without any fine-tuning. This highlights its robustness across four different domains of the office-home <cit.> dataset. This demonstrates the precision and accuracy of the features extracted by DINOv2, underscoring the effectiveness of the EUDA feature extraction approach.

Using DINOv2 as the feature extractor significantly enhances our model by employing its robust, self-supervised pre-training to extract general-purpose features from images. The pre-trained DINOv2, with its weights frozen, ensures the extraction of high-quality features and reduces the number of trainable parameters. It can greatly simplify integration and adaptation in resource-limited settings, providing a strong foundation for effective domain adaptation.





 §.§ Bottleneck


The bottleneck component in our architecture consists of a series of fully connected layers. At the core of our model, the bottleneck output serves two primary functions. Firstly, it feeds into a classification head composed of a simple linear layer. This setup utilizes minimal computational resources to classify images efficiently. Secondly, the output acts as the feature vector for calculating the MMD loss, which aims to minimize the distance between source and target feature vectors. This facilitates effective domain adaptation.

The design of this dual-function bottleneck simplifies our model architecture and enhances its generalization capabilities across various domains. Its straightforward structure helps prevent overfitting, a frequent challenge in domain-specific applications, ensuring robustness for real-world deployments. Additionally, the minimalistic approach in the bottleneck design allows for easier maintenance and adaptability, which is crucial for meeting the dynamic requirements of domain adaptation tasks.

The bottleneck follows a simple yet efficient design for flexible adjustments in the model’s complexity. This flexibility allows us to adjust the model to meet specific performance needs or computational limits, which is particularly useful in settings with restricted resources.



 §.§ Synergistic Domain Alignment Loss

The SDAL is a composite loss function designed for the UDA models. It combines the strengths of CE loss and MMD loss. Formally defined as:

    λℒ_CE + (1-λ) ℒ_MMD,

where ℒ_CE and ℒ_MMD are the CE and MMD losses, respectively, and λ is a tunable hyperparameter that balances the influence of each loss component, it allows for a flexible adjustment according to specific domain adaptation needs. 

CE loss is a widely used loss function in classification tasks.  
The CE loss increases as the predicted probability diverges from the actual label, thus providing a robust metric for optimizing classification models. The mathematical formulation of CE loss for a multi-class classification task is given by:

    ℒ_CE = -1/N∑_i=1^N ∑_c=1^C y_i,clog(ŷ_i,c),

where N is the number of samples, C is the number of classes, y_i,c is a one-hot vector indicating whether class label c is the correct classification for the i^th sample, and ŷ_i,c is the predicted probability of the i^th sample belonging to class c. This formula penalizes the deviation of each predicted class probability from the actual class labels.


MMD loss is defined as the squared distance between the mean embeddings of features from the two domains in an RKHS <cit.>. The MMD loss can be written as:

    ℒ_MMD = MMD^2(𝒟_s, 𝒟_t) = 
    1/n_s∑_i=1^n_sϕ(x_i^s) - 1/n_t∑_j=1^n_tϕ(x_j^t) ^2_ℋ,

where n_s and n_t denote the number of samples in the source and target domains, respectively, x_i^s and x_j^t are the data samples from these domains, and ϕ: 𝒳→ℋ is a feature mapping function that projects the data into RKHS ℋ. By minimizing the MMD loss, we aim to align the statistical properties of the source and target domains.






The loss defined in Eq. (<ref>) leverages the labeled data in the source domain to fine-tune the classification performance via CE loss while aligning the distribution of the source and target domain features through MMD. 
The synergy between these two loss components enhances the model's ability to perform effectively across different domains. This makes SDAL suitable for real-world applications where domain shift is a significant challenge.
Additionally, minimizing the distance between the source and target domain distributions ensures that the feature representations from both domains are closely aligned, facilitating smoother and more reliable domain adaptation. Empirical studies have shown that MMD effectively reduces domain shift and improves model performance across various tasks <cit.>.



§ EXPERIMENTAL STUDIES





 §.§ Datasets

We examine the effectiveness of our proposed method across three benchmarks. These datasets include: 
Office-31 <cit.>, which consists of 4,652 images from three domains: Amazon, Webcam, and DSLR, across 31 categories; 
Office-Home <cit.>, which consists of 15,500 images spread over four domains: Art, Clipart, Products, and Real World, within 65 categories;
VisDA-2017 <cit.>, which includes 280,000 images from four domains: Caltech-256, ImageNet, ILSVRC2012, and PASCAL VOC 2012 as the source domain and real images as the target, belonging to 12 categories; and DomainNet <cit.>, which consists of 48,129 images from six domains: Clipart, Real, Sketch, Infograph, Painting, and Quickdraw, across 345 categories. 











 §.§ Evaluation

To evaluate the effectiveness of our UDA model, following the procedure in <cit.>, We train our model by alternately using each domain as the source with labeled data and another as the target with unlabeled data, then evaluate using labeled data from the target domain. This procedure is repeated until all domains have been used as the source domain. This setup ensures that the model's ability to generalize to new, unseen environments is properly tested. The primary metric for evaluation is accuracy, specifically how accurately the model classifies new samples when trained on the source domain and tested on the target domain. This metric provides a clear measure of the model's performance in bridging the gap between disparate data distributions of the source and target domains.



 §.§ Baselines

We compare our model against a broad spectrum of state-of-the-art models. We categorize these models into ResNet- and ViT-based models. 
The ResNet-based models include RevGrad <cit.>, CDAN <cit.>, TADA <cit.>, SHOT <cit.>, JAN <cit.>, BNM <cit.>, MCD <cit.>, SWD <cit.>, and DTA <cit.>. While ViT-based models include CDTrans <cit.>, PMTrans <cit.>, and TVT <cit.>.



 §.§ Implementation Details

In our domain adaptation model, we designed the bottleneck component with varying complexities to assess how architectural depth influences feature processing. Our configurations ranged from a simple single-layer network with 256 neurons (designated S for Small) to more complex setups like 2048-1024-512-256 (B for Base model), 4096-2048-1024-512-256 (L for Large model), and 8192-4096-2048-1024-512-256 (H for Huge model). We also employed different sizes of the DINOv2 model, base and large, as feature extractors to examine their effects on domain adaptation performance. We used a naming convention in XY format for clarity, where X represents the feature extractor size (B for base and L for large) and Y the bottleneck size. For instance, BB indicates that both the feature extractor and bottleneck are base size.

To optimize domain alignment and classification accuracy, we experimented with different values of the hyperparameter λ, and based on the experiments conducted on Section <ref>, the value of 0.7 was selected for further experiments. Adjustments were made to batch sizes and learning rates according to the model size to ensure computational efficiency on a 1080Ti GPU, with batch sizes set to 32, 16, and 8, and learning rates starting at 3e-2 and gradually decreasing.






 §.§ Comparison with the baseline models

Performance. Tables <ref>, <ref>, <ref> and <ref> show the accuracy rates of EUDA and baseline models on Office-Home, Office-31, VISDA-2017 and DomainNet, respectively. 
As can be seen, our model consistently outperformed all ResNet-based models across all datasets. Specifically, for the Office-Home dataset, our LL model surpassed CDTrans and TVT while delivering comparable results to PMTrans. For the Office-31 dataset, our model exceeded the performance of TVT and matched that of CDTrans and PMTrans. In the case of the VISDA-2017 dataset in Table <ref>, our model performed closely to other SOTA models. For the DomainNet dataset, our findings were particularly striking. Despite the large number of classes the dataset contains, our smallest model configuration, the BS, outperformed most of the SOTA models, including CDTrans, and delivered results comparable to PMTrans. 




Learnable parameters. Table <ref> compares the number of learnable parameters of our proposed EUDA model with baseline models.  EUDA requires significantly fewer trainable parameters compared to ViT-based baseline models while still achieving competitive performance. Specifically, our best model uses approximately 83% fewer parameters for the Office-Home and Office-31 datasets, 43% fewer parameters for the VISDA-2017 dataset, and an impressive 99.7% fewer parameters for the DomainNet dataset. This substantial reduction in model complexity highlights the efficiency of our approach, making it more suitable for deployment in resource-constrained environments without compromising accuracy. 
This indicates a higher efficiency in our modeling approach. EUDA employs a pre-trained DINOv2 model with frozen parameters as a feature extractor, i.e., there are no learnable parameters in this component, and learnability is confined to the bottleneck layers, the normalization layers associated with the feature extractor, and the classification heads. This streamlines the training process and reduces computational overhead, making EUDA particularly suitable for applications where resource efficiency is critical. This efficiency combined with the demonstrated effectiveness of our model in adapting to various domains underscores the practical advantages of our approach in real-world settings.




 §.§ Ablation Study

We test the effectiveness of the SDAL loss on the performance of EUDA model.
To achieve this, we remove MMD loss from Eq. (<ref>), i.e., the model only relies on the source domain and excludes learning from the target domain data. In Table <ref>, we assessed the performance of our BS and BL configurations on the Office-Home dataset using a source-only approach. The results clearly show that incorporating SDAL led to 1.2% and 3.5% improvements in the BS and BL configurations, respectively. Similarly, on the Office-31 dataset (see Table <ref>), our BB configuration improved by +0.4, and BL improved by +0.6 when using MMD loss. For the VisDA-2017 dataset (see Table <ref>), implementing SDAL with our BB and BL models perform approximately 8.1% and 7.4%, respectively better than only using CE loss.



 §.§ Sensitivity Analysis



  §.§.§ Effect of λ


Table <ref> shows the impact of λ in Eq. (<ref>) on the BS configuration for the Office-Home dataset.  In this experiment, we used four different values for λ: 0.3, 0.5, and 0.7. As can be seen, λ = 0.7 managed to produce the best results. The effectiveness of λ = 0.7 comes from its balanced approach that minimizes domain discrepancies through MMD loss while maintaining classification accuracy, ensuring robust performance across various domain shifts.



  §.§.§ Performance of the EUDA Framework Under Various Configurations

In this experiment, we test our model in different configurations across various datasets. Our goal is to find which model best suits each dataset. Our findings indicate that while more complex models perform better on complex datasets, our simpler models, which have significantly fewer trainable parameters, can also achieve comparable results. This flexibility allows users to adjust the model's complexity to match their datasets' specific requirements. This adaptive capability is a distinctive feature of our approach which provides a unique advantage by offering a scalable solution that adjusts to varying data complexities without sacrificing performance. 

Table <ref> demonstrates our model’s performance on the Office-Home dataset with B and L feature extractors and S, B, L, and H bottleneck configurations. Our testing on Office-Home helped us identify optimal configurations, which reducing the need for extensive trials on other datasets. The L feature extractor was notably effective due to its ability to handle the significant domain variance within the dataset, and benefits from higher-dimensional features that capture more informative details. The LL configuration provides the best balance of complexity and performance.

Table <ref> shows the results of Office-31 dataset using B and L feature extractors and B and L bottleneck configurations. It can be seen that LL configuration produces the best results. Insights from the Office-Home tests informed the decision not to use the H bottleneck, as higher complexity had not resulted in improved performance in previous tests.

For the VisDA-2017 dataset (see Table <ref>), the BH configuration stood out, particularly suited to managing the transition from simulation to reality. This confirms the benefit of a more complex bottleneck in handling extensive domain shifts between real and simulation data. This emphasizes the importance of matching architectural choices to specific dataset challenges.









While we have conducted extensive testing across multiple configurations and datasets, time constraints limited the breadth of our experiments. We therefore encourage researchers and practitioners to further explore various bottleneck configurations to meet the specific demands and complexities of their datasets, enabling customized solutions that optimally address their unique challenges.



§ CONCLUSION


In this paper, we highlighted the potential of using a self-supervised pre-trained ViT for UDA by introducing a versatile framework that maintains simplicity, efficiency, and adaptability to ensure its applicability in practical scenarios. Specifically, we leveraged DINOv2, a self-supervised learning method in ViTs,
to extract general features, and employed a simple yet effective bottleneck of fully connected layers to refine the extracted features.
Additionally, we utilized the MMD loss to effectively align the source and target domains. 
Our model produces comparable results to state-of-the-art UDA methods
with significantly fewer trainable parameters. This makes our method particularly suitable for real-world applications, including on-edge devices.
Our proposed framework demonstrates promising results, achieving top-tier performance with 43% to 99.7% fewer trainable parameters across benchmark datasets compared to other methods. In our future research, we will explore additional UDA approaches based on self-supervised pre-trained ViT backbones and expand the applications of UDA in fields such as Autonomous Vehicles and other demanding areas where UDA is crucial.




IEEEtran



    [
    < g r a p h i c s >
]Ali Abedi (Graduate Student Member, IEEE) received his B.Sc. in Computer Software Engineering in 2019 and completed his M.Sc. in Artificial Intelligence Engineering at the University of Isfahan in 2022. 
    He has accumulated extensive academic and industrial experience in the fields of artificial intelligence and machine learning. Currently, he is pursuing a Ph.D. in Electrical and Computer Engineering at the University of Windsor. His primary research interests include deep learning, machine learning, and computer vision, with a particular focus on transformers architecture and vision transformers.



    [
    < g r a p h i c s >
]Q. M. Jonathan Wu (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from the University of Wales, Swansea, U.K., in 1990. He was affiliated with the National Research Council of Canada for ten years beginning, in 1995, where he became a Senior Research Officer and a Group Leader. He is currently a Professor with the Department of Electrical and Computer Engineering, University of Windsor, Windsor, ON, Canada. He has authored or coauthored more than 300 peer-reviewed papers in computer vision, image processing, intelligent systems, robotics, and integrated microsystems. His research interests include 3-D computer vision, active video object tracking and extraction, interactive multimedia, sensor analysis and fusion, and visual sensor networks. Prof. Wu held the Tier 1 Canada Research Chair in automotive sensors and information systems from 2005 to 2019. He is an Associate Editor for IEEE TRANSACTIONS ON CYBERNETICS, the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, the Journal of Cognitive Computation, and Neurocomputing. He was on technical program committees and international advisory committees for many prestigious conferences. He is a Fellow of the Canadian Academy of Engineering.



    [
    < g r a p h i c s >
]Ning Zhang (Senior Member, IEEE) is an Associate Professor in the Department of Electrical and Computer Engineering at University of Windsor, Canada. He received the Ph.D degree in Electrical and Computer Engineering from University of Waterloo, Canada, in 2015. After that, he was a postdoc research fellow at University of Waterloo and University of Toronto, respectively. His research interests include connected vehicles, mobile edge computing, wireless networking, and machine learning. He is a Highly Cited Researcher. He serves as an Associate Editor of IEEE Transactions on Mobile Computing, IEEE Internet of Things Journal, IEEE Transactions on Cognitive Communications and Networking, and IEEE Systems Journal; and a Guest Editor of several international journals, such as IEEE Wireless Communications, IEEE Transactions on Industrial Informatics, and IEEE Transactions on Intelligent Transportation Systems. He also serves/served as a TPC chair for IEEE VTC 2021 and IEEE SAGC 2020, a general chair for IEEE SAGC 2021, a track chair for several international conferences and workshops. He received 8 Best Paper Awards from conferences and journals, such as IEEE Globecom and IEEE ICC.



    [
    < g r a p h i c s >
]Farhad Porpanah (Senior Member, IEEE) received the Ph.D. degree in computational intelligence from the University of Science Malaysia (USM), Malaysia, in 2015. He is currently a Postdoctoral Research Fellow with the Department of Electrical and Computer Engineering, Queens University, Kingston, ON, Canada. Prior to his current position, he was an Associate Research Fellow with the Department of Electrical and Computer Engineering, University of Windsor, Windsor, ON, Canada. From 2019 to 2021, he held the position of Associate Research Fellow with the College of Mathematics and Statistics, Shenzhen University (SZU), Shenzhen, China. Before his tenure at SZU, he was a Postdoctoral Research Fellow with the Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. His research interests include machine learning and deep learning, driven by an emphasis on creating smart environments that improve the quality of life for the elderly population.




 Paper Title: EAR: Edge-Aware Reconstruction of 3-D vertebrae structures from bi-planar X-ray images.


UTF8gbsn
1
.001




EAR


L. Tan et al.


mode = title]EAR: Edge-Aware Reconstruction of 3-D vertebrae structures from bi-planar X-ray images                      




























1]Lixing Tan[style=Chinese,orcid=0000-0002-0323-8802]
Conceptualization, Methodology, Software, Validation, Formal
analysis, Investigation, Writing & original draft, Visualization

1]Shuang Song[style=chinese]
Conceptualization, Methodology, Formal
analysis, Investigation, Writing & review & editing, Supervision, Visualization

2]Yaofeng He[style=chinese]
Validation, Formal Analysis
3]Kangneng Zhou[style=chinese]
Formal Analysis
4]Tong Lu[style=chinese]
Resources

1,2]Ruoxiu Xiao[style=Chinese]
[1]
Conceptualization, Resources, Project administration, Funding acquisition


[1]organization=School of Computer and Communication Engineering,
    addressline=University of Science and Technology Beijing, 
    postcode=100083 Beijing, 
    country=China
[3]organization=College of Computer Science, addressline=Nankai University,
    
    postcode=300000 Tianjin, 
    country=China
[2]organization=Shunde Innovation School, addressline=University of Science and Technology Beijing,
    
    postcode=100024 Foshan, 
    country=China
[4]organization=Visual 3-D Medical Science and Technology Development Co. Ltd,,
    
    postcode=100070 Beijing, 
    country=China


[1]Lixing Tan and Shuang Song contributes equally.
[cor1]Corresponding author at: The School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China. E-mail address: xiaoruoxiu@ustb.edu.cnxiaoruoxiu@ustb.edu.cn


§ ABSTRACT 

X-ray images ease the diagnosis and treatment process due to their rapid imaging speed and high resolution. However, due to the projection process of X-ray imaging, much spatial information has been lost. To accurately provide efficient spinal morphological and structural information, reconstructing the 3-D structures of the spine from the 2-D X-ray images is essential. It is challenging for current reconstruction methods to preserve the edge information and local shapes of the asymmetrical vertebrae structures. In this study, we propose a new Edge-Aware Reconstruction network (EAR) to focus on the performance improvement of the edge information and vertebrae shapes. In our network, by using the auto-encoder architecture as the backbone, the edge attention module and frequency enhancement module are proposed to strengthen the perception of the edge reconstruction. Meanwhile, we also combine four loss terms, including reconstruction loss, edge loss, frequency loss and projection loss. The proposed method is evaluated using three publicly accessible datasets and compared with four state-of-the-art models. The proposed method is superior to other methods and achieves 25.32%, 15.32%, 86.44%, 80.13%, 23.7612 and 0.3014 with regard to MSE, MAE, Dice, SSIM, PSNR and frequency distance. Due to the end-to-end and accurate reconstruction process, EAR can provide sufficient 3-D spatial information and precise preoperative surgical planning guidance.















3-D reconstruction Deep learning Edge attention Bi-planar X-ray


[
    [
    August 5, 2024
==================





§ INTRODUCTION



As the main structure supporting the human body, spine is of great significance in providing stability and flexibility to body posture and enabling coordinated movement. Spine-related diseases, including scoliosis, spinal dysraphism, and intervertebral disc protrusion, are greatly threatening human health. The misdiagnosis or delayed treatment can lead to physical deformities and even cardiovascular and pulmonary abnormalities, and further significantly impact the quality and daily functioning of patients. Compared with natural images, medical images contain higher ambiguity and complexity <cit.>. As the imaging gold standard of spinal disease diagnosis and treatment, X-ray images can visualize bone structures, locations, and abnormalities with fast imaging speed, high resolution and low radiation dosage. However, due to the projection from 3-D space to 2-D space is a dimension reduction process, much spatial information has been lost <cit.>, which makes it difficult for the surgeons to accurately measure the spinal morphology parameters, such as the spinal curvature, rotation and local deformation. In addition, the asymmetric structural distribution and occlusion also increase the difficulty of the reconstruction. Though computed tomography (CT) can clearly present the detailed spinal structures from the 3-D views, the imaging speed is slow. Meanwhile, due to the high doses of radiation, unavoidable harm will be caused to both the patients and clinicians. Hence, it is highly necessary to reconstruct the 3-D spine from the X-ray images with limited viewing angles. 

Given that much information is lost during the projecting procedure of X-ray images, it is an ill-posed problem to reconstruct the 3-D volume structure from 2-D projections. The traditional solutions mainly include the kriging-based <cit.> and statistical shape model (SSM)-based <cit.> methods. These methods are motivated by the deformation of 3-D flexible structures observed in consistent 2-D projections of X-ray images in specific imaging views. However, these methods often require additional prior information, such as anatomical structures.

Recently, along with the explosive development of deep learning models, 3-D reconstruction can be realized by directly learning to map the distribution of the 2-D X-rays to the distribution of the 3-D volumes. While the possibility and advantages of reconstruction with deep learning-based approaches have been demonstrated, drawbacks still exist and can be summarized in three aspects: (1) due to the dimension difference between the input and output images, there is a considerable semantic gap between the feature maps; (2) Reconstruction relies on feature matching, necessitating a balance between accuracy and real-time performance; (3) To improve the accuracy of the reconstruction, related researchers focus on disentangling spatial transformation and content information. It generally results in lower accuracy within edge regions. 

In this study, to relieve the above-mentioned problems, we proposed the Edge-Aware Reconstruction network (EAR) which is based on the auto-encoder architecture. During data preparation, we first expanded the dimension by replicating the bi-planar X-ray images along both anterior-posterior (AP) and lateral (Lat) directions. In such a way, the reconstruction can be realized with 3-D convolution by learning the potential correlations of features in the higher dimension. In the proposed network, we use the symmetrical auto-encoder architecture with five layers as our backbone. Convolutional block attention module (CBAM) is embedded in the shortcuts between the symmetrical downsampling and upsampling layers to increase the weights utilizing the spatial and channel attention mechanisms. In addition, to ameliorate the precision of edge reconstruction, we took advantage of feature similarity in the frequency domain and designed the edge attention module (EAM) and frequency enhancement module (FEM) to further strengthen the weights of edge structures. In our loss function, we used the edge displacement as an auxiliary constraint to enlarge the influence of the edge region. Moreover, we utilized the maximum intensity projections (MIP) of the outcome as the additional shape to speed up the training and improve the edge shape consistency. Finally, we defined a frequency distance (FD) loss term to convert the reconstruction results into the frequency domain and thus gain improved quality of high-frequency components corresponding to the edge and sharp regions.


The contributions of this study are threefold as follows:


    
    
  * We propose a novel and effective edge-aware reconstruction network by combining the frequency enhancement module and edge attention module, improving the accuracy of 3-D reconstruction from X-ray. The proposed EAR can not only handle the detailed parts of the vertebral body and spinous process, but also provides an end-to-end way to reconstruct the asymmetrical vertebrae by preserving the edge information and local shapes.

    
    
  * The frequency enhancement module transforms the feature maps into the frequency domain. It aggregates the high-frequency information corresponding to edge structures and eases the perception ability of the network to the edge regions. Additionally, the design of frequency distance loss further enhances the reconstruction accuracy of high-frequency details.

    
    
  * We provide a general way to generate the X-ray images from different view angles, which can provide more databases to train and evaluate the reconstruction networks.

    
  * We rigorously evaluate the proposed method both quantitatively and qualitatively using three public spine datasets, with the comparison of four other state-of-the-art 3-D reconstruction methods. The results show that our proposed EAR can not only effectively reconstruct the vertebrae body but also preserve the detailed edge structures.

    





§ RELATED WORK


The traditional methods for 3-D reconstruction from X-ray radiographic images are based on the deformation models. Generally, a 3-D deformable reference structure is firstly pre-defined. By transforming the reference structure with an affine matrix, the 3-D reconstruction can then be realized when the errors between the projected images of the transformed reference structure and the target 2-D images reach the minimum value. The earlier methods are mainly derived from the kriging- and SSM-based approaches <cit.>, which assume that the spine is a rigid structure and always with similar morphologies. In the kriging-based method, a series of anatomical landmarks need to be selected in the X-ray images. By calibrating the corresponding landmarks from different X-ray images, the 3-D locations of these landmarks are obtained. Then the landmarks in the 3-D space are sent into an interpolation process to generate detailed points of the target model. Such kinds of methods can be optimized by combining with multi-view geometry techniques. Mitton et al. performed deformation on an elastic object so that the stereo-corresponding and non-stereo-corresponding observations in different views can be maintained <cit.>. This technique requires the annotation of representative anatomical landmarks on the projections, and the reconstructed edge regions exhibit significant distortion. Furthermore, this method demonstrated a significantly slow reconstruction speed, which cannot meet the real-time requirements of surgical procedures. Humbert et al. constructed a parametric model based on longitudinal and transversal inferences for fast reconstruction <cit.>. Generally, to personalize the reconstruction function, large amounts of parameters need to be defined and optimized which greatly decreases the model efficiency. To reduce the influence of the difference between the pre-defined deformable structure and the target reconstructed object to reconstruction accuracy, a mean statistical shape of spines is spatially deformed to realize the reconstruction. The reconstruction process is to find a series of optimal parameters by maximizing the similarity between the projected structures of the transformed images and real 2-D spines. Benameur et al. <cit.> modeled the prior geometric structure of each vertebra and realized the 3-D reconstruction by using the first mode of variation in the Karhunen-Loeve expansion. In their later study <cit.>, they divided the reconstruction into two stages which include a global deformation of the whole spine and a local deformation of each vertebra. The global one is based on statistical modal analysis while the local one is described by the first-order Markov. A stochastic optimization algorithm is then utilized to solve the two independent minimization processes. Boisvert et al. <cit.> included the 3-D positions of landmarks in the local coordinate system to capture the scoliosis of spines. However, due to the sparse estimation, global distortions tend to occur with regard to the vertebrae morphology. Considering the intuitive appearance reconstruction of spines, Zheng et al. <cit.> proposed the point distribution model to build the SSM. By computing the deformable field between the point cloud from input 2-D images and the SSM using an iterative closet point algorithm, the correspondences between 2-D and 3-D spaces can be obtained. The thin-plate splines method is then utilized to deform the shape of SSM.







Due to traditional reconstruction methods depend on the camera parameters of capturing X-ray images, it is much more complicated and time inefficient to iteratively solve the deformation attributes. Additionally, the deformation-based methods are more suitable for the mesh modeling instead of the volume images.



To ease above mentioned questions, solutions to 3-D reconstruction increasingly employ deep learning methods <cit.>. Kim et al. <cit.> extracted features from convolutional neural network (CNN) and then combined with the SSM to optimize the reconstruction process. Henzler et al. <cit.> first attempted to reconstruct the 3-D volumes under the auto-encoder architecture. Since the proposed method is based on the single-view image, it is difficult to restore comprehensive structures and also tends to introduce ambiguities. Shen et al. proposed a three-layer Patient-Specific Reconstruction (PSR) to reconstruct a CT volume from X-ray images <cit.>. In this framework, they demonstrated the possibility of reconstructing CT images from single-planar X-ray images, and their experiments demonstrated that the results could be further refined by increasing the number of views. However, PSR has some limitations: due to the deep network architecture and its pure CNN backbone instead of auto-encoder backbone, the training of the network requires a substantial amount of GPU memory and training time. Furthermore, due to the absence of additional prior information, the network can only differentiate between soft tissues and bones based on intensity, which increases the complexity of the reconstruction. In comparison, X2CT-GAN designed a generative adversarial network to fuse information from two orthogonal X-rays so that the dimension of the data can increase from 2-D to 3-D <cit.>. Katen et al. proposed the dimensional expansion scheme by replicating the bi-planar X-ray images along the orthogonal axes and further fed into the auto-encoder network. Since the operation is in the 3-D space, the pre-processing step greatly reduces the reconstruction difficulty of feature mismatching within cross-dimension.

Deep learning-based approaches have potentially made it possible to reconstruct the whole structures with complicated morphology. In order to pay more attention to local shape reconstruction, attention mechanism is gradually becoming an essential part of the networks. The principle of attention mechanisms is to emulate the human visual cognition system by assigning distinct weights to target regions. Such strategy is capable of catching complex semantic relationships in medical images of multi-modalities <cit.>. To fuse the features from multi-scale stage, Chen et al. <cit.> introduced a feature attention guidance module (FFAG) to aggregate image features from multi-resolutions. Considering the asymmetric shape and edge sharpening of the whole spine, Zhang et al. <cit.> implemented an edge attention module to guide the segmentation process and gained significant accuracy. Fan et al. utilized reverse attention to combine current feature maps and edge attention maps <cit.>. By regarding the feature map from low-level convolution as edge attention, 
they concatenated it with multi-resolution feature maps through multi-scale down-sampling, thus implicitly learning edge features. Wang et al. <cit.> proposed the edge attention preservation module to improve the perceptibility of the network to the edge regions. Xia et al. <cit.> introduced the reverse edge attention module and an edge-reinforced loss term to jointly enhance the voxel weights close to the edges. Though spatial edge information can be preserved, research concerning edge reconstruction is still scarce.



§ METHODOLOGY

Fig. 1 shows the overall workflow of the proposed 3-D reconstruction framework. In our framework, the dataset after dimension expansion is extended to two volumes with the shape of 128× 128× 128 so that the dimensional gap between 2-D X-rays and 3-D volumes is minimized. The proposed pipeline increases the gap between high- and low-frequency signals and employs an edge-awareness decoding process. The ABS improves the reconstruction quality by suppressing the discrete noise region. 





 §.§ Data Preparation


Considering the limitations of imaging system, it is difficult to capture the X-ray spine images from multi-views in a single imaging procedure. Compared to the work from Ying et al. who synthesized the X-ray images from CT scans while using CycleGAN to map the real X-rays to the synthesized style to minimize the gap between synthesized images and real X-rays <cit.>, we directly generated much more realistic X-ray images of multi-views by modifying DeepDRR <cit.> by which paired CT and X-rays of the same patient can be made to facilitate the reconstructions of CT scans from multi-view X-ray images using the deep networks. The illustration of digitally reconstructed radiograph (DRR) is shown in <ref>. In the generation, we resampled the CT to a resolution of 1 mm × 1 mm × 1 mm and then extracted single vertebrae from the uniformly resampled CTs using masks and constructed the projection matrix with variable Euler angles (alpha and beta) and fixed distances, both source to object distance (SOD) and source to detector distance (SDD). As for the attributes of DeepDRR, we set the energy spectrum to '60KV_35AL' and used more than 1000000 photons to simulate the realistic scatter process. We focus on the single vertebra reconstruction instead of the whole spine (12 thoracic vertebrae and 5 lumbar vertebrae) considering the limited GPU memory. The whole spine can then be composed by reconstructing each vertebrae respectively. 

All volumetric scans are resized to 128×128×128 and the 2-D projections are resized to a resolution of 128× 128 due to the limitation of memory and the purpose of computational efficiency. To improve convergence, a z-score normalization step was employed where the intensities of the volumetric data were manipulated to have a zero mean and a unit variance.



 §.§ Backbone Network of 3-D Reconstruction


The backbone network utilizes the typical U-Net structure with five layers. In the encoder part of the network, the images with enhanced dimensions are regarded as the input. In each layer of the down-sampling path, the module begins with a convolution operation with a kernel size of 3× 3× 3 and a stride of 2. Afterward, it adds a convolution operation with a kernel size of 3× 3× 3 and a stride of 1. Then a batch normalization (BN) and a leaky rectified linear unit (leaky ReLU) follow the convolution operation. At the end of the first three layers in the encoder, we added a simple yet effective frequency enhancement module (FEM) to better discriminate the high- and low-frequency contents of the feature maps.

In comparison to the encoder, the decoder performs an up-sampling operation on the feature maps and restores the resolution of the feature maps layer by layer. 
In the decoder, the designed layer consists of a transposed convolution layer with a kernel size of 2× 2× 2 and a stride of 2 and a convolution layer with the same setting as the second convolution operation used in the down-sampling layers. In addition, due to the lack of edge information, we introduced 3-D edge attention module (EAM) into each decoding layer to retain more edge information. Lastly, we designed an attention-based shortcut (ABS) to get a more fine-grained outcome.

In the next sections, we will look at the two aspects: edge enhancement (in sections 3.3.1 and 3.3.2) and optimized shortcut connection (in section 3.4).









 §.§ Edge-aware Attention Module


Considering the irregular and asymmetrical shape of the vertebrae, the most difficult part of reconstructing the vertebrae is to accurately preserve the edge information. Hence, in the reconstruction process, to better perceive the edge information, we designed the FEM and further combined it with the 3-D EAM to improve the attention to the edge regions. 





  §.§.§ Frequency Enhancement Module


Considering that the receptive field in the image domain is limited, the features of edge regions cannot be aggregated and differentiated from other flattened regions. On the contrary, the edge regions in the frequency domain all gather in the high-frequency part which makes it much easier to pay more attention to the edge regions. Hence, by applying the Fourier transform to the feature maps in the encoder path, we design the FEM to emphasize the edge regions, as shown in <ref>.



In the architecture of this module, the feature maps are first fed into the instance normalization (IN) layer to focus on the statistical property instead of over-smoothing the specificity of the feature maps. Next, a fast Fourier transform (FFT) layer is connected to transform the normalized feature maps into the frequency domain. In this procedure, the feature maps are split into the real and imaginary parts in the frequency domain. Subsequently, two parallel convolution layers are conducted to the real and imaginary parts, respectively. The magnitude of their output complex values from the two layers are then computed by an inverse fast Fourier transform (IFFT). To better complement the features both from the image and frequency domains, the input feature maps to this module are concatenated with the one processed by the frequency layers and further fed into a convolution block (Conv3-D+IN+ReLU). 



    ℛ,ℐ   =Conv(FFT(IN(𝐅_l)))
    𝐅_FEM   = ReLU(IN(Conv(IFFT(ℛ + iℐ)⊕𝐅_l)))




  §.§.§ 3-D Edge Attention Module


In the process of feature transferring to the decoder part, the edge information will be gradually weakened due to the down-sampling. To force the edge information to provide more constraints to the reconstruction procedure <cit.>, we designed a 3-D EAM to capture the edge information from both image and frequency domains. 

In the 3-D EAM, the low-level features extracted by the frequency enhancement module are regarded as the inputs. Inspired by the EANet <cit.>, this module is constituted of the residual blocks comprising the modified gated convolution blocks (m-GCB), as shown in <ref>. In the residual blocks, two convolution blocks (Conv3-D+BN+ReLU) are connected. Gated convolution (GC) was first proposed for the image inpainting task <cit.> which distinguishes between the internal and external content of a specified section and  was proved efficient in medical tasks <cit.>. By emphasizing the external content over internal redundancy, the weight of edge regions is increased. The m-GCB aims to constrain the network to pay more attention to edge information. In this way, feature maps from the current level and the previous level are concatenated and then fed into two connected convolution blocks to compute the gated maps. In the second convolution block, we use the Sigmoid function as the activation function to capture more edge information. After obtaining the gated maps, the features from the current level are multiplied by the gated maps, enabling the edge attention module to focus solely on the edge information. 





Given feature map 𝐅∈ R^C× H× W× D and its gating vector 𝐅_g∈ R^C'× H'× W'× D', the GC can be calculated as follow:

    α = ((𝐅×σ(BN(𝐅⊕𝐅_g)))+𝐅)

where ⊕ denotes concatenation, σ refers to sigmoid function.



 §.§ Attention-based Shortcut


Although dimensional enhancement greatly reduces the dimension gap between 2-D X-ray images and 3-D CT volumes, we noticed discrete parts appearing in the outcome due to redundant information generated during dimensional enhancement. To suppress the effect of redundant noise and focus on the bone regions' features, we introduced CBAM to the shortcuts of each layer.

Given that CBAM was originally designed to manipulate 2-D data, we modified it to 3-D version according to our dataset. In this module, by using the lth layer feature map 𝐅_l∈R^C× H× W× D as input, CBAM first derives the channel attention module 𝐀_c∈R^C× 1× 1× 1 along the channel dimension and multiplies the feature map with the calculated channel attention map, which can be described as follows:

    𝐅'_l = 𝐅_l×𝐀_c(𝐅_l)


Then the spatial attention module 𝐀_s∈R^1× H× W× D along the spatial dimension is connected, also we multiply the computed spatial attention map with the input feature maps, as follows: 

    𝐅”_l=𝐀_s(𝐅'_l)  ×𝐅'_l


By adding the CBAM module to the shortcuts, the network can force the learning process opt to discard the possible redundant information and hence lead to a better reconstruction result.



 §.§ Loss function


In the design of the loss function, we take into account the loss from both image and frequency domains. In the image domains, to reduce the blur regions during reconstruction, we choose the mean absolute error (MAE) as our reconstruction loss, as follows:

    ℒ_recon = E_x,y,z|| G-V ||_1

where x,y and z are the voxel indexes along x, y and z axes respectively.

As proposed in section 3.2, the EAM provides rich structural and textural details of the edge region, which can contribute more to the 3-D accurate reconstruction. Hence, considering that edge regions only occupy a small proportion in the computing of the overall reconstruction loss, we also adopt the output of the edge-aware attention module to design a loss focused on the edge regions. To be more specific, we calculate the binary cross-entropy (BCE) between the extracted edges from ground truth (GT) and reconstructed vertebrae, respectively. The details are as follows: 


    ℒ_edge = -E_x,y,z[   E(G)ln(E(V)+
       (1-E(G))ln(1-E(V)]


where G represents GT vertebrae, V represents reconstructed vertebrae and E(·) denotes extract edge operation.


To further improve the reconstruction accuracy of the edge regions, we take the statistical properties of high-frequency information from the frequency domains into account. In this way, the weight of the loss caused by the edge regions can be increased which can make the network pay more attention to the edge, contour and the corresponding texture information. To evaluate the loss, we define the loss term using the Euclidean distance, as follows:

    𝒟_real = E_x,y,z||(ℛ(G)-ℛ(V))||_2  
       𝒟_imag = E_x,y,z||(ℐ(G)-ℐ(V))||_2


where ℛ and ℐ denote the real and image components of the feature maps in the frequency domain after performing the FFT, respectively.

Furthermore, the mean value calculation potentially causes the weakening of high-frequency information. Hence, we still retain the original Euclidean distance but utilize the logarithm calculation to reduce the magnitude of the excessive accumulation in this loss term. Here, we name the loss term as the frequency distance (FD), as follows:


    ℒ_freq = log_10(𝒟_real+D_imag)


Considering that MIP cannot only preserve the whole shape of structures but also the relative position of different structures, we compute the MIP both of the reconstructed and the real vertebrae from three orthogonal views, respectively. In this procedure, the three views are consistent with the conventional direction in the medical imaging community (axial, coronal, and sagittal). In this study, we utilize the L1 norm to evaluate the similarity of the MIP images. The projection loss is defined as follows:

    ℒ_proj = 1/3[   E_x|| max (G_x(y,z))-max ( V_x(y,z))||_1 +
       E_y|| max (G_y(x,z))- max (V_y(x,z))||_1
        +
       E_z||max (G_z(x,y))- max (V_z(x,y))||_1]



Finally, the total objection loss function can be summarized as follows:

    ℒ_total = λ_1ℒ_recon+λ_2ℒ_edge+λ_3ℒ_freq+λ_4ℒ_proj


where λ_1, λ_2, λ_3, and λ_4 are the weights controlling the contribution of different loss terms.




§ EXPERIMENTS



 §.§ Datasets

To evaluate the proposed method, it is necessary to provide a dataset with bi-planar X-rays and their corresponding CT volumes. Obtaining such a dataset is challenging and constructing one from scratch is impractical. As an alternative, we adopt the following strategy: we first select three publicly available datasets to synthesize the corresponding simulated X-ray images using DeepDRR <cit.>. This technique has been proven to be effective in prior works <cit.>. More specifically, we validate the broad applicability of our proposed EAR for bi-planar 3-D reconstruction on those datasets: Verse19 <cit.>, VerSe20 <cit.>, CTSpine1K <cit.>. These datasets are used for vertebrae segment task. These datasets contain the original CT scans and masks for each vertebrae. We first extract individual vertebrae from CT scans using the offered masks and resample them to a 1× 1× 1 mm^3 resolution. These resampled vertebrae scans serve as the GT. The 2-D inputs are then constructed by synthesizing DRRs from these GT volumes.

The accurate reconstruction of vertebrae highly depends on the correspondence of features from multiple views. However, during the imaging process of vertebrae, variations in patient pose or the position of the hospital bed can decrease the reconstruction quality due to partial overlapping of the same structure. To mitigate potential interference from these imaging challenges, we utilize the DeepDRR model to obtain projected images from different imaging angles, focusing solely on the influence of the network and loss terms on reconstruction accuracy. To improve the generalization ability of the proposed method, we utilize three public datasets, including Verse19 <cit.>, VerSe20 <cit.> and CTSpine1K <cit.>. The details are as follows:

VerSe19 is a vertebral labeling and segmentation dataset prepared for a grand challenge hosted at the 2019 Medical Image Computing and Computer Assisted Intervention (MICCAI), including 160 CT scans.

VerSe20 is also prepared for a grand challenge hosted at the 2020 MICCAI, containing 319 scans from 300 patients.

CTSpine1K is collected by the Key lab of Information Processing of the Chinese Academy of Sciences, which contains 807 public CT scans. 

After removing the duplicated or incomplete scans, we eventually extracted 5838 vertebrae as the dataset. The dataset is divided into training set, validation set and test set according to a ratio of 8:1:1. 




 §.§ Evaluation Metrics


To quantitatively evaluate the proposed method and conduct the comparison experiments, we adopt six different metrics, including mean square error (MSE), MAE, Dice coefficient (Dice), peak signal noise ratio (PSNR), structural similarity index measure (SSIM) and FD. The generation quality and difference between the reconstruction and GT can be measured by calculating these metrics.


MSE and MAE are both utilized to evaluate the proximity of the reconstructed results and GT, but utilize L1 and L2 norm, respectively. In this way, we can pay more attention to the outlier removal. The closer to 1 means the higher reconstruction accuracy. In detail, MSE is computed as follows:

    MSE = E_x,y,z ||GT - pred||_2


Dice can be utilized to measure the overlap ratio between the predicted results and GT. The range of Dice coefficient is [0,1]. The closer to 1 means the higher reconstruction accuracy while closer to 0 means the lower accuracy. The Dice can be computed as follows:

    Dice = 2× (pred ⋃ GT)/pred ⋂ GT

where pred denotes the predicted results, while ⋃ denotes the intersection set and ⋂ denotes the union set.



PSNR can be utilized to evaluate the quality of the reconstructed results by calculating the ratio of the energy of the peak signal to the average energy of the noise. A higher PSNR indicates a higher percentage of signal components in the reconstruction results with better quality.

    PSNR = 10×log_10(Max Values^2/MSE)

Since the CT is often recorded with 12 bits, we set the Max Values to 4096.


SSIM can be exploited to objectively evaluate the quality from the view of the human visual system. The value of SSIM is between [-1,1], and the closer to 1 indicates that the result is more consistent with the high quality of human vision as well as high similarity. SSIM can be computed as follows:

    SSIM = (2μ_predμ_GT+C_1)(2σ_predGT+C_2)/(μ_pred^2+μ_GT^2+C_1)(σ_pred^2+σ_GT^2+C_2)

where μ denotes mean value, σ denotes variance, C1 and C2 is added to prevent dividing by zero.

FD is designed as a metric to evaluate the accuracy of the reconstructed results concerning the edge regions since edge information is crucial for preserving the structure of spines. Theoretically, the range of FD is in [0,+∞], where a lower value indicates closer proximity to the GT in the frequency domain. To enhance result comparability, we apply a logarithmic operation and min-max normalization to the FD, resulting in a range of [0,1]. A lower value now signifies a higher similarity in the frequency domain.





 §.§ Methods for comparison


To evaluate the performance of the proposed method, we compare it with four state-of-the-art (SOTA) methods: PSR <cit.>, Single Image Tomography (SIT) <cit.>, End-to-End reconstruction (E2E) <cit.> and BX2S-Net (BX2S) <cit.>. We achieve PSR and BX2S using public implementation. We re-implemented SIT and E2E in strict accordance with the original papers. The vanilla SIT is designed for single view input, we modified it to support bi-planar input. Specifically, we changed the first layer of SIT to a convolution layer with an input channel of 2. The E2E is originally designed for bi-planar input. Therefore, we reproduce the E2E as a 3-D auto-encoder with a dimensionality enhancement following <cit.>. To be fairly compared, all the models are trained using the same parameters and settings.




 §.§ Implement details

We implemented all the algorithms using PyTorch1.12 under the Ubuntu 20.04 environment and conducted all the experiments on a local computer with an NVIDIA GeForce RTX 3090 GPU and 2.4GHz Intel CPU. We use Adam as the optimizer, with an initial learning rate of 0.01, a batch size of 2, and a weight decay rate of 0.00001. A cosine annealing learning rate policy with a linear warmup for the first 10 epochs is used. The maximum epoch is 100. In addition, the λ_1,λ_2,λ_3 and λ_4 in objective function is empirically set as 1, 0.1, 0.1 and 0.1. For a fair experimental comparison, all the comparative networks are trained using the same setting.



§ EXPERIMENTAL RESULTS

To demonstrate the effectiveness of the proposed model, we first visualized the reconstructed results of a spine, as shown in <ref>. The prediction results are CT volumes. To better visualize the reconstruction results, we used the Marching Cubes algorithm to extract meshes from the predicted volumetric results. In this figure, the image on the first column refers to the original CT volume of the spine. Structures in the second column are the vertebrae from top T10 to bottom L5 which can be distinguished by the masks with different colors. The reconstructed vertebrae are shown from the third to fifth columns in different view angles. It can be seen that the proposed method is capable of reconstructing the stitched spine. In each row of the figure, we display the reconstructed results of  T11 thoracic vertebrae, L2 and L5 lumbar vertebrae, respectively. It can be observed that the proposed method accurately reconstructs the shapes and local details of different vertebrae. In particular, as illustrated by T11 vertebrae, our method also demonstrates its possibility even when dealing with the local vertebral deformities.



 
To better visualize the superiority of EAR, <ref> shows the heat maps by computing the surface distance error (SDE) between GT and reconstructed results by different methods, respectively. In the visual results, the PSR reconstructs the vertebral body and spinous process as a cylinder, resulting in large deviations. The SIT fails to handle uneven areas. The E2E and BX2S exhibit poor performance in morphological irregular areas such as the protrusions at the bottom of the spinous process and the depressions on the lateral sides of the vertebral body. In addition, due to the introduction of FFAG, the BX2S tends to generate discrete noise. In contrast, the proposed EAR achieved better reconstruction results in recovering the sharp regions and maintaining detailed vertebrae structures. 



 
To quantitatively compare the proposed method with other methods, <ref> presents the results by computing MSE, MAE, Dice, PSNR, SSIM and FD. The proposed EAR outperforms the other methods concerning all six metrics. Specifically, the proposed method achieves 25.32%, 15.32%, 86.44%, 23.7612, 80.13% and 0.3014 in terms of MSE, MAE, Dice, PSNR, SSIM and FD, respectively, which are 4.74%, 2.09%, 2.44%, 0.9633, 1.72% and 0.0587 higher than other comparative methods. The PSR shows better performance in terms of the Dice score while it performs the worst on other metrics. The possible reason is the cylinder assumption makes the overlap area between the GT and reconstructed results larger. In particular, the Dice score of BX2S is 3.74% lower than that of EAR, which further demonstrates the contribution of the EAM and FEM to edge preservation. 
 





 §.§ Ablation Study of the Network Modules



To verify the effectiveness of the proposed modules, we conducted ablation studies based on Baseline. The results are illustrated in <ref> and <ref>, while the visualization results of the proposed modules are demonstrated in <ref> and <ref>, from which we draw several conclusions.
















   



The comparison results in <ref> show that ABS improves the reconstruction results by 0.73%, 8.46% in terms of MSE and Dice, which shows that ABS benefits from removing redundant noises and improving reconstruction. We also visualize the reconstructed results in <ref>. After adding ABS, the reconstruction results are displayed in the second column. Lighter colors on the reconstructed surface indicate a closer match to the GT.

Based on the above network, we also present the results in <ref> after adding the FEM. 
It is observed that FEM improves the results with MSE, MAE, PSNR and SSIM increasing by 1.37%, 0.18%, 1.0597, 0.18%, respectively, which preliminarily explains that the FEM obtains more detailed sharp structures. Moreover, FD is significantly improved. We also further visualize reconstructions of the network with FEM in <ref>. Thanks to FEM, the color of the sharp structure is relatively light, which implies that FEM tends to discover the high-frequency component of the vertebrae and adequate edge information is captured. 


In the evaluation of EAM, the comparison results in <ref> show that EAM improves the reconstruction performance by 0.27%, 1.47% and 0.3217 in terms of MSE, Dice and PSNR, respectively, demonstrating better edge reconstruction ability of the network with EAM. As shown in <ref>(C), both the red and blue areas are lighter than the previous results, indicating that the EAM contributes to a more accurate reconstruction. 




We further visualize feature maps in the proposed EAM, as show in <ref>. Three samples were randomly selected to visualize the edge attention maps from the sagittal, coronal and transverse views. Brighter regions indicate a high probability of being identified as an edge region. With EAM, the surface part of the vertebrae appears particularly bright, indicating that EAM tends to detect the edge part and capture richer spatial information.


To further verify that FEM can improve the accuracy of high-frequency edge region and clarify the necessity of combining the FEM and EAM, we visualize the effect of FEM. Specifically, we randomly select a sample and predict the results using models with only EAM and with both EAM and FEM, respectively. Then we visualize the difference by subtracting the results without FEM from the results with FEM. The subtraction result is shown in <ref>. It can be observed that after subtraction, residual parts mainly concentrate in the high-frequency edge regions, indicating that incorporating FEM results in finer and more complete edge reconstruction, validating the effectiveness of FEM.









 §.§ Ablation Study of Loss Function


The performances of different loss terms are summarized in <ref>. We set the reconstruction loss as the base loss following other methods <cit.> and gradually add loss terms to evaluate the effectiveness of each loss term. As the first row shows, reconstruction loss performs the worst on MSE and FD due to the lack of edge constraints. The edge loss contributes slightly to improving the results. After adding the frequency loss, significant improvement has been made by constraining the high-frequency feature similarity and hence can lead the network to reconstruct the sharp edge features. The projection loss brings additional improvement in the reconstruction quality by constraining the shape contours of the reconstructed volume from three axes during training. Therefore, the projection loss can improve the shape consistency. From the table, the values of PSNR, SSIM and FD of the network with all the loss terms outperform all the other combinations. The results indicate that the projection loss improves the shape similarity of the prediction and GT. 








§ DISCUSSION

In this study, we proposed a novel and robust edge-aware reconstruction method. From a methodological perspective, while previous 3-D reconstruction methods focus only on the distribution of voxels within CT volumes, our approach prioritizes edge information. The proposed reconstruction method was evaluated by using three public datasets comprising 5838 vertebrae from 1162 CT scans. A comparative quantitative validation was conducted to assess whether our proposed technique is superior to previously developed PSR, SIT, E2E, and BX2S. In PSR, the 2-D inputs are directly fed into the network. This may slow down convergence, cause false reconstructions, and introduce significant noise. SIT, on the other hand, smooths out many high-frequency features during the 2-D to 3-D transformation, resulting in inaccuracies in reconstructing uneven areas of the vertebrae. As for E2E, while dimension expansion reduces the dimension gap, it overlooks edge information from the vertebrae, thereby inadequately handling complex and asymmetrical vertebrae surfaces. Regarding BX2S, the FFAG may guide the network's attention to redundant parts brought by dimension enhancement, leading to noise outside the vertebrae. In the validation of the reconstruction performance of the proposed EAR, we obtained MAE of 0.1532 and PSNR of 23.7613 for test sets. The results indicate that incorporating FEM and EAM can improve the reconstruction accuracy of sharp edge features. Previous works have attempted to improve edge constraint by adding an edge channel of the output to predict the edge of the reconstructions <cit.>. In contrast, we utilized an edge loss term to directly enforce edge similarity. Moreover, the designed frequency loss term is as substantial to the sharp edge region as edge loss. 









To study the potentiality of the proposed method when the angle of the bi-planar X-rays varies, especially in the situation of non-strictly orthogonal, we trained the proposed network when the angle of two views gradually increases from 60^∘ to 140^∘, as shown in <ref>. As can be seen from the figure, when the angle is 60^∘, the reconstruction exhibits a lot of red in the SDE, indicating that the reconstructed surface is mostly inside the GT surface. When the angle is 80^∘ and 110^∘, the overall color of the reconstruction is lighter, which denotes an acceptable reconstruction. As the angle increases to 120^∘, the color of the reconstructed vertebrae processes is darker, indicating a decrease in the reconstruction performance of the edge region. When the angle reaches 140^∘, since the X-rays are fluoroscopic, the non-overlapping information provided by the two views is greatly reduced, resulting in wrong reconstructions.
 



§ CONCLUSION

This study tackles the challenging task of reconstructing 3-D structures from bi-planar X-ray images using deep neural networks, particularly when faced with limited perspective information. The proposed EAR can achieve sparse-view 3-D reconstruction, directly recovering lost 3-D spatial information from 2-D X-ray images taken at limited viewing angles, and compute relatively clear vertebral morphologies. The proposed EAR is automated and has a very short inference time, making it highly suitable for real-time applications, including intraoperative reconstruction. Given that EAR prioritizes edge information and enhances the recovery of edge details from both spatial and frequency domains, it achieves a more precise reconstruction of asymmetric and uneven areas of the vertebral surface, thereby providing support for clinicians to better identify vertebral lesions. This achieves the challenge of inaccurate prediction in edge regions existed in current methods. Specifically, to enhance feature fusion between encoder and decoder layers, we incorporate the CBAM into the shortcuts. Additionally, we also propose the EAM to extract crucial edge information from shallow layers, aiding in the reconstruction process’s accuracy, especially concerning edge details. The versatility of our proposed EAM allows it to serve as a plug-in module for edge-related learning across various 3-D reconstruction tasks. Furthermore, we devise the FEM to amplify network attention towards high-frequency information. Empirical evidence demonstrates that EAR outperforms SOTA methods such as PSR, SIT, E2E and BX2S in terms of MSE, MAE, Dice, PSNR and SSIM. These results underscore the significant potential of our method for accurate 3-D reconstruction from bi-planar X-ray images.

However, the limitations and challenges are as follows: First, with regard to X-ray images, the information provided represents a perspective structure within the human body. When the two X-ray image angles input into EAR are not orthogonal, the non-overlapping information that can be provided is significantly reduced. This situation results in the network's inability to accurately infer the 3-D structure of the overlapping parts. Since diffusion model-based methods can generate multi-view consistent images from single-view inputs <cit.>, our future work will explore the extension of diffusion model-based methods to generate dense multi-view X-ray images. Secondly, our method is based on explicit voxel training, which requires significant memory, and the model requires fully supervised training, thus limiting its generalization capability. Therefore, future work will explore extensions based on more effective 3-D representation techniques like Neural Radiance Field (NeRF) <cit.> and 3-D Gaussian Splatting <cit.>, which eliminate the need for training neural network models and instead enable direct reconstruction of the target 3-D volume through iterative optimization. <cit.>.





§ DECLARATION OF COMPETING INTEREST

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.



§ ACKNOWLEDGEMENT

This work was supported in part by grants from National Key Research and Development Program of China
(2023YFB4706003), National Natural Science Foundation of China (62176268), and Beijing Natural Science Foundation-Joint Funds of Haidian Original Innovation Project (L232022).



model1-num-names









 Paper Title: Joint Neural Networks for One-shot Object Recognition and Detection.







Joint Neural Networks for One-shot Object Recognition and Detection
    Camilo J. Vargas,
        Qianni Zhang,
        Ebroul Izquierdo.
C. J. Vargas, Q. Zhang and E. Izquierdo are with the School of Electronic Engineering and Computer Science, Queen Mary University of London, London. UK.
E-mail: {c.j.vargascortes, qianni.zhang, ebroul.izquierdo}@qmul.ac.uk


    August 5, 2024
======================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
This paper presents a novel joint neural networks approach to address the challenging one-shot object recognition and detection tasks. Inspired by Siamese neural networks and state-of-art multi-box detection approaches, the joint neural networks are able to perform object recognition and detection for categories that remain unseen during the training process. Following the one-shot object recognition/detection constraints, the training and testing datasets do not contain overlapped classes, in other words, all the test classes remain unseen during training. The joint networks architecture is able to effectively compare pairs of images via stacked convolutional layers of the query and target inputs, recognising patterns of the same input query category without relying on previous training around this category. The proposed approach achieves 61.41% accuracy for one-shot object recognition on the MiniImageNet dataset and 47.1% mAP for one-shot object detection when trained on the COCO dataset and tested using the Pascal VOC dataset. Code available at <https://github.com/cjvargasc/JNN_recog> and <https://github.com/cjvargasc/JNN_detection/>.



One-shot detection, One-shot recognition,  multi-box detection, pattern recognition.








§ INTRODUCTION


The automatic recognition, description, classification, and grouping of patterns are relevant tasks in a variety of engineering and scientific disciplines such as biology, psychology, medicine, marketing, security, transport, and remote sensing. Regardless of almost 50 years of research, the design of a general-purpose machine pattern recognizer remains an elusive goal <cit.><cit.>.

Deep Neural Networks (DNNs) have achieved the state-of-the-art performance on a variety of pattern recognition tasks, most notably visual classification problems. Typically, a very large amount of training samples are required to achieve state-of-art results <cit.>. In contrast, real world applications often require to achieve high recognition performance with very limited datasets. In those cases the classic deep neural networks fall short, leading to overfitting on the training sets and poor generalizations on the test sets <cit.>. Thus, the capability to generalize visual concepts from small amounts of data becomes a highly desired characteristic in modern computer vision systems

This paper introduces Joint Neural Networks to tackle the one-shot object recognition and detection tasks. Here, the Joint Neural Network model is trained as a matching network to perform recognition or detection in pairs of images. The proposed approach is tested using input query and target images belonging to categories that remain unseen during the training stage. Such approach is applicable to any field where data availability may be reduced to a single sample, where traditional machine learning and data augmentation approaches fall short.

Two different architectures akin to Siamese neural networks (SNNs) are presented in this paper to tackle the recognition and detection tasks. Both Joint Neural Network architectures rely on the joint layer key concept. The joint layers proposed in this paper are convolutional layers that merge the convolutional filters resulting from the two branches of the Joint Network architectures via concatenation. The outputs of the joint layers are built to match the defined backbone architecture used for the Joint Neural Network.

For recognition, the proposed novel approach is akin to the well known SNN with an AlexNet backbone. Their performance is compared in the experimental section along with several few-shot learning approaches in the context of one-shot recognition. For detection, the proposed recognition approach is adapted inspired by one-stage multi-box detection approaches similar to that introduced by YOLO <cit.> <cit.>. The main contributions of this paper are:

- A scalable framework for object one-shot recognition and detection inspired by Siamese Neural Networks and anchor based one-stage multibox detectors, that additionally do not require support sets or additional training/fine-tuning for unseen classes.

- The introduction of two novel Joint Neural Networks architectures for one-shot recognition and detection respectively.



§ RELATED WORK


Previous to modern machine learning approaches, the initial approaches to engage in the one-shot object detection and recognition problems were based on image processing techniques. The most popular image processing based approaches relied on the localisation and description of interest points or keypoints, as it is the case of SIFT <cit.> and SURF <cit.> for object detection. Approaches based on keypoint detection and description are often designed to offer feature scale, rotational, and illumination invariance. When applied in realistic scenarios, one of the main withdraws of such approaches is the limited capability of matching feature keypoints under illumination or structural variations in particular when the background scenery is complex.

As the performance of image processing based approaches reached a plateau new machine learning approaches were introduced to address the one-shot problem. The goal of machine learning algorithms is to discover statistical structures within data, but building modern models such as convolutional neural networks (CNNs) from scratch requires a large amounts of labeled training data <cit.>. Following <cit.>, the seminal work towards one-shot learning dates back to the early 2000’s with work in <cit.> where the authors developed a variational Bayesian framework for one-shot image classification using the premise that previously learned classes can be leveraged to help forecast future ones when very few examples are available from a given class <cit.> <cit.>. However, these approaches are still based on keypoints, inheriting the same drawbacks and the problem remains unsolved for real world scenarios.

Data synthesis and transfer learning are commonly used techniques to support models carrying out the detection or recognition tasks when the labeled training data is limited. Data synthesis focuses on producing artificial data either manually from scratch or using advanced data manipulation techniques to produce novel and diverse training examples.  When creating new exemplars from existing instances, the available data must be enough to render novel and diverse instances to produce novel and diverse training examples <cit.>. In general, the data synthesis approach serves well for cases in which at least a few samples of the target categories are available but its effectiveness is limited for one-shot object recognition/detection.

Transfer learning is one of the most relevant tools in machine learning, proposed to solve the insufficient training data problem. It aims to transfer the knowledge from a source domain to a target domain by relaxing the assumption that the training data and the test data must be independent and identically distributed. This leads to a great positive effect on many domains that are difficult to engage with because of insufficient training data <cit.><cit.>. Fine-tuning remains the method of choice for transfer learning with neural networks in which a model is pre-trained on a source domain (where data is often abundant), the output layers of the model are adapted to the target domain, and the network is fine-tuned via back-propagation. This approach was pioneered in <cit.> by transferring knowledge from a generative to a discriminative model, and has since been generalized with great success <cit.><cit.><cit.>. Traditionally, this method also requires at least a few samples of the target domain categories to be available in order to perform the fine-tuning training process, which becomes a limiting factor when dealing with one-shot detection or recognition. 

Recently, a set of approaches for few-shot recognition and suitable for one-shot object recognition have been proposed <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. The work presented in <cit.> proposes a general and model-agnostic meta-learning algorithm. The algorithm is model-agnostic in the sense that it can be directly applied to any learning problem and model that is trained with a gradient descent procedure. The key idea behind the proposed approach is to train the model’s initial parameters such that the model has maximal performance on a new task after the parameters have been updated through one or more gradient steps computed with a small amount of data from that new task.

A set of architectures called Siamese Neural Networks (SNNs) have been also proposed to tackle to one-shot recognition task. A SNN consists of twin neural networks with tied parameters that takes two inputs and join them using an cost function at the top. This function computes a distance metric between resulting the highest-level feature representation on each side. Once trained, SNNs can then capitalise on powerful discriminative features to generalise the predictive power of the network not just to new data, but to entirely new classes from unknown distributions. Furthermore, the shared weights constraint on the twin CNN's guarantees that two extremely similar images could not possibly be mapped by their respective networks to very different locations in feature space given that each network computes the same function <cit.>.

Siamese Neural Networks have been applied for a wide set of tasks including character recognition <cit.>, person re-identification <cit.>, face verification <cit.> and tracking <cit.>; given their balance between accuracy and speed <cit.> for one-shot recognition. Traditionally, Siamese networks were trained with the so called “contrastive loss” <cit.> <cit.>. The main idea here is to construct a Euclidean space where the outputs of the sister networks are close to each other for matching pairs while simultaneously negative pairs are pushed away <cit.>.

Although, most of the work found in the literature is focused on one-shot object recognition, recent research works have started pioneering possible solutions to the challenging one-shot detection problem <cit.><cit.> and extending it to fields such as semantic segmentation <cit.> and visual tracking<cit.><cit.>. In contrast to the work presented in this paper, the aforementioned approaches are based on two-stage object detection approaches.

This work introduces an approach inspired by SNNs to tackle the one-shot object recognition problem, that additionally uses concepts from one-stage detectors such as YOLO <cit.> to extend its application to the one-shot object detection problem. Following the traditional object detection literature case where enough data is available for training, one-stage detectors offer a trade-off between accuracy and speed favorable for real world applications. The novel architecture proposed is based on the idea of allowing the model to Figure out by itself the convolutional activations that lead to pattern matching and locations definition, in contrast to the common encoding comparison at the top of the Siamese sister networks.



§ PROPOSED APPROACH


The fundamental idea behind the proposed joint neural networks is to allow the convolutional layers to learn the activations that define matches between pairs of images, instead of having a single comparison layer at the top of the architecture as in SNNs.

For this purpose we define joint convolutional layers, which are the backbone of Joint neural networks and can be used within any convolutional neural network architecture. The so called joint layers are convolutional layers like those found in common convolutional neural networks; where, the feature vector taken as input is the result of a concatenation operation between the previous convolutional layers on each branch of the Joint neural network. Figure <ref> illustrates the structure of a joint layer. The output of the proposed joint layers is defined to match the architecture used as backbone of the Joint neural network. In this work the AlexNet and DarkNet19 architectures are used to tackle the recognition and detection problems respectively.



In Figure<ref> f_i(x^(1)) and f_i(x^(2)) are the feature vectors outputted by the ith convolutional layer for the first and second branch respectively. F_j(f_i(x^(1)) ⊕ f_i(x^(2))) is the feature vector outputted by the jth joint layer. And ⊕ is the standard concatenation operation merging the output vectors f_i(x^(1)) and f_i(x^(2)).



 §.§ Recognition


When undertaking the one-shot recognition problem the main goal is to perform pattern matching for a pair of images x^(1), x^(2); outputting a predicted similarity metric P in the same manner as traditional Siamese neural networks. Given that AlexNet is an architecture commonly used as the backbone of SNNs <cit.> we adopt the same base architecture for demonstration and comparison purposes.



Figure <ref> depicts the proposed architecture showing the two branches of the network. The main branch of the architecture at the top of Figure <ref> is built over the AlexNet architecture with the additional joint layers that will integrate outputs from the second branch at the bottom, thus supporting information sharing between the network branches. The convolutional layers in the secondary branch of the architecture share weights with the main branch, the same way as sister networks in traditional SNN approaches. Additionally, the last fully connected layer of the architecture consists of a single neuron that will define the similarity metric P. The joint layers will take as input the concatenation of the previous outputs on both branches of the network and keep the AlexNet structure for following layers. Table <ref> shows a detailed description of the architecture layers.



In contrast to the traditional contrastive loss function
this work uses the binary cross entropy (BCE) loss function for training the proposed architecture as defined in equation <ref>. The loss function is defined to comply with the SNN architecture previously defined and the specified one-shot constraints, where, the model is trained to output a similarity measure that is small for matches and big for dissimilar mismatches.


    ℓ(x^(1),x^(2)) = -1/N∑_i=1^N y_i(x^(1),x^(2)) · log(p(x^(1),x^(2)))
    
    + (1 - y_i(x^(1),x^(2)) · log(1 - p(x^(1),x^(2))),


where y is the matching label between the inputs x^(1),x^(2), which equals to 1 for matching input pairs and 0 otherwise. p(x^(1),x^(2)) is the predicted label and N is the batch size.

In order to achieve the one-shot detection task the proposed model is trained with categories ignored during testing. For this purpose the training data loader will select a set of random queries for each epoch with one single query image per batch matched with a random target image that contains the query image 50% of times. Each query and target image is then passed through the model and paired with the ground truth annotations to compute the loss function.



 §.§ Detection



The presented recognition approach is adapted to perform multi-box one-shot object detection given a query and target images containing a pattern of interest from a category unseen during training. The proposed Joint Neural Network is defined as a multi-box detector based on the Darknet19 architecture <cit.>, taking the query and target images x^(1), x^(2) as input and outputting a tensor P̂ representing the locations where the object of interest may be present in x^(2).

Following <cit.> we define a S × S grid over the target image x^(2). Each grid cell predicts B anchor window coordinates and confidence score (x, y, w, h, conf); where, the output predictions are encoded as a S × S × (B × 5) tensor. In contrast with <cit.> the category to be detected in the target image x^(2) is dictated by the input query image x^(1), so the output tensor does not require additional position to define prediction classes. The confidence scores are trained as intersection over union predictor which will have a higher value when its respective anchor window correctly contains the object of interest.


The approach and notation for defining the anchor windows are inspired by <cit.> is used to define the anchor windows. For each anchor window in the S × S grid the model yields to five output values t_x, t_y, t_w, t_h, and t_o. The cell offsets from the top left corner of the image are defined as (c_x, c_y) and the bounding box prior has width and height p_w, p_h. The resulting predictions are obtained using equations <ref> - <ref> as illustrated in Figure <ref>.


    b_x = σ(t_x) + c_x


    b_y = σ(t_y) + c_y


    b_w = p_we^t_w


    b_h = p_he^t_h


    conf = σ(t_o)






The proposed model is essentially based on Darknet19 <cit.>, adapted to take two inputs on separated branches. Both branches share weights and are connected by convolutional layers (joint layers) at several points on the pipeline, ending in a single convolutional layer encoding the S × S × (B × 5) predictions tensor; where S=14 for the proposed architecture. Figure <ref> shows the base network structure for illustrative purposes in accordance with Table <ref>. The architecture used for experiments slightly differs from Figure <ref> following the results presented in section <ref>.



The input sizes for the target and query images are 448 × 448 and 224 × 224 respectively. By adapting the architecture to apply the first pooling layer on the target sister network only, the output sizes for the first 2 convolutional layers become even. This small adaptation makes the concatenation operation and further convolutional layers computation straightforward. In addition,the called joint convolutional layers in Figure <ref> are designed to take the concatenation of the previous convolutional layer outputs for each sister network as input and accommodate its output following the DarkNet19 architecture <cit.> in the same way as the residual residual layer.Finally, the last convolutional layer outputs a 14 × 14 tensor composed of B × 5 filters that will encode the model predictions.

The loss function ℓ used for this work is defined as the weighted sum of losses for localisation and confidence components. The first term being the location loss ℓ_loc, deals with how close the predicted box location is to the ground truth annotation; then, the confidence loss term ℓ_conf represents how certain the model is that the pattern of interest is located inside a specific anchor box.


    ℓ = ℓ_loc + ℓ_conf,



    ℓ_loc = λ_coord∑_i=1^S^2∑_j=1^B1_ij^obj  (y - y_pred)^2,



    ℓ_conf = λ_obj∑_i=1^S^2∑_j=1^B[1_ij^obj c· log(c_pred)+
    
    (1-c)· log(1-c_pred)] + 
    λ_noobj∑_i=1^S^2∑_j=1^B[1_ij^noobj c· log(c_pred)+
    
    (1-c)· log(1-c_pred)],




where S is the size of the grid; B is the number of anchors defined per grid cell; y and y_pred are the ground truth and prediction tensors containing the b_x, b_y, b_w, b_h values; c and c_pred are the ground truth and predicted confidences (conf); 1_ij^obj is equals to 1 where the anchor window j at cell i has an Intersection of Union IoU>0.5 with the target pattern and 0 otherwise; 1_ij^noobj is 1_ij^obj complement; and λ_coord, λ_obj, and λ_noobj are weights to deal with the relevance of each term in the loss function. In contrast to <cit.> we use the Binary Cross Entropy function for the confidence terms along with the λ_obj, and λ_noobj weights to deal with the learning priority when keeping the true positives or discarding false positives. The data loading for learning follows the same protocol as in recognition; where the target has a 50% chance of containing the query pattern.



§ EXPERIMENTS


This section presents the performance evaluation of the proposed approach initially for the recognition task using the AlexNet adaptation of the joint neural network architecture and afterwards using the Darknet variation for object detection. The experiments are built over three different datasets: The QMUL-OpenLogo, MiniImagenet, and pascalVOC datasets.

QMUL-OpenLogo Dataset: The OpenLogo dataset introduced in <cit.> is presented as a challenging benchmark to evaluate logo detection methods under realistic conditions. The dataset contains approximately 27,000 images from more than 300 logo classes captured from different sources. Figure <ref> (row 2) presents a small sample set that depicts the variance between the captures and the classes in the dataset.

The OpenLogo dataset exploits existing logo detection datasets; i.e., the dataset contains logo images from several different existing datasets and heterogeneous sources in an attempt to reproduce realistic scenarios <cit.>. These datasets are: FlickrLogos-27 <cit.>, FlickrLogos-32 <cit.>, Logo32plus <cit.>, BelgaLogos <cit.>, WebLogo-2M <cit.>, Logo-In-The-Wild <cit.>, SportsLogo <cit.>. To the best of our knowledge, the OpenLogo dataset has not been used by any published work in the context of one-shot recognition or detection.

The dataset is split into 210 categories for training and 125 categories for testing. For recongition tests both input images are cropped, while only the query image is cropped for detection experiments. Again, there is no category overlap between the training and testing sub-sets which guarantees that the embedded CNN has no knowledge of any of the tested query inputs and complies with the one-shot constraints. The categories on the QMUL-OpenLogo dataset are unbalanced, with some of the logos having as little data as 2 instances. Given that the proposed architecture works with pairs of images it is possible to make enough pairs testing purposes while keeping data balance. Therefore, approximately 20,000 input pairs are formed from the 125 testing categories available with a 50/50 split on matches and mismatches for the unseen testing categories.

MiniImageNet Dataset: The MiniImageNet dataset is derived from the ImageNet <cit.>, first proposed by <cit.>. MiniImageNet consists of 60,000 84 × 84 images representing 100 classes with 600 instances each. The exact 600 images that comprise each of the 100 classes in miniImageNet can be found at <https://goo.gl/e3orz6> and the images can be directly downloaded from <https://github.com/yaoyao-liu/mini-imagenet-tools>. The miniImageNet dataset splits 80 classes for training and 20 for testing. Again, the training and testing classes do not overlap following the one-shot problem constraints. Figure <ref> presents (row 1) a set sample images from the miniImageNet dataset.

Pascal VOC Dataset: Pascal VOC is a collection of datasets for object detection <cit.> used for the Pascal Visual Object Classes challenge. The most commonly used combination for benchmarking traditional object detection approaches is using the VOC 2007 trainval and 2012 trainval sets for training and the VOC 2007 test set for validation. These datasets contain 10.000 and 22.500 images respectively, and each image is annotated with one or several labels, corresponding to 20 object categories. Figure <ref> (row 3) presents a set of sample images from the VOC dataset.

COCO Dataset: COCO is a dataset that places the question of object recognition in the context of the broader question of scene understanding. The dataset is built from images of complex everyday scenes containing common objects in their natural context. The COCO dataset contains over a hundred thousand annotated images over 80 categories. The object categories in the data include animals, transport vehicles and furniture <cit.>.



 §.§ Recognition results



In order to evaluate the performance of the proposed approach for the one-shot recognition task results are presented using true positive rates, false positive rates, accuracy, precision, and the ROC curve. In this subsection the proposed JNN based recognition approach is compared with the traditional SNN approach using the AlexNet architecture backbone over the OpenLogo dataset. Additionaly, the proposed approach is compared with few-shot learning methods <cit.>, <cit.>, <cit.>, <cit.>, <cit.> over the MiniImageNet dataset.

Given that different thresholds over the learned similarity metric P lead to differences in the performance of the binary classifier, a set of 20 thresholds is used over the similarity metric within the range of responses returned by the Siamese neural network in order to compute the ROC curve. The threshold with the best accuracy is latter used for comparison purposes.

The proposed AlexNet variation of the architecture was trained 200 epochs with a 0.005 learning rate. Each training batch contained 64 pairs of images randomly selected from the training set. The training process was around 9 hours to complete on a GeForce GTX 1080. The proposed approach as well as the Siamese AlexNet architecture used for experiments are implemented using the pythorch library.

Figure <ref> shows the performance of the proposed JNN for logo recognition in terms of the ROC curve obtained by thresholding the similarity metric S. For each one of the models, the similarity metric is thresholded in the range of responses of the model to compute and plot the true positive (TPR) and false positive (FPR) rates. The accuracy, precision, and area under the ROC curve (AUC), reported for each method is measured for the threshold with the highest accuracy. 



Table <ref> shows the result scores for the approaches in Figure <ref>. The proposed JNN shows a significant improvement over the AUC, FPR and accuracy scores when compared with the standard SNN AlexNet approach (AUC=0.76, FPR=0.25, acc=0.70).



Table <ref> shows the accuracy scores for state-of-the-art one-shot learning approaches assessed in miniImageNet for object recognition. The proposed approach outperforms these methods without any fine-tuning or learning associated to the novel tested categories. This means that, compared to few-shot based learning, the presented JNN can fit a wider range of application scenarios, with reasonably accurate prediction outcomes at a significant smaller cost.




Figure <ref> depicts the predicted value P for a few arbitrarily selected patterns from the OpenLogo and MiniImageNet datasets unseen during training.

 







 §.§ Detection results



The proposed JNN based detection approach is tested under four experimental settings using the QMUL-OpenLogo, VOC, and COCO datasets. First, we follow the experiment proposed in <cit.> by training the proposed model in 16 of the 20 classes of the VOC dataset and testing on the additional 4 classes. The model is trained using the VOC 2007 and 2012 train&val set and evaluated using the VOC 2007 test set. Second, we use the experimental approach presented in <cit.>; where the proposed model is trained in the COCO dataset and tested in the VOC excluding the overlapping classes during training. Third, following <cit.> the model is trained and tested using the COCO dataset using four different splits for the training and testing classes. Finally, the model is tested on the QMUL-OpenLogo dataset using a 60/40 split for training and testing classes selected arbitrarily and complying with the one-shot constraints. 

The proposed JNN is trained using the SGD optimizer with a learning rate equals to 0.0001, a 0.9 momentum for 160 epochs, with a 16 batch size, over a single GeForce GTX 1080 GPU. All the experiments are performed using an IoU threshold of 0.5.

Table <ref> shows the results for the first experimental setting, along with the train and test class splits for the VOC dataset as in <cit.>. Under this experimental settings, results show a 13% mAP difference to the top performing approach <cit.>. However, it should be noted that in contrast to the results presented from the previous approaches we do not pre-train our model. The results table presented in <cit.> use the pretrained models on the full ImageNet dataset where the test classes may be present, meaning that the one-shot constraint is not in strict use.


The authors in <cit.> go further, experimenting with one-shot object detection using the COCO and VOC datasets. Following their approach, the second experimental setting presented in this paper for one-shot object detection trains the proposed model on the COCO dataset excluding the VOC classes used for testing. Table <ref> shows the results compared to the M-FCOS approach <cit.>. Results show a mAP improvement using the JNN based detection JNN based detection approach presented in this paper when compared to the top performing approach <cit.> in Table <ref>.

In addition, the performance of the JNN is also compared to the state-of-the-art approaches using the experimental approach presented in <cit.> and <cit.>. In this case, four different splits of training and testing categories are defined over the COCO dataset. The fully detailed category splits can be found in the first appendix of <cit.>. Each training split will contain 60 categories leaving 20 for testing. Table <ref> shows the results of the proposed approach compared to the SiamRPN <cit.> and coAcoE <cit.> approaches. Under this experimental setting our approach is down 8.5% mAP when compared with the CoAE approach <cit.>.



Finally, the proposed approach is assessed in the context of one-shot logo detection using the QMUL-OpenLogo dataset. Here, 210 categories are arbitrarily selected for training and 125 for testing in the same way a the recognition experiments in section <ref>. Table <ref> shows the average precision (AP50) results for the top performing 15 unseen classes and the mean average precision (mAP) for the all 125 unseen classes used for testing. Figure <ref> show some results of the proposed approach in action.



 



 §.§ Ablation study



This paper introduces Joint Neural Networks as an adaptable framework that can be used with any defined backbone architecture to perform the one-shot object recognition or detection tasks, where the joint layers are the key concept used build the Joint Neural Networks. This section explores how the use of joint layers on specific locations on the architecture may affect the performance of the proposed approach.

Figure <ref> presented in section <ref> shows the proposed convolutional and joint layers used to tackle the one-shot object detection task. In order to test the firstly proposed architecture, Table <ref> shows the performance of the proposed framework under different joint layer configurations. The test are carried out using the first experimental settings described in section <ref> (Table <ref>).




Each Joint Layer (JL) in Figure <ref> is depicted as a column in Table <ref>. The experiments explore the performance of the architecture when adding joint layers sequentialy as well as for groups of two and three layers. Results show that for the one-shot object detection task using the VOC dataset the combination of the joint layers 1, 2, and 4 in DarkNet19 architecture yields the best overall mAP performance (56.1%).



§ CONCLUSION


This paper presents a novel convolutional neural network based framework named Joint neural networks to tackle the challenging tasks of one-shot object recognition and detection. Joint neural networks are inspired by Siamese neural networks and one-stage multi-box object detection approaches. The proposed joint layers are the fundamental concept supporting the data sharing between the branches of the presented Joint architectures. This approach is developed for one-shot object recognition and detection tasks where traditional deep learning approaches fall short due to the lack of training data available to build state-of-the-art models.

Results indicate that the proposed approach can compete with the top performing state-of-the-art recognition and detection approaches, and even outperform them under the right experimental settings by using the most basic building blocks of convolutional neural networks. The work presented in this paper leaves open room to experiment with new architectures and joint layer configurations to further improve the state-of-the-art performance for the one-shot object recognition and detection tasks.



99

paper1
A. Jain, R. P. W. Duin, Jianchang Mao. “Statistical pattern recognition: a review". IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4–37, Jan 2000. 

Jref1
Z. Zhao, P. Zheng, S. Xu and X. Wu, “Object Detection With Deep Learning: A Review", IEEE Trans. Neural Netw. Learn. Syst, vol. 30, no. 11, pp. 3212-3232, Nov. 2019.

paper2
A. Antoniou, A. Storkey, H. Edwards. “Data augmentation generative adversarial networks", 2017. [Online] Available: <https://arxiv.org/abs/1711.04340>.

paper3
A. Nguyen, J. Yosinski, J, Clune. “Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images", December 2014. [Online] Available: <https://arxiv.org/abs/1412.1897>.

YOLO1
Redmon, J., Divvala, S., Girshick, R., Farhadi, A. “You only look once: Unified, real-time object detection", In Proc. CVPR, Las Vegas, NV, USA 2016.

YOLO2
J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger", In Proc. CVPR, 2017, pp. 6517–6525.

SIFT
Lowe, David G. “Distinctive Image Features from Scale-Invariant Keypoints", IJCV, 60(2):91–110, November 2004.

SURF
Bay, Herbert, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. “Speeded-Up Robust Features (SURF)", Computer Vision and Image Understanding, Volume 110, Issue 3, June 2008.

paper5
G. Koch, R. Zemel, R. Salakhutdinov. “Siamese neural networks for one-shot image recognition". ICML Deep Learning workshop, 2015.

paper6
L. Fe-Fei, R. Fergus, P. Perona. “A Bayesian approach to unsupervised one-shot learning of object categories". In Proc. ICCV, vol. 2, pp. 1134-1141, 2003.

paper7
L. Fei-Fei, R. Fergus, P. Perona. “One-shot learning of object categories".  IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no.4, pp. 594–611, April 2006.

thesis37
Nikolenko, S. I. “Synthetic data for deep learning", 2019. [Online]  Available: <https://arxiv.org/abs/1909.11512>.

thesis36
N. Tajbakhsh. “Convolutional neural networks for medical image analysis: Full training or fine tuning?", IEEE Trans. Med. Imag, vol. 35, no. 5, pp. 1299-1312, May 2016. 

Jref2
L. Shao, F. Zhu and X. Li, “Transfer Learning for Visual Categorization: A Survey", IEEE Trans. Neural Netw. Learn. Syst, vol. 26, no. 5, pp. 1019-1034, May 2015.

thesis31lost
R. Girshick, J. Donahue, T. Darrell, and J. Malik. “Rich feature hierarchies for accurate object detection and semantic segmentation". Proc. CVPR, 2014.

thesis32lost
M. D. Zeiler and R. Fergus. “Visualizing and understanding convolutional networks". In Proc. CVPR, 2014.

thesis32lost2
G. E. Hinton and R. R. Salakhutdinov. “Reducing the dimensionality of data with neural networks". Science, 313(5786):504–507, 2006.

thesis32lost3
Y. Wang, D. Ramanan, and M. Hebert, “Growing a brain: Fine-tuning by increasing model capacity", in IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit, Jul. 2017, pp. 3029–3038.

46fs1
C. Finn, P. Abbeel, and S. Levine. “Model-agnostic metalearning for fast adaptation of deep networks". In International Conference on Machine Learning, 2017

46fs2
J. Snell, K. Swersky, and R. S. Zemel. “Prototypical networks for few-shot learning". In Neural Information Processing Systems, 2017. 

46fs3
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. “A simple neural attentive meta-learner". In International Conference on Learning Representations, 2018.

46fs4
B. N. Oreshkin, P. Rodriguez, and A. Lacoste. “TADAM: Task dependent adaptive metric for improved few-shot learning", May 2018. [Online] Available: <https://arxiv.org/abs/1805.10123>

46fs5
Sun, Q., Liu, Y., Chua, T.S., Schiele, B. “Meta-transfer learning for few-shot learning". In Proc. CVPR, 2019.

Jref3
H. -G. Jung and S. -W. Lee, “Few-Shot Learning With Geometric Constraints", IEEE Trans. Neural Netw. Learn. Syst, vol. 31, no. 11, pp. 4660-4672, Nov. 2020.

paper13
E. Ahmed, M. Jones, T. K. Marks. “An improved deep learning architecture for person re-identification". In Proc. CVPR, 2015.

paper14
Y. Taigman, M. Yang, M. Ranzato, L. Wolf: Deepface. “Closing the gap to human-level performance in face verification". In Proc. CVPR, 2014.

paperref24 
Z. Zhang, H. Peng, and Q. Wang. “Deeper and wider siamese networks for real-time visual tracking". In Proc. CVPR, 2019.

paperref23 
L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr. “Fully-convolutional siamese networks for object tracking". In Proc. ECCV workshop, 2016.

paperref26
Bromley, J., Bentz, J. W., Bottou, L., Guyon, I., LeCun, Y., Moore, C., et al. “Signature verification using a Siamese time delay neural network". Proc. Advances in Neural Information Processing Systems, 6, pp. 737-744, 1994.

paperref27
S. Chopra, R. Hadsell, and Y. LeCun. “Learning a similiarty metric discriminatively, with application to face verification". In Proc. CVPR, 2005.

paperref28
M. Harandi, S. R. Kumar, and R. Nock. “Siamese Networks: A Thing or Two to Know". 2017. [Online]  Available: <https://pdfs.semanticscholar.org/c03c/e09b419b6a15c1228e344a900d8c54bddc78.pdf>

coAE
Hsieh, T.I., Lo, Y.C., Chen, H.T., Liu, T.L. “One-Shot Object Detection with Co-Attention and Co-Excitation"". In Proc. NeurIPS, 2019.

MFCOS
Xiang Li, Lin Zhang, Yau Pun Chen, Yu-Wing Tai, Chi-Keung Tang. “One-Shot Object Detection without Fine-Tuning", 2020. [Online]  Available: <https://arxiv.org/abs/2005.03819>.

SiamRPN
Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. “High performance visual tracking with siamese region proposal network". In Proc. CVPR, 2018, Salt Lake City, UT, USA, June 18-22, 2018. 

SiamFC
Miaobin Cen and Cheolkon Jung. “Fully convolutional siamese fusion networks for object tracking". In Proc. ICIP 2018, Athens, Greece, October 7-10, 2018, pages 3718–3722.

CompNet
Tengfei Zhang, Yue Zhang, Xian Sun, Hao Sun, Menglong Yan, Xue Yang, and Kun Fu. “Comparison network for one-shot conditional object detection", 2019. [Online]  Available: <https://arxiv.org/abs/1904.02317>.

paper24
Hang Su, Xiatian Zhu, Shaogang Gong. “Open Logo Detection Challenge". In Proc. BMVC, 2018.

52
Y. Kalantidis, LG. Pueyo, M. Trevisiol, R. van Zwol, Y. Avrithis. “Scalable Triangulation-based Logo Recognition". In Proc. ICMR, Trento, Italy, April 2011.

53
S. Romberg, L. G. Pueyo, R. Lienhart, and R. Van Zwol. “Scalable logo recognition in real-world images". in Proc. ICMR, 2011.

54
Simone Bianco, Marco Buzzelli, Davide Mazzini, and Raimondo Schettini. “Deep learning for logo recognition". Neurocomputing, 245:23–30, 2017.

55
Alexis Joly and Olivier Buisson. “Logo retrieval with a contrario visual query expansion". in Proc. ACM Multimedia, Beijing, China, 2009, pp. 581–584.

56
Hang Su, Shaogang Gong, and Xiatian Zhu. “Weblogo-2m: Scalable logo detection by deep learning from the web". In Proc. ICCV Workshop, 2017.

57
Andras Tüzkö, Christian Herrmann, Daniel Manger, and Jürgen Beyerer. “Open set logo detection and retrieval", 2017. [Online] Available: <https://arxiv.org/abs/1710.10891>.

58
Yuan Liao, Xiaoqing Lu, Chengcui Zhang, Yongtao Wang, and Zhi Tang. “Mutual enhancement for detection of multiple logos in sports videos". In Proc. ICCV, 2017.

44.1
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. “ImageNet: A Large-Scale Hierarchical Image Database". In Proc. CVPR, 2009.

47
Vinyals, Oriol, Blundell, Charles, Lillicrap, Timothy, Kavukcuoglu, Koray, and Wierstra, Daan. “Matching networks for one shot learning", 2016. [Online] Available: <https://arxiv.org/abs/1606.04080>

VOC
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. “The Pascal Visual Object Classes (VOC) Challenge". IJCV, pages 303–338, 2010.

COCO
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick. “Microsoft COCO: Common objects in context". In Proc. ECCV. 2014.

XiangLi
Xiang Li, Lin Zhang, Yau Pun Chen, Yu-Wing Tai, Chi-Keung Tang. “One-Shot Object Detection without Fine-Tuning", 2020. [Online] Available: <https://arxiv.org/abs/2005.03819>





 Paper Title: PLANesT-3D: A new annotated dataset for segmentation of 3D plant point clouds.


WIP: An Engaging Undergraduate Intro to Model Checking in Software Engineering Using 
    
  
    Konstantin Läufer1,
    Gunda Mertin2,
    and
    George K. Thiruvathukal1
  
  
    1Software and Systems Laboratory, 
      Department of Computer Science,
      Loyola University Chicago

      {laufer,gkt}@cs.luc.edu
    

    2Institute for Software Engineering and Programming Languages,
      University of Lübeck

      gunda.mertin@student.uni-luebeck.de
  

    August 5, 2024
============================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Creation of new annotated public datasets is crucial in helping advances in 3D computer vision and machine learning meet their full potential for automatic interpretation of 3D plant models.  In this paper, we introduce PLANesT-3D; a new annotated dataset of 3D color point clouds of plants. PLANesT-3D is composed of 34 point cloud models representing 34 real plants from three different plant species: Capsicum annuum, Rosa kordana, and Ribes rubrum. Both semantic labels in terms of "leaf" and "stem", and organ instance labels were manually annotated for the full point clouds. As an additional contribution, SP-LSCnet, a novel semantic segmentation method that is a combination of unsupervised superpoint extraction and a 3D point-based deep learning approach is introduced and evaluated on the new dataset. Two existing deep neural network architectures, PointNet++ and RoseSegNet were also tested on the point clouds of PLANesT-3D for semantic segmentation. 









§ INTRODUCTION



Cultivation of horticultural crops is one of the most labor-intensive economic activities, and is central to global food security 
<cit.>. The accuracy and efficiency of horticultural practices, such as pruning, weed control, disease management, and harvesting directly influence the yield and quality of the crops <cit.>. The increasing global demand for high-quality crops, hence the demand for intensive labor motivates development of technologies for automation in horticulture <cit.>. Automated horticultural practices are expected to adapt to plant traits, which are diverse across plant species <cit.>. Automation of trait measurements is also key to high throughput plant phenotyping, a fundamental process in plant research, breeding, and crop management <cit.>. Modeling of traits within a population, and capturing the structure of single plants or plant communities for particular operations have become major applications of computer vision <cit.>.



3D computer vision is particularly relevant in many automated horticultural operations. Robotic systems designed to perform pruning, weeding, and harvesting require correct identification of plant parts, and accurate estimation of the location, size, shape, and orientation of the structures in 3D space <cit.>. Advances in 3D sensing technologies and machine learning fuel the development of algorithms for automatic segmentation of plants into their structural parts as well as organ-level trait estimation <cit.>.

Despite the considerable progress in 3D geometry modeling and analysis of plants in the last decades <cit.>, creation of benchmark datasets, a fundamental component for advancement of relevant research, is still lagging behind compared to other domains <cit.>. Annotated public datasets of plant models are of particular importance both for comparison of competing algorithms and also for providing training data to machine learning processes <cit.>. 




In the last decade, open datasets of annotated 2D plant images have been proliferated to assist evaluation of performance and development of machine learning models for plant phenotyping <cit.>. However, open datasets containing annotated and complete 3D plant models are rare. Table <ref> gives a list of such existing datasets. Plant3D <cit.> contains a total of 714 3D laser scans of Tomato, Tobacco, Sorghum, and Arabidopsis plants obtained within 20-30 days of development. Information on the growth conditions of the plants are also provided. The 3D models do not contain color information. <cit.> manually labeled 54 models of the Plant3D data into stem and lamina points, and evaluated a range of machine learning approaches on this subset. <cit.> annotated 546 of the plants of the Plant3D dataset with semantic and instance labels. The labels are available upon request from <cit.>. ROSE-X dataset <cit.>
contains 3D models of 11 rosebush plants acquired through X-ray computer tomography. The voxels in the volumetric models were labeled into three semantic categories: "Leaf", "Stem", and "Flower". The point cloud versions of the volumetric models are also available. Pheno4D <cit.> is a dataset of 3D point clouds of 7 maize plants and 7 tomato plants. The plants were scanned with a laser scanner at different growth stages resulting in 244 point clouds. 126 of them were manually annotated with semantic and instance labels. The Soybean-MVS dataset <cit.> is fundamentally different from the rest in terms of data acquisition modality. The plants were captured with an RGB camera in a controlled setup and their corresponding point clouds were created through multi-view stereo. A total of 102 point cloud models of five different soybean varieties were reconstructed at 13 stages of the whole growth period.




These 3D plant datasets have been often used for assessing off-the-shelf machine learning tools for semantic and instance segmentation <cit.>. The datasets also enabled training and evaluation of new approaches specifically developed for plant data <cit.>, including deep learning architectures such as PSegNet <cit.>, PlantNet <cit.>, RoseSegNet <cit.>, and FF-Net <cit.>.


Although smaller in terms of the number of models as compared to Pheno4D, Plant3D, and Soybean-MVS (Table <ref>), the new dataset (PLANesT-3D [The PLANesT-3D dataset is publicly available at <https://www.dropbox.com/s/8gl0r0zo0j6kmdo/PLANesT-3D.rar?dl=0>]) introduced in this work contributes to the diversity of publicly available annotated 3D plant models in terms of acquisition modality, number of distinct plants, and inclusion of other species. The data is acquired with an RGB camera and reconstructed through structure from motion (SfM) and  multi-view steoroscopy (MVS). Together with the Soybean-MVS dataset, the PLANesT-3D dataset will balance the dominant acquisition modality, which is laser scanning. Reconstruction with SfM and MVS does not require expensive equipment or elaborate acquisition setups, while yielding both geometric and textural information. 

While Pheno4D and Soybean-MVS datasets are extremely valuable since they track the development of individual plants across time, the number of distinct plants  are 14 for Pheno4D and 5 for Soybean-MVS. The PLANesT-3D dataset includes models of 34 distinct plants belonging to three species: namely Capsicum annuum (pepper), Rosa kordana (rose), and Ribes rubrum (ribes). Also, point clouds from two species (pepper and ribes), 3D models of which were not available before, are added to the public domain. The diversity of training and evaluation data in terms of acquisition modality, noise levels, plant species, plant instances, plant architecture, organ geometry, etc. plays an important role in assessing the robustness as well as the generalization ability of machine learning approaches.

The objective of this publication is twofold: First is to introduce PLANesT-3D, a new dataset containing complete 3D color point clouds of plants together with their semantic and instance labels. Second is to present a new segmentation approach, the SP-LSCnet, that combines an unsupervised clustering scheme and an adaptive network for point cloud classification. We also provide semantic segmentation results on the dataset with two point-based deep learning architectures, PointNet++ and RoseSegNet. The main contributions of this work can be summarized as follows:



  * We introduce PLANesT-3D, a new dataset containing RGB point clouds representing 34 real plants from three species.

  * We propose SP-LSCnet as a new semantic segmentation approach where an unsupervised clustering method based on t-distributed stochastic embedding (t-SNE) is combined with a point cloud classifier network. The classifier network employs two adaptive modules to adjust local region organization for feature extraction. 

  * We tested RoseSegNet, a semantic segmentation algorithm we had previously developed for plant data <cit.> on the new dataset, and demonstrated that it is effective without requiring hyperparameter re-adjustment.



The rest of this article is organized as follows. In Sections <ref> and <ref>, we describe the data acquisition, reconstruction, and pre-processing steps leading to the creation of 3D point clouds. We give detailed information about the PLANesT-3D dataset in Section <ref>. The segmentation methods evaluated in this work are described in Section <ref>. Our new segmentation approach, SP-LSCnet, is introduced in Section <ref>. The performances of the semantic segmentation methods on the PLANesT-3D dataset are reported in Section <ref>. A discussion of the work is provided in Section <ref>, followed by our conclusion in Section <ref>.







§ MATERIALS AND METHODS





 §.§ Data acquisition



In order to obtain full 3D plant models, 2D color images of 34 plants were acquired using a handheld DSLR camera (EOS 6D MARK II, Canon, Japan). The acquisition was performed on 10 potted pepper plants (Capsicum annuum), 10 potted rose plants (Rosa kordana), and 14 ribes plants (Ribes rubrum), planted directly to the soil in a greenhouse. Multiple color images of resolution 6240 × 4160  were captured manually around each plant via positioning the camera so as to cover as much plant surface as possible with a high degree of overlap. Three examples of the sets of camera locations and orientations are illustrated in Fig. <ref>. 



Table <ref> gives the number of images used to reconstruct the 3D plant point clouds. Three pairs of Ribes were very closely planted (Ribes 02 & 12, Ribes 04 & 13, Ribes 05 & 14). A single point cloud was reconstructed for each pair using the set of images where the pair appeared together. After reconstruction, noise removal and pose normalization, the two plants were manually segmented into separate point clouds of two individual plants. 





 §.§ Reconstruction and pre-processing of 3D point clouds



For reconstruction of 3D color point clouds from 2D color images, Agisoft Metashape Professional (Agisoft LLC, St. Petersburg, Russia) was employed. Agisoft Metashape is a software that performs photogrammetric processing of digital images and generates 3D models through principles of structure from motion (SFM) and multi-view reconstruction <cit.>. Examples of the raw point clouds produced by Metashape Professional are given in Fig. <ref>. The software also provides confidence maps for the 3D points. The confidence value for a point corresponds to the number of images that "see" and contribute to reconstruct the point. 

As can be observed from Fig. <ref>, the raw point clouds include structures from the background and are noisy. The correct scale of the scene is lost. Automatic segmentation of plants from each other and from the background is in itself a research problem. However, we aimed to provide a dataset of 3D models that correspond to isolated plants for evaluation of algorithms that focus on the analysis of single plants. 




In order to recover the correct scale of the point cloud, remove background structures and noise, and segment the plant points from the scene, we devised a simple, semi-automatic procedure. We denote the raw point cloud as 𝒫_R, where a point p ∈𝒫_R is represented by its coordinates p = (x, y, z) defined in the coordinate frame returned by Metashape. The steps of the procedure are detailed as follows: 




  * Recovering the scale: 
The multiview reconstruction pipeline employed by Metashape Professional provides the scene geometry upto a scale, meaning that the correct scale of the structure is lost. To enable recovery of the correct scale, we installed rulers and objects with distinct patterns in the vicinity of the plants during image acquisition. Once the 3D point cloud is reconstructed, the markers on the rulers and on the patterns were picked manually. Let the set of these landmark points be {p_r}⊂𝒫_R with r=1,2,...,N_r, where N_r is the number of landmark points. The distance between a pair of these reference points on the point cloud was determined as d̂_rs =  p_r - p_s, where · corresponds to the Euclidean norm. The average of the distances of all the pairs is equal to

    D̂_R = 1/N_r(N_r-1)∑_r ≠ s∑_sd̂_rs .


Let D_R be calculated using the true metric distances (in cm units) between the landmark points d_rs measured manually, or read through the ruler. The scale factor is then determined as 


    S = D_R/D̂_R .


The point cloud is scaled such that the average of the reference distances on the point cloud is equal to the measured distances on the real objects, i.e. the point coordinates were updated as p'= S · p for all p ∈𝒫_R. The scaled point set is denoted as  𝒫'_R.




  * Rotating the point cloud to a normalized pose: We aimed to rotate the reference frame such that the XY-plane corresponded to the ground or the plane holding the plant pot. In order to estimate the parameters of this plane, we used M-estimator SAmple Consensus (MSAC) algorithm given in <cit.>, which is a variant of RANdom SAmple Consensus (RANSAC) algorithm. We manually picked a point, p'_base, close to the plant base. Using the estimated normal vector 𝐧 = [ n_x,n_y,n_z ] of the plane in the scene and the point p'_base at the plant base, we translated and rotated the point cloud 𝒫'_R into a new reference frame using


    p̂ = 𝐑(p'-p'_base)


for all p' ∈𝒫'_R. The point cloud rotated to this normalized pose is denoted as  𝒫̂_R. In Eq. <ref>, 𝐑 corresponds to the rotation matrix which transforms the normal 𝐧 to [0, 0, 1] in the new reference frame. The origin of this reference frame corresponds to the hand-picked point at the plant base. The XY plane coincides with the plane detected by the MSAC algorithm. The positive Z-direction is oriented from the plant base towards the shoot. 


  * Extracting plant points:
Once the origin of the reference coordinate frame is established at the plant base, and the positive Z-direction points toward the plant shoot, we removed all points with a negative z coordinate, assuming all plant points remain above the plant base. We verified this assumption for all the models in the dataset. We applied connected component analysis to the rest of the point cloud and retained the points belonging to the largest connected component as the final plant point set 𝒫⊂𝒫̂_R. Examples of the outputs of this semi-automatic process is given in Fig. <ref>-(b) for a rose plant and in Fig. <ref>-(d) for a ribes plant. 







 §.§ The PLANesT-3D dataset



The PLANesT-3D dataset consists of 34 plant point clouds, where the plant points are isolated from the background as described in the previous section. Accompanying the locations and color information of the points in each model, three scalar fields are provided: The confidence map as delivered by Metashape Professional, semantic labels, and instance labels. Examples of point clouds from the PLANesT-3D dataset and their corresponding confidence maps and labels are depicted in Fig. <ref>.





Both semantic labels and instance labels are obtained through manually labeling each point using CloudCompare point cloud processing software <cit.>. Semantic labels for PLANesT-3D correspond to "leaf" and "stem" classes. Points on the leaf blades were labeled as "leaf", main stem, branches, and petioles were included in the "stem" class. Instance labels are identity numbers representing individual leaflets within a plant model. All points labeled as "stem" were assigned a single instance label. 




Table <ref> gives five properties of each of the 34 plant point clouds in the dataset: 1) Plant height, 2) Total number of points, 3) Percentage of "leaf" points, 4) Percentage of "stem" points, and 5) Number of individual leaflets. Plant height is in the range of 25 to 105cm. The number of points exceed 1 million for most plants. Number of individual leaflets varies between 25 to 105, indicating the variation of complexity in plant structure.



 §.§ Methods for semantic segmentation





<cit.> provided a comprehensive review of typical steps involved in the processing and analysis of 3D representations of plants, including point clouds. Models in PLANesT-3D can be used to evaluate a diverse set of tools developed for plant analysis and phenotyping. In this study, we focus on the application of segmentation of 3D point clouds of plants into their semantic units.

The objective of semantic segmentation is to assign each point p in the point set 𝒫 one of the semantic labels. The semantic categories in the PLANesT-3D dataset correspond to "leaf" and "stem", where "leaf" points are on leaf blades, and "stem" points belong to the main stem, branches, and petioles. 


We developed a novel semantic segmentation method that is a combination of a superpoint extraction scheme <cit.> and an adaptive 3D object classification network <cit.>. We abbreviate this method as SP-LSCnet, where SP stands for superpoint, while LSCnet is a leaf-stem classification network. In order to provide reference performance results for further research, we also tested PointNet++ <cit.> and RoseSegNet <cit.> on the new PLANesT-3D  dataset.



  §.§.§ SP-LSCnet



SP-LSCnet consists of two stages: Superpoint extraction and leaf-stem classification. In superpoint extraction stage, the point cloud is over-segmented into superpoints through an unsupervised process that operates on 2D points embedded by t-SNE. Then, each superpoint generated in the first stage is classified through a deep neural network equipped with attention-based modules that adaptively determine receptive fields for feature extraction. The flowchart of the method is depicted in Fig. <ref>, and the details of the two stages are given below:



Superpoint extraction: For accomplishing this task, the superpoint extraction scheme proposed in <cit.> is selected. The plant point cloud is first downsampled with voxel grid average filtering. t-SNE, as introduced in <cit.>, is used to embed the downsampled 3D point cloud into 2D space. Superpoint extraction is then performed exclusively in 2D domain. t-SNE inherently provides clusters that are formed based on the implicit manifolds to which the 3D points belong. 

The first step of the superpoint extraction scheme corresponds to the identification of these clusters by Euclidean clustering in 2D. Line-like structures are detected through examining the local geometry of the points. An iterative procedure that involves spectral clustering and solidity computation is employed to segment non-convex clusters into convex regions. Once all 2D points are assigned to their corresponding superpoints, the assignments are carried to their 3D counterparts, and propagated to the high-resolution point cloud through nearest neighbor interpolation.

In this work, for the PLANesT-3D dataset, the grid size for voxel grid average filtering is selected as 0.12cm, the Perplexity for t-sNE embedding is set to 30. The distance threshold used in Euclidean clustering in 2D space is chosen as 1. The other parameters were kept in their default settings as given in <cit.>.





Leaf-stem classification network: Each superpoint generated by the previous step is considered to be a separate point cloud. A point-based deep neural network, introduced by <cit.>, is utilized to classify the superpoints into leaf and stem classes (Fig. <ref>). The network is formed through integrating two modules to the PointNet++ architecture for object classification (Fig. <ref>). The modules are based on attention mechanisms to analyze point interactions within and between point neighborhoods, and are named as Center Shift Module (CSM) and Radius Update Module (RUM) <cit.>. Their task is to update adaptively the centers and radii of the spherical regions on which PointNet++ encodes local features. This operation is referred to as adaptive local region inference <cit.>.

The classification network architecture of PointNet++ <cit.>, shown in Fig. <ref>, includes three set abstraction (SA) layers. The input point cloud with N = 1024 points is reduced to N_1 = 512, and N_2 = 128 representative points during SA layer-1 and SA layer-2, respectively. At each SA layer L, the representative points c_j^L are selected using farthest point sampling (FPS) algorithm. N_L spherical local regions with radius r^L is defined around the center points c_j^L. Within each region, K points are randomly selected and the features of K points are mapped to higher dimensions with MLPs. The abstracted features of each local region are calculated by taking the maximum among the feature channels of K grouped points.  

The global descriptors representing the input point cloud are extracted at the last layer, where a single region enclosing the entire object is encoded. The global descriptors are passed through fully connected layers to compute the category scores. In this study, the classification network calculates two category scores for each input point cloud: One for the "leaf" category, and one for the "stem" category.

The adaptive local inference modules, CSM and RUM can be integrated to either or both of the first two layers as shown in Fig. <ref>. If CSM is "ON" at a particular layer, for each representative point c_j^L, CSM calculates the shift amount Δ c_j^L, through attention-based subnetworks. The representative point is updated as:


    ĉ_j^L = c_j^L + Δ c_j^L .

If RUM is "ON", it computes the amount of radius update Δ r_j^L for each region j. The new radius then becomes:


    r̂_j^L = r^L + Δ r_j^L .

The grouping and feature encoding by MLPs of K points are then performed using these new region definitions. These modules allow the receptive fields to be adaptively shifted and resized through examining local and global point interactions through attention. The network thus adjusts the receptive fields and encodes the 3D points in them in accordance with the main task of the network.

Various alternatives of CSM and RUM were suggested by <cit.>. As a result of experimental trials, the CSM-II (sub) and RUM-II (cum) versions were selected for the PLANesT-3D dataset. Please, refer to <cit.> for detailed descriptions of of the modules CSM-II (sub) and RUM-II (cum). 






  §.§.§ PointNet++



We adopted the PointNet++ architecture <cit.> for semantic segmentation of large point clouds. The data preparation for PointNet++ involves two operations: Subsampling and partitioning the big point set into subsets enclosed by blocks.

The subsampling is performed using voxel-grid filtering, where the edge length of the voxel cube is set as 1mm. Each voxel is represented with a single point which is the average of the original points occupying the voxel. The sampled point is assigned the label of the closest point in the original cloud. This subsampling operation produces a point cloud with reduced size and homogeneous point density. 

3D point-based deep neural networks requires input clouds with a fixed number of points. One way is to subsample the full plant point cloud to have the required number of points. However, such strategy will result in significant loss of geometric information, especially for small structures. Instead, we selected to partition the plant point cloud into subsets of points, each of which is then processed by the network individually. We followed a similar procedure of partitioning the point clouds into blocks as in <cit.>.

The horizontal region, i.e. the extent of the XY plane, encompassing the point cloud is partitioned into squares of edge length of 10cm. All points with x and y coordinates falling into a square region is considered belonging to the point subset within a block. The points in the block are subsampled to obtain a cloud of fixed number of points, as N = 8192. The partitioning operation is conducted twice for each plant point cloud with offset values 0 and 5cm. 

 

We used the default PointNet++ architecture for semantic segmentation as given in <cit.>. PointNet++ segmentation network is composed of five grouping/abstraction layers. The radius parameter, which defines the size of local regions for grouping and abstraction, is set for each of the first four layers as 0.5cm, 1cm, 2cm, and 4cm, respectively. During training, batch size was selected as 16. The learning rate was updated as 0.005. Other hyperparameters of the PointNet++ were kept at default values.




  §.§.§ RoseSegNet



RoseSegNet was developed for the specific application of semantic segmentation of rosebush models. It involves attention-based modules that encode point interactions within and between local regions. The details of the RoseSegNet architecture can be found in <cit.>.

As with the case of PointNet++, we partition the plant point cloud into blocks and the point subset in each block is separately processed by RoseSegNet. The block partitioning process is the same as with PointNet++. 

All the hyperparameters of the RoseSegNet are kept at their default values as given in <cit.>, where the parameter search was performed on the ROSE-x dataset <cit.>. 






§ RESULTS



In this section, we provide semantic segmentation results on the PLANesT-3D dataset obtained by the three methods described in Section <ref>. We trained a separate network for each of the three plant species. The hyperparameters of the networks were kept fixed for all the species. 70% of the point clouds were selected for training the networks. The rest is reserved for test. The split for training and test sets for the three species are given in Table <ref>. The IDs of the plants in the test sets are also provided.






Before being processed by the networks, the point clouds were filtered such that points with confidence values less than 6 were discarded. For PointNet++ and RoseSegNet, the point cloud of a plant is partitioned into blocks and each block is separately processes as described in Section <ref>. For SP-LSCnet, the point clouds are partitioned into superpoints, and the superpoints are treated as separate objects for the object classification network. Table <ref> gives the number of point subsets used for training and test for each plant species. 



We report the semantic segmentation results of the three methods in Table <ref>. Precision, Recall, and Intersection over Union (IoU) measures specific to "stem" and "leaf" classes are given in the table, as well as overall accuracy (Acc) and mean IoU (MIoU). In terms of overall accuracy and mean IoU, RoseSegNet yielded the best performance for all three species. For MIoU, RoseSegNet surpassed the performance of PointNet++ by close to 1%. SP-LSCnet is in between, performing slightly better than PointNet++ for the Pepper and Ribes sets, while yielding similar peformance to that of PointNet++ for the Rose set.





The results on rose plant models are lower compared to those obtained on pepper and ribes plants. The recall of the "stem" class is more challenging for the rose plant models, since the petioles in between the leaflets are hard to discern. The misclassifications of petiole points of roses as belonging to leaf blades are observable in Fig. <ref> and Fig. <ref> where segmentation results are visually depicted for sample plants.





Other examples to erroneous cases are given in Fig. <ref>. The errors produced by PointNet++ and RoseSegNet appear as isolated points scattered in particular regions. Although context information heavily influences the features of single points in PointNet++ and RoseSegNet, the classification is performed individually for each point. Misclassifications occur particularly at cluttered regions where leaves and stems are close to each other.

The treatment of each superpoint as a single entity by SP-LSCnet produces smooth results; however, it suffers from two disadvantages: First, the superpoints might not be always homogeneous; some leakage from other classes is inevitable due to the unsupervised nature of superpoint extraction. Second, the correct classification of the superpoints becomes particularly critical. As can be seen in Fig. <ref>, portions of the stem were classified as leaf points if the single superpoint to which they belong has the geometrical characteristics of leaves as modeled by the network. Likewise, small leaves, where the point resolution is not sufficient for proper sampling of the leaf surface, are classified as stem. Low point resolution due to occlusion (see bottom-left example in Fig. <ref>) can also cause isolated regions separated into superpoints which are not large enough to possess distinguishing shapes.

All three methods can benefit from post-processing such as label smoothing for PointNet++ and RoseSegNet, and graph-based re-evaluation of superpoint predictions for SP-LSCnet. However, the limitations of these methods in dealing with scarce annotated data, noisy point clouds, complex architecture of plants, and variable point density call for further developments in plant segmentation methods.



§ DISCUSSION



The PLANesT-3D dataset is composed of 3D point clouds of plants reconstructed through structure from motion and multi-view stereo. Capturing 3D geometry through SfM has many advantages such as low cost setup requirements, high resolution, and provision of color information <cit.>. It can be used both in controlled environments and in the field without extensive training for acquisition and elaborate protocols. A public dataset obtained through this modality as opposed to laser scanning is expected to be instrumental in assessing the robustness of 3D plant processing and phenotyping tools.


As can be observed in Table <ref>, the size of each model in terms of number of points is high, ranging from 600K to 5.5M points. One question is whether such high resolution is necessary for certain tasks related to plant analysis and management. The number of points can be reduced systematically through downsampling to measure the effect of resolution on the performance of particular 3D shape processing tools <cit.>. 

We also used quite large numbers of images of the plants for reconstruction (<ref>). This amount of image capture might not be feasible in cases where manual data acquisition is performed. However, advances in robotics enable fast acquisition 
with autonomous crop surveying robots equipped with robotic arms and multiple cameras <cit.>, as well as techniques for automatic positioning of the robot arms for optimal data acquisition <cit.>. Nevertheless, limiting the number of images for reconstruction without a performance degradation in particular tasks would substantially reduce the computational demand.

Another important aspect of the new dataset is the availability of color information. Although not demonstrated in this work, use of color information together with 3D geometry can boost the performance of a segmentation procedure <cit.>. The PLANesT-3D dataset will be a valuable resource for measuring the contribution of color information for automated plant analysis and phenotyping tasks.

Semantic parts annotated in the PLANesT-3D dataset are limited to "leaf" and "stem" classes; however, a more fine-grained annotation is possible, for example, in terms of "leaf blades", "petioles", "main stem", "nodes", and "internodes" <cit.>. Leaflets, together with petioles, can be grouped into their corresponding individual leaves. The architecture of each plant can be modeled with a graph representation. We envision undertaking this level of annotation as a future work.

Another possible future work is a study on the transferability of the neural network models learned from a plant species into another species. In the current work, we trained a separate network for each of the three plant species. The extent of generalization ability of network models among plant species is of special interest; since manual annotation of plants for training is highly labor-intensive. PLANesT-3D, with a collection of three different species, allows application and evaluation of algorithms such as small sample learning and, transfer learning.





§ CONCLUSIONS



We introduced the PLANesT-3D dataset, which is composed of annotated color point clouds of 34 plants belonging to three different species. We described the acquisition, reconstruction, and labeling processes involved in dataset construction. As a use case of the dataset, we reported semantic segmentation results yielded by PointNet++, RoseSegNet, and a novel method abbreviated as SP-LSCnet. RoseSegNet achieved highest segmentation performance, reaching 95.5 %, 92.0 %, and 95.1 % MIoU, for pepper, rose, and ribes plants, respectively. SP-LSCnet also produced comparable results as 95.0 %, 89.8 %, and 94.5 % MIoU on the three plant species. We illustrated sample cases where the three methods failed to produce correct results. The PLANesT-3D dataset will be instrumental for developing new methods that effectively address these challenges.




§ ACKNOWLEDGEMENTS:

The authors acknowledge the support of The Scientific and Technological Research Council of Turkey (TUBITAK), Project No: 121E088.





§ DECLARATION OF COMPETING INTEREST

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.



elsarticle-num-names



 Paper Title: Multi-Site Class-Incremental Learning with Weighted Experts in Echocardiography.





Multi-Site Class-Incremental Learning with Weighted Experts




Bransby et al.


Ultromics Ltd., Oxford, United Kingdom

 agis.chartsias@ultromics.com 
Queen Mary University of London, United Kingdom

 k.m.bransby@qmul.ac.uk 


Multi-Site Class-Incremental Learning with Weighted Experts in Echocardiography
    Kit M. Bransby1,2, Woo-Jin Cho Kim1, Jorge Oliveira1, Alex Thorley1, Arian Beqiri1, Alberto Gomez1, Agisilaos Chartsias1
    August 5, 2024
============================================================================================================================



§ ABSTRACT 
Building an echocardiography view classifier that maintains performance in real-life cases requires diverse multi-site data, and frequent updates with newly available data to mitigate model drift. Simply fine-tuning on new datasets results in “catastrophic forgetting”, and cannot adapt to variations of view labels between sites. Alternatively, collecting all data on a single server and re-training may not be feasible as data sharing agreements may restrict image transfer, or datasets may only become available at different times. Furthermore, time and cost associated with re-training grows with every new dataset. We propose a class-incremental learning method which learns an expert network for each dataset, and combines all expert networks with a score fusion model. The influence of “unqualified experts” is minimised by weighting each contribution with a learnt in-distribution score. These weights promote transparency as the contribution of each expert is known during inference. Instead of using the original images, we use learned features from each dataset, which are easier to share and raise fewer licensing and privacy concerns. We validate our work on six datasets from multiple sites, demonstrating significant reductions in training time while improving view classification performance.  






§ INTRODUCTION


Echocardiography (echo) view classification is often a necessary first step in automated image interpretation and analysis, as different tasks may require different views as input <cit.>. Several deep learning methods have been proposed to this end <cit.> demonstrating excellent performance. To generalise and maintain performance in real-life cases, view classifiers need to be trained on a diverse multi-site dataset to accommodate varying acquisition, demographic, and clinical parameters. However, these parameters may change over time, e.g. with machine upgrades, modification of protocols, or changes in demographics. Such changes require updating the classifier with newly available data to mitigate model drift <cit.>. Naive approaches like fine-tuning on a new dataset may result in “catastrophic forgetting” <cit.>, in which previous knowledge is forgotten at the expense of the new information. Retaining previous knowledge while learning on new data is the goal of incremental-learning <cit.>. 

Furthermore, newly available data (e.g. from sites with different protocols) may have different sets of view labels. Labels can be characterised as “base”, “novel”, or “overlapping”, depending on whether they are present in the original data, the new data, or both. The goal of class-incremental learning (CIL) <cit.> is to learn to classify the increasing label set over time. A straightforward solution is to retrain using all data combined. However, this is not always feasible as data sharing agreements may limit access or transfer outside the acquisition site; or different datasets may be available at different times. In addition, retraining a classifier when a new dataset becomes available is inefficient as the total training time and cost grows considerably with the number and size of datasets.

A common approach for CIL is to use a single model, and update parts or all of the weights using knowledge distillation <cit.> or weight regularisation <cit.>. Others learn additional parameters with dynamic architecture changes <cit.>, or by duplicating the model for novel data and pruning <cit.>. Wu et al. <cit.> expand on this by duplicating and fine-tuning a base model for each new dataset, resulting in set of expert model branches, each specialised on a single dataset. They combine the output logits of the independently trained branches using a learnt score fusion (SF) network that enables knowledge transfer between base and novel classes.
This leverages shared information between representations in different branches, but is sensitive to one branch contributing ineffective information to another branch. We term this the “unqualified expert” problem and may happen when there are differences in data distributions, caused for instance by different label sets, patient demographics or scanner manufacturers. As a result view predictions for out-of-distribution (o.o.d.) images can be incorrect, but also highly confident. 



In this paper we address the problem of building an echo view classifier model with multiple datasets from different sites (see Fig. <ref>) that are not simultaneously available and have different, overlapping sets of labels. To this end, we build upon <cit.>, and
mitigate the “unqualified expert” problem by weighing the contribution of each branch by a learnt in-distribution (i.d.) score. This minimises the influence of logits which are o.o.d. for a given input image, while maximising the i.d. logits. Our weighting improves transparency as the contribution of each model is known during inference, indicating the similarity between input data and different training sets. Our method has the benefit of using learnt features as input rather than images, which provides a simple workaround in cases where licensing prevents image sharing but not sharing of byproduct features. 

Contributions: We apply CIL to echo view classification and propose a new i.d. score weighting that improves performance by minimising the influence of “unqualified experts” and promotes transparency. We validate our method on six datasets from multiple sites, demonstrating improved view classification without needing to transfer images out of remote servers. By bypassing the need for re-training with all data, the cumulative training time and cost is reduced with further reductions achieved through more incremental steps.



§ METHOD




 §.§ Problem Setting


Given a dataset D = {(x_i, y_i)}_i=1^N, where x_i represents an ultrasound image and y_i denotes the corresponding ground truth view label, the objective of view classification is to learn an image encoder ϕ and linear classifier W ∈ℝ^k × |𝒴|, where 𝒴 is the set of labels in D and k is the size of the image features. The view label is learnt by minimising the cross entropy loss between y and the label prediction ŷ = σ(ϕ(x)W) where σ(·) is a softmax function. 

We first train a base model ℳ_b = {ϕ_b,W_bb} on a large dataset 𝒟_b with label set 𝒴_b of base classes. We consider t incremental steps n ∈{n1, n2,..,nt} each with a dataset 𝒟_n and label set 𝒴_n. There can be varying degrees of intersection between base classes 𝒴_b and the new label set 𝒴_n. For instance, there may be novel classes not present in the base classes (𝒴_n∩𝒴_b = ∅) or overlapping classes (𝒴_b∩𝒴_n≠∅). Our goal is to accurately predict all classes, regardless of whether they are introduced in an incremental step, are novel, base, or overlapping. 

Simply combining base and incremental datasets into a single set and retraining ℳ_b from scratch is not ideal due to cost, time, and restrictive data licensing as motivated in Section <ref>. Similarly, naively fine-tuning ℳ_b on successive incremental datasets 𝒟_n1, 𝒟_n2, .., 𝒟_nt is difficult due to the changing size of the label set, and ultimately results in catastrophic forgetting of the base classes 𝒴_b. 





 §.§ Score Fusion


We follow Wu et al. <cit.> where ℳ_b is cloned and fine-tuned for each incremental dataset 𝒟_n, yielding t+1 expert models  {ℳ_b, ℳ_n1,.. , ℳ_nt} [We unfreeze all layers instead of freezing the first two convolutional blocks as in <cit.>, which we find improves performance.]. This approach mitigates the problem of forgetting by retaining all weights; however, it introduces a new challenge of selecting or combining experts during inference. 

This is addressed with SF <cit.> that allows an expert to make a logit prediction on its label set as well as the label sets of other experts. This allows for knowledge sharing between branches and is useful for overlapping classes or shared semantics. Specifically, the weights of an expert {ϕ_d, W_dd}, where d ∈{b, n1,.., nt}, are frozen and used to compute a logit score  z_dd = h_dW_dd for its label set 𝒴_d using feature vector h_d∈ℝ^k and the weight W_dd. To compute logit predictions on label sets of other experts 𝒴_d', each expert learns an additional set of weights W_dd' for all d' ∈{b, n1, .., nt}, d ≠ d'. On each branch, we compute the final logits z_d by summing the logits contributed by each expert as follows: 


    z_d = ∑_d'=b, n1,..,nt^ h_dW_dd'
 

The logits from each branch are concatenated to give z_a. As overlapping classes may be present between incremental steps, a logit for the same class may appear several times in z_a. We follow <cit.> with a “knowledge pooler” to obtain final logits z̃_a by maxpooling the overlapping classes (See Fig <ref>). 
This enables information sharing between experts, however some experts may have significant differences in knowledge and contribute unhelpful information that deteriorates performance. In the next section, we address the “unqualified expert” problem.



 §.§ Weighted Score Fusion


Given an input image, we compute an i.d. score to weigh each expert's logits such that the influence of unqualified o.o.d experts is minimised and the influence of i.d. experts is maximised. We do so by learning an attention-like  weighting vector A ∈ℝ^ | d  | using a feed-forward network ϕ_A consisting of 3 linear layers, the first two with ReLU activation and the last with a softmax. 

We propose two strategies to learn A: (1) attention-weighted score fusion (attn-wSF) where the concatenated feature vectors of all experts h ∈ℝ^k  | d  | are used as input to ϕ_A; and (2) neural mean discrepancy weighted score fusion (nmd-wSF), which uses the neural mean discrepancy (nmd) <cit.> as input. Nmd is an out-of-distribution metric which measures the discrepancy between neural means of an input image and the average neural mean of the i.d. training set. The neural mean is calculated by averaging output activations of the convolutional layers. Like with attn-wSF, we compute a nmd vector g_d for each expert and concatenate them to g ∈ℝ^p  | d  | where p is the size of the nmd vector. 

Similar to Equation <ref>, we compute the final logits for each branch as follows:

    z_d = ∑_d'=b, n1,..,nt^ h_dW_dd'A_d





 §.§ Multi-Site Training


Assuming that the incremental datasets are acquired at different medical sites, which often impose data sharing restrictions, we propose a multi-site training paradigm, which does not require transferring the image data. For every step, the view classifier is cloned, transferred into the remote server, and fine-tuned on dataset 𝒟_n. We then make five forward passes of the fine-tuned classifier ℳ_n on randomly augmented data from 𝒟_n to generate five different g and h vectors for each example. We then transfer these vectors back to the local server, which are used as input to train the weighted score fusion network.





§ EXPERIMENTS & RESULTS




 §.§ Datasets


As base dataset 𝒟_b, we use WASE-Normals (WASE) <cit.>, a large multi-site proprietary dataset of 2,009 healthy volunteers acquired at 18 sites from 15 countries. We also use three incremental datasets: (1) CAMUS <cit.>, a public dataset acquired from 500 patients at University Hospital of St Etienne, half of which are considered at pathological risk; (2) Medstar WASE-Covid (Medstar) <cit.>, a multi-site proprietary database of 870 patients with COVID-19 at 13 sites from 9 countries; 3) St George's (StG), a proprietary stress echo dataset of 420 patients some with coronary artery disease. We validate our method on an internal test set sampled from the four datasets and an independent external test set from two proprietary datasets: (1) Mazankowski Alberta Heart Institute (MAHI) consisting of 250 patients undergoing chemotherapy with cardiotoxic drugs; (2) University of Chicago (UoC) consisting of 391 patients with cardiomyopathy. 

We extract images from 15 echo view labels
and artificially decrease the overlap between datasets by removing non-contrast examples from StG and contrast examples from Medstar. As shown in Table <ref>, the resulting datasets have different label sets with varying degrees of overlap.
Each dataset is split into train (80%), validation (10%) and test (10%) sets at a patient level. A single random frame (112×112) was sampled from each echo video. The final dataset has 42,622 train, 5,175 validation, 5,189 internal test, and 13,999 external test frames. 



 §.§ Implementation & Training


View classifiers use ResNet18 <cit.> architecture and were trained for 200 epochs on a NVIDIA GeForce RTX 2080Ti with cross-entropy loss, Adam optimiser, batch size of 64, and learning rate of 1e-3. The score fusion networks were trained for 50 epochs with the same settings. Weights from the epoch with the highest validation accuracy were saved. Hyperparameters were tuned on a held-out validation set. We evaluate classification performance using accuracy and F1-score metrics. Our method is implemented in PyTorch, and the code is available here: https://github.com/kitbransby/class-incremental-learning-echo







 §.§ Comparison to Existing Methods & Ablation Study


We validate our method on an internal test set (WASE, CAMUS, Medstar, StG) to measure performance on incremental datasets seen during training, and an external set (MAHI, UoC) of data not seen during training to measure generalisability. Results are presented in Tables <ref> and <ref>, respectively. 

We demonstrate that naively fine-tuning a classifier on successive datasets results in “catastrophic forgetting”. We test two configurations, one where the classifier head is replaced at every step, and a second that expands to accommodate novel classes <cit.>. These do not require data transfers and perform well on the final dataset (StG) but forget learning from previous data. We also explore other methods that do not require data transfer such as using a single expert and find they perform well on i.d. data but do not generalise well on o.o.d data. 


Our primary comparison is to the training oracle where the model is retrained with all data combined at each incremental step. This method is preferred given unlimited compute, time constraints and data access, which are not typically available. On average our method outperforms the oracle on both the internal and external sets, including four out of six individual datasets. 

In addition, we ablate our weighting method, demonstrating that attn-wSF and nmd-wSF lead to an improvement in performance when compared to SF <cit.>. We find that nmd-wSF performs best on the internal test set, however, attn-wSF generalises better to the external test set. We theorise that nmd vectors for each dataset are quite distinct, therefore the i.d. scores are pushed towards 0 and 1 during training (see Fig. <ref>). When presented with datasets from the external test set the attention mechanism deteriorates as it considers the input image o.o.d. for all experts. 
 
Finally, we compare different confidence-based methods that select the best prediction from multiple experts, these include: (1) Max Logit <cit.> that selects the max logit for each class; (2) MSP <cit.>, that selects the max softmax for each class; (3) Confidence Routing <cit.> that selects the expert with the highest softmax probability, and uses its softmax prediction as the final prediction. All such approaches perform poorly, perhaps as classifiers are optimised to be overconfident leading to predictions which are uncorrelated with confidence measures. 


Fig. <ref> shows that our wSF methods reduce the cumulative training time, and this grows significantly with the number of datasets. Our attention-based weighting module has minimal computational overhead. While inference time is larger when using multiple experts, this can be reduced with parallelisation. 



§ CONCLUSION


Echo view classifiers require training on diverse multi-site data, and frequent updates with newly acquired data to avoid model drift. We address the problem of updating such a model using multiple datasets with partly overlapping label sets that are from different sites and may not be simultaneously available. We propose attn-wSF and nmd-wSF, two CIL methods that train and combine predictions from multiple experts models with a learnt i.d. weighting which minimises the influence of o.o.d predictions from “unqualified experts”. Our results demonstrate improved view classification with significantly reduced cumulative training time and without needing to transfer images out of remote servers. We aim to extend our wSF methods by training multiple experts within a single branch, which would reduce the memory requirements and inference time. 









splncs04




 Paper Title: Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs.





 



PAI






Liu et al.




State Key Lab of CAD&CG, Zhejiang University Ant Group 

{liushi0927,zkechengzk}@gmail.com 

chenvis@zju.edu.cn

Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
    
  Shi Liu1 Kecheng Zheng1,2^† Wei Chen1^†

    
=============================================================================================

[2]^† Corresponding authors.



§ ABSTRACT 
Existing Large Vision-Language Models (LVLMs) primarily align image features of vision encoder with Large Language Models (LLMs) to leverage their superior text generation capabilities.

However, the scale disparity between vision encoder and language model may led to LLMs assuming a predominant role in multi-modal comprehension.

This imbalance in LVLMs may result in the instances of hallucinatory. 

Concretely, LVLMs may generate consistent descriptions with or without visual input, indicating that certain outputs are influenced solely by context text.

We refer to this phenomenon as “text inertia.”

To counteract this issue, we introduce a training-free algorithm to find an equilibrium point between image comprehension and language inference. 

Specifically, we adaptively involve adjusting and amplifying the attention weights assigned to image tokens, thereby granting greater prominence to visual elements.

Meanwhile, we subtract the logits of multi-modal inputs from ones of pure text input, which can help LVLMs be not biased towards LLMs.

By enhancing images tokens and reducing the stubborn output of LLM, we can let LVLM pay more attention to images, towards alleviating text inertia and reducing the hallucination in LVLMs.

Our extensive experiments shows that this method substantially reduces the frequency of hallucinatory outputs in various LVLMs in terms of different metrics.

Project page is available at <https://lalbj.github.io/projects/PAI/>




§ INTRODUCTION


Recently, Large Vision-Language Models (LVLMs) have made significant strides, exhibiting impressive capabilities across a multitude of tasks <cit.>. 

However, these models still struggle with the phenomenon of hallucination. 

Specifically, there is often a mismatch between the textual content generated by the model and the actual visual input it receives <cit.>.

  


 

Hallucination in LVLMs is often attributed to issues with modality alignment, leading to the development of mitigating strategies through alignment training optimization <cit.>. 

However, is hallucination in LVLMs merely a result of the model's capacity, and can it only be alleviated through additional training?

We propose a scenario where LVLMs generate a hallucinated object description. 

Specifically, even when the image input is removed and only the generated text preceding the hallucinated object word is retained, the LVLMs persist in producing the same hallucinated description, as depicted in  <ref>.     

In order to empirically investigate this behavior, we conducted tests on three LVLMs within the context of image describing tasks on the COCO dataset. 

We identified and conducted a statistical analysis on instances where LVLMs generated identical hallucinated object descriptions, even when the input was exclusively historical response text without any image.

The observation from <ref> clearly indicates that, even with the application of rigorous identification settings, the phenomenon continues to represent a substantial proportion.

We referred this phenomenon as “Text Inertia”. 

Our hypothesis is that text inertia arises due to the current generative paradigms mapping image representations onto the text representation space as text tokens. 

In this mechanism, the LLM becomes the dominant character, and the inference process lacks additional handling of image tokens, leading to their neglect during the generation process. 

To validate this hypothesis, we have analyzed the attention values ratios of the LLaVA model during the inference process in <ref>. 

Our findings show that despite image tokens occupying a significant proportion, they do not receive substantial attention under the current mechanism. 

This multimodal chat resembles more of an automatic completion based on context rather than a continuous attention to the image for completion.

To close this gap, we introduce a method refered to as Pay Attention to Image (PAI). At a high level, PAI intervenes in the inference process to make it more image-centric, following the original image perception direction.

To achieve this, we focus on the self-attention heads in the decoder layers of LVLMs. 

We enhance the attention weights for image tokens in their original directions during inference. 

This allows us to use the updated attention matrix to calculate the hidden states for the generated token, thereby incorporating more consideration for image representation during the generation process. 

To further mitigate text inertia, we construct the input using instruction tokens and historical response tokens, and subtract the model logits of this input from the logits of original model with image tokens input. 

This strategy helps to reduce the influence of language priors during the generation process. 

Unlike previous methods for mitigating hallucination that require additional training or external tools, our approach is training-free. 

Moreover, we are the first to propose an inference intervention method for mitigating hallucination in LVLMs.

We then evaluate the response accuracy in the image description task from long sequence generation perspective by employing the CHAIR metric <cit.> and GPT-4V. 

In addition, we use POPE <cit.> and MMHal-Bench <cit.> to more comprehensively evaluate the model's hallucination performance on VQA task. 

Furthermore, we have constructed single-turn and multi-turn chat evaluations for POPE.

Since our model intervenes in the inference process, it can be used for any decoding method. 

Therefore, we conducted experiments on three decoding methods of the three models. 

The experimental results proved the effectiveness of our method in mitigating hallucinations.









 


§ RELATED WORK



 §.§ Large Vision-Language Models

The development of pre-training techniques <cit.> and instruction tuning techniques <cit.> has rapidly advanced LLMs technology, such as LLaMA <cit.> and Vicuna <cit.>, further leading to the prosperity of LVLMs technology. Early works, like Flamingo <cit.> and BLIP-2 <cit.>, have successfully adapted LLMs to visual tasks, demonstrating notable generative capabilities and in-context learning abilities. Recently, the capabilities of LVLMs have further advanced under the influence of visual instruction tuning techniques <cit.>. Using different projectors to map images to the text domain, thereby endowing language generation models with image understanding capabilities, is also a hot research topic <cit.>. Additionally, several studies focus on visual language tasks such as grounding capabilities <cit.> and reasoning capabilities <cit.>. However, recent LVLMs still face the issue of hallucination generation <cit.>.



 §.§ Mitigation of LVLMs Hallucination

Hallucination in LVLMs refers to contradictions between the image input and the textual output. Various methods have been proposed to mitigate hallucination. The most direct reason for hallucination generation is that hallucination arises from data bias and knowledge gaps between vision and language. Therefore, better data filtering methods <cit.> and higher quality annotated data <cit.> are introduced. Simultaneously, these methods also imply the need for more alignment training <cit.> or adjustments in the model architecture <cit.>. These methods can achieve good results, but they are time-consuming and require high computational resources.

Apart from addressing the ability of LVLMs itself, hallucination can also be mitigated through post-processing methods. This method usually involves using an additional module or external tools to edit the response. Recent methods such as LURE <cit.> utilize additional data to train a state detector and when hallucination issues are detected, content is regenerated by a revisor model. Woodpecker <cit.> introduces an external visual model to inspect entities extracted from the response, and then the detection results are handed over to the generation model to regenerate better answers. These methods also extend the inference chain and increase inference costs.

Training-free hallucination mitigation methods have so far only been attempted in decoding methods. OPERA <cit.> discovered an abnormal attention pattern that accompanies model decoding. It was statistically found that this pattern often accompanies hallucination descriptions, and thus a detection and mitigation method was proposed based on this pattern to alleviate the hallucination faced by the model. VCD <cit.> introduced the notion that visual uncertainty increases hallucination descriptions and, based on this discovery, proposed a contrast decoding method to alleviate hallucination issues.


§ PRELIMINARIES


The architecture of LVLMs typically comprises of three main components: an image encoder, a projector, and a language decoder. Both the image encoder and language decoder are usually pre-trained. The image encoder is employed to transform images into image tokens, which are subsequently mapped to the text representation space by the projector. This process enables the concatenation of image tokens with text tokens that are then fed into the language decoder. The language decoder subsequently generates corresponding responses based on the provided instructions.

The existing projectors. 
Currently, projectors predominantly fall into two categories: linear projectors and resamplers. A projector takes N visual features from the image encoder and transforms them into M visual tokens. The linear projector employs a multilayer perceptron to transform all visual features, maintaining a one-to-one transformation which meaning that M equals N. In contrast, the resampler does not preserve all visual features but instead samples visual cues (M, where M  N). For instance, Q-former <cit.> utilizes M learnable queries and Bert <cit.> to extract information from visual features. Given that the knowledge of images during the generation process solely originates from the output image tokens of the projectors, our attention is concentrated on the image tokens post-projection, irrespective of their preceding modeling process.

Autoregressive language decoders. 
Nearly all LVLMs adopt LLaMA-family models as their language decoders, which employ the self-attention mechanism.

The visual tokens processed by the projector are concatenated with text tokens and fed into the LLaMA, which carries out the forward decoding process.

From the perspective of a single attention head in a single layer, each head repeatedly performs the following attention operation with the same input shape:

    _h = _h _h,   _h = (_h _h^⊤/√(d_k)).



Each attention head h performs an attention operation using its own set of queries _h ∈^n × d_k, keys _h ∈^n × d_k, and values _h ∈^n × d_k, where n represents the sequence length and d_k represents the hidden dimensions.

The output _h ∈^n × d_k is modeled by multiplying _h and the attention weights _h ∈^n × n, where each row represents the weights for each token during feature mixing.

This operation enables the model to focus on different parts of the input for each head through attention weights, thereby capturing various parts of the information from the sequence token representations.

The final output is the current generated token vocabulary conditional probability distribution ∈^v based on the input instruction representations _I, image representations _V and history generated token representations _H, where v is the size of the vocabulary. This process can be formatted as:

    ∼ p_model(|𝐗_I, 𝐗_V, 𝐗_H), 
       ∝softmax(logit_model(|𝐗_I, 𝐗_V, 𝐗_H)),


which calculates the distribution of one token and iterates for the entire response. The sequence generation continues until an EOS (End of Sentence) token is produced, marking the end of the generation and resulting in a complete response.





§ METHOD




At the core of our method is a solution for image neglect and text inertia, both of which are fundamentally interconnected.

Essentially, as paying more attention to the image, there is a corresponding reduction in the reliance on language priors.

Intuitively, in a conversation centered around an image, the model should devote more attention to the image, thereby allowing it to have a significant impact on the response.

As such, we identify the self-attention map in the token-level generation and augment the image attention in its original directions.

This strategy promotes a more image-centric latent representations.

Additionally, to further mitigate the influence of text inertia, we devide the logits distribution of pure text input into the model's output.



 §.§ Pay More Attention to Image


Extracting the Self-Attention Matrix. 
We start from a token-level perspective.

The response process in LVLMs is fundamentally generated token by token. 

Each token is generated based on the input image, instruction, and the historically generated response. 

This process is facilitated through a multi-layer attention decoder architecture. 

Consequently, this results in a probability distribution of the vocabulary for the currently generated token.

Our goal is to extract the attention matrix of each attention head at every layer, indicating the influence of each content during inference.

When generating the k-th token in the sequence, the input representation for the attention head in the forward process includes the instruction representation _I=[_i_1, …, _i_n_I], image representation _V = [_v_1, …, _v_n_V], and the representation of the historically generated response _H = [_h_1, …, _h_n_H]. 

Notably, the image representation considered here is the one that has been processed by the projector.

Essentially, the hidden states of each input layer are = concat(_I[1:m], _V, _I[m+1:n_I], _H), where the notation _I[1:m] indicates the first m elements in the instruction representation.

Each attention head assigns different degrees of attention to each element during the current token representation decoding process. Our aim is to enhance the attention paid to the image. Therefore, we extract the attention weight values related to the image token for the current generated token, intervene, and then redistribute the attention values of each element through softmax.

Excite model in a trustful direction. 
There have been attempts in some LLMs works to make the answers generated by LLMs more trustworthy by means of intervention <cit.>. 

The implemented approach typically involves intervening with the hidden states. 

As for defining what constitutes a more trustworthy direction, it usually requires additional projection and training to probe this trustful direction. 

In our case, a response that is more image-based is considered more trustworthy. 

Since LVLMs have undergone alignment training, the original attention values provide a direction based on image content. 

As illustrated in <ref>, by amplifying the attention values of image tokens based on the original attention values, we can enhance the trustworthiness of our results.

Another nuance involves our avoidance of choosing the attention head used to shift intervention. 

In the ITI method <cit.>, it is stated that not all attention heads should be subjected to intervention. 

Therefore, they introduce a trustful score to rank each head across all layers and select the top-k heads for intervention. 

In our case, the less trustworthy heads with lower attention values receive less intervention. 

We first extract the attention weights of the image tokens for the 
current generated token from the attention weights  before softmax opeartion. We then use the hyper-parameter α to control the step size for intervention. From a single attention head perspective, our method can be expressed as follows: 

    _n, j = _n, j + α· |_n, j|   for  j = m+1  to  m+n_V.

The model's final vocabulary probability distribution is derived from the projection of the hidden states of the last token in the sequence. 

Therefore, we extract the attention weights of the last token n on the image tokens by indexing _n, j. Following the intervention, we use the softmax function to redistribute the attention values of each token during the reassignment of encoded hidden states.

This procedure is repeated for each subsequent token prediction in an autoregressive manner and is independent of the choice of the decoding algorithm.

    

Excite more precisely with an attention mode prior. The presence of the BOS token, an attention sink pattern <cit.>, in a sentence results in higher attention values during the attention computation process, which may seem counterintuitive. 

The BOS token typically signifies the start of a sentence and, as such, does not carry significant semantic content. 

However, the generation of tokens is significantly influenced by this particular token, a similar pattern that also manifests itself in visual models <cit.>. 

As mentioned in StreamLLM <cit.>, the pattern of the attention sink emerges when redundant attention values are present.

Naturally, one might infer that when the sink pattern appears, we excite the image token. 

To further investigate this phenomenon, as depicted in <ref>, we find that the sink phenomenon is not overtly evident in the shallow layers. This is because the shallow layers tend to focus more on encoding semantically rich information <cit.>. 

When the encoding of semantically rich tokens stabilizes, the attention sink phenomenon arises. 

Therefore, we build upon the judgement of intervention timing by calculating the similarity of the hidden states.



 §.§ Image-Centric Logit Refine

In <ref>, we observe a peculiar phenomenon where LVLMs continue to generate identical hallucinated text even when the image is removed from the input. 

This observation naturally leads us to the concept of using the output distribution (when no image is 
in input) as a reference to penalize our initial prediction distribution. 
Therefore, we update the distribution of the generated token by:

    = γ·( | _V, _I, _H) 
          - (γ - 1) ·( | _I, _H).

This equation effectively reduces the predicted probability based on text alone. The weight γ is used to control the degree of penalty applied to the initial prediction distribution. 

This operation is conceptually similar to LLM-CFG <cit.>. 

Essentially, it provides a guided generation mechanism that allows the model to make informed choices between outputs based on image content and those based on language logic. 

This way, the model can better balance the influence of visual and textual information in its outputs, leading to more contextually accurate and relevant results.





§ EXPERIMENTS



 §.§ Setup

Baselines. 
We evaluate the effectiveness of our method on three different models. 

To better compare the impact of image feature tokens after different projectors on our method, we selected two models that use linear projectors, LLAVA and Shikra, as well as one model that uses resamplers, Minigpt4. 

Additionally, for a more convincing comparison, we report on three decoding methods for comparison: greedy, beam search, and nucleus sample. 

We also selected the OPERA <cit.> method, which is an improvement on beam search, and the VCD <cit.> method, which is an improvement on nucleus sampling, to compare with our results. We used the default hyperparameters from the open-source versions of these two methods.

Implementation Details. 
As different models have different lengths of image tokens, leading to different degrees of image neglect, to better align with the image sequence length of the model, we set α=0.5 for LLAVA, α=0.6 for Shikra with long image token sequence lengths and α=0.2 for resampler models with short image token sequences. 

As text inertia is independent of the image token length, we continuously use γ=1.1. 

Apart from this, in the beam search tests, the beam number is set to 5 for all methods, and in the nucleus sample tests, all the common parameters are consistent.



 §.§ Benchmark & Evaluation Metrics

CHAIR <cit.>. Caption Hallucination Assessment with Image Relevance (CHAIR) is a widely used metric in image captioning tasks. CHAIR operates by creating a set of ground-truth object labels for each image. Any object mentioned in the caption that does not exist in the label set is considered a hallucinated object. CHAIR comprises two evaluation dimensions: instance-level and sentence-level, represented as CHAIR_I and CHAIR_S, respectively. These are calculated in the following manner:

    CHAIR_I = |{hallucinated objects}|/all mentioned objects, 
    CHAIR_S = |{captions with hallucinated objects}|/all captions.
   
We conducted experiments based on the validation set of MSCOCO 2014. 

Given “” as prompt, we subsequently employ the CHAIR metric to evaluate the generated description.

To evaluate hallucinations in long sequence generation, we adopted the same setup as used in LURE <cit.> and OPERA <cit.>, setting the max_new_tokens parameter to 512 and randomly sampled 500 instances for evaluation.

POPE <cit.>. The Polling-based Object Probing Evaluation (POPE) is a evaluation metric designed in the VQA paradigm.

POPE serves as a metric for assessing object hallucination, evaluating hallucinations by asking LVLMs questions such as “” Here,  is replaced with the constructed ground-truth object from three different types of splits. 

In the “random” split, objects are randomly selected from the entire dataset for evaluation. 

In the “popular” split, objects are chosen from those most frequently appearing in the dataset. 

In the “adversarial” split, objects that are highly related to the image objects are selected for evaluation. 

We conduct our evaluation on the COCO dataset with 500 images, with each image having 6 questions for each split of POPE. 

We evaluate the performance of the model in object recognition tasks using both accuracy score and F1 score.

Besides, in order to more comprehensively examine hallucinations in multi-turn chats, we have structured the evaluation in both single-turn and multi-turn dialogue forms.

MMHal-Bench <cit.>. For further evaluation of our method on some challenging datasets, we choose MMHal-Bench, which is designed with 96 image-question pairs, spread across 8 question categories × 12 object topics. 

It contains eight types of questions about object attributes, adversarial objects, comparisons, counting, spatial relations, environment, holistic descriptions, and others to comprehensively assess the model’s hallucination performance on high-difficulty datasets. 

Essentially, it is also a VQA-based evaluation, but unlike the existence-based examination in POPE, its questions also include some logical considerations. 

Therefore, for evaluation on MMHal-Bench, we first need to answer questions and then use GPT-4 to score the answer based on the response and the ground-truth answer. 

The evaluation results include the model’s scores across all question categories, and the overall score represents the average of these scores.

GPT-4v Assisted Evaluation. To further evaluate the model’s performance in image description tasks, we can move beyond the CHAIR metric, which is based on information extraction and only considers object hallucination. 

We can use GPT-4v for open evaluation. As with previous evaluations <cit.>, we sample 50 images on the COCO dataset for evaluation. 

We construct prompts, input images into GPT-4v, along with the descriptions responses from two assistants. 

GPT-4v evaluation takes into account two dimensions: Accuracy and Detailedness, denoted respectively as C and D.

Detailed prompt construction can be found in the appendix.



  



 §.§ Experimental Results

In this section, we analyze the performance of PAI across various hallucination evaluation tasks, including long image description, simplified VQA answer, construction of metric evaluation, and leveraging the near-human cognitive capabilities of GPT-4/GPT-4v as evaluation methods. 

For further analysis, please refer to the appendix.
































































































Results on long sequence hallucination evaluation. 
The experimental results are presented in <ref>. 

As our approach is an inference intervention method, it differs from previous decoding hallucination mitigation methods that primarily concentrate on improving a single decoding method.

We have tested our method on three decoding techniques. 

Our method has achieved hallucination mitigation on all three decoding methods used by the three models. 

However, when integrated with the nucleus, a sampling-based method, the hallucination reduction brought about by our method is not significant. 

This may be because even though our method has increased the priority of trustful tokens, the sample set during nucleus decoding still contains many hallucination tokens.

Moreover, while OPERA significantly mitigates hallucinations, its time efficiency is considerably higher compared to vanilla. 

In contrast, our method not only has almost the same time efficiency as vanilla, but it also performs better in reducing hallucination issues. 

Compared to VCD, during the generation process of long sequence tasks, the introduction of visual uncertainty during decoding sometimes leads to more hallucination descriptions. 

However, our method can reduce the proportion of hallucination words in the sample pool.

Results on single-turn and multi-turn hallucination evaluation. Unlike the CHAIR evaluation, POPE is in a VQA format, so the response is brief, answering only “Yes” or “No”. The phenomena of text inertia and image neglect may not be as noticeable under this setting, especially for single-turn. However, our method still achieved a notable improvement compared to the vanilla decoding method in single-turn. 

As for multi-turn evaluation, which involves a longer context, our method achieves more significant improvement as presented in <ref>. 

       

Results on hallucination evaluation in comprehensive general scenarios. 
The experimental results, as shown in <ref>, indicate that for some more image-based question types, such as object attributes, adversarial objects, and holistic questions, the answers are more accurate when inference intervention with PAI is applied, and there is a certain degree of improvement across all models. However, for some logical questions, such as comparisons and relations, there is no noticeable improvement after intervention. In summary, through the overall metric, i.e., the average of the eight evaluation dimensions, there is a certain degree of improvement compared to the baseline after incorporating PAI.

Results on human-like GPT-4v assisted hallucination evaluation. 
The experimental results, as shown in <ref>, indicate that even when more comprehensive hallucination evaluation dimensions are added, our method, compared to the greedy decoding method, can provide more accurate responses on all three models without losing detail in the description. Given that GPT-4v’s visual understanding and language logic capabilities have reached a level close to that of humans, it can more comprehensively illustrate the performance improvements brought about by our method.
 






















    














  





































 §.§ Ablation Study

Our method, PAI, consists of two stages of interventions. In the first stage, during forward inference, the hyperparameter α is utilized to set the scale of intervention. Simultaneously, the layer prior, represented as “L”, is used to determine the attention layer for intervention. The second stage unfolds during the decoding process, where we mitigate text inertia by subtracting the logits distribution that results from inputs devoid of image information. In this stage, the scale is managed by the parameter γ.

We use LLaVA-1.5 as the representative LVLM baseline and the greedy decoding method as the basic baseline to compare the impact of our hyperparameters on the task of long sequence image description. To evaluate our method, we choose the CHAIR metric. However, since CHAIR only evaluates the hallucination problem, we have incorporated the F1 score to consider information richness and accuracy. This makes the comparison with the CHAIR metric fairer when the F1 scores are similar. In the above, the F1 scores of the various methods differ slightly, so we have not included this somewhat redundant measure. The F1 score is calculated by counting the objects included in the description, the objects in the ground-truth sets, and the hallucinated objects.

Effects of α in Exciting Image Attention.
In the process of exciting the attention values of image tokens, we introduce a parameter α to control the amplification scale. As shown in  <ref>, different LVLMs exhibit varying sensitivity to the amplification scale. This sensitivity not only depends on the length of the model's image tokens (e.g., the image token length of LLaVA-1.5 is 576, while that of Minigpt4 is only 32), but also the original attention weights distribution. 

However, a commonality across these models is that an appropriate amplification scale can achieve a balance between the number of hallucinated objects in the description and the amount of information conveyed. If the scale is too small, the description may still contain many hallucinated objects. Conversely, if the scale is too large, the amount of information in the response will decrease.



Effects of Layer Prior in Exciting Image Attention. 
We further investigate the control of the intervention layer for exciting image attention. As observed in <ref>, the introduction of this prior does indeed improve the performance of our method. However, when there is no control over the intervention layer and interventions are applied to all layers, the performances of different models exhibit some variations. For Shikra, in the absence of the layer prior, our method's results regress to the baseline. For both LLaVA and Minigpt4, the loss of the layer prior causes some unfavorable fluctuations in both the CHAIR metric and the F1 score.

Effects of γ in Mitigating Language Prior. 
<ref> presents the results of an ablation study focusing on γ, which adjusts the balance between output distributions from conditioned inputs with excited image tokens and pure text inputs. Unlike the other two models, the Minigpt4 model is highly sensitive to γ. When γ is too large, it can lead to uncontrollable model behavior. The experimental results suggest that maintaining γ within a relatively small range, such as 1.1 - 1.2, yields the most stable performance.



§ CONCLUSION AND LIMITATION

In this paper, we first analyze the causes and manifestations of hallucinations in LVLMs. We propose a phenomenon termed "text inertia," wherein the model continues to produce the same hallucinatory descriptions even when no image input is provided. This issue fundamentally stems from the model's neglect of image tokens. Consequently, we introduce the PAI method to intervene in the model's inference process, steering it towards an image-based and trustworthy direction. This is a training-free method and does not require any external tools. Extensive experiments on multiple benchmarks and LVLMs have validated the effectiveness of PAI in mitigating hallucination issues.

Limitations: (1) The language decoders of existing open-source LVLMs are primarily models from the LLaMA-family. It is worth exploring whether the issues of image neglect and text inertia are introduced by LLaMA. (2) As we described, our method fundamentally alleviates the image neglect issue during model inference. Its upper limit depends on the capabilities of the well-trained model. Therefore, it is worth investigating whether incorporating this issue as a loss during the training process could lead to further performance improvements.


§ ACKNOWLEDGEMENTS

This work is supported by the National Science Foundation of China (62132017), Zhejiang Provincial Natural Science Foundation of China (LD24F020011).
splncs04












§ APPENDIX




§ TEXT INERTIA DETECTION PROCESS


To detect the text inertia phenomenon in LVLMs during image description tasks, we approach this issue in two stages. In the first stage, we employ the CHAIR metric to identify the indices of hallucinated objects in the LVLMs’ descriptions. In the following stage, we extract the tokens preceding the hallucinated objects, feed them into the LVLMs, and proceed with generation, excluding the image from the input. In other words, we only provide the LVLMs with the task instruction and history response up to the index of the hallucinated object. Finally, we extract the first ten tokens from the newly generated output and use GPT-4 to determine whether the same object has been generated. Our prompt structure is shown in  <ref>.

    

Here, {object} is replaced by the hallucinated object that we extract each time, and {description} is replaced by the first ten tokens of the newly generated conditioned description. When the model answers “Yes”, it indicates the presence of the text inertia phenomenon.

  
  
  



§ WHAT IF MODEL SCALES UP?

We performed experiments on the LLaVA-1.5-13B scale model to assess how the performance of PAI varies with increasing model scale. As shown in  <ref>, our method maintains its robustness regardless of the scale of the model. Notably, it continues to alleviate the hallucination issue in both long sequence and short VQA tasks as the model scale increases.



§ COMPARATIVE EXPERIMENTAL RESULTS

We compare our method with a different intervention approach during model inference. Prompt Highlighter <cit.> proposes an interactive technique for constructing visual prompts to steer model generation. Essentially, it adds a value of the same size to the attention mask of the tokens in the user's guidance area to influence the model's output. To automate the intervention process for evaluation, we adopt an approach similar to Prompt Highlighter. This involves adding a constant value of 1.0 to the attention mask of all image tokens, where 1.0 is the default value provided by their open-sourced code.

The final experimental results are presented in <ref>. The findings indicate that uniformly applying intervention across the entire image is less effective than enhancing image attention based on the original scale.







 



§ OBTAINING EXPLAINABLE LVLMS.
 
The extent to which attention can serve as an explanatory mechanism has been extensively studied <cit.>. 

Within the scope of LVLMs, the self-attention maps in the LLaMA decoder have been regarded as a natural explanation for the model. 

However, these explanatory results are only available for LLAVA and Shikra, due to the usage of linear projection that preserves all image patch token features. 

The model, which employs a resampler to derive the explanation, has to reverse-engineer the abstracted image token back to the input image patch token. However, tracing back the image patch after it has been encoded by the visual encoder and resampled into an image token proves to be challenging.

As illustrated in <ref>, the original LLAVA identifies the correct object, a mouse, and hallucinates an object, a motorboat, both of which have limited interpretability as the results do not focus on the main body of the described object. 

With our method of layer intervention, the response becomes more image-centric. 

The description aligns better with the image location, which can all be considered as a more faithful explanation.


  


 


 


 















 Paper Title: MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities.




[NO \title GIVEN]
    [NO \author GIVEN]
    August 5, 2024
======================



[remember picture,overlay,shift=(current page.north west)]
[anchor=north west,xshift=0.7cm,yshift=-2cm]-1[1]
    < g r a p h i c s >
;





§ ABSTRACT 
MM-Vet, with open-ended vision-language questions targeting at evaluating integrated capabilities, has become one of the most popular benchmarks for large multimodal model evaluation. MM-Vet assesses six core vision-language (VL) capabilities: recognition, knowledge, spatial awareness, language generation, OCR, and math. However, its question format is restricted to single image-text pairs, lacking the interleaved image and text sequences prevalent in real-world scenarios. To address this limitation, we introduce MM-Vet v2, which includes a new VL capability called “image-text sequence understanding”, evaluating models' ability to process VL sequences.
Furthermore, we maintain the high quality of evaluation samples while further expanding the evaluation set size.
Using MM-Vet v2 to benchmark large multimodal models, we found that Claude 3.5 Sonnet is the best model with a score of 71.8, slightly outperforming GPT-4o which scored 71.0. Among open-weight models, InternVL2-Llama3-76B leads with a score of 68.4.




§ INTRODUCTION


 


Large multimodal models (LMMs) <cit.> evolve rapidly, demonstrating emergent abilities to solve complex tasks <cit.> that required multiple integrated capabilities, such as GUI navigation <cit.>, screenshot to code <cit.>, video understanding <cit.>, . To comprehensively evaluate LMMs, multiple benchmarks have been proposed, such as MME <cit.>, MMBench <cit.>, SEED-Bench <cit.>, MMMU <cit.>, and MM-Vet <cit.>. Notably, MM-Vet is designed to evaluate LMMs from a capability integration perspective, defining tasks based on their required core capabilities. The benchmark accepts open-ended responses and takes GPT-4 <cit.> to score model predictions, which better align with real-world scenarios. Such effective evaluation designs make MM-Vet widely utilized as a standard benchmark for LMM evaluation, as indicated by its leaderboard [MM-Vet leaderboard: <https://paperswithcode.com/sota/visual-question-answering-on-mm-vet>].

Despite its popularity, MM-Vet and other concurrent evaluation benchmarks <cit.> are beginning to fall short in assessing the more advanced capabilities that have emerged in the latest LMMs, such as GPT-4V <cit.> and its successors <cit.>. Specifically, one major limitation is the question format. The questions in MM-Vet are limited to a single image-text pair, lacking the capability to handle interleaved image and text sequences. This design choice was natural at the time of prior studies <cit.>, given that most LMMs only supported single image inputs. However, the ability to process arbitrarily interleaved image-text sequences is crucial for advanced LMMs and should be included in LMM evaluation. 

In addition to the six core capabilities defined in MM-Vet, we introduce an additional capability: “image-text sequence understanding.” This measures the LMMs' ability to process image-text sequential data, as shown in Figure <ref>. For example, to complete the task in Figure <ref>(c), the LMM needs to understand the text question followed by multiple sequential images. This task requires image-text sequence understanding alongside the capabilities of recognition, OCR, knowledge, and language generation defined in MM-Vet. The image-text sequence can contain multiple images of drastically different usages, such as multiple video frames in Figure <ref>(c) for temporal understanding, and spot the difference challenge in Figure <ref>(d) for image comparison. These capabilities to comprehend arbitrarily interleaved image-text sequences is one fundamental step towards stronger general intelligence.


Another limitation is the size of the evaluation set. MM-Vet only has 217 evaluation samples due to the difficulty of high-quality data collection. We aim to maintain the same high quality while further expanding the evaluation set size. 
We break down the challenge of creating high-quality evaluation samples into two steps: generating high-quality questions and producing reference answers. For the question generation, we find it difficult for crowd-sourcing workers to propose a meaningful and complicated question that covers various scenarios, , in Figure <ref> (a,b). 
Instead, we have researchers design and collect 517 questions covering various scenarios from daily life to expert/industry applications, extended from an exploratory report on GPT-4V <cit.>. For straightforward questions that can be answered in a few words, our experts directly annotate the reference answer. For questions that need long paragraphs to answer, we first employ the GPT-4V <cit.> to draft the response. Next, our experts correct draft if there is any error and then rephrase it into the final ground truth. 


After developing MM-Vet v2, we evaluate multiple advanced LMMs. Claude 3.5 Sonnet <cit.> achieves the highest performance with a score of 71.8, slightly surpassing GPT-4o <cit.> by 0.8 points. Notably, InternVL2-Llama3-76B <cit.>, an open-weight model, also delivers a very competitive score of 68.4.






























§ DATASET AND EVALUATOR





The same as MM-Vet <cit.>, we aim to build a high-quality evaluation set for large multimodal models. MM-Vet <cit.> defines six core vision-language capabilities, including recognition (Rec), knowledge (Know), OCR, spatial awareness (Spat), language generation (Gen), and Math. MM-Vet's question format is only an image-text pair, which obviously cannot measure the capability of processing sequential image and text data. To fill this gap, we introduce a new capability:

    
  * Image-text sequence understanding (Seq). It refers to the capability to understand and reason the relationships among sequential image and text streaming data. 


Then we carefully design 517 questions that require one or more capabilities, extended from MM-Vet <cit.> and an exploratory report based on <cit.>. Specifically, 218 questions are directly from MM-Vet <cit.>, and the others are newly collected in various domains from daily life to expert/industry applications. For those questions that can be answered shortly, we annotated the ground truths directly. For questions requiring long text to respond, we first utilize GPT-4V <cit.> to generate the answer drafts, then we carefully correct the wrong information in the drafts and rephrase them into final ground truths. Because this new dataset is the extension of MM-Vet <cit.>, we name it . See Figure <ref> for the proportions of each core capability and capability integration.








Next, we design the evaluator for , based on that of MM-Vet <cit.>. Different from MM-Vet, the input format of  is not only an image-text pair, it can also be image-text sequences. To demonstrate the position of the image in the sequences, we utilize <image> to represent the image position. The few-shot prompt for  is shown in Table <ref>.






§ EXPERIMENTS








 §.§ Experiment settings

We evaluate two types of LMMs on our : (1) open-weight LMMs including OpenFlamingo <cit.>, Otter <cit.>, LLaVA <cit.>, CogAgent <cit.>, Emu2-Chat <cit.>, InternLM-XComposer2 (IXC2) <cit.>, CogVLM <cit.> and InternVL <cit.>; (2) Closed source models including Claude <cit.>, Qwen-VL Max <cit.>, Gemini <cit.>, GPT-4V <cit.> and GPT-4o <cit.>).

As illustrated in <ref>, for each sample, we complete the prompt template with its question, ground truth, and the output from a specific LMM. When this filled prompt is input into GPT-4, it generates a score ranging from 0 to 1 for each sample. Although the temperature is set to 0, we observe some variance in GPT-4's outputs. To address this, we evaluate the outputs of the LLMs using GPT-4 five times. Due to space constraints, we present the average scores for capabilities and for some capability integrations, and include both the average and variance for the total score.





 §.§ Results

The main results of the different methods are presented in Table <ref> for each capability, and in Table <ref> for 16 integrations with the highest proportions.
Claude 3.5 Sonnet <cit.> and GPT-4o <cit.> are the leading models, achieving scores of 71.8 and 71.0, respectively. Claude 3.5 Sonnet <cit.> excels in recognition, language generation, OCR, spatial awareness, and knowledge. On the other hand, GPT-4o <cit.> surpasses in image-text sequence understanding and math. Among open-weight models, InternVL2-Llama3-76B stands out with a competitive performance score of 68.4.






















§ CONCLUSION

In this paper, we aim to evaluate the integrated capabilities of large multimodal models and extend MM-Vet into  by introducing a new core capability: Image-text sequence understanding, which assesses the ability to process vision-language sequences. Additionally, we ensure the high quality of evaluation samples while expanding the evaluation set size. Using  to benchmark large multimodal models, we found that Claude 3.5 Sonnet achieved the highest score of 71.8, narrowly surpassing GPT-4o, which scored 71.0. Among open-weight models, InternVL2-Llama3-76B emerged as the leader with a score of 68.4.












plain





 Paper Title: From Feature Importance to Natural Language Explanations Using LLMs with RAG.






2024
Copyright for this paper by its authors.
  Use permitted under Creative Commons License Attribution 4.0
  International (CC BY 4.0).



MAI-XAI'24: Multimodal, Affective and Interactive eXplainable AI,
  October 19–20, 2024, Santiago de Compostela, Spain










1]Sule Tekkesinoglu[
  orcid=0000-0002-1232-346X,
  email=sule@robots.ox.ac.uk,
  ]
[1]
[1]Oxford Robotics Institute, University of Oxford, Oxford, UK

1,2]Lars Kunze[
  orcid=0000-0001-5302-1938,
  email=lars@robots.ox.ac.uk,
]
[2]Bristol Robotics Laboratory, University of the West of England, Bristol, UK




[1]Corresponding author.





§ ABSTRACT 
  As machine learning becomes increasingly integral to autonomous decision-making processes involving human interaction, the necessity of comprehending the model's outputs through conversational means increases. Most recently, foundation models are being explored for their potential as post hoc explainers, providing a pathway to elucidate the decision-making mechanisms of predictive models. In this work, we introduce traceable question-answering, leveraging an external knowledge repository to inform the responses of Large Language Models (LLMs) to user queries within a scene understanding task. This knowledge repository comprises contextual details regarding the model's output, containing high-level features, feature importance, and alternative probabilities. We employ subtractive counterfactual reasoning to compute feature importance, a method that entails analysing output variations resulting from decomposing semantic features. Furthermore, to maintain a seamless conversational flow, we integrate four key characteristics – social, causal, selective, and contrastive – drawn from social science research on human explanations into a single-shot prompt, guiding the response generation process. Our evaluation demonstrates that explanations generated by the LLMs encompassed these elements, indicating its potential to bridge the gap between complex model outputs and natural language expressions.






  Explainable AI LLMs Subtractive counterfactual reasoning Retrieval-Augmented Generation (RAG) 









From Feature Importance to Natural Language Explanations Using LLMs with RAG
    [
    Received XXX; accepted YYY
============================================================================





§ INTRODUCTION


As we become increasingly reliant on AI applications in our daily lives, it becomes imperative to interact with autonomous decision-making systems in human-understandable terms <cit.>. Regulatory frameworks and standards are also evolving to require transparency and accountability in AI-driven systems <cit.>. The technical necessity for explanations to facilitate model debugging and rectifying potential ethical and legal risks arising from biases and errors has been extensively discussed in the literature <cit.>.

The increasing demand for transparent and interpretable AI models has prompted the exploration of various explanation methods, resulting in significant progress in both inherently interpretable and post-hoc explainability methods <cit.>. While effective, these methods struggle to provide explicit interpretability or intuitive explanations for non-technical users. Consequently, there has been a concerted effort to integrate interpretability approaches with Natural Language Processing (NLP) <cit.>. Recently, Large Language Models (LLMs) have gained acclaim as post-hoc explainers, showcasing their potential to elucidate decisions made by other predictive models. Despite their success, a persistent challenge for these models is the occurrence of object hallucinations—instances where plausible yet incorrect outputs are generated, such as featuring objects that do not exist in the images <cit.>. This raises questions about the fidelity of explanations to the underlying model. To address this issue, we propose a traceable question-answering, which informs LLM responses through an external knowledge source that provides insights into the model's output. By utilizing this data, the LLM acts as a reasoning engine to process the information rather than a source of information. Figure 1 provides a high-level depiction of this process.  

Additionally, to cultivate a seamless conversational experience, we incorporated key elements derived from social science research on human explanations into the system prompt, directing the response generation process. We present our vision for creating this integrated approach, outlining each component, and illustrating its viability through a real-world dataset. This integration has the potential to bridge the gap between intricate model outputs and user comprehension towards advancing the development of human-understandable explanations. 







§ BACKGROUND



 §.§ Recent advances in image explanations

The demand for transparent and interpretable AI models has led to exploring diverse explanation methods for vision tasks <cit.>. Recent years have witnessed significant progress in both intrinsic and post-hoc explainability methods. Gradient-based techniques, which analyse the gradients of the model's output with respect to its input features, have been extensively studied <cit.>. While effective, these methods may fall short in providing intuitive explanations for non-experts. In another line of work, attention mechanisms are proposed by assigning weights to different parts of the input data based on their relevance to the model's decision <cit.>. Similarly, attention mechanisms may lack explicit interpretability, and some argue that attention alone is insufficient. As a result, there have been concerted efforts to integrate these approaches with Natural Language Processing (NLP) <cit.>. Long Short-Term Memory (LSTM) encoders have been employed to extract features from video clips, with LSTM decoders generating descriptive texts based on these features. While effective in image captioning, these methods provide textual descriptions rather than visual explanations. Moreover, researchers have proposed perturbation-based approaches—a model-agnostic method that perturbs input data and observes changes in model predictions. Building on this research, we integrate semantic perturbation analysis with pretrained language models, translating the results into the text modality for enhanced explainability.  



 §.§ LLMs as post hoc explainers

Recent strides in Large Language Models (LLMs), exemplified by GPT-4 (OpenAI), Bard (Google), Claude-2 (Anthropic), and Llama-2 (Meta), mark a transformative era in Natural Language Processing (NLP) research. These models have become pervasive across diverse applications, ranging from machine translation and question-answering to text generation <cit.>. Notably, LLMs have recently gained attention as post hoc explainers, highlighting their potential to elucidate the decisions made by other predictive models <cit.>. Leveraging their in-context learning capabilities by fine-tuning for specific tasks enhances their proficiency in generating contextually relevant explanations <cit.>. Despite their success, a persistent challenge for these models is the occurrence of object hallucinations—instances where plausible yet incorrect outputs are generated, featuring objects that do not exist in the images in vision-based tasks. This issue raises questions about the faithfulness of explanations to the underlying model. One way to address this is by recognizing the role of the semantic map layer in capturing the meaning and context of the physical surroundings <cit.>. By effectively encoding valuable semantic information into LLMs, we can truthfully represent and comprehend the intricate details of the model output within the language space.



 §.§ Characteristics of explanations and XAI


In the context of XAI, the audience for explanations is predominantly human; thus, understanding what makes an explanation human-friendly is paramount. While accuracy is important, the clarity and presentation of explanations are equally crucial in ensuring comprehensibility. According to Miller <cit.>, current work on interpretability in machine learning relies solely on researchers' intuition regarding what constitutes an appropriate explanation for humans. Miller's survey highlights four major characteristics of explanations drawn from findings in social science research on human explanation: explanations are social, causal, selective, and contrastive <cit.>. In the following section, we briefly discuss these characteristics in relation to XAI before demonstrating how we integrate them into our application.



  §.§.§ Explanations are social

An explanation represents an interactive exchange between two roles: the explainer and the explainee, and is governed by certain `rules' <cit.>. Interactive explanations in a dialogue structure consist of three fundamental components: opening statements, clarification questions, and closing statements <cit.>. As such, it should follow the basic rules of conversation, which are captured by Grice's conversational maxims <cit.>. These include principles such as quality, quantity, relation, and manner, which can be understood as `only say what you believe', `only say as much as is necessary', `only say what is relevant', and `say it nicely'. Manner further encompasses various sub-maxims, including avoiding unclear expression, eliminating ambiguity, maintaining brevity to avoid unnecessary wordiness, and presenting information in an orderly fashion. 

Moreover, explanatory agents employing anthropomorphic traits like politeness markers (e.g., “Thank you!”), warm and friendly tones, and empathetic expressions (e.g., “I understand.”) are pivotal for effective human interaction <cit.>. Theory of Mind also plays a crucial role in social explanations.[Theory of mind refers to our ability to understand other people by attributing mental states to them, including beliefs, desires, intentions, emotions, and thoughts that may differ from our own <cit.>.] As an integral part of a dialogue, an explanatory agent should keep track of what has already been explained, maintaining a simple mental model of others. This enables intelligent agents to tailor explanations based on past interactions and evolving contexts.



  §.§.§ Explanations are causal

Causal explanations aim to elucidate the cause-and-effect relationship underlying a decision. When explaining a prediction made by a machine learning model, identifying the most influential features can provide insight into the key causal connections to the prediction. This process, also known as causal inference,  starts with observations (e.g., what if a feature had a different value) and selecting some of those causes as the explanation. Miller divides casual inference into two parts: counterfactuals and abductive reasoning <cit.>.

The reasoner uses a model of hypothetical counterfactual cases to derive an explanation, such as perturbing inputs, to see how they causally impact the prediction (See Section <ref>). Abductive reasoning is inferring the causes from those observations to form the most probable explanation. One can think of abductive reasoning as the following process: observing unexpected or surprising events, generating one or more hypotheses about these events, judging the plausibility of the hypotheses, and selecting the `best' hypothesis as the explanation <cit.>. It is important to note that new information (e.g., different feature perturbations) can alter the most likely explanation.




  §.§.§ Explanations are selected


Explanations are contextual by nature. While an event may have numerous causes, the recipient of the explanation typically cares about only a specific subset relevant to the context <cit.>. The explainer selects a subset based on various criteria called casual selection. Hilton <cit.> argues that explanation selection is used for cognitive reasons since causal chains are often too large to comprehend. The criteria people use for explanation selection include abnormality, relevancy, and simplicity. 

Abnormality is a key criterion for explanation selection; as such, people select unusual causes to explain events <cit.>. In machine learning interpretability, identifying abnormal events corresponds to deviant or higher feature importance values. As pointed out within the conversational maxims, relevancy also plays a role in explanation selection—only say what is necessary and relevant. People select explanations to adhere to these maxims whilst persuading the explainee to new information or viewpoints. Another strong criterion in explanation selection is simplicity, which refers to those that cite fewer causes. People prefer uniquely sufficient causes that bring about the effect without any other causes, but there might be cases with multiple sufficient causes.  




  §.§.§ Explanations are contrastive


Studies have shown that people only request contrastive explanations, essentially `why' questions framed to imply the differences between two possible outcomes <cit.>. As a result, explanations are typically offered in relation to the cause of one event compared to another that did not occur.

Similarly, in the context of ML interpretability, people are not specifically interested in all the factors that led to the prediction but instead in the factors that need to change so that the prediction would also change. Explanations that present some contrast between the instance to explain and a reference point (i.e., a hypothetical instance) are preferable because the cognitive burden of complete explanations is too high. However, most existing work considers contrastive questions but not contrastive explanations, providing two complete explanations individually. This could be because the contrastiveness is application-dependent, given that a reference point may not be apparent in all tasks <cit.>. For instance, in sentiment analysis or anomaly detection, contrastive cases might not be as straightforward as other tasks, such as banking, e.g., loan accepted vs rejected cases.



§ MATERIALS AND METHOD


This section outlines the key elements of the traceable question-answering methodology. We detail the proposed explainability approach for extracting feature importance and the prompting technique to generate accurate and human-friendly explanations. Additionally, we provide an application example illustrating how we establish a connection between the model output, the explainability technique, and language models to produce natural language explanations in an interactive format. 











 §.§ Explanations through subtractive counterfactual reasoning



Subtractive counterfactual reasoning is the process of removing an event to understand its causal influence on an outcome <cit.>. Such reasoning is widely used as an XAI technique, exemplified by occlusion analysis and feature importance attribution, built on the same foundation - quantifying the variation in model output through systematic modifications of input components <cit.>. There are distinctions in terms of how these methods remove features, elucidate model behaviour, and summarize feature importance. Nonetheless, they provide valuable insights into the relative significance of each feature influencing the decision-making process <cit.>. 

This study proposes a decomposition-based approach to examine the output variations by decomposing input values. By measuring such variations, we determine the degree of impact on an outcome and reason about the importance of each feature value. As a function, the model assigns a class label and prediction probability f:x→ f(x) given input y. To measure the effect of each input value, we observe the model's prediction p for y without the knowledge of event T_i=T_k, where T_k is the value of feature T_i. To explicitly represent the absence of information, the feature value T_k is replaced with an undefined value, i.e., NaN (Not a Number). NaN values are treated as invalid or masked elements and are not considered in the computation. We attain an array containing probabilities for all perturbations resulting from decomposition. In order to determine the importance of an individual feature T_i, we begin by identifying the max and min values within this array, which establishes the decision boundaries for the given input. Next, we compute the feature importance as the position of the probability– after decomposing the feature– relative to the max and min probabilities. The computation is expressed as follows:


    importance(T_i) = p(x|y ∖ T_k)_max - p(x|y ∖ T_k)/p(x|y ∖ T_k)_max - p(x|y ∖ T_k)_min


This approximates an importance value between zero and one for each input feature. Considering that evaluating fractions has a higher cognitive load, the values are rounded to the nearest integer to increase comprehensibility. Moreover, this analysis is carried out for the categories following the main prediction simultaneously to generate explanations for the contrastive cases. 





 §.§ Prompting for traceable question-answering with RAG


Traceable question-answering, through a process known as Retrieval-Augmented Generation (RAG), integrates external knowledge sources to furnish responses to user inquiries. This method improves the performance of LLMs by referencing an external knowledge source prior to generating a response (See Figure <ref>). This approach mitigates the problem of `hallucination' and ensures factual consistency, particularly in scenarios where facts may evolve over time <cit.>. Depending on the specific implementation, RAG supports various file formats, including text, PDF files, word documents, structured data (tables), and images. 

In our work, the knowledge repository contains tabular information about model output, features, contrastive cases, and feature importance values obtained through the decomposition-based explanation method described in Section <ref>. This information is saved as the output of the explanation module as a .csv file. Then, the file, the system prompt (See Figure <ref>), and the user input query are fed to the LLM model for response generation. The resulting responses contextualise the feature importance grounded in the models' inner workings, thereby rendering the explanations traceable to the underlying model.   

Furthermore, our aim is to imbue LLM with the characteristics of explanations discussed in Section <ref> to foster a human-friendly interaction. Ideally, LLM-generated responses should adhere to conversational norms, encompassing opening, clarification, and closing statements while appropriately integrating social cues. The model is expected to deduce causal relationships from the observations within the provided knowledge repository, selecting the most pertinent causes to form an explanation and presenting it alongside a contrastive case. To accomplish this goal, we devised a system prompt that provides explicit instructions to the model to generate efficient responses to user queries. The system prompt includes a single-shot example illustrating the desired interaction, potential user inquiries and outlining how the model can effectively address them, enabling the language model to produce consistent and predictable outcomes, as presented in Figure <ref>.


















 §.§ Application example
 

To demonstrate our approach to traceable question-answering, we selected a visual scene understanding task. This involves scene classification, categorizing images based on their content and inferring high-level semantic scenes from low-level visual features. Many critical applications of scene classification include human-robot collaboration, autonomous driving, and other autonomous physical systems that rely on semantic segmentation to inform decision-making processes <cit.>. We experimented with the GoogleNet Places365 model designed specifically for scene recognition and classification tasks <cit.>. The model is trained on the Places365 dataset, containing more than 365 scene categories.  We focus on semantic urban scene understanding with the road segmentation dataset CamVid (Cambridge-driving Labelled Video Database). The dataset helps to understand driving scenes, comprising images capturing street-level views from urban driving scenarios. It includes manual annotations across 32 classes, encompassing elements such as building, driveway, pavement, tree, traffic sign, car, pedestrian, and bicyclist. The Deeplab v3+ deep learning model is trained on the CamVid dataset for semantic segmentation <cit.>. Figure <ref> shows the segmentation result for the example presented in the next section. Since we utilized off-the-shelf models for illustration purposes, the emphasis is not on these models' performances.



Semantic information plays a critical role in autonomous vehicle (AV) decision-making processes, as it captures the meaning and context of the physical surroundings observed in the road scene. This includes understanding the appearance of various elements such as buildings and roads, identifying shapes such as cars and pedestrians, and recognizing spatial relationships to contextualize the scene <cit.>. Particularly in unstructured off-road environments, the accurate understanding of the surroundings concerning semantic classes such as trail, grass, or rock is important for safe and deliberate navigation <cit.>. Moreover, semantic segmentation provides a way to represent and comprehend the details of the environment in the language space. By leveraging the labelled information within the image scene, we can semantically map the crucial components based on their feature importance. This granularity enables a more insightful analysis of feature importance, contributing to a nuanced understanding of the model's decision-making process. Such insights can be helpful for post-incident forensic analysis in case of misreading a scene causing collisions or accidents. In this work, we suppose situations where the user checks in with an AI assistant in an AV system in non-critical driving scenarios while engaging in non-driving related tasks.































Table <ref> reveals the content of the knowledge source - only the first contrastive case is displayed. The effect of removal is the probability when a feature is removed, as listed respectively. Given the system prompt and the data presented in Table <ref>, Figure <ref> showcases the dialogue generated for this case. From a social standpoint, the resulting responses balance between being informative and supportive while maintaining a friendly and engaging tone. It effectively justifies the model's output by emphasising its most significant feature in a concise manner. Additionally, the dialogue delves into contrastive explanations by detailing three consecutive classes of lower likelihood, demonstrating the flexibility to explore different scenarios. When comparing the top prediction to the second most likely class, it highlights the distinguishing features contributing to the output. 













§ EVALUATION RESULTS

In this section, we present our experimental findings. We analyzed nine distinct scene classes — parking lot, street, residential neighbourhood, crosswalk, highway, industrial area, gas station, shopfront, and general store (outdoor)— across different scenarios (totalling 42), where the model displayed varying degrees of confidence in its output. We evaluated the responses in terms of sociability, causality, selectiveness, and contrastiveness generated by GPT-3.5 () and GPT-4 () models. 



 §.§ Sociability


In examining the sociability within the responses, our initial focus centres on the overall tone of the responses. We employed the Valence Aware Dictionary and Sentiment Reasoner (VADER) to gauge the sentiment conveyed within the text, encompassing dimensions of positivity, negativity, and neutrality. Sentiment scores for each response are aggregated to delineate overall sentiment trends. Figure <ref> illustrates a noticeable positive trendline by both models, indicating a prevalent inclination towards positivity in the sentiment expressed. A qualitative inspection of the text reveals a prevalence of polite, helpful, and informative social cues across all responses, supporting this observation. Consequently, the neutral sentiment score remains notably low, underscoring the prevalence of emotional expression throughout the corpus. Exploring negative sentiment scores unveils instances where the models acknowledge their limitations, particularly when unable to retrieve answers from the external knowledge repository. 

 
Label Positive Neutral Negative

GPT3.5   202  5  8
GPT4  195  6  14











Considering the three foundational phases of dialogue—opening, clarification, and closing statements—next, we analyzed the indicators such as politeness markers and emotional expressions within each phase. To facilitate this analysis, we created a distinct dictionary for each phase and computed the term frequency of social cues present in each phase.[Term frequency is defined as the number of times a term occurs in the document over the total number of terms in a document.] Opening statements include greetings such as `hey' and `hello.' Clarification statements are characterized by social cues indicative of helpfulness, continued support, and awareness of its limitations, including terms such as `absolutely', `sorry', `further', `curious', `questions', `help', `feel', and `free'. Closing statements commonly conveyed sentiments of well-wishing or safety, with phrases including terms such as `welcome', `enjoy', `safe', `great', `pleasant', and `glad'.




Figure <ref> shows that both models utilize these terms fairly across the respective phases of the dialogue. Our qualitative assessment of the responses further confirms the contextual relevance of these terms, which align with conversational maxims (See Section <ref>). The higher frequency value observed in closing statements can be attributed to their typically brief nature, which encapsulates most of the social cue terms. Additionally, both models occasionally repeat certain phrases in their responses, albeit with the application of a frequency and presence penalty (0.3).[Frequency penalty discourages repetitive use of words or phrases, while presence penalty promotes diversity in the generated text.] In addition to analysing social cues, we examined part-of-speech (POS) pronouns as social interaction markers, encompassing terms such as `I', `you', `we', `my', `our', and `your.' Quantitative analysis and qualitative evaluation of the results indicate that both models adequately and appropriately applied POS pronouns.


 


 §.§ Expression of Causality


To evaluate the expression of causality, we initially extracted responses indicating cause and effect, primarily found within clarification statements. We then identified key terms related to causality, such as `because', `if', `then', `albeit', `due', `contribute', `influence', `affect', `impact', and `effect', and tallied their occurrences within each answer to measure the presence of causal reasoning. In Figure <ref>, we compare the frequency of these terms used by two models in the response set. Notably, `if' emerges as the most frequent term used by both models, often utilized for expressing counterfactual reasoning (e.g., `if it was not', `if there is/were'), while `influence' typically indicates the cause of a decision. Overall, GPT-4 demonstrated a higher frequency of causality terms and showcased greater diversity in expression compared to GPT-3.5.




Through the qualitative review, we examined existing patterns of dependency parsing signifying causal relationships. Two primary grammatical structures emerged, suggesting cause-effect relationships within the text. The first structure pertains to how the presence or absence of certain features influences the likelihood of specific scenarios. The second structure involves a cause-effect relationship between feature importance and probability, indicating that features with higher importance scores exert a greater impact on probability. It is important to acknowledge that while these grammatical structures and terms are widely used, their accuracy is not always consistent across models. In the subsequent section, we will further evaluate the selection and accuracy of inferred causal relationships.



 §.§ Selectiveness


In this section, we evaluate whether the models effectively detect unusual values within the provided knowledge repository, infer potential causes from those observations, and select the most relevant ones to construct an explanation. Our evaluation encompasses analysing responses across the number of causes given, graded selection, and sort order to assess whether expected causes are addressed correctly. First, we specified the maximum number of uniquely sufficient causes for each case, which are feature importance values >5. The number of causes varies across scenarios; some exhibit only a single highly important feature >5, while others might involve multiple high-importance features with varying degrees of significance. Graded selection is crucial for discerning important values such that the language model must prioritize the most relevant causes when encountered with multiple features of comparable importance levels. Additionally, graded selection entails disregarding features with lower importance scores (<5). Finally, the sort order concerns the sequence of selected causes, which should follow a hierarchy from the most significant and relevant features to those of lesser importance. Thereafter, responses are categorized as fulfilled, partial, subpar, or unfulfilled according to their arrangement with our specifications. 



The findings presented in Table <ref> indicate that GPT-4 outperformed GPT-3.5 in all categories. Notably, GPT-4 had a much higher success rate (76%) than GPT-3.5 (24%) in identifying the most relevant cause (fulfilled). Additionally, GPT-4 had a lower rate of subpar and unfulfilled responses. GPT-3.5 referenced features not present in the knowledge source as potential causes in two instances. These results suggest that GPT-4 is more accurate and selective than GPT-3.5 in identifying the most relevant cause(s) from a set of possible causes. 

Furthermore, an additional criterion for assessing explanation selection is simplicity, which involves utilizing concise language, avoiding technical terminology, and referencing fewer causes. Table <ref> evaluates responses by comparing three metrics: the frequency of technical jargon, response length, and the number of causes cited. Lower values across all three metrics signify simpler responses.



To measure the term frequency of technical jargon, we compiled a list of technical terms in the responses and calculated their frequency within the generated explanations. The list contains terms such as `prediction', `feature importance', `score', `contrastive cases', and `confidence', which may not be easily comprehensible to all users. Table 3 indicates that GPT-3.5 exhibits a slightly higher term frequency (0.0162) compared to GPT-4 (0.0157), suggesting that GPT-3.5 utilizes technical jargon marginally more frequently. The length metric refers to the median lengths of the responses generated by each model, which are quite similar (45 tokens for GPT-3.5 and 44 tokens for GPT-4). In terms of the number of causes cited, GPT-4 cites fewer causes (median of 2) than GPT-3.5 (median of 3), suggesting that GPT-4’s responses tend to be simpler in terms of causal reasoning. Overall, the results suggest that GPT-4 generates simpler responses than GPT-3.5 across all three assessed metrics. 

Regarding the out-of-scope questions posed, such as the number of cars or pedestrians, the intent was to assess whether the model would fabricate an answer. Both models have expressed their inability to provide such information. This demonstrated an awareness of their limitations rather than attempting to generate a response.  




 §.§ Expression of Contrastiveness


Ideally, initial explanations should include a contrast with an alternative case to justify the output. Then the user would investigate it further if desired. In our experiment, we observed that in most cases, models, particularly with GPT3.5, failed to mention alternative scenarios unless explicitly prompted (e.g., inquiries such as “What could it be otherwise if it wasn’t X?” or “How does X differ from other options?”). Nevertheless, to measure the presence of contrastiveness across models, we analysed the occurrence of contrastive terms and comparative phrases within the generated responses. We compiled a dictionary consisting of terms such as `distinguish', `different', `contrast', `compared to', `in contrast', `while', `both', `on the other hand', `whereas', and `conversely', which introduce contrasting explanations. Figure <ref> shows that `distinguish' is the most frequently used term by GPT-3.5, whereas `different' predominates in GPT-4's responses. Other frequently encountered terms include `compared to', `contrast', and `differentiate.' Overall, the results suggest that GPT-4 employs a wider array of contrastive terms than GPT-3.5 in its responses. Our qualitative observations further confirm that while GPT-3.5 tends to adhere to one type of contrastive explanation, GPT-4 explores diverse ways to express such contrasts.




Although both models frequently employ these terms, indicating the generation of alternative viewpoints in their responses, the effectiveness of contrastive explanations varies considerably. Our qualitative review found that well-developed contrasting explanations included alternative scenarios with counterfactuals to strengthen the case. Conversely, weakly constructed contrastive explanations failed to explore `what if' scenarios to elucidate the disparity between the two situations under comparison.





§ CONCLUSION

This study contributes to advancing post-hoc explainability by proposing a traceable question-answering approach. By integrating LLMs with the decomposition-based explainability technique, we generate natural language explanations in an interactive format. Our approach promotes transparency and interpretability in scene-understanding tasks by fusing LLMs with semantic feature importance. The proposed approach has the potential for application across various domains, offering comprehensible explanations for decision-making processes, including critical areas such as medical applications. While this is a promising development, there are still areas for further exploration and improvement. One area that requires attention is the overuse of social cues by LLMs. While polite and helpful phrases can enhance user experience, their misuse can become vexing over time. It is important to evaluate such interactions through human subject studies to understand user perceptions of the explanations provided by the LLM. Future research could also explore dialogue management strategies to address issues related to interaction and personalization. On another note, the dynamic nature of the external knowledge source offers the potential for further expansion with additional observations. This not only ensures that the system remains relevant and up to date but also allows the system to answer different types of questions. Additionally, incorporating multimodal elements, such as combining text with visual explanations, can enhance the effectiveness of the responses (e.g., highlighting the specific elements contributing to the prediction). Another consideration is the estimation of feature importance values, which are subject to perturbations and computational processes. It is important to assess and reconcile differences in feature importance across algorithms to ensure stability and consistency. 
 

 







This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation. 


















 
  




 Paper Title: Multimodal Fusion and Coherence Modeling for Video Topic Segmentation.


Few-shot Defect Image Generation based on Consistency Modeling
    Qingfeng Shi1,2 0009-0000-8480-9314Jing Wei1,2 0000-0001-7697-8070Fei Shen1,2,3()0000-0001-9263-4489 Zhengtao Zhang1,2,30000-0003-1659-7879
    August 5, 2024
===============================================================================================================================================

§ ABSTRACT 
The video topic segmentation (VTS) task segments videos into intelligible, non-overlapping topics, facilitating efficient comprehension of video content and quick access to specific content. VTS is also critical to various downstream video understanding tasks. Traditional VTS methods using shallow features or unsupervised approaches struggle to accurately discern the nuances of topical transitions. Recently, supervised approaches have achieved superior performance on video action or scene segmentation over unsupervised approaches. In this work, we improve supervised VTS by thoroughly exploring multimodal fusion and multimodal coherence modeling. Specifically, (1) we enhance multimodal fusion by exploring different architectures using cross-attention and mixture of experts. (2) To generally strengthen multimodality alignment and fusion, we pre-train and fine-tune the model with multimodal contrastive learning. (3) We propose a new pre-training task tailored for the VTS task, and a novel fine-tuning task for enhancing multimodal coherence modeling for VTS. We evaluate the proposed approaches on educational videos, in the form of lectures, due to the vital role of topic segmentation of educational videos in boosting learning experiences. Additionally, we introduce a large-scale Chinese lecture video dataset to augment the existing English corpus, promoting further research in VTS. Experiments on both English and Chinese lecture datasets demonstrate that our model achieves superior VTS performance compared to competitive unsupervised and supervised baselines[The code and model checkpoints will be released upon acceptance].




§ INTRODUCTION

The proliferation of digital video content over the last few decades has underscored the importance of efficient content navigation and comprehension. As the unstructured nature of videos poses significant challenges for users seeking to quickly grasp or reference specific topics, Video Topic Segmentation (VTS) has emerged as a vital tool in addressing these demands. By delineating videos into coherent non-overlapping topics, VTS not only facilitates intuitive understanding of video content but also enables swiftly pinpointing and accessing topics of interest. This is particularly pertinent for the furtherance of various video understanding tasks, where VTS serves as a foundational component.

Traditional VTS techniques predominantly hinge on shallow features <cit.> and unsupervised methods <cit.> due to the scarcity of labeled data. These methods often fall short in capturing the semantic cues that signal topical shifts in video streams, hence suffer from limited precision. Recent advancements in supervised learning paradigms, benefiting from the availability of large-scale labeled data, have achieved notable performance improvements in various video segmentation tasks, such as video action segmentation <cit.>, scene segmentation <cit.>, and topic segmentation <cit.>, surpassing unsupervised methods. Performance can be further enhanced by pre-training on vast volumes of unlabeled data <cit.> or initializing models from pre-trained models <cit.> and then fine-tuning the model. Hence, in this work, we focus on further improving supervised methods for VTS. 

Compared to text topic segmentation <cit.>, videos contain rich and diverse multimodal contextual information. Fully utilizing multimodal information, such as visual cues and textual data (e.g., screen text and subtitles), could facilitate more detailed content understanding and in turn more accurate semantic segmentation than relying on text only. Also, coherence is essential for understanding logical structures and semantics. Enhancing coherence modeling has achieved significant improvements in long document topic segmentation <cit.>. Therefore, we improve supervised VTS methods by thoroughly exploring multimodal fusion and multimodal coherence modeling. We enhance multimodal fusion from the perspectives of model architecture and pre-training and fine-tuning tasks. Specifically, we compare various multimodal fusion architectures built upon Cross-Attention and Mixture-of-Experts (MoE). We investigate the effect of multi-modal contrastive learning for general pre-training and fine-tuning for strengthening cross-modal alignment. For enhancing multimodal coherence modeling, we propose a new pre-training task tailored for the VTS task, and a novel fine-tuning task by elevating intra-topic multimodal feature similarity and inter-topic multimodal feature differences. The proposed approaches are extensively evaluated on educational videos, in the form of lectures, due to the pivotal contributions of topic segmentation of educational videos in bolstering the learning experiences.

Our contributions can be summarized as follows.

    
  * We propose a supervised multimodal sequence labeling model for VTS, denoted MMVTS model. We compare various multimodal fusion architectures.
    In addition to applying multimodal contrastive learning for strengthening cross-modal alignment, we propose a new self-supervised pre-training task tailored to the VTS task
    and a novel fine-tuning task for enhancing multimodal coherence modeling.
    
  * We introduce a large-scale Chinese Lecture Video Topic Segmentation dataset (CLVTS) to promote the research of VTS.
    
  * Experiments show that our model sets new state-of-the-art (SOTA) performance on both English and Chinese lecture video datasets, outperforming competitive unsupervised and supervised baselines. Comprehensive ablation study further confirms the effectiveness of our approaches.






§ RELATED WORK

Text Topic Segmentation
The objective of text topic segmentation is to automatically partition text into topically consistent, non-overlapping segments <cit.>. By automatically mining clues of topic shifts from large amounts of labeled data <cit.>, contemporary supervised models <cit.> have demonstrated superior performance compared to unsupervised approaches <cit.>. Notably, supervised models that excel at modeling long sequences <cit.> are capable of capturing longer contextual nuances and thereby achieve better topic segmentation performance, compared to models that model local sentence pairs or block pairs <cit.>. In addition, recent works <cit.> have demonstrated that strengthening coherence modeling can improve the topic segmentation performance. Therefore, in this work, we explore enhancing coherence modeling for video topic segmentation under the multimodal configurations. 



Video Topic Segmentation
Distinct from text topic segmentation, video topic segmentation is inherently characterized by its multimodal data context; hence, comprehensive utilization of the multimodal information could improve semantic understanding capabilities of a model. Specifically,  <cit.> introduce an unsupervised method that employs the TWFINCH algorithm <cit.> to cluster video clips into distinct topics based on learned visual and text features, which are derived from self-supervised tasks of matching the narration with the temporally aligned visual content.  <cit.> simply concatenate the sampled encoded visual features and text embedding as the input to a pre-trained language model, which may cause inferior fine-tuning due to the discrepancy between the pre-training stage of the language model and the fine-tuning stage combining textual and visual inputs. Other works related to VTS are summarized and compared in Appendix <ref>.




§ THE CLVTS DATASET


Collection Procedure
The CLVTS dataset is primarily sourced from Public Video Platforms[<https://study.163.com/>][<https://www.bilibili.com/v/knowledge>]. Videos are first transcribed by a competitive automatic speech recognition (ASR) system[<https://tingwu.aliyun.com/home>]. We then ask annotators to combine visual and textual (ASR 1-best) information to mark the timestamp (in seconds) of the end of each topic. We ensure high accuracy and reliability of annotations from three aspects, including annotator training, hierarchical topic labeling, and a multi-annotator strategy. Details of these strategies and the rigorous quality assessment and quality control we conduct can be found in Appendix <ref>.

Dataset Statistics
To measure the extent of consensus for VTS annotations, following <cit.>, we compute the F_1@k score (described in Appendix <ref>) based on the absolute distance between two topic boundary sequences, varying the threshold k from 0 to 8 seconds with a step size of 2 seconds, where 8 seconds are approximately the average duration of a clip. By averaging the F_1 scores across all three pairs of annotated topic boundaries from three annotators on the same video, we obtain the consistency score. The more similar the annotations from all annotators on the same video are, the higher the consistency score is. Figure <ref> shows that the consistency scores of the majority of videos exceed 0.5, indicating a decent degree of consensus for VTS annotations <cit.>.

Table <ref> compares our CLVTS dataset against existing VTS datasets. Among annotated videos in CLVTS, 47% are presentations showing slides, 34% are blackboard demonstrations, and 19% are miscellaneous types. We also collect 1027 hours of unlabeled videos from the same sources for pre-training. Besides the linguistic distinctness, CLVTS is characterized by its natural and uninterrupted long videos, a stark contrast to the English lecture dataset, AVLecture <cit.>, since nearly two-thirds of AVLecture are reassembled pre-segmented short videos. Figure <ref> and <ref> show a diverse distribution of video and topic durations and a broad spectrum of subjects in CLVTS.



§ METHODOLOGY

Figure <ref> depicts the architecture of our VTS model. Section <ref> presents the problem definition of multimodal VTS and the overall architecture. We enhance multimodal fusion from the perspectives of model architecture, pre-training, and fine-tuning tasks. Specifically, we compare different architectures built upon Merge- and Cross-Attention, and Mixture-of-Experts for multimodal fusion (Section <ref>). We explore multimodal contrastive learning for cross-modality alignment and propose a new pre-training task tailored for VTS (Section <ref>). For fine-tuning, we also propose a novel task for multimodal coherence modeling (Section <ref>).






 §.§ Multi-Modal Video Topic Segmentation




  
Overall Architecture Following prior works <cit.>, we define video topic segmentation as a clip-level sequence labeling task and propose our MultiModal Video Topic Segmentation (MMVTS) model. As illustrated in Figure <ref>, we apply unimodal pre-trained encoders for each modality and then fuse multimodal information at the intermediate representation level (i.e., `middle fusion' <cit.>) through Multimodal Fusion Layers. Given a video, we transcribe it with a competitive automatic speech recognition (ASR) system[<https://tingwu.aliyun.com/home>] and use ASR 1-best as the text modality. We then divide the video into n clips (c^v_i, c^t_i)_i=1^n, with clips segmented at the sentence boundaries predicted on ASR 1-best. c^v_i = {f^i_1,..., f^i_k} denotes evenly sampled k frames from the i-th clip and is fed to a visual encoder E_v to extract visual features. c^t_i = {w^i_1,...,w^i_s_i+1} denotes the sequence of words from ASR 1-best within the i-th clip, where w^i_1 is the inserted special token [BOS] and s_i denotes the number of words of i-th clip. c^t_i is fed to a text encoder E_t for extracting textual features. After extracting unimodal features, we first apply trainable projection matrices W_v and W_t to convert unimodal features into the same dimension, as visual feature sequence v={v_1,...,v_n} and textual feature sequence t={t_1,...,t_n} (Eq. <ref>). Then we fuse the multimodal information with M Multimodal Fusion Layers MFL_M and obtain the updated visual features h^v={h^v_1,...,h^v_n} and textual features h^t={h^t_1,...,h^t_n} (Eq. <ref>), which are then concatenated into the multimodal features m={m_1,...,m_n} (Eq. <ref>). Finally, the multimodal features m are fed into the predictor composed of a linear layer W_p to obtain the probability of binary classification p={p_1,...,p_n} (Eq. <ref>), where p_i indicates whether the i-th clip is at the boundary of the topic that it belongs to. We use the standard binary cross-entropy loss (Eq. <ref>) to train the model, where y_i∈{0, 1} is the label. The last clip is excluded from loss computation.

Compared to `late fusion' where no cross-modal interaction happens until after the classifiers, middle fusion and `early fusion' are found to generally outperform late fusion <cit.>, probably because early and middle fusion aligns better with human perception where multimodal fusion happens early in sensory processing. On the other hand, compared to early fusion, middle fusion yields superior or comparable performance <cit.> and is much less computationally expensive since we could freeze partially strong pre-trained unimodal encoders and only train a small number of parameters in the Multimodal Fusion Layers.


    v = W_v·E_v({c^v_1,...,c^v_n})
    t = W_t·E_t({c^t_1,...,c^t_n})


    h^v; h^t = MFL_M(v;t)


    m_i = h^v_i;h^v_i


    p = W_p·m


    l_vts = -∑_i=1^n-1[y_iln p_i+(1-y_i)ln(1-p_i)]




  
Multimodal Fusion Layer (MFL) We compare four different cross-modal interaction mechanisms for multimodal fusion layers. We investigate the Merge-Attention and Co-Attention multimodal fusion layers proposed in  <cit.>. With Merge-Attention (Figure <ref>), features from unimodal encoders are sequentially concatenated and then input into a standard transformer encoder layer <cit.>, which shares attention parameters across modalities. A feed forward layer is added on top to produce the final output representation. In contrast, with Co-Attention (Figure <ref>),  features from each unimodal encoder first go through self-attention with modality-specific attention parameters, then we perform symmetrical cross-attention to integrate information from all other modalities to enhance the representation of the considered modality, followed by a feed forward layer.

Inspired by <cit.>, which interleaves MoE encoder layers and standard dense encoder layers for image-text multimodal models, we also investigate two new architectures by replacing the traditional single feed-forward layers in Figure <ref> and <ref> with a Mixture-of-Experts (MoE) <cit.> module respectively, depicted by Figure <ref> and <ref>. The motivation is that adding MoE on top of the fused representations may facilitate deeper cross-modal integration of information and improve model capacity without a proportional increase in computational complexity. Specifically, experts are MLPs activated depending on the input. Firstly, we concatenate fused features output from self-attention or cross-attention. Then, we implement the Noisy Top-k Gating mechanism <cit.> to select K experts from a total of E candidates (Eq. <ref> - <ref>), where SN() denotes stand normal distribution, W_n denotes tunable Gaussian noise to help load balancing, W_g is a trainable weight matrix, K and E are hyper-parameters. Finally, the outputs of the K activated experts are linearly combined according to the learned gating weights  (Eq. <ref>). For the MoE training objective, we sum importance loss and load loss <cit.> to balance expert utilization as Eq. <ref>.


    G(x) = Softmax(KeepTopK(H_x, k))


    H(x)_i = (W_g· x)_i + SN() · Softplus((W_n· x)_i)


    KeepTopK(x, k)_i =
        
            x_i if x_i is in top-k.
    
            -∞ otherwise.


    MoE(x) = ∑^K_e=1 G(x)_e· MLP_e(x)


    l_balance = l_importance + l_load







 §.§ Pre-training with Unlabeled Data


Prior works have demonstrated that standard self-supervised denoising pre-training (even using only the downstream task data) <cit.> or pre-training adapted to the downstream task <cit.> often perform substantially better than randomly initializing the parameters. Therefore, to better initialize the parameters of the Multimodal Fusion Layers, we explore pre-training with unlabeled video data before fine-tuning on the labeled training set. Firstly, we introduce a general cross-modality alignment pre-training task to learn the multi-modal representation. We use contrastive learning loss to adjust the features learned by the model fusion layer, by maximizing the cosine similarity of the visual features and textual features of the same clip, while reducing the similarity of the modality features between different clips, as show in Eq. <ref>, where ϵ is used to prevent division by 0 and τ is a temperature hyper-parameter to scale the cosine similarity.


    l_cma = -1/n∑_i=1^n e^sim(h^v_i, h^t_i)/∑_i=1^n∑_j=1^n e^sim(h^v_i, h^t_j)+ϵ


    sim(x_1, x_2) = x_1^T· x_2/x_1·x_2 / τ

Secondly, we introduce a novel pre-training task tailored for the downstream VTS task, focusing on utilizing unlabeled data for learning pseudo topic boundaries and also enhancing modality alignment. Initially, a Kernel Density Estimation (KDE) <cit.> model is employed to approximate the topic duration distribution of the training set; then, videos are split into segments matching durations sampled from the KDE-derived distribution. Subsequently, for every resulting segment, we undertake one of three actions with an equal probability: inserting segments randomly selected from other videos either before or after the current segment, replacing the current segment with a segment randomly selected from other videos, or no change. The resulting segments within the modified video are then treated as distinct topics, thereby enabling the model to learn pseudo topic boundaries during pre-training. This task-adaptive pre-training task has the same l_vts objective as shown in Eq. <ref>. The overall pre-training objective is shown in Eq. <ref>, where α and β are hyper-parameters to adjust the loss weights.

    l_pretrain = l_vts + α l_cma + β l_balance




 §.§ Fine-tuning with Multimodal Coherence Modeling


For fine-tuning, we introduce two auxiliary tasks to enhance multimodal coherence modeling. The cross modal alignment task is the same task in Eq. <ref> used in pre-training. This continuity ensures that the modalities retain their coherence through both pre-training and fine-tuning, fostering a consistent interplay between different modalities. 
In addition, we adapt the Contrastive Semantic Similarity Learning (CSSL) task proposed by <cit.>, which leverages the inherent characteristics of topic-related coherence, to the multimodal context. We adopt the same strategy for selecting positive and negative sample pairs <cit.>, but extend the features to the multimodal representations, as shown in Eq. <ref>, where k_1 and k_2 are hyper-parameters that determine the number of positive and negative pairs. For each clip's multimodal representation m_i,  m_i,j^+ denotes the multimodal representation of the j-th similar clip in the same topic as clip i, while m_i,j^- denotes the multimodal representation of the j-th dissimilar clip in a different topic from clip i. We hypothesize that this extension could improve multimodal representation learning by identifying relative consistency relations within and across topics.

    l_mcssl=-1/n∑_i=1^nlog ∑_j=1^k_1 e^sim(m_i, m_i,j^+)/∑_j=1^k_1 e^sim(m_i, m_i,j^+)+∑_j=1^k_2 e^sim(m_i, m_i,j^-)

The overall fine-tuning objective combines Eq. <ref>,  <ref>,  <ref>, and  <ref>,  as shown in Eq. <ref>, where σ, θ, and γ are hyper-parameters to adjust loss contribution. When the multimodl fusion layer does not contains MoE structure, β in Eq. <ref> and σ are set to zero.

    l_finetune=l_vts + σ l_balance + θ l_mcssl + γ l_cma





§ EXPERIMENTS



 §.§ Experimental Setup

Datasets We partition the labeled data within AVLecture into 70% for training, 10% for validation, and 20% for test sets. We also split the CLVTS dataset into train, validation, and test sets with identical ratios.  The unlabeled data of AVLectures and CLVTS are used for pre-training on each dataset, respectively. 

Baselines
We carefully select the following representative baselines:

- UnsupAVLS <cit.> is an unsupervised approach that clusters video clips into discrete topics based on visual and text embeddings learned from matching the narration with temporally aligned visual content. The actual number of topics is used as the desired number of clusters.

- Visual-only BaSSL <cit.> is initially proposed for video scene segmentation. We use their released checkpoints to initialize our model and fine-tune on the VTS task to evaluate the performance of a visual-only model. 

- Text-only LongFormer is evaluated on long document topic segmentation by <cit.>. We fine-tune LongFormer and LongFormer_cssl in <cit.> to evaluate the performance of a text-only model on the VTS task.

- Llama-3-8B[<https://huggingface.co/meta-llama/Meta-Llama-3-8B>] Our pre-training (Section <ref>) is conducted on the relatively limited unlabeled videos of AVLectures and CLVTS datasets. To investigate the effect of fine-tuning a powerful pre-trained text large language model (LLM) on VTS, we fine-tune Llama-3-8B with two prompting methods (see Appendix <ref> for details).

- SWST_seq is our adapted version of the multimodal video scene and topic segmentation model <cit.> to the sequence labeling VTS task, for comparing performance between their early fusion and our middle fusion strategy on VTS. SWST_seq uses a pre-trained LongFormer <cit.> as backbone.

Evaluation Metrics We adopt four commonly used metrics for VTS, including positive F_1 <cit.>(denoted as F_1 to simplify notations), BS@k <cit.>, mIoU <cit.>, and F_1@k. Definitions of the four metrics are in Appendix <ref>. We follow  <cit.> and set k to 30 seconds. Additionally, we use the average score (Avg) of these four metrics to measure the overall performance of a model.

Implementation Details
We use the same maximum sequence length 2048 as  <cit.> for a fair comparison with the text-only models. All results are the mean values over three runs with different random seeds. More implementation details are in Appendix <ref>.



 §.§ Results and Analysis

Table <ref> reports the performance of baselines (the first group) and our MMVTS model variants (the second and third group) on the AVLecture and CLVTS test sets.

Unimodal performance. The text-only Longformer (Row3) substantially outperforms the visual-only BaSSL (Row2) on BS@30 by a large gain (+25.31), and also surpasses the unsupervised approach UnsupAVLS by a notable gain (+13.25). Such results are expected since the textual modality inherently conveys more precise and richer information than the visual modality for VTS. 

Mutimodal performance. As can be seen from Table <ref>, Avg of SWST_seq closely aligns with that of the text-only LongFormer, suggesting that more data may be necessary to mitigate the discrepancy between fine-tuning with early fusion and pre-training the language model, in order to fully exploit the potential of the early fusion strategy. Our MMVTS Baseline_1,2,3 simply concatenate unimodal features to predict topic boundaries. Without pre-training on unlabeled data (PT, Eq. <ref>) and the two auxiliary fine-tuning tasks to enhance multimodal coherence modeling (FT-Coh, Eq. <ref>), on F_1, MMVTS Baseline_1 outperforms the text-only Longformer by 2.28 and 2.90 on AVLecture and CLVTS respectively, while on Avg score,  MMVTS Baseline_1 outperforms Longformer by 1.1 on AVLecture yet slightly underperforms Longformer by 0.65 on CLVTS. These results suggest that simply concatenating unimodal features to predict topic boundaries does not bring consistent gains over unimodality. The third group in Table <ref> compares the four Multimodal Fusion Layer architectures with pre-training (PT) and fine-tuning (FT-Coh). Overall, compared to all the competitive unsupervised and supervised baselines, after pre-training and fine-tuning, our MMVTS model using Co-Attention with MoE as Multimodal Fusion Layers achieves the best Avg, BS@30, and F_1@30 results and sets new SOTA on AVLecture, and achieves the best F_1 and a Avg score comparable to the best result on CLVTS. The gains on Avg from our Co-Attention MoE with PT and FT-Coh over LongFormer_cssl and MMVTS Baseline_1 are statistically significant (p-values are less than 0.05). It is also notable that the performance of models on CLVTS is generally much lower than that on AVLecture, with the best Avg score between AVLecture and CLVTS differing by 17.50 (68.84 versus 51.34), indicating a greater challenge posed by our CLVTS dataset than the AVLecture dataset.

Table <ref> also shows that on CLVTS, fine-tuning the powerful pre-trained Llama-3-8B with our Discrete prompt achieves the best Avg score 51.34, although very close to the performance of Merge-Attn with MoE and Co-Attn with MoE after PT and FT-Coh. However, on AVLecture, the performance of fine-tuning Llama-3-8B is much lower than Multimodal Fusion Layer with PT and FT-Coh; particularly, F_1 on both AVLecture and CLVTS are much lower because the model's predicted boundaries often have a clip offset, but its F_1@30 and mIoU results are still best on CLVTS. 


We conduct ablation studies and analysis to validate effectiveness of the proposed multimodal fusion layers, pre-training and fine-tuning tasks.


(1) Effect of Multimodal Fusion Layers. Comparing the third group and Baseline_3 in Table <ref> shows that with the same pre-training and fine-tuning, the best performing architecture using Multimodal Fusion Layers always substantially outperforms Baseline_3. Specifically, with PT and FT-Coh, on AVLecture, both Co-Attention and Co-Attention with MoE notably outperform MMVTS Baseline_3 by 1.32 on Avg; on CLVTS, all four Multimodal Fusion Layer architectures achieve remarkable gains on Avg over MMVTS Baseline_3, from 2.65 to 4.18. These results demonstrate that deep cross-modal interaction has notable advantage for multimodal fusion over simple unimodal feature concatenation for VTS. Moreover, adding MoE on top consistently improves Co-Attention on both AVLecture and CLVTS, by 0.47 and 1.07 on Avg; whereas, the effect of MoE on top of Merge-Attention is inconsistent, with a slight degradation on AVLecture and 1.53 gain on Avg on CLVTS. In addition, we conduct more analysis of the effect of Co-Attn with MoE with different numbers of multimodal fusion layers in Appendix <ref>.

(2) Effect of Pre-training tasks. Table <ref> shows that for simple concatenation of unimodal features, pre-training before fine-tuning (as Baseline_3) outperforms Baseline_2 (w/o PT). We conduct ablation studies of the proposed pre-training and fine-tuning tasks on the AVLecture dataset. We apply the same fine-tuning with multimodal coherence modeling (FT-Coh, Eq. <ref>) and compare (a) random initialization of parameters of Multimodal Fusion Layers (w/o pre-training) (b) pre-training the model on unlabeled data. Table <ref> shows that w/o pre-training,  Co-Attn slightly improves Avg over Merge-Attn by 0.4 pts, and MoE further improves Avg by 0.87 pts. Pre-training improves the performance on all four Multimodal Fusion Layer architectures, with the average Avg score increased by 1.36 pts (66.29 → 67.65). Pre-training also improves stability of a model as the standard deviations of all w/ PT experiments are less than 1. Table <ref> in Appendix further compares the fine-tuning performance after applying different pre-training tasks. Compared to using both pre-training tasks, removing loss l_cma or l_vts leads to Avg reduction of 0.4 and 1.64 respectively,  indicating that the pretraining task more closely aligned with the downstream task, i.e., l_vts, yields greater benefits.


(3) Effect of Fine-tuning tasks. Table <ref> compares the efficacy of different fine-tuning tasks after pre-training. Compared to the standard l_vts, adding the two auxiliary losses l_cma and l_mcssl notably improves  Avg by 1.12 pts, while decreases F_1 by 0.73. 

These results suggest that while multimodal coherence modeling may slightly compromise the precision of exact matches, it enhances the overall contextual comprehension of a model for VTS. Adding l_cma or l_mcssl individually improves Avg by 0.53 and 0.57 respectively, with improvements mainly on BS@30 and mIoU, suggesting that feature alignment at different granularities may improve fuzzy matching of a model. 



§ CONCLUSION

In conclusion, we propose a supervised MultiModal Video Topic Segmentation (MMVTS) model to advances VTS by meticulously investigating multimodal fusion and coherence modeling, coupled with an innovative pre-training task tailored for VTS. Furthermore, we introduce a large-scale Chinese Lecture Video Topic Segmentation (CLVTS) dataset to promote research in VTS.




§ LIMITATIONS

It is essential to acknowledge that our methodology solely capitalizes on the multimodal information derived from visual and textual data, with the parameters of the visual encoder fixed due to consideration about computational complexity. This specific limitation potentially results in a less-than-optimal exploitation of the vast visual information contained within the dataset. Concurrently, features from additional modalities, such as audio, are not incorporated. Additionally, we will conduct more investigations and study approaches to make Co-Attn with MoE more stable in future work. Finally, the integration of general multimodal pre-trained models and large language models could be investigated, offering a more holistic and effective exploration of multimodal information for VTS.












§ RELATED WORK TO VTS


 <cit.> utilized visual, audio and textual information, and focus on hierarchical modeling techniques to simultaneously learn the boundaries of scenes, stories and topics, which suggests that the segmentation can be benefited by introducing the hierarchical structures of videos. Nevertheless, their approach did not delve into optimizing the integration of multimodal information, which is one of the motivations of our work. The integration of audio features represents a direction for our future work. <cit.> encode the image and text sequences with respective encoders and use an asymmetric cross-modal cross-attention mechanism to produce text-aware visual representations, followed by a BiLSTM, to fuse multimodal features. Our approaches are drastically different from  <cit.> as we investigate symmetric Cross-Modal Cross-Attention and also Mixture-of-Experts mechanism to fuse multimodal information. <cit.> also investigate cross-modal and sentence level intra-modal contrastive learning to empower unsupervised domain adaptation, whereas we explore extending the topic level Contrastive Semantic Similarity Learning (CSSL) task proposed by <cit.> to the multimodal setting.



§ DETAILS OF ANNOTATIONS FOR THE CLVTS DATASET



For the CLVTS dataset, we ensure high accuracy and reliability of annotations from three aspects. 
Firstly, before the actual annotation process, the annotators take two rounds of training. Each annotator needs to annotate 5 videos in each round; at the end of each round, we assess the annotation quality, provide feedback, and ensure that the annotators address the issues and understand the annotation guideline clearly at the end of training. Secondly, during annotation, to help the annotators thoroughly understand the course content, we ask the annotators to annotate topics hierarchically, that is, they label both coarse-grained topic (large topic) and fine-grained topic (small topic) boundaries while the large topic boundaries are a subset of the small topic boundaries. We take the small topic boundaries as the final topic boundary labels. 
Thirdly, we employ a multi-annotator strategy. All data is annotated in batches, with two annotators annotating each sample independently. The third annotator reviews the annotations by the first two annotators, rectifies errors, and provides the final annotations. After the annotations of each batch, we randomly select 5% of a batch for quality assessment. If the unacceptable rate (the proportion of the wrongly annotated topic boundaries) is less than 10%, the data are deemed satisfactory and accepted; otherwise, after communicating quality assessment results and possible reasons for errors, the annotators are requested to re-annotate the batch based on the feedback. This quality control process is repeated until the unacceptable rate is lower than 10% for all batches; in this work, we could finish quality control within 3 iterations.



§ TRAINING DETAILS


Our experiments are implemented with transformers package[< https://github.com/huggingface/transformers>]. For video with the number of clips greater than the max sequence length, we use sliding window and take the last clip of the prior sample as the start clip of the next sample. All supervised models use a threshold strategy, where clips with scores above a threshold 0.5 are predicted as topic boundaries. Following <cit.>, for each video clip, we extract three visual feature types: OCR, 2D and 3D. Specifically, the OCR feature is derived by encoding the textual output obtained from the OCR API[<https://help.aliyun.com/zh/viapi/developer-reference/api-sy75xq>] of the clip's central image. This encoding task is performed using the BERT-based sentence transformer model[<https://www.sbert.net/docs/sentence_transformer/pretrained_models.html>], where all-mpnet-base-v2 and paraphrase-multilingual-mpnet-base-v2 models are employed for English and Chinese language experiments, respectively. The 3D feature are extracted using the same video feature extraction pipeline as in  <cit.>, while the 2D feature is extracted by visual encoder of CLIP <cit.> which is pre-trained to predict if an image and a text snippet are paired together. Specifically, the images from AVLecture and CLVTS are processed to extract 2D features using CLIP_ViT-B/16[<https://github.com/openai/CLIP>] and CN-CLIP_ViT-B/16[<https://github.com/OFA-Sys/Chinese-CLIP>], respectively.

After extracting the visual features, we concatenate them as shown in Eq. <ref> to get v_i, which then will be fed into following projection layer. During pre-training and fine-tuning, the parameters of visual encoders are kept fixed. The learning rate is 5e-5 and dropout probability is 0.1. AdamW <cit.> is used for optimization. The batch size is 8 and the epoch for pre-training and fine-tuning is 1 and 5, respectively. The loss weight α and γ for l_cma is 0.5, β and σ for l_balance is 1.0 when MoE is in multimodal fusion layer. θ for l_mcssl is 0.5. k_1 and k_2 of Eq. <ref> is 1 and 3 following <cit.>. We select one multimodal fusion layer to comprehensively compare different types of fusion structure. We also investigate the performance of different number of multimodal fusion layer in Figure <ref>. As for the MoE layer in multimodal fusion layer, we choose 4 candidate experts and activate 2 experts for each input feature, the intermediate size of expert is 3072. Therefore, the total number of trainable parameters is shown in Table <ref>.

    v_i = v^2d_i;v^3d_i;v^ocr_i






§ EVALUATION METRICS


F_1 is a metric used to evaluate the accuracy of text topic segmentation <cit.>, which focus on the performance of exact matching of the positive class and balances precision and recall rate.

BS@k <cit.> is the average number of predicted boundaries matching with the ground truth boundaries within a k second interval, which can be regard as the recall rate based on fuzzy matching.

F_1@k denotes the F_1 score calculated based on matching predicted and ground truth boundaries within k seconds. Considering subjectivity and uncertainty in VTS annotations, we introduce F_1@k as a supplement to BS@k to enabling a more comprehensively assessment of model performance in dealing with ambiguously defined boundaries.

mIoU is commonly used in video action segmentation <cit.> and video scene segmentation <cit.>. While the F_1, BS@k and F_1@k focus on the accuracy of positive predictions (either exact match or fuzzy match), the mIoU measures the overlapping area between predicted and ground truth segments, providing a generalized assessment of how well the model's predicted segments match the actual segments in the segmentation task.

Our implementation of BS@30 draws upon the code publiced by <cit.>[<https://github.com/Darshansingh11/AVLectures/>], while the approach to implement mIoU is guided by <cit.>[<https://github.com/kakaobrain/bassl>]. We have relied on the scikit-learn package[<https://scikit-learn.org/stable/>] to compute  F_1, following the implementation by <cit.>. The definitions provided in the aforementioned sources also inform our implementation of F_1@k.






§ ARTIFACT USE CONSISTENT WITH INTENDED USE


All of our use of existing artifacts is consistent with their intended use, provided that it was specified. For the CLVTS data set we created, its license will be for research purpose only.




§ FINE-TUNE LLAMA-3-8B FOR VIDEO TOPIC SEGMENTATION


Considering the computational complexity and data volume, we employ LoRA <cit.> for fine-tuning Llama-3-8B[<https://huggingface.co/meta-llama/Meta-Llama-3-8B>] with a maximum sequence length of 2048. Our training configuration includes a batch size of 32 and a total of 6 epochs, utilizing a learning rate of 5e-5 and cosine scheduler <cit.>. Table <ref> shows two prompts we used to fine-tuning Llama-3-8B on text data of VTS. We first try the Generative type that <cit.> has experimented with due to the better zero-shot and one-shot performance than the Discriminative type. However, compared with Longformer, the Avg score of Generative prompting Llama-3-8B is lower 8.3 and 6.27 on AVLecture and CLVTS respectively. We suspect that this is due to the issue of sparse labels in binary classification, leading us to refine the Discriminative into the Discrete prompt as outlined in the Table <ref>. Although the F_1 on AVLecture and CLVTS is much lower (the prediction boundary of the model usually has a clip offset), the Avg score of Llama-3-8B on AVLecture is merely 2.26 points behind LongFormer. Moreover, Llama-3-8B_Discrete secures the highest Avg score on CLVTS, surpassing LongFormer by 4.53 points. which suggests that the Discrete prompt is more suitable for stimulating Llama-3-8B and future work can further explore how to better use large language models for VTS.




§ PERFORMANCE OF MFLS ACROSS VARIED LAYER NUMBERS


Utilizing Co-Attention with Mixture of Experts (MoE) as the multimodal fusion layer, we investigated the Avg performance of models featuring various numbers of Multimodal Fusion Layers (MFLs) on AVLecture, as depicted in Figure <ref>. The findings indicate that when the model is fine-tuned directly on labeled data, a single MFL yields the best performance, surpassing the no-layer configuration by 2.34 points. However, adding more layers leads to diminished performance and training instability, particularly noticeable with three layers. By incorporating a pre-training phase followed by fine-tuning through a coherence modeling task, performance enhancements of 4.0, 2.87, 2.37, and 13.51 points are observed, respectively. These results clearly demonstrate the substantial benefits of our pre-training and coherence modeling strategies in boosting the model's effectiveness. Notably, they facilitate the convergence of a model with three layers, achieving optimal results that marginally exceed the performance of a single-layer model by 0.1 points.




