Paper Title: A Systematic Review on Long-Tailed Learning.















§ ABSTRACT 
Long-tailed data is a special type of multi-class imbalanced data with a very large amount of minority/tail classes that have a very significant combined influence. Long-tailed learning aims to build high-performance models on datasets with long-tailed distributions, which can identify all the classes with high accuracy, in particular the minority/tail classes. It is a cutting-edge research direction that has attracted a remarkable amount of research effort in the past few years. In this paper, we present a comprehensive survey of latest advances in long-tailed visual learning. We first propose a new taxonomy for long-tailed learning, which consists of eight different dimensions, including data balancing, neural architecture, feature enrichment, logits adjustment, loss function, bells and whistles,  network optimization, and post hoc processing techniques. Based on our proposed taxonomy, we present a systematic review of long-tailed learning methods, discussing their commonalities and alignable differences. We also analyze the differences between imbalance learning and long-tailed learning approaches. Finally, we discuss prospects and future directions in this field.



Long-tailed learning, Long-tailed data, Imbalance learning,  Deep learning, Deep imbalance learning.


A Systematic Review on Long-Tailed  Learning
    Chongsheng Zhang,  Senior Member, IEEE,
        George Almpanidis,
         Gaojuan Fan1,
         Binquan Deng,
         Yanbo Zhang,
         Ji Liu,
         Aouaidjia Kamel,
         Paolo Soda,
        João Gama,  Fellow, IEEE

Chongsheng Zhang, George Almpanidis, Gaojuan Fan, Binquan Deng, Yanbo Zhang and Aouaidjia Kamel are with Henan University, Henan Key Lab of Big Data Analysis and Processing, Kaifeng, China. E-mail: cszhang@ieee.org; almpanidis@ieee.org; {fangaojuan,bqdeng,zhangyanbo,kamel}@henu.edu.cn. 
 Gaojuan Fan is the corresponding author of this work.
Ji Liu is with Baidu Inc., Beijing, China. E-mail: jiliuwork@gmail.com.
Paolo Soda is with University Campus Bio-Medico di Roma, Rome, Italy.  Email: p.soda@unicampus.it.
João Gama is with the Laboratory of Artificial Intelligence and Decision Support, Faculty of Economics, University of Porto, Porto, Portugal. Email: jgama@fep.up.pt. 

    
======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================







§ INTRODUCTION


The term “Long Tail Theory” was popularized by Chris Anderson's book  “The Long Tail: Why the Future of Business Is Selling Less of More” <cit.>, in which the shift in business from advertising and marketing a small number of bestseller products to paying attention to the vast amount of niche products was observed and analyzed, since their combined sales/revenues could be as significant as that of the head (most popular) products. It is closely related to “the Pareto Principle”, also known as the “80/20 Rule”, the Zipf's law, or the “Vital Few Rule”. Generally speaking, they are like two sides of the same coin. While the Pareto Principle focuses on the heads/hits and cuts off the rest, the Long Tail Theory aims to capture the tails and emphasizes the combined importance of the tail  classes/niche items.

Long-tailed data is essentially a special type of multi-class imbalanced data with a sufficiently large number of tail (minority) classes. Moreover, the combined importance of these tail classes is very significant, although each tail class by itself only has  a small number of samples (sales). Long-tailed distributed data is a relatively common phenomenon in real-world scenarios, which often needs to be handled by artificial intelligence (AI) systems and applications, e.g. high-speed train fault diagnosis <cit.>, escalator safety monitoring <cit.>, etc. Indeed, AI should not only satisfy the head/hit applications, but also be able to cover a large number of tail cases to ensure the robustness and generalization capability of the techniques and systems.

Long-tailed learning (hereafter referred to as LTL for short) is a sub-domain of artificial intelligence/machine learning that aims to build effective models for applications/tasks having long-tailed distributed data. Its main goal is to significantly improve the recognition accuracy on the tail (rare/minority) classes or cases while maintaining the same or similar accuracy on the head (frequent/majority) classes or cases. Specifically, 1) in object recognition/classification, researchers have proposed many LTL methods <cit.> that can substantially improve the recognition/prediction accuracy on the tail classes/cases, e.g. rare species, defective industrial products. Studies in <cit.> have shown promising results by disentangling deep feature representation learning and classifier training. There are also a large number of loss reweighting approaches that endow different weights to samples of the head and tail classes to adjust the decision boundary under the long-tailed settings <cit.>. 2) in object detection, a few LTL methods <cit.> have also been designed to automatically locate rare objects or cases from images or videos, e.g. locating cracks in industry equipment or detecting the engineering vehicles near natural parks; 3) in image segmentation, researchers have also developed methods <cit.> that can identify and segment the rare objects/cases in images, such as abnormal actions, pathology areas in medical images, etc. In addition, specific data augmentation techniques for LTL have also been devised to mitigate the data scarcity problem in the tail classes <cit.>.



However, due to the rapid development of this field, keeping pace with recent advances in LTL is becoming increasingly difficult. As such, a comprehensive survey of existing methodologies in this field is urgent and beneficial to the community. This motivates us to conduct an in-depth survey of recent advances in long-tailed visual learning to gain insights into their principles and technical aspects in a systematic way. Based on the inherent learning process, we first propose a novel taxonomy that categorizes existing long-tailed visual learning approaches into eight groups, depicted in Figure <ref>, which are data balancing, neural architecture, feature enrichment, logits adjustment, loss function, bells and whistles,  network optimization, and post hoc processing approaches. Based on this new taxonomy, we present a comprehensive survey of state-of-the-art LTL approaches and discuss their ideas and characteristics. Moreover, we make comparisons between imbalance learning and long-tailed learning, in which we elucidate their connections and differences. Finally, we discuss challenges and future research opportunities in this field.

Though a few recent papers <cit.> also provide literature reviews on long-tailed learning, we distinguish our survey from them by the following differences: 1) Existing reviews often employ the conventional taxonomy that categorizes existing LTL approaches into three types, which are the data resampling and augmentation category, the loss reweighting category, and the transfer learning category. However, such a taxonomy does not facilitate a comprehensive understanding of state-of-the-art LTL methods since it can not sufficiently cover LTL's whole learning process. In this work,  we propose a taxonomy that provides a more unified perspective on LTL based on its internal learning process, where we identify four major steps with eight main categories, as shown in Figure <ref>. Using this new taxonomy, we provide a more up-to-date and insightful review of existing LTL methods, discussing their commonalities and alignable differences. 2) We make detailed comparisons between LTL and imbalance learning. 3) We include the latest advances in LTL and point out future directions in this field.

The main contributions of this work can be summarized as follows:




  * We propose a  unified taxonomy with eight dimensions to characterize and organize existing long-tailed learning methodologies.


  * We present a comprehensive, taxonomy-guided survey of state-of-the-art long-tail visual learning approaches, focusing on recent advances and trends.



  * We compare long-tailed learning with imbalance learning, elucidate their connections and  differences.


  * We summarize and analyze the results of different LTL methods in different downstream tasks, using the corresponding benchmark datasets.


  * Finally, we discuss a number of future research directions and trends that can be interesting to researchers in this field.



The remainder of this paper is organized as follows. In Section <ref>, we present the background and related survey papers in long-tailed visual learning. In Section <ref>, based on our proposed taxonomy, we present a comprehensive overview of state-of-the-art LTL methods.  In Section <ref>, we provide detailed comparisons between imbalance learning and long-tailed learning, then present a preliminary discussion on long-tailed distribution in Section <ref>.  In Section <ref>, we summarize and analyze the experimental results of different LTL methods in different downstream tasks. We discuss future directions and trends in Section <ref>, and conclude the  paper in Section <ref>. 





§ BACKGROUND AND RELATED WORK
 



 §.§ Background
 
Data imbalance has been a long-standing research issue in both machine learning and computer vision. In the past, researchers commonly used the term “imbalance” to refer to datasets with skewed class proportions <cit.>. However, in the last few years, many deep learning-based methods have emerged for overcoming the data imbalance challenges in computer vision, making this research field even more popular. For instance, <cit.> are among the first methods that adopt deep learning to learn feature representations from imbalanced image data. In the meantime, people have been increasingly using the term “long tail” in these works. Today, these two terms are often used interchangeably.

One reason for the popularity of the “long tail” term in computer vision in recent years is that the training datasets nowadays are often of large scale and present highly imbalanced distributions, and they have a large number of tail (minority) classes. Another reason is the use of deep learning for overcoming the challenges of class-imbalanced distributions and the new problems and research opportunities brought by such deep learning methods. Specifically, the built-in data-driven feature extraction mechanism in deep learning is significantly influenced by the scale of the datasets and their distributions. For image datasets with long-tailed distributions, deep learning-based methods tend to become class-biased in the feature extraction step, hence the need for more advanced  LTL approaches that can effectively extract representative features for the tail classes.

In comparison, traditional computer vision methods use hand-crafted feature extraction methods, which are often model-driven and independent of data distributions. After the feature extraction step, if the resulting features are class-imbalanced, these approaches rely on the subsequent data classification stage to counter such imbalance issues. That is, traditional methods mainly rely on the data classification algorithms/stage for tackling the imbalance problem, where many existing imbalance learning algorithms <cit.> can be straightforwardly applied. Early approaches such as <cit.> fall into this category, which modifies the kernel features of SVM to optimize the class boundary when the extracted features are class-imbalanced.

Besides computer vision, the term “long tail” has also been used in many others fields. 1) In search engines and recommender systems, the long tail issue has been studied in depth, aiming to suggest items or results that are highly relevant to infrequent queries <cit.> or specific users <cit.>. 2) In network traffic modeling,  it is widely accepted that network traffic follows a long-tailed distribution, and different distribution models such as hyper-exponential distribution <cit.> and mixed Erlang model <cit.> have been proposed to model such  long-tailed network traffic data. 





 §.§ Related Work
 

In the literature, there are a few related survey papers on long-tailed learning <cit.>. In <cit.>, the authors propose a taxonomy of three main categories, which are information augmentation, class re-balancing and module improvement.  Their first category consists of data augmentation and transfer learning techniques; the second category contains resampling, class-sensitive learning and logit adjustment methods; the third category consists of  metric learning, classifier design, decoupled training and ensemble learning approaches. In <cit.>, the authors uses a taxonomy for the training stage of LTL that consists of data augmentation, resampling, cost-sensitive loss, multiple experts and transfer learning. In <cit.>, the authors present a taxonomy of data processing, cost-sensitive weighting, decoupled learning and other methods. 

In our taxonomy,  the LTL process is organized into four major modules, containing a total of eight parts/categories. As can be observed from Figure <ref>, the first module is the (input) data pre-processing step, which aims to balance the number of samples of different classes to be used for neural network training via data balancing. The second module is the neural network design  (modeling)  step for effective representation / embedding learning. Based on the internal processing flow of a neural network, it consists of  five parts/categories, which are neural architecture, feature enrichment, logits adjustment, loss function, and bells and whistles (i.e., various training strategies, techniques and practices for model performance improvement). The third module is the (internal) network optimization step, which  aims to effectively update the huge amount of  weights in a network via optimization algorithms and computing the gradients which are the partial derivatives of the loss function with respect to the weights for guiding the update process. The last module is the post-processing (output) step, which adjusts the confidence of the trained long-tailed classifiers or refines their predictions to better fit to the scenarios of real-world applications. 

Comparing with the above surveys <cit.>, our taxonomy is significantly different. First of all, we divide the overall LTL process into four major modules based on the internal learning process of a neural network, including the (input) data pre-processing/balancing step, the neural network design (modeling) step, the  (internal) network optimization step, and the post hoc processing step,  which is more holistic and natural, and easier to understand and accept for those who are new to the LTL field. Second, our taxonomy is more comprehensive, systematic and and in-depth. 1) Previous overview papers in LTL use the term/category “loss reweighting” or “cost-sensitive weighting/learning”, which can not cover other loss functions for the general embedding learning purposes, which we will address in subsection <ref>.  2) Previous overview papers <cit.> lack a systematic discussion on logit adjustment, which is a preliminary step ahead of loss function design. In subsection <ref>, we will specifically illustrate the logits calibration and Softmax margin learning techniques under this category. 3) We categorize weights and gradients rebalancing approaches <cit.> into network optimization <cit.>, which can not be covered by taxonomies in existing survey papers. 












§ LONG-TAILED LEARNING METHODOLOGIES
 

In Figures <ref> and <ref>, we present a new taxonomy for long-tailed learning, in which we divide existing techniques into eight categories, which are 1) data balancing; 2) neural architecture; 3) feature enrichment; 4) logits adjustment; 5) loss functions; 6) bells and whistles; 7) network optimization; and 8) post hoc processing. 

Data balancing approaches utilize data resampling or data augmentation or data synthesis techniques to build class-balanced training sets for training neural network models. Neural architecture approaches design specific network architectures to improve LTL performance. Feature enrichment methods aim to augment the features of the tail samples using a memory bank or the perturbation strategy, or utilize the additional or multi-modal features extracted from pre-trained/foundation  models. Logits adjustment approaches adjust the logit values or enlarge the classification margins to improve LTL performance. Methods in loss function design either upweight the loss values of the tail-class samples or difficult samples, or devise comprehensive loss functions as optimization objectives to enhance the embedding/representation learning effectiveness. “Bells and whistles” contains various strategies and technical details for network training and performance enhancement. Network optimization mainly involves the weights and gradients updating techniques for the internal network optimization.  Finally, post hoc processing methods calibrate the confidence of a  long-tail model to fit to real-world scenarios.  In the following, we will introduce representative techniques in each category.






 §.§ Data Balancing
 

Data balancing aims to increase the volume and diversity of the training samples of the minority (tail) classes to make the training samples class-balanced. Along this line, there are three representative subcategories of approaches, which are a) data resampling, which balances the number of samples in different classes; b) data augmentation, which expands the size of the training data via transformation operations; c) data synthesis, which generates synthetic samples using more advanced techniques such as GANs, distribution estimation/transfer methods, foundation models (such as GPT-4V, DALL-E <cit.>), etc. Table <ref> summarizes representative methods in each subcategory. 






  §.§.§ Data Resampling
 
In machine learning, oversampling/upsampling the minority classes and undersampling/downsampling the majority classes are two commonly used strategies for handling class-imbalanced distributions <cit.>. In deep learning, class-wise sampling (e.g.<cit.>) and instance-wise sampling (e.g. <cit.>) are two frequently used methods for balancing the number of per-class training samples in the mini-batches. Essentially, class-wise resampling may involve both oversampling (the samples of the minority classes) and undersampling (the samples of the majority classes), while instance-wise resampling (i.e. instance-wise resampling) adopts random/uniform sampling (undersampling) to select samples from the original datasets, regardless of their class labels. In <cit.>, the authors show that class-balanced re-sampling can learn discriminative feature representations when the training samples are highly semantically related to their target labels; otherwise, uniform sampling is even better than class-balanced re-sampling.

In <cit.>, the authors first perform clustering within each class, then select samples through a  quintuplet sampling scheme that enforces both inter-cluster and inter-class margins. Such a sampling strategy can alleviate the information loss brought by random under-sampling. 

Existing approaches often treat samples of the same class equally, without taking into account the  difficulty of individual instances. Hard example mining is a commonly adopted technique in deep learning for enhancing the learning capability of the models <cit.>. To improve the recognition performance on datasets with extremely imbalanced image attribute distributions, the authors in <cit.> propose to mine both the hard positives, i.e. samples from the minority classes that are predicted to belong to other classes or with low prediction scores on the ground-truth class, and the hard-negatives, i.e. samples from other classes that are misclassified to one of the minority classes, which can help enlarge the margins between classes.





  §.§.§ Data Augmentation

Data augmentation is a commonly used technique in deep learning for artificially expanding the training set to improve the accuracy of the models while avoiding overfitting. It includes basic image transformations such as cropping and colour space transformations, and more advanced approaches such as image mixing <cit.>.

RandAugment and its variants. AutoAugment <cit.> and RandAugment <cit.> aim to find a group of transformation operations for data augmentation such that the deep model trained by a neural network can obtain better or the best accuracy on the target dataset. AutoAugment automatically searches for the best image transformation choices/policies for a dataset. RandAugment further investigates how to reduce the search space, where straightforward grid search is found to be very effective. Unlike RandAugment, CUrriculum of Data Augmentation (CUDA) <cit.> is designed to find more fine-grained class-wise augmentations, i.e., the number of candidates for sequential augmentation operations for different classes. They reveal that strong/more data augmentations should be applied to samples of the majority (easy) classes instead of the minority (difficult) classes. Therefore, minority classes are given shallow-degree sequential augmentation operations in the training process for curriculum learning. 

Mixup and its variants. Mixup <cit.> is a data augmentation technique that generates synthetic images with soft labels by performing linear interpolations in both the raw input space and label space. The soft label describes the prior inter-class relationship between the two input images. The main difference between Mixup and conventional transformation-based augmentation techniques lies in that Mixup alters both the raw input space and the label space when creating synthetic/augmented images. The idea of Manifold Mixup <cit.> is similar to Mixup <cit.>, with the only difference being that the former interpolates two images in the latent feature space. Remix <cit.> adapts Mixup to the long-tailed setting by assigning a soft label in favour of the minority class, giving a higher label mixing coefficient to the minority classes. Similarly, UniMix <cit.> extends Mixup to the long-tailed setting by adopting a tail-favoured Mixup factor and an inverse sampling strategy to encourage more occurrences of head-tail pairs. Recently, the authors in <cit.> propose BEM which is a novel data augmentation method for long-tailed semi-supervised learning. It uses a sampling strategy based on class-wise entropy to rebalance data uncertainty in data sampling, and the class activation map to achieve more accurate localization to avoid redundant areas in data mixing. <cit.>  proposes a dynamic Mixup approach for data augmentation, which determines the number of per-class images to select in the next training epoch according to the recognition performance in the previous training epochs. <cit.> proposes the Long-Tail Mixed Reconstruction (LMR) algorithm that reconstructs the few-shot videos through weighted combinations of the head-class samples/videos in the batch using Mixup, which reduces overfitting on few-shot classes.




Cutmix and its variants. Cutmix <cit.> is another image data augmentation strategy that replaces a randomly selected region in an image with a patch from another image during image mixing. The soft label of the generated image can be obtained in the same way as Mixup, but the weights to be used in the linear interpolations should be in proportion to the areas of the patches in the generated images. Based on the idea of Cutmix, the Context-Rich Minority Oversampling (CMO) method <cit.> creates synthetic minority images by simply pasting the real minority samples (as the foreground) onto the majority samples (as the background). This way, the rich contexts can be transferred from the majority samples to the synthetic minority samples, making them more diversified for training LTL models. <cit.> proposes a new augmentation method that first extracts the rich background images from the head classes using activation maps then pastes the tail class images upon them to generate tail-class images with diverse contexts. Instead of arbitrarily pairing the foreground and background images like <cit.>, OTmix <cit.> considers the semantic distances between the foreground and background images/distributions and selects the most suitable background image to generate more meaningful samples.

Other augmentation methods. ISDA <cit.> is a data augmentation method that produces diversified augmented samples by translating features along semantically meaningful directions (e.g. colour, backgrounds, visual angles). MetaSAug <cit.> extends ISDA to the long-tailed setting by learning the semantic directions via meta-learning. Major-to-minor Translation (M2m) <cit.> iteratively modifies (translates) a majority sample until it can be confidently classified as a minority sample by a pre-trained classifier built upon the imbalanced dataset.  M2m  leverages the richer information of the majority samples and does not overly use the minority samples, which can thus learn more generalisable features for the minority classes. <cit.> reveals that existing data augmentation methods have side effects since they sacrifice the performance of certain classes. They propose DODA that dynamically maintains class-wise data augmentation preference list by comparing the number of correctly predicted samples of each class in the current and previous epochs. 

Augmentation methods for long-tailed object detection. In <cit.>, the authors present a comprehensive review on the class imbalance problems in object detection. They summarize four types of imbalance problems in object detection: class imbalance, scale imbalance, spatial imbalance, and objective imbalance. <cit.> analyzes the issue of Intersection over Union (IoU) distribution imbalance in two-stage object detection and proposes the positive Region of Interest (pRoI) generator, which increases the number of positive examples, especially for those with greater IoU values, and improves the final object detection performance.

The authors in <cit.> point out that, for long-tailed object detection, image-level resampling alone is not enough to yield a sufficiently balanced distribution at the object level. They thus propose a joint resampling strategy, termed RIO (Resampling at Image-level and Object-level), which augments the minority classes in the current batch with the object-centric RoI features and box coordinates stored in memory over the past batches. MosaicOS <cit.> is another augmentation method that generates pseudo scene-centric images by stitching object-centric images (i.e. ImageNet) for long-tailed object detection. The difference between RIO and MosaicOS is that RIO uses a memory bank to reuse the object-centric RoI features and box coordinates over the past batches; in contrast, MosaicOS straightforwardly generates synthetic scene images by stitching several object-centric images into one image.







  §.§.§ Data Synthesis

While data augmentation techniques generate synthetic images based on image transformations, data synthesis approaches utilize more complex techniques for image generation, including GANs <cit.>, distribution estimation/transfer  <cit.>, pre-trained vision/vision-language models (foundation models <cit.>), etc. 

GAN based data generation. <cit.> finds that information at the lower resolutions tends to be class-independent thus can be shared by all the classes, while class-specific features are usually unveiled at higher solutions. Inspired by such observations, in GAN-based image synthesis for long-tailed data, it proposes to calculate the unconditional (class-independent) objective for low-resolution images, while injecting conditional information at higher-resolution images. 


Distribution based data synthesis. The authors in <cit.> point out that estimating the centroids of the tail classes is much noisier because they have few samples and propose DisRobuLT, which optimizes against the worst-case centroids within a safety hyper ball around the empirical centroid. Rare-class sample generator (RSG) <cit.> is a data synthesis method which obtains the feature displacement of a real frequent-class sample by subtracting its feature vector with the closest cluster centre of the same class to remove the class-relevant information and obtain the divergence value. It next adds the above divergence value of the frequent-class samples directly to the feature vector of each rare-class sample to create synthetic minority samples. The authors in <cit.> propose LEAP, which first uses the angles of the cosine similarity between each feature vector and the centre of the corresponding class centre to model a Gaussian distribution for each class, next forces the variance of the Gaussian distribution for each tail class to be the same as the averaged variance of the head classes, then randomly selects feature vectors using the new distribution of each tail class. LEAP enlarges the variance of the feature distribution of each tail class so that they become more separated from other classes. GistNet <cit.> also proposes to share the geometry of the head classes with the tail classes. However, unlike LEAP, it uses an alternative implementation of the Softmax function by adding the weight vector with a shape parameter that contains the geometry information shared by all the classes. Since the head classes tend to dominate the optimization of the shared parameters, the geometry of the head classes will be naturally transferred to the tail classes.  Overall, RSG forces the intra-class variation information of the tail classes to be the same as the head classes to generate synthetic samples for the tail classes, while DisRobuLT estimates the centroids of the tail classes using the distributional robustness theory. LEAP uses the averaged variance value of all the head classes as the variance of the tail classes to model the distribution of each tail class, whereas GistNet proposes to learn the shared geometry parameters from the head classes and then transfer them to the tail classes. 




Foundation model based data generation. With the rise and prosperity of large pre-trained models/foundation models, e.g.,  CLIP <cit.>, GPT-4V and DALL-E <cit.>, researchers  attempt using such large pre-trained models to generate synthetic data to address the long-tail problem. In <cit.>, the authors leverage Text-to-Image (T2I) model <cit.> to generate diverse images for the tail classes, then refine them by  measuring the cosine similarity between the  image and the corresponding text features extracted via CLIP <cit.>.








 §.§ Neural Architecture

Neural architecture design, i.e., devising an effective neural network structure/model  for a specified task, long with loss function design, are the two preliminary approaches to improving the overall performance of a neural network (model). Along this line, related methods can be divided into subcategories of decoupled learning framework, early-exit framework, knowledge distillation framework, dual streams inference framework, and multi-expert (ensemble) learning framework.






  §.§.§ The Decoupled Learning Framework

Decouple <cit.> is a seminal work that reveals representation learning and classifier training should be carried out separately (i.e. decoupled) for improving long-tailed recognition. Specifically, as shown in Figure <ref>, the proposed decoupled LTL framework first adopts random sampling (instance-based sampling) and the Cross-Entropy loss for training a conventional feature extraction module in the first stage. It then freezes the backbone and uses class-balanced (class-wise) sampling to retrain the classifier in the second stage. Under the same decoupling framework as  <cit.>, <cit.> proposes to train several teacher models upon batches built via instance sampling in the first stage so that features extracted by different models can be more complementary, then distils the knowledge from the teachers to a student model. <cit.> also adopts the decoupling framework to address the class imbalance issue in medical image datasets, the difference is that it utilizes contrastive learning <cit.> to learn the feature space in the first stage. In <cit.>, the authors use class-balanced sampling and conventional random sampling simultaneously to construct mini-batches of data for the feature extraction step.  They reveal that random sampling avoids the risk of overfitting on the tail classes, while class-balanced sampling prevents the training process from being dominated by head classes. 

The above methods have a clear boundary between the two decoupled learning stages. In Bilateral-Branch Network (BBN) <cit.>, the authors propose to use an annealing factor to gradually transits between the two stages, which can adjust the model to learn the global features from the original distributions first, then gradually transit to the tail data modeling and discrimination learning. In Domain Balancing (DB) <cit.>, the authors devise a soft gate to switch between a conventional feature learning module and a tail-class favoured feature learning module. For each tail-class image, the features from the two modules are integrated to improve the LTL performance. Hybrid-SC <cit.> proposes a hybrid network structure composed of a supervised contrastive learning branch to learn better feature representations from the long-tailed data and a  conventional CNN branch with a cross-entropy loss to learn the classifiers. Like BBN, it smoothly transits between the two branches via a  cumulative learning adaptor. The main difference between Hybrid-SC and BBN is that the former uses supervised contrastive learning in the first stage, which can learn better features. In <cit.>, the authors reveal that the inference of visual relationships in a scene faces the problem of the dual long-tailed distributions of the entity (object) and predicate (relation) classes. To address this problem, they propose the DT2 model, which is based on an iterative decoupled training scheme. Like BBN, DT2 first trains (warms up) the model on the original data with uniform sampling, then fixes the backbone part and fine-tunes the classifier on the (remaining) network with the proposed Alternating Class Balanced Sampling (ACBS) method to capture the interplay between the long-tailed distributions of entities and predicates using knowledge distillation.  

<cit.> points out that the performance bottleneck of the two-stage (decoupling) learning framework seems to be the classifier retraining issue in the second stage. It thus proposes DisAlign for confidence calibration to adjust the decision boundary in the second (classifier retraining) stage. In <cit.>, the authors propose the long-tailed class incremental learning problem and present a preliminary method under the two-stage LTL framework that adds a layer of learnable scaling weights to integrate the outputs of the classifier heads for class-incremental learning.




  §.§.§ The Early-Exit Framework
 
In <cit.>, the authors point out that existing approaches to long-tailed recognition reweight samples only by class size, but without considering the fact that there might be easy examples in the minority classes that get incorrectly up-weighted and hard examples in the majority classes that get erroneously down-weighted. They thus employ the early exiting framework (ELF) to early-exit easy examples in the shallow layers of the network to focus on the hard examples in each class in the deeper layers. In implementation, ELF augments a backbone network with several auxiliary classifier branches. During training, it finds and early-exits the correctly and confidently classified examples at the earliest possible exit branch, while harder examples will exit later in the network, sustaining their impact on the overall loss. Similarly, <cit.> proposes the Hardness Aware Reweighting (HAR) framework, which also adopts the early-exiting strategy in the training process. Specifically, HAR augments a backbone network with auxiliary classifier branches, and samples that are confidently and correctly classified at a branch will early exit from that branch.




  §.§.§ The Knowledge Distillation Framework

In <cit.>, the authors propose Self-Supervised Distillation for long-tailed visual recognition (SSD-LT). Adopting the decoupled learning framework, it trains two heads for knowledge distillation, which are the self-distillation head and the classification head, respectively. It thus owns the merits of rebalanced sampling and decoupled training strategy and is reported to obtain state-of-the-art performance in long-tailed recognition. The authors in <cit.> propose the “Distill the Virtual Examples” (DiVE) pipeline for long-tailed recognition. In DiVE, a teacher model is first trained for the long-tailed task with the existing methods, and the predictions from such a teacher model are used as virtual examples. It then uses knowledge distillation to transfer the knowledge from the teacher model to the student model.

<cit.> discusses the use of ViT/Transformers for long-tailed learning. It proposes DeiT-LT which distills knowledge from a CNN teacher to a ViT backbone/student using out-of-distribution images generated via CutMix and Mixup, in which it reweights the distillation loss to enhance the focus on the tail classes.  Recently, for long-tailed object detection, RichSem <cit.> adds a new branch to the detection framework for distilling semantics from CLIP, which can extract rich semantics for the bounding box patches within the image. 




  §.§.§ The Dual Stream Inference Framework
 
<cit.> designs a neural network architecture with two branches: one branch trains a baseline model on the original long-tailed distribution, while the other trains a second model upon the artificially balanced distributions. They add a cross-branch consistency loss based on their logit difference to enforce the two branches to train the multi-label visual recognition model collaboratively. For long-tailed face recognition, Center-dispersed (CD) Loss <cit.> first splits the original data into head-class data and tail-class data, then feeds it into the “unequal-training” framework that consists of a  backbone and two learning branches for the head classes and tail classes, respectively. The first learning branch is based on the head-class data and is used to stabilise face representation, supervised by a noise-resistant loss. The second learning branch is based on the tail-class data and is used to enhance the inter-class discrimination capability. The training of the framework alternates between the two branches. One difference between <cit.>  and <cit.> lies in that the latter trains two models on the split data (head-classes data and tail-classes data) separately, whereas the former trains two models on the original/complete data.

In SimCal <cit.>, the authors investigate instance segmentation over data with long-tail distributions. Upon Mask R-CNN, their proposed method makes three main modifications: i)  it changes the inputs of the original dataset with batches of class-balanced samples, then selects class-balanced object proposals for each class in the region proposal generation (RPN) stage;  ii) in the classification/segmentation stage, it trains a calibrated classifier with Mask R-CNN on the above proposals; iii) in the inference step (after the training is over), it employs the calibrated classifier and the classifier trained on the original dataset using the vanilla Mask R-CNN algorithm to predict the tail and head classes, respectively. Such a dual-head inference framework yields improved performance on the tail classes while still maintaining similar performance on the head classes.




  §.§.§ The Multi-Expert Learning Framework
 
Ensemble learning methods aim to utilize multiple experts/models to make more accurate predictions. Representative ensemble learning methods for LTL include ACE <cit.>, RIDE <cit.>,  CBD <cit.>, LFME <cit.>, BalPoE <cit.>, SHIKE <cit.>, MDCS <cit.> and LGLA <cit.>. 

The authors in <cit.> propose ACE, a multi-expert ensemble network for long-tailed recognition. It aims to achieve biased representation learning and unbiased classifier training in a single stage. The class indices are first ordered descendingly by their number of samples, the first branch (expert) trains a classifier/model with the samples of all the classes; the next branch (expert) first eliminates the current head class, then trains a new classifier with the samples of the rest classes, and so on. Therefore, the number of classes handled by different classifiers decreases monotonically, and the medium-shot or tail classes will have chances to dominate a classifier, thus mitigating the bias towards the majority classes. Overall, ACE can simultaneously improve the recognition accuracy of both the head and tail classes. RIDE <cit.> trains multiple recognition models/experts on randomly sampled subsets and uses their logits mean to make predictions. It also devises a routing algorithm to skip the inferences of certain experts. CBD <cit.> also trains multiple recognition models/experts on randomly sampled subsets, it proposes to distil the knowledge from the multiple teachers/experts into a single student model. Similarly, LFME <cit.> also propose to distil the knowledge from multiple teachers into a unified/single student model. LFME differs from CBD in that the teacher models of the LFME are trained over non-overlapped partitions of classes and samples, while CBD trains multiple teacher models over randomly sampled subsets which may have overlapped classes/samples. Compared to CBD and LFME, ACE <cit.> trains cascaded models/classifiers by iteratively removing the dominating group of classes and the corresponding samples before training the next teacher model, while RIDE  straightforwardly utilize logits mean/average instead of knowledge distillation to ensemble/ally the teacher models. 

The authors in <cit.> propose BalPoE, which is also an ensemble learning framework composed of logit-adjusted experts <cit.>. They prove that the ensemble is Fisher-consistent for minimizing the balanced error. SHIKE <cit.> is a knowledge distillation-based multi-expert ensemble learning framework for long-tailed recognition. The authors find that shallow features of a network (or a shallow network) can better represent the tail classes. They thus propose aggregating the deep features in different experts/branches with different layers of shallow features, respectively. Next, they use knowledge distillation techniques to let the experts learn from each other. Like other ensemble learning algorithms, the averaged predictions from the multi-experts will then be used as the final prediction. Nested Collaborative Learning (NCL) <cit.> also trains multiple experts on all the categories and some of the hard categories, respectively. It then uses knowledge distillation to learn the multiple experts collaboratively. MDCS <cit.> is a multi-expert learning framework that introduces a diversity loss (which is essentially the Balanced Softmax loss <cit.>) to control experts' focus on different categories with an adjustable weight to improve the diversity of experts, and performs  self-distillation between weakly and strongly augmented logits sets for each expert to distill the richer knowledge of weakly augmented instances to reduce model variance. 

Considering the fact that the class distributions of test data are not necessarily uniform but might be long-tailed or even inversely long-tailed, the authors in  <cit.> propose SADE which is a test-agnostic long-tailed recognition approach that trains three experts/models on uniform, long-tailed, and inversely long-tailed class distributions separately, then weighted aggregates them at the inference stage. DirMixE <cit.> is also an multi-expert learning approach, which uses the Monte Carlo method to estimate the mean and semi-variance of a meta-distribution for capturing both the global and local variations to adapt to test-agnostic distributions. LGLA <cit.> trains multiple experts on the entire dataset to extract more discriminative features for different subsets, which uses logit adjustment strategy to enlarge the discrepancy among the subsets. It also learns an expert excelling at inversely long-tailed distribution, next ensembles these experts to achieve superior prediction performance.

<cit.> proposes to leverage the powerful discrimination ability of the large-scale pre-trained vision models <cit.> to tackle long-tailed classification directly. It proposes LPT which first adapts the pre-trained model to the target domain using shared prompt tuning, next adopts the divide-and-conquer strategy for long-tailed recognition by using the pre-trained model and shared prompt to generate a  group-specific prompt set for each group of classes. 

Besides, lately, the Transformer architecture  <cit.> is being increasingly utilized in LTL <cit.>. Transformer based LTL methods, such as  LTGC <cit.> and VL-LTR <cit.> reported significantly better performance than CNN based methods in long-tailed recognition, while Step-wise <cit.>  which adopts the CNN-Transformer combined architecture, also showed superior performance in long-tailed object detection. 




 §.§ Feature Enrichment




Feature enrichment methods enrich/augments features of the tail classes  using fusion-based approaches <cit.>,  perturbation based approaches <cit.>, large pre-trained models/foundation models<cit.>, etc. 




  §.§.§ Fusion Based Feature Enrichment


As depicted in Figure <ref>, the Retrieval Augmented Classification (RAC) method <cit.> uses an external memory module to store features of the tail samples explicitly. Next, for each input image, it finds in the memory bank the Top-K most similar features of the same class, which are then fed into a BERT-like encoder to be transformed into logits, then fused with the logits derived from a conventional backbone. Likewise, the authors in <cit.> propose PFENet, a prior guided feature enrichment network that enriches query features with support features and prior masks to overcome the spatial inconsistency between the query and support samples in few-shot segmentation. LOCE  <cit.> also utilizes a memory module to store features of the tail-class samples for augmenting the features of the tail classes, which works collaboratively with the proposed  Equilibrium Loss (EBL) that adjusts the margins for the tail classes in long-tailed detection. In OLTR <cit.>, the authors combine the features extracted by a conventional CNN pipeline and the features stored in an additional memory module to generate features of the tail classes, which can transfer knowledge between head and tail classes, and increase the sensitivity of the model to novel classes.

<cit.> proposes to disentangle image features into  class-agnostic features (a.k.a. category-related/class-relevant) and class-generic features (a.k.a. category-independent/class-irrelevant)  for  one-shot image generation. It uses an auxiliary memory module to store class-generic features. The class-agnostic features from the input image and the class-generic features from the memory module are then fused and sent to the generator of a vanilla GAN for image generation. Likewise, in <cit.>, the authors point out that some features are shared by the head and tail classes, while other features are class-agnostic (unique to certain classes). The authors thus propose to transfer the class-generic features from the head to the tail classes. <cit.> propose to augment tail classes by grafting the diverse semantic information from head classes, where it replaces a portion of feature maps from tail classes with those belonging to head classes. Such fused features substantially enhance the diversity of tail-class representations. To prevent the neural networks from fitting the head classes first, MFW <cit.> weakens features of the head classes by mixing them with features of other classes in the same mini-batches but without altering the corresponding labels to equalize the training progress across classes.

CoSSL <cit.> devises a tail-class feature enhancement mechanism that gives a higher probability to the tail classes when blending features of unlabeled and labelled data.




  §.§.§ Perturbation Based Feature Augmentation

The authors in <cit.> propose the FASA method for long-tailed instance segmentation that generates class-wise virtual features by random perturbation over a Gaussian distribution on the real samples of each class. 

CC-SAM <cit.> proposes to improve the generalization capability of the LTL models by robust training against class-conditioned perturbations on the features of the last convolutional layer. Its idea is similar to that of LogitAdjust <cit.>. The main difference is that LogitAdjust performs perturbations over logit values in the output layer, whereas CC-SAM adds noises to the features in the last convolutional layer (the last hidden layer).



  §.§.§ Foundation Model Based Feature Enrichment


Recently, foundation models such as CLIP <cit.> and ALIGN <cit.> have demonstrated impressive representation and generalization capabilities. In <cit.>, researchers have  explored using such foundation models for feature-level enhancements. They simultaneously utilize the visual and linguistic representations extracted by the foundation models CLIP <cit.> for long-tailed recognition. For long-tailed object detection, RichSem <cit.> adds a new branch to the detection framework for distilling knowledge from  CLIP, which can extract rich visual and linguistic semantics to enhance the representations of the objects within the image.




 §.§ Logits Adjustment
 
In the output layer, logits contain the raw prediction values of the neural network, which will be used as the input to the Softmax activation function to turn the logit values into a probability distribution over the input classes. Methods that fall into the logits adjustment category either optimize the inherent process in deriving the logit values  <cit.>, or calibrate the logit values in a post hoc manner <cit.>. It also includes softmax margin learning methods <cit.> that enforce margins into the logit values so as to influence the resulting Softmax classifier to have larger classification margins <cit.>. We note that, sometimes it is challenging to clarify the differences between logits calibration and Softmax margin learning. For instance, LogitAdjust <cit.> can be considered to be belonging to both subcategories. In Table <ref>, we summarize the formulas of representative methods in this category.






  §.§.§ Logits Calibration


Inherent Logits Calibration. When deriving logits from the last hidden layer to the output layer, IR-Softmax <cit.> sets the weights for each class as their corresponding class centres in the feature space to avoid the shift between the weights and their centres. The authors in <cit.> propose Balanced MSE that straightforwardly uses the mean square error on each class to replace the corresponding logit element in imbalance regression. 

Post hoc Logits Calibration. In <cit.>, the authors argue that the obscure regions between classes in classification are caused by Softmax saturation. To alleviate this issue, they propose the Gaussian Clouded Logit adjustment (GCL) loss that perturbs the normalized logit with an additive margin that has a random disturbing parameter. In <cit.>, the authors reveal that pseudo-labels in semi-supervised learning are naturally imbalanced. They propose a Debiased Pseudo-Labeling method (DebiasPL) that deducts the probability for each class from the logit. After that, margin learning methods can still be applied. 

Like GCL <cit.>, in BLV <cit.>, category-wise variations are introduced into network predictions (logits) for long-tailed semantic segmentation, where tail classes obtain larger variations by multiplying the random value in a Gaussian distribution with a coefficient that is inversely proportional to the per-class sample size. This simple strategy improves the segmentation performance on the tail categories.

<cit.> finds that a classifier trained on long-tailed/imbalanced data produces skewed class probabilities on a solid color image, rather than uniform probabilities, it thus proposes to adjust the classifier's predicted logits on a test sample by subtracting its predicted logits on a white image, leading to improved recognition performance.

Most LTL methods train a model on long-tailed data but evaluate their performance on uniformly distributed data. To address this issue, the authors in <cit.> propose the LAbel distribution DisEntangling (LADE) method, which regularizes the logits so that they can be explicitly disentangled from the source label distribution and adapt to arbitrary target distributions.




  §.§.§ Softmax Margin Learning

Softmax margin learning algorithms alter the logit values before sending them to the Softmax classifier (Softmax activation function) to improve the discrimination power of the classifiers. Therefore, it can also be considered belonging to the “Post hoc Logits Calibration” sub-category. In label-distribution-aware margin loss (LDAM) <cit.>, when using the logits to calculate the  Softmax output,  the corresponding logit element for the ground-truth class of the input sample will subtract a class-wise margin parameter, which is reversely proportional to the quadratic root of the number of samples in the class. Equalization Loss (EQL) <cit.> removes the logit values of other tail classes in the denominator of the Softmax activation function. AutoBalance <cit.> adopts a generalized cross entropy loss with multiplicative and additive margin parameters, and uses a bi-level optimization procedure to learn these parameters on a validation set directly. The overall ideas of BALMS <cit.> and  LogitAdjust <cit.> are very similar to AutoBalance: the main difference lies in the multiplicative and additive margin parameters used in the Softmax function, as can be seen from Table <ref>. Similarly, in <cit.>, the authors propose the Category-Aware Angular Margin (C2AM) loss for long-tailed object detection, which adds a margin for every non-ground-truth class,  based on the ratio between the weight norms for the ground-truth class and every other class. In <cit.>, the authors deduce that margin-based binary classification provides a closed-form solution for the ideal margin of each object category in long-tailed object detection and propose the Effective Class-Margin (ECM) loss for optimizing margin-based classification.

In <cit.>, the authors propose a progressive margin loss (PML) for facial age classification. They propose a framework that consists of an ordinal margin learning head and a variational margin learning head specially tailored to long-tailed data. The weighted sum of the two margins is adopted to make adjusted predictions.

As a short summary, logits adjustment either intervenes the derivation process of logits, which occurs between  the last hidden layer and the output layer, or modifies the existing logit values, primarily by adding different margin parameters. Ultimately, the calibrated logits will be fed into the Softmax Activation Function, resulting in a better-performing Softmax classifier.




 §.§ Loss Function

An embedding space, also known as the latent space or latent feature space, is the new space produced by a learning algorithm where semantically similar objects are positioned close together, and dissimilar objects are placed far apart <cit.>, as illustrated in Figure <ref>. Roughly speaking, embedding (metric) learning refers to any technique or process that converts an input image into an effective feature representation. Loss function design, along with neural architectures,  are the main approaches to embedding learning and optimization. In fact, the aforementioned feature enrichment techniques, logit adjustment techniques, neural architecture,  as well as training strategies (bells and whistles), are all integral parts of the overall embedding learning process. 






  §.§.§ Loss Function For Embedding Learning
 

In this subsection, we will  introduce various loss functions for deep embedding learning in long-tailed settings.


Long-tailed  Recognition. The authors in <cit.>  propose Affinity Loss which jointly performs max-margin classification and feature space clustering for enhancing inter-class separability and reducing intra-class variations. DASO <cit.> generates unbiased pseudo-labels by blending two complementary pseudo-labels obtained via a linear classifier and the similarity computation between the feature vector of a sample and each class centre, respectively. In DLSA <cit.>, the authors propose to tackle long-tailed recognition by adjusting the label space of the samples via deep clustering. They first cluster the samples according to their likelihood in the feature space. The well-clustered samples are sent to a dedicated Cluster-aided Classifier, which exploits the cluster prior information in classification. The remaining samples are outliers and are sent to the next adjustment module to decrease the training difficulty further. SBCL <cit.> devises a triplet loss like contrastive loss  for long-tailed recognition. It initially divides the head classes into multiple sub-classes  and enforces a sample to be closer to samples of the same sub-class than other samples in the same class, while being closer to samples of a different sub-class under the same class than samples from different classes in the loss function. GLMC <cit.> is based upon the Siamese network architecture: for each pair of samples, it respectively uses the CMO <cit.> and Mixup <cit.> methods to create two synthetic images, then uses both the cosine distances between the features of the two images and the classification loss on the two images to optimize the network.  

In <cit.>, the authors define the distribution overlap coefficient to quantify the dominance severity of the head classes in the representation space and formulate the inter-class discrepancy and class-feature consistency losses to close the gap. The authors in <cit.> find that the distribution of the super-classes in imbalanced data is more balanced than the original data. They thus propose SuperDisco to discover the super-class in a hierarchy of semantic abstraction, then refine the original features of a sample using the corresponding super-class features. 

Contrastive learning  <cit.> has recently gained popularity since it can substantially improve the capability of the neural network in representation learning. The authors in <cit.> propose the K-positive Contrastive Loss (KCL) for LTL, which draws k instances from the same class to form the positive sample set for contrastive learning. Based upon KCL, targeted supervised contrastive learning (TSC) <cit.> adds a new loss term between the samples and the predefined optimal class centres to move the samples closer to the centres of their class and away from centres of other classes.  The PaCo <cit.> loss introduces learnable class-wise centers into the supervised contrastive learning process to make the probability that two samples are a true positive pair become more balanced across different classes, which is beneficial for long-tailed recognition. Model-Aware K-center (MAK) <cit.> sorts the external samples by their empirical contrastive loss expectation value, then selects the Top-N samples to perform K-means clustering and chooses the final samples. The authors in <cit.> propose  Balanced Contrastive Learning loss (BCL) that includes cluster compactness in the contrastive learning process. 

In <cit.>, the authors design a hybrid network comprising a prototypical supervised contrastive (PSC) learning module and a classifier training module. A curriculum is designed to control the weightings of these two modules during network training.  Motivated by the observation in <cit.> that model pruning negatively influences the prediction accuracy on samples of the infrequent classes, the authors in <cit.> design the  Self-Damaging Contrastive Learning (SDCLR) framework, which adds a pruned contrastive learning module to the (original/non-pruned) unsupervised contrastive learning module, to amplify the prediction differences between the pruned and non-pruned models and increase the weights of the infrequent class samples and hard samples in the overall loss. SDCLR increases both the overall accuracy and the balanceness of the learned representations.

<cit.> reveals that existing semi-supervised learning methods obtain low recall performance in the minority classes, but the corresponding precision performance is high. It thus proposes CReST for  semi-supervised LTL that iteratively samples pseudo-labelled data from the unlabeled set to expand the labelled set, then retrains the semi-supervised learning model. SimPro <cit.> is a simple probabilistic framework for long-tailed semi-supervised learning, which explicitly separates the estimation process into conditional and marginal distributions using the Expectation-Maximization algorithm.  ABC <cit.> attaches an auxiliary classifier head to the representation layer of a semi-supervised learning framework to avoid bias towards the majority classes in LTL. ACR <cit.> uses a dual-branch network to adaptively handle various class distributions of unlabeled data, in which it derives the scaling parameter using the bidirectional KL divergence between the estimated class distribution and three anchor distributions. 

Long-tailed  Regression. When quantizing the imbalanced regression target space into interval classes for imbalance regression, HCA <cit.> proposes a coarse-to-fine hierarchical classification strategy which first uses coarse classifiers to provide the range, next selects the corresponding finer classifier in the range, and achieves improved regression performance.

Long-tailed  Object Detection and Segmentation. The authors in <cit.> propose the Balanced Group Softmax method (BAGS), which groups the classes by the number of instances, then trains a separate Softmax classifier for each group to reduce the suppression by the head classes. However, finding a proper group partition number for  BAGS could be laborious and tricky. Thus, the authors in <cit.> devise the Adaptive Class Suppression Loss (ACSL), which utilizes the output confidence to prevent the classifiers of tail categories from being over-suppressed by the samples from head categories, and only the samples from visually confusing categories are kept for discriminative learning. Learning to Segment the Tail (LST) <cit.> also adopts the “divide-and-conquer” strategy for long-tailed segmentation. It divides the original data into several balanced parts and devises a knowledge distillation loss which measures the logit difference of the same class in neighbouring parts to preserve knowledge learned in previous parts. <cit.> adds an Adaptive Hierarchical Representation (AHR) loss to the traditional object detection loss for more discriminative embedding learning. It calculates both the coarse-grained classification loss for distinguishing each cluster clearly, and the fine-grained classification loss within each cluster by adopting adaptive margins. In <cit.>, the authors point out that, in contemporary deep learning-based object detectors, classification and regression are often optimized independently, which leads to inconsistent predictions with high classification scores but low localization accuracy, and vice versa. They thus propose a harmonic mutual-supervision method which devises a regression-aware classification loss to optimize the classification branch and a classification-aware regression loss to supervise the optimization of the regression branch. <cit.> proposes an unsupervised instance segmentation method that uses a pre-trained mask generation network to generate object proposals, next samples mask triplets from the proposals to capture the hierarchical relations and learn the embeddings. 

Considering the advantages of the Gumbel distribution in modeling extreme events with very low probabilities, the authors in <cit.> propose GOL that utilizes the Gumbel activation function for long-tailed object detection and segmentation. Through large-scale experiments, they demonstrate the superiority of Gumbel against Sigmoid and Softmax in detecting/segmenting rare-class objects.




  §.§.§ Loss Reweighting
 

After obtaining the loss on each class or sample, loss reweighting approaches aim at adjusting the importance of their loss values through the weighting factors. Existing approaches along this line can be divided into class-wise reweighting and instance-wise reweighting methods. In Table <ref>, we report the formulas of representative methods in this sub-category.





Class-wise Loss Reweighting. Class-wise loss reweighting methods assign the same weighting factor to instances of the same class or group, as demonstrated in Figure <ref>. CB Loss <cit.> is a class-balanced loss function for long-tailed distributions that reweights the samples by the inverse class frequency, i.e. the weight/factor of a class is inversely proportional to the number of samples in the class. AREA <cit.> proposes to re-weight the loss of each class based on the inverse Pearson correlation of the samples in the same class.  Range Loss <cit.> is also a class-wise reweighting method that consists of an intra-class loss and an inter-class loss: the former is defined as the harmonic mean of the two largest intra-class distances, while the latter is the shortest distance between any two class centres. The Learning to Reweight method (L2RW) <cit.> is a meta-learning algorithm that assigns weights to a batch of samples based on their similarities in gradient directions.  TCR <cit.> is a two-component reweighting loss function that simultaneously computes the class-wise weights using the CB Loss <cit.> and the conditional distribution between the source and target with L2RW <cit.>. 


Instance-wise Loss Reweighting. The above methods assign the same weight to all samples belonging to the same class. In contrast, instance-wise reweighting methods assign different weights to different samples based on their learning difficulty. Focal Loss <cit.> proposes to assign relatively larger weights to samples with high prediction loss, independent of their class labels. In <cit.>, the authors propose the Equalized Focal Loss (EFL) for long-tailed object detection, which modifies the original Focal loss by supplementing the focusing parameter with a new term that is the inverted ratio between the accumulated positive and negative gradients of the input sample's ground-truth class. In <cit.>, the authors propose the Population Stability Index (PSI) to measure the overall trend that an instance experiences loss increase over time. It pays more attention to difficult instances, including those from the majority classes.  In IB Loss <cit.>,  the authors propose a method that reweights each sample differently using the designed influence function to alleviate overfitting in the decision boundary. The authors in <cit.> discover that models trained on long-tailed datasets are more miscalibrated and over-confident than those trained on balanced data. They thus propose the label-aware smoothing soft label method  (MiSLAS) that assigns different weights to the head and tail classes to handle different degrees of over-confidence. 




 §.§ Bells and Whistles

In the context of long-tailed learning, “bells and whistles” refer to various techniques, training strategies and practices that go beyond the neural architectures and loss functions but are also essential for improving model performance. Common “bells and whistles” include network training and  optimization strategies and choices, such as advanced optimizer, learning rate schedulers, regularization, and commonly adopted techniques in deep learning such as data augmentation, pre-training and transfer learning, knowledge distillation, and ensemble learning for improving the performance of deep learning models. 

There are different types of optimization algorithms used in neural networks <cit.>, among them,  Stochastic Gradient Descent (SGD) is the major algorithm for optimizing the parameters (weights and biases) in neural networks for long-tailed learning. Learning rate (LR) schedulers are methods that adjust the learning rate during network training, such as linear decay, step decay (such as CosineAnnealing), time-based decay and exponential decay to optimize model performance over time. Regularization techniques in deep learning are methods used to prevent overfitting and improve the generalization ability of neural networks.  Commonly used regularization techniques include L2 regularization (Weight Decay) <cit.>, dropout, batch normalization, label smoothing (which replaces hard labels with smoothed distributions to reduce the model's confidence and encourage it to be more robust and generalize better) <cit.>, etc. 

Since data augmentation and synthesis, knowledge distillation, ensemble (multi-expert) learning methods have already been addressed in the previous subsections,  we only present a few pre-training and transfer learning techniques, and foundation models based approaches. 

Pre-training and transfer learning for long-tailed learning. SSP <cit.> introduces self-supervised pre-training into class-imbalanced learning to yield good network initialisation, which can alleviate the label bias issue in imbalanced datasets and enhance LTL performance. FBGL <cit.> uses both contrastive pretraining and normalization for improving the feature extractor and classifier for long-tailed recognition, which consists of a new balanced contrastive loss to improve  long-tailed pretraining and a generalized normalization method for the classifier (feature and weight vectors). The authors in <cit.> propose a simple yet effective step-wise learning framework based on Deformable DETR <cit.> that combines fine-tuning and knowledge transfer for long-tailed object detection. In specific, after selecting representative examplars for the head and tails classes,  it first pre-trains a model on all the classes, next learns a head-class model via fine-tuning, then transfers knowledge  from the head-class model to the final model. 

Leveraging large foundation models and multi-modal models for long-tailed learning. Lately, there has been a popular trend in utilizing the powerful feature representation capability of foundation models <cit.> and pre-trained multi-modal models <cit.>  to improve the LTL performance, which can also be considered as a “trick” for long-tailed learning. For instance, integration of textual and visual features  enhances the model's ability to discriminate between classes with fewer examples <cit.>, or enables the model to understand and reason about the semantic relationships between concepts depicted in images and described in texts <cit.>.

Finally, we note that, these strategies and techniques are often collectively used to make LTL models more effective and efficient. 




 §.§ Network Optimization

Network optimization refers to the internal mechanism for updating and optimizing the neural networks to  minimize the loss/objective function. Besides neural architecture and loss function design, optimizing the huge amount of internal learnable parameters of a neural network, mainly the weights and biases, plays a very important role in efficiently and effectively training and optimizing a deep learning model. In particular, this process involves techniques in updating the weights, and the gradients with respect to the loss function and weights.  Existing approaches in this line of research can be divided into weights rebalancing and gradients rebalancing approaches.

We clarify that, “network optimization” refers to techniques that optimize or utilize the weights and gradients of the network for internal network optimization, while “Bells and Whistles” in the above subsection refer to different external training strategies, choices, techniques and tricks for obtaining better learning performance.




  §.§.§ Weights Rebalancing

The authors in <cit.> show an obvious correlation between sampling frequency and the norm of the weight vectors, and propose to raise the weight vector of each class by multiplying the ratio between the number of samples in the most frequent class and the current class to adjust the decision boundary in long-tailed recognition. The authors in <cit.> also reveal that, for long-tailed data, a normally trained deep learning model has imbalanced norms of classifier weights, with the majority classes having dominating weights. They propose to balance network weights using standard regularization techniques such as weight decay (L2 regularization) and notably improve the long-tailed recognition performance. <cit.> discusses why weight balancing <cit.> is effective in two-stage long-tailed learning. It reveals that  weight decay helps achieve better feature extraction, while  class-balanced loss enables implicit logit adjustment in the training. Curvature-Balanced  Regularizer (CR) <cit.> uses the sum of the logarithm of the inverse of the maximum normalized Gaussian curvature mean of each class as a regularizer to enable the model to learn curvature-balanced manifolds. 




  §.§.§ Gradients Rebalancing

<cit.> reveals that long-tailed data suffers from gradient distortion problem in which the overall gradient is shifted toward the head gradient,  it thus proposes a two-phase long-tailed recognition method which first updates model parameters  using the gradient on  the head classes only, next grows the classifier on the added tail classes. The authors in <cit.> propose BSGAL which is a gradient-based generated data contribution estimation method for long-tailed instance segmentation that calculates the loss and gradient of the model on a batch of data with and without augmentation to actively accept or reject the generated data. SeeSaw Loss <cit.> dynamically rebalances the gradients of positive and negative samples for each category with two complementary factors to reduce punishments on tail categories and increase the penalty of misclassified instances. For long-tailed class-incremental learning, <cit.> proposes to separately reweight the gradients for the new tasks and learned tasks, based on the accumulated magnitude of gradients and the volume of the lost training data of each class, respectively. 

Overall, properly tuning the above factors is crucial for the internal optimization of the neural network to achieve high model accuracy and prevent overfitting in long-tailed learning.





 §.§ Post Hoc Processing



As demonstrated in Figure <ref>, post hoc processing methods modify the outputs of the (conventionally) trained models to tackle the imbalance problem at the inference stage, instead of improving the feature representation capability or adjusting the decision boundary at the model training process. The expected calibration error (ECE) <cit.> is often used to measure how well the predicted probabilities match the ground-truth probabilities, which first partitions the confidence scores into several equal-size bins,  then computes the difference between the accuracy and the averaged confidence in each bin.




  §.§.§ Post Hoc Confidence Calibration

Temperature Scaling (TS) <cit.> uses a single scalar parameter T to rescale the logits of a trained neural network, which will then be transformed into calibrated confidence scores using the Softmax activation function. However, this approach has yet to be validated in long-tailed settings. 

For confidence calibration in long-tailed object detection, NORCAL <cit.>  multiplies the prediction (output) scores with a vector of factors that are inversely proportional to the training sample size of each class. 

For confidence calibration in long-tailed object segmentation, the Pairwise Class Balance (PCB) loss <cit.> post-calibrates a sample's prediction probability on each class by multiplying the corresponding pairwise misclassification bias obtained from the confusion matrix. 




  §.§.§ Confidence Aggregation

When a neural network has multiple heads for inference, e.g. the dual streams inference framework and the multi-expert learning framework addressed in subsections <ref> and <ref>, these methods commonly use the logits mean <cit.> or the weighted aggregation of the logits <cit.>, as well as the confidence mean to make the final prediction.





§ COMPARISONS BETWEEN LONG-TAILED LEARNING AND IMBALANCE LEARNING
 

Imbalance learning is a long-standing research direction in machine learning <cit.>.  Long-tailed learning  belongs to imbalance learning, but they also have differences in the learning approaches. While conventional imbalance learning algorithms commonly deal with rational/tabular imbalanced data, long-tailed learning often handles image data, which needs the feature representation learning step as a prerequisite. That is, for image data, LTL needs deep representation learning techniques to extract features from images, in which neural architecture, feature enrichment and logit adjustment techniques are needed. 

However, for relational/tabular data, its features are its attribute  values, therefore,  features are already available before imbalance learning, thus feature representation techniques such as neural architecture, feature enrichment and logits adjustment, are not essential in imbalance learning. But we note that, in recent years,  deep imbalance learning approaches have also emerged, which use neural networks to improve the imbalance learning performance on relational/tabular data. 

In the following, we will mainly compare the data balancing and loss functions/cost-sensitive learning methods, as well as the multi-expert/ensemble learning methods adopted in both LTL and imbalance learning. 




 §.§ Data Balancing





  §.§.§ Data Resampling


In long-tailed learning, only random sampling is utilized, either at class-level or instance-level, referred to as class-wise sampling and instance-wise sampling, respectively. However, in imbalance learning, more advanced undersampling and oversampling techniques have been proposed. 

Undersampling techniques in imbalance learning. In Condensed Nearest Neighbor Rule <cit.>, for each sample from the majority set, if it can not be correctly classified by the Nearest Neighbor (NN) rule, it is kept; otherwise, it is discarded. In <cit.>, the authors propose an intuitive geometric space partition (GSE) based method for imbalance classification which iteratively uses a new hyper-plane classifier to cut the current geometric data space into two partitions, then removes the partition that only contains the majority of samples. 

Oversampling techniques in imbalance learning. ROS  <cit.> is the most straightforward oversampling technique, which  balances the data by replicating the minority class samples. The Majority Weighted Minority Oversampling Technique (MWMOTE) <cit.> first identifies the most important and hard-to-learn minority class samples, then assigns weights to them according to their Euclidean distance to the nearest majority class samples.




  §.§.§ Data Augmentation


For image data, in long-tailed learning, Mixup <cit.>, ReMix <cit.> and UniMix <cit.> create synthetic samples with interpolated feature vectors and labels out of two original samples.

For relational/tabular data, since the features are already available before imbalance learning, it is easy and common to use linear interpolation algorithms to generate synthetic minority samples between two real minority samples of the same class. SMOTE <cit.> creates synthetic minority samples by randomly selecting a new point on the line segment between a pair of neighbouring minority samples. Borderline SMOTE <cit.> extends SMOTE by emphasizing the borderline samples that have both majority and minority class points in their neighbourhood and ignoring the rest of the minority samples. ADASYN <cit.> creates synthetic minority samples according to data density, where a comparatively larger number of synthetic samples are created (using SMOTE) in regions of low density of the minority class than in higher-density regions. SMOTE-RSB <cit.> first uses SMOTE to generate synthetic instances for the minority classes, then employs the rough set theory as a cleaning (undersampling) method to select only the synthetic minority examples that belong to the lower approximation of their class.

In general, the idea of Mixup, ReMix and UniMix is similar to SMOTE  and its variants, but the former interpolates the label space as well, while SMOTE and its variants only handle samples of the same minority classes,  but without modifying the labels of the synthetic samples. 




  §.§.§ Data Synthesis


In LTL, there are distribution based data synthesis methods, such as (RSG) <cit.>, LEAP <cit.> and GistNet <cit.>, GAN based <cit.> and foundation model based data generation <cit.> methods. In comparison, in imbalance learning,  GANs are commonly utilized to generate synthetic tabular data, such as TableGAN <cit.>, CTGAN <cit.>, medGAN <cit.> and QAST <cit.>. 

TableGAN <cit.> was the first attempt to synthesize tables/relational databases using GANs. It adopts the Deep Convolutional Generative Adversarial Network (DCGAN) architecture, but adds a new neural network module called “classifier” and a corresponding loss function to measure the discrepancy between the label of a generated record and the label predicted by the classifier for this record. CTGAN <cit.> models the distribution of the values with a few Gaussian distributions (modes) for each continuous attribute. For discrete attributes, CTGAN introduces a conditional generator and training-by-sampling method to ensure that each category of a discrete attribute gets a fair chance to be included during sampling. This way, it can simultaneously generate a mix of discrete and continuous attributes. Medical Generative Adversarial Network (medGAN) <cit.> uses Auto-Encoders to learn the salient features of discrete attributes and GAN to generate realistic samples in the embedding space. To address the usability of GAN-generated samples in imbalanced learning,  QAST <cit.> designs a semantic pseudo-labelling module to control the quality of the generated features, then calibrates their corresponding semantic labels using a classifier committee consisting of multiple pre-trained shallow classifiers. The above approaches only generate synthetic examples for the minority classes. In <cit.>, the authors propose using GAN to generate a subset of the majority of samples that follow the same distribution as the original dataset. Next, they construct a few balanced subsets, each containing all the minority samples and an equal-sized subset of GAN-generated majority samples. Then, upon each decomposed two-class subset, they use deep metric/embedding learning to build a two-class classification model. These models are then integrated to make the final predictions.  Recently, a diffusion model <cit.> based imbalance learning method, namely TabDDPM  <cit.>,  has been proposed for generating rational samples.




 §.§ Loss Function




  §.§.§ Loss Function for Embedding Learning


The idea behind embedding learning is to map complicated (often high-dimensional) metric spaces into easier (often low-dimensional) metric spaces, where inter-class and intra-class distances can be optimized simultaneously. In LTL, there are many specially designed embedding learning approaches, such as <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, etc.

In imbalance learning over relational data, many traditional (non-deep learning based) embedding learning methods have been proposed to transform the learning space. IML <cit.> is an imbalance learning method that aims to find a more stable neighbourhood space for the testing data by performing metric learning on the data and training sample selection according to the testing data iteratively until the selected training samples in two adjacent iterations are the same. MLFP <cit.>  expands the decision boundaries around the positive samples and learns better metric space using a triplet-based cost function on the false negative and false positive samples. 

In recent years, many deep imbalance learning approaches have also been proposed which use neural networks to transform the metric space and learn better embedding/feature representations. Khan et al. <cit.> propose a cost-sensitive (CoSen) deep neural network method to learn feature representations for both the majority and minority classes. <cit.> addresses the problem of imbalanced data for the multi-label classification problem using deep learning. It introduces incremental minority class discrimination learning by formulating a class rectification loss regularization, which imposes an additional batch-wise class balancing on top of the cross-entropy loss. This can rectify model learning bias due to the over-representation of the majority classes. GAMO <cit.> is an end-to-end deep oversampling model for imbalance classification, which effectively integrates sample generation with classifier training. 




  §.§.§ Loss Reweighting

In LTL, loss-level reweighting methods assign different weights to the loss values of the samples, which include class-wise reweighting algorithms such as CB Loss <cit.> and L2RW <cit.>, instance-wise reweighting algorithms such as IB Loss <cit.>, as illustrated in subsection <ref>. 

In imbalance learning, there are also a few cost-sensitive learning approaches that modify the internal cost function/objective function by assigning different weights to different samples based on their difficulty. Such methods usually assign a higher cost to minority classes to boost the importance of the minority classes during the learning process <cit.>. To adapt SVM to imbalanced data, cost-sensitive SVM (CostSVM) <cit.> adds a class-specific classification penalty in its cost function. 

Overall, the general ideas for loss-level reweighting in LTL and cost-sensitive learning in imbalance learning are very similar, i.e., they all give higher weights to samples of the minority classes or the misclassified samples. 




 §.§ Multi-Expert Learning


Multi-expert learning (ensemble learning) techniques are applicable to both LTL and imbalance learning.  For instance, the idea of ACE <cit.> to train cascaded learning models for LTL is similar to that of BalanceCascade <cit.> and AdaCost <cit.> for imbalance learning. 

There are also a few differences, for instance, i) Knowledge distillation-based ensemble techniques have been adopted in LTL <cit.>, but not in imbalance learning; ii) Dichotomy-based ensemble technique for multi-class imbalance learning <cit.> are not needed in long-tailed learning, since cross-entropy loss based LTL classifiers are naturally multi-class classifiers.




 §.§ Evaluation Metrics


In terms of evaluation metrics,  existing LTL methods commonly assume that the test sets are class-balanced, that is, all the classes have the same number of test instances in the testing/inference stage. Under such an assumption, only the Top-1 accuracy/overall accuracy metric is reported. However, we note that, in real-world applications, the test sets might also be long-tailed, or at least non-uniformly distributed.  In contrast, in imbalance learning the test sets are often considered to be imbalanced, and various evaluation metrics have been proposed <cit.>, including the overall accuracy, G-mean (the geometric mean of the per-class accuracy), Macro-F1, etc. 

In <cit.>, the authors argue that the average accuracy metric used in LTL incurs little penalty on classes with very low per-class recalls. They thus propose three new metrics for evaluating long-tailed recognition algorithms: the geometric mean and harmonic mean of the per-class recall (referred to as G-mean and H-mean for short), and the lowest per-class recall. They also propose the Geometric Mean Loss (GML), which essentially calculates the logarithm of the averaged prediction probability for samples of each class. 

As a short summary, when the test set distributions are non-uniform or skewed, major evaluation  metrics including overall accuracy, G-mean,  Macro-F1 and the lowest per-class recall, should  be adopted in LTL and imbalance learning to more comprehensively assess the performance of the proposed algorithms.





§ LONG-TAILED DISTRIBUTIONS: A DISCUSSION
 




 §.§ Comparing Long-tailed Data with Imbalanced  Data


When referring to long-tailed data, one may neglect the fact that long-tailed data is a special type of imbalanced data, and long-tailed learning (LTL)  is essentially part of imbalance learning (IL). 

The long tail phenomenon occurs in imbalanced data when a small number of classes are highly frequent and are thus easy to model. In the meantime, there are also a substantial number of tail classes that have insufficient samples and are thus difficult to model. Moreover, the combined quantity or influence of all the tail classes is comparable to that of the head classes. Specifically, there are three main differences between long-tailed distributed data and imbalanced data:

First, in long-tailed data, there should be a significantly large number of minority (tail) classes compared to the majority (head) classes. It thus has a very long tail, also named “fat tail” or “heavy tail”. However, in the case of imbalanced data, the number of classes may be as few as two (two-class imbalanced data) or only a few (multi-class imbalanced data). Therefore, long-tailed data must be class-imbalanced data, but class-imbalanced data is not necessarily long-tailed data.
  
Second, since long-tailed data has a very long tail, the overall trend/pattern of event frequency/sample size per class should decrease exponentially from the head classes to the tail classes, but the sub-pattern/trend among the tail classes should be a relatively flat curve with a slowly decreasing pattern. Ordinary class-imbalanced data does not necessarily have this property.
  
Third, the combined quantity or influence of the tail class samples in long-tailed data should be significant, e.g. 60% or 20% of the total quantity/influence of the head classes. This unique property does not hold in ordinary class-imbalanced data.




 §.§ Comparing Long-tailed Distribution with Pareto Distribution


A Pareto distribution is a power-law probability distribution with heavy, slowly decaying tails. In comparison, a “long-tailed distribution”  essentially corresponds to a set of qualitative properties/constraints added to a class-imbalanced distribution rather than being a statistical distribution that can be rigorously modelled. In the following, we empirically study the differences between a Pareto distribution and a long-tailed distribution.




In Figure <ref>, we first depict the empirical rank-size distributions of two real-world datasets, iNaturalist 2018 and LVIS, typically used in long-tailed learning settings. A rank-size distribution gives the empirical relative frequencies  of the data with the classes ordered in descending order in size across the x-axis[When calculating the relative frequencies, for normalisation reasons, we divide the sample size/frequency of each class by the maximum frequency (the frequency of the Top-1 class), instead of the total number of samples of all the classes.], which corresponds to the empirical survival distribution function (ESDF).



    S(x)=[ 1 + ( x-μ/σ) ^ 1/γ] ^ -α , x≥μ , μ∈ℝ , σ , γ > 0, α≥ 1


For comparing a long-tailed distribution with a Pareto distribution, in Figure <ref> we fit each dataset with a Generalised Pareto Type IV (DPIV) survival function (SF), using Equation <ref>, where  μ is the location, σ is the scale (which corresponds to the minimum value of x),  γ is the inequality parameter, and α is the tail index (shape). Robust estimators for the parameters of DPIV have been proposed in the literature. Here, after fixing parameters μ and σ in relation to the empirical distribution of each dataset, we exhaustively vary the values of α and γ with a step of s=0.1 to heuristically find the set of parameter values that minimises the difference between the ESDF and the fitting Pareto survival function, measured in terms of the relative errors/distances between them, which can be observed from the bottom sub-figures of Figure <ref>.

As the Pareto distribution (DP) embodies a heavy (fat) tail, it is natural to present the survival rates of both the theoretical and the empirical distribution on a logarithmic scale. In Figure <ref>, on the two real-world long-tailed datasets iNaturalist and LVIS, we see that the derived Pareto distribution is not guaranteed to consistently fit to all segments of the ESDF of the real long-tailed data.

In summary,  unlike the Pareto distribution, a long-tailed distribution is not a specific statistical distribution that can be precisely modelled, but rather an imbalanced distribution with a set of properties/constraints.




 §.§ When Using the Pareto Distribution to Model the Long Tail Phenomena



In case a Pareto distribution is used to approximately model the long tail phenomena, we need to add the following constraint to the Pareto distribution:

In long-tailed data, the combined quantity/significance of the tail class samples should exceed a user-specified ratio compared to the head classes, e.g. 60%, or 50%, or 20%.

The Cumulative Distribution Function (CDF) of the Pareto Distribution is given by:


    F(x)  = 1 - (x_m/x)^α


where x_m is the scale parameter, e.g. x_m = 10 or 3, which can be considered as the minimum number of samples that a tail class has.  α is the shape parameter, which is essentially the power law slope. 

Based on the Pareto Distribution, the constraint added by the long-tailed data depicted above can be formulated as: 


    F̅(h_l)= P(X ≥ h_l) = (x_m/h_l)^α≤ 1 - r


where h_l is the lower bound frequency of the head classes, i.e. the number of samples of the least-ranked head class, and r is a user-specified threshold which is the ratio between the combined quantity/influence of the tail classes and the total quantity/influence of all the classes.  Different r values can be set in different circumstances; for instance, r can be 60%, 50%, or 20%.  Essentially, for modeling long-tailed data using the Pareto distribution, Equation <ref> adds to the Pareto distribution a joint constraint on the parameters x_m, α, h_l and r.





§ PERFORMANCE ANALYSIS
 




 §.§ Datasets



For long-tailed recognition, benchmark datasets include CIFAR-100-LT <cit.>, ImageNet-LT <cit.>, Places-LT <cit.>, and iNaturalist 2018 <cit.>. For long-tailed object detection and instance segmentation, LVIS <cit.> is the primary real-world dataset. LVIS v0.5 consists of 1,230 categories, whereas LVIS v1.0 contains 1,203 categories, with 100K images and 19.8K images for the training and test sets, respectively. 




 §.§ Evaluation Metrics

For long-tailed recognition tasks, since existing LTL methods all assume that the test set is uniformly distributed, i.e.,  the sample size for each class in the test set is equal, therefore only the overall accuracy metric is needed for performance evaluation. Besides, these methods often separately report the specific performance on three super groups, which are “Many-shot” (if a class has > 100 instances), “Medium-shot” (20-100 instances) and “Few-shot” (< 20 instances).

For long-tailed detection and segmentation tasks,  the main evaluation metric is Average Precision (AP). For long-tailed segmentation, separate performance on three super groups are also reported, which are AP_f for frequent classes (having > 100 instances), AP_c for  common class (having 11-100 instances), and AP_r for rare classes (having < 11 instances).





 §.§ Performance Comparison





  §.§.§ Long-tailed Recognition


Table <ref> summarizes the performance of representative long-tailed classification algorithms on four benchmark datasets CIFAR-100-LT (with two different imbalance ratios (IR) which are 50 and 100), ImageNet-LT,  Places-LT, and iNaturalist 2018. It can be observed that, under the CNN backbone (ResNet), MDCS <cit.>, LGLA <cit.> and SHIKE <cit.> achieve the best overall performance on all the four benchmark datasets. Besides, NCL <cit.>, SADE <cit.>,  RIDE <cit.> and ACE <cit.> also obtain remarkable performance, all of which adopt the multi-expert learning framework, owing to its effectiveness. While under the Transformer backbone <cit.>, due to the utilization of foundation models, LTGC <cit.> and VL-LTR <cit.> achieve significantly better performance than CNN based methods, in particular, their performance improvements over MDCS <cit.> and LGLA <cit.> on ImageNet-LT  are near 20% in terms of overall accuracy, which is very substantial. 

Besides, we see that the performance of loss reweighting methods tend to saturate, since almost all the possible forms have been exhaustively examined, as can be observed from Table <ref>. 

In Table <ref>, we report the experimental settings of representative long-tailed recognition algorithms on the iNaturalist 2018 dataset. 

Overall, on ImageNet-LT and iNaturalist 2018, the state-of-the-art performance is only 82.5% in terms of overall accuracy, while the best overall performance is only 54.1% on Places-LT, which can hardly satisfy the requirements of real-world applications. Therefore, there is still a large room for performance improvement.




  §.§.§ Long-tailed Object Detection and Segmentation

Table <ref> reports the performance of representative long-tailed object detection and segmentation methods on the LVIS v1.0 benchmark dataset. We see that  BSGAL <cit.>, RichSem <cit.>, Step-wise <cit.>, PCB <cit.>, ECM <cit.>, C2AM <cit.> achieve the best long-tailed object detection performance, while BSGAL <cit.>,  GOL <cit.>, ECM <cit.>, PCB <cit.>, C2AM <cit.> and NORCAL <cit.> are among the best performers in long-tailed segmentation. We would like to note that, Step-wise <cit.> is implemented based upon Deformable DETR <cit.> which is essentially a CNN-Transformer combined detection framework, while the other methods are commonly based on Mask R-CNN with ResNet as the backbone. 

Overall, the state-of-the-art performance is below 35.4% in both  long-tailed object detection and segmentation tasks, and the segmentation performance on rare classes is even lower, which is less than 25.4%. Therefore, there is considerable room for future research.



















§ FUTURE DIRECTIONS
  
While numerous approaches have been proposed to address the long tail issue, many research opportunities remain.





 §.§ Federated Long-Tailed Learning

Federated learning enables multiple clients to learn a global model collaboratively without transmitting the local private data on each client to a centralized server <cit.>. The problem becomes more challenging when the training data across the clients is both heterogeneous and long-tailed. 

Fairness refers to whether a model can perform equally for diverse types of categories. Although there are a few preliminary studies <cit.>, ensuring fairness within a federated learning system in long-tailed settings still needs in-depth investigations.




 §.§ Long-Tailed Out-of-Distribution Detection



Out-of-Distribution (OOD) samples are samples that do not match the training data distribution, regardless of whether they belong to known (existing) categories or entirely new (open-set) categories. In <cit.>, the authors point that it is sometimes challenging to distinguish between the in-distribution tail class samples and the OOD samples that belong to entirely new (open-set) categories. To address this issue, <cit.> expands the in-distribution class space by introducing multiple abstention classes and adopts mixture of experts for classification. It also augments the context-limited tail classes using both the head and OOD classes via Cutmix. At test time, the sum of probabilities for the k abstention classes is used for OOD detection. In <cit.>, the authors also investigate the open-set long-tailed recognition problem, in which a long-tailed distribution also contains samples of new/unknown/open-set categories. They propose to simultaneously handle long-tailed classification, few-shot learning and open-set recognition (OOD detection) in a unified framework. 




 §.§ Active Learning for Long-Tailed Distributions


Active learning approaches iteratively select the most informative samples from underrepresented classes for annotation and training to optimize the use of limited labeled data to improve model performance over time. The authors in <cit.>  incorporate class imbalance into active learning by considering both class imbalance and labelling cost. <cit.> proposes a progressive sampling mechanism that explores active learning of OOD samples for enhancing the performance in both long-tailed classification and OOD detection. 




 §.§ Long-Tail Class Incremental Learning


To acquire novel knowledge from long-tailed class-incremental settings, <cit.> proposes to use sub-protypes to discover and store data distributions for sub-tasks with long-tailed distributions, and adopts the knowledge distillation technique to avoid catastrophic forgetting when training a new task. 




 §.§ Long-tailed  Domain Generalization


Domain generalization aims to learn from multiple training domains to generalize to previously unseen target domains. In <cit.>, the authors introduces the maximum square loss which has a linearly increasing gradient to address the long-tailed distribution issue in domain generalization.  While <cit.> focuses on single domain imbalance, <cit.> investigates  the multiple domains LTL problem which aims to learn from multi-domain imbalanced data and generalize to all domains. It proposes a loss for bounding the transferability statistics based on the contrastive loss over the positive and negative cross-domain pairs. 




 §.§ Adversarial Training Under Long-Tailed Distributions


Adversarial training is a defense strategy aimed at enhancing the robustness of machine learning models against adversarial attacks by incorporating adversarial examples into the training process. 
<cit.> investigates the adversarial vulnerability in long-tailed recognition and proposes RoBal, which includes a scale-invariant cosine classifier and a two-stage LTL framework. In <cit.>, the authors reveal that adversarial training under long-tailed distributions suffers from robust overfitting, i.e., the long-tailed model overfits to current adversarial samples. They show that simply combining Balanced Softmax Loss <cit.> and data augmentation can simultaneously alleviate robust overfitting and improve model robustness against adversarial attacks. 




 §.§ Diverse Applications with Long Tail Concerns

Long-tailed data widely exists in many real-world domains, e.g., defect identification <cit.>,  remote sensing <cit.>, nutrition ingredients analysis <cit.>, etc. While we can apply long-tailed learning techniques in these domains, such real-world applications often pose new challenges to be addressed,  which  will advance the long-tailed learning techniques in turn. 





§ CONCLUSIONS
  
In this paper, we propose a new taxonomy for long-tailed learning that divides existing techniques into eight categories, which are data balancing, neural architecture, feature enrichment, logits adjustment, loss function, bells and whistles, network optimization, and post hoc processing techniques. Based on this taxonomy, we systematically review long-tailed learning methods in each category, including the latest advances. We also examine the distinctions between imbalance learning and long-tailed learning approaches. We summarize the experimental results of representative methods, and show that there is still a very large room for further improvements. Finally, we discuss future directions in this field.

IEEEtran




 Paper Title: Non-convolutional Graph Neural Networks.


[
    Bruno S. Dandogbessi^2
    August 5, 2024
==========================





















0.3in












§ ABSTRACT 
Rethink convolution-based graph neural networks (GNN)—they characteristically suffer from limited expressiveness, over-smoothing, and over-squashing, and require specialized sparse kernels for efficient computation.
Here, we design a simple graph learning module entirely free of convolution operators, coined random walk with unifying memory (RUM) neural network, where an RNN merges the topological and semantic graph features along the random walks terminating at each node.
Relating the rich literature on RNN behavior and graph topology, we theoretically show and experimentally verify that RUM attenuates the aforementioned symptoms and is more expressive than the Weisfeiler-Lehman (WL) isomorphism test.
On a variety of node- and graph-level classification and regression tasks, RUM not only achieves competitive performance, but is also robust, memory-efficient, scalable, and faster than the simplest convolutional GNNs.




§ INTRODUCTION: CONVOLUTIONS IN GNNS

Graph neural networks (GNNs) <cit.>—neural models operating on representations of nodes (𝒱) and edges (ℰ) in a graph (denoted by 𝒢 = {𝒱, ℰ}, ℰ⊆𝒱×𝒱, with structure represented by the adjacency matrix A_ij = 1[(v_i, v_j) ∈ℰ])—have shown promises in a wide array of social and physical <cit.> modeling applications.
Most GNNs follow a convolutional scheme, where the D-dimensional node representations 𝐗∈ℝ^|V| × D are aggregated based on the structure of local neighborhoods:

    𝐗' = Â𝐗.

Here, Â displays a unique duality—the input features doubles as a compute graph.
The difference among convolutional GNN architectures, apart from the subsequent treatment of the resulting intermediate representation 𝐗', typically amounts to the choices of transformations (Â) of the original adjacency matrix (A)—the normalized Laplacian for graph convolutional networks (GCN) <cit.>, a learned, sparse stochastic matrix for graph attention networks (GAT) <cit.>, powers of the graph Laplacian for simplifying graph networks (SGN) <cit.>, and the matrix exponential thereof for graph neural diffusion (GRAND) <cit.>, to name a few. 
For all such transformations, it is easy to verify that permutation equivariance (Equation <ref>) is trivially satisfied, laying the foundations of data-efficient graph learning.
At the same time, this class of methods share common pathologies as well:



  
Limited expressiveness.
(Figure <ref>)

<cit.> groundbreakingly elucidates that GNNs cannot exceed the expressiveness of Weisfeiler-Lehman (WL) isomorphism test <cit.>.
Worse still, when the support of neighborhood multiset is uncountable, no GNN with a single aggregation function can be as expressive as the WL test <cit.>.
As such, crucial local properties of graphs meaningful in physical and social modeling, including cycle sizes (Example <ref>) and diameters (Example <ref>) <cit.>, cannot be realized by convolution-based GNNs.



  
Over-smoothing. 
(Figure <ref>)

As one repeats the convolution (or Laplacian smoothing) operation (Equation <ref>), sandwiched by linear and nonlinear transformations, the inter-node dissimilarity, measured by Dirichlet energy, 

    ℰ(𝐗) = 1/N∑_(u, v) ∈ℰ_𝒢 || 𝐗_u - 𝐗_𝐯 ||^2,

will decrease exponentially as the number of message-passing steps l increases <cit.>, 
ℰ(𝐗^(l))
≤
C_1 exp(-C_2 l)
with some constants C_1, 2,
resulting in node representations only dependent upon the topology, but not the initial embedding.



  
Over-squashing. 
(Figure <ref>)

As the number of Laplacian smoothing grows, the reception field of GNNs increases exponentially, while the dimension of node representation, and thereby the possible combinations of neighborhood environment, stays unchanged <cit.>.
Quantitatively, <cit.> quantifies this insight using the inter-node Jacobian and relates it to the powers of the adjacency matrix through sensitivity analysis:

    | ∂𝐗_v ^(l+1)/∂𝐗_u^(0) |
    ≤
    | ∇ϕ |^(l+1) (Â^l+1)_uv,

where ϕ is the node-wise update function, whose Jacobian is typically diminishing.
If this Equation <ref> converges to zero, the node representation is agnostic to the changes happening l edges away, making convolutional GNNs difficult to learn long-range dependencies.



  
Main contributions: Non-convolutional GNNs as a joint remedy.
In this paper, we propose a variant of GNN that does not engage the convolution operator (Equation <ref>) at all.[Code at: <https://anonymous.4open.science/r/rum-834D/>]
Specifically, we stochastically sample a random walk terminating at each node and use the anonymous experiment associated with the random walk to describe its topological environment.
This is combined with the semantic representations along the walk and fed into a recurrent neural network layer <cit.> to form the node embedding, which we call the unifying memory.
We theoretically (Section <ref>) and experimentally (Section <ref>) show that the resulting model, termed random walk with unifying memory (RUM) relieves the aforementioned symptoms and offers a compelling alternative to the popular convolution-based GNNs.





§ RELATED WORKS: WAYS TO WALK ON A GRAPH




  
Walk-based GNNs.
RAW-GNN (<cit.>, compared and outperformed in Table <ref>) also proposes walk-based representations for representing node neighborhoods, which resembles our model without the anonymous experiment (ω_u in Equation <ref>).
CRaWl (<cit.>, outperformed in Table <ref>) also incorporates a similar structural encoding for random walk-generated subgraphs to feed into an iterative 1-dimensional convolutional network.
AWE (<cit.>, Table <ref>) and <cit.>, like ours, use anonymous experiments for graph-level unsupervised learning and temporal graph learning, respectively.
More elaborately, <cit.> and AgentNet (<cit.>, Table <ref>) use agent-based learning on random walks and paths on graphs.



  
Random walk kernel GNNs.
In a closely related line of research, RWK (<cit.>, Table <ref>) employs the reproducing kernel Hilbert space representations in a neural network for graph modeling and GSN (<cit.>, Table <ref>) counts the explicitly enumerated subgraphs to represent graphs.
The subgraph counting techniques intrinsically require prior knowledge about the input graph of a predefined set of node and edge sets.
For these works, superior expressiveness has been routinely argued, though usually limited to a few special cases where the WL test fails whereas they do not, and often within the unlabelled graphs only.

More importantly, focusing mostly on expressiveness, no aforementioned walk-based or random walk kernel-based GNNs address the issue of over-smoothing and over-squashing in GNNs.
Some of these works are also de facto convolutional, as the random walks are only incorporated as features in the message-passing scheme.
Interestingly, most of these works are either not applicable to, or have not been tested on, node-level tasks.
In the experimental Section <ref>, we show that RUM not only outperforms these baselines on most graph-level tasks (Table <ref>) but also competes with a wide array of state-of-the-art convolutional GNNs on node-level tests (Table <ref>).
Moreover, random walk kernel-based architectures, which explicitly enumerate random walk-generated subgraphs are typically much slower than WL-equivalent convolutional GNNs, whereas RUM is faster than even the simplest variation of convolutional GNN (Figure <ref>).



  
State-of-the-art methods to alleviate oversmoothing.

Stochastic regularization. 
DropEdge (<cit.>, Figure <ref>) regularizes the smoothing of the node representations by randomly disconnecting edges.
Its associated Dirichlet energy indeed decreases slower, though eventually still diminishes as the number of layers increases.
Graph rewiring.
<cit.> and GPR-GNN (<cit.>, Appendix Table <ref>) rewire the graph using personalized page rank algorithm <cit.> and generalized page rank on graphs, respectively.
Similar to JKNet <cit.>, they mitigate over-smoothing by allowing direct message passing between faraway nodes.
Constant-energy methods.
<cit.> constrain the pair-wise distance or Dirichlet energy among graphs to be constant.
Nevertheless, the non-decreasing energy does not necessarily translate to better performance, as they sometimes come with the sacrifice of expressiveness, as argued in <cit.>.
Residual GNNs.
Residual connection <cit.> can naturally be added to the GNN architecture, such as GCNII (<cit.>, Table <ref>), to restraint activation to be similar to the input to allow deep networks.
They however can make the model less robust to perturbations in the input.
In sum, these works have similar, if not compromised expressiveness compared to a barebone GCN.









§ ARCHITECTURE: COMBINING TOPOLOGIC AND SEMANTIC TRAJECTORIES OF WALKS




  
Random walks on graphs. 
An unbiased random walk w on a graph 𝒢 is a sequence of nodes w = (v_0, v_1, …) with landing probability:

    P(v_j | (v_0, …, v_i-1)) = 
    1[(v_i, v_j) ∈ℰ_𝒢] / D(v_i),

where D(v_i) = ∑ A_ij is the degree of the node v_i.
Walks originating from or terminating at any given node v can thus be easily generated using this Markov chain.
We record the trajectory of embeddings associated with the walk as
ω_x(w) = (𝐗_i) = (𝐗_0, 𝐗_1, …, 𝐗_l).
In this paper, we only consider finite-long l-step random walk |w| = l ∈ℤ^+.

In our implementation, the random walks are sampled ad hoc during each training and inference step directly on GPU using Deep Graph Library <cit.> (see Appendix Section <ref>).

Moreover, the walk considered here is not necessarily a path, as repeated traversal of the same node v_i = v_j, i ≠ j is not only permitted, but also crucial to effective topological representation, as discussed below.




  
Anonymous experiment.
We use a function describing the topological environment of a walk, termed anonymous experiment <cit.>,  ω_u(w): ℝ^l→ℝ^l  that records the first unique occurrence of a node in a walk (Appendix Algorithm <ref>).
To put it plainly, we label a node as the number of unique nodes insofar traversed in a walk if the node has not been traversed, and reuse the label otherwise.
Practically, this can be implemented using any tensor-accelerating framework in one line ( is the node sequence of a walk) and trivially parallelized [
Note that this particular implementation introduces an intermediate 𝒪(l^2) complexity term, though it is empirically faster than the linear-complexity naive implementation, since only integer indices are involved and thus the footprint is negligible.

]:




  
Unifying memory: combining semantic and topological representations.
Given any walk w, we now have two sequences ω_x(w) and ω_u(w) describing the semantic and topological (as we shall show in the following sections) features of the walk.
We project such sequential representations onto a latent dimension to combine them (illustrated in Table <ref>):

    h(w) = f(ϕ_x(ω_x(w)), ϕ_u(ω_u(w))),

where ϕ_x: ℝ^l × D→ℝ^D_x maps the sequence of semantic embeddings generated by a l-step walk to a fixed D_x-dimensional latent space,
ϕ_u: ℝ^l→ℝ^D_u maps the indicies sequence to another latent space D_u,
and f: ℝ^D_x⊕ℝ^D_u→ℝ^D combines them.
We call Equation <ref> the unifying memory of a random walk.
Subsequently, the node representations can also be formed as the average representations of l-step (l being a hyperparameter) walks terminating (for the sake of gradient aggregation) at that node:

    ψ(v) = ∑_{w}, |w|=l, w_l = v p(w) h(w),

which can be stochastically sampled with unbiased Monte Carlo gradient and used in downstream tasks as such node classification and regression.
We note that this is the only time we perform SUM or MEAN operations.
Unlike other GNNs incorporating random walk-generated features (which are sometimes still convolutional and iterative), we do not iteratively pool representations within local neighborhoods.
The likelihood of the data written as:

    P(y|𝒢, 𝐗) = ∑_{w}, |w|=l, w_l = v p(w) p(y|𝒢, 𝐗, w)

The node representation can be summed 

    Ψ(𝒢) = ∑_v ∈𝒱⊆𝒢ψ(v)

to form global representations for graph classification and regression.
We call ψ in Equation <ref> and Ψ in Equation <ref> the node and graph output representations of RUM.



  
Layer choices. 
Obvious choices to model f include a feed-forward neural network after concatenation, and ϕ_x, ϕ_u recurrent neural networks (RNNs).

This implies that, different from most convolutional GNNs, parameter sharing is natural and the number of parameters is going to stay constant as the model incorporates a larger neighborhood.
Compared to dot product-based, transformer-like modules <cit.>, RNNs not only have linear complexity (see detailed discussions below) w.r.t. the sequence length but also naturally encodes the inductive bias that nodes closer to the origin have stronger impact on the representation. 

The gated recurrent unit (GRU)<cit.> variant is used everywhere in this paper.
Additional regularizations are described in Appendix Section <ref>.



  
Runtime complexity. 
To generate random walks for one node has the runtime complexity of 𝒪(1), and for a graph 𝒪(| 𝒱 |), where |𝒱| is the number of nodes in a graph 𝒢 = {𝒱, ℰ}.
To calculate the anonymous experiment, as shown in Appendix Algorithm <ref>, has 𝒪(1) complexity (also see Footnote <ref>).
If we use linear-complexity models, such as RNNs, to model ϕ_x, ϕ_u, the overall complexity is 𝒪(|𝒱|lkD) where l is the length of the random walk, k the samples used to estimate Equation <ref>, and D the latent size of the model (assumed uniform).
Note that different from convolutional GNNs, RUM does not depend on the number of edges | ℰ | (which is usually much larger than |𝒱|) for runtime complexity, and is therefore agnostic to the sparsity of the graph.
See Figure <ref> for an empirical verification of the time complexity.
In Appendix Table <ref>, we show, on a large graph, the overhead introduced by generating random walks and computing anonymous experiments accounts for roughly 1/1500 of the memory footprint and 1/8 of the wall time.



  
Mini-batches.

RUM is naturally compatible with mini-batching.
For convolutional GNNs, large graphs that do not fit into the GPU memory have traditionally been a challenge, as all neighbors are required to be present and boundary conditions are hard to define <cit.>.
RUM, on the other hand, can be inherently applied on subsets of nodes of a large graph without any alteration in the algorithm—the random walks can be generated on a per-node basis, and the FOR loop in Algorithm <ref> can be executed sequentially, in parallel, or on subsets.

Empirically, in Appendix Table <ref>, RUM can be naturally scaled to huge graphs.



§ THEORY: RUM AS A JOINT REMEDY.


We have insofar designed a new graph learning framework—convolution-free graph neural networks (GNNs) that cleanly represent the semantic (ω_x) and topological (ω_u) features of graph-structured data before unifying them.
First, we state that RUM is permutation equivariant,

For any permutation matrix P, we have

    P𝐗_v(𝒢) = 𝐗_v(P(𝒢)),


which sets the foundation for the data-efficient modeling of graphs.
Next, we theoretically demonstrate that this formulation jointly remedies the common pathologies of the convolution-based GNNs by showing that:
(a) the topological representation ω_u is more expressive than convolutional-GNNs in distinguishing non-isomorphic graphs;
(b) the semantic representation ω_x no longer suffers from over-smoothing and over-squashing.



 §.§ RUM is more expressive than convolutional GNNs.

For the sake of theoretical arguments in this section, we assume that in Equation <ref>:


ϕ_x, ϕ_u, f are universal and injective.



Graph 𝒢 discussed in this section is always connected, unweighted, and undirected.

Assumption <ref> is easy to satisfy for feed-forward assuneural networks <cit.> and RNNs <cit.>. 
Composing injective functions, we remark that h(w) is also injective w.r.t. ω_x(w) and ω_u(w);
despite of Assumption <ref>, our analysis can be extended to disjointed graphs by restricting the analysis to the connected regions in a graph.
Under such assumptions, we show, in Remark <ref> (deferred to the Appendix), that ψ is injective, meaning that nodes with different random walks will have different distributions of representations ψ(v_1) ≠ψ(v_2).
We also refer the readers to the Theorem 1 in <cit.> for a discussion on the representation power of anonymous experiments on unlabelled graphs.
Combining with the semantic representations and promoting the argument from a node level to a graph level, we arrive at:


Up to the Reconstruction Conjecture <cit.>, RUM with sufficiently long l-step random walks can distinguish non-isomorphic graphs satisfying Assumption <ref>.

The main idea of the proof of Theorem <ref> (in Appendix Section <ref>) involves explicitly enumerating all possible non-isomorphic structures for graphs with 3 nodes and showing, by induction, that if the theorem stands for graph of N-1 size it also holds for N-sized graphs.
We also show in Appendix Section <ref> that a number of key graph properties such as cycle size (Example <ref>) and radius (Example <ref>) that convolutional GNNs struggle <cit.> to learn can be analytically expressed using ω_u.
As these are solely functions of ω_x, they can be approximated arbitrarily well by universal approximators.
These examples are special cases of the finding that RUM is stricly more expressive than Weisfeiler-Lehman isomorphism test <cit.>:

Two graphs with N nodes 𝒢_1, 𝒢_2 labeled as non-isomorphic by Weisfeiler-Lehman isomorphism test is the necessary, but not sufficient condition that the representation resulting from RUM with walk length N+1 are also different.

    Ψ(𝒢_1) ≠Ψ(𝒢_2)


Thus, due to <cit.>, RUM is also more expressive than convolutional GNNs in distinguishing non-isomorphic graphs.
This also confirms the intuition that RUM with longer walks are more expressive.
(See Figure <ref> on an empirical evaluation.)
The proof of this corollary is straightforward to sketch—even if we only employ the embedding trajectory ω_x, it would have the effect of performing the function ϕ_x in Equation <ref> on each traversal of the WL expanded trees.



 §.§ RUM alleviates over-smoothing and over-squashing


Over-smoothing refers to the phenomenon where the node dissimilarity (e.g., measured by Dirichlet energy in Equation <ref>) decreases exponentially and approaches zero with the repeated rounds of message passing.

<cit.> relates Dirichlet energy directly with the convolutional operator:
*dirichlet-convLemma 3.1 from <cit.>


    ℰ((1 - Δ̃)𝐗)
    ≤
    (1 - λ)^2 ℰ(𝐗)

where λ is the smallest non-zero eigenvalue of Δ̃, the normalized Laplacian of a graph.

Free of convolution operators, it seems only natural that RUM does not suffer from this symptom (Figure <ref>).
We now formalize this intuition by first restricting ourselves to a class of non-contractive mappings for f in Definition <ref>.


A map f is non-contractive on region Ω if ∃α∈ [1, +∞) such that |f(x) - f(y)| ≥α  || x - y ||, ∀ x, y ∈Ω.

A line of fruitful research has been focusing on designing non-contractive RNNs <cit.>, and to be non-contractive is intimately linked with desirable properties such as preserving the long-range information content and non-vanishing gradients.
From this definition, it is easy to show that, for each sampled random walk in Equation <ref>, the Dirichlet energy is greater than its input.
One only needs to verify that the integration in Equation <ref> does not change this to arrive at:


If ϕ_x, f are non-contractive w.r.t. all elements in the sequence, the expected Dirichlet energy of the corresponding RUM node representation in Equation <ref> is greater than its initial value

    E(ℰ(ψ(𝐗))) ≥ℰ(𝐗).



This implies that the expectation of Dirichlet energy does not diminish even when l → +∞, as it is bounded by the Dirichlet energy of the initial node representation, which is consistent with the trend shown in Figure <ref>, although the GRU is used out-of-box without constraining it to be explicitly non-contractive.




  
RUM alleviates over-squashing is deferred to Appenxix Section <ref>, where we verify that the inter-node Jacobian | ∂𝐗_v ^(l+1) / ∂𝐗_u^(0)| decays slower as the distance between u, v grows vis-à-vis the convolutional counterparts.
Briefly, although RUM does not address the information bottleneck with exponentially growing reception field (the 1 / (Â^l+1)_uv term in Equation <ref>), it nevertheless can have a non-vanishing (nor exploding) gradient from the aggregation function (| ∇ϕ_x |). 



§ EXPERIMENTS


On a wide array of real-world node- and graph-level tasks, we benchmark the performance of RUM to show its utility in social and physical modeling.
Next, to thoroughly examine the performance of RUM, we challenge it with carefully designed illustrative experiments.
Specifically, we ask the following questions in this section, with Q1, Q2, and Q3 already theoretically answered in Section <ref>:
Q1:
Is RUM more expressive than convolutional GNNs?
Q2:
Does RUM alleviate over-smoothing?
Q3:
Does RUM alleviate over-squashing?
Q4:
Is RUM slower with convolutional GNNs?
Q5:
Is RUM robust?
Q6:
How does RUM scale up to huge graphs?
Q7:
What components of RUM are contributing most to the performance of RUM?



  
Real-world benchmark performance.
For node classification, we benchmark our model on the popular Planetoid citation datasets <cit.>, as well as the coauthor <cit.> and co-purchase <cit.> datasets common in social modeling.
Additionally, we hypothesize that RUM, without the smoothing operator, will perform competitively on heterophilic datasets <cit.>—we test this hypothesis.
For graph classification, we benchmark on the popular TU dataset <cit.>.
We also test the graph regression performance on molecular datasets in MoleculeNet <cit.> and Open Graph Benchmark <cit.>.
In sum, RUM almost always outperforms, is within the standard deviation of, the state-of-the-art architectures, as shown in Tables <ref>, <ref>, <ref>, <ref>, as well as in Tables <ref>, <ref>, <ref>, <ref> moved to the Appendix due to space constraint.








  
On sparsity: the subpar performance on the Computer dataset.
The most noticeable exception to the good performance of RUM is that on the Computer co-purchase <cit.> dataset, where RUM is outperformed even by GCN and GAT.
This dataset is very dense with an average node degree (| ℰ | / | 𝒱|) of 18.36, the highest among all datasets used in this paper.
As the variance of the node embedding (Equation <ref>) scales with the average degree, we hypothesize that dense graphs with very high average node degrees would have high-variance representations from RUM.

On the other hand, RUM outperforms all models surveyed in two large-scale benchmark studies on molecular learning, GAUCHE <cit.> and MoleculeNet <cit.>.
The atoms in the molecules always have a degree of 2 ∼ 4 with intricate subgraph structures like small rings.
This suggests the utility of unifying memory in chemical graphs and furthermore chemical and physical modeling.



























  
Graph isomorphism testing (Q1).
Having illustrated in Section <ref> that RUM can distinguish non-isomorphic graphs, we experimentally test this insight on the popular Circular Skip Link dataset <cit.>.
Containing 4-regular graph with edges connected to form a cycle and containing skip-links between nodes, this dataset is artificially synthesized in <cit.> to create an especially challenging task for GNNs.
As shown in Appendix Table <ref>, all convolutional GNNs fail to perform better than a constant baseline (there are 10 classes uniformly distributed).
3WLGNN <cit.>, a higher-order GNN of at least 𝒪(2) complexity that operates on explicitly enumerated triplets of graphs, can distinguish these highly similar 4-regular graphs by comparing subgraphs.
RUM, with 𝒪(| 𝒩 | ) linear complexity, achieves similarly high accuracy.
One can think of RUM as a stochastic approximation of the higher-order GNN, with all of its explicitly enumerated subgraphs being identified by RUM with a probability that decreases with the complexity of the subgraph.




  
Effects of walk lengths and number of samples on performance (Q2, Q3).
Having studied the relationship between inference speed and the walk lengths and number of samples, we furthermore study its impact on performance.
Using Cora <cit.> citation graph and vary the walk lengths and number of samples from 1 to 9, where the performance of RUM improves as more samples are taken and longer walks are employed, though more than 4 samples and walks longer than L > 4 yield qualitatively satisfactory results;
this empirical finding has guided our hyperparameter design.
In Figure <ref>, we also compare the Dirichlet energy of RUM-generated layer representations with not only baselines GCN <cit.> and GAT <cit.>, but also strategies to alleviate over-smoothing discussed in Section <ref>, namely residual connection and stochastic regularization <cit.>, and show that when L gets large, only RUM can maintain Dirichlet energy.
Traditionally, since <cit.> (see its Figure 5 compared to Figure <ref>), the best performance on Cora graph was found with 2 or 3 message-passing rounds, since mostly local interactions are dominating the classification, and more rounds of message-passing almost always lead to worse performance.
As theoretically demonstrated in Section <ref>, RUM is not as affected by these symptoms.
Thus, RUM is especially appropriate for modeling long-range interactions in graphs without sacrificing local representation power.



  
Long-range neighborhood matching (Q3).
To verify that RUM indeed alleviates over-squashing (Section <ref>), in Figure <ref>, we adopt the tree neighborhood matching synthetic task in <cit.> where binary tree graphs are proposed with the label and the attributes of the root matching a faraway leave.
The full discussion is moved to the Appendix Section <ref>.



  
Speed (Q4).
Though both have linear runtime complexity (See Section <ref>), intuitively, it might seem that RUM would be slower than convolution-based GCN due to the requirement of multiple random walk samples.
This is indeed true for CPU implementations shown in Figure <ref> left.
When equipped with GPUs (specs in Appendix Section <ref>), however, RUM is significantly faster than even the simplest convolutional GNN—GCN <cit.>.
It is worth mentioning that the GCN and GAT <cit.> results were harvested using the heavily optimized Deep Graph Library <cit.> sparse implementation whereas RUM is implemented naïvely in PyTorch <cit.>, though the popular GRU component <cit.> have already undergone CUDA-specific optimization.






  
Robustness to attacks (Q5).
With the stochasticity afforded by the random walks, it is natural to suspect RUM to be robust.
We adopt the robustness test from <cit.> and attack by randomly adding fake edges to the Cora <cit.> graph and record the test set performance in Figure <ref>.
Indeed, RUM is much more robust than traditional convolutional GNNs including GCN <cit.> and GAT <cit.> and is even slightly more robust than the convolutional GNN specially designed for robustness <cit.>, with the performance only decreased less than 10% with 10% fake edges added.



  
Scaling to huge graphs (Q6) and Ablation study (Q7)
are deferred to Appendix Section <ref>.



§ CONCLUSIONS


We design an innovative GNN that uses an RNN to unify the semantic and topological representations along stochastically sampled random walks, termed random walk with unifying memory (RUM) neural networks.
Free of the convolutional operators, our methodology does not suffer from symptoms characteristic of Laplacian smoothing, including limited expressiveness, over-smoothing, and over-squashing.
Most notably, our method is more expressive than the Weisfeiler-Lehman isomorphism test and can distinguish all non-isomorphic graphs up to the reconstruction conjecture.
Thus, it is more expressive than all of convolutional GNNs equivalent to the WL test, as we demonstrate theoretically in Section <ref>.
RUM is significantly faster on GPUs than even the simplest convolutional GNNs (Section <ref>) and shows superior performance across a plethora of node- and graph-level benchmarks.



  
Limitations. 
Very dense graphs.
As evidenced by the underwhelming performance of the Computer <cit.> dataset and discussed in Section <ref>, RUM might suffer from high variance with very dense graphs (average degree over 15).
Tottering. 
In our implementation, we have not ruled out the 2-cycles from the random walks, as that would require specialized implementation for walk generation.
This, however, would reduce the average information content in a fixed-length random walk (known as tottering <cit.>).
We plan to characterize the effect of excluding these walks.
Biased walk. 
Here, we have only considered unbiased random walk, whereas biased random walk might display more intriguing properties as they tend to explore faraway neighborhoods more effectively <cit.>.
Directed graphs. 
Since we have only developed RUM for undirected graph (random walk up to a random length is not guaranteed to exist for directed graphs), we would have to symmetrize the graph to perform on directed graphs (such as the heterophilic datasets <cit.>);
this create additional information loss and complexity.



  
Future directions.
Theoretical. 
We plan to expand our theoretical framework to account for the change in layer width and depth to derive analytical estimates for realizing key graph properties.
Applications. 
Random walks are intrinsically applicable to uncertainty-aware learning.
We plan to incorporate the uncertainty naturally afforded by the model to design active learning models.
On the other hand, the physics-based graph modeling field is also heavily dominated by convolutional GNNs.
Inspired by the superior performance of RUM on chemical datasets, we plan to apply our method to the equivariant modeling of n-body physical systems <cit.>.



  
Impact statement. 
We here present a powerful, robust, and efficient learning algorithm on graphs.
Used appropriately, this algorithm might advance the modeling of social <cit.> and physical <cit.> systems, which can oftentimes modeled as graphs.
As with all graph machine learning methods, negative implications may be possible if used in the design of explosives, toxins, chemical weapons, and overly addictive recreational narcotics.





  
Disclosures. 
YW has limited financial interests in Flagship Pioneering, Inc. and its subsidiaries.



  
Funding. 
YW acknowledges support from the Schmidt Science Fellowship, in partnership with the Rhodes Trust, and the Simons Center for Computational Physical Chemistry at New York University.





§ EXPERIMENTAL DETAILS




  
Code availability.
All architectures, as well as scripts to execute the experiment, are distributed open-source under MIT license at <https://anonymous.4open.science/r/rum-834D/>.
Core dependencies of our package include PyTorch <cit.> and Deep Graph Library <cit.>.



  
Hyperparameters.
All models are optimized using Adam <cit.> optimizer and SiLU <cit.> activation functions.
4 random walk samples are drawn everywhere unless specified.
Other hyperparameters—learning rate (10^-5∼ 10^-2), hidden dimension (32 ∼ 64), L2 regularization strength (10^-8∼ 10^-2), walk length (3∼ 16), temperature for ℒ_𝚌𝚘𝚗𝚜𝚒𝚜𝚝𝚎𝚗𝚌𝚢 (0 ∼ 1), coefficient for ℒ_𝚌𝚘𝚗𝚜𝚒𝚜𝚝𝚎𝚗𝚌𝚢 (0 ∼ 1), coefficient for ℒ_𝚜𝚎𝚕𝚏, and dropout probability—are tuned using the Ray platform <cit.> with the default Ax <cit.> search algorithm with 1000 trails or 24 hours tuning budget on a Nvidia A100® GPU.



§ ADDITIONAL TECHNICAL DETAILS




 §.§ Self-supervised regularization. 


The stochasticity encoded in our model naturally affords it with some level of regularization.
Apart from using the consistency loss (ℒ_𝚌𝚘𝚗𝚜𝚒𝚜𝚝𝚎𝚗𝚌𝚢) used in <cit.> for classifications, we further regularize the model by using the RNNs in ϕ_x to predict the semantic representation of the next node on the walk given ω_u and jointly maximize this likelihood:

    ω̂_x_i+1 = g({ω_x_1, ω_x_2, …, ω_x_i}, ω_u | θ);
    ℒ_𝚜𝚎𝚕𝚏(θ) = -log P(ω̂_x_i+1 | θ),

where g(· | θ) is modeled as the sequence output of the RNN ϕ_x in Equation <ref>.
The total loss is modeled as a combination of:

    ℒ(θ) =  
    -log P(y|𝒢, 𝐗, θ)
    + ℒ_𝚜𝚎𝚕𝚏
    + ℒ_𝚌𝚘𝚗𝚜𝚒𝚜𝚝𝚎𝚗𝚌𝚢




 §.§ RUM attenuates over-squashing


Similarly, we can show that if the composing neural networks defy the vanishing gradient problem (w.r.t. the input) <cit.>, the sensitivity analysis in Equation <ref> <cit.> has a lower bound for RUM.


If ϕ_x, f have lower-bounded derivatives, the inter-node Jacobian for nodes u, v separated by a shortest path of length l, RUM with walk length l also has a lower bound:

    | ∂𝐗_v ^(l)/∂𝐗_u^(0) |
    ≥
    | ∇ϕ_x | |∇ f| (Â^l)_uv,

where Â_ij = A_ij / ∑_jA_ij is the degree-normalized adjacency matrix. 

Like the upper bound in Equation <ref>, this lower bound is also controlled by the power of the (normalzied) adjacency matrix, albeit the absence of self-loop will result in a slightly looser bottleneck.
The term (Â^l)_uv corresponds to the probability of the shortest path among all possible walks as a product of inverse node degrees (see Equation <ref>).
There is no denying that the lower bound is still controlled by the power of the adjacency matrix, which corresponds to the exponentially growing receptive fields.
One can also argue that, without prior knowledge, the contribution of the sensitivity analysis by the power of the graph adjacency matrix can never be alleviated, since there are always roughly 1 / (Â^l+1)_uv (assuming uniform node degree) structurally equivalent nodes.
Nevertheless, since ϕ_x is not necessarily an iterative function, we alleviate the over-squashing problem by eliminating the power of the update function gradient term.

Now, we plug in the layer choices of ϕ_x—a GRU <cit.> unit.
Its success, just like that of long short-term memory (LSTM) <cit.>, can be attributed to the near-linear functional form of the long-range gradient.
The term |∇ϕ_x| is controlled by a sequence of sigmoidal update gates, which can be optimized to approach 1 (fully open).
If we ignore the gradient contribution of 𝐗_u^0 to these gates, the non-linear activation function has only been applied exactly once on 𝐗_u^0;
therefore, the gradient | ∂𝐗_v ^(l+1) / ∂𝐗_u^(0)| is neither rapidly vanishing nor exploding.



 §.§ Long-range neighborhood matching (Q3).


Once identifying the target leaf, this task seems trivial;
nonetheless, this piece of information needs to be passed through layers of aggregation and non-linear update and is usually lost in the convolution.
Since, on this binary tree, the receptive field grows exponentially, <cit.> argues that there is a theoretical lower boundary for the layer width D for the convolutional GNN to be able to encode all possible combinations of leaves, which is 2^32D for single-precision floating point ().
This corresponds to the structural Â^l term in Equation <ref> and Equation <ref>.
Evidently, when D=32, as is the adopted experimental setting in  Figure <ref>, this limit is far from being hit.
So we hypothesize that the reason why convolutional GNNs cannot overfit the training set is because of the limitation of the functional forms, which are remedied by RUM, which shows 100% accuracy up to tree depth or problem radius r=5, and a relatively moderate decrease afterward.

Note that when the problem radius exceeds r=7, RUM's performance is not significantly different from the convolutional counterparts.




 §.§ Scalaing to large graphs (Q6).

In Appendix Table <ref>, we apply RUM on an ultra-large graph , that cannot fit easily on a single GPU, and compare RUM with architectures specifically designed for large graphs <cit.>.



 §.§ Ablation study (Q7). 



In Table <ref>, we conduct a brief ablation study where we delete, one by one, the components introduced in Section <ref>.
ω_u = 0 and ω_x = 0 refer to the deletion of the topological and semantic representations of walks, respectively.
Neglecting topological information results in a moderate decrease in performance, whereas neglecting semantic representation is more detrimental.
The ω_u = 0 also resembles <cit.> albeit with different walk-wise aggregation.
This offers a qualitative comparison between our work and <cit.> as no source code was released for this package so no rigorous comparison was possible.
We also see that the regularization methods are helpful to the performance, with self-supervision being more crucial.
We attribute this effect to firstly the relative simplicity of the Cora classification task, and secondly the flexibility of the (overparametrized) RNNs.




§ ADDITIONAL RESULTS






For two nodes v_1, v_2 with distribution of random walks terminating at v_1, v_2 not equal in distribution p(ω_u(w_1)) ≠ p(ω_u(w_2)) or p(ω_x(w_1)) ≠ p(ω_x(w_2)), the node representations in Equation <ref> are also different
ψ(v_1) ≠ψ(v_2).

One way to construct h function in Equation <ref> is to have h(w) positive only where p(ω_u(w_1)) > p(ω_u(w_2));
the same thing can be argued for ω_x.
In other words, one only needs to prove two walks terminating at two nodes p(ω_u(w_1)) ≠ p(ω_u(w_2)) or p(ω_x(w_1)) ≠ p(ω_x(w_2)) are not equal in distribution to verify that RUM can distinguish two nodes.
Conversely, we can also show that p(ω_u(w_1)) = p(ω_u(w_2)) implies that v_1, v_2 are isomorphic without labels—




we refer the readers to the Theorem 1 in <cit.> for a discussion on reconstructing unlabelled graphs using anonymous experiments.












§ MISSING MATHEMATICAL ARGUMENTS.



 §.§ Examples of Theorem <ref>


[Cycle detection.]

A k-cycle 𝒞_k is a subgraph of 𝒢 consisting of k nodes, each with degree two.
The existence of k-cycle can be determined by:

    1(𝒞_k ⊆𝒢)
    = 1[P(
    ω_x_j+1 = ω_x_0,
    ω_x_i≠ω_x_j, ∀ i < j
    )>0]




[Diameter.]

The diameter, δ_𝒢 of graph 𝒢 which equals the length of the longest shortes path in 𝒢, can be expressed as

    δ_𝒢 = 
    l=| ω_x |, ω_x_i≠ω_x_j, ∀ i ≠ jargmax
    | ω_x |







 §.§ Proof of Theorem <ref>



First, we enumerate all possible unlabelled graphs with three nodes satisfying Assumption <ref>—one with two edges, one with three edges.
(Note that there is only one non-isomorphic graph with two nodes.)
Now we consider random walks of length l=3, where

    P(ω_u_3 = ω_u_0) > 0

only stands for the graph with three edges, but not with two edges, just like Example <ref>.

Furthermore, we can also distinguish between the 2-degree node and the 1-degree node in the graph with 2 edges and 3 nodes simply by verifying that

    P(ω_u_1≠ω_u_3) > 0

only stands when v_2 is the 2-degree node.

Moving on to the labeled 3-node graph case, we can reduce the problem to investigate whether RUM can distinguish 3-node graphs that are isomorphic when unlabeled, but non-isomorphic when labeled.
For the three-edged graph, ω_x uniformly samples the labels of three nodes.
For the two-edged graph, suppose the node labels are A, B, C, and we start from the node with 2 degrees B (with nodes bearing A and C labels locally, structurally isomorphic), 

    P(B|ω_x(w_t)) = 
    1, t = 2n, n∈ℕ,
    
    0, t = 2n + 1, n ∈ℕ
    
    P(A|ω_x(w_t)) = P(C|ω_x(w_t)) = 
    0, t = 2n, n∈ℕ,
    
    1/2, t = 2n + 1, n ∈ℕ

If graphs have the same ω_x, they have the same B and the same or swapped A, C.
As such, we have proven Theorem <ref> for graphs with 3 nodes.

Now we prove that Theorem <ref> stand for graphs of N nodes, they also stand for graphs of N+1 nodes, the Reconstruction Conjecture <cit.>

Suppose we have two non-isomorphic graphs with N+1 nodes 𝒢_1 and 𝒢_2 with the same RUM embedding Ψ_1 = Ψ_2.
We enumerate all N + 1 subgraphs with each node deleted for each of these two graphs.
By the Reconstruction Conjecture, at least one pair of subgraphs are non-isomorphic.
For this pair, suppose the deleted vertex is v (ruling out the trivial case where the label or connectivity of v is different for these two graphs), and two remaining subgraphs 𝒢_1^\ v, 𝒢_2^\ v;
since ϕ_x, ϕ_u, f are injective, Ψ_1 = Ψ_2 implies ω_u, ω_x are equal in distribution for 𝒢_1, 𝒢_2.
As such, the walk distribution

    P(ω_x(w), ω_u(w) | w_0 = v, w_i ≠ v, i > 0) 
    
    = P(ω_x(w_0), ω_u(w_0) | w_0 = v) P(ω_x(w_1…), ω_u(w_1…) | ω_x(w_1…), ω_u(w_1…), w_i ≠ v, i > 0)

are also equal in distribution for 𝒢_1, 𝒢_2.

If there is a link between v and the nodes in 𝒢_1, 𝒢_2, we assign a new label to contain both the old label and the connection.
As such, if the second term is not equal in distribution for 𝒢_1, 𝒢_2, we would have 

    Ψ(𝒢_1^\ v) ≠Ψ𝒢_2^\ v),

which is in conflict with the assumption that Theorem <ref> stands for graphs with N nodes.




 §.§ Proof of Lemma <ref>


By the definition of Dirichlet energy (Equation <ref>) and non-contractive mappings (Definition <ref>),

    ℰ(ψ(𝐗))
            = 1/N∑_u, v ∈ℰ_𝒢 || ψ(ω_x(𝐗_u)) - ψ(ω_x(𝐗_v)) || ^2
        = 1/N∑_u, v ∈ℰ_𝒢 || f(ϕ_x(ω_x(𝐗_u))) - f(ϕ_x(ω_x(𝐗_v))) || ^2
       ≥1/N∑_u, v ∈ℰ_𝒢 || ϕ_x(ω_x(𝐗_u)) - ϕ_x(ω_x(𝐗_v)) || ^2
        = 1/N∑_u, v ∈ℰ_𝒢 || ϕ_x(∑_{w}, |w|=l, w_l = u p(w)𝐗_u, u ∈ w) - ϕ_x(∑_{w}, |w|=l, w_l = v p(w)𝐗_v, v ∈ w) || ^2
       ≥1/N∑_u, v ∈ℰ_𝒢 || (∑_{w}, |w|=l, w_l = u p(w)𝐗_u, u ∈ w) - (∑_{w}, |w|=l, w_l = v p(w)𝐗_v, v ∈ w) || ^2
       ≥ℰ(𝐗)

The last inequality is due to the fact that ϕ_x is non-contractive w.r.t. all, and therefore, the last element of the sequence, which are always 𝐗_u and 𝐗_v.

































































































































    

























    



































































 Paper Title: On the Limitations and Prospects of Machine Unlearning for Generative AI.



[
On the Limitations and Prospects of Machine Unlearning for Generative AI














equal*


Shiji Zhouthu
Lianzhe Wangthu
Jiangnan Yeic
Yongliang Wuseu
Heng Changthu





thuTsinghua University
seuSoutheast University
icImperial College London

Heng Changchangh17@tsinghua.org.cn




Machine UnLearning, Generative AI

0.3in
]










 

§ ABSTRACT 
Generative AI (GenAI), which aims to synthesize realistic and diverse data samples from latent variables or other data modalities, has achieved remarkable results in various domains, such as natural language, images, audio, and graphs. However, they also pose challenges and risks to data privacy, security, and ethics. Machine unlearning is the process of removing or weakening the influence of specific data samples or features from a trained model, without affecting its performance on other data or tasks. While machine unlearning has shown significant efficacy in traditional machine learning tasks, it is still unclear if it could help GenAI become safer and aligned with human desire. To this end, this position paper provides an in-depth discussion of the machine unlearning approaches for GenAI. Firstly, we formulate the problem of machine unlearning tasks on GenAI and introduce the background. Subsequently, we systematically examine the limitations of machine unlearning on GenAI models by focusing on the two representative branches: LLMs and image generative (diffusion) models. Finally, we provide our prospects mainly from three aspects: benchmark, evaluation metrics, and utility-unlearning trade-off, and conscientiously advocate for the future development of this field.



§ INTRODUCTION










    

















“Remembrance is a form of meeting. Forgetfulness is a form of freedom.”Kahlil Gibran (1926)



In an era marked by the burgeoning influence of Generative AI  (GenAI) <cit.>, we are rapidly progressing toward a digital future dominated by AI-generated content. This technological advancement has become a cornerstone in various domains, including natural language processing, image synthesis, audio generation, and graph-based applications. While GenAI heralds an era of innovation and efficiency, it simultaneously raises pressing concerns about data privacy, security, and ethical implications <cit.>.

The training datasets employed in GenAI often contain sensitive information encompassing private, copyrighted, or potentially harmful content <cit.>. This situation raises significant risks of sensitive data leakage <cit.>, directly conflicting with the growing legislative emphasis on the “right to be forgotten” <cit.>. Instances such as the proliferation of copyright infringement cases post the release of models like Stable Diffusion <cit.>, and The New York Times's lawsuit against OpenAI for content leakage[https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec2023.pdf], underscore the urgency of addressing these issues.

In response to these challenges, Machine unlearning <cit.> has emerged as a potentially promising solution. Machine unlearning aims to compel models to forget sensitive information, thereby fundamentally eliminating the risk of content leakage. This approach, which seeks to erase sensitive memories directly, stands in contrast to filtering-based solutions that are often susceptible to bypassing or direct attacks. Current research on Machine unlearning spans various generative models, including Large Language Models (LLMs) <cit.>, image generative models <cit.>, and multi-modal generative models <cit.>. These studies have demonstrated machine unlearning's potential in removing elements like copyrighted styles <cit.>, fictional characters <cit.>, and private data <cit.>.

However, the exploration of MU in the context of GenAI is still nascent, with several limitations hindering its full potential. As illustrated in Figure <ref>, we summarise the urgent limitations of the current machine unlearning methods on GAI into three perspectives. Firstly, as an emerging technique, machine unlearning for GenAI is still distant from achieving a level of efficacy requisite for practical applications. Secondly, the evaluation metrics currently employed in MU research are insufficiently robust and fail to capture the multifaceted impact of unlearning on generative models. Thirdly, the side effects of unlearning, including its impact on model performance, generalization, and safety, are significant concerns that must be addressed. As a basis of this position paper, we aim to raise public awareness of these limitations and provide insights that guide future research to address these problems.



In this position paper, we systematically discuss the limitations from the angles of efficacy, side effects, and measurement. Our discussion ranges from LLMs to image generative models. Based on these current inadequacies of unlearning approaches, we advocate future research to focus on three fundamental paths: the benchmarking of unlearning methods, the development of robust and complete evaluating metrics, and the investigation of the balance between utility and unlearning. We believe exploring these three paths should pave the way for further enhancing machine unlearning on GenAI. Overall, our contributions can be summarized as follows:


  * This is, to the best of our knowledge, the first attempt to comprehensively review the limitations of current research on machine unlearning in GenAI. 

  * We offer a systematic examination of the limitations in current unlearning approaches, covering a broad spectrum of models from LLMs to image generative models. This examination provides researchers with a clearer insight into the urgent problems in this area.

  * Tackling these limitations, we propose three prospects based on the examinations, aiming to chart a course for future research in machine unlearning for GenAI.


Through this paper, we seek to contribute to the growing body of knowledge in GenAI, specifically in the area of machine unlearning, and catalyze further research that addresses the critical challenges identified.



§ BACKGROUND



 §.§ Unlearning Formulation
 
Let 𝒟_tr={(x_i)}_i=1^N be the training data where x_i∈𝒳 is the training input. Suppose we have the original model by optimizing towards the training data:

    θ_0 = min_θ∈𝒦𝔼_x∼𝒟_trℒ(θ,x).

Let 𝒟_f⊆𝒟_tr be a subset of training data that is harmful to the model and needs to be forgotten, and 𝒟_r = 𝒟_tr∖𝒟_f be remaining training data of which information we want to retain. The goal of machine unlearning is to successfully unlearn 𝒟_f, and the exact unlearning is to obtain the gold standard model retrained from scratch with only 𝒟_r:

    θ^* = min_θ∈𝒦𝔼_x∼𝒟_rℒ(θ,x).

Exact unlearning can be obtained by retraining, which causes tremendous cost that is not affordable to frequently updating models. In practice, approximate unlearning aims to finetune the original model to obtain the unlearned model θ^u whose output distribution P_θ^u(·) approximates the distribution of gold standard model P_θ^*(·). Unless otherwise indicated, we only discuss the approximate unlearning in the next context in this paper.





 §.§ Unlearning for LLM

Machine unlearning for LLMs is a crucial technique to align LLMs with human preferences and values and to ensure their ethical and responsible use. The existing methods for machine unlearning for LLMs can be broadly classified into:



  
Parameter Optimization Methods. These methods update the model parameters by minimizing a loss function that penalizes the undesirable outputs or behaviors of the model.  <cit.> proposed a gradient-based unlearning method that minimizes the cross-entropy loss between the model outputs and a predefined target distribution for the data samples that need to be unlearned. They applied their method to three scenarios of unlearning for LLMs: removing harmful responses, erasing copyright-protected content, and eliminating hallucinations.



  
Parameter Merging Methods. These methods reduce the model size and complexity by merging or pruning the model parameters that are most affected by the data samples that need to be unlearned. <cit.> proposed the concept of a task vector, which, through arithmetic operations like negation or addition between task vectors, can selectively modify the model’s output with minimal impact on other model behaviors.



  
In-context Learning Methods. These methods modify the model inputs or outputs by adding or removing certain tokens or features that indicate the data samples or modalities that need to be unlearned.  To unlearn a particular instance in the forget set, <cit.> provided the instance alongside a flipped label and additional correctly labeled instances which are prepended as inputs to the LLM at inference time. These contexts are shown to be able to effectively remove specific information in given instances while maintaining comparable performance with other unlearning methods that need to access the LLM parameters.
















 §.§ Unlearning for Image Generative Model

Image generative models have various applications, such as image editing, style transfer, super-resolution, and data augmentation <cit.>. However, image generative models also face challenges and risks, such as violating data privacy, infringing data ownership, and generating inappropriate or misleading images.
Based on the degree of influence removal achieved, the existing methods for machine unlearning for image generative models can be broadly classified into two categories <cit.>:



  
Exact Unlearning Methods. These methods focus on removing the influence of targeted data points from the model through retraining at the algorithmic level completely. It usually involves censoring images from the training dataset such as removing all people's images <cit.> or excluding undesirable classes of data and then performing model retraining <cit.>. The retraining process of large models is often costly, which makes this dataset removal-based approach less practical.



  
Approximate Unlearning Methods. These methods aim to minimize the influence of target data points in an efficient manner through limited parameter-level updates to the model. This approach is post-hoc and efficient to test and deploy. Among them, <cit.> introduced a new concept of weight saliency and used a gradient-based approach to estimate the influential weights and then conduct unlearning accordingly.<cit.> defined the unlearning process from the knowledge perspective and proposed an entanglement-reduced mask (ERM) structure to reduce the knowledge entanglement during training. <cit.> mainly focused on erasing the high-level visual concept from the text-to-image models. <cit.> migrated Elastic Weight Consolidation (EWC) and Generative Replay (GR) from continual learning to perform unlearning effectively. <cit.> formulated the unlearning problem as a bi-level optimization problem and proposed a first-order method to solve it accordingly.



















§ LIMITATIONS
 


 §.§ The Efficacy of Unlearning

We divide the limitations on the efficacy of the current methods into four subsections. These include the discussions on the overarching efficacy limitations, followed by specific efficacy of the LLMs and the image generative models.


	 





  §.§.§ General Weaknesses



  
Dependence on the Original Training Data.
Many of the existing unlearning methods assume that the unlearning targets are a subset of the training set, and therefore require access to the original training data <cit.>. However, it can be difficult to obtain the original training data at times. Training data can be confidential when they are concerned with privacy or contractual issues. For models trained through distributed learning, the model and data are decentralized and can be aggregated from complex sources. Moreover, the training data can often be lost or corrupted due to the limit of storage space, cyber-attacks, or hardware failures. Unlearning methods that are dependent on the training data can not fit into these scenarios, which hinders the generalization of unlearning methods to more real-world applications.



  
Scalability Issue. 
The parameter optimization technique is still the major fundamental idea of the existing unlearning methods for generative models, which involves the iterative updating of parameters <cit.>. This updating process can be computationally expensive and time-consuming, especially when dealing with large-scale models or datasets. Besides, even though the unlearning of target instances can be conducted in a batch-wise manner to improve the efficiency <cit.>, experiments show that a larger batch size brings more degradation to the performance of the model <cit.>. An alternative way is to feed the unlearning instances sequentially, which is beneficial for maintaining the capability of the model <cit.>. The sequential approach compromises system efficiency and poses significant scalability challenges. Particularly in practical applications, there is a frequent necessity to consecutively apply unlearning on the large generative models within constrained timeframes to maintain the relevance and accuracy of models in dynamic environments. Current SOTA methods for unlearning are not effective in addressing scalability issues, which are critical in production.



  
Information Leakage of What was Forgotten. 
Machine unlearning is often used to remove user data for privacy reasons. In this case, users who require the deletion of data may not expect their identity to be identifiable through the requests. In other words, if a model that has gone through the unlearning process still contains clues revealing the identity of the user who is related to the deleted contents, the unlearning should not be considered complete or effective. Nonetheless, this leakage issue has not been comprehensively tested and the risk can be hidden for most of the unlearning methods designed for generative models. Membership Inference Attack (MIA) is a kind of attack specifically designed to test whether the deleted data can be inferred from the output of the unlearned models. Typically, a binary classifier is trained to distinguish the samples in the forgotten set from those in the retained set. A model without information leakage of the deleted data should be able to puzzle the classifier to output the probability close to 0.5 for all samples. Although MIA has been applied to test the unlearning methods in <cit.> showing their robustness against the leakage of the deletion information, this limitation of leakage can still exists in other unlearning methods of generative models.



  §.§.§ For LLMs



  
Task Dependence. 
LLMs learn general representations of both syntactic and semantic knowledge during their pretraining on the large corpus. This enables them to serve as a more general-purpose tool to solve different tasks. The patterns in the representations are intertwined and can be vulnerable when combined with downstream tasks. Catastrophic forgetting is a typical example of such vulnerability, which often occurs during transfer learning. The model can lose its generalization ability and overfit to the target domain in a catastrophic forgetting <cit.>. Thus, when testing the efficacy of an unlearning method for the LLMs, it is important to conduct comprehensive fidelity experiments on datasets from various domains. Nevertheless, most of the existing work tests the retaining and forgetting performance of the unlearning methods on specific datasets <cit.>. The impact of the unlearning process on the model's generalization ability to other tasks is rarely verified. Although the model preserves its performance on the current task, it remains uncertain whether the nuanced modifications in parameters during unlearning force the model to compromise its capability in other tasks. We argue that an effective unlearning method should minimize the performance degradation of the target model on diverse tasks. 



  
Forget or Lie? 
Nowadays, we hold a higher expectation of generative models than before. This gap of expectation is more prominent in terms of LLMs. In the previous era, we mainly focused on improving the fluency and stability of the generation. Therefore, as the unlearning results from <cit.> shown in Figure <ref>, a worse performance or a fabricated description for the sensitive instances could be viewed as a successful unlearning result. However, simply generating incorrect outputs can no longer be enough when we are expecting factual and reliable generation. LLMs are being transformed to function as knowledge bases <cit.> consisting of structural representations of facts and relations. Besides, substantial efforts have been devoted to reducing hallucinations <cit.>. Fake outputs have become increasingly unacceptable after we force the LLMs to forget target knowledge. Contrary to the conventional methods that might lead to a distorted output distribution, an effective unlearning method should teach the model to generate appropriate explanations for the absence of unlearned information. This necessitates an additional objective in the training of unlearning schemes, which should be specially designed to inhibit the models from producing deceptive or hallucinatory responses. This objective aims to maintain the integrity and reliability of the model's output after unlearning.







  §.§.§ For Image Generative Models



  
Visual Perception Inconsistencies.
Human visual systems exhibit high sensitivity to inconsistencies in images. Conversely, image generative models after unlearning tend to generate images with nuanced discrepancies, which are noticeably contradictory to human perceptual norms. For models with deletion of features <cit.>, the discrepancies can be subtle changes in the details of the image including abnormal colors, shapes, or textures of objects and backgrounds. For models trained to unlearn a large block of an image <cit.>, imbalanced foreground and background, unrealistic patterns, or visual artifacts can occur in the images. This kind of abnormal factor is a critical obstacle to the landing of the image generation unlearning techniques and can also be potentially risky for the privacy of users. Thus, it is significant for unlearning methods to minimize the perception inconsistencies in the generated images to become more reliable and effective. 



  
Creation of Visual Bias. 
Some of the unlearning methods for image generation aim to remove certain features or concepts <cit.> in the images. This process requires the unlearned model to recover the part that was occupied by the patterns related to the concept. The limitation is that the recovery can introduce undesired biases that may not have existed in the original image. 

These biases could manifest in the form of gender-specific disparities or patterns indicative of racial biases. Assuming the primary goal of employing concept-removing unlearning is to eliminate biased features, the introduction of other types of biases could bring unpredictable deficiencies to the data, which might be even more challenging to detect and address.



  
Ambiguity in Prompts. 
Image generative models, especially text-to-image models, generate images that share closely matched representations with the corresponding texts in the same latent space. In the unlearning of text-to-image models, textual concepts usually serve as the prompt to guide the removal of visual features. Under this setting, one of the key challenges is to retain the intrinsic similarities between text and image representations. However, some textual prompts can be ambiguous and some concepts may possess vague boundaries with others. The ambiguity may lead to a mismatch between the unlearning prompt and the image features, thereby introducing incomplete unlearning results. As evidence, we find both <cit.> and <cit.> report difficulties in forgetting concepts that are abstract, ambiguous, and intertwined with others. Consequently, the ambiguity in prompts remains a significant challenge for the unlearning of text-to-image models. This issue necessitates further investigation to enhance the efficacy and accuracy of these models in diverse applications.




 §.§ The Side Effect of Unlearning





  
Impact on Utility. 
During the process of machine unlearning, particularly in text-to-image generative models, it becomes evident that forgetting specific concepts can have some negative consequences on the performance of associated concepts <cit.>. This phenomenon is especially pronounced when considering the intricate relationship between artistic styles. For instance, unlearning the style of Van Gogh may inadvertently impact the style of Claude Monet. Previous studies largely overlooked the inclusion of mechanisms to mitigate these negative outcomes and propose proper methods to control these effects <cit.>.




  
Impact on Generalization. Machine unlearning methods usually inversely process the forget set samples, which potentially influences the test performance. On the one hand, a majority method tries to decay the performance of the unlearned model, by flipping the label <cit.> or updating with inverse gradient <cit.>. However, the forget set performance can be viewed as the test performance of the unlearned model. On the other hand, some methods <cit.> aims to align the output distribution of the forget set and an unseen set of the unlearned model. However, the inherent difference between the forget set and the unseen set may lead to mismatching between the two targets. Therefore, forcing the model to inversely learn the forget set samples may impair the generalization of the test set. 






 §.§ The Difficulty of Measuring

Addressing the complexities of evaluating the efficacy of unlearning across different generative scenarios, such as Language Models and image generative models, poses a multifaceted challenge. Each scenario presents unique obstacles, necessitating a tailored approach. After examining individual scenarios, we will explore the overarching challenges that pervade the field of generative model unlearning.





  
Evaluation for Large-scale Models.
Evaluating unlearning in LMs, particularly large-scale models, encounters the challenge of using general metrics like Membership Inference Attacks (MIA) or other classifier-based methods. <cit.> emphasize this difficulty, pointing out the often inaccessible nature of the full training corpus and the complexities in implementing MIA-like methods within LLMs. However, alternative methods such as using fixed models to evaluate unlearning effects also face limitations, for instance, they may struggle to gauge the similarity between unlearned and original model outputs for utility evaluation, indicating a need for further methodological advancements.



  
Evaluation for Diffusion Models.
Current evaluation methods often employ classifier-based approaches and image quality metrics like Frechet Inception Distance (FID) <cit.>. However, these methods also present limitations. For instance, the use of classifiers may not capture the subtle influences of unlearned data comprehensively. While FID is a measure of image quality and utility, it cannot fully evaluate the differences between images pre- and post-unlearning, nor does it guarantee that generated images adhere to intended conditions or assess changes in image content corresponding to unlearned aspects.








  
Lack of Standardized Metrics. There is a absence of universally accepted metrics in each generative scenario. Even within image generative models, where some consensus on methods like classifier-based evaluation for unlearning exists, these metrics are implemented and interpreted differently across studies <cit.>, underscoring the need for standardized approaches.



  
Measuring Residual Information. Lacking universally accepted metrics to capture the residual information for unlearning, and the controversy surrounding the popular use of metrics like MIA and classifier-based methods persists <cit.>. For example, their application in generative models is debated due to inherent limitations. Furthermore, the direct application of intuitively appealing standards like MIA in generative models proves challenging <cit.>.





  
Limited Access to Original Data and Shadow Models. Having adequate access to original data and shadow models can significantly aid in evaluating unlearning <cit.>. However, especially in large-scale generative models, the volume of data often makes this impractical, hindering effective measurement.







  
Condition-Output Alignment. In conditional generative models, maintaining alignment between output and conditions post-unlearning is crucial. For image generative models, tools like CLIP <cit.> offer some solution, but for language models, evaluating whether responses align with prompts remains a non-trivial task.
































§ FUTURE PROSPECTS















 §.§ Towards More Effective Unlearning for GenAI

We provide three directions where future research can focus in terms of improving the efficacy of unlearning for GenAI. 




  
Transferable and Scalable Unlearning.
In the forthcoming period, with the extremely increasing number of data being pooled into the training of large generative models, the demand for the right to be forgotten will inevitably intensify. This can induce the massive use of the unlearning methods on a significant amount of data. In this case, we emphasize that future unlearning methods should be able to effectively adapt to large-scale applications. The unlearning should be agnostic to the statistics of the original data to enable transferability between different tasks. Moreover, instead of retraining the whole model, more attention needs to be paid to parameter-efficient methods that only adjust a small portion of weights. Only lightweight unlearning methods can become prevalent and permeate into any downstream areas where sensitive data could potentially be located. When designing an unlearning algorithm, researchers may need to care more about a fast and accurate tackling of the privacy problem instead of sacrificing efficiency for a minor performance improvement.



  
Unlearning for General Generative Models.
As we are pacing into the era of Artificial General Intelligence(AGI), we have to consider the demand from general generative models when designing unlearning methods. Since the emergence of generalisability usually comes from extreme scaling up at a very high cost, the loss of such ability will not be affordable. We need unlearning methods that maintain the general ability of the target model on all tasks. This certainly should be accompanied by a comprehensive evaluation benchmark, which will be discussed in later sections. Additionally, the unlearning method should also be curated to prevent hallucinations. As aforementioned in <ref>, existing unlearning methods can lead the generative models to output fabricated facts which may require more effort in cleaning. To address this problem, different from knowledge editing which performs precise alterations of the knowledge, in unlearning we may avoid the injection of new knowledge and seek alternative ways to represent the forgetting of knowledge. 



  
Safe Unlearning
The unlearned model should not become more vulnerable to the attack of privacy or bias. Even though, as we mentioned in <ref>, most of the existing methods ignore the necessity to minimize the revealing of privacy or the creation of bias under hostile attacks. We must consider whether the unlearning algorithm will amplify the bias or privacy issues in the original training data and whether the models after unlearning will exhibit excessive reactions to the modified data points. We may consider integrating differential privacy techniques during the unlearning phase, which can provide a mathematical guarantee of privacy protection. Additionally, fairness-aware algorithms could be adapted to monitor and adjust the model's outputs, ensuring that unlearning does not unfairly impact certain groups. By prioritizing the development of such comprehensive approaches, the field can move towards unlearning methods that not only remove data effectively but also uphold the ethical standards required for responsible AI development.






 §.§ Utility-Unlearning Trade-off

As delineated in the preceding section, extant unlearning algorithms substantially impair the performance of the original model. Consequently, in the design of unlearning algorithms, it is imperative to consider the trade-off between model performance and the efficacy of unlearning. In this section, we envisage several potential solutions that may address this Utility-Unlearning Trade-off in the future.


  
Regularization.
Owing to the small size of the forget set, the disparity in parameters between the retrained model and the original model is typically minimal. However, in current unlearning algorithms, there is often a substantial deviation of the unlearned model from the original model due to an overemphasis on unlearning efficacy. This deviation results in the loss of a considerable amount of useful information, thereby substantially diminishing the utility of the model. Consequently, a very direct approach would be to employ various regularization methods to constrain the changes to the parameters during the unlearning process. Techniques from parameter-efficient fine-tuning methods, such as those employed in LoRA <cit.> or Adapter <cit.> methods, could be adapted for this purpose. 

This approach achieves a more optimal Utility-Unlearning Trade-off and facilitates a faster unlearning efficiency.


  
Multi-Objective Optimization.
Merely simplistically applying regularization constraints could yield unforeseen outcomes, and due to the presence of conflicting training objectives, this approach may result in suboptimal unlearning efficiency. An alternative method worth exploring involves the employment of gradient surgery techniques, commonly employed to address conflicting gradients present in the multi-task learning framework <cit.>. By pruning gradients that conflict with the direction of knowledge preservation, it becomes feasible to retain the performance manifestations of the samples in the retained set intact.





 §.§ Evaluating Metrics


In the pursuit of advancing the field of machine unlearning in generative models, it is imperative to address the notable limitations in current evaluative practices. The intricate nature of generative models, ranging from language to image generation, demands a nuanced approach to metric development. This endeavor is not merely a technical challenge but a fundamental requirement to ensure that unlearning processes align with ethical standards, maintain utility, and adapt to diverse applications. The following directions are proposed not only as responses to identified gaps but as strategic advancements that acknowledge the evolving landscape of GenAI.



  
Holistic and Standardized Metrics.The development of holistic and standardized metrics is crucial for creating a uniform framework for evaluating unlearning across different generative models. This approach will facilitate comparative studies and benchmarking, enabling a clearer understanding of the effectiveness of various unlearning methods. By integrating measures of residual information, utility retention, and model integrity, these metrics can provide a comprehensive assessment that is currently lacking in fragmented and scenario-specific evaluations.



  
Advanced Residual Information Assessment.Advanced methods for residual information assessment are essential to address the limitations of current metrics, which often fail to capture the nuanced effects of unlearning. This necessitates the exploration of novel approaches, such as utilizing AI interpretability techniques, to trace and quantify the lingering influences of unlearned data. Such methods could provide deeper insights into the effectiveness of unlearning processes, bridging the gap between theoretical unlearning and its practical implications.



  
Integrated Efficacy-Utility Metric.A nuanced approach involves formulating an integrated metric that encapsulates both unlearning efficacy and utility preservation. This could be operationalized as a composite score, merging domain-specific measures of unlearning with utility indicators (e.g., FID for image models, ROUGE for text models). Adjusting the weightage in the composite score would allow for flexibility depending on domain-specific requirements.



  
Emphasizing Condition-Output Alignment.In conditional generative models, maintaining the fidelity of outputs to their conditions post-unlearning is critical. Developing metrics that rigorously evaluate this alignment is imperative, especially considering the subjective nature of outputs in these models. 



  
Incorporating Global Influence Evaluation.Evaluating the global influence of unlearning is crucial for understanding its broader impact on a model's utility across various tasks and domains. This perspective is vital for ensuring that unlearning specific data does not inadvertently compromise the model's overall performance and applicability. 



  
Addressing Subjectivity in Output Evaluation.
The subjective nature of outputs from generative models necessitates an evaluation framework that incorporates both quantitative metrics and qualitative assessments. This framework is especially important in scenarios where content, style, or ethical considerations play a significant role in the outputs.







  
Cost-efficiency and Scalability Metrics. Finally, the practical aspects of unlearning necessitate metrics that evaluate cost-efficiency and scalability. This is particularly pertinent for large-scale models, where resource constraints play a crucial role. Metrics that assess computational resources, time, and sample efficiency are vital for understanding the feasibility and practicality of unlearning methods, ensuring they are accessible and implementable in diverse contexts.






 §.§ Unlearning Benchmark

Benchmarking is indeed an important aspect of evaluating and comparing different machine unlearning methods for GenAI. However, there are not many existing benchmarks that specifically address this problem. Therefore, there is a need to develop more comprehensive, reproducible, and interpretable benchmarks for machine unlearning in GenAI.









  
Benchmarking LLMs.
It was only recently that the urgent of benchmarking machine unlearning in LLMs aroused the attention of researchers. As the pioneering research, TOFU (Task of Fictitious Unlearning) benchmark <cit.> involves a dataset of 200 synthetic author profiles, each with 20 question-answer pairs, and a subset known as the ‘forget set’ for unlearning.  
TOFU allows for a controlled evaluation of unlearning with a suite of metrics, offering a dataset specifically designed for this purpose with various task severity.

While TOFU is a significant contribution to the field of machine unlearning in GenAI by introducing the first comprehensive benchmark for unlearning in the context of LLMs, it also has some limitations. 

These limitations inspire us for future directions on further benchmarking LLMs from different perspectives: (1) The synthetic nature of the data and the scenarios may not capture the real-world challenges and risks of machine unlearning in LLMs, such as the diversity and complexity of the data sources, the ambiguity and subjectivity in data contents, and the ethical and legal implications of the data ownership and consent. Therefore, real-world datasets especially where the exact retrain set is inaccessible would be a valuable add-on for effective evaluation. 
(2) The evaluation metrics and datasets may not be sufficient or representative of the forget quality and model utility, as they only cover a limited range of tasks and domains with a small quantity. Therefore, more complex tasks that account for the trade-offs and interactions between different metrics and datasets, such as directly forgetting a specific person rather than a set of people, are important to demonstrate the forget quality and model utility. 
(3) The baseline methods may not be adequate or competitive for machine unlearning in LLMs, as they only include four methods that are based on parameter optimization. In the future, more LLM-native techniques such as parameter merging or in-context learning should be considered to make the baselines more comprehensive.



  
Benchmarking Image Generative Models 
While there have been some efforts to address the issue of unlearning in image generative models, a clearly defined benchmark for evaluating this process is yet to emerge <cit.>.

To develop a comprehensive benchmark for the unlearning capabilities of image generative models, it is essential to consider multiple dimensions: 
(1) The diversity of unlearning goals. These should encompass tasks such as eradicating distinct image aesthetics, excising particular objects from scenes, and filtering out content that is not suitable for all users. 
(2) The situations involving the simultaneous unlearning of multiple concepts. Past endeavors in dataset construction primarily focused on unlearning individual concepts. Yet, it holds paramount importance and possesses practical significance to carry out examinations that evaluate the eradication of multiple concepts concurrently. 
(3) The unlearning objective of synonym concepts. For example, the model's competency in unlearning the artistic style of Van Gogh should be tested. This should include the model's ability to dissociate from works like “Starry Night," which, despite not explicitly naming Van Gogh, could still be indicative of his distinctive style.



§ CONCLUSIONS

In this position paper, we have systematically examined the challenges and limitations in the field of machine unlearning within GenAI. Our analysis reveals critical areas requiring attention: efficacy limitations in LLMs and image generative models compounded by issues of scalability, potential data leakage, the side effects on model utility and safety, and the difficulty of measurement design. We posit more effective unlearning for GenAI, the establishment of robust and nuanced evaluation metrics, and a balanced approach to the utility-unlearning trade-off. These steps are crucial for the development of sophisticated benchmarks, and realizing GenAI's full potential while adhering to ethical standards.






§ IMPACT STATEMENTS

Our contribution lays a foundation for future research, emphasizing the need for innovative, responsible strategies in machine unlearning. We assert that addressing these challenges is imperative for the ethical advancement of GenAI, ensuring its alignment with societal values and legal norms. This paper serves as a call to action for the research community to prioritize responsible and effective unlearning methods in the rapidly evolving landscape of GenAI.


icml2024




























 Paper Title: Hilbert curves for efficient exploratory landscape analysis neighbourhood sampling.











Hilbert curves for efficient exploratory landscape analysis neighbourhood sampling
     Johannes J. Pienaar

	Department of Computer Science

	University of Pretoria 

        Pretoria, South Africa 

	 

	
	https://orcid.org/0000-0003-3546-1467
    < g r a p h i c s >
Anna S. Bosman 

	Department of Computer Science

	University of Pretoria 

        Pretoria, South Africa 

	 

 https://orcid.org/0000-0002-6070-2632
    < g r a p h i c s >
Katherine M. Malan 

	Department of Decision Sciences

	University of South Africa 

        Pretoria, South Africa 

	 



    August 5, 2024
======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 

Landscape analysis aims to characterise optimisation problems based on their objective (or fitness) function landscape properties. 
The problem search space is typically sampled, and various landscape features are estimated based on the samples.
One particularly salient set of features is information content, which requires the samples to be sequences of neighbouring solutions, such that the local relationships between consecutive sample points are preserved. Generating such spatially correlated samples that also provide good search space coverage is challenging. It is therefore common to first obtain an unordered sample with good search space coverage, and then apply an ordering algorithm such as the nearest neighbour to minimise the distance between consecutive points in the sample. However, the nearest neighbour algorithm becomes computationally prohibitive in higher dimensions, thus there is a need for more efficient alternatives.
In this study, Hilbert space-filling curves are proposed as a method to efficiently obtain high-quality ordered samples. Hilbert curves are a special case of fractal curves, and guarantee uniform coverage of a bounded search space while providing a spatially correlated sample. We study the effectiveness of Hilbert curves as samplers, and discover that they are capable of extracting salient features at a fraction of the computational cost compared to Latin hypercube sampling with post-factum ordering. Further, we investigate the use of Hilbert curves as an ordering strategy, and find that they order the sample significantly faster than the nearest neighbour ordering, without sacrificing the saliency of the extracted features.














§ INTRODUCTION

Search landscape analysis has established itself as a useful approach for understanding complex optimisation problems and analysing evolutionary algorithm behaviour <cit.>. Many landscape analysis techniques have been developed over the years, with the most widely used techniques including fitness distance correlation <cit.>, local optima networks <cit.>, and exploratory landscape analysis (ELA) <cit.>. When landscape analysis produces numeric outputs, the resulting feature vectors can be used as abstract representations of problem instances, where instances with similar feature vectors are assumed to fall into similar problem classes. If these feature vectors effectively capture the important characteristics of problems, they can be used as the feature component for automated algorithm design (AAD) – specifically, automated algorithm configuration and selection. A number of recent studies have achieved different aspects of AAD using landscape analysis in specific contexts <cit.>. 

Landscape analysis approaches differ in terms of what they measure or predict (e.g. ruggedness, modality, presence of funnels, and so on), and also on what they produce (e.g. numerical results or a visualisation of a phenomenon) <cit.>. They can also be distinguished based on the scale of the analysis <cit.> – a global approach attempts to characterise the features of the search space as a whole, while a local approach will consider the features of the landscape in the neighbourhood of solutions. Fitness distance correlation <cit.> and local optima networks <cit.> are both examples of global approaches, whereas the average length of an adaptive walk <cit.> is an example of a local landscape feature. 

Many landscape analysis techniques that measure local features are based on samples that are spatially correlated, i.e. sequences of neighbouring solutions, as opposed to a sample of independent solutions from the whole search space.  Techniques that require such sequences of sampled solutions include correlation length for measuring ruggedness <cit.>, entropic profiles of ruggedness and smoothness with respect to neutrality <cit.> – adapted as a single measure of ruggedness for continuous spaces <cit.>, approximations of gradient <cit.>, information content features <cit.>, and measures of neutrality <cit.>. 

In the context of numerical optimisation, a wide range of landscape analysis approaches have been implemented in the R package  <cit.>, which has also been re-implemented in Python ([<https://pypi.org/project/pflacco/>]). Over 300 landscape metrics are included in , organised into 17 sets, which include the original ELA metrics <cit.> covering six of the feature sets, and the set of information content features <cit.> consisting of five metrics. In a study of the subset of the “cheap" feature sets in , Renau et al. <cit.> found that the two most salient (in terms of distinguishing between problems) and robust feature metrics were from the information content and ELA meta-model feature sets. Unlike the other feature sets in , the information content metrics require spatially correlated samples.
 The saliency of the information content feature metrics and the additional requirement of neighbourhood ordering is the motivation for this study.  

To date, the two main strategies proposed for generating spatially correlated samples in continuous search spaces are random walks <cit.> and post-factum ordering of global samples <cit.>. Desirable properties of a sampling strategy for estimating local features are that the solutions provide good coverage of the search space <cit.>, successive points are positioned close to each other (compared to other solutions in the sample) to capture landscape changes in the neighbourhood, and that the process has low computational expense. The most commonly used approaches to neighbourhood sampling for numerical optimisation are uniform random sampling or Latin hypercube sampling (LHS) <cit.>, followed by either random or nearest-neighbour ordering. These are the approaches implemented in  and .

In this paper, we investigate the use of Hilbert curves <cit.> as a new sampling strategy. Hilbert curves are fractal space-filling curves with two desirable properties: (1) they guarantee uniform search space coverage, and (2) neighbouring points on the curve are located close to one another. We show that Hilbert curves are comparable to ordered LHS in terms of the saliency of the extracted features, but are significantly cheaper to compute. Additionally, we show that a Hilbert curve can be used to spatially order an LHS sample, resulting in significant computational gains compared to the commonly used nearest neighbour ordering method.   







§ HILBERT CURVES

A space-filling curve is a surjective continuous function from the unit interval [0,1] to a unit hypercube [0,1]^d. Surjectivity implies that every point in the hypercube maps to at least one point in the interval, and continuity ensures that no areas in [0,1]^d are missed <cit.>. Space-filling curves are a special case of fractal curves, and are guaranteed to fill a continuous space in the limit. 

Our interest in space-filling curves derives from two useful properties that they offer, namely (1) uniform coverage of a bounded d-dimensional space, and (2) the ability to provide a unique mapping between points in the d-dimensional space and points on the  1-dimensional curve <cit.>. Specifically, this study considers the Hilbert space-filling curve, first proposed by D. Hilbert in 1891 <cit.>. The Hilbert curve is defined recursively, and can be constructed through a limit process of iteration. Each successive iteration creates an approximation of the true Hilbert curve that passes through more points in the d-dimensional unit hypercube. For practical purposes, the number of iterations is chosen to be finite, and is further referred to as the order of the Hilbert curve. 

Consider the construction of a Hilbert curve in 2D. For the first iteration, a line on the closed interval [0,1] and a square [0,1]^2 are taken. Four equidistant points are selected on the line, where the starting point is at 0 and the end point is at 1. The square is subdivided into four equal parts. Intervals of the line connecting each pair of points are then mapped onto the square such that the intervals adjacent on the line share a common edge on the square. This results in a simple U-shape, illustrated in Figure <ref>(hc:p1). At each subsequent iteration, the curve from the previous iteration is divided into four equal parts. Each part is then shrunk by a factor of 1/2, rotated, and repositioned such that the four curves connect at their endpoints in a U-shaped or reverse U-shaped pattern. Figure <ref> shows 2D Hilbert curves of order 1 to 3. 





The Hilbert curve is bijective, i.e. the mapping between points on the curve and a d-dimensional space is reversible. Bijectivity of space-filling curves has been useful in organising multi-dimensional data storage and retrieval systems, since it allows a linear index to the data to be constructed and searched <cit.>. Ideally, closely related data in the d-dimensional space would also be close together on the 1-dimensional curve. Faloutsos and Roseman <cit.> investigated the clustering properties of Hilbert curves compared to other space-filling curves, and found Hilbert curves to be superior when measuring the average distance (along the curve) to the nearest neighbours of a given point. This finding suggests that Hilbert curves may serve as a viable alternative to nearest neighbour sorting of a sample. 

For the Hilbert curve implementation in this study, we use the  Python package (<https://pypi.org/project/hilbertcurve/>), based on the algorithm presented by Skilling <cit.>. All code used for the experiments conducted in this study can be found at <https://github.com/jpienaar-tuks/MIT807>.



§ HILBERT CURVES AS SAMPLERS


Given a Hilbert curve of a particular order, the vertices can be used as a basis for a sample in the associated multidimensional space. Table <ref> shows how the number of vertices grows with the order of the curve and the dimension of the search space. When sampling for ELA, it is common practice to use sample sizes of 10^2× d to 10^3× d <cit.>. 

Table <ref> highlights in bold the order and dimension combinations where the number of vertices exceeds a common sample budget of 10^3 × d. To remain within a sampling budget, we sub-sample randomly from the set of vertices on a Hilbert curve with a minimum order of 3. 



 §.§ Stochastic sampling using Hilbert curves



Since the generation of a Hilbert curve is deterministic, the following two strategies were evaluated to introduce stochasticity:

    
  * Selecting random points along the edges of the Hilbert curve, illustrated in Figure <ref>(fig:hilbert_along_edges): Given any two sequential vertices, P_i and P_i+1, a new point is selected P_j = rP_i+(1-r)P_i+1, where r ∼ U(0,1). 
    
  * Selecting points near vertices of the curve, illustrated in Figure <ref>(fig:hilbert_near_vertexes): For each vertex P_i, generate a new point drawn from a normal distribution centred on P_i with a standard deviation σ (constrained by the bounds of the search space). The step size of the Hilbert curve before scaling is 1, so σ was empirically chosen to be 0.3 to prevent excessive potential overlap with points generated by neighbouring vertices.




Neighbouring vertices on a Hilbert curve are equidistant, so the sequence of vertices has a constant step size. The two randomisation techniques introduce variation into the step size. 
Figure <ref> shows that 

the randomisation near vertices strategy produces step sizes with a more normal distribution. For all further experiments we have used the randomisation around vertices strategy.



 §.§ Search space coverage


We now investigate the extent to which Hilbert curve sampling covers the search space compared to competing strategies such as Latin hypercube sampling (LHS). To investigate the search space coverage, the following three strategies were used to draw samples from search space [-5,5]^d of sizes n ∈{100d, 316d, 1000d} for dimensions d ∈{5, 10, 20, 30}:

    
  * Hilbert curve: randomly select the required number of points from the curve (without replacement).
    
  * LHS: generate the required sample using a Latin hypercube design <cit.>.
    
  * Random walk: use a simple random walk <cit.> with maximum step size of 1 to generate the sample.

For each sample size and dimension combination, a random uniform sample was drawn as a reference set. Thirty independent runs of each sampling strategy were implemented, and for each run, the Hausdorff distance <cit.> to the reference set was calculated. The Hausdorff distance as a measure of search space coverage was first proposed and investigated by Lang and Engelbrecht <cit.>, and their methodology is followed in this study. Statistical significance tests were performed for each sampling strategy as proposed by Derrac et al. <cit.> and described by Lang and Engelbrecht <cit.>.





Table <ref> shows that the Hilbert curve and LHS have similar Hausdorff distances, implying a similar coverage of the search space, whereas the random walk provides the worst coverage. Lower values were achieved in most cases by the Hilbert curve sampler, confirmed by the statistical significance tests presented in Table <ref>. The null hypothesis is that there is no significant difference between the Hausdorff distances for each of the sampling strategies. Results show that the null hypothesis can be rejected, given the p-values and a significance level of 0.05. We therefore conclude that the Hilbert curve sampler achieves a more uniform search space coverage than LHS.

    
    




 §.§ Computational cost

We now compare the cost of the Hilbert curve as a sampler to alternative sampling strategies. 
Generating the Hilbert curve is not computationally cheap, but we expect that this upfront investment will be justified if the sample is subsequently used to calculate landscape metrics that require ordered samples. 


To evaluate the computational cost, a performance counter[Python's ] was used to keep track of the time to a) generate the sample and b) calculate information content metrics for both the Hilbert curve and LHS. Using the  library, the information content () metrics were calculated for each of the 24 BBOB functions defined as part of the COCO platform <cit.>. This was done for each dimension and sample size listed in Table <ref>. Note that for LHS to be used for information content metrics, the sample needs to be ordered. We evaluated both the random ordering and the nearest neighbour ordering strategies. Also note that the information content function provided by the  library was modified to accept  as an ordering argument to use the pre-ordered Hilbert curve samples.

Figure <ref>(fig:timing:generation) 
shows that the Hilbert curve sampler is slower than LHS to generate a sample of the same size. While the computational cost of the Hilbert curve sampler seems to grow at least quadratically, it remains acceptable even for large sample sizes (just over 1.2 seconds to generate a sample of 30 000 points). 

Figure <ref>(fig:timing:total) 
shows that for the information content metrics, Hilbert curve sampling strategy is substantially faster (especially at larger sample sizes) than LHS with nearest neighbour ordering.























    























 §.§ Predictive performance of Hilbert curve samples



We have shown that Hilbert curve sampling provides an efficient method for generating spatially correlated sequences of solutions from continuous search spaces, and that these sequences provide good coverage of the search space. We now investigate whether the landscape features extracted from these samples provide good predictive performance for algorithm selection. 

Following the approach by Muñoz et al. <cit.>, we use the task of predicting the class of each BBOB function as a proxy for algorithm selection: if the landscape features of function instances can be used to discriminate between problem classes, then they should also be effective predictor variables for algorithm selection. Five target classes corresponding to the BBOB function groupings as outlined in <cit.> were used, namely (1) separable functions: {f_1… f_5}, (2) functions with low or moderate conditioning: {f_6… f_9}, (3) unimodal functions and functions with high conditioning: {f_10… f_14}, (4) multimodal functions with adequate global structure: {f_15… f_19}, (5) functions with low or moderate conditioning: {f_20… f_24}.

The experimental setup is as follows: 24 BBOB functions in dimensions d ∈{2,5,10,20} are used, with an evaluation budget of 1 000 × d. Each BBOB function has 15 predefined instances, and three random instances per function are held out as testing data[This leave-one-instance out approach is used instead of the leave-one-problem out approach due to the small number of classes.]. For each combination of instance and dimension, 1 000d samples are drawn from [-5,5]^d, using three sampling strategies: Hilbert curve (HC), Latin hypercube (LH) and random walk (RW). Four feature sets from the  package were selected that did not require further function evaluations, namely: dispersion (), ELA y-distribution (), ELA meta model (), and information content ().=-1

Table <ref> gives the testing accuracy of a decision tree, k-nearest neighbour, and random forest classifier 

for the task of predicting the function class from the landscape metrics. The  version 1.1.3 default settings were used in all instances. Results show that a random forest model was the most effective at predicting the function class across all sampling strategies. Overall, the RW strategy emerges as the least competitive performer, while the Hilbert curve and Latin hypercube sampling strategies were comparable for all three classifiers.  

    

Table <ref> gives the performance for each of the three sampling strategies based on the four feature sets separately. An observable trend is that both the  and  features are very salient features and allow the classifiers to accurately discriminate between the function classes, confirming results of Renau et al. <cit.>. 

    
 











    




§ HILBERT CURVES AS AN ORDERING TOOL

In this section, we investigate the use of Hilbert curves as an ordering aid for samples generated from other sampling methodologies. 
A Hilbert curve of order p maps each point on a 1-dimensional curve [0,…,2^dp] to a point in [0,…,2^p-1]^d and vice versa, as illustrated earlier in Figure <ref>. The mapping is bijective, and allows for an efficient mapping from a point in the d-dimensional space to a point on the 1-dimensional Hilbert curve. 

This d-D to 1-D mapping can be exploited, given that a Latin hypercube sample of n points divides each axis into n intervals. By selecting the appropriate curve order p, i.e. 
    p=⌈log_2(n+1)⌉,
we can ensure that the resulting Hilbert curve is of sufficient length to accurately map all LHS points to points on the Hilbert curve. The ordering of the points on the Hilbert curve can then be used to provide a spatially correlated ordering of the Latin hypercube sample.

Given that the current best practice of using a greedy nearest neighbour strategy to order LHS requires calculating at least half of the pairwise distance matrix between all points in the sample, the Hilbert curve ordering strategy has the potential to be significantly faster. Furthermore, the nearest neighbour strategy tends to start with relatively short step sizes, which increase as the number of unvisited points decrease. Hilbert curve ordering is likely to provide more consistent step sizes.



 §.§ Step size consistency

Figure <ref> shows a visual comparison of the Hilbert curve and nearest neighbour ordering strategies on a 2D Sphere function sample. The underlying sample X for both strategies is identical, and was obtained using LHS. In the figures the orderings produced by the Hilbert curve (HC) and the nearest neighbour (NN) approach are depicted as a red line. Colour scale is used to indicate the fitness values of the sampled points. It is evident from Figure <ref> that the maximum step size is smaller for HC than for NN. The step size distributions shown in Figures <ref>(fig:step_size_viz:hc) and <ref>(fig:step_size_viz:nn) confirm that the maximum step size of HC is lower, and show a greater skew in the step size distribution for NN.







 §.§ Computational cost

To evaluate the computational cost of the Hilbert curve ordering, we compare it to the nearest neighbour and random (RND) ordering strategies applied to a sample generated using LHS. The Latin hypercube samples of sizes n ∈{100d, 316d, 1000d} were drawn from search space [-5,5]^d for dimensions d ∈{5, 10, 20, 30}. For each configuration, 24 independent samples were drawn, and timed with a performance counter (Python's ). 

Figure <ref> shows the performance comparison between the various ordering strategies. Hilbert curve sampling as discussed in Section <ref> is included for completeness, as it does not require additional ordering. From Figure <ref>, NN ordering performs the worst, especially for large sample sizes. Furthermore, RND sorting and Hilbert curve sampling are equally fast (red and blue curves overlap). Hilbert curve ordering produces an intermediate result, slightly slower than random ordering, but significantly faster than the NN ordering. Since the underlying sample is generated by LHS, Hilbert curve ordering retains all the benefits of LHS for ELA features insensitive to order (such as , or ).






 §.§ Evaluation of features generated using Hilbert curve ordering


We now compare the information content features generated from a Latin hypercube sample using the HC, NN, and RND ordering. Since the feature values are evaluated rather than the computational cost, this experiment was restricted to dimensions d∈{2,5} and sample sizes n∈{100d, 1000d}. A single sample set X was generated and evaluated on each of the BBOB functions to generate corresponding y = f(X) results. Information content metrics were then calculated on these (X, y) pairs using each of the ordering strategies. This procedure was repeated 30 times.

Figures <ref> and <ref> present initial partial information (M_0) and maximum information content (H_max) landscape features based on the entropy of the sample. We consider BBOB problems in 5D (problem indices listed on the x-axis), and calculate feature values based on NN, RND, and HC ordering for n=1000d.

It is evident from Figure <ref> that the HC ordering produces values that lie between RND and NN, but follow a pattern similar to that of NN. It is clear that M_0^RND converges to the same value regardless of the underlying function (confirming the results of  <cit.>). M_0^NN and M_0^HC, on the other hand, produce values that allow for discrimination between functions. Since larger values for M_0 indicate a more rugged landscape (more changes in concavity over the length of the walk), the HC ordering would on average indicate a more rugged landscape than the NN ordering.

According to Figure <ref>, H_max^RND varies over a wider range than M_0^RND. However, the H_max^RND values still tend to cluster around the upper edge of the graph, closer to the maximum value of 1. In comparison, H_max^NN and H_max^HC vary over a larger part of the [0,1] range. Larger values for H_max indicate a larger variety of objects in the landscape (pits, peaks, plateaus). Thus, HC ordering again indicates a more rugged landscape than the NN ordering.















To evaluate the relative saliency of the features produced by the various ordering strategies, permutation-based feature importance technique <cit.> is employed. We train a random forest (RF) classifier using information content features as input ({ϵ_s, 
ϵ_max, ϵ_0.5, H_max and  M_0}, as defined in <cit.>), and the function class as output (as defined in <cit.> and used in Section <ref>). Feature saliency is gauged by randomly permuting each input feature in turn across the dataset, and observing the corresponding reduction in RF accuracy. For this purpose, the features are grouped by the ordering strategy, and a RF is trained on 2/3 of the data, with the remaining 1/3 held out for testing.  

After training, the base accuracy of the RF classifier is evaluated on the held out data. The accuracy of the RF classifier is then re-evaluated as each feature in the held-out dataset is randomly permuted in turn, and the difference to the base accuracy is recorded. This procedure is repeated 10 times for each sampling strategy. The results are shown in Table <ref>. 

Table <ref> shows that  ϵ_s was the most salient feature across all sample orderings, with the RF trained on the HC ordered features relying on ϵ_s more heavily than the RF trained on the NN or RND ordered features. RND sample ordering made the M_0 feature near useless for the classifier, in line with the observations made in Figure <ref>. Conversely, RF trained on the NN ordering relied on M_0 more than RF trained on the HC ordered features. Overall, HC ordering yielded marginally better RF performance compared to the NN and RND ordering approaches.






    






§ CONCLUSION

This paper proposed the use of Hilbert space-filling curves in the context of optimisation problem landscape analysis for the purpose of (1) sampling the search space in a spatially correlated manner that also guarantees uniform coverage, and (2) spatially ordering samples generated using other  sampling algorithms such as the Latin hypercube. Experiments were conducted to evaluate the relative computational efficiency of the Hilbert curves, as well as the saliency of the landscape features extracted using Hilbert curve sampling and Hilbert curve ordering. In the context of sampling, Hilbert curves were significantly faster than Latin hypercube sampling for the purpose of generating order-sensitive features such as the information content metrics. Features extracted by the Hilbert curves were informative, and allowed for successful discrimination between problem classes. As an ordering tool, Hilbert curves performed significantly faster than the commonly used nearest neighbour ordering, and also yielded salient landscape features. Thus, Hilbert curves present a viable computationally efficient alternative to both Latin hypercube sampling, and nearest neighbour ordering of a sample for the purpose of landscape analysis.
 


    



Displayed equations are centered and set on a separate
line.

    x + y = z

Please try to avoid rasterized images for line-art diagrams and
schemas. Whenever possible, use vector graphics instead (see
Fig. <ref>).



<cit.>








splncs04




 Paper Title: Open-Vocabulary Audio-Visual Semantic Segmentation.







  National Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
  Beijing
  China
ruohguo@stu.pku.edu.cn



  Department of Electrical and Computer Engineering,
  Carnegie Mellon University
  Pittsburgh, PA
  USA
liaoq@andrew.cmu.edu



  Berkeley AI Research,
  University of California, Berkeley
  Berkeley, CA
  USA
Bias_88@berkeley.edu



  College of Information and Electrical Engineering,
  China Agricultural University
  Beijing
  China
yanyu.qi@cau.edu.cn



  National Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
  Beijing
  China
yuewenzhen@stu.pku.edu.cn



  National Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
  Beijing
  China
sjj118@pku.edu.cn



  National Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
  Beijing
  China
xingbowei@pku.edu.cn


Xianghua Ying is the corresponding author.

  National Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
  Beijing
  China
xhying@pku.edu.cn




§ ABSTRACT 
Audio-visual semantic segmentation (AVSS) aims to segment and classify sounding objects in videos with acoustic cues. However, most approaches operate on the close-set assumption and only identify pre-defined categories from training data, lacking the generalization ability to detect novel categories in practical applications. In this paper, we introduce a new task: open-vocabulary audio-visual semantic segmentation, extending AVSS task to open-world scenarios beyond the annotated label space. This is a more challenging task that requires recognizing all categories, even those that have never been seen nor heard during training. Moreover, we propose the first open-vocabulary AVSS framework, OV-AVSS, which mainly consists of two parts: 1) a universal sound source localization module to perform audio-visual fusion and locate all potential sounding objects and 2) an open-vocabulary classification module to predict categories with the help of the prior knowledge from large-scale pre-trained vision-language models. To properly evaluate the open-vocabulary AVSS, we split zero-shot training and testing subsets based on the AVSBench-semantic benchmark, namely AVSBench-OV. Extensive experiments demonstrate the strong segmentation and zero-shot generalization ability of our model on all categories. On the AVSBench-OV dataset, OV-AVSS achieves 55.43% mIoU on base categories and 29.14% mIoU on novel categories, exceeding the state-of-the-art zero-shot method by 41.88%/20.61% and open-vocabulary method by 10.2%/11.6%. The code is available at  https://github.com/ruohaoguo/ovavsshttps://github.com/ruohaoguo/ovavss.



<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224.10010245.10010248</concept_id>
       <concept_desc>Computing methodologies Video segmentation</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Video segmentation



Open-Vocabulary Audio-Visual Semantic Segmentation
    Xianghua Ying
    August 5, 2024
==================================================







§ INTRODUCTION

Generally speaking, objects can be characterized jointly by their appearance and the sounds they make. By incorporating visual and acoustic information in a collaborative manner, it is beneficial for a better perception and understanding of the concepts of objects. Recently, a novel audio-visual learning task, i.e., audio-visual segmentation (AVS), has been proposed in <cit.>. The goal of AVS is to output pixel-level maps of sound-emitting objects within video frames. This requires aligning two different modalities and serving audio as prompt to localize and segment visual objects. Different from AVS being tasked with binary foreground segmentation, audio-visual semantic segmentation (AVSS) is to classify sound sources and  generate semantic maps associating one category with each pixel. To accomplish the above tasks, many works adopt convolutional- or Transformer-based encoder-decoder architectures to establish audio-visual relationships and segment sounding objects. Despite promising results, these approaches only recognize pre-defined categories in the training set, resulting in poor generalization capacity to novel concepts. In real-world applications, this closed-set paradigm lacks practical value, since it often encounters objects from novel categories during training.



If the model has never seen an object and heard its sound, could it still accurately localize sound source in audible videos? Wang et al. <cit.> present a prompt-based model, GAVS, which attempts to pinpoint sounding objects real-world videos, including novel categories unseen during training. Specifically, they insert Adapter <cit.> into the large-scale pre-trained segment-anything model (SAM) <cit.> to construct audio-visual correlations and learn downstream tasks. With the help of the adapter-based fine-tunning technique, GAVS adapts the visual foundational model, i.e., SAM, to the AVS task, and generalize the prior knowledge to unseen objects and different datasets. However, GAVS is designed for binary segmentation, that is, it can only localize visual position of sounding objects without predicting their categories. In addition, GAVS takes only an image along with one-second audio as input and fuse multi-modal in spatial domains, leading to the inability to capture frame-by-frame relationships in temporal domains. Yu et al. <cit.> explore the feasibility of promoting visual foundational models for the AVS task in a zero-shot setting. They use CLAP <cit.>, a large-scale audio-language pre-trained model, to classify the input audio from open-set categories. Then, the predicted class names are fed into Grounding DINO <cit.> to obtain box predictions, and these boxes prompt SAM <cit.> to generate the corresponding masks. Nevertheless, only using audio information to predict the category of sounding objects is weak and inadequate in complex scenarios. For example, humans can imitate cat meows, and cars and airplanes can produce similar engine sounds. Moreover, due to the lack of visual appearances and motion cues, this model tends to segment all objects belonging to the predicted class instead of those sounding objects.

In this paper, we propose a novel multi-modal task, i.e., open-vocabulary audio-visual semantic segmentation, which aims at segmenting and classifying sound-emitting objects in videos from open-set categories, as illustrated in Fig. <ref>. Unlike the above zero-shot AVS methods that generate binary masks, we focus on the semantic segmentation and require the network to assign semantic labels to each pixel in given video frames. Open-vocabulary audio-visual semantic segmentation is a more challenging task, facing the following problems: 182 Audio is a 1D signal that exhibits high information density, meaning that multiple objects could make sound simultaneously and their audio could entangle at any timestamp. Thus, it is not easy to build correct spatial alignment from mixed-source audio to visual content. 183 Video is continuous and only using a single image-audio pair is not optimal. Temporal information and motion cues play an indispensable role in sound source localization and classification. So how to effectively integrate audio and visual features in the temporal dimension? 184 In the open-vocabulary setting, the model is trained on a closed set and then localizes while identifying sounding objects within categories beyond the annotated label space. This calls for the model to possess strong generalization abilities: not only localizing sounding objects, but also suppressing silent objects and background disturbances, such as noise or sounds from objects outside the frame.

To address the above problems, we design the following modules. For problem 182, an audio-visual early fusion module is introduced to align audio signals with image features in the spatial dimension. It takes an image feature map from the visual backbone and the corresponding audio embedding from the pre-trained audio encoder as inputs, and then computes bi-directional cross-attention between them. On the one hand, the audio embedding as the query can be enhanced by correlating with image pixels of those semantic-relevant objects. On the other hand, the image feature as the query can be supplemented and activated with audio information. For problem 183, we present an audio-conditioned Transformer decoder to establish frame-by-frame relationships and capture audio-visual dependencies in the temporal dimension. Specifically, a set of learnable object queries is first fed into the decoder and performs the cross- and self-attention sequentially to obtain object-centric representations, following Mask2Former <cit.>. Then, these queries extract audio temporal information via a cross-attention layer, where the key elements are the all audio embeddings from the audio-visual early fusion module. For problem 184, to localize sounding objects from all categories, we discard the class head but added an sound head which acts as a binary classifier only responsible for determining whether the object is making sound in the current frame. This enables our model to not be constrained by the close-set classification annotations. Additionally, we employ the pre-trained CLIP <cit.> model to predict the categories of the sounding objects.

On the whole, we propose an open-vocabulary audio-visual semantic segmentation model, termed OV-AVSS, which consists of two main components: universal sound source localization module (USSLM; detailed in Section <ref>) and open-vocabulary classification module (OVCM; detailed in Section <ref>). Equipped with the audio-visual early fusion module and audio-conditioned Transformer decoder, USSLM integrates audio and visual features in spatial and temporal domains, separately, and then outputs class-agnostic sounding objects. With the aid of the rich knowledge from CLIP, OVCM classifies these potential sounding objects without limiting to closed-label space. To assess the model’s generalization capability on unseen and unheard objects, we build a open-vocabulary audible video dataset based on AVSBench-Semantic benchmark <cit.>. Experimental results demonstrate that OV-AVSS achieves superior generalizable segmentation performance.

To sum up, our contributions are threefold:

(1) We propose a new multi-modal task, open-vocabulary audio-visual semantic segmentation, which aims at segmenting and classifying sounding objects of arbitrary open-set categories in videos.

(2) A strong baseline model is developed to generate class-agnostic masks for sounding objects and predict their categories by leveraging the prior knowledge of large-scale vision-language models.

(3) Extensive experiments indicate that our model can achieve state-of-the-art results in terms of performance and generalization. It has been ready-to-use for open-vocabulary audio-visual semantic segmentation in the real-world applications.




§ RELATED WORKS



 §.§ Audio-Visual Segmentation

Audio-visual segmentation (AVS) aims to segment the objects that emit sound within each video frame. Recent AVS research primarily focuses on supervised learning on the sparsely annotated AVSBench dataset, which can be divided into FCN-based and Transformer-based approaches. For FCN-based methods, Zhou et al. <cit.> propose temporal pixel-wise audio-visual interaction (TPAVI), which is a non-local block with cross-modal attention to locate the sound source. ECMVAE <cit.> employs three latent encoders to achieve latent space factorization, yielding both modality-shared and specific representations to model audio-visual contributions explicitly. In addition, Transformer-based methods accomplish audio-visual fusion and mask decoding with vision Transformer <cit.>. For example, AVSegFormer <cit.> uses audio-based channel attention to dynamically adjust visual features, and adopts a DETR-like architecture <cit.> to segment sounding objects. Liu et al. <cit.> associate the audio signals with visual information by aligning predicted audio class labels with instance segmentation masks, thereby highlighting sounding objects while suppressing silent ones. Audio-visual semantic segmentation (AVSS) extends the AVS task by additionally predicting the category of each sound source. Due to the semantic entanglement in audio, tackling multi-source AVSS presents a greater challenge. Zhou et al. <cit.> follow the TPAVI module <cit.> to perform audio-visual fusion, and then attach a class head for semantic predictions. CATR <cit.> separately carries out audio-visual interactions in a decoupled manner. Additionally, audio-constrained queries are introduced to select sounding objects during decoding. 




 §.§ Zero-Shot Audio-Viusal Segmentation

Current AVS approaches mainly focus on in-domain and close-set situations. When faced with unseen classes in real-world scenarios, these approaches will no longer be applicable, owing to the limited training data. To overcome the drawback, Wang et al. <cit.> design an encoder-prompt-decoder framework to improve the generalization of the AVS model by leveraging the prior knowledge of the visual foundation model. Specifically, they use the adapter technique <cit.> to fine-tune the pre-trained segment-anything model (SAM) <cit.> and generalize knowledge learned from the training set to unseen objects. Yu et al. <cit.> employ the large-scale audio-language model, CLAP <cit.>, to predict the category to which the audio belongs. For the single-source and multi-source audio, they select the class names with the highest and the top two highest scores, respectively. The predicted class names are input into Grounding DINO <cit.> and SAM <cit.> to sequentially generate box predictions and the corresponding masks. Unlike the above zero-shot methods applied to the binary segmentation, this work focuses on the audio-visual semantic segmentation and exploits the large-scale pre-trained CLIP model <cit.> to classify sounding objects from novel categories.




 §.§ Open-Vocabulary Segmentation

Compared with traditional segmentation methods <cit.>, open-vocabulary segmentation seeks to locate and recognize categories beyond the annotated label space. The main difference between zero-shot learning and open-vocabulary learning is that the latter is capable of leveraging visual-related language vocabulary data <cit.>. For example, LSeg <cit.> aligns the image embeddings with the text embeddings of category labels from CLIP text encoder. This allows LSeg to use the knowledge of vision-language models and segment objects that are not pre-defined but depend on the input texts. Recently, more works focus on video instance segmentation and attempt to build an open-vocabulary tracker based on close-set training data. Wang et al. <cit.> collect a large-scale video dataset, and propose an end-to-end approach, called OV2Seg, for open-vocabulary video instance segmentation. OV2Seg adopts a momentum-updated module to track objects and a CLIP-based classifier for recognizing novel categories. OpenVIS <cit.> adapts the pre-trained CLIP with instance guidance attention and generates object proposals and the corresponding classes. Moreover, a rollout association mechanism is designed to associate instances of any categories across frames thereby improving tracking performance. In this paper, we extend open-vocabulary learning to audio-visual semantic segmentation: segmenting and classifying sound-producing objects of arbitrary open-set categories simultaneously.






§ METHOD

In this section, we present an open-vocabulary audio-visual semantic segmentation framework, namely OV-AVSS, as shown in Fig. <ref>. In the first stage, the universal sound source localization module (Section <ref>) is applied to predict class-agnostic masks of sounding objects covering all possible classes. In the second stage, the open-vocabulary classifier (Section <ref>) empowered by CLIP <cit.>, is employed to identify the category of each sounding object mask.



 §.§ Problem Definition

Let D_train be a training dataset containing pixel-level annotations for a set of base categories C_base, open-vocabulary audio-visual semantic segmentation aims to train a model f_θ (·) on D_train, and then test on D_test for both base categories C_base and novel categories C_novel with the help of large language vocabulary knowledge C_voc. Note that C_voc is not strictly required to contain C_base or C_novel, as the language vocabulary may not cover all the class names in the vision or audio data. Given a test video clip 𝐕∈ℝ^T × H × W × 3 and the corresponding audio 𝐀∈ℝ^T, f_θ (·) is supposed to predict the segmentation mask {𝐦_𝐢}_i=1^T∈ℝ^T × H × W and the category label 𝐜∈(C_base∪ C_novel) for each sounding object of all the frames:

    {{𝐦_1, 𝐦_2, …, 𝐦_𝐓}, 𝐜}_p^P=f_θ(𝐕,𝐀),

where P is the total number of categories of the sounding objects. 
In the experiments, we designate the categories used for training as “base categories". Categories that fall outside the established base categories are referred to as novel categories.



 §.§ Multi-modal Representation

Given an input video sequence containing both visual and audio tracks, we divide it into T non-overlapping audio and visual snippet pairs {𝐕, 𝐀} = {𝐯_𝐢, 𝐚_𝐢}_i^T, where each snippet is 1-second long and T denotes the number of snippets as well as the video duration.

Audio Representation. For each audio snippet 𝐚_𝐢, we use the pre-trained VGGish <cit.> model to extract the audio feature as 𝐟^𝐀_𝐢∈ℝ^C, where C=128 is the feature dimension. The VGGish model, a VGG-like 2D convolutional neural network, is pre-trained on AudioSet dataset <cit.>. Due to the input being 2D audio spectorgram, we first convert the audio 𝐀 into a mono-waveform at a sampling rate of 16kHz and apply short-time Fourier transform to obtain the mel spectrograms. Then, they are fed into VGGish model and the audio features are extracted as 𝐅^𝐀 = {𝐟^𝐀_1, 𝐟^𝐀_2, ..., 𝐟^𝐀_𝐓}. Note that the audio representation is extracted offine and the parameters of VGGish model are frozen during the training process.

Visual Representation. For each visual snippet 𝐯_𝐢, we extract image-level hierarchical features using the convolutional-based ResNet-50 <cit.> or Transformer-based Swin-B <cit.> backbones. We represent the features as 𝐟^𝐕_𝐢,𝐤∈ℝ^H_k × W_k × D_k, where (H_k, W_k)=(H,W)/2^k+1, k=1,2,3,4, (H, W) is the resolution of each visual snippet 𝐯_𝐢 and k is the number of backbone levels. The final visual representation can be formulated as 𝐅^𝐕 = {𝐟^𝐕_1, 𝐟^𝐕_2, ..., 𝐟^𝐕_𝐓}.




 §.§ Universal Sound Source Localization
 
To enhance the audio features 𝐅^𝐀 with the image contexts and embed audio information into image features 𝐅^𝐕 simultaneously, an audio-visual early fusion module is adopted, allowing modality interaction at the early stage (before feature decoding). Specifically, the audio feature 𝐟^𝐀_𝐢 and multi-level image features {𝐟^𝐕_𝐢,2, 𝐟^𝐕_𝐢,3, 𝐟^𝐕_𝐢,4} are linearly projected to the same dimension C_av via pointwise convolution and group normalization. Then, we flatten image features and concatenate them into a 1D sequence 𝐬^𝐕_𝐢. Finally, a bi-directional cross-attention module (BC_A → V and BC_V → A) is applied to establish audio-visual spatial dependence and obtain cross-modal feature embeddings 𝐟^𝐀𝐕_𝐢, 𝐟^𝐕𝐀_𝐢. This process can be formulated as:

    𝐟^𝐀𝐕_𝐢 = BC_A → V(𝐟^𝐀_𝐢, 𝐬^𝐕_𝐢) = Softmax(𝐟^𝐀_𝐢𝐖^𝐐·(𝐬^𝐕_𝐢𝐖^𝐊)^T/√(d_k)) ·𝐬^𝐕_𝐢𝐖^𝐕 + 𝐟^𝐀_𝐢


    𝐟^𝐕𝐀_𝐢 = BC_V → A(𝐬^𝐕_𝐢, 𝐟^𝐀_𝐢) = Softmax(𝐬^𝐕_𝐢𝐖^𝐐·(𝐟^𝐀_𝐢𝐖^𝐊)^T/√(d_k)) ·𝐟^𝐀_𝐢𝐖^𝐕 + 𝐬^𝐕_𝐢

where 𝐖^𝐐, 𝐖^𝐊, 𝐖^𝐕∈ℝ^C_av× d_k are learnable parameter matrices. In BC_A → V, it serves audio feature 𝐟^𝐀_𝐢 as query and 1D image feature 𝐬^𝐕_𝐢 as key/value, and then perform multi-head attention between them. Owing to the image feature 𝐬^𝐕_𝐢 being multi-level and processed only once, the audio-visual early fusion module can efficiently leverage both strong semantics and fine-grained details. In BC_V → A, query is 𝐬^𝐕_𝐢 and key/value is 𝐟^𝐀_𝐢. It treats the audio signals as auxiliary information, and the embedded semantical consistency is used to highlight the corresponding spatial regions.

Given the processed image features 𝐟^𝐕𝐀_𝐢, we feed them into the pixel decoder, a multi-scale deformable attention Transformer <cit.>, to output the enhanced feature 𝐨^𝐕𝐀_𝐢 and high-resolution per-pixel embeddings 𝐩_𝐢. Considering sound source localization requires learning more accurate audio-object matching relationship, we propose the audio-conditioned Transformer decoder (AudioMaskDec) to incorporate audio information and decode the masks for sound-emitting objects. As shown in Fig. <ref>, AudioMaskDec consists of five key components: a spatio-temporal cross-attention layer, an audio self-attention layer, an object self-attention layer, an audio-aware cross-attention layer, and a feedforward neural network. 

To be specific, we first initialize a set of learnable object queries 𝐪∈ℝ^N × C_o supplemented with positional encodings, where N is the number of object queries. Then, the spatio-temporal cross-attention layer computes cross-attention between object queries and all frames' features. This empowers object queries 𝐪' with visual semantics and associates them with all potential objects, following previous works <cit.>. 
After that, the object queries 𝐪' and audio embeddings {𝐟^𝐀𝐕_𝐢}_i=1^T are fed into the object self-attention layer and audio self-attention layer, separately. With the help of two separated self-attention layers, it isolates 𝐪' and {𝐟^𝐀𝐕_𝐢}_i=1^T instead of concatenating together as inputs, so as to facilitate object interactions across spatial domains and audio interactions across time domains, respectively. We denote the processed object queries as 𝐪^∗ and audio embedding as {𝐟^∗_𝐢}_i=1^T. To better judge when objects emit sound, we introduce an audio-aware cross-attention layer that performs cross-attention between 𝐪^∗ and {𝐟^∗_𝐢}_i=1^T, where the former is query and the latter is key/value. In this way, T-second audio is embedded into object queries to guide sound source localization temporally. Inspired by <cit.>, the audio-aware cross-attention layer is placed behind spatio-temporal cross-attention layer to avoid forgetting audio information as the decoder layer goes deep.



To detect the sounding objects from all categories while ignoring other non-sounding objects, a sound head is attached to the AudioMaskDec, which predicts the binary sounding scores 𝐒∈ℝ^N × 1, indicating if a query represents an sounding object:

    𝐒 = Softmax(MLP_sound(AudioMaskDec(𝐪)))

Due to the utilization of the sound head instead of a class head, object queries are class-agnostic, enabling our model to detect objects from all categories. This universal object proposal has been widely proved in previous open-vocabulary works <cit.>. For the mask prediction, the class-agnostic object queries are passed into a mask head and then dot-producted with the per-pixel embeddings 𝐩_𝐢, yielding the final masks {𝐌_𝐢}_i=1^T.

    𝐌_𝐢 = MLP_mask(AudioMaskDec(𝐪)) ⊛𝐩_𝐢, ∀ i ∈[1, T]




 §.§ Open-Vocabulary Classification
 
Once sounding object candidates are localized, we employ the frozen pre-trained vision-language model, i.e., CLIP <cit.>, to classify each candidate. Concretely, the category names with prompt templates are input into the CLIP text encoder CLIP_text to generate the text embeddings offline, such as:

    𝐄_𝐭𝐞𝐱𝐭 = CLIP_text(“This is a photo of a {violin}”)

where the category names contain both base and novel classes. Since CLIP model is trained on full sentences, we ensemble multiple prompt templates to improve classification accuracy. Our list of prompt templates is shown in Appendix.



In addition, we crop sounding image regions with the proposed masks 𝐌_𝐢^𝐧 and feed into the CLIP image encoder CLIP_image to compute image embeddings 𝐄_𝐢𝐦𝐚𝐠𝐞, where n represents the number of sounding objects. Inspired by <cit.>, we first overlay the mask 𝐌_𝐢^𝐧 onto the original image, resulting in the masked images. Then, we compute the object's center coordinate and the length of its longer side. Next, the masked images are cropped into a square-shaped images 𝐈_𝐢^𝐧. Lastly, we resize 𝐈_𝐢^𝐧 to 224 × 224 resolution required by the CLIP image encoder, such as ViT-B <cit.>. Compared with no cropping or simple cropping in <cit.>, this square cropping strategy can avoid objects occupying too small an area of the image and ensure 𝐈_𝐢^𝐧 without distortion.

After acquiring the text embeddings 𝐄_𝐭𝐞𝐱𝐭∈ℝ^cls × D_c and image embeddings 𝐄_𝐢𝐦𝐚𝐠𝐞∈ℝ^n × D_c, we compute the similarity matching map Sim ∈ℝ^n × cls between them as:

    Sim(i,j) = softmax(𝐄_𝐢𝐦𝐚𝐠𝐞^i · (𝐄_𝐭𝐞𝐱𝐭^j)^T ·ϵ)

where cls is the number of all categories, including base and novel classes. ϵ is the temperature hyper-parameter. The class label of each class-agnostic mask can be obtained by the argmax operation.






 §.§ Training and Loss

In the training phase, suppose there are K sounding objects in an image, we select K (K<N) object queries that best refers to the sounding objects via a bipartite matching strategy. Technologically, we use the sounding scores and mask predictions to compute the assignment cost metrics. The cost metrics is composed of two parts: one for segmentation that includes binary focal loss and dice loss, and the other is binary classification for sounding objects presence. Then the K best matched object queries {𝐪̂_𝐤}_k=1^K are applied for model training. Given the above matching, each prediction is supervised with a sound score loss and a mask loss. The former is binary cross-entropy loss and the latter consists of a focal loss and a dice loss. The total loss function between the query {𝐪̂_𝐤}_k=1^K and ground-truth {𝐲_k}_k=1^K can be written as: 

    ℒ(𝐪̂_𝐤, 𝐲_k) = λ_ceℒ_ce + λ_focalℒ_focal + λ_diceℒ_dice

where λ_ce=2.0, λ_focal=5.0, and λ_dice=5.0 are the weights to balance the loss function.




§ EXPERIMENTS



 §.§ Datasets and Evaluation Metrics

To facilitate a comprehensive evaluation of OV-AVSS, we partition a novel dataset called AVSBench-OV, derived from AVSBench-Semantic <cit.>, an extension of the initial AVSBench-object dataset <cit.> for semantic segmentation. AVSBench-Semantic encompasses 70 categories, incorporating both single and multiple sound source scenarios. Each video within AVSBench-Semantic is truncated to either 5 or 10 seconds in duration, with one frame extracted per second. Following the partitions in LVIS <cit.>, we split the categories within AVSBench-OV into 40 base categories representing those seen during training and inherited from frequent and common categories, and 30 novel (unseen and unheard) categories disjoint from the base categories. To ensure consistency, we eliminated sample videos from the training subset whenever a novel category appeared in annotations, thereby restricting the training data exclusively to base categories. The dataset statistics of AVSBench-OV are reported as follows: the training set encompasses 5,184 real-world videos comprising 40,095 frames across 40 base categories, while the validation and test sets consist of 1,240 and 1,490 videos across all 70 categories. For specific category names and corresponding video counts per category, please refer to the Appendix.

Following the previous open-vocabulary semantic segmentation task <cit.>, we adopt the mean intersection-over-union (mIoU) to evaluate the open-vocabulary AVSS performance. We compute overall mIoU, Base mIoU for seen categories, Novel mIoU for unseen categories, along with their harmonic mean (Harmonic). Harmonic is defined as follows:

    Harmonic=2 * Base * Novel/Base + Novel










 §.§ Implementation Details
    
We use both ResNet-50 <cit.> and Swin-B <cit.> as the image encoders and VGGish <cit.> as the audio encoder. Parameters in the audio encoder, CLIP image encoder and CLIP text encoder are frozen during the training. For each video clip, we set the total number of video frames T to 5 by default. The number of object queries is set to 100 for all experiments. We train our model on the AVSBench-OV dataset for 82,000 iterations (about 10 epochs) with a batch size of 1. We use the AdamW optimizer <cit.> and the step learning rate schedule. The initial learning rate is 1e-4 and scaled by a decay factor of 0.1 at the 72,171 iterations. If no other specification, the shorter side of frames are resized to 240, 360 or 480 during training. The experiments are conducted on an NVIDIA Quadro 6000 GPU.




 §.§ Comparison to State-of-the-art Methods

Zero-Shot Audio-Visual Segmentation. Given that our proposed open-vocabulary AVSS task employs a new division of base and novel categories, there are no baselines with identical training settings for an entirely fair comparison. Therefore, we use baselines from a similar but not identical tasks, specifically, the zero-shot AVS task. First, we consider GAVS <cit.>, which introduces an encoder-prompt-decoder paradigm to enhance the generalization of the AVS model by leveraging the prior knowledge of SAM <cit.>. However, due to the absence of CLIP or alternative foundation models, GAVS is limited to generate binary masks and cannot be applied to AVSS. Additionally, we compare with Sam4AVS <cit.>, which first employs CLAP <cit.> to predict categories of the input audio, and then feed into Grounding DINO <cit.> and SAM for mask prediction. A significant constraint of Sam4AVS stems from its dependency on the CLAP for class identification, which neglects the visual cues. This exclusion can result in inaccuracy classification for sounding objects. Also, Sam4AVS segments all instances of the predicted category, lacking the ability to identify specific sounding source. The original Sam4AVS is used for binary segmentation. We reproduce it and make it suitable for the open-vocabulary AVSS task. As shown in Table <ref>, our OV-AVSS model surpasses Sam4AVS in all metrics by a significant margin (+41.88% mIoU in base categories, +20.61% mIoU in novel categories, +27.73% harmonic mIoU and +32.34% overall mIoU). These substantial improvements can be attributed to our designed universal sound source localization and open-vocabulary classification modules, which effectively leverages both audio and visual information for accurate segmentation and categorization of both seen and unseen sounding objects.






Open-Vocabulary Audio-Visual Semantic Segmentation. As shown in Table <ref>, we first compare OV-AVSS with two closed-set trained methods, namely TPAVI <cit.> and CATR <cit.>. These methods adhere to the traditional paradigm based on the close-set assumption, which can only predict pre-defined categories that are present in the training set. Consequently, their performance on the novel set yields zero across all metrics. To provide a thorough comparison, we evaluate TPAVI with different backbones, including ResNet-50 and PVT-v2. The results demonstrate that even on the base categories, the performance of these closed-set methods falls short of our approach. Furthermore, we compare OV-AVSS to several state-of-the-art open-vocabulary video instance segmentation methods. These methods do not incorporate audio cues, making it challenging to localize sounding objects in frames. Our proposed OV-AVSS significantly outperforms the best-performing method, CLIP-VIS <cit.>, by +9.772% in base categories, +7.2% in novel categories, +8.84% in harmonic mIoU, and +8.6% in overall mIoU. These results highlight the effectiveness of our approach in leveraging both visual and audio information to achieve superior performance in open-vocabulary audio-visual semantic segmentation.


Qualitative Results. In Fig. <ref>, we present qualitative segmentation results of the proposed model on both base and novel categories. Our model can precisely localize the sounding object within each frame, e.g. “emergency car” in Fig. <ref> (c), while effectively omitting similar objects (vehicles) that do not emit sound. The multi-source scenarios depicted in Fig. <ref> (b) show our method's remarkable ability to accurately segment sounding objects not only from base categories but also from unseen categories, i.e. “violin”. This highlights the effectiveness of our approach in handling complex scenes containing a mix of seen and unseen sound sources. Despite the prevalence of “human” as the most easily confused sounding object in the training set, our method also accurately isolates silent humans, as evidenced by the examples shown in Fig. <ref> (a) and (b). 





 §.§ Ablation Study


Impact of Audio-Visual Early Fusion Strategy. The setting on the first row of Table <ref> serves as a baseline, where no early fusion is applied. While this baseline approach can generate reasonably good results for both base and novel categories, there is still room for improvement. The audio embeddings and processed image features, when used independently, may not capture sufficient semantic information to guide the localization process effectively. To address this limitation, we introduce an early fusion mechanism that allows us to acquire audio-aligned visual features and visual-enriched audio embeddings. The second row in Table <ref> presents a straightforward fusion approach, where audio embeddings are broadcast and added to the visual features in a pixel-wise manner. However, it is important to note that this fusion strategy is a one-way process, where the audio embeddings are used to enhance the visual features, but the audio embeddings themselves do not receive any visual priors to aid in sound source localization. To further improve the fusion mechanism, we introduce a bi-directional cross-attention mechanism, where each audio embedding can acquire corresponding semantic information from the visual features. Moreover, the cross-attention operation is much stronger than that of simple one-way summation, as it can adaptively focus on relevant regions in the visual features based on the audio embeddings. This brings extra performance increase as shown in the 4^th row of Table <ref>. This process is performed at multiple levels, allowing the model to incorporate audio information at different scales. 





Impact of Audio Prompt Method. For the baseline, we directly feed the object queries into the decoder, disregarding the audio embeddings obtained through our audio-visual early fusion. Due to the pixel-wise audio information incorporated into the image features through early fusion, this approach achieves promising results. However, it fails to fully utilize the temporal audio information, limiting its potential for enhancing performance. To overcome this obstacle, we concatenate the audio embeddings in the temporal dimension and add them to the object queries, similar to positional encoding. While this method yields modest improvements across all categories, it does not fully capture the rich time-dependent correspondence present in the audio modality. We further examine the impact of performing a single cross-attention operation between the object queries and audio embeddings before feeding them into the decoder. This approach yields further improvements in performance, highlighting the importance of effectively integrating audio and visual information. However, compared to our proposed AudioMaskDec, this audio prompting method may suffer from information loss as the decoder depth increases, limiting its ability to fully exploit the audio-visual synergy. In contrast, AudioMaskDec places an audio-aware cross-attention layer behind each spatio-temporal cross-attention layer in the decoder. This design enables the model to iteratively refine the integration of audio and visual features throughout the decoding process. By facilitating the interaction between object queries and audio embeddings at multiple layers, AudioMaskDec ensures that pertinent audio information is retained and effectively utilized before the sound head and mask head. As shown by the results in row 4 of Table <ref>, this approach further enhances the model's performance.




Impact of Image Crop Strategy. Previous work on zero-shot semantic segmentation <cit.> has shown that cropping an object's masked region by using its bounding box and resizing to a fixed size (e.g. 224 × 224) provides a more suitable input representation for CLIP compared to the full image. However, naively resizing the masked region can introduce significant aspect ratio distortion relative to the object's natural proportions. This unnatural stretching may degrade CLIP's zero-shot classification performance. We instead compute the center coordinate and longer side length of the object's bounding box, and extract a square crop around the object center, yielding a distortion-free representation. As shown in Table <ref> row 2 and 4, SquareCrop approach outperforms the CropResize by a sizeable margin across all metrics, including a +2.16% gain on base classes, +1.28% on novel classes, +1.65% on harmonic mean, and +1.77% on mIoU. Furthermore, we investigate the impact of using different visual encoders of CLIP. As shown in the 3^rd row of Table <ref>, changing the visual encoder does not yield further improvements in performance. This suggests that the classification effect of CLIP is largely dependent on the segmentation quality of the universal module and the input image quality, rather than the choice of different visual encoder.





§ CONCLUSION

This paper introduces a new task of open-vocabulary AVSS with the goal of segmenting and classifying sounding objects of arbitrary categories in a given video. Furthermore, we present the first open-vocabulary AVSS framework, termed OV-AVSS, generalizing from annotated (seen) object classes to other (unseen) categories with the knowledge of CLIP. Quantitative and qualitative experiments on AVSBench-OV dataset show strong zero-shot generalization ability on novel categories unseen during training. We hope that our proposed OV-AVSS model can further promote the generalization study of audio-visual segmentation in zero-shot, open-vocabulary and real-world scenarios. As a pioneering work, the current approach is not perfect. We outline the limitations of the model. 1) Base classes over-fitting: Our model may easily overfit the base ones when both classes share similar appearances or sounds, since these classes are trained with higher confidence scores. 2) Real-world challenges: When audio-visual inconsistency and noisy background disturbances are involved, our model’s performance may be limited, leading to inaccurate segmentation results.




This work was supported by the National Natural Science Foundation of China (NSFC) (No. 62371009 and No. 61971008).



ACM-Reference-Format





 Paper Title: EUDA: An Efficient Unsupervised Domain Adaptation via Self-Supervised Vision Transformer.








Preprint


EUDA: An Efficient Unsupervised Domain Adaptation via Self-Supervised Vision Transformer
    Ali Abedi, Graduate Student Member, IEEE,
        Q. M. Jonathan Wu, Senior Member, IEEE,
        Ning Zhang, Senior Member, IEEE,
        Farhad Pourpanah, Senior Member, IEEE
A. Abedi, Q. M. J. Wu, and N. Zhang are with the Department of Electrical and Computer
Engineering, University of Windsor, Windsor, ON N9B 3P4, Canada (e-mail:
abedi3@uwindsor.ca, jwu@uwindsor.ca, ning.zhang@uwindsor.ca).
F. Pourpanah is with the Department of Electrical and
Computer Engineering, Queen’s University, Kingston, ON K7L 3N6, Canada (e-mail: f.pourpanahnavan@queensu.ca).
Manuscript
    August 5, 2024
=================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Unsupervised domain adaptation (UDA) aims to mitigate the domain shift issue, where the distribution of training (source) data differs from that of testing (target) data. 
Many models have been developed to tackle this problem, and recently vision transformers (ViTs) have shown promising results. However, the complexity and large number of trainable parameters of ViTs restrict their deployment in practical applications. This underscores the need for an efficient model that not only reduces trainable parameters but also allows for adjustable complexity based on specific needs while delivering comparable performance.
To achieve this, in this paper we introduce an Efficient Unsupervised Domain Adaptation (EUDA) framework.
EUDA employs the DINOv2, which is a self-supervised ViT, as a feature extractor followed by a simplified bottleneck of fully connected layers to refine features for enhanced domain adaptation. Additionally, EUDA employs the synergistic domain alignment loss (SDAL), which integrates cross-entropy (CE) and maximum mean discrepancy (MMD) losses, to balance adaptation by minimizing classification errors in the source domain while aligning the source and target domain distributions. The experimental results indicate the effectiveness of EUDA in producing comparable results as compared with other state-of-the-art methods in domain adaptation with significantly fewer trainable parameters, between 42% to 99.7% fewer. This showcases the ability to train the model in a resource-limited environment. The code of the model is available at: https://github.com/A-Abedi/EUDA.




Unsupervised domain adaptation, vision transformers, Max mean discrepancy, self-supervised Learning













§ INTRODUCTION

Unsupervised domain adaptation (UDA) has shown promising results in addressing the domain shift problem, where the distribution of the training data (source domain) differs from that of the test data (target domain), and knowledge transfer <cit.>. The domain shift problem can significantly degrade the performance of conventional deep neural networks (DNNs) when applied directly to the target domain <cit.>. UDA overcomes this issue by leveraging unlabeled data from the target domain to align feature distributions and enhance model adaptation <cit.>.
Additionally, traditional DNNs often rely on a large number of annotated samples, which limits their applicability in conditions where labeling is expensive, such as autonomous driving <cit.>. UDA solves this problem by transferring knowledge from previously labeled data (source domain) to unlabeled data in the target domain, thereby enhancing model performance without the need for extensive new annotations. 

Existing UDA methods can be classified into: (i) adversarial methods, which use generative adversarial networks to generate domain-invariant features <cit.>; (ii) reconstruction methods, which employ encoder-decoder structures to learn invariant features by reconstructing domains <cit.>; (iii) transformation methods, which optimize inputs through transformations to enhance adaptation <cit.>; and (iv) discrepancy methods, which align domain features through statistical metrics like maximum mean discrepancy (MMD) <cit.> and correlational alignment (CORAL) <cit.>. 
Regardless of which UDA method is chosen, the backbone of the network for feature extraction plays a crucial role. The backbone is responsible for extracting features from the input data, and the more generalized and domain-invariant these features are, the better the UDA results will be. Recently, vision transformers (ViTs) <cit.>  have revolutionized computer vision by adapting the transformer architecture, originally designed for natural language processing <cit.>, to process images as sequences of patches. This approach enables ViTs to capture global image dependencies, offering a more comprehensive understanding than the local feature extraction typical of convolutional neural networks (CNNs) <cit.>. This capability makes ViTs well-suited for complex tasks such as object recognition and scene understanding <cit.> and particularly effective in UDA <cit.>. 

The success of ViTs in large-scale pre-training also shows significant performance improvements on benchmarks, particularly in UDA, making the integration of self-supervised learning (SSL) strategies effective <cit.>. SSL allows ViTs to learn detailed representations from vast, unlabeled datasets, enhancing generalization and robustness across various visual domains <cit.>. Among these methods, DINOv2 <cit.> stands out for its ability to extract general-purpose features <cit.>. This positions DINOv2 as a powerful tool for advancing UDA, highlighting its transformative potential in modern visual computing.

Despite the promising performance of the state-of-the-art UDA models, they often follow complex frameworks. Particularly those utilizing ViT architectures as backbone, which involve a significant amount of learnable parameters. These parameters need powerful computational resources for training. This complexity not only increases the cost of deployment but also limits their applicability in environments with limited resources.
Therefore, there is a critical need for UDA models that are inherently simpler and have fewer trainable parameters. Such models would ensure more stable training and better generalization, and offer the versatility to adjust their complexity based on specific task demands and the complexity of various applications. By achieving this, we can facilitate more efficient domain adaptation across a wider range of practical settings, making advanced UDA techniques more accessible and sustainable.

In this paper, we introduce a novel Efficient Unsupervised Domain Adaptation framework, known as EUDA, that utilizes the capabilities of self-supervised ViTs to address the challenges of UDA. Specifically, EUDA integrates DINOv2 as a feature extractor with a simple yet effective bottleneck consisting of fully connected layers. 
Using the fully connected layers aims to refine the features extracted by DINOv2 for effective domain adaptation. Additionally, inspired by <cit.> EUDA uses the synergistic domain alignment loss (SDAL), which combines cross-entropy (CE) loss and MMD loss. This hybrid loss function effectively balances the adaptation by minimizing classification errors on the source domain while aligning the distributions between the source and target domains, as shown in Fig. <ref>. By leveraging the self-supervised pre-training of DINOv2 and the computational efficiency of the MMD, our approach offers a promising solution to UDA. It reduces the dependency on large, complex models and extensive computational resources, making it not only effective but also suitable for real-world applications where adaptability and efficiency are crucial. 
To the best of our knowledge, this is the first application of DINOv2 within a UDA framework, marking a significant advancement in leveraging self-supervised learning techniques for addressing domain adaptation challenges.



In summary, the main contributions of this paper are as follows:


    
  * To the best of our knowledge, we are the first to apply DINOv2, a self-supervised vision transformer, for UDA, utilizing its strength to generate robust, domain-invariant features.
    
  * We propose EUDA (Efficient Unsupervised Domain Adaptation), a novel framework that leverages self-supervised ViTs and MMD to tackle the challenges of UDA. EUDA features a flexible architecture with a simple bottleneck of fully connected layers, which require significantly fewer parameters to be tuned and can be adjusted according to the complexity of the problem. 
    
  * We perform comprehensive experiments and ablation studies to validate the effectiveness of our proposed method on multiple benchmarks.


The rest of this paper is organized as follows: Section <ref> reviews UDA, ViTs, and self-supervised learning approaches within ViTs. Section <ref> details the proposed EUDA model. The experimental results across various configurations are presented in Section <ref>. Finally, Section <ref> concludes the paper with insights and implications of our findings.



§ RELATED WORK




 §.§ Unsupervised Domain Adaptation


As stated in the introduction, the domain shift problem is the major concern in UDA. Over the years, numerous methods have been developed to tackle this issue, aiming to enable effective model performance on the target domain without relying on labeled data <cit.>. 
Adversarial methods use adversarial training to create domain-invariant features. For example, DANN <cit.> leverages a gradient reversal layer that inverts the gradient sign during training. ADDA <cit.> trains a source encoder on labeled images, and mixes source and target images to confuse the discriminator. Adversarial methods are computationally intensive, often exhibit unstable training processes, and do not always guarantee accurate feature mapping.

Transformation methods enhance domain adaptation by pre-processing input samples to optimize their condition for model training. DDA <cit.> preprocesses input data to align signal-to-noise ratios and reduce domain shifts, while TransPar <cit.> applies the lottery ticket hypothesis to identify and adjust transferable network parameters for better cross-domain generalization. These approaches are simple and effective but may not capture all domain differences.

Reconstruction methods utilize an encoder-decoder setup to harmonize features across domains by reconstructing target domain images from source domain data. MTAE <cit.> utilized a multitask autoencoder to reconstruct images from multiple domains. DSNs <cit.> enhance model performance by dividing image representations into domain-specific and shared subspaces. This improves generalization and surpasses other adaptation methods. They usually face challenges with high computational costs, training instability, and potential overfitting to the source domain.

Discrepancy methods have emerged as particularly effective for UDA. DAN <cit.> embeds task-specific layer representations into a reproducing kernel Hilbert space (RKHS) and uses MMD to explicitly match the mean embeddings of different domain distributions. WeightedMMD <cit.> introduces a weighted MMD that incorporates class-specific auxiliary weights to address class weight bias in domain adaptation. This approach optimizes feature alignment between source and target domains by considering class prior distributions. Joint adaptation networks <cit.> align the joint distributions of multiple domain-specific layers across domains using a Joint MMD criterion to improve domain adaptation by considering the combined shift in input features and output labels.



 §.§ Vision Transformer


Transformers, initially introduced by Vaswani et al. <cit.>, have demonstrated exceptional performance across various language tasks. The core of their success lies in the attention mechanism, which excels at capturing long-range dependencies. 
ViT <cit.> represents a groundbreaking approach to applying transformers in vision tasks. It treats images as sequences of fixed-size, non-overlapping patches. Unlike CNNs that depend on inductive biases such as locality and translation equivariance, ViT leverages the power of large-scale pre-training data and global context modeling. ViT offers a straightforward yet effective balance between accuracy and computational efficiency <cit.>.

In the context of UDA, ViTs have demonstrated remarkable potential. TVT <cit.> introduces a transferability adaptation module to guide the attention mechanism and a discriminative clustering technique to enhance feature diversity. CDTrans <cit.> consists of a triple-branch structure with weight-sharing and cross-attention to align features from source and target domains, alongside a two-way center-aware pseudo labeling strategy to improve label quality. WinTR <cit.> uses two classification tokens within a transformer model to learn distinct domain mappings with domain-specific classifiers. This enhances the cross-domain knowledge transfer through source-guided label refinement and single-sided feature alignment. PMTrans <cit.> combines patches from both domains using game-theoretical principles, mixup losses, and attention maps for effective domain alignment and feature learning.

While these methods have shown promising performance in solving UDA problems, they typically rely on complex architectures with extensive trainable parameters and sophisticated training regimes, including multi-branch transformers, cross-attention, adversarial training, game-theoretical principles, and mixup losses. Furthermore, these models generally require training the entire network, resulting in a substantial computational burden. As a result, achieving promising outcomes necessitates extensive training on large-scale models, limiting their practical applicability in resource-constrained environments <cit.>.



 §.§ Self-supervised Vision Transformer

SSL has revolutionized the field of computer vision by enabling models to learn effective representations from unlabeled data, eliminating the dependency on large annotated datasets. These models learn representations by performing pre-text tasks, such as rotation prediction <cit.> and image colorization <cit.>, and then apply the learned representations to downstream tasks.
In the context of ViTs, SSL has been pivotal in enhancing their ability to extract robust, domain-invariant features. Jigsaw-ViT <cit.> integrates the jigsaw puzzle-solving problem into vision transformer architectures for improving image classification. EsViT <cit.> utilizes a multi-stage Transformer architecture to reduce computational complexity and introduces a novel region-matching pre-training task.


Among recent advancements in SSL for ViTs, DINO <cit.> and DINOv2 <cit.> have notably enhanced SSL by scaling up ViTs to effectively match representations across different views of the same image. With improvements like automatic data curation and innovative loss functions, DINOv2 excels in stability and efficiency. It can learn domain-invariant features essential for image classification and other vision tasks. Its capacity to generate robust feature maps makes DINOv2 an excellent choice for a feature extractor and representation generator. Because of these properties, we use DINOv2 in this research to significantly boost performance and generalization across various domains.




§ METHODOLOGY








 §.§ Problem Formulation

UDA aims to learn a function f: 𝒳→𝒴 that performs well on an unlabeled target domain by leveraging information from a labeled source domain. Let 𝒳 and 𝒴 denote the input and label spaces, respectively, 𝒟_s = {(x_i^s, y_i^s)}_i=1^n_s indicates the source domain data, where x_i^s ∈𝒳 and y_i^s ∈𝒴 represent the i^th input-output pairs, and 𝒟_t = {x_j^t}_j=1^n_t indicates the target domain data, where x_j^t ∈𝒳 is the j^th input samples without labels, n_s and n_t are the number of samples in the source and target domains, respectively. The goal of UDA is to train a model on labeled source data 𝒟_s and unlabeled target data 𝒟_t in such a way that it performs well on the predicting target data labels y_i^t.





 §.§ Model Overview

Fig. <ref> shows an overview of our framework for addressing UDA problems. 
The model receives images from labeled source and unlabeled target domains. It then extracts features using a pre-trained self-supervised ViT, while its weights are frozen to ensure stability and efficiency. The extracted features from both domains are fed into a bottleneck composed of multiple fully connected layers to refine and condense the information.
The bottleneck's output serves a dual purpose. Firstly, it inputs into the classification head to compute the CE loss on source domain.
Secondly, it contributes to the computation of the MMD loss to minimize the distance between source and target domain distributions in RKHS to align their feature representations. The pseudocode for our training procedure of EUDA is outlined in Algorithm <ref>. In the following subsections, we present different components of our model in detail.





 §.§ Feature Extractor

In our effort to leverage the capabilities of ViTs for domain adaptation, we adopt DINOv2 <cit.> as a self-supervised pre-trained model for feature extraction. 
It utilizes self-distillation to derive insights from unlabeled data autonomously. Central to DINOv2's design is its twin-network structure, which includes a student and a teacher network. Both networks employ the same underlying architecture based on ViTs. During training, these networks process 
different augmentations of the same image. They aim to extract consistent features regardless of the input variations.

During the training phase (see Fig. <ref>), the student network's parameters are continually updated, while the teacher network's parameters are progressively updated through an exponential moving average of the student's parameters. This ensures that the teacher model remains robust and generalizable. Moreover, DINOv2 uses registers <cit.> to improve the performance and interpretability of ViTs by addressing the problem of artifacts in feature maps, commonly observed in both supervised and self-supervised ViT networks. Registers are additional tokens added to the input sequence of ViTs to absorb high-norm token computations that typically occur in low-information areas of images. 

In our study, we leverage DINOv2 as the primary feature extractor due to its robust training on a large-scale, diverse dataset through self-supervised learning. To enhance efficiency, we freeze the model's parameters. This approach reduces the computational burden and significantly decreases the number of trainable parameters. This makes our method notably more efficient compared to other UDA techniques that require extensive training. Consequently, our streamlined model is well-suited for deployment in real-world scenarios and on-edge devices, where computational resources are often limited. Fig. <ref> shows the attention map of the pre-trained DINOv2 base model without any fine-tuning. This highlights its robustness across four different domains of the office-home <cit.> dataset. This demonstrates the precision and accuracy of the features extracted by DINOv2, underscoring the effectiveness of the EUDA feature extraction approach.

Using DINOv2 as the feature extractor significantly enhances our model by employing its robust, self-supervised pre-training to extract general-purpose features from images. The pre-trained DINOv2, with its weights frozen, ensures the extraction of high-quality features and reduces the number of trainable parameters. It can greatly simplify integration and adaptation in resource-limited settings, providing a strong foundation for effective domain adaptation.





 §.§ Bottleneck


The bottleneck component in our architecture consists of a series of fully connected layers. At the core of our model, the bottleneck output serves two primary functions. Firstly, it feeds into a classification head composed of a simple linear layer. This setup utilizes minimal computational resources to classify images efficiently. Secondly, the output acts as the feature vector for calculating the MMD loss, which aims to minimize the distance between source and target feature vectors. This facilitates effective domain adaptation.

The design of this dual-function bottleneck simplifies our model architecture and enhances its generalization capabilities across various domains. Its straightforward structure helps prevent overfitting, a frequent challenge in domain-specific applications, ensuring robustness for real-world deployments. Additionally, the minimalistic approach in the bottleneck design allows for easier maintenance and adaptability, which is crucial for meeting the dynamic requirements of domain adaptation tasks.

The bottleneck follows a simple yet efficient design for flexible adjustments in the model’s complexity. This flexibility allows us to adjust the model to meet specific performance needs or computational limits, which is particularly useful in settings with restricted resources.



 §.§ Synergistic Domain Alignment Loss

The SDAL is a composite loss function designed for the UDA models. It combines the strengths of CE loss and MMD loss. Formally defined as:

    λℒ_CE + (1-λ) ℒ_MMD,

where ℒ_CE and ℒ_MMD are the CE and MMD losses, respectively, and λ is a tunable hyperparameter that balances the influence of each loss component, it allows for a flexible adjustment according to specific domain adaptation needs. 

CE loss is a widely used loss function in classification tasks.  
The CE loss increases as the predicted probability diverges from the actual label, thus providing a robust metric for optimizing classification models. The mathematical formulation of CE loss for a multi-class classification task is given by:

    ℒ_CE = -1/N∑_i=1^N ∑_c=1^C y_i,clog(ŷ_i,c),

where N is the number of samples, C is the number of classes, y_i,c is a one-hot vector indicating whether class label c is the correct classification for the i^th sample, and ŷ_i,c is the predicted probability of the i^th sample belonging to class c. This formula penalizes the deviation of each predicted class probability from the actual class labels.


MMD loss is defined as the squared distance between the mean embeddings of features from the two domains in an RKHS <cit.>. The MMD loss can be written as:

    ℒ_MMD = MMD^2(𝒟_s, 𝒟_t) = 
    1/n_s∑_i=1^n_sϕ(x_i^s) - 1/n_t∑_j=1^n_tϕ(x_j^t) ^2_ℋ,

where n_s and n_t denote the number of samples in the source and target domains, respectively, x_i^s and x_j^t are the data samples from these domains, and ϕ: 𝒳→ℋ is a feature mapping function that projects the data into RKHS ℋ. By minimizing the MMD loss, we aim to align the statistical properties of the source and target domains.






The loss defined in Eq. (<ref>) leverages the labeled data in the source domain to fine-tune the classification performance via CE loss while aligning the distribution of the source and target domain features through MMD. 
The synergy between these two loss components enhances the model's ability to perform effectively across different domains. This makes SDAL suitable for real-world applications where domain shift is a significant challenge.
Additionally, minimizing the distance between the source and target domain distributions ensures that the feature representations from both domains are closely aligned, facilitating smoother and more reliable domain adaptation. Empirical studies have shown that MMD effectively reduces domain shift and improves model performance across various tasks <cit.>.



§ EXPERIMENTAL STUDIES





 §.§ Datasets

We examine the effectiveness of our proposed method across three benchmarks. These datasets include: 
Office-31 <cit.>, which consists of 4,652 images from three domains: Amazon, Webcam, and DSLR, across 31 categories; 
Office-Home <cit.>, which consists of 15,500 images spread over four domains: Art, Clipart, Products, and Real World, within 65 categories;
VisDA-2017 <cit.>, which includes 280,000 images from four domains: Caltech-256, ImageNet, ILSVRC2012, and PASCAL VOC 2012 as the source domain and real images as the target, belonging to 12 categories; and DomainNet <cit.>, which consists of 48,129 images from six domains: Clipart, Real, Sketch, Infograph, Painting, and Quickdraw, across 345 categories. 











 §.§ Evaluation

To evaluate the effectiveness of our UDA model, following the procedure in <cit.>, We train our model by alternately using each domain as the source with labeled data and another as the target with unlabeled data, then evaluate using labeled data from the target domain. This procedure is repeated until all domains have been used as the source domain. This setup ensures that the model's ability to generalize to new, unseen environments is properly tested. The primary metric for evaluation is accuracy, specifically how accurately the model classifies new samples when trained on the source domain and tested on the target domain. This metric provides a clear measure of the model's performance in bridging the gap between disparate data distributions of the source and target domains.



 §.§ Baselines

We compare our model against a broad spectrum of state-of-the-art models. We categorize these models into ResNet- and ViT-based models. 
The ResNet-based models include RevGrad <cit.>, CDAN <cit.>, TADA <cit.>, SHOT <cit.>, JAN <cit.>, BNM <cit.>, MCD <cit.>, SWD <cit.>, and DTA <cit.>. While ViT-based models include CDTrans <cit.>, PMTrans <cit.>, and TVT <cit.>.



 §.§ Implementation Details

In our domain adaptation model, we designed the bottleneck component with varying complexities to assess how architectural depth influences feature processing. Our configurations ranged from a simple single-layer network with 256 neurons (designated S for Small) to more complex setups like 2048-1024-512-256 (B for Base model), 4096-2048-1024-512-256 (L for Large model), and 8192-4096-2048-1024-512-256 (H for Huge model). We also employed different sizes of the DINOv2 model, base and large, as feature extractors to examine their effects on domain adaptation performance. We used a naming convention in XY format for clarity, where X represents the feature extractor size (B for base and L for large) and Y the bottleneck size. For instance, BB indicates that both the feature extractor and bottleneck are base size.

To optimize domain alignment and classification accuracy, we experimented with different values of the hyperparameter λ, and based on the experiments conducted on Section <ref>, the value of 0.7 was selected for further experiments. Adjustments were made to batch sizes and learning rates according to the model size to ensure computational efficiency on a 1080Ti GPU, with batch sizes set to 32, 16, and 8, and learning rates starting at 3e-2 and gradually decreasing.






 §.§ Comparison with the baseline models

Performance. Tables <ref>, <ref>, <ref> and <ref> show the accuracy rates of EUDA and baseline models on Office-Home, Office-31, VISDA-2017 and DomainNet, respectively. 
As can be seen, our model consistently outperformed all ResNet-based models across all datasets. Specifically, for the Office-Home dataset, our LL model surpassed CDTrans and TVT while delivering comparable results to PMTrans. For the Office-31 dataset, our model exceeded the performance of TVT and matched that of CDTrans and PMTrans. In the case of the VISDA-2017 dataset in Table <ref>, our model performed closely to other SOTA models. For the DomainNet dataset, our findings were particularly striking. Despite the large number of classes the dataset contains, our smallest model configuration, the BS, outperformed most of the SOTA models, including CDTrans, and delivered results comparable to PMTrans. 




Learnable parameters. Table <ref> compares the number of learnable parameters of our proposed EUDA model with baseline models.  EUDA requires significantly fewer trainable parameters compared to ViT-based baseline models while still achieving competitive performance. Specifically, our best model uses approximately 83% fewer parameters for the Office-Home and Office-31 datasets, 43% fewer parameters for the VISDA-2017 dataset, and an impressive 99.7% fewer parameters for the DomainNet dataset. This substantial reduction in model complexity highlights the efficiency of our approach, making it more suitable for deployment in resource-constrained environments without compromising accuracy. 
This indicates a higher efficiency in our modeling approach. EUDA employs a pre-trained DINOv2 model with frozen parameters as a feature extractor, i.e., there are no learnable parameters in this component, and learnability is confined to the bottleneck layers, the normalization layers associated with the feature extractor, and the classification heads. This streamlines the training process and reduces computational overhead, making EUDA particularly suitable for applications where resource efficiency is critical. This efficiency combined with the demonstrated effectiveness of our model in adapting to various domains underscores the practical advantages of our approach in real-world settings.




 §.§ Ablation Study

We test the effectiveness of the SDAL loss on the performance of EUDA model.
To achieve this, we remove MMD loss from Eq. (<ref>), i.e., the model only relies on the source domain and excludes learning from the target domain data. In Table <ref>, we assessed the performance of our BS and BL configurations on the Office-Home dataset using a source-only approach. The results clearly show that incorporating SDAL led to 1.2% and 3.5% improvements in the BS and BL configurations, respectively. Similarly, on the Office-31 dataset (see Table <ref>), our BB configuration improved by +0.4, and BL improved by +0.6 when using MMD loss. For the VisDA-2017 dataset (see Table <ref>), implementing SDAL with our BB and BL models perform approximately 8.1% and 7.4%, respectively better than only using CE loss.



 §.§ Sensitivity Analysis



  §.§.§ Effect of λ


Table <ref> shows the impact of λ in Eq. (<ref>) on the BS configuration for the Office-Home dataset.  In this experiment, we used four different values for λ: 0.3, 0.5, and 0.7. As can be seen, λ = 0.7 managed to produce the best results. The effectiveness of λ = 0.7 comes from its balanced approach that minimizes domain discrepancies through MMD loss while maintaining classification accuracy, ensuring robust performance across various domain shifts.



  §.§.§ Performance of the EUDA Framework Under Various Configurations

In this experiment, we test our model in different configurations across various datasets. Our goal is to find which model best suits each dataset. Our findings indicate that while more complex models perform better on complex datasets, our simpler models, which have significantly fewer trainable parameters, can also achieve comparable results. This flexibility allows users to adjust the model's complexity to match their datasets' specific requirements. This adaptive capability is a distinctive feature of our approach which provides a unique advantage by offering a scalable solution that adjusts to varying data complexities without sacrificing performance. 

Table <ref> demonstrates our model’s performance on the Office-Home dataset with B and L feature extractors and S, B, L, and H bottleneck configurations. Our testing on Office-Home helped us identify optimal configurations, which reducing the need for extensive trials on other datasets. The L feature extractor was notably effective due to its ability to handle the significant domain variance within the dataset, and benefits from higher-dimensional features that capture more informative details. The LL configuration provides the best balance of complexity and performance.

Table <ref> shows the results of Office-31 dataset using B and L feature extractors and B and L bottleneck configurations. It can be seen that LL configuration produces the best results. Insights from the Office-Home tests informed the decision not to use the H bottleneck, as higher complexity had not resulted in improved performance in previous tests.

For the VisDA-2017 dataset (see Table <ref>), the BH configuration stood out, particularly suited to managing the transition from simulation to reality. This confirms the benefit of a more complex bottleneck in handling extensive domain shifts between real and simulation data. This emphasizes the importance of matching architectural choices to specific dataset challenges.









While we have conducted extensive testing across multiple configurations and datasets, time constraints limited the breadth of our experiments. We therefore encourage researchers and practitioners to further explore various bottleneck configurations to meet the specific demands and complexities of their datasets, enabling customized solutions that optimally address their unique challenges.



§ CONCLUSION


In this paper, we highlighted the potential of using a self-supervised pre-trained ViT for UDA by introducing a versatile framework that maintains simplicity, efficiency, and adaptability to ensure its applicability in practical scenarios. Specifically, we leveraged DINOv2, a self-supervised learning method in ViTs,
to extract general features, and employed a simple yet effective bottleneck of fully connected layers to refine the extracted features.
Additionally, we utilized the MMD loss to effectively align the source and target domains. 
Our model produces comparable results to state-of-the-art UDA methods
with significantly fewer trainable parameters. This makes our method particularly suitable for real-world applications, including on-edge devices.
Our proposed framework demonstrates promising results, achieving top-tier performance with 43% to 99.7% fewer trainable parameters across benchmark datasets compared to other methods. In our future research, we will explore additional UDA approaches based on self-supervised pre-trained ViT backbones and expand the applications of UDA in fields such as Autonomous Vehicles and other demanding areas where UDA is crucial.




IEEEtran



    [
    < g r a p h i c s >
]Ali Abedi (Graduate Student Member, IEEE) received his B.Sc. in Computer Software Engineering in 2019 and completed his M.Sc. in Artificial Intelligence Engineering at the University of Isfahan in 2022. 
    He has accumulated extensive academic and industrial experience in the fields of artificial intelligence and machine learning. Currently, he is pursuing a Ph.D. in Electrical and Computer Engineering at the University of Windsor. His primary research interests include deep learning, machine learning, and computer vision, with a particular focus on transformers architecture and vision transformers.



    [
    < g r a p h i c s >
]Q. M. Jonathan Wu (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from the University of Wales, Swansea, U.K., in 1990. He was affiliated with the National Research Council of Canada for ten years beginning, in 1995, where he became a Senior Research Officer and a Group Leader. He is currently a Professor with the Department of Electrical and Computer Engineering, University of Windsor, Windsor, ON, Canada. He has authored or coauthored more than 300 peer-reviewed papers in computer vision, image processing, intelligent systems, robotics, and integrated microsystems. His research interests include 3-D computer vision, active video object tracking and extraction, interactive multimedia, sensor analysis and fusion, and visual sensor networks. Prof. Wu held the Tier 1 Canada Research Chair in automotive sensors and information systems from 2005 to 2019. He is an Associate Editor for IEEE TRANSACTIONS ON CYBERNETICS, the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, the Journal of Cognitive Computation, and Neurocomputing. He was on technical program committees and international advisory committees for many prestigious conferences. He is a Fellow of the Canadian Academy of Engineering.



    [
    < g r a p h i c s >
]Ning Zhang (Senior Member, IEEE) is an Associate Professor in the Department of Electrical and Computer Engineering at University of Windsor, Canada. He received the Ph.D degree in Electrical and Computer Engineering from University of Waterloo, Canada, in 2015. After that, he was a postdoc research fellow at University of Waterloo and University of Toronto, respectively. His research interests include connected vehicles, mobile edge computing, wireless networking, and machine learning. He is a Highly Cited Researcher. He serves as an Associate Editor of IEEE Transactions on Mobile Computing, IEEE Internet of Things Journal, IEEE Transactions on Cognitive Communications and Networking, and IEEE Systems Journal; and a Guest Editor of several international journals, such as IEEE Wireless Communications, IEEE Transactions on Industrial Informatics, and IEEE Transactions on Intelligent Transportation Systems. He also serves/served as a TPC chair for IEEE VTC 2021 and IEEE SAGC 2020, a general chair for IEEE SAGC 2021, a track chair for several international conferences and workshops. He received 8 Best Paper Awards from conferences and journals, such as IEEE Globecom and IEEE ICC.



    [
    < g r a p h i c s >
]Farhad Porpanah (Senior Member, IEEE) received the Ph.D. degree in computational intelligence from the University of Science Malaysia (USM), Malaysia, in 2015. He is currently a Postdoctoral Research Fellow with the Department of Electrical and Computer Engineering, Queens University, Kingston, ON, Canada. Prior to his current position, he was an Associate Research Fellow with the Department of Electrical and Computer Engineering, University of Windsor, Windsor, ON, Canada. From 2019 to 2021, he held the position of Associate Research Fellow with the College of Mathematics and Statistics, Shenzhen University (SZU), Shenzhen, China. Before his tenure at SZU, he was a Postdoctoral Research Fellow with the Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. His research interests include machine learning and deep learning, driven by an emphasis on creating smart environments that improve the quality of life for the elderly population.




 Paper Title: Multi-Site Class-Incremental Learning with Weighted Experts in Echocardiography.





Multi-Site Class-Incremental Learning with Weighted Experts




Bransby et al.


Ultromics Ltd., Oxford, United Kingdom

 agis.chartsias@ultromics.com 
Queen Mary University of London, United Kingdom

 k.m.bransby@qmul.ac.uk 


Multi-Site Class-Incremental Learning with Weighted Experts in Echocardiography
    Kit M. Bransby1,2, Woo-Jin Cho Kim1, Jorge Oliveira1, Alex Thorley1, Arian Beqiri1, Alberto Gomez1, Agisilaos Chartsias1
    August 5, 2024
============================================================================================================================



§ ABSTRACT 
Building an echocardiography view classifier that maintains performance in real-life cases requires diverse multi-site data, and frequent updates with newly available data to mitigate model drift. Simply fine-tuning on new datasets results in “catastrophic forgetting”, and cannot adapt to variations of view labels between sites. Alternatively, collecting all data on a single server and re-training may not be feasible as data sharing agreements may restrict image transfer, or datasets may only become available at different times. Furthermore, time and cost associated with re-training grows with every new dataset. We propose a class-incremental learning method which learns an expert network for each dataset, and combines all expert networks with a score fusion model. The influence of “unqualified experts” is minimised by weighting each contribution with a learnt in-distribution score. These weights promote transparency as the contribution of each expert is known during inference. Instead of using the original images, we use learned features from each dataset, which are easier to share and raise fewer licensing and privacy concerns. We validate our work on six datasets from multiple sites, demonstrating significant reductions in training time while improving view classification performance.  






§ INTRODUCTION


Echocardiography (echo) view classification is often a necessary first step in automated image interpretation and analysis, as different tasks may require different views as input <cit.>. Several deep learning methods have been proposed to this end <cit.> demonstrating excellent performance. To generalise and maintain performance in real-life cases, view classifiers need to be trained on a diverse multi-site dataset to accommodate varying acquisition, demographic, and clinical parameters. However, these parameters may change over time, e.g. with machine upgrades, modification of protocols, or changes in demographics. Such changes require updating the classifier with newly available data to mitigate model drift <cit.>. Naive approaches like fine-tuning on a new dataset may result in “catastrophic forgetting” <cit.>, in which previous knowledge is forgotten at the expense of the new information. Retaining previous knowledge while learning on new data is the goal of incremental-learning <cit.>. 

Furthermore, newly available data (e.g. from sites with different protocols) may have different sets of view labels. Labels can be characterised as “base”, “novel”, or “overlapping”, depending on whether they are present in the original data, the new data, or both. The goal of class-incremental learning (CIL) <cit.> is to learn to classify the increasing label set over time. A straightforward solution is to retrain using all data combined. However, this is not always feasible as data sharing agreements may limit access or transfer outside the acquisition site; or different datasets may be available at different times. In addition, retraining a classifier when a new dataset becomes available is inefficient as the total training time and cost grows considerably with the number and size of datasets.

A common approach for CIL is to use a single model, and update parts or all of the weights using knowledge distillation <cit.> or weight regularisation <cit.>. Others learn additional parameters with dynamic architecture changes <cit.>, or by duplicating the model for novel data and pruning <cit.>. Wu et al. <cit.> expand on this by duplicating and fine-tuning a base model for each new dataset, resulting in set of expert model branches, each specialised on a single dataset. They combine the output logits of the independently trained branches using a learnt score fusion (SF) network that enables knowledge transfer between base and novel classes.
This leverages shared information between representations in different branches, but is sensitive to one branch contributing ineffective information to another branch. We term this the “unqualified expert” problem and may happen when there are differences in data distributions, caused for instance by different label sets, patient demographics or scanner manufacturers. As a result view predictions for out-of-distribution (o.o.d.) images can be incorrect, but also highly confident. 



In this paper we address the problem of building an echo view classifier model with multiple datasets from different sites (see Fig. <ref>) that are not simultaneously available and have different, overlapping sets of labels. To this end, we build upon <cit.>, and
mitigate the “unqualified expert” problem by weighing the contribution of each branch by a learnt in-distribution (i.d.) score. This minimises the influence of logits which are o.o.d. for a given input image, while maximising the i.d. logits. Our weighting improves transparency as the contribution of each model is known during inference, indicating the similarity between input data and different training sets. Our method has the benefit of using learnt features as input rather than images, which provides a simple workaround in cases where licensing prevents image sharing but not sharing of byproduct features. 

Contributions: We apply CIL to echo view classification and propose a new i.d. score weighting that improves performance by minimising the influence of “unqualified experts” and promotes transparency. We validate our method on six datasets from multiple sites, demonstrating improved view classification without needing to transfer images out of remote servers. By bypassing the need for re-training with all data, the cumulative training time and cost is reduced with further reductions achieved through more incremental steps.



§ METHOD




 §.§ Problem Setting


Given a dataset D = {(x_i, y_i)}_i=1^N, where x_i represents an ultrasound image and y_i denotes the corresponding ground truth view label, the objective of view classification is to learn an image encoder ϕ and linear classifier W ∈ℝ^k × |𝒴|, where 𝒴 is the set of labels in D and k is the size of the image features. The view label is learnt by minimising the cross entropy loss between y and the label prediction ŷ = σ(ϕ(x)W) where σ(·) is a softmax function. 

We first train a base model ℳ_b = {ϕ_b,W_bb} on a large dataset 𝒟_b with label set 𝒴_b of base classes. We consider t incremental steps n ∈{n1, n2,..,nt} each with a dataset 𝒟_n and label set 𝒴_n. There can be varying degrees of intersection between base classes 𝒴_b and the new label set 𝒴_n. For instance, there may be novel classes not present in the base classes (𝒴_n∩𝒴_b = ∅) or overlapping classes (𝒴_b∩𝒴_n≠∅). Our goal is to accurately predict all classes, regardless of whether they are introduced in an incremental step, are novel, base, or overlapping. 

Simply combining base and incremental datasets into a single set and retraining ℳ_b from scratch is not ideal due to cost, time, and restrictive data licensing as motivated in Section <ref>. Similarly, naively fine-tuning ℳ_b on successive incremental datasets 𝒟_n1, 𝒟_n2, .., 𝒟_nt is difficult due to the changing size of the label set, and ultimately results in catastrophic forgetting of the base classes 𝒴_b. 





 §.§ Score Fusion


We follow Wu et al. <cit.> where ℳ_b is cloned and fine-tuned for each incremental dataset 𝒟_n, yielding t+1 expert models  {ℳ_b, ℳ_n1,.. , ℳ_nt} [We unfreeze all layers instead of freezing the first two convolutional blocks as in <cit.>, which we find improves performance.]. This approach mitigates the problem of forgetting by retaining all weights; however, it introduces a new challenge of selecting or combining experts during inference. 

This is addressed with SF <cit.> that allows an expert to make a logit prediction on its label set as well as the label sets of other experts. This allows for knowledge sharing between branches and is useful for overlapping classes or shared semantics. Specifically, the weights of an expert {ϕ_d, W_dd}, where d ∈{b, n1,.., nt}, are frozen and used to compute a logit score  z_dd = h_dW_dd for its label set 𝒴_d using feature vector h_d∈ℝ^k and the weight W_dd. To compute logit predictions on label sets of other experts 𝒴_d', each expert learns an additional set of weights W_dd' for all d' ∈{b, n1, .., nt}, d ≠ d'. On each branch, we compute the final logits z_d by summing the logits contributed by each expert as follows: 


    z_d = ∑_d'=b, n1,..,nt^ h_dW_dd'
 

The logits from each branch are concatenated to give z_a. As overlapping classes may be present between incremental steps, a logit for the same class may appear several times in z_a. We follow <cit.> with a “knowledge pooler” to obtain final logits z̃_a by maxpooling the overlapping classes (See Fig <ref>). 
This enables information sharing between experts, however some experts may have significant differences in knowledge and contribute unhelpful information that deteriorates performance. In the next section, we address the “unqualified expert” problem.



 §.§ Weighted Score Fusion


Given an input image, we compute an i.d. score to weigh each expert's logits such that the influence of unqualified o.o.d experts is minimised and the influence of i.d. experts is maximised. We do so by learning an attention-like  weighting vector A ∈ℝ^ | d  | using a feed-forward network ϕ_A consisting of 3 linear layers, the first two with ReLU activation and the last with a softmax. 

We propose two strategies to learn A: (1) attention-weighted score fusion (attn-wSF) where the concatenated feature vectors of all experts h ∈ℝ^k  | d  | are used as input to ϕ_A; and (2) neural mean discrepancy weighted score fusion (nmd-wSF), which uses the neural mean discrepancy (nmd) <cit.> as input. Nmd is an out-of-distribution metric which measures the discrepancy between neural means of an input image and the average neural mean of the i.d. training set. The neural mean is calculated by averaging output activations of the convolutional layers. Like with attn-wSF, we compute a nmd vector g_d for each expert and concatenate them to g ∈ℝ^p  | d  | where p is the size of the nmd vector. 

Similar to Equation <ref>, we compute the final logits for each branch as follows:

    z_d = ∑_d'=b, n1,..,nt^ h_dW_dd'A_d





 §.§ Multi-Site Training


Assuming that the incremental datasets are acquired at different medical sites, which often impose data sharing restrictions, we propose a multi-site training paradigm, which does not require transferring the image data. For every step, the view classifier is cloned, transferred into the remote server, and fine-tuned on dataset 𝒟_n. We then make five forward passes of the fine-tuned classifier ℳ_n on randomly augmented data from 𝒟_n to generate five different g and h vectors for each example. We then transfer these vectors back to the local server, which are used as input to train the weighted score fusion network.





§ EXPERIMENTS & RESULTS




 §.§ Datasets


As base dataset 𝒟_b, we use WASE-Normals (WASE) <cit.>, a large multi-site proprietary dataset of 2,009 healthy volunteers acquired at 18 sites from 15 countries. We also use three incremental datasets: (1) CAMUS <cit.>, a public dataset acquired from 500 patients at University Hospital of St Etienne, half of which are considered at pathological risk; (2) Medstar WASE-Covid (Medstar) <cit.>, a multi-site proprietary database of 870 patients with COVID-19 at 13 sites from 9 countries; 3) St George's (StG), a proprietary stress echo dataset of 420 patients some with coronary artery disease. We validate our method on an internal test set sampled from the four datasets and an independent external test set from two proprietary datasets: (1) Mazankowski Alberta Heart Institute (MAHI) consisting of 250 patients undergoing chemotherapy with cardiotoxic drugs; (2) University of Chicago (UoC) consisting of 391 patients with cardiomyopathy. 

We extract images from 15 echo view labels
and artificially decrease the overlap between datasets by removing non-contrast examples from StG and contrast examples from Medstar. As shown in Table <ref>, the resulting datasets have different label sets with varying degrees of overlap.
Each dataset is split into train (80%), validation (10%) and test (10%) sets at a patient level. A single random frame (112×112) was sampled from each echo video. The final dataset has 42,622 train, 5,175 validation, 5,189 internal test, and 13,999 external test frames. 



 §.§ Implementation & Training


View classifiers use ResNet18 <cit.> architecture and were trained for 200 epochs on a NVIDIA GeForce RTX 2080Ti with cross-entropy loss, Adam optimiser, batch size of 64, and learning rate of 1e-3. The score fusion networks were trained for 50 epochs with the same settings. Weights from the epoch with the highest validation accuracy were saved. Hyperparameters were tuned on a held-out validation set. We evaluate classification performance using accuracy and F1-score metrics. Our method is implemented in PyTorch, and the code is available here: https://github.com/kitbransby/class-incremental-learning-echo







 §.§ Comparison to Existing Methods & Ablation Study


We validate our method on an internal test set (WASE, CAMUS, Medstar, StG) to measure performance on incremental datasets seen during training, and an external set (MAHI, UoC) of data not seen during training to measure generalisability. Results are presented in Tables <ref> and <ref>, respectively. 

We demonstrate that naively fine-tuning a classifier on successive datasets results in “catastrophic forgetting”. We test two configurations, one where the classifier head is replaced at every step, and a second that expands to accommodate novel classes <cit.>. These do not require data transfers and perform well on the final dataset (StG) but forget learning from previous data. We also explore other methods that do not require data transfer such as using a single expert and find they perform well on i.d. data but do not generalise well on o.o.d data. 


Our primary comparison is to the training oracle where the model is retrained with all data combined at each incremental step. This method is preferred given unlimited compute, time constraints and data access, which are not typically available. On average our method outperforms the oracle on both the internal and external sets, including four out of six individual datasets. 

In addition, we ablate our weighting method, demonstrating that attn-wSF and nmd-wSF lead to an improvement in performance when compared to SF <cit.>. We find that nmd-wSF performs best on the internal test set, however, attn-wSF generalises better to the external test set. We theorise that nmd vectors for each dataset are quite distinct, therefore the i.d. scores are pushed towards 0 and 1 during training (see Fig. <ref>). When presented with datasets from the external test set the attention mechanism deteriorates as it considers the input image o.o.d. for all experts. 
 
Finally, we compare different confidence-based methods that select the best prediction from multiple experts, these include: (1) Max Logit <cit.> that selects the max logit for each class; (2) MSP <cit.>, that selects the max softmax for each class; (3) Confidence Routing <cit.> that selects the expert with the highest softmax probability, and uses its softmax prediction as the final prediction. All such approaches perform poorly, perhaps as classifiers are optimised to be overconfident leading to predictions which are uncorrelated with confidence measures. 


Fig. <ref> shows that our wSF methods reduce the cumulative training time, and this grows significantly with the number of datasets. Our attention-based weighting module has minimal computational overhead. While inference time is larger when using multiple experts, this can be reduced with parallelisation. 



§ CONCLUSION


Echo view classifiers require training on diverse multi-site data, and frequent updates with newly acquired data to avoid model drift. We address the problem of updating such a model using multiple datasets with partly overlapping label sets that are from different sites and may not be simultaneously available. We propose attn-wSF and nmd-wSF, two CIL methods that train and combine predictions from multiple experts models with a learnt i.d. weighting which minimises the influence of o.o.d predictions from “unqualified experts”. Our results demonstrate improved view classification with significantly reduced cumulative training time and without needing to transfer images out of remote servers. We aim to extend our wSF methods by training multiple experts within a single branch, which would reduce the memory requirements and inference time. 









splncs04




 Paper Title: Resilience and Security of Deep Neural Networks Against Intentional and Unintentional Perturbations: Survey and Research Challenges.






Resilience and Security of Deep Neural Networks Against Intentional and Unintentional Perturbations]Resilience and Security of Deep Neural Networks Against Intentional and Unintentional Perturbations: Survey and Research Challenges








[3]
These authors contributed equally to this research.
sayyed.a@northeastern.edu

[1][3]
zhang.mil@northeastern.edu


[1][3]
rifat.s@northeastern.edu

[2]
ananthram.swami.civ@army.mil

[2]
michael.j.delucia2.civ@army.mil

f.restuccia@northeastern.edu[3]


   Institute for the Wireless Internet of Things, Northeastern University
  
  Boston
  Massachussetts
  United States
  02120


  † DEVCOM Army Research Laboratory
  
  
  
  United States
  02120















§ ABSTRACT 
  In order to deploy dnn in high-stakes scenarios, it is imperative that dnn provide inference robust to external perturbations – both intentional  and unintentional. Although the resilience of dnn to intentional and unintentional perturbations has been widely investigated, a unified vision of these inherently intertwined problem domains is still missing. In this work, we fill this gap by providing a survey of the state of the art and highlighting the similarities of the proposed approaches. We also analyze the research challenges that need to be addressed to deploy resilient and secure dnn. As there has not been any such survey connecting the resilience of dnn to intentional and unintentional perturbations, we believe this work can help advance the frontier in both domains by enabling the exchange of ideas between the two communities.      

















































[
    Francesco Restuccia
    August 5, 2024
=======================





§ INTRODUCTION


Thanks to their ability of performing critical tasks such as object detection <cit.>, language translation <cit.>, image classification <cit.>, and efficient pose estimation <cit.>, dnn have become essential in our everyday life <cit.>. For a comprehensive survey on dnns, we refer the reader to <cit.>. Breakthroughs in the form of large language models such as GPT-4 <cit.>, universal segmentation models such as Segment Anything <cit.> and diffusion models such as Stable Diffusion <cit.> have   advanced the frontier of ai and captured the interest of ordinary citizens in using ai in their day-to-day lives. 

The unprecedented benefits of dnn come with their own set of social and ethical challenges, mainly in the form of privacy, safety and security issues <cit.>. For example, it has been shown that a dnn is extremely sensitive to intentional perturbations where changing a few pixels in the input can lead to misclassifications <cit.>. In addition, dnn are vulnerable to unintentional perturbations due to natural phenomena, for example, frost, rain, shot noise, impulse noise, defocus blur, motion blur and zoom blur, as well as natural shifts in distribution of labels, which is also known as semantic shift. To ensure ai systems based on dnn can be deployed in real-world systems, it is imperative to ensure resilience and security from both intentional and unintentional perturbations. Achieving this goal can be ensured, for example, by guaranteeing that the output of a dnn will be inherently robust and accurate, or by implementing a rejection scheme that detects inputs on which the prediction is likely to be incorrect due to perturbation or semantic shift, so that the dnn can be adapted to provide a correct prediction for such inputs under specific constraints. This makes the study of resilience and security of dnn a timely yet extremely complex issue. 














 §.§ Motivation and Novel Contributions


Although intentional and unintentional perturbations share several critical aspects, a unified vision  has so far been elusive. Since its inception in 2014 <cit.>, the study of resilient dnn design has been fragmented into separate domains. Some approaches have been studying intentional perturbations – also known as adversarial machine learning – while some approaches have been proposed to guarantee robustness against ood samples. Since a dnn needs to be  resilient to  both types of perturbation, it is important to study the connection between these domains. <cit.> discusses the relation between ood detection and anomaly detection, open-set recognition, and novel set recognition. The survey in <cit.> focuses on the detection of adversarial inputs and bench-marking some of the detection approaches. To the best of our knowledge, literature lacks surveys focusing on the connection between the detection of adversarial input and ood input. As such, in this paper we  discuss the literature from the perspective of resilience of dnn encompassing detection of both adversarial and ood. Specifically, the main contributions of this paper can be summarized as follows:




    
  * We categorize and discuss the seminal, significant, and recent work in the domain of ood detection (i.e., unintentional interference) and adversarial sample detection (i.e., intentional interference);
    
  * We investigate the commonalities among intentional and unintentional perturbation detection and the corresponding defense strategies, while remarking the strengths and weaknesses of these approaches. We believe these two communities can benefit from this study as they can discover common approaches, similarities in these two fields, and adopt new perspective from the other community.
    
  * We conclude the paper by pointing out some open questions and research directions regarding ensuring the resilience of dnn in real-world inference systems. 






§ VULNERABILITIES OF DEEP NEURAL NETWORKS: BACKGROUND AND TAXONOMY




We define the term "vulnerability"  as any action that causes the dnn to perform in not its intended manner as compared to when the action is absent. Under the scope of this survey, we consider  actions where external perturbations are introduced to the input samples during inference. In this section, we provide background on different approaches to introduce these perturbations intentionally. Next, we illustrate different scenarios that occur when external perturbations are added to the input samples in a natural fashion. 
      



 §.§ Intentional Perturbation: Adversarial Attack






We consider a target dnn f(x)=y, where x∈𝒳 and y∈𝒴 represent input and output samples respectively, with 𝒳, 𝒴 respectively representing input and output space. The objective of an adversary is to find a perturbation δ that misleads the target DNN. Based on their intent or objectives, adversarial attacks can be categorized as untargeted and targeted. The former aim at causing an incorrect classification without specifying a particular target class, i.e., 

    f(x+δ) ≠ y,   δ≤ϵ .

Equation <ref> describes the untargeted attack scenario where ϵ is a constraint determining the magnitude of the perturbation. On the other hand, the targeted attack aims at inducing a specific incorrect classification toward a specific class ŷ≠ y, 

    f(x+δ) = ŷ,   δ≤ϵ .

Based on the attacker's knowledge of the target dnn, attacks can be classified as white-box <cit.>, black-box <cit.>, and gray-box <cit.>. In the white-box setting, the attacker has full knowledge of the targeted dnn architecture, parameters, and training data, which enables highly-effective perturbations. A common strategy in white-box attacks involves formulating the attack as an optimization problem, facilitating the utilization of gradient descent to identify adversarial examples. Consequently, these techniques are often referred to as gradient-based attacks. For example, fgsm <cit.> uses a single step in the direction of gradient of the loss function ℒ(·) with respect to the input to obtain the necessary perturbation to evade correct detection of the the input sample as shown in equation <ref>. bim  <cit.> improved the approach by searching iteratively. Its variant pgd <cit.> with random initialization and multiple restarts is considered as a baseline approach to assess adversarial robustness in literature. 

    x_adv = x + ϵ· sgn(▿_x ℒ(θ,x,y)) ,


cw <cit.> demonstrated the importance of the loss function and optimizer. Furthermore, they proposed a surrogate loss given by Equation <ref>,

    ℒ(𝐳,y) = z_i≠ ymax z_i - z_y.

which  denotes the margin loss where 𝐳 is the output vector, z_i is the score of i-th class of the target dnn and y is the ground-truth label of the sample. mt <cit.> enhances the gradient attack by a surrogate loss function designed for multiple targeted classes τ∈ T,

    ℒ(𝐳,y) = τ∈ T∑z_τ - z_y,    y ∉ T.

Equation <ref> denotes the multi-targeted loss, which aims at optimizing adversarial examples across multiple targeted classes and thus bypass the local minima associated with a single class. As a result, it demonstrates strong capability to evade defense algorithms. Among other notable work eot <cit.> employs input augmentation, gama <cit.> utilizes guidance from function mapping of unperturbed samples, aa <cit.> uses ensemble of attacks, acg <cit.> applies conjugate directions to guide the gradient descent, and MIFPE <cit.> minimizes effect of floating point error on the gradient step.

On the other hand, black-box attacks have only limited knowledge about the victim model. As a result, black-box attacks leverage indirect information such as model transferability and output score to generate effective adversarial examples. Black-box attacks encompass a diverse array of strategies, including transfer-based <cit.>, and query-based <cit.> attacks, among others. In the transfer attack scenario, attackers perform gradient attacks on a group of surrogate models by assuming that adversarial samples misleading one dnn model are also likely to mislead others. Therefore, transfer attacks consists of two parts: surrogate model training and transferable adversarial sample generation. me <cit.> first proposed to use averaged prediction from ensemble of models to improve transferability. cwa <cit.> targets the common weaknesses of multiple surrogate models based on cosine similarity and smoothness. Other notable approaches are self-ensemble (gne <cit.>, tsea <cit.>), input transformations (<cit.>, <cit.>, Admix <cit.>, ssa <cit.>, sit <cit.>, pam <cit.>), gradient calibration (mim <cit.>, nim <cit.>, lbp <cit.>, pgn <cit.>, tim <cit.>, taig <cit.>, vt <cit.>, taig <cit.>,gra <cit.>), and surrogate model training (kegn <cit.>, dfme <cit.>, dast <cit.>).  

In query-based attacks, attackers can only access the input and corresponding dnn output (i.e., probability scores or hard labels). Adversarial samples are generated based on an iterative search by sending multiple quires to the target model. The approaches of query based attacks follow one of two streams - score based and decision based scenario. <cit.> first proposes zeroth order optimization approach and applies gradient estimation to find perturbations without substitute model training. nes <cit.>, N-attack <cit.>, AdvFlow <cit.>, and NP-attack <cit.> try to approximate the gradient information. As gradient estimation demands a large number of queries, SimBA <cit.>, Square Attack <cit.>, PPBA <cit.>, and BABIES<cit.> utilize random search to find the optimal perturbation for attacking the model. 

Gray-box adversarial attacks, also known as partial knowledge attacks, are a class of adversarial machine learning techniques where the attacker has limited knowledge about the target model. Unlike black-box attacks, where the attacker has no knowledge of the model's architecture or parameters, and white-box attacks, where the attacker has full access to the model, gray-box attacks assume the attacker has some information—such as knowledge of the model's architecture but not its exact parameters, or access to some but not all training data.  <cit.> demonstrated that adversarial examples crafted for a substitute model (one that approximates the target model) can effectively transfer to the target model, indicating the potency of gray-box attacks . Another study by <cit.> explored ensemble methods to enhance the robustness of models against gray-box attacks by using multiple substitute models to generate adversarial examples, showing that such attacks can still circumvent defenses designed for black-box or white-box scenarios .

Based on the imperceptibility of perturbations, adversarial attacks can be classified as L_p norm constrained <cit.>, sparse <cit.>, and unrestricted <cit.> attacks. In most of the literature, the threat model is bounded in l_2 or l_∞ norm. In other word, the constraint in Eqn. <ref> and Eqn. <ref> are refined as ||δ||_p ≤ϵ, where p=2 or p=∞. Sparse attacks investigate a more difficult problem where attackers can only perturb a small number of pixel of each input. Unrestricted attacks leverage small digital transformations of inputs (i.e., rotation, scaling, brightness) or generate noise that contains semantic difference in physical world (i.e., shadows, raindrops, laser beams) to attack victim models. 

Based on the specificity of perturbations, adversarial attacks can be classified as instance-aware <cit.> and universal attack <cit.>. Instance-aware attack aims to generate adversarial examples tailored to each specific instance. This approach considers the unique characteristics of each input when crafting perturbations to maximize their effectiveness. Universal adversarial perturbation, on the other hand, is a perturbation that can be applied universally to a wide range of samples, regardless of their specific characteristics. Unlike instance-aware perturbations, universal perturbations are designed to have a broad impact across different instances, potentially affecting various types of inputs. They are crafted to exploit vulnerabilities in machine learning models consistently across diverse datasets or input distributions.



 §.§ Unintentional Perturbation: Determining Out-of-Distribution Samples


Figure <ref> shows a taxonomy of existing work in unintentional perturbation. To study the resilience of dnn to unintentional perturbation, existing work makes assumption regarding the nature of the perturbation. If the perturbation has no resemblance to the distribution of the training data, it is extremely challenging to adapt the dnn without additional knowledge and/or labeled samples. In this case, perturbation detection can be considered as a viable option.



 We introduce some notation and define 𝑋 and 𝑌 as the random variables respectively representing input and labels. 𝒟_S(X) and 𝒟_S(Y) is used to denote the marginal distribution of input samples and the labels in the domain from where the training dataset is sampled.  The joint distribution of input and label 𝒟_S(X, Y) is consequently known as training distribution, source distribution or in-domain distribution. Similarly, 𝒟_T(X) and 𝒟_T(Y) are used to denote the marginal distribution of input and label in the target domain while their joint distribution 𝒟_T(X,Y) is termed as the target distribution, or out-of-distribution. We define the input space 𝒳∈ℝ^d and the corresponding label space 𝒴, where d indicates the dimension of the input. The dimension of 𝒴 varies depending on the task, e.g., for classification tasks, 𝒴=[C]≜{1, ...,C}, where C denotes the number of classes, and 𝒴∈ℝ^d for semantic segmentation tasks. We define a dnn f_θ_S:𝒳↦𝒴 composed of a feature extractor g_θ_S:𝒳↦𝒵 and a predictor h_θ_S:𝒵↦𝒴 such that f=h∘ g, where 𝒵 is the intermediate representation space and 𝑍 is the corresponding random variable. 


Out of distribution (OOD) samples can be defined as any input drawn from any distribution different from training distribution. The term distribution shift encompasses four types of shifts in the literature as summarized in Figure <ref>. The first is non-semantic or covariate shift (e.g. noise, blur, frost and different types of corruptions, change in geographical location, brightness etc.) and the second is semantic shift or label set shift, which corresponds to the emergence of new semantic category or class label.  The third is prior shift, and corresponds to a shift in the distribution of the classes observed.  For example, an object detection algorithm will be more likely to encounter “animal” objects in a rural area than in an urban area. The fourth is the generalized label shift, where the prior shift and the covariate shift happens simultaneously. We formally define the four types of distribution shifts below.

Definition of Covariate Shift: The covariate shift is a change in the distribution 𝒟(X) which preserves their label distribution 𝒟(Y). In other words, 𝒟_S(Y|X) = 𝒟_T(Y|X). Notice that the target feature distribution 𝒟_T(X) is assumed to be known. Since the covariate shift does not change the semantic content of the input, it is also named non-semantic shift. Direct perturbation, either intentional or unintentional, may cause such shift. Among the unintentional perturbations, we have common corruptions such as Gaussian blur, Gaussian noise, motion blur, defocus blur, frost, fog and rain, among others <cit.>. This also includes shift in input distribution due to a change in geographical location, operating hardware (e.g., different camera or sensors) or viewpoint <cit.>. Due to the change in the distribution 𝒟(𝒳), the distribution 𝒟(𝒳,𝒴) changes. This brings the problem of dataset shift <cit.>. Additionally, we can consider intentionally perturbed inputs a.k.a. adversarial inputs <cit.> as covariate shifts, since the perturbation only changes the input distribution. This highlights the connection between unintentional and intentional perturbations. 

Definition of Semantic Shift: Any kind of shift that changes the semantic content and as a result the marginal distribution of label 𝒟(𝒴) of the dataset is a semantic shift or label shift. A semantic shift affects both the image space and the label space as the distribution of input 𝒟_X is shifted from the source distribution and new labels are introduced in the label space. The detection of such shifts can be considered as encompassing the tasks of novel class recognition and open set recognition. In the open set recognition problem <cit.>, the dnn is presented with sample from classes which were not present in training data representing a shift in the label distribution. One-class novel class recognition can be thought of as an extreme version of open set recognition where the dnn is presented with a single class during training and is required to detect new incoming classes during testing or inference.

Definition of Prior Shift:  Prior shift refers to a scenario where the marginal distribution of labels in source and target domain are different, i.e.,  𝒟_S(Y) ≠𝒟_T(Y). On the other hand, the class conditional distribution of data given labels are assumed to be same, i.e., 𝒟_S(X|Y) = 𝒟_T(X|Y). Prior shift might affect domain adaptation processes, also known as tta in literature <cit.>. Although the marginal label distribution 𝒟_S(Y) is uniform in most of the cases, in tta the data is observed as a small batch of samples at a particular time. Hence, we might observe dominance of certain labels based on the current scenario. This is a very common example of prior shift that might happen which is also termed as 'correlated label distribution' in tta literature.  

Definition on Generalized Label Shift.  Generalized label shift is a more challenging data drift that was first introduced in <cit.>. It occurs  when both covariate shift 𝒟_S(X) ≠𝒟_T(X) and prior shift 𝒟_S(Y) ≠𝒟_T(Y) occurs simultaneously. Moreover, it is assumed that there exists some feature representation Z = g(X) for which the conditional distributions based on labels both on source and target domain are equal, i.e.,  𝒟_S(Z|Y) = 𝒟_T(Z|Y). This situation might arise when a dnn continuously experiences different data corruptions while also being adapted with a skewed label distribution.








§ A UNIFIED VISION FOR INTENTIONAL AND UNINTENTIONAL PERTURBATION


There are two main approaches to ensure resilience of dnn. The first is by training the dnn to be inherently resilient. Examples include adversarial training <cit.> to avoid intentional perturbations, and training with different augmentation schemes <cit.> for unintentional perturbations. The second way is to augment the dnn with another binary classifier that differentiates between perturbed and unperturbed samples. Ideally, the auxiliary classifier would be able to differentiate between unperturbed and perturbed inputs for both intentional and unintentional perturbation. In reality, the detection of intentional and unintentional perturbation have developed as two separate domains. In the following, we highlight the connection among the methods for detection of intentional and unintentional perturbations. As such, we have categorized the approaches from the perspective of input space, latent (i.e., model) space, and  logit (i.e., output) space. Figure <ref> overviews our categorization, which is detailed below.

Input Space: Prior work has attempted to model the input distribution to detect intentional and unintentional perturbations. For example,  <cit.> proposed to model adversarially perturbed inputs with a diffusion model, which is also known as generative modeling. The authors propose the expected perturbation score which is the expected value of the gradient of the distribution of the output of the diffusion process at a given time, and show that the distribution of the expected perturbation score is different for the unperturbed and intentionally perturbed samples. For unintentional perturbation detection, the authors in <cit.> propose a technique to learn the input distribution using generative models and corrects for the bias for complexity of the input samples.  




Model (or Latent) Space: Most of the existing work either adapts the model space or create scores based on the latent representations. The works in this space can befurther categorized into four types: (i) augmentation, (ii) regularization, (iii) model adaptation, and (iv) scoring functions. Some prior work has used ood samples available in the wild to train the dnn to distinguish between the id and ood samples by regularizing the loss function. The first work in this direction was <cit.>, which was later improved in <cit.> by improving the sampling of the ood data as well as the regularization methodology. Prior work in ood detection adapts the model to better differentiate between the id and ood inputs. For example, <cit.> shape the model features during test-time for better detection of the ood inputs. The work in <cit.> discretizes the output of a layer of the model into binary or quaternary codes and detects intentional perturbation by observing deviations from the codes produced by unperturbed samples. In addition, the authors in <cit.> focus on intentional perturbation detection and design a partial Bayesian network by learning the likelihood of the weights for the last few layers. This network is then used to measure the uncertainty associated with the input. A vast body of work in ood crafts different scoring functions based on the latent representation of the dnn. For example <cit.> craft scores using intermediate layer activations, while <cit.> also uses the score for detection of intentionally perturbed inputs.

Output (or Logit) Space: By output space, we indicate either the logit layer of a dnn or its softmax probability distribution. The work in such output space has mostly focused on model adaptation and scoring functions. To the best of our knowledge, <cit.> is the only work we have found that focuses on the output space during training time. This work normalizes the output logits to mitigate the overconfidence issue of dnn. Conversely, <cit.> examines logits to detect the intentionally perturbed inputs and potentially correct the prediction. The work in <cit.> first used the maximum softmax probability to detect ood samples, while <cit.> studied the detection of ood using score-functions derived from the output of a dnn.





§ DETECTION OF INTENTIONAL PERTURBATION



Based on the classification in Section <ref>, we describe existing work in perturbation detection as well as the related defense strategies. The detection of intentional perturbation  – also known as adversarial perturbations in literature –  is mainly based on the hypothesis that adversarial samples reside in a manifold different than that of the unperturbed inputs <cit.>. The key objective behind adversarial sample detection is to accurately detect and minimize the influence of adversarial samples on dnn. The general framework for this detection is: (a) characterizing the adversarial perturbation; (b) formulating a score function based on the characterization; and (c) deciding on a threshold to accept or reject a sample as adversarially perturbed. In this section, we first review some seminal work in this field and then provide key insights into recent work. We summarize all the reviewed work in Table <ref>.



 §.§ Use of Intermediate Representations



One of the earliest approaches utilize the features from intermediate layers of a dnn to detect adversarial samples. The authors of <cit.> train an adversary detector that receives inputs from the intermediate feature representations of a classifier. This detector aims to distinguish between samples from the original dataset and adversarial examples. The author considers two different scenarios: (i) a static scenario, where adversaries have access to the classification network and its gradients only; and (ii) a dynamic scenario, where adversaries have access to both the classification and the proposed detector network along with its gradients. The adversary detector network is trained in a supervised manner using training images and their corresponding adversarially perturbed images. The authors showed that a static detector cannot perform well against a dynamic adversary. 

To tackle dynamic adversaries, the author proposes a dynamic adversary training method inspired by the approach in <cit.>, where for each mini-batch the adversarial samples are computed on the fly. For each mini-batch, a dynamic adversary modifies a data point with a parameter σ selected randomly from the range [0, 1], as it depends on the gradient of the detector which evolves over time. By training the detector, one implicitly makes it resilient to dynamic adversaries for various values of σ. Although it shows excellent results, it does not show robustness against random noise. One key takeaway from this work is that supervised training with specific attack strategy is unlikely to hold against adaptive attackers and a robust characterization of adversarial perturbation is required to be utilized by the detector so that it can generalize across different attack mechanisms instead of overfitting to a single type of attack. 

The detector sub-network proposed in <cit.> remains vulnerable to the adversarial samples that are not used during the detector training process. To address this issue,  <cit.> proposes a new approach called SafetyNet that relies on the hypothesis that adversarial attacks work by producing different patterns of activation in late-stage relu to those produced by natural examples. As a result, it focuses on discrete codes produced by the quantization of each relu function at later stages of the classification network. SafetyNet consists of the original classifier along with an adversary detector that examines the internal state of the later layers in the original classifier.



|p1.8cm|p.6cm|p2.8cm|p3.5cm|p7cm|p1cm|p0.6cm|
Studies on Adversarial Sample Detection Techniques. The metrics used refer to A: AUROC, B: Attack Failure Rate, C: Detection Accuracy/Detection Rate/Detection Success Rate, D: TPR(FPR@x), E : TPR, F : FPR, G: Reclassification Accuracy, H: Robust Accuracy, I : Robust Average Precision, J : Recovery Rate , K : Equal Error Rate, L : Label Change Rate, M : Detection Cost Function




Work     Year     Task(s)     Dataset(s)     Attack(s)     Metrics     Code 

Work     Year     Task(s)     Dataset(s)     Attack(s)     Metrics     Code 



Metzen et al. <cit.>     2017     Image Classification     CIFAR10<cit.>, 10-Class Imagenet<cit.>     fgsm, bim, DeepFool     C     

Feinman et al. <cit.>       2017              Image Classification                                                                                   MNIST<cit.>, CIFAR10<cit.>, SVHN<cit.>                                                                                                     fgsm, bim, jsma, cw                                                                     A                                                             https://github.com/rfeinman/detecting-adversarial-samplesCode 

Lu et al. <cit.>            2017              Image Classification                                                                                   CIFAR10<cit.>, Imagenet<cit.>                                                                                                        fgsm, DeepFool, ta, bim,                                 C, E                                           https://github.com/Jianbo-Lab/ML-LOO.Code 

Grosse et al. <cit.>        2017              Image Classification                                                                                   MNIST<cit.>, DREBIN<cit.>, MicroRNA<cit.>                                                                                                  fgsm, jsma, svma, dta,                                                                    C     

Ma et al. <cit.>            2018              Image Classification                                                                                   CIFAR10<cit.>, MNIST<cit.>, SVHN<cit.>                                                                                                     fgsm, bim, jsma, cw                                                        A, B                                        https://github.com/xingjunm/lid_adversarial_subspace_detectionCode 

Zheng et al. <cit.>         2018              Image Classification                                                                                   MNIST<cit.>, F-MNIST<cit.>                                                                                                           fgsm, DeepFool                                                                          A, E                                                         

Tao et al. <cit.>           2018              Facial recognition                                                                                     VGGFace<cit.>, LFW<cit.>, CelebA<cit.>     fgsm, bim, cw, Patching Attack, Glasses                               F, C                                           https://github.com/AmIAttribute/AmICode 

Lee et al. <cit.>           2018              Image Classification                                                                                   CIFAR10<cit.>, CIFAR100<cit.>, SVHN<cit.>                                                                                                  fgsm, bim, DeepFool, cw                                                                 A                                         https://github.com/pokaxpoka/deep_Mahalanobis_detectorCode 




Roth et al. <cit.>          2019              Image Classification                                                                                   CIFAR10<cit.>, Imagenet<cit.>                                                                                                        pgd, cw,                                                                                C, G                      https://github.com/yk/icml19_publicCode 

Wang et al. <cit.>          2019              Image Classification                                                                                    CIFAR10<cit.>, MNIST<cit.>                                                                                                           fgsm, jsma, DeepFool, cw, Black-Box                                                     A, C, L                https://github.com/dgl-prc/m_testing_adversatial_sampleCode 

Cohen et al. <cit.>         2020              Image Classification                                                                                   CIFAR10<cit.>, CIFAR100<cit.>, SVHN<cit.>                                                                                                  fgsm, jsma, DeepFool, cw, pgd, ead,  Adaptive                                           A, B                                        https://github.com/giladcohen/NNIF_adv_defenseCode 

Yang et al. <cit.>          2020              Image Classification                                                                                   MNIST<cit.>, CIFAR10<cit.>, CIFAR100<cit.>                                                                                                 fgsm, cw, jsma     A, D                  

Li et al. <cit.>            2020              Speaker identification                                                                Voxceleb1 <cit.>                                                                                                              bim, jsma                                                                              K, M, C     

Shumailov et al. <cit.>     2020              Image Classification                                                                                    MNIST<cit.>, FashinMNIST<cit.>, CIFAR10<cit.>                                                                                              fgsm, bim, pgd, cw, Fully Decision Based Boundary Attack                           C                                                

Deng et al. <cit.>          2021              Image Classification, Face Recognition, Object Detection     Imagenet<cit.>, LFW<cit.>, CPLFW<cit.>, CALFW<cit.>, CFP<cit.>, VGGFace2<cit.>, AgeDB-30<cit.>, COCO<cit.>                                                               fgsm, bim, pgd, mim, cw, dim, tim                                                       A                                  https://github.com/thudzj/ScalableBDL.Code 

Li et al. <cit.>            2021              Image Classification                                                                                    CIFAR10<cit.>, Imagenet<cit.>                                                                                                      Patch-PGD, Adversarial Patch, Adaptive Attack                                                    C                                                    



Mekala et al. <cit.>        2021              Facial Recognition                                                                                    VGGFace2<cit.>, LFW<cit.>,                                                                                      fgsm, pgd, cw                                                                           C                                                

Xu et al. <cit.>          2018              Image Classification                                                                                                    MNIST<cit.>, CIFAR10<cit.>, Imagenet<cit.>                                                                                                                      fgsm, bim, cw, jsma, DeepFool                                                                                     H                                                                             

Picot et al. <cit.>         2023              Image Classification                                                                                   CIFAR10<cit.>, CIFAR100<cit.>                                                                                                        fgsm, bim, pgd, cw, DeepFool, hop, sa, sta, Adaptive                                     A, D                                                     https://github.com/MarinePICOT/HAMPERCode 

Chyou et al. <cit.>         2023              Image Classification                                                                                   CIFAR10<cit.>                                                                                                                  fgsm, bim, pgd, DeepFool, Auto Attack                                                   A, E, F                                                   https://github.com/CycleBooster/Unsupervised-adversarial-detection-without-extra-modelCode 

Zhang et al. <cit.>         2023              Image Classification                                                                                   CIFAR10<cit.>, Imagenet<cit.>                                                                                                        fgsm, pgd, bim, cw, Auto Attack, ta                                   A                                                             https://github.com/ZSHsh98/EPS-AD.gitCode 

Tarchoun et al. <cit.>      2023              Image Classification, Object Detection      Imagenet<cit.>,Pascal VOC 07<cit.>, INRIA<cit.>, CASIA<cit.>                                          Adversarial Patch, lavan, YOLO, Naturalistic Patch                    H, I, J           

Sun et al. <cit.>           2023              Image Classification                                                                                   CIFAR10<cit.>                                                                                                                  fgsm, bim, DeepFool, jsma, cw                                                           C                                            











The detector in SafetyNet employs a rbf-svm to discern adversarial examples based on binary or quaternary codes representing activation patterns. For the code denoted by b, rbf-svm uses the following formula to classify the samples:

    f(b) = ∑_i^Nα_iy_iexp(-∥ b-b_i∥^2/2σ^2) + c

The detector hardly generates any gradient unless the attacking code, denoted as b, closely resembles the code for a positive example b_i when the variance of the kernel, σ is very small. SafetyNet compels the attacker to solve a hard optimization problem that is discrete in nature. This work employs the idea of including non-differentiable operations so that the adversary cannot obtain the gradients necessary to compute adversarial perturbations. 




 §.§ Manifold-based detectors



In <cit.>, the key intuition is that the adversarial samples lie on a different manifold than unperturbed samples. The authors argue that if after perturbation the data x is transformed into x^*, it can leave the manifold c_x in favor of sub-manifold c_x^* in one of the following three ways illustrated in Figure <ref>:

    
  * x^* is distant from the submanifold of c_x^* but closer to the classification boundary between c_x and c_x^*;
    
  * x^* lies closer to c_x^* submanifold but is still outside. On the other hand, x^* is distant from the classification boundary that separates the classes c_x and c_x^*. As shown in Figure <ref>, here one of the submanifolds has a pocket.
    
  * x^* is close to the submanifold c_x^*, but is still outside. In addition, x^* is close to the classification boundary which separates the classes c_x and c_x^*. 


The authors estimate the manifold with kd estimation. They do so with the output of the last hidden layer based on the hypothesis presented in <cit.>, which states that the deeper layers of a dnn offer more linear and 'unwrapped' manifolds compared to the input space. Given an input point x and s set of training points X_l having label l, the kde f̂ can be obtained as f̂ = 1/|X_l  |∑_x_i ∈ X_l^k(x_i, x) 




 , where k(.,.) is the kernel function. The latter offers an indication of the distance between x and the submanifold for l. For the point x, if the last hidden layer activation map is ϕ(x), then the density estimate with predicted class l is k̂(x, X_l) = ∑_x_i ∈ X_l^k_σ(ϕ(x),ϕ(x_i)) 




where σ is a tunable bandwidth. While this approach exhibits effective performance in the detection of adversarial samples that are located far away from the c_x^* submanifold, it performs poorly in scenarios where adversarial samples x^* lie in proximity to the c_x^* submanifold. As a result, in addition to the kde, the author proposes bu to identify low-confidence regions within the input space. The bu shows considerably different distributions for normal and adversarial samples - supporting the intuition of different manifolds for adversarial examples. The drawback of their method is that the uncertainty estimate hinges on the use of "dropout" which limits its application. Also, the performance is poor on more challenging datasets and stronger attacks. 




 §.§ lid



The objective of lid <cit.> is to characterize the specific regions where adversarial examples may be located. Specifically, this work shows that the kd approach adopted in <cit.>  – which was based on the assumption that the adversarial subspaces are low probability regions – fails to detect some forms of adversarial attack. As an alternative, the authors propose lid to characterize the adversarial subspace. lid represents the dimension of the data submanifold local to the data point x under consideration. In connection to the classical expansion models, treating the probability mass as a proxy for volume may provide information about the dimensional structure of the data. lid considers the cdf of the number of data points encountered F(d), where d is a realization of the random variable D, i.e., the distance from data point x to other data points. The lid of x at a distance d can be defined as: 

    LID_F(d)≜lim_ϵ→0ln(F((1+ ϵ)· d)/F(d))/ln(1+ϵ) = d · F^'(d)/F(d)

where D is a positive random variable and the cdf F(d) of D is continuously differentiable at distance d>0. The local dimension at x in turn is defined as LID_F = lim_d →inf LID_F (d). The LID_F quantifies how quickly the cdf F(d) grows with the distance d. It can be approximated by considering the distances between a point x and its k nearest neighbors within the dataset. The work in <cit.> hypothesizes that for the estimation of lid of the adversarial samples, the nearest neighbors drawn should come not only from the manifold of the adversarial samples but also from the manifold of the normal samples, as the adversarial submanifold lies close to the data manifold. This will increase the dimension of the adversarial submanifold leading to higher value of lid. This approach exhibits better generalization across different attacks than kd. The authors train logistic regression model with lid feature to discriminate between perturbed and unperturbed samples. The major drawback is that it fails against stronger attacks, which indicates that the characterization of the adversarial manifold with lid is not universal for all adversarial perturbation. 




















 §.§ Interpretability-based Approaches











Conversely from adopting adversarial sample training <cit.>, the work in <cit.> examines the adversarial samples from the dnn interpretability point of view. Specifically, a novel adversarial sample detection named ami has been proposed for face recognition. In this work, the main hypothesis is that adversarial samples utilize complex features extracted by the dnn that are imperceptible to humans. As such, ami initially extracts a set of neurons called attribute witness, which are entangled with the face attributes. They substitute parts of the face from one image onto different images and look for unchanged neurons. This is named the Attribute Preservation step. Another step is to substitute the same parts from different images onto a single image - generating versions of the image only part-substituted. The neurons which change in this case are likely to be attributed to that part, which is the Attribute Substitution step. The common neurons obtained from attribute substitution and attribute preservation steps results in attribute witness neurons. The authors then construct an attribute-steered model by increasing the values of the witness neuron and decreasing the values of non-witness neurons. For a given test input, the inconsistency observed between the two models indicates that the input is adversarial. The major drawbacks of this approach are the necessary manual feature inspection, absence of features in test image, and limited generalizability to setting other than face recognition. The work in <cit.> constructs shap-based <cit.> feature attribution for the features of the penultimate layer. The knowledge base constructed from these shap signatures for both natural and adversarial images are used to train a binary classifier. The key drawback of this approach is the requirement of the shap signature from both natural and adversarially perturbed inputs. This creates a potential generalization problem for the binary classifier used for discriminating between the natural and adversarial inputs. 




 §.§ Statistical Approaches


The work in this category falls attempts to extract statistical information from different layers of the model to detect adversarial samples. The work in <cit.> proposes a statistical detection method based on mmd. The authors hypothesize that only a limited number of samples is needed to  observe a measurable difference between normal and adversarial samples using a statistical test. They use a two-sample hypothesis test on the distribution of mmd values to detect the difference between the normal samples and the adversarial samples. To detect single examples, they augment their classifier with an outlier class and train using the adversarial samples. The work in <cit.> proposed a gda-based approach where the model features from different layers of the dnn as class conditional multivariate Gaussian and calculate the confidence score for a sample as the  Mahalanobis Distance between the sample and the closest class conditional distribution. The authors extract such score from all the layers and integrate them using weighted averaging. Both <cit.> utilize adversarial samples either to train their detector or find appropriate weights. These approaches are also model and attack specific and require separate detectors for different attack approaches. This severely limits their applicability.

The work in <cit.> approaches the detection of adversarial patches from a statistical perspective in the input space. The authors postulate that adversarial patches should contain a statistically higher amount of information, from an information theory perspective, compared to any random neighborhood from a natural image distribution. This leads to the proposal of Jedi, which detects adversarial patches in images based on entropy thresholds. They use a 50-pixel × 50-pixel kernel and compute the entropy threshold dynamically based on the entropy distribution of the dataset and the image under consideration. The extracted high-entropy patch is passed through a sparse autoencoder for improved localization. Finally, they use coherence transport-based image inpainting <cit.> which aims at defusing the patch. They obtain high robust accuracy with respect to the baseline methods.



r0.35
    

    
    < g r a p h i c s >

    The embedding space of a dnn <cit.>. 

    







 §.§ Influence-based approaches



The work in <cit.> proposes a novel adversarial sample detection strategy by using an “influence function” <cit.>. This approach can be employed by any pre-trained dnn. The key intuition is that the training samples have close correspondence to the dnn classification. When this relationship is disrupted, it strongly suggests the presence of an adversarial input. As such, the influence function measures the impact of training data in the decision-making of the dnn. The influence of a training image x on the loss of a specific test image x_test can be measured as I_up, loss(x, x_test) = -∇_θL(x_test, θ)^TH_θ^-1∇_θL(x, θ) where L is the loss function and H is the Hessian of the machine learning model. The work also applies knn classifier at the embedding space of the dnn as the resemblance of the nearest neighbor in the embedding space also dictates the decision of the dnn. The combination of the influence function and knn classifier enables the detection of adversarial samples as the inference of a sample should be influenced most by the samples that are closest in the feature space. For adversarial samples, we observe the lack of this correlation. 

Figure <ref> illustrates this relation, where the black star and the brown X illustrate a normal and adversarial image from the CIFAR-10 validation set. We observe that the 25 nearest neighbors of the normal sample (blue circle) and the 25 helpful training samples (blue squares) for the normal sample lie very close in the pca projected embedding space. A sharp contrast to this relation is observed in the case of the adversarial sample. Although this approach exhibits good performance in terms of generalization, it requires longer computation time as it calculates the influence function for the entire training dataset. The work in <cit.> masks the pixels of the input image and computes pixel level feature attribution by measuring the change in the output. The authors expedite the feature attribution to input space by using different sampling strategies. All these approaches assume access to the adversarial samples, which limit the utility of such approaches as they fail to detect stronger attacks. 















































 §.§ Other Notable Approaches


Input Space Approaches: <cit.> introduce a diffusion based perturbation method and derive eps. They show that the distribution of eps is different for normal and perturbed images. The difference in perturbation is measured using mmd. This approach achieves an AUC of 1, and its performance does not degrade for unseen attacks. This is because eps models the distribution of the input data itself instead of modeling the feature space. One drawback of the eps score is that it cannot differentiate between adversarial perturbation and noise perturbation. The work in <cit.> proposes mlloo feature attribution based detection of adversarial samples. The authors observe that feature attribution or mapping of importance of input features to the final prediction behaves differently for unperturbed and adversarially perturbed images. Adversarial perturbation disperses the feature attribution scores with significant deviation from the normal samples. Equipped with this observation, the authors use simple statistics to characterize the deviation of the adversarial samples as it progress through the dnn and aggregate the statistics with a logistic regression model to differentiate between adversarial and normal samples. Although the authors utilize the leave-one-out feature attribution, their approach is generic to any feature attribution method. The work in <cit.> trains a binary classifier using the saliency data by concatenating the saliency map to the raw image along the channel, while <cit.> proposes to squeeze input features using bit-depth reduction and spatial smoothing (both local and non-local variants). The work compares the output probability distributions on the original input and the feature-squeezed inputs using L_1 distance which changes significantly for adversarial inputs while normal samples show no change. This is because feature squeezing removes unimportant non-robust features improving robustness. 

Model Space Approaches: The work in <cit.> proposed to modify the dnn and has implemented a self-contained toolkit named mMutant that integrates mutation testing and statistical hypothesis testing on dnn. The key observation is that the sensitivity of the mutation on dnn is more acute for adversarial samples compared to unperturbed samples. If the dnn is slightly altered, there is a greater chance that the mutated dnn will alter the label of the adversarial sample than the unperturbed sample. The empirical investigation confirms this inherent sensitivity of the adversarial samples against a group of dnn mutants in terms of lcr. However, with the increase of mutation rate, the distance of lcr between adversarial and normal samples decreases. This approach is also prone to the generation of some false positives during adversarial detection. Shumailov et al. <cit.> proposed a mechanism called ctt, which incorporated the Taboo Trap detection, as well as numerical bound propagation. It prioritizes on finding the overexcited neurons being driven by adversarial perturbations outside of a predetermined range. The incorporation of numerical bound propagation on ctt certifies the detection bounds on activation values of cnn against specific input perturbation sizes. The authors proposed three variants of ctt namely lite, loose, and strict. Although ctt-lite does not require fine-tuning, its defense capacity is limited. Further optimization through fine-tuning results in ctt-loose having most samples detected as adversarial. On the other hand, ctt-strict guarantees detection of adversarial samples withing a specific range of l_∞ bound. Deng et al. <cit.> proposed libre - a bnn based approach that does not require training an extra model. The authors combined the expressiveness of deep ensemble <cit.> and efficiency of last layer Bayesian learning <cit.>. Conversely from using the entire dnn for deep ensemble, they convert the last few layers – for example, the last residual block of ResNet50 –  to a deep ensemble-like architecture. They also proposed to quantify the uncertainty with feature variance instead of softmax variance. As libre does not use any adversarial samples for its training, it is adversarial attack agnostic and treats the adversarial samples as ood data. This broadens its application scenario to more than classification with no modification which is not observed for the supervised detection methods.



Finally, in <cit.> the authors proposed hamper to characterize the adversarial samples. They calculated the class conditional half-space mass depth for a subset of dnn layers and aggregate the scores as weighted sum. To tune the weight values, they utilized adversarial samples, and showed state-of-the-art performance in both attack-aware and blind-to-attack scenarios. It is shown that the last layers carry more important information for the detection of adversarial samples. hamper has the advantage of being robust against adaptive attacks. However, this work does not provide any understanding as to why the half-space mass depth metric is able to capture the difference between the adversarial and normal samples. 

Output Space Approaches: Prior work has attempted to characterize adversarial examples in the output space to facilitate their detection. I-defender <cit.> modeled the distribution of the output of the linear layers and shows that the distributions are different for the normal and adversarially perturbed images. The authors used a mixture of Gaussian models to approximate the ihsd for each class. If the class conditional probability is lower than a threshold for a sample, it is detected as adversarially perturbed. However, this approach performs poorly under moderate to strong attacks. Roth et al. <cit.> proposed a statistical metric for the detection of adversarial samples based on expected value of perturbed log-odds. They showed that the robustness properties of perturbed log-odds statistics are different for natural and adversarial samples. The idea is that geometrically optimal adversarial manipulations are embedded into a cone-like structure they call "Adversarial Cone". They also reported an intriguing finding, i.e.,  that adversarial samples are much closer to the ground truth unperturbed class than any other class. Based on these observations, they proposed the maximum expected deviation of the perturbed log-odds from its expected value as an indicator of an adversarial sample. Chyou et al. <cit.> proposed an unsupervised adversarial sample detection method without any extra model. They proposed new training losses to improve detection accuracy. The main idea is to remove unnecessary features for false outputs and strengthen the true outputs. This is achieved by forcing all the false raw outputs in a mini-batch to have a uniform distribution during training. By doing so, false outputs become adversarially robust and only true outputs can be attacked. Any attack on the true output changes the raw false output values triggering an adversarial detection. The proposed training loss keeps the accuracy of the original classification task almost the same, around 86% when using ResNet18 architecture and CIFAR10. Although the authors used adversarial examples to determine their threshold, they show that the estimated threshold for their binary detector generalizes for other stronger attacks. The achieved accuracy on CIFAR10 dataset for Resnet architecture falls short of the reported accuracy of >90% <cit.> although this can be related to the training strategy. 





§ DETECTION OF OOD SAMPLES


Existing approaches for ood detection mostly operate in the output space. Indeed,  it has been shown that working with activations of earlier layers does not provide much improvement in detector performance <cit.>. In this section, we first provide the discussion of seminal work in ood detection, and then focus on the state of the art approaches. Table <ref> provides a summary of the surveyed work regarding ood detection.  






 §.§ Input Space Approaches
 

Among the few works in this space, Li et al. <cit.> forced the classifier to implicitly learn the id data distribution instead of just learning features for classification. They applied a preprocessing step of masking and then learn to reconstruct the original image from the masked image. This calibrates the dnn for id and ood samples. Gao et al. <cit.> utilized the diffusion model to learn the distribution of id data. They learned a ddim conditioned to the semantic labels, while during inference time, they inverted the image to obtain the latent representation using the ddim and them reconstruct it from the latents. 




 §.§ Output Space Approaches


ODIN <cit.> is among the earliest work in the domain of ood detection in the output (or logit) space. It adopted the baseline <cit.> where the authors utilize the softmax probability score to distinguish between id and ood samples and improve it by incorporating temperature scaling and input pre-processing steps. ODIN <cit.> shows that by manipulating the temperature parameter T ∈ℝ_+, it is possible to increase the separation between the id and ood examples. The resulting score function S_ŷ(𝐱;T) is given by Equation <ref>:

    S_i (𝐱;T) = exp(f_i(x)/T)/∑_j=1^j=Nexp(f_j(x)/T)
    
        S_ŷ(𝐱;T) = _i S_i(𝐱;T)

Here, f_i(𝐱) denote the logit value corresponding to i-th class for dnn 𝐟. The authors complemented the temperature scaling by perturbing the input image before feeding it into the dnn. The perturbation procedure was inspired by <cit.>, which adds the perturbation to decrease the softmax score but here the perturbation boosts the softmax score prediction. The perturbation follows 𝐱 = 𝐱 - ϵ(-∇_𝐱S_ŷ(𝐱;T)).






The authors explained the effect of temperature scaling using U_1 = 1/1-N∑_i ≠ŷ [f_ŷ(𝐱) - f_i(𝐱)] and U_2 = 1/1-N∑_i ≠ŷ [f_ŷ(𝐱) - f_i(𝐱)]^2. They showed that id data contains some classes that are similar to each other, resulting in a higher value of U_2 even when U_1 value for id and ood data is the same. By taking the Taylor approximation, the softmax function can be expressed as S_ŷ = 1/N - (U_1 - U_2/2T)/T. This suggests that for very high values of T, the softmax score is dominated by the term U_1 compensating the negative effect of U_2 on the detection performance. They argue that this makes the id and ood data more separable. The authors argue from the Taylor expansion of the log softmax of the perturbed input x that id images have a larger norm of the gradient of the score function compared to the ood images which results in a higher score function values after perturbation.

    S_ŷ(𝐱;T) = S_ŷ(𝐱;T) + ϵ ||∇_𝐱S_ŷ(𝐱;T)||_1 +o(ϵ)

The key observations are  (i) dnn produce outputs with larger variance for id examples; and (ii) dnn have larger value of the gradients of the log-softmax score when applied to id images.



Generalized ODIN <cit.> improved <cit.> without utilizing any ood samples. The authors point out the limitation of the softmax classifier, as it is an approximation of the indicator function it gives a categorical distribution rather than a continuous distribution over the classes. To circumvent this limitation, the authors reformulated the posterior probability by incorporating domain of the input as a random variable, changing it from p(y | 𝐱) to p(y | 𝐱, d_in), where y, 𝐱,  d_in represent the output, the input and the input domain. This lead to the decomposition of the posterior as p(y | 𝐱, d_in) = p(y, d_in | 𝐱)/p(d_in, 𝐱). Due to the lack of out-of-domain knowledge, the authors proposed to utilize the prior knowledge of the dividend/divisor structure of the posterior to provide the classifier capacity to decompose the confidence of the class probability. They modeled the logits f_i(𝐱) as f_i(𝐱) = h_i(𝐱)/g(𝐱). While the work shows improvement over <cit.>, its key limitation is that it needs to change the structure of the dnn and perform training to learn h_i(𝐱) and g(𝐱).

[12]r0.65
    

    
    < g r a p h i c s >

    Decomposing the logit values into dividend/divisor structure. g(x) allows incorporating domain knowledge into training and disentangles the probability over the domain and probability over the classes. <cit.>
    


Liu et al. <cit.> proposed a generalized entropy-based approach. Specifically, the authors used the generalized entropy family defined by G(𝐩) = ∑_i p_i^γ(1-p_i)^γ, where 𝐩 denotes a categorical probability distribution. This score function, coupled with truncation of very small probabilities, showed competitive result for state-of-the-art models like Swin Transformer and BiT-S R101×1 on ImageNet benchmark. The work in <cit.> addressed the fact that OOD detection schemes underperform in real-world settings. This work introduces a new dataset called Species consisting of 700,000 images and over 1,000 anomalous species to test ood detection performance on the ImageNet dataset. They proposed to use the negative of the maximum of the logit values (Max Logit) of a dnn as the score to distinguish between id and ood samples and established a baseline for large-scale setting. They also introduced new benchmarks for anomaly segmentation and ood detection in multi-label prediction. Decoupling max-logit <cit.> decouples the magnitude and the direction of the logit vector. This decomposes the max-logit score into product of max cosine (cosine of the angle between the feature vector and respective weights for each class) and max norm (norm of the product between feature vector and respective weights for each class). While max-cosine outperforms max-logit score consistently, max norm falls short by a large margin. 

GradNorm <cit.> is among the approaches using the gradients of the parameters of the dnn.  The approach utilizes a label-agnostic score function to formulate the ood detection problem as a binary classification problem. For a dnn parameterized with 𝐰, GradNorm formulates the score function for an input 𝐱 as S(𝐱) = ||∂ D_KL(𝐮 || softmax(f(𝐱)))/∂𝐰||_p, where 𝐮 denotes the uniform distribution and D_KL denotes the KL-divergence. The intuition behind this formulation is that the prediction tends to concentrate around the target class for id inputs which should result in a larger value of the KL divergence and its derivative. The parameter vector 𝐰 contains the concatenated parameters from different layers in a single vector regardless of their original shape. The authors further show that the gradients from the last layer's parameters are sufficiently informative for distinguishing between the id and ood inputs. Through empirical results, they showed that the L_1 norm is the most effective for GradNorm. The authors attribute the improvement brought by GradNorm to the joint information from the feature space and the output space that is utilized in this framework. Specifically, they showed that the score function can be written as equation <ref>, where the x_is are the input features to the logit layer, C is the number of classes, T is the temperature, and f_j denotes the logit for class j. U=∑_j=1^m|x_i| and V represent the information about the feature space and output space respectively.





 §.§ Approaches based on Energy Functions


The work <cit.> provides a solution to ood detection problem in the logit space. It connects the posterior probability in Gibb's distribution and the posterior probability of the softmax layer and proposes the equivalent of Helmholtz energy function for the softmax function as the score function for discriminating between id and ood samples. The key observation is the similarity between Gibb's density function in Equation <ref> and the softmax function  in Equation <ref>. By connecting these two equations, the energy for a specific sample is defined by E(𝐱, y) = -f_y(𝐱):

    p(y | 𝐱) = e^-E(𝐱,y)/T/∫_y' e^-E(𝐱, y')/T = e^-E(𝐱,y)/T/ e^-E(𝐱)/T


    p(y | 𝐱) = e^f_y(𝐱)/T/∑_i=1^i=K e^f_i(𝐱)/T


This also leads to the equivalent of free-energy E(𝐱) for dnn as E(𝐱;f) = -T ∑_i=1^i=Ke^f_i(𝐱)/T. The authors show that the Energy function is affinely related to the logarithm of the probability of the input p(𝐱). On the other hand, the logarithm of the softmax score or the softmax score itself is not related to log p(𝐱) linearly and depends on the maximum logit value. The authors also proposed an energy-based regularizer for training the dnn, shown in Equation <ref> and Equation <ref>,  and demonstrated that it improves the performance of the detector. 

    min_θ 𝔼_(𝐱, y)∼𝒟_in^train [-log F_y(𝐱)] + λ L_Energy


    L_Energy = 𝔼_(𝐱, y)∼𝒟_in^train (max(0, E(𝐱_in) - m_in))^2 + 𝔼_(𝐱, y)∼𝒟_out^train (max(0, m_in - E(𝐱_out)))^2

Despite having the advantage of being easy in nature, the energy-based ood detection method requires access to the ood distribution either to determine the threshold or to train the network with regularization. 




 §.§ Approaches based on Activation Shaping




l0.35
    
    < g r a p h i c s >

    The per-unit activations for the penultimate layer for OOD data (iNaturalist) are considerably different when using true (top) statistics as opposed to mismatched (bottom) BatchNorm statistics <cit.>.

    
    


These approaches assume that the features of a dnn affect the output differently for id and ood samples. As such, they rely on some predefined transformation of the activation values of intermediate layers of dnn to differentiate between id and ood samples. ReAct <cit.> is a rectification operation on the activation in model space to facilitate the detection of ood samples by making id and ood samples more separable. The activations from the penultimate layer h(𝐱) of the dnn are truncated using ReAct operation given by h(𝐱) = (h(𝐱; c)), where (x;c)=min(x,c).  


The rectified activations are used to obtain the model output as f^ReAct(𝐱;θ)=𝐖^Th(𝐱)+𝐛. These outputs can be used with any score function for ood detection. The detection performance depends on the threshold value c. Through empirical study, the authors showed that setting the value of c to the 90th percentile of the activations works best for ReAct. The authors theoretically showed that ReAct suppresses the activations more for the ood samples than id samples. This translates to a larger reduction in the output value for the ood samples making them  separable from the id samples. When batch-normalization statistics calculated for id data are applied to ood data, significantly different activation patterns emerge as shown by the authors in Figure <ref>. The approach is also shown to work for dnn architectures using normalization techniques other than batch-normalization (e.g. weight normalization, group normalization).





Activation Shaping (ASH) <cit.> modifies the activations of a dnn in a post-hoc manner to facilitate the detection of ood samples using existing scoring methods. The work assumes that modern overparameterized dnn produce redundant representations for the task it is trained for. Therefore, the representation can be greatly simplified while preserving performance and providing additional benefits in ood detection. Based on this hypothesis, the authors propose three variants of the ASH algorithm where they set values smaller than p-th percentile of the representation to 0 as a form of activation pruning. The first variant, ASH-P, keeps the un-pruned activations unchanged. In the second variant, ASH-B, the un-pruned activations are assigned a value such that their total value equals that of the total value of the original activations. The third variant, ASH-S, calculates s_1 and s_2, the sums of the activations before and after pruning, and scales the unpruned values with exp(s_1/s_2). They apply this activation shaping to the penultimate layer of the dnn. The effect of the choice of the value of p for setting the pruning threshold is dependent on the architecture and dataset and takes values in the range of [65, 95]. While this method is relatively simple, it integrates the detector into the architecture impacting the performance on id data. The unique perspective of this method indicates that sparsity can be beneficial to ood detection.

    S(𝐱) = 1/CT(∑_i=1^m|x_i|)(∑_j=1^C|1 - C.e^f_j/T/∑_j=1^Ce^f_j/T|) Δ=1/CTU · V

This approach is label-agnostic, ood data agnostic, and can be utilized using back-propagation. Conversely, other approaches that are comparable in accuracy require considerable computational overhead and also require access to ood data <cit.>. The limitation of this work lies in the assumption that the softmax output probability would be more uniform for ood data. It has been shown by <cit.> that dnn can assign a high probability to a specific class even when presented with random noise. This limitation is reflected in the results presented in the work where it is shown that GradNorm still reports fpr of 43.16% at 95% tpr.

Similar to <cit.>, the work in <cit.>  suppresses the irrelevant activations. However it also suppresses weights based on Shapley value <cit.>. The impact on the performance for id requires further study in such cases where the dnn architecture is changed.  The work <cit.> takes a similar approachwhere they assume that the deep features follow a Gaussian distribution. They define features that fall into the high-probability region as typical features and they estimate the typical features with the batch-normalization layer statistics. For a batch-normalization layer with mean μ and standard deviation σ, the features that are within λσ from μ are estimated to be the typical features. The features are then clamped to have values between μ-λσ and μ+λσ meaning that only the typical features can pass unperturbed. 

vra <cit.> adopts a piece-wise activation shaping function similar to <cit.>. The difference is that they clamp the lower values to zero and shift the intermediate values by a fixed amount γ. The inspiration to shift the intermediate values comes from the variational formulation of the activation-shaping function, which shows the need to suppress abnormally low and high values and boosting intermediate feature values. Both <cit.> and <cit.> are  inspired by ReAct <cit.>. The recent work <cit.> formulates the design of feature-shaping functions as an optimization problem and shows that the previous approaches <cit.> approximate the optimal solution. The authors provide two versions of the optimization problem. The first one utilizes the ood data and the second one is ood data-free approach. For the ood data-free approach, the authors also provide a closed-form solution.





 §.§ Approaches based on Mitigating Overconfidence in Prediction



LogitNorm <cit.> observes that the norm of the logit vector increases as the training progresses. As up-scaling the logits increases the maximum softmax probability, to normalize the logit values to address the issue. Their key idea is to decouple the influence of the norm of the output from the training objective and optimization step. They decomposed the logit vector 𝐟 into its norm and directions as given by 𝐟 = ||𝐟|| ·𝐟̂. They proposed to constrain the logit norm to a fixed value α, leading to the constrained optimization problem given by Equation <ref>:

    𝔼_𝒫_𝒳𝒴[ℒ_CE(f(𝐱;θ, y))]
     ||f(𝐱;θ)||_2 = α

They showed that this constrained optimization does not work in practice and propose to use the logit vector normalized with its L_2 norm instead. This leads to the optimization objective given by Equation <ref>:

    ℒ_logit-norm(f(𝐱; θ), y) = - e^f_y/(τ ||𝐟||)/∑_i=1^ke^f_i/(τ ||𝐟||)

where the temperature parameter τ modulates the logit-norm. In this case, the compounded temperature parameter τ ||𝐟|| shows dependence on the input as 𝐟 = f(𝐱;θ). The authors argued that the input-dependent temperature scaling provides superior performance in terms of ood detection and calibration of the output. While providing a simple fix to the overconfidence issue of the dnn, LogitNorm approach has the drawback that the output distribution gets shifted due to the normalization. The paper does not provide any theoretical guarantees for improvement in calibration performance. While this approach can improve the performance of different scoring functions, it requires modification of the training objective. Hence,  it cannot be applied to pretrained dnn.


mood <cit.> shows that the overconfident prediction in the ood samples is due to incomplete learning of the id features. As such, it can be mitigated by using masked image modeling (MIM) as a pretext task. The authors argued that using the reconstruction of the masked image as the pretext task forces the dnn to learn the id data distribution instead of just patterns for classification and this improves the ood classification performance. The Mahalanobis distance <cit.> proves to be the most effective. This suggests a Gaussian distribution of features and well-separated class boundaries. The overall approach of mood consists of four steps - a) pretraining the Masked Image Modelling vit on ImageNet-21k., b) intermediate fine-tuning of vit on ImageNet-21k, c) fine-tuning of pre-trained vit on the id dataset, and d) extracting features from the pre-trained vit and using Mahalanobis distance metric to detect ood samples. Despite the superior performance shown by mood, a shortcoming is that only the auroc is used as the evaluation metric.


The work in <cit.> addresses the problem of lack of supervision from ood data during training resulting in overconfident predictions on data from unknown domains. The authors propose an unknown-aware training approach for object detection tasks without accessing ood data. Conversely, they create virtual outliers to regularize the model. They model the feature representation h(𝐱,𝐛) of object instances 𝐱 (𝐛 is the bounding box) with class-conditional multivariate Gaussian distribution having mean μ_k, and covariance matrix Σ. After fitting the features from the penultimate layer of the network to the class conditional Gaussian distributions, they sample feature instances (𝐯) from these distributions to use for regularizing the dnn. For regularization, they use an uncertainty loss ℒ_uncertainty as follows:

    ℒ_uncertainty = 𝔼_𝐯∼𝒱[-1/1 + exp^-ϕ(E(𝐯;θ))] + 𝔼_𝐱∼𝒟[-exp^-ϕ(E(𝐱;θ))/1 + exp^-ϕ(E(𝐱;θ))],

where 𝒱 and 𝒟 denote the virtual outlier and training datasets respectively, while ϕ(·) is a non-linear MLP function that allows learning flexible energy surface. This work provides a flexible approach for use in ood detection for object detection tasks, which is an underexplored area. However, the performance still needs to improve significantly. Indeed, the reported fpr gets close to 50%, which may be unacceptable in real-world tasks. 




 §.§ Approaches based on Training with Regularization


Some prior work assumes access to ood data – also referred to as  surrogate ood data – that can be used at training time. Specifically, these approaches propose a regularization term to be added to the training loss that helps improve detection of ood samples. The key issue with surrogate OOD data is that the OOD distribution may not be fully characterized. As a result, these methods perform poorly on unseen ood distributions. oe <cit.> is the first to propose to use such surrogate data. They assumed to have training OOD data that is disjoint from the test ood data and proposed a generalized training scheme using the regularizer ℒ_OE, which is task-dependent. The authors set ℒ_OE to cross-entropy loss between the dnn output and uniform distribution for supervised learning tasks. If the task is of density estimation and without any label, then ℒ_OE is set to margin ranking loss between output of dnn f - f(x) and f(x'), where x and x' are sampled from id and surrogate ood distribution respectively. 

doe <cit.> proposes an improvement over oe by making it distribution-agnostic. doe is composed of ood generation using model perturbation and worst OOD regret (WOR) based training. The perturbation is done with a matrix multiplying the dnn parameters, which leads to features resembling ones sampled from the ood distribution. The WOR score measures the worst performance of the ood detector and helps discover the hardest ood samples. The hardest ood samples are simulated by perturbing the model parameters. divoe <cit.> took a similar approach as doe. However, instead of generating new ood samples through model perturbation, they perturbed a portion of the surrogate ood data and use a mix of the perturbed (that maximize ℒ_OE loss) and unperturbed surrogate data to train the dnn.   

The work by Choi et al. <cit.> proposed a balanced energy regularization loss L_energy, bal, which is built upon the energy regularization loss L_energy proposed in <cit.>. The authors fine-tuned the linear layers of a dnn with auxiliary samples (i.e. ood samples) and used the proposed loss as a regularization factor, which increases the energy margin between id and ood samples. The proposed loss accounts for the imbalance in the ood data used for training the dnn. However, such training requires access to ood data and is computationally expensive. Another line of work considers adding regularizers to the loss function without using any surrogate ood data. Conversely, these approaches impose constraints on latent representations of id data that ood data are unlikely to achieve. For example, <cit.> proposed to constrain the embedding generated by dnn to a union of 1-dimensional subspaces where inputs from each class occupy one dimension. The authors achieve such embeddings by imposing a cosine similarity constraint between the embeddings and the weight vector corresponding to the class it belongs to. They also imposed orthogonality constraints among the weight vectors of different classes so as not to impact the id accuracy. The ood samples are  detected by estimating the probability of a test sample belonging to one of the classes using spectral discrepancy (angular deviation of the embedding from first singular vector corresponding to a class) measurement. If the probability of the sample belonging to any class is very low (ideally zero), then it is declared to be an ood sample. 

Du et al. <cit.> proposed Siren, which added a projection head after the penultimate layer of the dnn. Such projected features are modeled with a vmf distribution <cit.> and trained to align with class conditional priors. The score function is designed with maximum class conditional likelihood, while the regularizer used to train the projections ensures that the id representations align with their class conditional priors, which is unlikely for ood samples.  Finally, the work in <cit.> proposed CIDER, which utilized the distance from class prototypes in a hyper-spherical embedding space. The loss function in CIDER is divided into two sections. The first one makes the embedding compact around each class prototype, while the second increases the angular distance among the class prototypes. The authors show that using the two part loss provides better separability between id and ood samples. 





 §.§ Approaches based on Distance Functions


The intuition behind distance-based ood detection is that the latent representation of the id and ood samples can be divided into separate clusters. As such, the distance between representations of id samples should be smaller than the distance between id and ood representations. The typical distances used are Mahalanobis distance, Hamming distance, and L2 distance. As these approaches typically use class-conditional prototypes, they are a form of prototype learning. 

Lee et al. <cit.> proposed to model the features using class-conditional Gaussian distributions and use the Mahalanobis distance as the score for ood detection. The distance is calculated for each layer separately, while auxiliary ood data is used to fit the distances collected from different layers into a logistic regression model which greatly diminishes its utility as it requires access to the ood distribution to be detected and trains a separate logistic regression model for each ood distribution. In contrast, <cit.> proposes to use L2 distance of the test sample to the k-th nearest neighbor from the training dataset, thus leading to a DNN- and ood-agnostic approach. The authors show that the performance of the KNN based ood detector improves if the model is trained with contrastive learning. This is because the features form tighter clusters and the distance of the ood data from any particular cluster increases. 


Ming et al. <cit.> proposed an approach based on cosine similarity, where they measure the alignment of the features of an input image with concept vectors obtained from the language model. The similarity score is then used in place of logit to calculate the softmax probability over the concepts from the language model. The similarity represents the probability of the image aligning with a specific concept. The maximum of these probabilities work as the score for ood detection. The work in <cit.> uses von Mises-Fisher distribution to model the class prototypes in hyper-spherical embedding space. The recent work <cit.> used a similar loss function as <cit.> and showed that it improves the ood generalization performance of dnn. Conversely, Lu et al. <cit.> showed that using a single class prototype is detrimental to learning the representation properly. The authors propose to use multiple prototypes for each class and assign each sample to a prototype using soft assignment. They also model the clusters using vmf and use Mahalanobis distance as their ood scoring metric. Gomes et al. <cit.> proposes to use geodesic distance between two data distributions – i.e., the Fisher-Rao distance <cit.> – arguing that this information geometric distance is more suitable for measuring difference between two distributions than the Mahalanobis distance.  Finally, Olber et al. <cit.> proposed an approach based on the concept of neural activation pattern. Specifically, the authors extracted activations from all layers of a dnn, binarize them based on a threshold value, and concatenate them in a vector. If the Hamming distance of this binary vector, which represents the activation pattern of the dnn, to the nearest id sample is greater than a certain threshold, the input sample is considered as an ood sample.  




 §.§ Other Approaches


The work in <cit.> proposed nac, which considers the difference in behavior of neurons for id and ood samples. Specifically, nac measures the frequency of activation of a neuron for training data in the form of coverage scores. The intuition is that ood samples trigger neurons that are not usually triggered by id data. The coverage score  measures the alignment of the triggering of the neurons with training data and should be higher for id than ood input. Although  inspired by <cit.>, the key innovation of this work is that nac may also be used as a regularizer during training to improve ood generalization. 

Yi et al. <cit.> connected ood generalization performance to shift in distribution. It also showed that the generalization bound tightens for dnn robust against perturbation to the input. The authors proposed to use adversarial training to make the dnn robust and showed that the performance of dnn improves when adversarially trained. This work established a connection between adversarial robustness and ood robustness and provided the insight that the problem of resilience of dnn against intentional and unintentional perturbation is connected.

The work in <cit.> estimated the typicality of selected test-statistics T_n(x) using kde on sets of statistics and creating a density estimate for individual samples by using product-of-experts (POE) structures. The POE provides the probability that an input is jointly typical for all test statistics. The id samples are typical for all test statistics providing higher probability while ood samples provide lower probability as they are not typical for all test statistics. The authors use different test statistics like posterior/prior cross-entropy, posterior entropy , posterior/prior KL divergence, posterior expected log-likelihood, and so on.

Finally, Huang et al. <cit.> proposed two density regularization methods, i.e.,  density consistency regularization and contrastive distribution regularization, to reliably calibrate and estimate sample density to identify ood samples. Density consistency regularization enforces the agreement between analytical and empirical densities of categorical labels, while contrastive distribution regularization helps separate the densities between id and ood samples. The logarithm of the sample density function is used as the score function for ood detection. 




§ WHY ARE INTENTIONAL AND UNINTENTIONAL PERTURBATION CONNECTED?


A fundamental similarity between intentional and unintentional perturbations is that their effect is similar. Indeed, both can be modeled as a perturbation δ such that for a dnn f and input image 𝐱, f(𝐱+δ) ≠ f(𝐱). This suggests that intentional perturbation can be modeled as ood examples. This has been explored in <cit.>, where the adversarial inputs are modeled as a special case of ood samples. The presence of natural adversarial samples <cit.> strengthen the support for connection between intentionally and unintentionally perturbed samples.

Another similarity  is that the dnn becomes overconfident in the case of both intentional and unintentional perturbation. This indicates that detection algorithms will share common characteristics. Prior work also reflects this intuition, as they attempt to present a common detection framework <cit.>. On the other hand, the two communities for detection of intentional and unintentional perturbation can also benefit from exploring cross-domain approaches. For example, the detection of intentionally perturbed samples utilize the Bayesian Uncertainty estimation model <cit.>, yet this approach has not been explored for ood detection. Moreover, intentional perturbation detection use class-conditional Gaussian distribution <cit.>, mixture of Gaussian <cit.>, kernel density estimation <cit.>, and lid <cit.>. The community for unintentional perturbation detection has explored additional distributions - vmf <cit.> for example which is better at modeling embedding on a hypersphere. Another example of cross-domain opportunity is the modeling of the activation patterns for intentional and unintentional perturbation in terms of discrete states. The work in <cit.> models the activation from the penultimate layer using binary or quaternary codes,  while <cit.> extracts binary neural activation pattern from multiple layers to characterize the label shift. These connections signify that the detection of intentional perturbation can benefit from exploring approaches adopted in unintentional perturbation detection. On the other hand, the assumption of access to perturbed samples is a common bottleneck that can be jointly explored. This leads to incomplete modeling of the perturbed samples. 

Another connection lies in the intermediate representation in dnn. For example,<cit.> shows that adversarial inputs tend to activate all filters more or less uniformly. A similar phenomenon is observed in case of unintentional perturbation, especially in the case of label shift, where the ood samples show unusually high activations for some filters. Such samples can be effectively detected by reshaping the filters activations through suppression <cit.>, typical feature selection <cit.> and feature reshaping optimization <cit.>. Since mitigating overconfidence issue improves the detection of unintentional perturbation <cit.>, we need to explore such approach also for intentional perturbation detection. 












§ SUMMARY OF EXISTING RESEARCH CHALLENGES IN PERTURBATION DETECTION


Security and Privacy Issues: The work in <cit.> showed that a large difference between the training loss and the testing loss can lead to membership privacy risk, i.e., the adversary may increase their ability to conclude that
an entity is in the input dataset <cit.>. Traditional training algorithms provide high confidence on both training and testing samples provided the dnn has low generalization gap, which leads to overconfidence. In order to alleviate the overconfidence issue, some approaches propose modified training loss to facilitate the detection of adversarial or ood examples <cit.>. While these approaches reduce the test error, the change in generalization gap is neglected, which introduces possible  privacy vulnerabilities. For example, <cit.> imposes specific pattern for unperturbed samples and any deviation from the pattern is detected as adversarial sample. The effect on privacy is not studied as the generalization gap with this training approach is not mentioned. If this approach increases divergence between the training loss distribution and test loss distribution, then the privacy of the DNN would be affected. 

Research in Domains Different from Multi-class Classification: The ood detection literature is predominantly focusing on dnn for multi-class classification. Settings such as multi-label classification, object detection, segmentation tracking are largely understudied. We were able to find only the work <cit.> focusing on unintentional perturbation for segmentation tasks and only the work <cit.> focusing on unintentional perturbation object detection. 

Detecting Perturbations at Scale: Most of the work in ood detection reports detection performance on CIFAR10 and CIFAR100 benchmarks. On the other hand, the work in <cit.> has shown that ood detection performance degrades significantly when considering real-world datasets. There is also a varying degree of performance as a function of the ood dataset and architecture. For example, <cit.> reports the fpr for Imagenet-1k benchmark varying from 22.60% to 54% in the Swin transformer architecture, while in the BiT-S-R101×1 architecture it varies from 80.35% to 97.25%. This variability hints at a current lack of  a dataset-agnostic ood  detection approach. Another key issue is determining which detector might work for a specific task during deployment. The average performance of the detectors would be unreliable in real-world as one would not know the distribution of ood samples and might encounter worst-case distributions on which the detector provides high fpr. This opens up another research question: can we create a detector performance whose performance is ood distribution agnostic?   

Computation vs Performance Trade-off: The aspect of application and hardware specific constraints (e.g. computational cost, latency, energy etc.) is mostly ignored in the current literature of perturbation and semantic shift detection. There is no benchmark for latency, and computation cost metrics and these are rarely reported. <cit.> is the only work we could find that reports the computation time for detection of intentional perturbation but there is no analysis of the computational cost or energy expenditure. These are important metrics for implementation in edge devices. As a result, it is important that these metrics are reported as well. 

Lack of Curated Dataset for OOD Detection: Most of the OOD detection literature focuses on semantic ood and on the generalization capability of the dnn for non-semantic ood. An interesting observation was recently reported in<cit.>, where the authors introduced the Imagenet-OOD dataset to separatesemantic and covariate shifts. Moreover, the authors showed that existing methods are susceptible to detecting covariate shift rather than semantic shift. This suggests that existing methods are mostly designed for semantic ood. This calls for curated datasets that (i) disentangle covariate shift and semantic shift and (ii) are sufficiently large-scale to capture real-world scenarios. 

Lack of Explainability: The existing literature lacks an explainable way to separate semantic and non-semantic ood. Specifically, current approaches toood detection cannot explain which features of the input lead the dnn to generate the  embeddings or the scores used to perform the detection. A more explainable ood detection method can provide insights and drive further innovation.

Joint Resilience to Intentional and Unintentional Perturbation: In our view, guaranteeing resilience to intentional and unintentional perturbations cannot be seen as separate issues. Due to the disjoint communities of adversarial robustness and ood robustness, the problem of joint resilience against both types of perturbations has been under-explored. The only work we have found is <cit.>. Although the work shows impressive performance on five different datasets, the number of images in different datasets and the variety is not sufficient to represent real-world conditions. The PACS <cit.>, VLCS <cit.>, and OfficeHome <cit.> datasets used contain 7, 5, and 65 classes with 9991, 10729, and 15588 example images respectively. To experiment on more realistic settings, a reasonable approach would be to consider datasets like DomainNet <cit.> and ImageNet-C <cit.>.      




§ CONCLUSIONS AND CALL TO ACTION


Over the last few years, a substantial amount of research has  identified various issues regarding the resilience of dnn, including susceptibility to adversarial attacks and distributional shifts. This  has spurred a significant amount of research aimed at enhancing the robustness and reliability of dnn. Efforts to bolster resilience have led to the development of novel techniques such as adversarial training, robust optimization, and domain adaptation. These approaches aim at making DNNs more robust against potential threats, while also improving their generalization across operational settings. The ongoing research into the underlying mechanisms of vulnerability can only advance through a collaborative effort from both intentional and unintentional perturbation detection communities. In this work, we have surveyed the state of the art in both intentional and unintentional perturbation detection, with the key aim of consolidating the approaches being employed. We have also listed a set of challenging research directions in this field. We hope that this survey will spur excitement in both communities and set new research directions in this topic.  



ACM-Reference-Format.bst





 Paper Title: MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities.




[NO \title GIVEN]
    [NO \author GIVEN]
    August 5, 2024
======================



[remember picture,overlay,shift=(current page.north west)]
[anchor=north west,xshift=0.7cm,yshift=-2cm]-1[1]
    < g r a p h i c s >
;





§ ABSTRACT 
MM-Vet, with open-ended vision-language questions targeting at evaluating integrated capabilities, has become one of the most popular benchmarks for large multimodal model evaluation. MM-Vet assesses six core vision-language (VL) capabilities: recognition, knowledge, spatial awareness, language generation, OCR, and math. However, its question format is restricted to single image-text pairs, lacking the interleaved image and text sequences prevalent in real-world scenarios. To address this limitation, we introduce MM-Vet v2, which includes a new VL capability called “image-text sequence understanding”, evaluating models' ability to process VL sequences.
Furthermore, we maintain the high quality of evaluation samples while further expanding the evaluation set size.
Using MM-Vet v2 to benchmark large multimodal models, we found that Claude 3.5 Sonnet is the best model with a score of 71.8, slightly outperforming GPT-4o which scored 71.0. Among open-weight models, InternVL2-Llama3-76B leads with a score of 68.4.




§ INTRODUCTION


 


Large multimodal models (LMMs) <cit.> evolve rapidly, demonstrating emergent abilities to solve complex tasks <cit.> that required multiple integrated capabilities, such as GUI navigation <cit.>, screenshot to code <cit.>, video understanding <cit.>, . To comprehensively evaluate LMMs, multiple benchmarks have been proposed, such as MME <cit.>, MMBench <cit.>, SEED-Bench <cit.>, MMMU <cit.>, and MM-Vet <cit.>. Notably, MM-Vet is designed to evaluate LMMs from a capability integration perspective, defining tasks based on their required core capabilities. The benchmark accepts open-ended responses and takes GPT-4 <cit.> to score model predictions, which better align with real-world scenarios. Such effective evaluation designs make MM-Vet widely utilized as a standard benchmark for LMM evaluation, as indicated by its leaderboard [MM-Vet leaderboard: <https://paperswithcode.com/sota/visual-question-answering-on-mm-vet>].

Despite its popularity, MM-Vet and other concurrent evaluation benchmarks <cit.> are beginning to fall short in assessing the more advanced capabilities that have emerged in the latest LMMs, such as GPT-4V <cit.> and its successors <cit.>. Specifically, one major limitation is the question format. The questions in MM-Vet are limited to a single image-text pair, lacking the capability to handle interleaved image and text sequences. This design choice was natural at the time of prior studies <cit.>, given that most LMMs only supported single image inputs. However, the ability to process arbitrarily interleaved image-text sequences is crucial for advanced LMMs and should be included in LMM evaluation. 

In addition to the six core capabilities defined in MM-Vet, we introduce an additional capability: “image-text sequence understanding.” This measures the LMMs' ability to process image-text sequential data, as shown in Figure <ref>. For example, to complete the task in Figure <ref>(c), the LMM needs to understand the text question followed by multiple sequential images. This task requires image-text sequence understanding alongside the capabilities of recognition, OCR, knowledge, and language generation defined in MM-Vet. The image-text sequence can contain multiple images of drastically different usages, such as multiple video frames in Figure <ref>(c) for temporal understanding, and spot the difference challenge in Figure <ref>(d) for image comparison. These capabilities to comprehend arbitrarily interleaved image-text sequences is one fundamental step towards stronger general intelligence.


Another limitation is the size of the evaluation set. MM-Vet only has 217 evaluation samples due to the difficulty of high-quality data collection. We aim to maintain the same high quality while further expanding the evaluation set size. 
We break down the challenge of creating high-quality evaluation samples into two steps: generating high-quality questions and producing reference answers. For the question generation, we find it difficult for crowd-sourcing workers to propose a meaningful and complicated question that covers various scenarios, , in Figure <ref> (a,b). 
Instead, we have researchers design and collect 517 questions covering various scenarios from daily life to expert/industry applications, extended from an exploratory report on GPT-4V <cit.>. For straightforward questions that can be answered in a few words, our experts directly annotate the reference answer. For questions that need long paragraphs to answer, we first employ the GPT-4V <cit.> to draft the response. Next, our experts correct draft if there is any error and then rephrase it into the final ground truth. 


After developing MM-Vet v2, we evaluate multiple advanced LMMs. Claude 3.5 Sonnet <cit.> achieves the highest performance with a score of 71.8, slightly surpassing GPT-4o <cit.> by 0.8 points. Notably, InternVL2-Llama3-76B <cit.>, an open-weight model, also delivers a very competitive score of 68.4.






























§ DATASET AND EVALUATOR





The same as MM-Vet <cit.>, we aim to build a high-quality evaluation set for large multimodal models. MM-Vet <cit.> defines six core vision-language capabilities, including recognition (Rec), knowledge (Know), OCR, spatial awareness (Spat), language generation (Gen), and Math. MM-Vet's question format is only an image-text pair, which obviously cannot measure the capability of processing sequential image and text data. To fill this gap, we introduce a new capability:

    
  * Image-text sequence understanding (Seq). It refers to the capability to understand and reason the relationships among sequential image and text streaming data. 


Then we carefully design 517 questions that require one or more capabilities, extended from MM-Vet <cit.> and an exploratory report based on <cit.>. Specifically, 218 questions are directly from MM-Vet <cit.>, and the others are newly collected in various domains from daily life to expert/industry applications. For those questions that can be answered shortly, we annotated the ground truths directly. For questions requiring long text to respond, we first utilize GPT-4V <cit.> to generate the answer drafts, then we carefully correct the wrong information in the drafts and rephrase them into final ground truths. Because this new dataset is the extension of MM-Vet <cit.>, we name it . See Figure <ref> for the proportions of each core capability and capability integration.








Next, we design the evaluator for , based on that of MM-Vet <cit.>. Different from MM-Vet, the input format of  is not only an image-text pair, it can also be image-text sequences. To demonstrate the position of the image in the sequences, we utilize <image> to represent the image position. The few-shot prompt for  is shown in Table <ref>.






§ EXPERIMENTS








 §.§ Experiment settings

We evaluate two types of LMMs on our : (1) open-weight LMMs including OpenFlamingo <cit.>, Otter <cit.>, LLaVA <cit.>, CogAgent <cit.>, Emu2-Chat <cit.>, InternLM-XComposer2 (IXC2) <cit.>, CogVLM <cit.> and InternVL <cit.>; (2) Closed source models including Claude <cit.>, Qwen-VL Max <cit.>, Gemini <cit.>, GPT-4V <cit.> and GPT-4o <cit.>).

As illustrated in <ref>, for each sample, we complete the prompt template with its question, ground truth, and the output from a specific LMM. When this filled prompt is input into GPT-4, it generates a score ranging from 0 to 1 for each sample. Although the temperature is set to 0, we observe some variance in GPT-4's outputs. To address this, we evaluate the outputs of the LLMs using GPT-4 five times. Due to space constraints, we present the average scores for capabilities and for some capability integrations, and include both the average and variance for the total score.





 §.§ Results

The main results of the different methods are presented in Table <ref> for each capability, and in Table <ref> for 16 integrations with the highest proportions.
Claude 3.5 Sonnet <cit.> and GPT-4o <cit.> are the leading models, achieving scores of 71.8 and 71.0, respectively. Claude 3.5 Sonnet <cit.> excels in recognition, language generation, OCR, spatial awareness, and knowledge. On the other hand, GPT-4o <cit.> surpasses in image-text sequence understanding and math. Among open-weight models, InternVL2-Llama3-76B stands out with a competitive performance score of 68.4.






















§ CONCLUSION

In this paper, we aim to evaluate the integrated capabilities of large multimodal models and extend MM-Vet into  by introducing a new core capability: Image-text sequence understanding, which assesses the ability to process vision-language sequences. Additionally, we ensure the high quality of evaluation samples while expanding the evaluation set size. Using  to benchmark large multimodal models, we found that Claude 3.5 Sonnet achieved the highest score of 71.8, narrowly surpassing GPT-4o, which scored 71.0. Among open-weight models, InternVL2-Llama3-76B emerged as the leader with a score of 68.4.












plain





 Paper Title: Neural Graph Matching for Video Retrieval in Large-Scale Video-driven E-commerce.















  JD.COM
  P.O. Box 1212

  
  China
  43017-6221




	JD.COM
	P.O. Box 1212
	
	
	China
	43017-6221




	JD.COM
	P.O. Box 1212
	
	
	China
	43017-6221





	JD.COM
	P.O. Box 1212
	
	
	China
	43017-6221






	National University of Singapore
	P.O. Box 1212
	
	
	China
	43017-6221




	JD.COM
	P.O. Box 1212
	
	
	China
	43017-6221














	§ ABSTRACT 
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
		With the rapid development of the short video industry, traditional e-commerce has encountered a new paradigm, video-driven e-commerce, which leverages attractive videos for product showcases  and provides both video and item services for users.
		Benefitting from the dynamic and visualized introduction of items, video-driven e-commerce has shown huge potential in stimulating consumer confidence and  promoting sales.
		In this paper, we focus on  the video retrieval  task, facing the following challenges: (1)  How to handle the heterogeneities  among users, items, and videos?  (2) How to mine the complementarity  between items and videos for  better user understanding?
		In this paper, we first leverage the dual graph to model the co-existing of user-video and user-item interactions in video-driven e-commerce and innovatively reduce user preference understanding to a graph matching problem. 
		To solve it, we further propose  a novel bi-level Graph Matching Network (GMN),  which mainly consists of node- and preference-level   graph matching.
		Given a user, node-level  graph matching aims to match videos and items, while preference-level graph matching aims to match multiple user preferences extracted from both videos and items.
		Then the proposed GMN can generate and improve user embedding by aggregating matched nodes or preferences from the  dual graph in a bi-level manner.
		Comprehensive  experiments show the superiority of the proposed GMN with significant improvements over   state-of-the-art approaches (e.g., AUC+1.9% and  CTR+7.15%). 
		We have developed it on a well-known video-driven  e-commerce platform, serving hundreds of millions of users every day.
		
	
	










<ccs2012>
 <concept>
  <concept_id>00000000.0000000.0000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>00000000.00000000.00000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
 <concept>
  <concept_id>00000000.00000000.00000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
 <concept>
  <concept_id>00000000.00000000.00000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
</ccs2012>


[500]Do Not Use This Code Generate the Correct Terms for Your Paper
[300]Do Not Use This Code Generate the Correct Terms for Your Paper
Do Not Use This Code Generate the Correct Terms for Your Paper
[100]Do Not Use This Code Generate the Correct Terms for Your Paper











20 February 2007
[revised]12 March 2009
[accepted]5 June 2009




 Neural Graph Matching for Video Retrieval in Large-Scale Video-driven E-commerce
    Lei Su
    August 5, 2024
=================================================================================






 





§ INTRODUCTION








	
	
	
	
		
		
	
	







	In the era of the information explosion, the recommender system has become the most effective way to help users  discover what they are interested from enormous data.
	As two basic Internet applications, e-commerce and  content recommendation  both  provide  the recommender service but focus on either item recommendation or content (e.g., video and live) recommendation.  
	
	Recently, with the thriving of online applications, there is a surge of video-driven  e-commerce <cit.>,  which integrates  video    and e-commerce for better  services. Benefiting from   vivid and  attractive video shows, video-driven  e-commerce provides a new business paradigm which improves user stickiness and activeness, and has become a new driving force for e-commerce development.
	
	
	
	For example, TikTok and Kuaishou integrate e-commerce with videos, while
	Taobao and  JD.COM     leverage videos   to   improve the user experience in traditional e-commerce.   As shown in Figure <ref>(a),
	JD.COM first incentivizes authors to produce videos  for related items,  and then conducts video recommendations for users.
	Due to the large-scale video set,
	industrial recommendation systems usually adopt the classical two-stage architecture: retrieval and ranking <cit.>.
	Given a user, the retrieval stage  aims to find thousands of candidate videos from the whole video set, then  the  ranking stage aims to  precisely select   hundreds of videos from the retrieved candidate videos.
	
	



	
	
	
	
	
	
	
	
	
	
	
	





































	
	
	
	
		
	
	


	
	
	In this paper, we focus on the video retrieval stage with the following challenges:
	
	
(1) How to  handle the heterogeneities   among users, items, and videos?

The characteristics of items (e.g., price, sales, and shop) are markedly different from videos (e.g.,  resolution, duration, and author) and them aim to stimulate user consumption and attract users stay, respectively.


This disparity compels us to treat user-item interactions and user-video interactions with distinct considerations, given the unique characteristics each type of interaction manifests.

(2) How to mine the complementarity  between items and videos for better user understanding?

While at a cursory glance these interactions may seem discrete, they exhibit a strong complementary nature upon closer examination. As exemplified in Figure <ref>, when a user interacts with both an item i_1 symbolizing a keyboard and a corresponding video v_1, it reinforces the user's profound interest in keyboards. In a contrasting scenario, if a user interacts with a beer item i_4 but directs their video interactions exclusively towards devices, such as keyboards, it suggests a potential superficial interest in the beer item.









Based on the above analysis, when  performing video retrieval  in video-driven e-commerce, we need to address the following new requirements:

	
	
  * Heterogeneous Interaction Modeling.  
	
	Different from traditional e-commerce, the video-driven e-commerce consists of complex heterogeneous interactions among users, videos, and items.
	
	Intuitively, user interactions on videos and items will show similar patterns and preferences. 
	
	As shown in Figure <ref>(b), user u_1 who wants to buy a keyboard will watch related videos v1,v2 and   click item i_1.
	
	How to model the co-existing of heterogeneous interactions is the basis of user preference understanding and video retrieval.
	
	
  * Node-level Graph Matching. 
	
	
	
	As shown in Figure <ref>(b), video v_1 and item i_1 are both keyboard-related (high relevance) while video v_1 and item i_4 are belonging to different categories (low relevance).
	
	So, when jointly modeling heterogeneous interactions for user preference understanding, it is necessary to properly distinguish and match the relevance between videos and items.
	
	
  * Preference-level Graph Matching. 
	
	Intuitively, diverse user preferences extracted from both videos and items are similar to each other.
	
	As shown in Figure <ref>(b), items {i_1, i_2, i_3 }  indicate user preference on electronic products, which is similar to user preference on video {v_1, v_2}.
	
	But, item {i_4}  indicates user preference on food, which is different from his video preference.
	
	It is necessity to mine the  the differences and relevances for user preference understanding.
	
	Particularly, how to match multiple user preferences on both videos and items is still an open problem.
	
		























In this paper, we study a newly emerging  paradigm video-driven e-commerce,  and then  formulate it as the dual graph to model the co-existing of users, videos, and items as well as interactions among them. 
After that, we reduce the user preference understanding  in the dual graph to a graph matching problem. To solve it, we propose a novel Graph Matching Network (GMN), which mainly consists of node- and preference-level graph matchings, to improve user preference understanding. 
























The contributions of this work are summarized as follows:




•  To our knowledge,  this  is the first time that  leveraging the dual graph  to model  the newly emerging video-driven e-commerce,   and innovatively reduce  the user preference understanding  problem to a graph matching problem.

•  To solve this, we propose a novel  Graph Matching  Network,  called GMN, which mainly consists of node- and preference-level   graph matching. Specifically,   node-level   graph matching aims to mine the potential   relevance between  videos and items  in the dual graph.  Furtherly,    preference-level graph matching further captures the  relevance of user preferences extracted from both  videos and items.








•  Extensive experimental results on both offline and online data demonstrate the superiority of the proposed GMN. Currently, it has been deployed on one of the largest video-driven e-commerce, affecting hundreds of millions of users every day.






















§ RELATED WORK

Recommendation System (RS), which mainly focuses  on  user preference understanding and recommends diverse types of candidates (e.g., items and videos) to them, 
has accompanied  diverse paradigms, such as  item recommendation <cit.>, video  recommendation <cit.>, and business-driven recommendation  <cit.>. 

Real-world    RS usually consists of two stages: retrieval   <cit.>
and   ranking   <cit.>.
YoutubeDNN <cit.> retrieves videos for online video services, while MIND <cit.> retrieves items for e-commerce.




Unlike them, 

we study a newly emerging video-driven e-commerce, which models the co-existing of items and videos.

Cross-domain Recommendation (CDR) attempts to learn useful knowledge from the source domain   to help the target domain. 
Hu et al.  <cit.> propose CoNet  to perform 
dual transfer at the unit level.   <cit.> unify both source and target domains as
graph-structured data  and perform CDR via graph learning.


Significantly different from the above works, we 

we first model video-driven e-commerce 
as a dual graph and reduce it to a graph matching problem.






Graph Neural Networks  (GNNs) generalize deep learning to graph-structured data, where graph convolution <cit.> aims to aggregate  neighbors information  and update node embedding,
while graph pooling


<cit.>  aims to aggregate all node to update graph embedding.


Recently, many works <cit.> apply GNNs to      recommendation scenarios. 
SURGE <cit.> reduces   user behaviors into graphs   and then extracts users' core interests via cluster-aware   graph pooling. 
Furthermore, several works <cit.> generalize GNN to multiple interactions (or domains).
MGFN <cit.> proposes a multi-graph learning for cross domain video recommendation.


Different from the above works, we first propose a novel bi-level graph matching network  to align  user behaviors and  preferences on both videos  and items    for   retrieval.




























































































































































§ PRELIMINARIES






	Dual   Graph. 

The dual graph,  termed as 𝒢=(𝒢^U-V,𝒢^U-I), describes the co-existing of  user  behaviors on both videos and items in video-driven e-commerce. Here  𝒩=𝒩^U∪𝒩^V∪𝒩^I is the union set  of  User, Video, and Item,  respectively. Also,   ℰ=ℰ^U-V∪ℰ^U-I is the edge set, where	ℰ^U-V=⟨𝒩^U, 𝒩^V⟩ denotes User-Video interaction  and ℰ^U-I=⟨𝒩^U, 𝒩^I⟩ denotes User-Item interaction. 




Example. 
Figure <ref>(b) gives an illustrative example of the dual graph in video-driven e-commerce.
Given  a user u and the  dual graph  𝒢, we can obtain his neighbor sets, including 𝒩_u^U-V={ v|⟨ u, v ⟩∈ℰ^U-V} from 𝒢^U-V and
𝒩_u^U-I={ i|⟨ u, i ⟩∈ℰ^U-I} from 𝒢^U-I.
In the dual graph, user u_1's  neighbor sets  are  𝒩_u_1^U-I= {i_1,  i_2, i_3, i_4 }
and   𝒩_u_1^U-V= {v_1, v_2  }, respectively.
Furthermore, we also  extract user-centric  sub-graphs  for user u  from the dual graph, termed as    𝒢_u=(𝒢_u^U-V,𝒢_u^U-I).  
In detail, 𝒢_u^U-V  contains multi-hops neighbors of user u and himself (i.e., 𝒱_u^U-V = 𝒩_u^U-V∪ u), as well as induced edges ℰ_u^U-V. 
Note that,  dual graphs can be widely used in many other related fields, such as  the co-existing of user-music and user-movie, or even  the co-existing of user-item(old/warm domain) and user-item(new/cold domain).




Finally, we formulate the video retrieval problem in video-driven e-commerce as follows:


    max_v ∈𝒩^V Pr(v| u; 𝒢),

which aims to retrieve candidate videos v from the whole video set 𝒩^V for user u based on the dual graph 𝒢.




















§ THE PROPOSED MODEL



In this section, we present a novel  Graph Matching  Network, which aims the  to match the users’ behaviors as well as preferences via both node- and preference-level graph matching.



The overall framework of GMN is 
shown in Figure <ref>.
































 §.§ Embedding Initialization




In this section, we first initialize the node embeddings of users, videos, and items via   feature embeddings, and then update them via GNNs.






Taking   embedding initialization   of video v as an example,  we  have:



    𝐱_v= ||_f𝐱_v^f,


where || denotes the vector concatenation, 𝐱_v and 𝐱_v^f denote the initial embedding of video v and the  f-th feature embedding of video v, respectively. 












































After that, we further utilize two interaction-specific    GNNs (i.e., GNN^U-V and GNN^U-I) to update corresponding embeddings.


Given  video v and 𝒢^U-V,  GNN^U-V is able to update the video 𝐱_v^U-V embedding via aggregating its neighbors 𝒩_v^U-V, as follows:


    𝐱_v^U-V = GNN^U-V(𝐱_u| u ∈𝒩_v^U-V).



Note that,   we focus on bi-level graph matching and 


simply instantiate  GNN^U-V  as the  MeanPooling <cit.>. 
Similarly,  the embedding of item i via  GNN^U-I is 𝐱_i^U-I.
For user u, we can also get his embeddings on the  dual graph    𝐱_u^U-V and  𝐱_u^U-I.










 































































































































































































































Until now, 
a method way for
video retrieval   is  
to fuse user preferences on   videos and items i.e., 𝐱_u^U-V|| 𝐱_u^U-I, and then retrieve top videos based on the similarity of their embeddings, 

suffering from  the following weaknesses:

(1) In video-driven e-commerce,  video production  actually revolves around items   and  aims to promote the item purchase. Although videos and items are different types of nodes, there exists potential relevance between them.
(2) It only models each user as a single node embedding while fails to explicitly capture the  graph structure  surrounding users, which  also provides  valuable information for understanding user preferences. So, it is necessary to  encode the user-centric sub-graph as a whole for video retrieval.




 §.§ Node-level Graph Matching


In this section, we propose node-level graph matching to mine the  relevance between videos and items in the dual graph,  and then enhance their connections via  aggregating  cross-graph neighbors based on    the  matching scores.




To handle large-scale video-driven e-commerce, we 
first extract user-centric  sub-graphs (i.e.,  𝒢_u^U-V and 𝒢_u^U-I) for user u, 
and then actually perform the node-level graph matching GM_NL on them, shown as follows:


    𝐑^u_ = GM_NL(𝒢_u^U-V,𝒢_u^U-I).

Here   𝐑^u  ∈ℝ^|𝒱_u^U-V| × |𝒱_u^U-I| 
denotes the  personalized relevance   matrix of user u in   node-level.


Given a   user u, 

the  personalized   relevance   score r^u_v,i  between   video v ∈𝒢_u^U-V and   item i ∈𝒢_u^U-I is shown as follows:


    r^u_v,i
    =(𝐱_v^U-V)^T ·𝐌·𝐱_i^U-I.

Here (·)^T denotes the transposition operator. 

𝐌 is  a learnable metric matrix, harmonizing the embeddings of heterogeneous nodes (video v.s. item).


Obviously, the  personalized relevance  score  r^u_v,i is dynamically changed with regard to users, videos, and items.






After obtaining the personalized relevance  scores {r^u_v,i}, we normalize them via the softmax function to get the personalized matching scores s^u_v← i and s^u_v→ i, 
shown as follows:


    s^u_v← i=  softmax_i(r^u_v,i)= exp(r^u_v,i)/∑_i'∈𝒱_u^U-Iexp(r^u_v,i'),



    s^u_v→ i=  softmax_v(r^u_v,i)= exp(r^u_v,i)/∑_v'∈𝒱_u^U-Vexp(r^u_v',i).


Note that  the above matching scores are asymmetric since they are normalized on different node sets (i.e., 𝒱_u^U-I and 𝒱_u^U-V).
Given a user u,  the personalized matching score s^u_v← i
measures the    matching level from item i to video v.  
Intuitively, aggregating relevant items embeddings into video embeddings is able to  the alleviate the  heterogeneity and sparsity problem,  improving user preference understanding and  the effectiveness of video retrieval.








Taking personalized matching scores (e.g., {s^u_v← i})
as coefficients, we can properly summarize all information ∑_i ∈𝒱_u^U-I r^u_v← i·𝐱_i^U-I which needs to propagate from relevant items i ∈𝒢_u^U-I to video v,  and then  utilize it to enhance  the video embedding.




The final video embedding 𝐡_v^U-V of video v is shown as  follows:


    𝐡_v^U-V  =  𝐱_v^U-V||   ∑_i ∈𝒱_u^U-I r^u_v← i·𝐱_i^U-I  .


Also, we   conduct information propagation from    videos to items, and  learn  the final   embedding 𝐡_i^U-I  of item i.





Although node-level graph matching is able to learn the potential connections between videos and items and improves their embeddings via bi-directional information propagation, it only focuses on node-level modeling and largely ignores the user preferences inferred from user interactions.













 §.§ Preference-level Graph Matching






In this section, we introduce a novel method for preference-level graph matching. This method mines the relevance of user preferences for videos and items. Subsequently, this relevance is utilized in cross-graph propagation to refine preference embeddings.







  §.§.§ Extracting User Preferences via Dual Graph Pooling


Specifically, we propose a novel dual graph pooling method (i.e., GPN^U-V and GPN^U-I) to cluster user-centric sub-graphs (i.e.,  𝒢_u^U-V and 𝒢_u^U-I) as preference graphs, where each node indicates  a cluster of similar videos or items and  corresponding user preferences.





Given a user u and  𝒢_u^U-V, a graph pooling network   GPN^U-V is able to cluster all video embeddings  {𝐡_v^U-V   | v ∈𝒱_u^U-V} into k_1 preferences, and constructs the condensed preference graph 𝒵_u^U-V:




    𝒵_u^U-V = GPN^U-V(𝒢_u^U-V),





    {𝐳_u^1^U-V,  ⋯, 𝐳_u^k_1^U-V}= GPN^U-V(𝐡_v^U-V  ),

where 𝐳_u^k_1^U-V is the embedding of  the k_1-th user preference.



If we set k_1 > 1, then   GPN^U-V actually extracts   multiple  types of user preferences. 
Therefore, we simply instantiate GPN^U-Vas the MeanPooling <cit.>


Similarly, we use    GPN^U-I to extract   k_2   preference embeddings   {𝐳_u^1^U-I,   ⋯, 𝐳_u^k_2^U-I | u^k_2∈𝒵_u^U-I}    for user u  from 𝒢^U-I.







  §.§.§ Improving Preference Embedding via Preference-level Graph Matching















































Intuitively, user preferences on both videos and items also have potential relevance. 

So we further propose the preference-level graph matching GM_PL, which  learns the   preference-level   relevance   matrix  𝐏^u ∈ℝ^k_1 ×  k_2 between preference graphs (i.e.,  𝒵_u^U-V and 𝒵_u^U-I), shown as follows:


    𝐏^u_ = GM_PL(𝒵_u^U-V,𝒵_u^U-I),

where each element p^u_k_1,k_2 indicates the relevance between the k_1-th video preference  and  the  k_2-th item preference of user u,  as follows:


    p^u_k_1,k_2
    =(𝐳_u^k_1^U-V )^T ·𝐳_u^k_2^U-I.



Then, we also normalize  p^u_k_1,k_2 on 𝒵_u^U-V and 𝒵_u^U-I  via softmax, and get corresponding matching scores s^u_k_1← k_2 and s^u_k_1→ k_2. Then, the embeddings of preference nodes can be enhanced via bi-directional propagation, as follows:

    𝐞_u^k_1^U-V  =  𝐳_u^k_1^U-V||   ∑_u^k_2∈𝒵_u^U-I s^u_k_1← k_2·𝐳_u^k_2^U-I  ,



    𝐞_u^k_2^U-I  =  𝐳_u^k_2^U-I||   ∑_u^k_1∈𝒵_u^U-V s^u_k_1→ k_2·𝐳_u^k_1^U-V  .


Interactively performing graph pooling and graph matching on preference graphs for L times, we can  progressively compress the original graph into  a single  node  (i.e., k_1=k_2=1),  termed as 𝐞_u^U-I and 𝐞_u^U-I,  representing the overall user preferences.






 §.§ Prediction and Optimization



To comprehensively understand user preferences, we   fuse user preferences on both videos and items, and get 
the final user embedding 𝐳_u of user u, as follows:





    𝐳_u= MLP(𝐞_u^U-V || 𝐞_u^U-I|| 𝐱_u^U-V|| 𝐱_u^U-I).


Since the feature of   video  si  much simpler and more stable than   users, we only perform GMN for user modeling,  and the final embedding  𝐳_v of  video  v is actually 𝐱_v^U-V. Similar to previous works<cit.>,  we do not use a more sophisticated fusion network (e.g., attention-based fusion), so the effectiveness of bi-level graph matching can be clearly verified. 



































































The estimated preference ŷ_u,v of  user u  towards the target video v is   the   inner product of their embeddings:


    ŷ_u, v= (𝐳_u)^T ·𝐳_v.




The overall loss function is as follows:


    ℒ = ∑_(u, v, v_neg) ∈𝒟-lnσ( ŷ_u,v - ŷ_u,v_neg) + λ· ||Θ||^2_2,






where Θ denotes all trainable model parameters, and λ
controls the L2 regularization strength to prevent overfitting. The whole model can be optimized via back-propagation.








 §.§ Model Analysis

 Here we give the analysis of the proposed GMN as follows:
 
 •  Considering the   characteristics of video-driven e-commerce, 
 we first represent it as the dual graph, and then reduce user preference 
 understanding to a bi-level graph matching problem. 
 So, the proposed GMN focused on matching relevant nodes (e.g., videos and items) and preferences properly in a bi-level manner, rather than designing better GNN architectures to learn nodes and preferences embeddings.  Similar to previous works<cit.>, we adopt basic GNNs as backbones, so the effectiveness of  bi-level graph matching  could be clearly verified.
 
 


•  We  analyze  the time complexity of GMN in both the node- and preference-level graph matching. In the  node-level graph matching,  if we directly  match all nodes between   𝒢^U-V and 𝒢^U-I, the computational cost is  𝒪(|𝒩^V| · |𝒩^I|), which is unacceptable for real-world scenarios. Fortunately, we actually match  user-centric  sub-graphs (i.e.,  𝒢_u^U-V and 𝒢_u^U-I) , which significantly reduce the time complexity to    𝒪(|𝒢_u^U-V| · |𝒢_u^U-I|). Similar to the node-level, the    time complexity of preference-level graph matching is     𝒪(|𝒵_u^U-V| · |𝒵_u^U-I|).


•  Then, we further  analyze the space complexity of our GMN, which mainly depends on the learnable parameters in embedding matrixes, rather than deep neural networks. Assuming we have total N nodes and F features in the dual graphs, the space complexity of GMN with feature embedding is  𝒪(F · d), which is significantly less than ID embedding 𝒪(N · d). In summary, the proposed GMN is able to efﬁciently handle  large-scale e-commerce. 

 





























§ OFFLINE EVALUATIONS



















 §.§ Datasets.
 









Although video-driven e-commerce has grown rapidly on many platforms, related data is protected by privacy   and security policies.
Therefore, we sample   user logs  from a real-world video-driven e-commerce platform, and then extract two large-scale dual graph datasets from different time periods (7-days, and 30-days) for verification.








We   also use a public dataset MovieLens-1M released by GIFT<cit.> to mimic  video-driven e-commerce, which takes user-movie (warm) as user-item interaction and user-movie (cold) as user-video interaction, respectively.  





The details of the datasets  are shown in Table <ref>. 






















































 §.§ Baselines.

To validate the effectiveness of  GMN, 
we  first select single interaction  based methods (i.e., YoutubeDNN<cit.>, MIND<cit.>, and SURGE<cit.>) as well as  multiple interactions based methods (i.e., GIFT<cit.>, MGFN<cit.>, and CCDR<cit.>). 

Since MIND and SURGE cannot be directly applied to the dual graph, we also provide corresponding dual versions (i.e., MIND+ and SURGE+) for better performance. The baselines are shown below:










 



 
  * YoutubeDNN<cit.>: 

It is a classical model for large-scale   video retrieval, utilizing two-tower architecture to learn the embeddings of users and videos based on their features.




 
  * MIND/MIND+<cit.>: 

 MIND is a classical retrieval   model, which   utilizes a dynamic routing mechanism to extract   users' diverse interests from historical behaviors.


 Since MIND cannot directly apply to the dual graph, we extend MIND as MIND+ to learn diverse user interests from both user-video and user-item interactions,  and  then concatenate them as the final user interests for   video retrieval.



 
  * SURGE/SURGE+<cit.>: 
 SURGE is a basic graph-based model that takes graph convolution and graph pooling to extract and fuse users' interest from user  interaction for recommendation. 

 We also extend SURGE as SURGE+ to handle the dual graph. 
 The experimental setting is the same as MIND/MIND+.



 
  * GIFT<cit.>: It is a graph-based   model that utilizes diverse types of relations r  to guide the feature propagation between cold- and warm-domain for video  recommendation. 

Here we take   user-video interaction    as cold-domain  and   user-item interaction as warm-domain for experiment.




  * MGFN<cit.>: It is a multi-graph based video recommendation model which 
 encapsulates interaction patterns across scenarios via  multi-graph fusion network. 


 
  * CCDR<cit.>: It is a contrastive cross-domain model for retrieval, which  simultaneously utilizes both intra- and inter-domain contrastive learning. Here we take user-video and user-item as two domains.



 
  * GMN: It is our complete model, which performs both node- and preference-level graph matching on the dual graph for video retrieval.



   






















 §.§ Implementation.

For all models, we random initialize parameters with  Xavier initializer and select Adam as the optimizer.

For a fair comparison, we use

the following hyper-parameters for all models:


the dimension of  node embedding is 128, 

the learning rate is 0.0015,

the L2 regularizer is 0.01, and the dropout rate is 0.75. 
For MIND and SURGE, we use the multiple preferences version and set the maximum  number of preferences to 5.
For YoutubeDNN, we use 3 layer DNN (1024-512-256)  for prediction.

For offline evaluation,  we  select  AUC, Precision, Recall, and Loss as evaluation metrics.
We implement all models with TensorFlow 1.15 and run them on  Nvidia  A100 Cluster.
The  code of GMN  and the anoymous version of the  industrial dataset will be released after acceptance.

























































 §.§ Performance Comparison
 
 
The comparison results are shown in Table <ref>, where we have the following observations:


•  The proposed GMN consistently performs better than all baselines  with significant gaps.



Compared to the best performance of baselines, the improvements  of GMN on AUC metric is up to 1.2%-1.9% on  large-scale video-driven e-commerce, which indicates the effectiveness of both node- and preference-level  modeling in the proposed GMN for   video  retrieval. 



•  Multiple interactions   based methods (i.e., MIND+ and CCDR) outperform single graph methods (i.e., MIND and SURGE). It makes sense because both  user-video and user-item interactions are able to provide valuable information for understanding user preferences.






By capturing the correlation  between two types of interaction,

GIFT, MGFN, CCDR, and GMN 
always show their superiority  over 
concatenate based models (i.e., MIND+ and SURGE+).





• GNN-based models, including SURGE, GIFT, MGFN, CCDR, and GMN, perform better than traditional DNN-based models (i.e., MIND and YoutubeDNN), indicating the effectiveness of graph structure modeling in the recommendation.









































 §.§ Ablation Study


In this section, we conduct both model-level and graph-level ablation studies for further verification, shown in Table <ref>.


•  Model-level.













 
We first conduct a model-level ablation study to show how  the  delicate designs in GMN affect   performance. 

As shown in Table <ref>, we test two variants of GMN (GMN_\  NL and GMN_\ PL), which remove node- and preference-level graph matching, respectively. 





Obviously,  GMN performs better than  both GMN_\  PL and GMN_\ NL), which clearly verifies  the   superiority of node- and graph-level  matching.


Note that the degradation of GMN_\ NL is   more significant than GMN_\ PL,   implying that the node-level graph matching contributes more significantly.


The reason is that node-level graph matching is able to capture the potential relevance between videos and items and integrate them seamlessly, serving as the basis of preference extraction and preference-level graph matching.




•  Graph-level.







We also conduct graph-level ablation study    to verify the effectiveness of   dual graphs modeling, where 



GMN_\   U-V and GMN_\ U-I    remove the user-video graph and user-item graph, respectively.

Note that since graph-level ablation studies
are conducted only on
a single graph (user-video or user-item), so they naturally  build upon model-level ablation studies.

In Figure  <ref>, we find that  removing either the user-video graph or the  user-item graph fails to comprehensively capture user preference, leading to suboptimal performance.    



It suggests the necessity of dual graph modeling in video-driven e-commerce.

































































































 §.§ Case Study

We further present a case study to show the potential interpretability of GMN. 
Taking a young boy as an example who is interested in the Apple iPhone and the Casio watch, he has interacted with related items and videos.

Figure <ref> shows the heatmap on both node- and preference-level relevance  matrix, where darker color means higher relevance.
Since item i_9 and video v_8  are both related to iPhone,  the node-level relevance score  r^u_v_8, j_9 is high. A similar phenomenon can be observed at the  preference-level (e.g., p^u_1,1).






 §.§ Parameter Study









 In this section, we investigate the sensitivity of parameters and report the performance of GMN on the offline 7-days dataset with various parameters in Figure <ref>.
 
•  Dimension of Metric Matrix. As shown in Figure <ref>(a),  we  test the effect of the dimension  of the metric matrix 𝐌. Obviously, with the growth of the  dimension, the performance of GMN continues to rise. The reason is that, a larger dimension enables the proposed GMN to properly measure the similarity between videos and items. 



•   Number of User Preferences.   Naturally, most   users have diverse  preferences, and one preference fails to comprehensively describe the characteristics of users. 
Figure <ref>(b) shows that  
  more  user preferences  (i.e., larger k_1) extracted from  preference-level modeling   improve the overall performance of GMN. 
And, GMN achieves the best performance when  the number of user preferences  is set to 4.
 
 




























§ ONLINE EVALUATIONS

 









 §.§ Online A/B Testing











We conduct A/B testing during 0907-0925  (19 days) for online evaluations, shown in Figure <ref>. 
Although MIND+ is not the strongest benchmark in offline evaluation, it is the previously deployed model,  affecting hundreds of millions of users every day. Therefore, we use MIND+ as  the   baseline in online A/B testing.


















 


Here, we select the widely used CTR[ CTR= Number  of  clicked  videos / Number  of  exposed  videos × 100(%)]  to evaluate   online performance.


The larger CTR means users are willing to click the exposed videos recommended  by our system and indicates better performance. 




As shown in Figure <ref>, the proposed GMN  consistently shows significant superiority over the previously  deployed MIND+ on all days.
 

More specifically,  the average  CTR of the proposed GMN is  3.74% while MIND+ achieves only 3.49%, resulting in a CTR improvement of up to 7.15%.


Following <cit.>, we also use  TP99[TP99 is a minimum time that 99% of queries have been served.] response time to 
evaluate the efficiency of our system, shown in Figure <ref>.











 









Full-day observation shows that the response time of the proposed  GMN system is only about 1 ms higher than the previously developed MIND+,
meeting response time requirements and ensuring user experience.




Currently, we have deployed GMN on a well-known video-driven e-commerce platform, affecting hundreds of millions of users every day.

































































 §.§ System Architecture and Online Serving.
 

Here we present how to deploy the proposed GMN system for real-world video-driven e-commerce, shown in Figure <ref>.
For offline training,  we collect real-world user clicks as positive samples and random sample several videos as negative samples.   Then, we sample user-centric sub-graphs and associate them with training samples.  as training data. 
The trained model will be divided into two parts and developed in different platforms: (1) The user part, which is developed to the model center, will generate user embedding based on real-time user behavior. (2) The video part, which is developed to the retrieval center, will retrieve candidate videos based on user embedding.








§ CONCLUSION


The thriving of e-commerce is accompanied by a newly emerging paradigm,  video-driven e-commerce,  which utilizes attractive videos to improve user experience and stimulate consumption.

In this paper, we propose the dual graph to represent video-driven e-commerce and innovatively reduce the user preference understanding problem to a graph matching problem. 

To solve it, we further propose a novel  Graph Matching Network (GMN),   which mainly consists of node- and preference-level graph matching.

By properly matching the heterogeneous interactions and preferences, the proposed GMN achieves significant improvements over the state-of-the-art models in both offline and online evaluation (e.g., AUC+1.9% and  CTR+7.15%).

Currently, the proposed GMN  has been developed in a well-known video-driven e-commerce platform, affecting hundreds of millions of users every day.


































ACM-Reference-Format





 Paper Title: From Feature Importance to Natural Language Explanations Using LLMs with RAG.






2024
Copyright for this paper by its authors.
  Use permitted under Creative Commons License Attribution 4.0
  International (CC BY 4.0).



MAI-XAI'24: Multimodal, Affective and Interactive eXplainable AI,
  October 19–20, 2024, Santiago de Compostela, Spain










1]Sule Tekkesinoglu[
  orcid=0000-0002-1232-346X,
  email=sule@robots.ox.ac.uk,
  ]
[1]
[1]Oxford Robotics Institute, University of Oxford, Oxford, UK

1,2]Lars Kunze[
  orcid=0000-0001-5302-1938,
  email=lars@robots.ox.ac.uk,
]
[2]Bristol Robotics Laboratory, University of the West of England, Bristol, UK




[1]Corresponding author.





§ ABSTRACT 
  As machine learning becomes increasingly integral to autonomous decision-making processes involving human interaction, the necessity of comprehending the model's outputs through conversational means increases. Most recently, foundation models are being explored for their potential as post hoc explainers, providing a pathway to elucidate the decision-making mechanisms of predictive models. In this work, we introduce traceable question-answering, leveraging an external knowledge repository to inform the responses of Large Language Models (LLMs) to user queries within a scene understanding task. This knowledge repository comprises contextual details regarding the model's output, containing high-level features, feature importance, and alternative probabilities. We employ subtractive counterfactual reasoning to compute feature importance, a method that entails analysing output variations resulting from decomposing semantic features. Furthermore, to maintain a seamless conversational flow, we integrate four key characteristics – social, causal, selective, and contrastive – drawn from social science research on human explanations into a single-shot prompt, guiding the response generation process. Our evaluation demonstrates that explanations generated by the LLMs encompassed these elements, indicating its potential to bridge the gap between complex model outputs and natural language expressions.






  Explainable AI LLMs Subtractive counterfactual reasoning Retrieval-Augmented Generation (RAG) 









From Feature Importance to Natural Language Explanations Using LLMs with RAG
    [
    Received XXX; accepted YYY
============================================================================





§ INTRODUCTION


As we become increasingly reliant on AI applications in our daily lives, it becomes imperative to interact with autonomous decision-making systems in human-understandable terms <cit.>. Regulatory frameworks and standards are also evolving to require transparency and accountability in AI-driven systems <cit.>. The technical necessity for explanations to facilitate model debugging and rectifying potential ethical and legal risks arising from biases and errors has been extensively discussed in the literature <cit.>.

The increasing demand for transparent and interpretable AI models has prompted the exploration of various explanation methods, resulting in significant progress in both inherently interpretable and post-hoc explainability methods <cit.>. While effective, these methods struggle to provide explicit interpretability or intuitive explanations for non-technical users. Consequently, there has been a concerted effort to integrate interpretability approaches with Natural Language Processing (NLP) <cit.>. Recently, Large Language Models (LLMs) have gained acclaim as post-hoc explainers, showcasing their potential to elucidate decisions made by other predictive models. Despite their success, a persistent challenge for these models is the occurrence of object hallucinations—instances where plausible yet incorrect outputs are generated, such as featuring objects that do not exist in the images <cit.>. This raises questions about the fidelity of explanations to the underlying model. To address this issue, we propose a traceable question-answering, which informs LLM responses through an external knowledge source that provides insights into the model's output. By utilizing this data, the LLM acts as a reasoning engine to process the information rather than a source of information. Figure 1 provides a high-level depiction of this process.  

Additionally, to cultivate a seamless conversational experience, we incorporated key elements derived from social science research on human explanations into the system prompt, directing the response generation process. We present our vision for creating this integrated approach, outlining each component, and illustrating its viability through a real-world dataset. This integration has the potential to bridge the gap between intricate model outputs and user comprehension towards advancing the development of human-understandable explanations. 







§ BACKGROUND



 §.§ Recent advances in image explanations

The demand for transparent and interpretable AI models has led to exploring diverse explanation methods for vision tasks <cit.>. Recent years have witnessed significant progress in both intrinsic and post-hoc explainability methods. Gradient-based techniques, which analyse the gradients of the model's output with respect to its input features, have been extensively studied <cit.>. While effective, these methods may fall short in providing intuitive explanations for non-experts. In another line of work, attention mechanisms are proposed by assigning weights to different parts of the input data based on their relevance to the model's decision <cit.>. Similarly, attention mechanisms may lack explicit interpretability, and some argue that attention alone is insufficient. As a result, there have been concerted efforts to integrate these approaches with Natural Language Processing (NLP) <cit.>. Long Short-Term Memory (LSTM) encoders have been employed to extract features from video clips, with LSTM decoders generating descriptive texts based on these features. While effective in image captioning, these methods provide textual descriptions rather than visual explanations. Moreover, researchers have proposed perturbation-based approaches—a model-agnostic method that perturbs input data and observes changes in model predictions. Building on this research, we integrate semantic perturbation analysis with pretrained language models, translating the results into the text modality for enhanced explainability.  



 §.§ LLMs as post hoc explainers

Recent strides in Large Language Models (LLMs), exemplified by GPT-4 (OpenAI), Bard (Google), Claude-2 (Anthropic), and Llama-2 (Meta), mark a transformative era in Natural Language Processing (NLP) research. These models have become pervasive across diverse applications, ranging from machine translation and question-answering to text generation <cit.>. Notably, LLMs have recently gained attention as post hoc explainers, highlighting their potential to elucidate the decisions made by other predictive models <cit.>. Leveraging their in-context learning capabilities by fine-tuning for specific tasks enhances their proficiency in generating contextually relevant explanations <cit.>. Despite their success, a persistent challenge for these models is the occurrence of object hallucinations—instances where plausible yet incorrect outputs are generated, featuring objects that do not exist in the images in vision-based tasks. This issue raises questions about the faithfulness of explanations to the underlying model. One way to address this is by recognizing the role of the semantic map layer in capturing the meaning and context of the physical surroundings <cit.>. By effectively encoding valuable semantic information into LLMs, we can truthfully represent and comprehend the intricate details of the model output within the language space.



 §.§ Characteristics of explanations and XAI


In the context of XAI, the audience for explanations is predominantly human; thus, understanding what makes an explanation human-friendly is paramount. While accuracy is important, the clarity and presentation of explanations are equally crucial in ensuring comprehensibility. According to Miller <cit.>, current work on interpretability in machine learning relies solely on researchers' intuition regarding what constitutes an appropriate explanation for humans. Miller's survey highlights four major characteristics of explanations drawn from findings in social science research on human explanation: explanations are social, causal, selective, and contrastive <cit.>. In the following section, we briefly discuss these characteristics in relation to XAI before demonstrating how we integrate them into our application.



  §.§.§ Explanations are social

An explanation represents an interactive exchange between two roles: the explainer and the explainee, and is governed by certain `rules' <cit.>. Interactive explanations in a dialogue structure consist of three fundamental components: opening statements, clarification questions, and closing statements <cit.>. As such, it should follow the basic rules of conversation, which are captured by Grice's conversational maxims <cit.>. These include principles such as quality, quantity, relation, and manner, which can be understood as `only say what you believe', `only say as much as is necessary', `only say what is relevant', and `say it nicely'. Manner further encompasses various sub-maxims, including avoiding unclear expression, eliminating ambiguity, maintaining brevity to avoid unnecessary wordiness, and presenting information in an orderly fashion. 

Moreover, explanatory agents employing anthropomorphic traits like politeness markers (e.g., “Thank you!”), warm and friendly tones, and empathetic expressions (e.g., “I understand.”) are pivotal for effective human interaction <cit.>. Theory of Mind also plays a crucial role in social explanations.[Theory of mind refers to our ability to understand other people by attributing mental states to them, including beliefs, desires, intentions, emotions, and thoughts that may differ from our own <cit.>.] As an integral part of a dialogue, an explanatory agent should keep track of what has already been explained, maintaining a simple mental model of others. This enables intelligent agents to tailor explanations based on past interactions and evolving contexts.



  §.§.§ Explanations are causal

Causal explanations aim to elucidate the cause-and-effect relationship underlying a decision. When explaining a prediction made by a machine learning model, identifying the most influential features can provide insight into the key causal connections to the prediction. This process, also known as causal inference,  starts with observations (e.g., what if a feature had a different value) and selecting some of those causes as the explanation. Miller divides casual inference into two parts: counterfactuals and abductive reasoning <cit.>.

The reasoner uses a model of hypothetical counterfactual cases to derive an explanation, such as perturbing inputs, to see how they causally impact the prediction (See Section <ref>). Abductive reasoning is inferring the causes from those observations to form the most probable explanation. One can think of abductive reasoning as the following process: observing unexpected or surprising events, generating one or more hypotheses about these events, judging the plausibility of the hypotheses, and selecting the `best' hypothesis as the explanation <cit.>. It is important to note that new information (e.g., different feature perturbations) can alter the most likely explanation.




  §.§.§ Explanations are selected


Explanations are contextual by nature. While an event may have numerous causes, the recipient of the explanation typically cares about only a specific subset relevant to the context <cit.>. The explainer selects a subset based on various criteria called casual selection. Hilton <cit.> argues that explanation selection is used for cognitive reasons since causal chains are often too large to comprehend. The criteria people use for explanation selection include abnormality, relevancy, and simplicity. 

Abnormality is a key criterion for explanation selection; as such, people select unusual causes to explain events <cit.>. In machine learning interpretability, identifying abnormal events corresponds to deviant or higher feature importance values. As pointed out within the conversational maxims, relevancy also plays a role in explanation selection—only say what is necessary and relevant. People select explanations to adhere to these maxims whilst persuading the explainee to new information or viewpoints. Another strong criterion in explanation selection is simplicity, which refers to those that cite fewer causes. People prefer uniquely sufficient causes that bring about the effect without any other causes, but there might be cases with multiple sufficient causes.  




  §.§.§ Explanations are contrastive


Studies have shown that people only request contrastive explanations, essentially `why' questions framed to imply the differences between two possible outcomes <cit.>. As a result, explanations are typically offered in relation to the cause of one event compared to another that did not occur.

Similarly, in the context of ML interpretability, people are not specifically interested in all the factors that led to the prediction but instead in the factors that need to change so that the prediction would also change. Explanations that present some contrast between the instance to explain and a reference point (i.e., a hypothetical instance) are preferable because the cognitive burden of complete explanations is too high. However, most existing work considers contrastive questions but not contrastive explanations, providing two complete explanations individually. This could be because the contrastiveness is application-dependent, given that a reference point may not be apparent in all tasks <cit.>. For instance, in sentiment analysis or anomaly detection, contrastive cases might not be as straightforward as other tasks, such as banking, e.g., loan accepted vs rejected cases.



§ MATERIALS AND METHOD


This section outlines the key elements of the traceable question-answering methodology. We detail the proposed explainability approach for extracting feature importance and the prompting technique to generate accurate and human-friendly explanations. Additionally, we provide an application example illustrating how we establish a connection between the model output, the explainability technique, and language models to produce natural language explanations in an interactive format. 











 §.§ Explanations through subtractive counterfactual reasoning



Subtractive counterfactual reasoning is the process of removing an event to understand its causal influence on an outcome <cit.>. Such reasoning is widely used as an XAI technique, exemplified by occlusion analysis and feature importance attribution, built on the same foundation - quantifying the variation in model output through systematic modifications of input components <cit.>. There are distinctions in terms of how these methods remove features, elucidate model behaviour, and summarize feature importance. Nonetheless, they provide valuable insights into the relative significance of each feature influencing the decision-making process <cit.>. 

This study proposes a decomposition-based approach to examine the output variations by decomposing input values. By measuring such variations, we determine the degree of impact on an outcome and reason about the importance of each feature value. As a function, the model assigns a class label and prediction probability f:x→ f(x) given input y. To measure the effect of each input value, we observe the model's prediction p for y without the knowledge of event T_i=T_k, where T_k is the value of feature T_i. To explicitly represent the absence of information, the feature value T_k is replaced with an undefined value, i.e., NaN (Not a Number). NaN values are treated as invalid or masked elements and are not considered in the computation. We attain an array containing probabilities for all perturbations resulting from decomposition. In order to determine the importance of an individual feature T_i, we begin by identifying the max and min values within this array, which establishes the decision boundaries for the given input. Next, we compute the feature importance as the position of the probability– after decomposing the feature– relative to the max and min probabilities. The computation is expressed as follows:


    importance(T_i) = p(x|y ∖ T_k)_max - p(x|y ∖ T_k)/p(x|y ∖ T_k)_max - p(x|y ∖ T_k)_min


This approximates an importance value between zero and one for each input feature. Considering that evaluating fractions has a higher cognitive load, the values are rounded to the nearest integer to increase comprehensibility. Moreover, this analysis is carried out for the categories following the main prediction simultaneously to generate explanations for the contrastive cases. 





 §.§ Prompting for traceable question-answering with RAG


Traceable question-answering, through a process known as Retrieval-Augmented Generation (RAG), integrates external knowledge sources to furnish responses to user inquiries. This method improves the performance of LLMs by referencing an external knowledge source prior to generating a response (See Figure <ref>). This approach mitigates the problem of `hallucination' and ensures factual consistency, particularly in scenarios where facts may evolve over time <cit.>. Depending on the specific implementation, RAG supports various file formats, including text, PDF files, word documents, structured data (tables), and images. 

In our work, the knowledge repository contains tabular information about model output, features, contrastive cases, and feature importance values obtained through the decomposition-based explanation method described in Section <ref>. This information is saved as the output of the explanation module as a .csv file. Then, the file, the system prompt (See Figure <ref>), and the user input query are fed to the LLM model for response generation. The resulting responses contextualise the feature importance grounded in the models' inner workings, thereby rendering the explanations traceable to the underlying model.   

Furthermore, our aim is to imbue LLM with the characteristics of explanations discussed in Section <ref> to foster a human-friendly interaction. Ideally, LLM-generated responses should adhere to conversational norms, encompassing opening, clarification, and closing statements while appropriately integrating social cues. The model is expected to deduce causal relationships from the observations within the provided knowledge repository, selecting the most pertinent causes to form an explanation and presenting it alongside a contrastive case. To accomplish this goal, we devised a system prompt that provides explicit instructions to the model to generate efficient responses to user queries. The system prompt includes a single-shot example illustrating the desired interaction, potential user inquiries and outlining how the model can effectively address them, enabling the language model to produce consistent and predictable outcomes, as presented in Figure <ref>.


















 §.§ Application example
 

To demonstrate our approach to traceable question-answering, we selected a visual scene understanding task. This involves scene classification, categorizing images based on their content and inferring high-level semantic scenes from low-level visual features. Many critical applications of scene classification include human-robot collaboration, autonomous driving, and other autonomous physical systems that rely on semantic segmentation to inform decision-making processes <cit.>. We experimented with the GoogleNet Places365 model designed specifically for scene recognition and classification tasks <cit.>. The model is trained on the Places365 dataset, containing more than 365 scene categories.  We focus on semantic urban scene understanding with the road segmentation dataset CamVid (Cambridge-driving Labelled Video Database). The dataset helps to understand driving scenes, comprising images capturing street-level views from urban driving scenarios. It includes manual annotations across 32 classes, encompassing elements such as building, driveway, pavement, tree, traffic sign, car, pedestrian, and bicyclist. The Deeplab v3+ deep learning model is trained on the CamVid dataset for semantic segmentation <cit.>. Figure <ref> shows the segmentation result for the example presented in the next section. Since we utilized off-the-shelf models for illustration purposes, the emphasis is not on these models' performances.



Semantic information plays a critical role in autonomous vehicle (AV) decision-making processes, as it captures the meaning and context of the physical surroundings observed in the road scene. This includes understanding the appearance of various elements such as buildings and roads, identifying shapes such as cars and pedestrians, and recognizing spatial relationships to contextualize the scene <cit.>. Particularly in unstructured off-road environments, the accurate understanding of the surroundings concerning semantic classes such as trail, grass, or rock is important for safe and deliberate navigation <cit.>. Moreover, semantic segmentation provides a way to represent and comprehend the details of the environment in the language space. By leveraging the labelled information within the image scene, we can semantically map the crucial components based on their feature importance. This granularity enables a more insightful analysis of feature importance, contributing to a nuanced understanding of the model's decision-making process. Such insights can be helpful for post-incident forensic analysis in case of misreading a scene causing collisions or accidents. In this work, we suppose situations where the user checks in with an AI assistant in an AV system in non-critical driving scenarios while engaging in non-driving related tasks.































Table <ref> reveals the content of the knowledge source - only the first contrastive case is displayed. The effect of removal is the probability when a feature is removed, as listed respectively. Given the system prompt and the data presented in Table <ref>, Figure <ref> showcases the dialogue generated for this case. From a social standpoint, the resulting responses balance between being informative and supportive while maintaining a friendly and engaging tone. It effectively justifies the model's output by emphasising its most significant feature in a concise manner. Additionally, the dialogue delves into contrastive explanations by detailing three consecutive classes of lower likelihood, demonstrating the flexibility to explore different scenarios. When comparing the top prediction to the second most likely class, it highlights the distinguishing features contributing to the output. 













§ EVALUATION RESULTS

In this section, we present our experimental findings. We analyzed nine distinct scene classes — parking lot, street, residential neighbourhood, crosswalk, highway, industrial area, gas station, shopfront, and general store (outdoor)— across different scenarios (totalling 42), where the model displayed varying degrees of confidence in its output. We evaluated the responses in terms of sociability, causality, selectiveness, and contrastiveness generated by GPT-3.5 () and GPT-4 () models. 



 §.§ Sociability


In examining the sociability within the responses, our initial focus centres on the overall tone of the responses. We employed the Valence Aware Dictionary and Sentiment Reasoner (VADER) to gauge the sentiment conveyed within the text, encompassing dimensions of positivity, negativity, and neutrality. Sentiment scores for each response are aggregated to delineate overall sentiment trends. Figure <ref> illustrates a noticeable positive trendline by both models, indicating a prevalent inclination towards positivity in the sentiment expressed. A qualitative inspection of the text reveals a prevalence of polite, helpful, and informative social cues across all responses, supporting this observation. Consequently, the neutral sentiment score remains notably low, underscoring the prevalence of emotional expression throughout the corpus. Exploring negative sentiment scores unveils instances where the models acknowledge their limitations, particularly when unable to retrieve answers from the external knowledge repository. 

 
Label Positive Neutral Negative

GPT3.5   202  5  8
GPT4  195  6  14











Considering the three foundational phases of dialogue—opening, clarification, and closing statements—next, we analyzed the indicators such as politeness markers and emotional expressions within each phase. To facilitate this analysis, we created a distinct dictionary for each phase and computed the term frequency of social cues present in each phase.[Term frequency is defined as the number of times a term occurs in the document over the total number of terms in a document.] Opening statements include greetings such as `hey' and `hello.' Clarification statements are characterized by social cues indicative of helpfulness, continued support, and awareness of its limitations, including terms such as `absolutely', `sorry', `further', `curious', `questions', `help', `feel', and `free'. Closing statements commonly conveyed sentiments of well-wishing or safety, with phrases including terms such as `welcome', `enjoy', `safe', `great', `pleasant', and `glad'.




Figure <ref> shows that both models utilize these terms fairly across the respective phases of the dialogue. Our qualitative assessment of the responses further confirms the contextual relevance of these terms, which align with conversational maxims (See Section <ref>). The higher frequency value observed in closing statements can be attributed to their typically brief nature, which encapsulates most of the social cue terms. Additionally, both models occasionally repeat certain phrases in their responses, albeit with the application of a frequency and presence penalty (0.3).[Frequency penalty discourages repetitive use of words or phrases, while presence penalty promotes diversity in the generated text.] In addition to analysing social cues, we examined part-of-speech (POS) pronouns as social interaction markers, encompassing terms such as `I', `you', `we', `my', `our', and `your.' Quantitative analysis and qualitative evaluation of the results indicate that both models adequately and appropriately applied POS pronouns.


 


 §.§ Expression of Causality


To evaluate the expression of causality, we initially extracted responses indicating cause and effect, primarily found within clarification statements. We then identified key terms related to causality, such as `because', `if', `then', `albeit', `due', `contribute', `influence', `affect', `impact', and `effect', and tallied their occurrences within each answer to measure the presence of causal reasoning. In Figure <ref>, we compare the frequency of these terms used by two models in the response set. Notably, `if' emerges as the most frequent term used by both models, often utilized for expressing counterfactual reasoning (e.g., `if it was not', `if there is/were'), while `influence' typically indicates the cause of a decision. Overall, GPT-4 demonstrated a higher frequency of causality terms and showcased greater diversity in expression compared to GPT-3.5.




Through the qualitative review, we examined existing patterns of dependency parsing signifying causal relationships. Two primary grammatical structures emerged, suggesting cause-effect relationships within the text. The first structure pertains to how the presence or absence of certain features influences the likelihood of specific scenarios. The second structure involves a cause-effect relationship between feature importance and probability, indicating that features with higher importance scores exert a greater impact on probability. It is important to acknowledge that while these grammatical structures and terms are widely used, their accuracy is not always consistent across models. In the subsequent section, we will further evaluate the selection and accuracy of inferred causal relationships.



 §.§ Selectiveness


In this section, we evaluate whether the models effectively detect unusual values within the provided knowledge repository, infer potential causes from those observations, and select the most relevant ones to construct an explanation. Our evaluation encompasses analysing responses across the number of causes given, graded selection, and sort order to assess whether expected causes are addressed correctly. First, we specified the maximum number of uniquely sufficient causes for each case, which are feature importance values >5. The number of causes varies across scenarios; some exhibit only a single highly important feature >5, while others might involve multiple high-importance features with varying degrees of significance. Graded selection is crucial for discerning important values such that the language model must prioritize the most relevant causes when encountered with multiple features of comparable importance levels. Additionally, graded selection entails disregarding features with lower importance scores (<5). Finally, the sort order concerns the sequence of selected causes, which should follow a hierarchy from the most significant and relevant features to those of lesser importance. Thereafter, responses are categorized as fulfilled, partial, subpar, or unfulfilled according to their arrangement with our specifications. 



The findings presented in Table <ref> indicate that GPT-4 outperformed GPT-3.5 in all categories. Notably, GPT-4 had a much higher success rate (76%) than GPT-3.5 (24%) in identifying the most relevant cause (fulfilled). Additionally, GPT-4 had a lower rate of subpar and unfulfilled responses. GPT-3.5 referenced features not present in the knowledge source as potential causes in two instances. These results suggest that GPT-4 is more accurate and selective than GPT-3.5 in identifying the most relevant cause(s) from a set of possible causes. 

Furthermore, an additional criterion for assessing explanation selection is simplicity, which involves utilizing concise language, avoiding technical terminology, and referencing fewer causes. Table <ref> evaluates responses by comparing three metrics: the frequency of technical jargon, response length, and the number of causes cited. Lower values across all three metrics signify simpler responses.



To measure the term frequency of technical jargon, we compiled a list of technical terms in the responses and calculated their frequency within the generated explanations. The list contains terms such as `prediction', `feature importance', `score', `contrastive cases', and `confidence', which may not be easily comprehensible to all users. Table 3 indicates that GPT-3.5 exhibits a slightly higher term frequency (0.0162) compared to GPT-4 (0.0157), suggesting that GPT-3.5 utilizes technical jargon marginally more frequently. The length metric refers to the median lengths of the responses generated by each model, which are quite similar (45 tokens for GPT-3.5 and 44 tokens for GPT-4). In terms of the number of causes cited, GPT-4 cites fewer causes (median of 2) than GPT-3.5 (median of 3), suggesting that GPT-4’s responses tend to be simpler in terms of causal reasoning. Overall, the results suggest that GPT-4 generates simpler responses than GPT-3.5 across all three assessed metrics. 

Regarding the out-of-scope questions posed, such as the number of cars or pedestrians, the intent was to assess whether the model would fabricate an answer. Both models have expressed their inability to provide such information. This demonstrated an awareness of their limitations rather than attempting to generate a response.  




 §.§ Expression of Contrastiveness


Ideally, initial explanations should include a contrast with an alternative case to justify the output. Then the user would investigate it further if desired. In our experiment, we observed that in most cases, models, particularly with GPT3.5, failed to mention alternative scenarios unless explicitly prompted (e.g., inquiries such as “What could it be otherwise if it wasn’t X?” or “How does X differ from other options?”). Nevertheless, to measure the presence of contrastiveness across models, we analysed the occurrence of contrastive terms and comparative phrases within the generated responses. We compiled a dictionary consisting of terms such as `distinguish', `different', `contrast', `compared to', `in contrast', `while', `both', `on the other hand', `whereas', and `conversely', which introduce contrasting explanations. Figure <ref> shows that `distinguish' is the most frequently used term by GPT-3.5, whereas `different' predominates in GPT-4's responses. Other frequently encountered terms include `compared to', `contrast', and `differentiate.' Overall, the results suggest that GPT-4 employs a wider array of contrastive terms than GPT-3.5 in its responses. Our qualitative observations further confirm that while GPT-3.5 tends to adhere to one type of contrastive explanation, GPT-4 explores diverse ways to express such contrasts.




Although both models frequently employ these terms, indicating the generation of alternative viewpoints in their responses, the effectiveness of contrastive explanations varies considerably. Our qualitative review found that well-developed contrasting explanations included alternative scenarios with counterfactuals to strengthen the case. Conversely, weakly constructed contrastive explanations failed to explore `what if' scenarios to elucidate the disparity between the two situations under comparison.





§ CONCLUSION

This study contributes to advancing post-hoc explainability by proposing a traceable question-answering approach. By integrating LLMs with the decomposition-based explainability technique, we generate natural language explanations in an interactive format. Our approach promotes transparency and interpretability in scene-understanding tasks by fusing LLMs with semantic feature importance. The proposed approach has the potential for application across various domains, offering comprehensible explanations for decision-making processes, including critical areas such as medical applications. While this is a promising development, there are still areas for further exploration and improvement. One area that requires attention is the overuse of social cues by LLMs. While polite and helpful phrases can enhance user experience, their misuse can become vexing over time. It is important to evaluate such interactions through human subject studies to understand user perceptions of the explanations provided by the LLM. Future research could also explore dialogue management strategies to address issues related to interaction and personalization. On another note, the dynamic nature of the external knowledge source offers the potential for further expansion with additional observations. This not only ensures that the system remains relevant and up to date but also allows the system to answer different types of questions. Additionally, incorporating multimodal elements, such as combining text with visual explanations, can enhance the effectiveness of the responses (e.g., highlighting the specific elements contributing to the prediction). Another consideration is the estimation of feature importance values, which are subject to perturbations and computational processes. It is important to assess and reconcile differences in feature importance across algorithms to ensure stability and consistency. 
 

 







This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation. 


















 
  




