Text-Guided Mixup Towards Long-Tailed Image Categorization
Richard Franklin Jiawei Yao Deyang Zhong Qi Qian Juhua Hu


SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local Editing
    Zhihu Hu
    September 9, 2024
====================================================================================



§ ABSTRACT 
In many real-world applications, the frequency distribution of class labels for training data can exhibit a long-tailed distribution, which challenges traditional approaches of training deep neural networks that require heavy amounts of balanced data. Gathering and labeling data to balance out the class label distribution can be both costly and time-consuming. Many existing solutions that enable ensemble learning, re-balancing strategies, or fine-tuning applied to deep neural networks are limited by the inert problem of few class samples across a subset of classes. Recently, vision-language models like CLIP have been observed as effective solutions to zero-shot or few-shot learning by grasping a similarity between vision and language features for image and text pairs. Considering that large pre-trained vision-language models may contain valuable side textual information for minor classes, we propose to leverage text supervision to tackle the challenge of long-tailed learning. Concretely, we propose a novel text-guided mixup technique that takes advantage of the semantic relations between classes recognized by the pre-trained text encoder to help alleviate the long-tailed problem. Our empirical study on benchmark long-tailed tasks demonstrates the effectiveness of our proposal with a theoretical guarantee. Our code is available at https://github.com/rsamf/text-guided-mixuphttps://github.com/rsamf/text-guided-mixup.





§ INTRODUCTION



In recent years, deep learning has made state-of-the-art advancements in computer vision tasks such as image categorization, object detection, and semantic segmentation <cit.>. Deep learning models are highly dependent on large-scale and balanced training data, but real-world data are typically class-imbalanced <cit.>. When training data is abundant for a subset of classes (i.e., head classes) but scarce for the other (i.e., tail classes), the distribution of the data is said to be long-tailed <cit.>. Taking image categorization as an example, deep neural networks (DNNs) aim to minimize the empirical risk on the training data by incrementally adjusting the learnable parameters. However, given a long-tailed training data, this happens more on the head-class instances that appear more frequently, augmenting the model's performance bias towards head classes but reducing the model's generalization performance on tail classes <cit.>.


Long-tailed learning proves to be a significantly challenging task as addressed by many previous studies <cit.>. Intuitively, under-sampling the head classes and over-sampling the tail classes is a reasonable technique. Although class-level re-sampling or re-weighting can help balance out the data distribution and mitigate the model's performance bias on head classes, these techniques can cause the model's overfitting on tail classes and/or degenerate the performance on head classes <cit.>. There is evidently more success in module improvement techniques <cit.>, especially those that use ensemble learning <cit.>. There are a number of additional techniques <cit.> that aim to mitigate the long-tailed problem such as class-level re-margining <cit.>, data augmentation <cit.>, and transfer learning <cit.>. However, these methods are still limited by the scarce information found among tail classes. 



r0.38
  

    
    < g r a p h i c s >

  

  The decision boundary of `tiger' stretches towards that of `leopard' and `cat' and away from `bicycle' as text-guided mixup allows semantically similar classes to be mixed more frequently.


Recently, vision-language models such as CLIP <cit.> and ALIGN <cit.> have demonstrated good performance in zero-shot classification and few-shot learning <cit.>. These models are trained on large-scale data containing image-text pairs that elicit the forming of connections between text and image embedding. By capturing the contrastive locality of image and text features, vision-language models can generalize to unseen categories well, which is a potential information source of tail classes in long-tailed learning. However, existing multi-modal works <cit.> are limited by the general domain knowledge of the CLIP's pre-trained text encoder and must continue linguistic training on the downstream task. 



In this work, we propose to leverage the frozen CLIP text encoder to obtain prompt embedding as additional supervision for long-tailed learning in vision tasks.
Considering the observation that semantic relationships between class names (e.g., `tiger' and `cat') correlate with their localities of visual features in vision-language models, we can utilize semantically similar classes to assist the generalization among tail classes (e.g., the head class `cat' can help assist the tail class of `tiger' as shown in Fig. <ref>). However, the intra-class variance of the tail class can still be ignored. Therefore, we further propose a novel text-guided mixup strategy, named local feature mixup (LFM), to shift the label towards tail classes, so as to alleviate the long-tailed problem. The main contributions of this work are summarized as follows.

    
  * We leverage the frozen CLIP text encoder to enhance the performance of long-tailed visual recognition tasks.
    
  * We construct a novel mixup technique that takes advantage of the text encoder to boost the performance of tail classes with a theoretical guarantee.
    
  * Our extensive experiments on several benchmark long-tailed data demonstrate the effectiveness of our proposal.





§ RELATED WORK

 
In long-tailed visual recognition, numerous methods have been proposed to boost the performance of tail classes <cit.>. Module improvement methods including ensembling have shown recent success <cit.>. In mixture of experts, TADE <cit.> and SHIKE <cit.> output an aggregation of multiple expert modules, where each expert in TADE strives to perform well in a different training distribution, and each expert in SHIKE focuses on modeling a different depth of image features. Although ensembling can boost performance, these methods are still limited by the scarce information found among instances of the tail classes.

Moreover, class re-balancing such as class-level re-sampling <cit.>, re-weighting <cit.> (e.g. Balanced Cross Entropy <cit.>), and re-margining (e.g., LDAM <cit.>) can adjust the model's attention to classes with a lower sample rate. However, class-balanced sampling or re-weighting can lead to overfitting of the tail classes, under-represent the intra-class variance of the head classes <cit.>, and thus decrease the model's overall performance <cit.>. Alternatively, it can be effective to train a model with meta sampling <cit.>, in which the optimal sample rate per class is estimated by applying a learnable parameter for each class label. Using this method can slightly avoid the overfitting of tail classes, but finding the optimal parameter or trade-off between class labels for multi-class classification is difficult. 



Another instance of success is found through pre-training vision transformers <cit.> in an autoencoder setup <cit.>. Once the encoder is sufficiently trained, it feeds into a classification layer that is trained using a balanced binary cross-entropy loss <cit.>. However, these methods still lack sufficient performance on the set of tail classes as it is an inert challenge to train deep neural networks for classes with small sample rates. Recently, pre-trained vision-language models like Contrastive Language-Vision Pre-training (CLIP) <cit.> have demonstrated strong zero-shot performance. CLIP embodies multi-modal learning through unsupervised training of image-caption pairs available on the wild web to capture the contrastive locality of image and text features. This makes CLIP more adaptable to new tasks, so that they can be leveraged to make zero-shot predictions, that is, generalize to unseen categories. Thereafter, a pre-trained vision-language model can be further fine-tuned on a downstream task in few-shot learning <cit.> or long-tailed learning (e.g., RAC <cit.>, VL-LTR <cit.>, LPT <cit.>, TeS <cit.>, and VPT <cit.>). However, most of them have been focusing more on the text encoder. For example, VL-LTR <cit.> requires manually retrieving text descriptions of each class from the Internet to augment the text data in preparation for linguistic training, which is resource expensive, so we instead freeze the text encoder. 


 


§ THE PROPOSED METHOD




Given a long-tailed training data D = {(x_i,y_i)}, x_i is an image associated with its target class y_i∈{1, …, C}. We construct a set of text snippets T, where each T_k describes a class label for k∈{1, …, C}. For example, the text snippet describing class name “dog” is a tokenized sequence generated from the string as “a photo of a dog”. 

We feed image and text snippets T to the image and text encoders, respectively, pre-trained by CLIP <cit.> as shown in Fig. <ref>, for which we denote as ℱ_I and ℱ_T, respectively. Both of these encoders output feature vectors of size d. We denote the output from the text encoder as f_T and f_T = ℱ_T (T) and allow f_T_k to denote the feature vector for class k, which does not change during the long-tailed learning. To better separate the tail-class feature embeddings from that of the head-classes following <cit.>, we append a fully connected layer W_I ∈ℝ^d× d that is learnable to ℱ_I. Thereafter, we can extract the feature vector for each image x_i as f_I = W_Iℱ_I(x_i). Additionally, we normalize both f_T_k and f_I to be of a unit norm. 


After obtaining f_I and f_T, image classification is performed as shown in Fig. <ref> by computing the cosine similarity between f_I and f_T_k for all k, and finally, the predicted class label, ŷ, for each image is computed as ŷ = *arg max_k ∈{1, …, C} f_I · f_T_k. 



Thereafter, we can adopt a decoupled training approach as suggested by <cit.> to learn better embeddings for tail-classes compared to joint training. In stage 1, we open ℱ_I and freeze W_I for training, and in stage 2, we freeze ℱ_I and open W_I. At the beginning, W_I is initialized as the identity matrix with f_I = ℱ_I(x). However, by minimizing the empirical risk directly based on the training data with a long-tailed distribution, both ℱ_I and W_I can still be biased to the head classes. Therefore, we propose a novel text-guided mixup technique. 





  
Local Feature Mixup
A statistical measure of class imbalance in a dataset can be defined as the imbalance factor γ = n_1 / n_C, where n_k is the number of examples in class k and n_1≥ n_2≥ ...≥ n_C is ordered from high to low, and typically, we have n_1≫ n_C. Our main goal is to increase the few-shot accuracy (i.e., those with low n_k), while not attenuating the model's accuracy on many-shot classes (i.e., those with high n_k). We strive to boost the few-shot accuracy by making two assumptions about the data. First, we assume that classes with low n_k are underrepresented because a few examples may not fully express the complete diversity (or variance) of their associated class. For example, a cat can look different from another cat in terms of their features such as their sizes, their eye colors, and the color/pattern of their furs. When limited to observing a few examples of cats, it is difficult for DNNs to grasp the full range of features that a cat can express. Therefore, we assume that every tail class has a larger intra-class variance than that can be learned from long-tailed data.

Secondly, because both CLIP's image and text encoders map their respective inputs to d-dimensional feature vectors, we say that every class can be represented by certain feature space in ℝ^d. The pre-trained text encoder already has an understanding of the local relationships between words. For example, words “frog” and “toad” are close in the language model feature space, since they have similar meanings. Part of our learning objective is to closely align the outputs of our image encoder to the outputs of the pre-trained language model. That is, if we feed an image of a frog and an image of a toad to our image encoder, their extracted feature vectors should be close in proximity as in the text feature space. Therefore, we also assume that if two classes have similar meanings (i.e., nearby in the text encoder's feature space), these two classes also share a subset of visual features and thus should also be nearby within the image encoder's feature space. In the following construction of local feature mixup, we incorporate these two critical ideas separately, that is, local sampling and label shift.



  
Local Sampling
Existing mixup strategies often randomly sample y_i and y_j uniformly across the training data <cit.>. However, we aim to choose pairs that are semantically related supervised by the pre-trained text encoder. First, we sample an instance from class y_i uniformly across the training data as p(y = y_i) = n_i/∑^C_kn_k. Then, we sample another instance from class y_j with probability p_ls(y = y_j | y_i) given by Eqn. (<ref>). 


    p_ls(y = y_j | y_i) = 
         exp( f_T_i· f_T_j / τ ) /∑^C_k=1 exp( f_T_i· f_T_k / τ )     i ≠ j
    
            0    o.w.

where the hyperparameter τ > 0 controls the temperature scaling on the softmax equation. A lower τ increases the likelihood that similar class pairs are chosen for mixup, but a too low temperature can lead to oversampling of nearby classes. We set τ = 0.05 for most experiments. Using this strategy, we hope to extend the variance of minor class samples towards neighboring classes as our assumption is that semantically similar classes share a subset of visual features as depicted in Fig. <ref>.




  
Label Shift
Then, we perform mixup by mixing images x_i and x_j sampled through our above local sampling method. With mixing factors λ_x, λ_y ∈ [0,1], we propose

    x̃^LFM   = λ_x x_i + (1-λ_x)x_j
    ỹ^LFM   = λ_y y_i + (1-λ_y)y_j

where y_i, y_j are one-hot vectors and factor λ_x is chosen randomly from the beta distribution. More importantly, we generate λ_y by

    λ_y = clamp( λ_x - αn_i - n_j/n_i + n_j, 0, 1 )

where hyperparameter α≥ 0 adjusts the intensity of label shift and the resulting value is clamped between 0 and 1. In order to expand the margin for tail classes, we shift the decision boundary away from tail classes and towards head classes according to the difference of n_i and n_j. For example, if n_i > n_j (i.e., class y_i has more samples than class y_j), we shift the target to be more in favor of class y_j, thus increasing the model's margin on the class with fewer samples. Algorithms are summarized in the supplementary and we provide a theoretical guarantee for our proposal as follows, while the overall framework is illustrated in Fig. <ref>.


Letting p=n_i/(n_i+n_j), λ_y can be obtained by balancing the distribution between x_i and x_j

    λ_y = min_λ∈[0,1] (λ - λ_x)^2/2+α R(λ)

where R(λ) = (λ-1/2)^2 - (λ - p)^2.




  
Remark The former term constrains that the obtained weight for the label should be close to the weight for the example, while the latter term is a balance regularization to incorporate the prior distribution p between two examples. By minimizing the regularization, it aims to push λ from the imbalanced initial distribution to a balanced one. When p=1/2, it degenerates to the standard weight for mixup.







§ EXPERIMENTS

To demonstrate the proposed LFM method, following the common practice in long-tailed learning, we use publicly available long-tailed datasets, that is, CIFAR10-LT and CIFAR100-LT <cit.>, ImageNet-LT <cit.>, and Places-LT <cit.>. 



  
Experiment Setup
For CIFAR10/100-LT, we fine-tune CLIP with a single GPU, and for ImageNet-LT and Places-LT, we fine-tune CLIP with three GPUs. Each GPU is an Nvidia GeForce RTX 2080 Ti with 11GB of memory. During training, each GPU receives a batch size of 32, so for ImageNet-LT and Places-LT the effective batch size is 96. Training is performed with a fixed seed to allow for reproduceability. The hyperparameters chosen for LFM are fixed (i.e., α = 1, τ = 0.05) for all experiments on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT, while they are adjusted on Places-LT as α = 1.25, τ = 1.00 in stage 1 and α = 1.50, τ = 1.00 in stage 2, due to the imbalance severity as explained in the next section. Low learning rates were picked to avoid the risk of catastrophic forgetting and losing CLIP's zero-shot performance advantage. 
The detailed hyperparameters used can be found in the supplementary. CLIP's default text prompt template is “a photo of a {CLASS}”. For all experiments, we utilize the default text prompt template provided. 

A model's performance is not necessarily stable across all classes, each with different sample counts, so it is important that we quantify the performance of our model in subdivisions relative to every n_k. Across all datasets, we subdivide the resulting model's accuracy into four categories, namely many-shot, medium-shot, few-shot, and overall following <cit.>. Many-shot classes have n_k > 100, medium-shot classes have 20 ≤ n_k ≤ 100, and few-shot classes have n_k < 20. For each performance category, we report the top-1 accuracy of our model against the balanced validation set for each subdivision of our chosen datasets. 

We compare our proposed method with vision-focused baseline methods and strategies that perform well in tackling the long-tailed problem. We also fine-tune the competitive image encoder (i.e., ViT-B/32) with different existing losses as baselines, i.e., Cross Entropy (CE), Balanced Cross Entropy (BalCE) <cit.>, Focal <cit.>, Label Distribution Aware Margin (LDAM) <cit.>, and Margin Metric Softmax (MMS) <cit.>. All losses except CE were proved to be helpful for the class imbalance problem. In summary, we compare with the following baselines based on the pre-trained CLIP <cit.>: 1) Zero-shot: The pre-trained image and text encoders by CLIP <cit.> are directly used to do prediction on the balanced test data, in which ViT-B/32 is adopted; 2) CE: Fine-tuned ViT-B/32 using the cross entropy loss; 3) BalCE: Fine-tuned ViT-B/32 using the balanced loss <cit.>; 4) Focal: Fine-tuned ViT-B/32 using the Focal loss <cit.>; 5) LDAM: Fine-tuned ViT-B/32 using the loss in LDAM <cit.>; and 6) MMS: Fine-tuned ViT-B/32 with MMS <cit.>.




  
CIFAR10/100-LT
As in the literature, we can create CIFAR10-LT and CIFAR100-LT by taking a subset of the original balanced CIFAR10 and CIFAR100 datasets <cit.>, and the imbalance factor γ is variable. We experiment with multiple imbalance factors in {10, 50, 100}.




First, on CIFAR100 with the imbalance factor of 100, we compare all methods based on CLIP. For our proposal, 
we set ViT-B/32 as the backbone 
and apply LFM with two different losses, i.e., cross entropy and MMS <cit.> that is the best in the literature. The comparison results to all baselines are summarized in Table <ref>. Based on the zero-short performance, we can observe that the pre-trained CLIP can help balance the performance in different categories, which demonstrates the effectiveness of pre-trained vision-language model to alleviate the class imbalance issue. Then, by fine-tuning the pre-trained image encoder, the overall accuracy can be improved. However, due to the severe imbalance, the performance of the tail classes is still lacking even when balanced losses are utilized. Our proposal can help improve the accuracy in all categories, where LFM combined with a loss well-suited for CLIP can further help improve the performance.









Then, we compare the fine-tuned CLIP models (ViT-B/32 is adopted) including our proposal with multiple existing state-of-art long-tailed learning methods in Table <ref> under different imbalance factors. We can observe that by fine-tuning the pre-trained CLIP image encoder, the performance can be significantly improved in all scenarios. Moreover, the state-of-the-art imbalance loss MMS <cit.> is very helpful, while our proposal can further significantly improve the performance in most cases. This further demonstrates the proposal of alleviating the class imbalance problem using pre-trained vision-language model and the effectiveness of LFM. It should be noted that methods using the backbone of ResNet50 are performing worse in general, and thus ResNet50 is not adopted in the following experiments. 



  
ImageNet-LT and Places-LT
We construct ImageNet-LT <cit.> by forming a subset of the ImageNet 2014 dataset <cit.>. The resulting imbalance ratio of ImageNet-LT is 256. As shown in Table <ref>, we can observe that compared to existing methods, our method begets better performance especially on few-shot accuracy (i.e., for tail classes) by both rebalancing and leveraging semantic similarities of classes. Observing that the LFM + MMS performance for minor classes falls behind LFM + CE, we hypothesize that MMS's sole focus on exercising semantic similarities and ignorance of class sample frequencies may overfit the many classes. A technique that only focuses on one may be problematic for tasks where semantic similarities happens to exist more frequently among the many classes.



In addition, we conduct experiments on Places-LT <cit.> using LFM with CE and MMS. Places-LT is a long-tailed subset of the original dataset Places2 <cit.>. It is a dataset for scene classification containing 365 classes, and it suffers from extreme imbalance (γ = 996). To account for its imbalance severity, we adjust local feature mixup hyperparameters to be highly in favor of the minority classes. We increase the value of τ, so that the probability distribution constructed by local sampling is more balanced. Additionally, we increase the value of α, so that the label is shifted to the tail classes, more heavily as shown in the supplementary. 



Table <ref> summarizes the results. The benefit from the pre-trained model by CLIP can be observed from the zero-shot performance on tail classes, which further demonstrates the advantage of the text supervision from CLIP. However, fine-tuning using our proposal is necessary to improve the performance. It should also be noted that due to the severe imbalance factor of this data, our proposal with CE is expected to be less effective compared to that with MMS <cit.>. LFM with MMS shows significantly better performance compared to state-of-the-arts, especially on medium-shot and few-shot classes, and demonstrates strong performance on many-shot classes as well. This further demonstrates the effectiveness of our proposed method on the long-tailed problem.




  
Effect of Mixup Techniques
To demonstrate the proposed LFM, we also compare it with the standard Mixup <cit.> and Remix <cit.> on CIFAR100-LT. Specifically, we fine-tune CLIP with the same hyperparameters and decoupled stages, using different mixup techniques. Each model is trained using cross entropy loss with the ViT-B/16 backbone. Remix is a mixup method that addresses the class-imbalance issue, and it makes a trade-off between many-shot and few-shot performances. For example, compared to the standard mixup, Remix can help improve the performance on few-shot classes but sacrifice the performance on many-shot classes. However, Remix ignores the semantic relationship between each class pair that CLIP can be used for. Our proposal shows significantly better performance in Table <ref>.







  
Visualization
To demonstrate our assumption on the local semantic relationship, we illustrate the geometric effect of fine-tuning CLIP with our proposal in Fig. <ref>. We demonstrate the effect by revealing the contrastive locality of image feature vector outputs, where the input is comprised of a set of randomly sampled images from 10 chosen classes {apple, pear, …, motorcycle} in CIFAR100. Our illustration contains the following 5 pairs of semantically related categories from CIFAR100: (apple, pair), (lobster, crab), (snake, worm), (bed, couch), and (bicycle, motorcycle). The legend contains the name of the class and the sample count in parenthesis. We choose these pairs to show that their semantic relations are aligned with their visual relations in terms of contrastive locality, as perceived by the image encoding layers.




With the ViT-B/32 vision encoder and fully connected layer, we obtain a 512-dimensional feature vector for each image. To reduce the high-dimensional feature vectors to three dimensions for human readability, we convert them using t-SNE trained for 1000 iterations and seed set to 1. At zero shot, we can observe that semantically related classes are located nearby (e.g., `apple' vs. `pear' and `bed' vs. `couch'), although some are poorly clustered (e.g., `lobster' vs. `crab'). This confirms our assumption that pre-trained vision-language model can align semantically related classes together. However, the separation between non-related classes are not clear in the pre-trained model. By fine-tuning the image encoder with cross entropy loss, the separation between non-related classes becomes clear, thanks to the help of head-class training data. However, we can observe that tail-class instances are largely overlapping with semantically related head-class instances (e.g., `lobster' vs. `crab'). Fortunately, by incorporating our proposal of LFM, tail-class instances can be pushed a bit away from their semantically related head-class instances without sacrificing the clear boundaries between non-related classes, which further demonstrates our proposal. 



§ CONCLUSION

Considering CLIP's ability to generalize to unseen categories, we leverage a fixed text encoder to enhance the performance of image categorization over long-tailed training distributions.


We enable the accuracy boost with the construction of a novel mixup technique that takes advantage of the semantic relationships between classes by probabilistic sampling based on their locality in the text encoder's feature space and slightly shift the label towards tail classes. Our extensive experiments on several benchmark long-tailed training data demonstrate the effectiveness of our proposal in alleviating the class imbalance issue with an efficient strategy that incorporates a fixed text encoder. Local feature mixup can be easily applied to not only vision-language backbones but also non multi-modal methods (i.e. vision-only architectures), which will be studied in our future work. However, both LFM and vision-language image classification are limited by the domain knowledge of the text encoder. Without further training, pre-trained CLIP performs poorly on domain-specific tasks as suggested by <cit.> due to its generic knowledge. 
Our method relies on the ability of the text encoder to capture pairwise semantic similarities among the class names present in the dataset which proves to be performant for the common domain such as CIFAR and ImageNet but not for biological names present in iNaturalist <cit.>, which will be our future work.  



§ ACKNOWLEDGEMENT

Yao and Hu's research is supported in part by NSF (IIS-2104270) and Advata Gift Funding. Zhong's research is supported in part by the Carwein-Andrews Graduate Fellowship and Advata Gift Funding. All opinions, findings, conclusions and recommendations in this paper are those of the author and do not necessarily reflect the views of the funding agencies.









§ APPENDIX



 §.§ Effect of Local Sampling







As discussed in the main paper, at each training step, local sampling feeds the model an image pair that holds semantically-related images, where the semantic relation is determined by the text encoder. In constructing the pair, the label of the first image is determined by 

    p(y = y_i) = n_i/∑^C_kn_k

which is to uniformly sample an image without replacement from the dataset. However, the label of the second image is determined by

    p_ls(y = y_j | y_i) = 
         exp( f_T_i· f_T_j / τ ) /∑^C_k=1 exp( f_T_i· f_T_k / τ )     i ≠ j
    
            0    o.w.

which ignores the sample count for any class label. Due to the negligence of the second label's sample count, the amount of times that the model sees minority classes can be increased effectively balancing the data distribution by resampling. To observe the amount of resampling, we show the sample count before and after local sampling as follows. Allow Y to be the random variable in the event that local sampling yields an instance of class y ∈{y_i, y_j}, and allow y_i to be the event that y_i = y and y_j to be the event that y_j = y. The probability that the model observes an image with class label y can be calculated as

    p(Y=y)    = p(y_i) + (1 - p(y_i)) p(y_j)
       = p(y_i) + (1 - p(y_i))∑_k, k≠ i^C p(y_j | y_k)p(y_k)  .

Using Eqns. <ref> and <ref>, p(Y) can be evaluated for all y, and we illustrate the resulting p(y) for every dataset in Figs. <ref>-<ref>. Additionally, we indicate the new imbalance factor as γ'. We can observe that the imbalance severity and the magnitude of long-tailed distribution can be well reduced, which demonstrates the effectiveness of our local sampling method. 






 §.§ Comparison between Textual Similarity and Visual Categorization






To further confirm our assumption that semantically related classes are visually related, we make a comparison between class label textual similarities and CLIP's zero-shot performance. Fig. <ref> shows a comparison between our semantic probability distribution p_ls and a confusion matrix of CLIP's zero-shot classification performance using CIFAR100's validation set. It can be observed that p_ls is correlated with the performance of zero-shot classification. By observing the blue cells in the confusion matrix, we see that the model more frequently struggles to find a decision boundary between related classes. When we sample with p_ls, we expect that we are sharing information with related classes more frequently and thus establish a decision boundary more optimally positioned for inference on the balanced validation data.










 §.§ Algorithms and Training Configurations


In this section, we summarize the algorithms for LocalSample, Mix, and the entire training process, where the effect on the proposed mixup technique on the decision boundary between nearby head and tail classes is illustrated in Fig. <ref>. Upon acceptance of this paper, we will also publicly release the code.










During the training, we use the hyperparameters and other training properties listed in Table <ref>. Most experiments have the same setup, but some minor adjustments are made largely due to differences in class label distributions. Under the circumstances of heavy class imbalance, we can simply raise the values of α and τ, which we do for Places-LT <cit.>. Detailed information for each dataset is provided in Table <ref>. The original dataset imbalance is summarized by the imbalance factor γ.



 §.§ Additional Ablation Studies


Besides the ablation study conducted in the main paper, we also conducted the following ablation studies.









  §.§.§ Effect of α

We study the effect of the intensity in which we shift the training label assigned to each mixup, for which we can control with α. The α value directly affects the positioning of the model's decision boundaries between class pairs, and we can expect lower values to extend the boundary of many-shot classes and higher values to extend the boundary of few-shot classes. In this study, we change α among the range [0, 2] on CIFAR100-LT with an imbalance factor of 100 using CLIP's ResNet50 backbone with the same configuration settings. From Fig. <ref>, we can easily observe that an increasing of α can slowly degenerate the performance of many-shot classes while improve the performance of the other, especially that of the few-shot classes as expected. The result also reveals that setting α to 1 works best for all accuracies.



  §.§.§ Effect of τ

To study the effect of different temperature settings for p_ls, we run multiple experiments with τ={.002, .01, .05, .25, 1.25, 31.25}. At lower values, we increase the probability that nearby class samples (i,j) are paired together. At higher values, the probability of two nearby class samples becoming paired is mitigated, and the class sampling becomes more balanced. We run our experiments on CIFAR100-LT <cit.> with an imbalanced factor of 100 using CLIP's ViT-B/16 backbone, which is of the same configuration settings. 
Fig. <ref> reveals that when we increase τ from a small value, all classes can benefit from LFM by mixing semantically related samples, while after τ=0.05 it plateaued. Therefore, τ=0.05 is adopted in the rest of our experiments.



