_supplementary

# BEM: Balanced and Entropy-based Mix
for Long-Tailed Semi-Supervised Learning

Hongwei Zheng1,2, Linyuan Zhou1, Han Li2, Jinming Su1, Xiaoming Wei1, Xiaoming
Xu1 †
1 Meituan 2 Shanghai Jiao Tong University
{zhenghongwei04, zhoulinyuan, sujinming, weixiaoming,
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Data mixing methods play a crucial role in semi-supervised learning (SSL), but
their application is unexplored in long-tailed semi-supervised learning
(LTSSL). The primary reason is that the in-batch mixing manner fails to
address class imbalance. Furthermore, existing LTSSL methods mainly focus on
re-balancing data quantity but ignore class-wise uncertainty, which is also
vital for class balance. For instance, some classes with sufficient samples
might still exhibit high uncertainty due to indistinguishable features. To
this end, this paper introduces the Balanced and Entropy-based Mix (BEM), a
pioneering mixing approach to re-balance the class distribution of both data
quantity and uncertainty. Specifically, we first propose a class balanced mix
bank to store data of each class for mixing. This bank samples data based on
the estimated quantity distribution, thus re-balancing data quantity. Then, we
present an entropy-based learning approach to re-balance class-wise
uncertainty, including entropy-based sampling strategy, entropy-based
selection module, and entropy-based class balanced loss. Our BEM first
leverages data mixing for improving LTSSL, and it can also serve as a
complement to the existing re-balancing methods. Experimental results show
that BEM significantly enhances various LTSSL frameworks and achieves state-
of-the-art performances across multiple benchmarks.

(a)

(b)

(c)

(d)

Figure 1: Experimental results on CIFAR10-LT [35]. (a)-(c): Class distribution
of unlabeled data quantity and entropy for three typical settings, which have
the same labeled data quantity distribution but differ in unlabeled ones. Both
the data quantity and entropy are the statistical averages within one epoch
after model convergence. Unexpected discrepancies are observed across all
settings between the distribution of data quantity and entropy, particularly
for head and tail classes. Notably, classes 3-6 exhibit the highest entropy,
indicating greater uncertainty. (d): Test accuracy gain brought by BEM for
various LTSSL frameworks in consistent setting. ††footnotetext:  $\dagger$
Corresponding Author

##  1 Introduction

Semi-supervised learning (SSL) capitalizes on unlabeled data to reduce the
cost of data labeling and boost the performance of models [61, 47, 54, 22,
36]. The general paradigm of most approaches is to randomly generate two views
of an image with various augmentation methods and then use the output of one
as the pseudo label to supervise the other [55, 72, 12]. As a simple and
effective augmentation technique introduced in supervised learning [26, 30, 7,
62], data mixing is widely used in SSL algorithms [9, 8, 63], further
enhancing model generalization and performance.

However, most existing SSL algorithms assume a balanced dataset, ignoring the
real-world prevalence of long-tailed class distributions [6, 16, 75, 74, 27,
28]. To deal with this class imbalance scenario, various long-tailed semi-
supervised learning (LTSSL) methods have been proposed, such as re-sampling
[65], logit alignment [45, 66], and pseudo label alignment [46, 39].
Nevertheless, data mixing is rarely explored in LTSSL. The primary reason is
that existing data mixing methods (_e.g._ MixUp [73], CutMix [69], and
SaliencyMix [58]) often perform random mixing within a batch. Consequently,
the infrequent tail classes may not be adequately sampled when the batch size
is small. This hinders a balanced class distribution, which is crucial for
LTSSL.

To make data mixing suitable for LTSSL, let us consider two key questions: i)
How can we apply data mixing to effectively re-balance the data quantity for
each class? ii) Is it sufficient to solely focus on re-balancing data quantity
to achieve class balance?  For the second question, we notice that previous
LTSSL methods mainly focus on addressing the issue of long-tailed class
distribution in terms of data quantity. These methods ignore the fact that
class performance also depends on class-wise uncertainty [40, 43, 52, 42],
which is associated with the training difficulty for each class. For instance,
some classes with sufficient samples may still encounter high training
difficulty due to the high uncertainty induced by indistinguishable features.
As shown in Fig. 1 (a)-(c), we quantify the uncertainty by entropy [43] and
compare this class-wise entropy with data quantity under three typical
settings [66]. The results reveal a significant disparity between the class
distribution of entropy and data quantity across all settings. This finding
emphasizes the limitation of solely re-balancing data quantity, as it does not
consider classes with high uncertainty, ultimately limiting performance
improvement. Thus, it is crucial to also address the re-balancing of class-
wise uncertainty, _i.e._ entropy.

To tackle the above problems, this paper presents a novel data mixing
paradigm, called Balanced and Entropy-based Mix (BEM), for LTSSL.
Specifically, we first introduce a simple mixing strategy, named as CamMix,
which has a strong localization capability to avoid redundant areas for
mixing. Then, we establish a class balanced mix bank (CBMB) to store and
sample class-wise data for mixing. The sampling function follows the estimated
class distribution of data quantity and we adopt the effective number [16] to
represent the realistic data quantity of each class. Our CamMix incorporated
CBMB can effectively re-balance the class-wise data quantity in an end-to-end
optimized manner, which can not be achieved by the re-sampling methods [32,
10, 80, 65] with the complex training procedures.

Further, we present a novel entropy-based learning approach to re-balance
class-wise uncertainty. Entropy-based sampling strategy (ESS) integrates
class-wise entropy into the quantity-based sampling function. In addition,
entropy-based selection module (ESM) adaptively determines the sampled data
ratio between labeled and unlabeled data during mixing to manage the trade-off
between guiding high-uncertainty unlabeled data [5, 64] with confident labeled
data and maximizing the utilization of unlabeled data. Finally, we incorporate
the class balanced loss [16] with class-wise entropy to form entropy-based
class balanced (ECB) loss.

We highlight that our BEM is the first method that leverages data mixing to
enhance LTSSL. Our results demonstrate that BEM can effectively complement
existing re-balancing methods by boosting their performance across several
benchmarks. As shown in Fig. 1 (d), our method enhances FixMatch [55],
FixMatch+LA [45], FixMatch+ABC [39], FixMatch+ACR [66], achieving to 11.8%,
4.4%, 1.4% and 2.5% average gains on test accuracy, respectively.
Additionally, BEM proves to be a versatile framework, performing well across
different data distributions, diverse datasets, and various SSL learners.

##  2 Related Work

Data mixing. MixUp [73] and CutMix [69] are typical data mixing methods used
in various computer vision tasks. While performing mixing at element-wise and
region-wise levels respectively, they share a common limitation of neglecting
class content, thus introducing substantial redundant context irrelevant to
class content [19, 48]. To achieve class balance in LTSSL, it is essential to
ensure that the selected region for mixing contains related class content and
avoids redundancy. SaliencyMix [58] alleviates this issue by using a saliency
map to ensure that selected regions contain class content, but the resulting
region is still too coarse to avoid numerous redundant areas. In our paper,
CamMix achieves tighter localization of class regions to minimize redundant
areas, which is particularly well-suited for LTSSL.

Data mixing in semi-supervised learning. Data mixing is crucial in SSL,
enhancing model performance by creating diverse training samples. For
instance, MixMatch [9] utilizes MixUp [73] as the data mixing technique to
learn a robust model. ReMixMatch [8] adds distribution alignment and
augmentation anchoring to the MixMatch framework. ICT [63] employs the mean
teacher model and implements MixUp on unsupervised samples. Despite widely
used in SSL algorithms, almost no methods in LTSSL apply data mixing. This is
mainly due to the limitation of employing in-batch mixing, which fails to
address the class imbalance problem. Our method stands out as the first to
incorporate data mixing in LTSSL.

Long-tailed semi-supervised learning. LTSSL is gaining attention due to its
real-world applicability. For example, CReST [65] refines the model by
iteratively enriching the labeled set with high-quality pseudo labels in
multiple rounds. ABC [39] uses an auxiliary balanced classifier, trained by
down-sampling majority classes. DASO [46] mitigates class bias by adaptively
blending linear and semantic pseudo labels. ACR [66], the current state-of-
the-art method, proposes a dual-branch network and dynamic logit adjustment.
However, none of these methods utilizes data mixing to further enhance their
performance as in SSL. CoSSL [20] uses MixUp at the feature level for minority
classes and decouples representation learning and classifier learning.
However, this feature mixing method assumes identical class distributions for
labeled and unlabeled data and requires a complex training approach. Our study
considers non-ideal class distributions and designs a simple image-level
mixing method in an end-to-end training framework.

##  3 Preliminaries

Figure 2: Left: The overview of Balanced and Entropy-based Mixing (BEM),
incorporating with FixMatch [55] as an example in this figure. BEM consists of
two sub-modules: class balanced mix bank (CBMB) and entropy-based learning
(EL). CBMB re-balances data quantity through the proposed CamMix, guided by a
class-balanced sampling function. EL further re-balances class-wise
uncertainty using three techniques: entropy-based sampling strategy (ESS),
entropy-based selection module (ESM) and entropy-based class balanced loss
($L_{ecb}$). Right: The sampling and CamMix process of BEM. The sampling
process considers both the class distribution of data quantity and
uncertainty, which are estimated on the fly. CamMix extracts the bounding box
from the high response area of the CAM to form mixed data. (The lock icon
denotes the unknown distribution that needs estimation, and the $\oplus$ icon
denotes the process of CamMix.).

Semi-supervised learning. In SSL, the training data consists of labeled data
$X=\\{(x_{n},y_{n})\\}_{n=1}^{N}$ and unlabeled data
$U=\\{u_{m}\\}_{m=1}^{M}$. Here, $x_{n}$ and $u_{m}$ are training samples,
$y_{n}$ is the ground truth, $N$ and $M$ denote the quantity of labeled data
and unlabeled data, respectively. A representative framework of SSL is
FixMatch [55], which utilizes unlabeled data with the _Weak and Strong
Augmentation_. For an unlabeled sample $u_{m}$, it first takes a _weakly_
-augmented version of $u_{m}$ as the input of the model $f(\cdot)$ to compute
the prediction. Then, it uses $q_{m}=\mathrm{argmax}(f(A_{w}(u_{m})))$ as one-
hot pseudo label, while applying the prediction from a _strongly_ -augmented
of $u_{m}$ to calculate the cross entropy loss $L_{u}$:

$L_{u}=\sum_{m=1}^{B}\mathbb{I}(\max(f(A_{w}(u_{m})))>\tau)\underbrace{\mathcal{H}(f(A_{s}(u_{m})),q_{m})}_{L_{cls}},$
(1)

where $B$ denotes the batch size, $\mathcal{H}(\cdot)$ is cross entropy and
$L_{cls}$ denotes original classification loss term.
$\mathbb{I}(\max(q_{m})>\tau)$ is the mask to filter low-confidence pseudo
label with a threshold of $\tau$, abbreviated as $M_{u}(\cdot)$ in the
following part. $A_{w}$ and $A_{s}$ denotes the _weak augmentation_ (_e.g._ ,
random crop and flip) and _strong augmentation_ (_e.g._ , RandAugment [15] and
Cutout [18]), respectively.

Long-tailed semi-supervised learning. In LTSSL, a dataset with a long-tailed
distribution is characterized by the minority of classes possessing a large
number of samples, while the majority of classes contain only a few samples.
Given $C$ classes across the dataset, $N_{c}$ represents the quantity of
labeled data for class $c$. Without loss of generality, we assume that
$N_{1}\geq N_{2}\geq\cdot\cdot\cdot\geq N_{C}$ and the imbalanced ratio is
denoted by $\gamma_{l}=N_{1}/N_{C}$. Similarly, we can denote the quantity of
unlabeled data as $M_{c}$ for class $c$ and the imbalanced ratio as
$\gamma_{u}=\max_{c}M_{c}/\min_{c}M_{c}$.

##  4 Balanced and Entropy-based Mix (BEM)

The Balanced and Entropy-based Mix (BEM) is a plug-and-play method based on
the existing SSL framework. Fig. 2 shows the overview of BEM, incorporating
FixMatch [55] as an example. Specifically, our entropy-based learning (EL)
takes the prediction of the weakly augmented samples as input to perform
entropy-based sampling and selection. Then, strongly augmented samples are
mixed with data from class balanced mix bank (CBMB) using our CamMix. Based on
the estimated distribution of data quantity and uncertainty, we employ the
entropy-based class balance (ECB) loss $L_{ecb}$ to train the overall
framework. Please refer to Appendix C for the pseudo-code of BEM.

###  4.1 CamMix

Most data mixing methods, such as MixUp [73] and CutMix [69], lack the
localization ability for class re-balancing. Although SaliencyMix [58] has
initial localization ability, it still tends to extract excessive redundant
context. To this end, we propose CamMix to replace the saliency map of
SaliencyMix with Class Activation Map (CAM) [79] to achieve more accurate
localization. Specifically, we feed images into the prediction model (_i.e._
ResNet50 [23]) to generate the CAM, where the last layer of the third block of
ResNet50 is used as the CAM layer. The resulting CAM is used to extract the
largest connected region using a threshold of $\tau_{c}$. Finally, we obtain
the bounding box of this region and paste the corresponding patch onto the
original image. The pseudo-code of CamMix can be found in the Appendix C.

###  4.2 Class Balanced Mix Bank (CBMB)

Previous in-batch data mixing methods used in SSL are limited to increasing
the data quantity of tail classes, thus failing to re-balance the class
distribution. To address this issue, we further propose a class balanced mix
bank (CBMB) that stores samples for each class and adequately selects samples
to be mixed based on a prior-based class-balancing rule. In essence, the more
frequent a class, the more samples are used in the data mixing process. As
noted in [16], there is overlap in the data, necessitating the use of the
effective number $E_{c}$ to measure the realistic class distribution of data
quantity:

$E_{c}=\frac{1-\beta^{N_{c}}}{1-\beta},$ (2)

where $N_{c}$ represents the data quantity of class $c$, while the hyper-
parameter $\beta$ is set to 0.999 in our experiments.

The effective number of labeled data, denoted as $E_{c}^{x}$, can be obtained
directly using Eq. 2. As the class distribution of unlabeled data is unknown,
we estimate it using a simple yet effective approach. Specifically, at each
iteration $t$, we obtain the class distribution of the pseudo label
$d_{c}^{ut}$ and update the class distribution of the entire unlabeled dataset
$d_{c}^{u}$, using an Exponential Moving Average (EMA) approach once the
training status stabilizes.

$d_{c}^{u}\leftarrow\lambda_{d}d_{c}^{u}+(1-\lambda_{d})d_{c}^{ut},$ (3)

where $\lambda_{d}$ denotes the EMA weight. To obtain the effective number of
unlabeled data for each class $E_{c}^{u}$, we substitute the class-wise data
quantity $N_{c}^{u}=Md_{c}^{u}$ into Eq. 2, where $M$ is the quantity of
entire unlabeled dataset. Then, we obtain the effective number of total data
for each class by $E_{c}=E_{c}^{x}+E_{c}^{u}$ and perform our CamMix using the
initial sampling function as follows:

$s_{c}=\frac{F_{c}}{\sum_{c=1}^{C}F_{c}},$ (4)

where $F_{c}=1/E_{c}$ and $s_{c}$ denotes the sampling probability for class
$c$. By accurately estimating the class distribution of the dataset, we can
enhance the precision of mixed data sampling. Our data mixing achieves class
balance among training samples, equivalent to re-sampling during training.

###  4.3 Entropy-based Learning (EL)

In the previous section, we re-balance training samples to initially alleviate
the long-tail distribution problem. However, class balance does not only
depend on data quantity. Class-wise uncertainty, which can be quantified by
entropy, is also vital for class performance as it reflects training
difficulty. Thus, we propose an entropy-based learning approach to re-balance
class-wise entropy, including entropy-based sampling strategy (ESS), entropy-
based selection module (ESM) and entropy-based class balanced (ECB) loss.

Entropy-based Sampling Strategy. To consider class-wise uncertainty in the
sampling process, we define the class-wise entropy $e_{c}^{x}$ and $e_{c}^{u}$
for the entire labeled and unlabeled dataset, and update them in EMA manner by
using the average entropy $e_{c}^{xt}$ and $e_{c}^{ut}$ at each training
iteration $t$ as follows:

$\displaystyle e_{c}^{xt}$
$\displaystyle=\frac{1}{N_{c}^{t}}\sum_{n=1}^{N_{c}^{t}}\sum_{c=1}^{C}-f_{c}(A_{w}(x_{n}))\log(f_{c}(A_{w}(x_{n})))$
(5) $\displaystyle e_{c}^{ut}$
$\displaystyle=\frac{1}{M_{c}^{t}}\sum_{m=1}^{M_{c}^{t}}\sum_{c=1}^{C}-f_{c}(A_{w}(u_{m}))\log(f_{c}(A_{w}(u_{m}))),$
$\displaystyle e_{c}^{x}$
$\displaystyle\leftarrow\lambda_{e}e_{c}^{x}+(1-\lambda_{e})e_{c}^{xt}$ (6)
$\displaystyle e_{c}^{u}$
$\displaystyle\leftarrow\lambda_{e}e_{c}^{u}+(1-\lambda_{e})e_{c}^{ut},$

where $N_{c}^{t}$ and $M_{c}^{t}$ represent the data quantity within one batch
belonging to class $c$ according to the ground truth and pseudo label
respectively, and $\lambda_{e}$ denotes the EMA weight. It’s worth noting that
we start estimating the entropy of data once the training status stabilizes.
Then, we obtain the total class-wise entropy, _i.e._
$e_{c}=e_{c}^{u}+e_{c}^{x}$, and subsequently compute the final sampling
probability $\hat{s}_{c}$:

$\hat{s}_{c}=\delta(\alpha s_{c}+(1-\alpha)s_{c}^{\prime}),$ (7)

where $s_{c}^{\prime}$ is the normalization of $e_{c}$, denoted as
$s_{c}^{\prime}=e_{c}/\sum_{c=1}^{C}e_{c}$, the hyper-parameter $\alpha$ is
used to balance between the effective number and entropy. The convex function
$\delta(\cdot)$ is utilized to map the sampling function better according to
FlexMatch [72]. Finally, we can obtain a more comprehensive sampling function
$\hat{s}_{c}$ for CamMix.

Entropy-based Selection Module. Previous work [9, 8, 63] in SSL primarily uses
unlabeled samples for data mixing. Yet, some pseudo labels possess high
uncertainty [5, 64], especially for challenging samples or in early training
stages, causing confirmation bias [5]. Our data mixing approach allows the
selection of both labeled and unlabeled data. We suggest augmenting high-
uncertainty unlabeled data with confident labeled data. However, this
beneficial mixing may under-utilize unlabeled data when the regions of labeled
data cover unlabeled ones, leaving them unexploited in training. This trade-
off is a crucial consideration in data mixing. Thus, we utilize sample-wise
entropy $e_{m}$ as the selection indicator between labeled and unlabeled
samples in data mixing as:

$e_{m}=\sum_{c=1}^{C}-f_{c}(A_{w}(u_{m}))\log(f_{c}(A_{w}(u_{m}))).$ (8)

We then define $M_{h}(\cdot)$ and $M_{l}(\cdot)$ as the masks of high and low
entropy for selecting labeled and unlabeled samples respectively. They are
also used to mask the unsupervised loss as in the Appendix A. These masks can
be expressed as:

$\displaystyle M_{h}(u_{m})=\mathbb{I}(e_{m}>\tau_{e})$ (9) $\displaystyle
M_{l}(u_{m})=\mathbb{I}(e_{m}<\tau_{e}),$

where $\tau_{e}$ is the selection threshold of the entropy mask, updated in
EMA manner:

$\tau_{e}\leftarrow\lambda_{\tau}\tau_{e}+(1-\lambda_{\tau})e^{t},$ (10)

where $\lambda_{\tau}$ denotes the EMA weight, $e^{t}$ is the average entropy
of unlabeled data at each training iteration $t$, _i.e._
$e^{t}=\frac{1}{B}\sum_{m=1}^{B}e_{m}$. In the early training stages, we
select more labeled samples for mixing due to the uncertainty of model
prediction on some unlabeled data. As training progresses and predictions
become more reliable, the utilization of unlabeled data increases.

Entropy-based class balanced loss. We further apply the class balanced loss,
which is first introduced in [16] to re-balance the class distribution by
utilizing the weighted loss based on the class-wise effective number as
$L_{cb}=L_{cls}/E_{c}$. By normalizing $1/{E_{c}}$ as Eq. 4, we can obtain
$L_{cb}=s_{c}L_{cls}$.

Moreover, to tackle the class-wise uncertainty problem in LTSSL, we propose
entropy-based class balanced loss $L_{ecb}$ on the unlabeled data. $L_{ecb}$
uses $\hat{s}_{c}$ to measure both the effective number and uncertainty as:

$L_{ecb}=\hat{s}_{c}^{u}L_{cls},$ (11)

where $\hat{s}_{c}^{u}$ is calculated by Eq. 7, but only based on unlabeled
data. Finally, $L_{ecb}$ can be weighted towards both tail classes and high
uncertainty classes, further re-balancing the training process. Unlike
previous entropy-based losses [22, 51], our loss focuses on class-wise
uncertainty instead of sample-wise, making it ideally suited for the LTSSL
problem characterized by large category gaps. A detailed description of the
loss functions can be found in the Appendix A.

##  5 Experiments

###  5.1 Experimental setup

Datasets. We perform evaluation experiments of our proposed method on widely
used long-tailed datasets, including CIFAR10-LT [35], CIFAR100-LT [35],
STL10-LT [14] and ImageNet-127 [20]. To create imbalanced versions of the
datasets, we randomly discard training samples to maintain the pre-defined
imbalance ratio. With the imbalance ratio $\gamma_{l}$ and maximum number
$N_{1}$ of labeled samples, we can calculate the number of labeled samples for
class $c$ as $N_{c}=N_{1}\times\gamma_{l}^{-\frac{c-1}{C-1}}$. Similarly,
using the parameters $\gamma_{u}$ and $M_{1}$, we can determine the class
distribution of unlabeled data quantity as in the labeled samples. For a
detailed introduction to the datasets, please refer to the Appendix B.

Implementation Details. Following DASO [46], we apply our method to various
baseline frameworks, including FixMatch [55], FixMatch + LA [45], FixMatch +
ABC [39] and FixMatch + ACR [66]. We compare our method with recent re-
balancing methods like DARP [33], CReST/CReST+ [65] and DASO [46]. For a fair
comparison, our code is developed based on DASO and ACR, implemented with
Pytorch [49]. We conduct our experiments on CIFAR10-LT, CIFAR100-LT and STL-10
using Wide ResNet-28-2 [70], and on ImageNet-127 using ResNet-50 [23]. The
top-1 accuracy on the test set is used as the evaluation metric. The mean and
standard deviation of three independent runs are reported. Due to the page
limitation, detailed training settings are provided in the Appendix B.

###  5.2 Results on CIFAR10/100-LT and STL10-LT.

We first consider the $\gamma_{l}=\gamma_{u}$ situation which is the most
common scenario in SSL. Then, we investigate the performance of the methods by
setting $\gamma_{l}\neq\gamma_{u}$, including uniform ($\gamma_{u}=1$) and
reversed ($\gamma_{u}=1/100$) scenarios.

In case of $\gamma_{l}=\gamma_{u}$. As shown in Tab. 1, we compare our method
with existing re-balancing methods under various baseline settings. When
setting FixMatch as the baseline, our BEM shows superior performance
improvement in most scenarios. When further adding LA to FixMatch for label
re-balancing, our BEM outperforms all other configurations. When integrating
ABC into FixMatch for pseudo label re-balancing, our BEM can benefit the
baseline more than the DASO. Finally, we also demonstrate that our methods can
complement ACR, achieving the SOTA performance with an average gain of 18.35%
over FixMatch for CIFAR10-LT. In summary, our BEM achieves consistent and
significant gain under all baseline settings, showing its great adaptability.
The main reason is that, unlike most previous methods with pseudo label or
logit adjustment, we directly re-balance the class distribution through data
mixing, a vital technique missing in them, thus complementing these methods.

Table 1: Comparison of test accuracy with combinations of different baseline
frameworks under $\gamma_{l}=\gamma_{u}$ setup on CIFAR10-LT and CIFAR100-LT.
The best results for each diversion are in bold.

|  | CIFAR10-LT | CIFAR100-LT
---|---|---|---
|  | $\gamma=\gamma_{l}=\gamma_{u}=100$ | $\gamma=\gamma_{l}=\gamma_{u}=150$ | $\gamma=\gamma_{l}=\gamma_{u}=10$ | $\gamma=\gamma_{l}=\gamma_{u}=20$
|  | $N_{1}=500$ | $N_{1}=1500$ | $N_{1}=500$ | $N_{1}=1500$ | $N_{1}=50$ | $N_{1}=150$ | $N_{1}=50$ | $N_{1}=150$
Algorithm |  | $M_{1}=4000$ | $M_{1}=3000$ | $M_{1}=4000$ | $M_{1}=3000$ | $M_{1}=400$ | $M_{1}=300$ | $M_{1}=400$ | $M_{1}=300$
Supervised | 47.3±0.95 | 61.9±0.41 | 44.2±0.33 | 58.2±0.29 | 29.6±0.57 | 46.9±0.22 | 25.1±1.14 | 41.2±0.15
w/LA [45] | 53.3±0.44 | 70.6±0.21 | 49.5±0.40 | 67.1±0.78 | 30.2±0.44 | 48.7±0.89 | 26.5±1.31 | 44.1±0.42
FixMatch [55] | 67.8±1.13 | 77.5±1.32 | 62.9±0.36 | 72.4±1.03 | 45.2±0.55 | 56.5±0.06 | 40.0±0.96 | 50.7±0.25
w/DARP [33] | 74.5±0.78 | 77.8±0.63 | 67.2±0.32 | 73.6±0.73 | 49.4±0.20 | 58.1±0.44 | 43.4±0.87 | 52.2±0.66
w/CReST+ [65] |  76.3±0.86 | 78.1±0.42 | 67.5±0.45 | 73.7±0.34 | 44.5±0.94 | 57.4±0.18 | 40.1±1.28 | 52.1±0.21
w/DASO [46] | 76.0±0.37 | 79.1±0.75 |  70.1±1.81 | 75.1±0.77 | 49.8±0.24 |  59.2±0.35 | 43.6±0.09 | 52.9±0.42
w/BEM (ours) | 75.8±1.13 |  80.3±0.62 | 69.7±0.91 |  75.7±0.22 |  50.4±0.34 | 59.0±0.23 |  44.1±0.18 |  54.3±0.36
FixMatch+LA [45] | 75.3±2.45 | 82.0±0.36 | 67.0±2.49 | 78.0±0.91 | 47.3±0.42 | 58.6±0.36 | 41.4±0.93 | 53.4±0.32
w/DARP [33] | 76.6±0.92 | 80.8±0.62 | 68.3±0.94 | 76.7±1.13 | 50.5±0.78 | 59.9±0.32 | 44.4±0.65 | 53.8±0.43
w/CReST [65] | 76.7±1.13 | 81.1±0.57 | 70.9±1.18 | 77.9±0.71 | 44.0±0.21 | 57.1±0.55 | 40.6±0.55 | 52.3±0.20
w/DASO [46] | 77.9±0.88 | 82.5±0.08 | 70.1±1.68 | 79.0±2.23 | 50.7±0.51 | 60.6±0.71 | 44.1±0.61 | 55.1±0.72
w/BEM (ours) |  78.6±0.97 |  83.1±0.13 |  72.5±1.13 |  79.9±1.02 |  51.3±0.26 |  61.9±0.57 |  44.8±0.21 |  56.1±0.54
FixMatch+ABC [39] | 78.9±0.82 | 83.8±0.36 | 66.5±0.78 | 80.1±0.45 | 47.5±0.18 | 59.1±0.21 | 41.6±0.83 | 53.7±0.55
w/DASO [46] |  80.1±1.16 | 83.4±0.31 | 70.6±0.80 | 80.4±0.56 |  50.2±0.62 | 60.0±0.32 |  44.5±0.25 | 55.3±0.53
w/BEM (ours) | 79.8±0.82 |  83.9±0.34 |  70.7±0.78 |  80.8±0.67 | 50.0±0.15 |  60.9±0.42 | 44.4±0.18 |  55.5±0.84
FixMatch+ACR [66] | 81.6±0.19 | 84.1±0.39 | 77.0±1.19 | 80.9±0.22 | 55.7±0.12 | 65.6±0.16 | 48.0±0.75 | 58.9±0.36
w/BEM (ours) |  83.5±0.33 |  85.5±0.28 |  78.1±0.99 |  83.8±1.12 |  55.8±0.32 |  66.3±0.24 |  48.6±0.45 |  59.8±0.37

Table 2: Comparison of test accuracy with combinations of different baseline
frameworks under $\gamma_{l}\neq\gamma_{u}$ setup on CIFAR10-LT and STL10-LT.
The $\gamma_{l}$ is fixed to 100 for CIFAR10-LT, and the $\gamma_{l}$ is set
to 10 and 20 for STL10-LT. The $N/A$ denotes the class distribution of data
quantity is unknown. The best results for each diversion are in bold.

|  | CIFAR10-LT($\gamma_{l}\neq\gamma_{u}$) | STL10-LT($\gamma_{u}=N/A$)
---|---|---|---
|  |  $\gamma_{u}=1$(uniform) |  $\gamma_{u}=1/100$(reversed) | $\gamma_{l}=10$ | $\gamma_{l}=20$
|  | $N_{1}=500$ | $N_{1}=1500$ | $N_{1}=500$ | $N_{1}=1500$ | $N_{1}=150$ | $N_{1}=450$ | $N_{1}=150$ | $N_{1}=450$
Algorithm |  | $M_{1}=4000$ | $M_{1}=3000$ | $M_{1}=4000$ | $M_{1}=3000$ | $M=100k$ | $M=100k$ | $M=100k$ | $M=100k$
FixMatch [55] | 73.0±3.81 | 81.5±1.15 | 62.5±0.94 | 71.8±1.70 | 56.1±2.32 | 72.4±0.71 | 47.6±4.87 | 64.0±2.27
w/DARP [33] | 82.5±0.75 | 84.6±0.34 | 70.1±0.22 | 80.0±0.93 | 66.9±1.66 | 75.6±0.45 | 59.9±2.17 | 72.3±0.60
w/CReST [65] | 83.2±1.67 | 87.1±0.28 | 70.7±2.02 |  80.8±0.39 | 61.7±2.51 | 71.6±1.17 | 57.1±3.67 | 68.6±0.88
w/CReST+ [65] | 82.2±1.53 | 86.4±0.42 | 62.9±1.39 | 72.9±2.00 | 61.2±1.27 | 71.5±0.96 | 56.0±3.19 | 68.5±1.88
w/DASO [46] | 86.6±0.84 | 88.8±0.59 |  71.0±0.95 | 80.3±0.65 |  70.0±1.19 | 78.4±0.80 |  65.7±1.78 | 75.3±0.44
w/BEM(ours) |  86.8±0.47 |  89.1±0.75 | 70.0±1.72 | 79.1±0.77 | 68.3±1.15 |  81.2±1.42 | 61.6±0.98 |  76.0±1.51
FixMatch+ACR [66] | 92.1±0.18 | 93.5±0.11 | 85.0±0.09 | 89.5±0.17 | 77.1±0.24 | 83.0±0.32 | 75.1±0.70 | 81.5±0.25
w/BEM(ours) |  94.3±0.14 |  95.1±0.56 |  85.5±0.21 |  89.8±0.12 |  79.3±0.34 |  84.2±0.56 |  75.9±0.15 |  82.3±0.23

In case of $\gamma_{l}\neq\gamma_{u}$. In real-world datasets, the class
distribution of unlabeled data remains unknown or inconsistent with labeled
data. For CIFAR10-LT, we consider two extreme scenarios: uniform and reversed.
For STL10-LT, where the class distribution of unlabeled data is unknown, we
set $\gamma_{l}\in\\{10,20\\}$ and $N_{1}\in\\{150,450\\}$.

As shown in Tab. 2, our methods yield an average improvement of 14.1% and
11.1% over FixMatch in two scenarios for CIFAR10-LT. However, our method is
less effective than DASO under the reversed setting. We speculate that data
mixing methods cannot achieve thorough re-balancing in challenging scenarios,
unlike approaches from the prediction perspective. However, integrating ACR
results in the best performance on CIFAR10-LT, even under the reversed
setting, with an average gain of 22.9% and 30.9% over FixMatch. Similarly, for
STL-10, our method enhances the performance of FixMatch and achieves the best
performance when combined with ACR. This highlights the value of our BEM for
re-balancing methods. Further comparisons of our method with more re-balancing
methods can be found in the Appendix D.

BEM on the SSL learner. We further validate the adaptability of BEM with
various SSL learners, including MeanTeacher [57], FlexMatch [72] and SoftMatch
[12]. Notably, FlexMatch and SoftMatch outperform FixMatch on balanced
datasets. For SoftMatch, we only apply $L_{cb}$ considering its training
process already re-weights the loss based on class-wise confidence. Following
DASO, we set $\gamma_{l}=100$ for CIFAR10-LT and $\gamma_{l}=10$ for
CIFAR100-LT and STL10-LT. As depicted in Tab. 3, our method enhances the
performance of all SSL learners under each setting. Specially, MeanTeacher
initially underperforms on the Long-Tailed dataset but achieves gains of
41.1%, 15.4%, and 37.9% on three datasets by applying BEM. SoftMatch, the
state-of-the-art SSL method, also gains an additional 5.5%, 9.7% and 9.7%
improvement with our BEM.

Table 3: Comparison of test accuracy with combinations of different SSL
learners, including MeanTeacher, FlexMatch and SoftMatch.

|  | C10-LT | C100-LT | STL10-LT
---|---|---|---|---
|  | $N_{1}=1500$ | $N_{1}=150$ | $N_{1}=450$
|  | $M_{1}=3000$ | $M_{1}=300$ | $M=100k$
Algorithm |  | $\gamma_{u}=100$ | $\gamma_{u}=1$ | $\gamma_{u}=10$ | $\gamma_{u}=N/A$
MeanTeacher[57] | 68.6±0.88 | 46.4±0.98 | 52.1±0.09 | 54.6±0.17
w/BEM(Ours) |  73.5±0.56 |  81.3±1.67 |  60.1±0.43 |  75.3±0.59
FlexMatch [72] | 79.2±0.92 | 82.2±0.23 | 62.1±0.86 | 74.9±0.42
w/BEM(Ours) |  81.2±0.50 |  88.0±0.17 |  68.4±0.79 |  81.2±0.92
SoftMatch [12] | 79.6±0.46 | 78.3±0.86 | 62.8±0.33 | 75.5±0.74
w/BEM(Ours) |  82.0±0.38 |  84.5±0.25 |  68.9±1.08 |  82.8±0.49

###  5.3 Results on ImageNet-127.

ImageNet127, initially introduced in [31] and later employed by CReST [65] for
imbalanced SSL, is a naturally imbalanced dataset with an imbalance ratio
$\gamma\approx 286$. It groups the 1000 classes of ImageNet [17] into 127
classes, based on the WordNet hierarchy. Due to resource constraints, we down-
sample the origin ImageNet127 images to $32\times 32$ or $64\times 64$ pixel
images [20] and randomly select 10% of training samples as the labeled set.
Given the long-tailed test set, we set $\alpha=0.2$ to reduce sampling and
loss weight bias towards tail classes, favoring high uncertainty classes
instead. Tab. 4 demonstrates the superiority of our method over FixMatch, even
without other re-balancing techniques. When combined with ACR, our method
achieves the best results for both image sizes (95.3% and 51.1% absolute gains
over FixMatch). This shows the applicability of our BEM to long-tailed test
datasets and its ability to enhance previous re-balancing methods.

Table 4: Comparison of test accuracy with combinations of different baseline
frameworks on ImageNet-127.

Algorithm |  | $32\times 32$ | $64\times 64$
---|---|---|---
FixMatch [55] | 29.7 | 42.3
w/DARP [33] | 30.5 | 42.5
w/DARP+cRT [33] | 39.7 | 51.0
w/CReST+ [65] | 32.5 | 44.7
w/CReST++LA [45] | 40.9 | 55.9
w/CoSSL [20] | 43.7 | 53.9
w/TRAS [67] | 46.2 | 54.1
w/BEM(Ours) | 53.3 | 58.2
w/ACR [66] | 57.2 | 63.6
w/ACR+BEM(Ours) | 58.0 | 63.9

###  5.4 Comprehensive analysis of the method.

We perform comprehensive ablation studies to further understand how our method
enhances baseline frameworks. Following DASO, we use CIFAR10-LT (C10) with
$N_{1}=500$, $\gamma=100$ and STL10-LT (STL10) with $N_{1}=150$,
$\gamma_{l}=10$ to cover both $\gamma_{l}=\gamma_{u}$ and
$\gamma_{l}\neq\gamma_{u}$ cases. Our baseline framework is FixMatch. More
results are provided in the Appendix D.

Ablation study on different mixing strategies. We compare our mixing method
with existing techniques including MixUp [73], CutMix [69] and SaliencyMix
[58] to demonstrate its effectiveness in Tab. 5. First, we mix data within the
same batch. SaliencyMix outperforms CutMix and MixUp on both datasets, and our
CamMix surpasses SaliencyMix, indicating better localization ability. By
further optimizing the in-batch mixing method, BEM achieves the best results.

Table 5: Ablation study on different mixing strategies. Apart from BEM, all other methods perform mixing within the same batch. Algorithm |  | C10 | STL10
---|---|---|---
FixMatch [55] | 67.8 | 56.1
w/MixUp [73] | 69.9 | 63.2
w/CutMix [69] | 70.2 | 62.5
w/SaliencyMix [58] | 70.8 | 64.0
w/CamMix(Ours) | 71.9 | 64.8
w/BEM(Ours) | 75.7 | 68.3

Ablation study on each component of BEM. We verify each component in BEM by
either removal or standard component replacement in Tab. 6. The accuracy on
both datasets reduces sharply when replacing CamMix with CutMix. It highlights
the importance of semantic region selection. We then remove CBMB and implement
random sampling, resulting in a maximum performance decrease of 5.1% and 4.9%,
respectively. This suggests our CBMB effectively tackles the long-tail
problem. Removing ESS, denoted as setting $\alpha=1$, also leads to a decline
in the model’s performance. When we remove ESM and merely use unlabeled data
mixing, it results in a 4.4% performance decrease on STL10. This implies
initial training phase guidance from confident labeled data resolves the
problem of pseudo label errors especially when $\gamma_{l}\neq\gamma_{u}$.
Finally, the removal of the ECB loss also causes a performance drop on both
datasets.

Table 6: Ablation study on each component of BEM. | C10 | STL10
---|---|---
BEM(Ours) | 75.7 | 68.3
w/o CamMix | 74.0 | 66.6
w/o CBMB | 72.1 | 65.0
w/o ESS | 74.7 | 67.0
w/o ESM | 75.3 | 65.3
w/o ECB Loss | 74.9 | 67.2

Furthermore, we conduct a qualitative analysis of the performance enhancement
achieved by BEM on CIFAR10-LT, setting $\gamma_{l}=\gamma_{u}=100$,
$N_{1}=500$ and $M_{1}=4000$. More visualization analysis can be seen in the
Appendix E.

Figure 3: Class distribution of data quantity and entropy in three settings.
Each mixed data is calculated as containing two classes.

Visualization of the class distribution of unlabeled data quantity and
entropy. To verify the effect of BEM on the re-balancing training process, we
visualize the class distribution of data quantity and entropy. Fig. 3 reveals
our method’s effect on re-balancing data quantity across all settings.
Moreover, our approach notably diminishes uncertainty via entropy and re-
balances class-wise entropy, particularly in uniform settings where higher
entropy classes engage more training samples, thus lowering uncertainty.

Visualization of T-SNE. Additionally, we visualize the learning representation
on the balanced test set using t-distributed stochastic neighbor embedding
(t-SNE) [59]. We apply our method to FixMatch and ACR respectively. The
results in Fig. 4 suggest that our method generates clearer classification
boundaries for representations.

(a)

(b)

(c)

(d)

Figure 4: Comparison of t-SNE visualization with combinations of FixMatch and
ACR.

Visualization of data mixing. As shown in Fig. 5, we compare intermediate
images from various data mixing methods on STL10 due to the high-resolution
input. Five images with different target sizes are selected to visualize.
CutMix shows strong randomness and tends to miss the class content, especially
when the target is small (see in (e)). Although SaliencyMix has initial target
localization ability, it often fails to accurately locate key areas and tends
to include numerous redundant contexts (see in (c) and (e)). CamMix shows the
best localization ability, accurately locating the class content based on CAM.
As $\tau_{c}$ increases, localization accuracy improves and inclusion of
redundant context decreases.

Figure 5: The visualization of data mixing process for CutMix, SaliencyMix,
and CamMix on STL10-LT. The red box indicates the image area selected by data
mixing.

##  6 Conclusion

In this work, we introduce a novel approach, Balanced and Entropy-based Mix
(BEM), to enhance long-tailed semi-supervised learning by re-balancing the
training process. Specially, we re-balance data quantity using the class
balanced mix bank and re-balance class-wise uncertainty through the entropy-
based learning approach. As the first method to leverage data mixing in LTSSL,
BEM significantly boosts the accuracy of various LTSSL frameworks across
multiple benchmarks, offering a complementary technique for other re-balancing
methods.

## References

  * Alpher [2002] FirstName Alpher.  Frobnication.  _IEEE TPAMI_ , 12(1):234–778, 2002.
  * Alpher and Fotheringham-Smythe [2003] FirstName Alpher and FirstName Fotheringham-Smythe.  Frobnication revisited.  _Journal of Foo_ , 13(1):234–778, 2003.
  * Alpher and Gamow [2005] FirstName Alpher and FirstName Gamow.  Can a computer frobnicate?  In _CVPR_ , pages 234–778, 2005.
  * Alpher et al. [2004] FirstName Alpher, FirstName Fotheringham-Smythe, and FirstName Gamow.  Can a machine frobnicate?  _Journal of Foo_ , 14(1):234–778, 2004.
  * Arazo et al. [2020] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8. IEEE, 2020.
  * Bengio [2015] Samy Bengio.  Sharing representations for long tail computer vision problems.  In _Proceedings of the 2015 ACM on International Conference on Multimodal Interaction_ , pages 1–1, 2015.
  * Berthelot et al. [2018] David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow.  Understanding and improving interpolation in autoencoders via an adversarial regularizer.  _arXiv preprint arXiv:1807.07543_ , 2018.
  * Berthelot et al. [2019a] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  _arXiv preprint arXiv:1911.09785_ , 2019a.
  * Berthelot et al. [2019b] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  _Advances in neural information processing systems_ , 32, 2019b.
  * Cao et al. [2019] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  _Advances in neural information processing systems_ , 32, 2019.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * Chen et al. [2023a] Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides.  Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning.  _arXiv preprint arXiv:2301.10921_ , 2023a.
  * Chen et al. [2023b] Xiaohua Chen, Yucan Zhou, Dayan Wu, Chule Yang, Bo Li, Qinghua Hu, and Weiping Wang.  Area: adaptive reweighting via effective area for long-tailed classification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 19277–19287, 2023b.
  * Coates et al. [2011] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 215–223. JMLR Workshop and Conference Proceedings, 2011.
  * Cubuk et al. [2020] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 702–703, 2020.
  * Cui et al. [2019] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9268–9277, 2019.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * DeVries and Taylor [2017] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  _arXiv preprint arXiv:1708.04552_ , 2017.
  * Du et al. [2023] Fei Du, Peng Yang, Qi Jia, Fengtao Nan, Xiaoting Chen, and Yun Yang.  Global and local mixture consistency cumulative learning for long-tailed visual recognitions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15814–15823, 2023.
  * Fan et al. [2022] Yue Fan, Dengxin Dai, Anna Kukleva, and Bernt Schiele.  Cossl: Co-learning of representation and classifier for imbalanced semi-supervised learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14574–14584, 2022.
  * Girshick [2015] Ross Girshick.  Fast r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1440–1448, 2015.
  * Grandvalet and Bengio [2004] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  _Advances in neural information processing systems_ , 17, 2004.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * He et al. [2021] Ruifei He, Jihan Yang, and Xiaojuan Qi.  Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6930–6940, 2021.
  * He et al. [2019] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.  Bag of tricks for image classification with convolutional neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 558–567, 2019.
  * Hong et al. [2023] Feng Hong, Jiangchao Yao, Zhihan Zhou, Ya Zhang, and Yanfeng Wang.  Long-tailed partial label learning via dynamic rebalancing.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Hong et al. [2024] Feng Hong, Jiangchao Yao, Yueming Lyu, Zhihan Zhou, Ivor Tsang, Ya Zhang, and Yanfeng Wang.  On harmonizing implicit subpopulations.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * Hu et al. [2021] Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang.  Semi-supervised semantic segmentation via adaptive equalization learning.  _Advances in Neural Information Processing Systems_ , 34:22106–22118, 2021.
  * Huang et al. [2017] Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger.  Snapshot ensembles: Train 1, get m for free.  _arXiv preprint arXiv:1704.00109_ , 2017.
  * Huh et al. [2016] Minyoung Huh, Pulkit Agrawal, and Alexei A Efros.  What makes imagenet good for transfer learning?  _arXiv preprint arXiv:1608.08614_ , 2016.
  * Kang et al. [2019] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  _arXiv preprint arXiv:1910.09217_ , 2019.
  * Kim et al. [2020a] Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin.  Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning.  _Advances in neural information processing systems_ , 33:14567–14579, 2020a.
  * Kim et al. [2020b] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song.  Puzzle mix: Exploiting saliency and local statistics for optimal mixup.  In _International Conference on Machine Learning_ , pages 5275–5285. PMLR, 2020b.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Laine and Aila [2016] Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  _arXiv preprint arXiv:1610.02242_ , 2016.
  * LastName [2014a] FirstName LastName.  The frobnicatable foo filter, 2014a.  Face and Gesture submission ID 324. Supplied as supplemental material fg324.pdf.
  * LastName [2014b] FirstName LastName.  Frobnication tutorial, 2014b.  Supplied as supplemental material tr.pdf.
  * Lee et al. [2021] Hyuck Lee, Seungjae Shin, and Heeyoung Kim.  Abc: Auxiliary balanced classifier for class-imbalanced semi-supervised learning.  _Advances in Neural Information Processing Systems_ , 34:7082–7094, 2021.
  * Li et al. [2022a] Bolian Li, Zongbo Han, Haining Li, Huazhu Fu, and Changqing Zhang.  Trustworthy long-tailed classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6970–6979, 2022a.
  * Li et al. [2021] Han Li, Bowen Shi, Wenrui Dai, Yabo Chen, Botao Wang, Yu Sun, Min Guo, Chenlin Li, Junni Zou, and Hongkai Xiong.  Hierarchical graph networks for 3d human pose estimation.  In _BMVC_ , 2021.
  * Li et al. [2023] Han Li, Bowen Shi, Wenrui Dai, Hongwei Zheng, Botao Wang, Yu Sun, Min Guo, Chenglin Li, Junni Zou, and Hongkai Xiong.  Pose-oriented transformer with uncertainty-guided refinement for 2d-to-3d human pose estimation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1296–1304, 2023.
  * Li et al. [2022b] Jun Li, Zichang Tan, Jun Wan, Zhen Lei, and Guodong Guo.  Nested collaborative learning for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6949–6958, 2022b.
  * Liu et al. [2021] Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda.  Unbiased teacher for semi-supervised object detection.  In _International Conference on Learning Representations_ , 2021.
  * Menon et al. [2020] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar.  Long-tail learning via logit adjustment.  _arXiv preprint arXiv:2007.07314_ , 2020.
  * Oh et al. [2022] Youngtaek Oh, Dong-Jin Kim, and In So Kweon.  Daso: Distribution-aware semantics-oriented pseudo-label for imbalanced semi-supervised learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9786–9796, 2022.
  * Ouali et al. [2020] Yassine Ouali, Céline Hudelot, and Myriam Tami.  An overview of deep semi-supervised learning.  _arXiv preprint arXiv:2006.05278_ , 2020.
  * Park et al. [2022] Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi.  The majority can help the minority: Context-rich minority oversampling for long-tailed classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6887–6896, 2022.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Qin et al. [2020] Jie Qin, Jiemin Fang, Qian Zhang, Wenyu Liu, Xingang Wang, and Xinggang Wang.  Resizemix: Mixing data with preserved object information and true labels.  _arXiv preprint arXiv:2012.11101_ , 2020.
  * Saito et al. [2019] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko.  Semi-supervised domain adaptation via minimax entropy.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 8050–8058, 2019.
  * Samuel and Chechik [2021] Dvir Samuel and Gal Chechik.  Distributional robustness loss for long-tail learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9495–9504, 2021.
  * Shi et al. [2022] Bowen Shi, Dongsheng Jiang, Xiaopeng Zhang, Han Li, Wenrui Dai, Junni Zou, Hongkai Xiong, and Qi Tian.  A transformer-based decoder for semantic segmentation with multi-level context mining.  In _European Conference on Computer Vision_ , pages 624–639. Springer, 2022.
  * Shim et al. [2020] Heereen Shim, Stijn Luca, Dietwig Lowet, and Bart Vanrumste.  Data augmentation and semi-supervised learning for deep neural networks-based text classifier.  In _Proceedings of the 35th annual ACM symposium on applied computing_ , pages 1119–1126, 2020.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ , 33:596–608, 2020.
  * Strudel et al. [2021] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 7262–7272, 2021.
  * Tarvainen and Valpola [2017] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _Advances in neural information processing systems_ , 30, 2017.
  * Uddin et al. [2020] AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, et al.  Saliencymix: A saliency guided data augmentation strategy for better regularization.  _arXiv preprint arXiv:2006.01791_ , 2020.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Van der Walt et al. [2014] Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu.  scikit-image: image processing in python.  _PeerJ_ , 2:e453, 2014.
  * Van Engelen and Hoos [2020] Jesper E Van Engelen and Holger H Hoos.  A survey on semi-supervised learning.  _Machine learning_ , 109(2):373–440, 2020.
  * Verma et al. [2019] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio.  Manifold mixup: Better representations by interpolating hidden states.  In _International conference on machine learning_ , pages 6438–6447. PMLR, 2019.
  * Verma et al. [2022] Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Arno Solin, Yoshua Bengio, and David Lopez-Paz.  Interpolation consistency training for semi-supervised learning.  _Neural Networks_ , 145:90–106, 2022.
  * Wang et al. [2022] Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, et al.  Freematch: Self-adaptive thresholding for semi-supervised learning.  _arXiv preprint arXiv:2205.07246_ , 2022.
  * Wei et al. [2021] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang.  Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10857–10866, 2021.
  * Wei and Gan [2023] Tong Wei and Kai Gan.  Towards realistic long-tailed semi-supervised learning: Consistency is all you need.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3469–3478, 2023.
  * Wei et al. [2022] Tong Wei, Qian-Yu Liu, Jiang-Xin Shi, Wei-Wei Tu, and Lan-Zhe Guo.  Transfer and share: semi-supervised learning from long-tailed data.  _Machine Learning_ , pages 1–18, 2022.
  * Xu et al. [2021] Zhengzhuo Xu, Zenghao Chai, and Chun Yuan.  Towards calibrated model for long-tailed visual recognition from prior perspective.  _Advances in Neural Information Processing Systems_ , 34:7139–7152, 2021.
  * Yun et al. [2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6023–6032, 2019.
  * Zagoruyko and Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ , 2016.
  * Zang et al. [2023] Yuhang Zang, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Semi-supervised and long-tailed object detection with cascadematch.  _International Journal of Computer Vision_ , 131(4):987–1001, 2023.
  * Zhang et al. [2021a] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  _Advances in Neural Information Processing Systems_ , 34:18408–18419, 2021a.
  * Zhang et al. [2017] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017.
  * Zhang et al. [2021b] Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun.  Distribution alignment: A unified framework for long-tail visual recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2361–2370, 2021b.
  * Zhang et al. [2023] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng.  Deep long-tailed learning: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Zheng et al. [2023] Hongwei Zheng, Han Li, Bowen Shi, Wenrui Dai, Botao Wang, Yu Sun, Min Guo, and Hongkai Xiong.  Actionprompt: Action-guided 3d human pose estimation with text and pose prompting.  In _2023 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 2657–2662. IEEE, 2023.
  * Zheng et al. [2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6881–6890, 2021.
  * Zhong et al. [2021] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia.  Improving calibration for long-tailed recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16489–16498, 2021.
  * Zhou et al. [2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2921–2929, 2016.
  * Zhou et al. [2020] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9719–9728, 2020.
  * Zhu et al. [2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  _arXiv preprint arXiv:2010.04159_ , 2020.

Supplementary Material

##  Appendix A Detailed Loss Functions

We detail the loss functions for the training in this section. For the labeled
data, we directly adopt the cross entropy $\mathcal{H}(\cdot)$ to calculate
the supervised loss $L_{s}$. For the unlabeled data, we first follow FixMatch
[55] to filter samples with low-confidence pseudo label by a mask
$M_{u}(u_{m})=\mathbb{I}(\max(f(A_{w}(u_{m})))>\tau)$. Then, we can obtain
$M_{h}$ and $M_{l}$, the masks of high and low entropy for selecting labeled
samples $(x_{m}^{s},y_{m}^{s})$ and unlabeled samples $u_{m}^{s}$ in data
mixing. Given the mixed samples $u_{m}^{\prime}$ from CAMmix, we can obtain
four types of unsupervised loss (_i.e._ $L_{u}^{h}$,
$L_{u}^{l},L_{u^{s}}^{h}$, and $L_{u^{s}}^{l}$), in which $L_{u}^{h}$,
$L_{u}^{l}$ and $L_{u^{s}}^{l}$ are weighted by the entropy-based class
balanced weight $\hat{s}^{u}$ to form the $L_{ecb}$. Specifically, $L_{u}^{h}$
and $L_{u}^{l}$ are supervised by the pseudo label of the original unlabeled
data $q_{m}$, while $L_{u^{s}}^{h}$ and $L_{u^{s}}^{l}$ are supervised by the
ground truth of the sampled labeled data $y_{m}^{s}$ and the pseudo label of
the sampled unlabeled data $q_{m}^{s}$, respectively. The final loss function
$L$ is weighted by $\lambda$ reflecting the proportion of area occupied by
original and sampled data as in CutMix [69]. Detailed loss functions are as
follows:

$\displaystyle L_{s}$
$\displaystyle=\sum_{n=1}^{B}\mathcal{H}(f(A_{w}(x_{n})),y_{n})$ (12)
$\displaystyle L_{u}^{h}$
$\displaystyle=\hat{s}^{u}\sum_{m=1}^{B}M_{u}(u_{m})M_{h}(u_{m})\mathcal{H}(f(u_{m}^{\prime}),q_{m})$
$\displaystyle L_{u}^{l}$
$\displaystyle=\hat{s}^{u}\sum_{m=1}^{B}M_{u}(u_{m})M_{l}(u_{m})\mathcal{H}(f(u_{m}^{\prime}),q_{m})$
$\displaystyle L_{u^{s}}^{h}$
$\displaystyle=\sum_{m=1}^{B}M_{h}(u_{m})\mathcal{H}(f(u_{m}^{\prime}),y_{m}^{s})$
$\displaystyle L_{u^{s}}^{l}$
$\displaystyle=\hat{s}^{u}\sum_{m=1}^{B}M_{u}(u_{m}^{s})M_{l}(u_{m})\mathcal{H}(f(u_{m}^{\prime}),q_{m}^{s})$
$\displaystyle L$
$\displaystyle=L_{s}+\lambda(L_{u}^{h}+L_{u}^{l})+(1-\lambda)(L_{u^{s}}^{h}+L_{u^{s}}^{l})$

##  Appendix B Detailed Experimental Setup

In this section, we provide additional information about the datasets and
implementation details.

Datasets. We evaluate our method in three scenarios, i.e., _1)_ the class
distribution of labeled data is consistent with the unlabeled data
($\gamma_{l}=\gamma_{u}$). _2)_ the labeled and unlabeled data fail to share
the same distribution ($\gamma_{l}\neq\gamma_{u}$). _3)_ The test data
possesses an imbalanced class distribution.

Algorithm 1 Balanced and Entropy-based Mix (BEM)

0: Labeled dataset $X$, unlabeled dataset $U$, model $f$, effective number of
labeled data $E_{c}^{x}$, CAM threshold $\tau_{c}$, area threshold $\tau_{a}$,
balanced parameter $\alpha$, number of iterations $T$.

0: Weak augmentation $A_{w}$, strong augmentation $A_{s}$.

1: for $t=1$ to $T$ do

2: $\\{(x_{n},y_{n})\\}_{n=1}^{B}\leftarrow X$,
$\\{u_{m}\\}_{m=1}^{B}\leftarrow U$

3: Pseudo label $q_{m}\leftarrow$ argmax $f(A_{w}(u_{m}))$

4: {Update training status}

5: Update CBMB according to $(x_{n},y_{n})$, $(u_{m},q_{m})$

6: Update class-wise data quantity $E_{c}$ via Eq. 2, 3

7: Update class-wise entropy $e_{c}$ via Eq. 5, 6

8: Update sampling probability $\hat{s}$, $\hat{s}^{u}$ via Eq. 7

9: Update sample-wise entropy $e_{m}$, entropy masks $M_{h}$, $M_{l}$ and
entropy selection threshold $\tau_{e}$ via Eq. 8, 9, 10

10: {Sampling}

11:
$\\{(x_{m}^{s},y_{m}^{s})\\}_{m=1}^{B},\\{u_{m}^{s}\\}_{m=1}^{B}\leftarrow$
Sample labeled and unlabeled data from CBMB following $\hat{s}$

12: {Selection and CamMix}

13: $\\{u_{m}^{\prime}\\}_{m=1}^{B}$, $\lambda\leftarrow$
CamMix($A_{s}(u_{m})$, $A_{w}(x_{m}^{s})$, $y_{m}^{s}$, $A_{w}(u_{m}^{s})$,
$f$) get mixed data and loss weight following $M_{h}$, $M_{l}$, $\tau_{c}$ and
$\tau_{a}$

14: {Compute losses}

15: Generate the mask of pseudo label $M_{u}$

16: $L_{s}\leftarrow\sum_{n=1}^{B}\mathcal{H}(f(A_{w}(x_{n})),y_{n})$

17:
$L_{u}^{h}\leftarrow\hat{s}^{u}\sum_{m=1}^{B}M_{u}(u_{m})M_{h}(u_{m})\mathcal{H}(f(u_{m}^{\prime}),q_{m})$

18:
$L_{u}^{l}\leftarrow\hat{s}^{u}\sum_{m=1}^{B}M_{u}(u_{m})M_{l}(u_{m})\mathcal{H}(f(u_{m}^{\prime}),q_{m})$

19:
$L_{u^{s}}^{h}\leftarrow\sum_{m=1}^{B}M_{h}(u_{m})\mathcal{H}(f(u_{m}^{\prime}),y_{m}^{s})$

20:
$L_{u^{s}}^{l}\leftarrow\hat{s}^{u}\sum_{m=1}^{B}M_{u}(u_{m}^{s})M_{l}(u_{m})\mathcal{H}(f(u_{m}^{\prime}),q_{m}^{s})$

21:
$L=L_{s}+\lambda(L_{u}^{h}+L_{u}^{l})+(1-\lambda)(L_{u^{s}}^{h}+L_{u^{s}}^{l})$

22: Update $f$ based on $\nabla L$ using SGD

23: end for

24: return

Algorithm 2 CamMix

0: Strong augmentation of unlabeled data $A_{s}(u_{m})$, weak augmentation of
sampled labeled data $A_{w}(x_{m}^{s})$, the label of sampled labeled data
$y_{m}^{s}$, weak augmentation of sampled unlabeled data $A_{w}(u_{m}^{s})$,
model $f$, high entropy mask $M_{h}$, low entropy mask $M_{l}$, CAM threshold
$\tau_{c}$, area threshold $\tau_{a}$, functions in skimage label($\cdot$) and
regionprops($\cdot$), the function of CutMix Mix($\cdot$).

0: Mixed data $\\{u_{m}^{\prime}\\}_{m=1}^{B}$, loss weight $\lambda$.

1: for $m=1$ to $B$ do

2: $q_{m}^{s}\leftarrow$ argmax $f(A_{w}(u_{m}^{s}))$

3: $CAM_{m}^{u}\leftarrow$ GradCAM$(A_{w}(u_{m}^{s}),q_{m}^{s})$

4: $S_{m}^{u}\leftarrow$ int$(CAM_{m}^{u}>\tau_{c})$

5: $P_{m}^{u}\leftarrow$ max(regionprops(label($S_{m}^{u}$))) get largest
connected region

6: if the area ratio of $P_{m}^{u}<\tau_{a}$ then

7: $bbox_{m}^{u}\leftarrow$ Random crop of $A_{w}(u_{m}^{s})$

8: else

9: $bbox_{m}^{u}\leftarrow$ The bounding box of $P_{m}^{u}$

10: end if

11: $bbox_{m}^{x}\leftarrow$ Calculate the bounding box for
$(A_{w}(x_{m}^{s}),y_{m}^{s})$ using a similar method in steps 3-10.

12: $u_{m}^{\prime}\leftarrow$ Mix($A_{s}(u_{m})$, $A_{w}(x_{m}^{s})$ or
$A_{w}(u_{m}^{s})$) following $bbox_{m}^{x}$, $M_{h}(u_{m})$, $bbox_{m}^{u}$,
$M_{l}(u_{m})$

13: $(1-\lambda_{m})\leftarrow$ The area ratio of $bbox_{m}^{x}$ or
$bbox_{m}^{u}$

14: end for

15: $\lambda\leftarrow$ The average of $\lambda_{m}$

16: return Mixed data $\\{u_{m}^{\prime}\\}_{m=1}^{B}$, loss weight $\lambda$

  * •

CIFAR10/100-LT CIFAR-10/100 [35] are originally class-balanced datasets, each
containing 500/5000 samples across 10 and 100 classes respectively. All images
are $32\times 32$ in size. Following previous work [46], we sample the
training data to create imbalanced versions of the datasets. We employ
different sampling ratios for labeled and unlabeled data to achieve various
data distributions, including $\gamma_{l}=\gamma_{u}$ and
$\gamma_{l}\neq\gamma_{u}$ scenarios. The test set contains $10k$ samples with
a balanced class distribution. The CIFAR dataset can be downloaded from
https://www.cs.toronto.edu/ kriz/cifar.html.

  * •

STL10-LT The STL-10 [14] dataset consists of 5000 class-balanced labeled data
and 1000k unlabeled data with an unknown distribution. To make an imbalanced
version of the dataset, we only sample the labeled data, while the
distribution of unlabeled data naturally differs from that of labeled data,
i.e.,$\gamma_{l}\neq\gamma_{u}$. All images are $96\times 96$ in size and the
dataset can be downloaded from https://cs.stanford.edu/ acoates/stl10/.

  * •

ImageNet-127 ImageNet-127 [20] is naturally an imbalanced dataset, thus it
doesn’t require any further processing. Moreover, it has an imbalanced test
set, which can validate scenario _3)_. To conserve computation resources, all
images are down-sampled to $32\times 32$ or $64\times 64$ in size and the
dataset can be downloaded from https://image-net.org/download-images.

Implementation details. Following previous training protocol [46], we conduct
our experiments on CIFAR10-LT, CIFAR100-LT and STL10-LT using Wide ResNet-28-2
[70], and on ImageNet-127 using ResNet-50 [23]. We train the model with a
batch size of 64 for $250k$ iterations, with an evaluation every 500
iterations. We use SGD with momentum as our optimizer and adopt a cosine
learning rate decay strategy by setting the learning rate to $\eta
cos(\frac{7\pi t}{16T})$, where $\eta$ is the initial learning rate, $t$ is
the current iteration number and $T$ is the total number of iterations. We set
the balance parameter $\alpha=0.5$ on CIFAR10-LT, CIFAR100-LT and STL10-LT,
and set it to 0.2 on ImageNet-127. We set all EMA update weights as
$\lambda=\lambda_{d}=\lambda_{e}=\lambda_{\tau}=0.999$. The CAM threshold
$\tau_{c}$ and area threshold $\tau_{a}$ are set to 0.8 and 0.1, respectively.
The epoch number for starting to estimate the data quantity and entropy of
unlabeled data is set to 5. We designate the final block as the CAM layer. We
adopt Softmax$(\cdot)$ as the mapping function $\delta(\cdot)$. Our
experiments are conducted on one NVIDIA Tesla V100 with the CentOS 7 system,
using PyTorch 1.11.0 and Torchvision 0.12.0.

##  Appendix C Pseudo-code for Our BEM Algorithm

We define the pseudo-code for our BEM and CamMix algorithm in Alg. 1 and 2,
respectively.

Table 7: Comparison of test accuracy with combinations of different baseline
models under $\gamma_{l}\neq\gamma_{u}$ setup on CIFAR10-LT and STL10-LT. The
$\gamma_{l}$ is fixed to 100 for CIFAR10-LT, and the $\gamma_{l}$ is set to 10
and 20 for STL10-LT. The best results for each diversion are in bold.

|  |  CIFAR10-LT($\gamma_{l}\neq\gamma_{u}$) |  STL10-LT($\gamma_{u}=N/A$)
---|---|---|---
|  |  $\gamma_{u}=1$(uniform) |  $\gamma_{u}=1/100$(reversed) | $\gamma_{l}=10$ | $\gamma_{l}=20$
|  | $N_{1}=500$ | $N_{1}=1500$ | $N_{1}=500$ | $N_{1}=1500$ | $N_{1}=150$ | $N_{1}=450$ | $N_{1}=150$ | $N_{1}=450$
Algorithm |  | $M_{1}=4000$ | $M_{1}=3000$ | $M_{1}=4000$ | $M_{1}=3000$ | $M=100k$ | $M=100k$ | $M=100k$ | $M=100k$
FixMatch [55] |  73.0±3.81 |  81.5±1.15 |  62.5±0.94 |  71.8±1.70 |  56.1±2.32 |  72.4±0.71 |  47.6±4.87 |  64.0±2.27
w/DASO [46] |  86.6±0.84 |  88.8±0.59 |  71.0 ±0.95 |  80.3±0.65 |  70.0±1.19 |  78.4±0.80 |  65.7±1.78 |  75.3±0.44
w/BEM(Ours) |  86.8±0.47 |  89.1±0.75 |  70.0±1.72 |  79.1±0.77 |  68.3±1.15 |  81.2±1.42 |  61.6±0.98 |  76.0±1.51
w/LA [45]+DASO [46] |  84.6±2.04 |  86.8±0.76 |  72.6 ±0.38 |  78.5±1.31 |  72.7±1.45 |  79.7±0.44 |  66.8±0.62 |  75.7±0.50
w/LA [45]+BEM(Ours) |  85.3±0.31 |  88.5±0.65 |  70.9±1.69 |  79.8±1.37 |  72.9±0.38 |  81.8±0.76 |  65.7±0.25 |  76.8±1.87
w/ABC [39]+DASO [46] |  85.2±1.56 |  88.4±0.82 |  70.1±1.25 |  79.8±0.21 |  71.8±1.17 |  78.4±0.58 |  67.3±2.06 |  75.9±0.43
w/ABC [39]+BEM(Ours) |  85.9±0.33 |  89.0±0.67 |  71.2±0.58 |  80.1±0.96 |  73.1±1.68 |  81.4±1.29 |  66.4±1.93 |  76.7±1.12
w/ACR [66] |  92.1±0.18 |  93.5±0.11 |  85.0±0.09 |  89.5±0.17 |  77.1±0.24 |  83.0±0.32 |  75.1±0.70 |  81.5±0.25
w/ACR [66]+w/BEM(Ours) |  94.3±0.14 |  95.1±0.56 |  85.5±0.21 |  89.8±0.12 |  79.3±0.34 |  84.2±0.56 |  75.9±0.15 |  82.3±0.23

Table 8: Comparison of test accuracy with combinations of different baseline
models under $\gamma_{l}\neq\gamma_{u}$ setup on CIFAR100-LT. The $\gamma_{l}$
is fixed to 10. The best results for each diversion are in bold.

|  |  CIFAR100-LT($\gamma_{l}\neq\gamma_{u}$)
---|---|---
|  |  $\gamma_{u}=1$(uniform) |  $\gamma_{u}=1/10$(reversed)
|  | $N_{1}=50$ | $N_{1}=150$ | $N_{1}=50$ | $N_{1}=150$
Algorithm |  | $M_{1}=400$ | $M_{1}=300$ | $M_{1}=400$ | $M_{1}=300$
FixMatch [55] |  45.5±0.71 |  58.1±0.72 |  44.2±0.43 |  57.3±0.19
w/DASO [46] |  53.9±0.66 |  61.8±0.98 |  51.0±0.19 |  60.0±0.31
w/BEM(Ours) |  54.8±0.55 |  63.6±0.91 |  50.8±0.25 |  60.7±0.12
w/LA [45]+DASO [46] |  54.7±0.40 |  62.4±1.06 |  51.1±0.12 |  60.5±0.23
w/LA [45]+BEM(Ours) |  56.5±0.43 |  64.1±0.87 |  51.7±0.20 |  61.3±0.17
w/ABC [39]+DASO [46] |  53.4±0.53 |  62.4±0.61 |  51.2±0.19 |  60.8±0.39
w/ABC [39]+BEM(Ours) |  55.2±0.35 |  64.7±0.87 |  51.1±0.10 |  61.4±0.29
w/ACR [66] |  66.0±0.25 |  73.4±0.22 |  57.0±0.46 |  67.6±0.12
w/ACR [66]+BEM(Ours) |  68.1±0.34 |  75.9±0.49 | 58.0±0.28 | 68.4±0.13

Table 9: Fine-grained results on CIFAR10-LT with
$N_{1}=1500,M_{1}=3000,\gamma_{l}=100$.

|  Consistent($\gamma_{u}=100$) |  Uniform($\gamma_{u}=1$) |  Reversed($\gamma_{u}=1/100$)
---|---|---|---
Algorithm | Many | Medium | Few | All | Many | Medium | Few | All | Many | Medium | Few | All
DASO | 95.1 | 78.6 | 60.4 | 78.1 | 89.6 | 84.4 | 85.7 | 86.3 | 84.0 | 71.6 | 68.2 | 74.3
BEM | 94.7 | 78.0 | 67.0 | 79.8 | 91.7 | 88.1 | 90.7 | 89.4 | 82.3 | 80.2 | 73.3 | 78.7
ACR | 93.9 | 81.6 | 75.3 | 83.4 | 92.8 | 90.6 | 97.9 | 93.5 | 90.7 | 83.8 | 96.4 | 89.7
ACR+BEM | 92.3 | 83.3 | 81.9 | 85.4 | 95.4 | 93.1 | 98.0 | 95.3 | 90.9 | 84.9 | 95.8 | 89.9

Table 10: Ablation study on different sampling strategies. EFF. denotes the
effective number.

| CBMB | ESS | EFF. | C10 | STL10
---|---|---|---|---|---
Random |  |  |  | 72.1 | 65.0
Quantity-based | ✓ |  | ✓ | 74.9 | 66.5
Entropy-based |  | ✓ |  | 74.4 | 65.9
w/o effective number | ✓ | ✓ |  | 75.2 | 67.3
Ours | ✓ | ✓ | ✓ | 75.7 | 68.3

Table 11: Ablation study on updating strategies of entropy selection threshold $\tau_{e}$. | C10 | STL10
---|---|---
Baseline | 67.8 | 56.1
$\tau_{e}=0.1$ | 74.7 | 66.6
$\tau_{e}=0.2$ | 75.2 | 67.2
$\tau_{e}=0.4$ | 75.1 | 66.9
$\tau_{e}=0.6$ | 74.4 | 66.4
w/ ours | 75.7 | 68.3

##  Appendix D Additional Experiment Results

In this section, we conduct a series of additional experiments to further
demonstrate the effectiveness of our BEM.

More results with re-balancing methods when $\gamma_{l}\neq\gamma_{u}$. We
present the results of combining with FixMatch and ACR under
$\gamma_{l}\neq\gamma_{u}$ setup in Tab. 2. As shown in Tab. 7, we further
combine our BEM with more re-balancing methods, including LA and ABC. Without
incorporating any re-balancing method, BEM’s performance is weaker than DASO
in some settings, particularly in the reversed setting. After combining two
re-balancing methods, BEM outperforms DASO in almost all settings. Further
integration with ACR achieves the state-of-the-art results in all scenarios
with an average 31.5% performance gain. In summary, our method needs to
combine with re-balancing methods to enhance the re-balancing ability in
challenging scenarios, and it in turn complements these methods.

More results on CIFAR100-LT. We also conduct experiments on CIFAR100-LT under
$\gamma_{l}\neq\gamma_{u}$ setup in Tab. 8. Results show that our BEM
outperforms DASO in almost all settings. By integrating with ACR, we can
achieve the best results in all scenarios (32.7% accuracy gain). It further
demonstrates that the complementation of BEM can boost the performance of most
re-balancing methods.

Table 12: Comparison of test accuracy on balanced datasets with combinations
of different SSL methods, including MeanTeacher, FixMatch, FlexMatch and
SoftMatch.

|  | CIFAR-10 | CIFAR-100 | STL-10
---|---|---|---|---
Algorithm |  | 40 | 250 | 4000 | 400 | 2500 | 10000 | 40 | 1000
MeanTeacher[57] |  29.81±1.60 |  62.54±3.30 |  91.90±0.21 |  18.89±1.44 |  54.83±1.06 |  68.25±0.23 |  28.28±1.45 |  66.10±1.37
w/BEM(Ours) |  43.13±2.55 |  74.31±1.79 |  92.65±0.23 |  30.92±3.69 |  60.73±2.14 |  72.54±0.19 |  37.31±2.59 |  78.74±1.38
FixMatch [55] |  92.53±0.28 |  95.14±0.05 |  95.79±0.08 |  53.58±0.82 |  72.97±0.16 |  77.80±0.12 |  64.03±4.14 |  93.75±0.33
w/BEM(Ours) |  93.96±0.37 |  95.37±0.03 |  95.93±0.11 |  55.24±0.93 |  73.12±0.14 |  77.95±0.11 |  66.45±3.29 |  93.98±0.65
FlexMatch [72] |  95.03±0.06 |  95.03±0.09 |  95.81±0.01 |  60.06±1.62 |  73.51±0.20 |  78.10±0.09 |  70.85±0.01 |  94.23±1.62
w/BEM(Ours) |  95.08±0.09 |  95.21±0.04 |  95.98±0.01 |  60.83±0.98 |  73.94±0.18 |  78.72±0.11 |  72.11±0.03 |  94.39±1.54
SoftMatch [12] |  95.09±0.12 |  95.18±0.09 |  95.96±0.02 |  62.90±0.77 |  73.34±0.25 |  77.97±0.03 |  78.58±3.48 |  94.27±0.24
w/BEM(Ours) |  95.11±0.08 |  95.37±0.06 |  96.12±0.07 |  63.13±0.92 |  73.56±0.08 |  78.14±0.08 |  79.09±3.87 |  94.43±0.38

Fine-grained results. In this experiment, we present the fine-grained results
in Tab. 9. We compare our BEM with DASO and ACR in three settings. Our method
surpasses DASO in all scenarios and further enhances the state-of-the-art
method (ACR). In particular, our method significantly improves the performance
of few-shot classes at the cost of negligible drop on head classes in the
consistent setting. Moreover, in all settings, our method shows a large
improvement in medium classes, which is brought by entropy-based learning.

BEM on balanced datasets. To verify the effect of our BEM on balanced
datasets, we conduct experiments on balanced datasets with combinations of
different SSL methods, including MeanTeacher, FixMatch, FlexMatch and
SoftMatch. Specifically, we set $\alpha=0$, meaning that we only consider the
differences in class-wise uncertainty distribution. As shown in Tab. 12, our
BEM enhances the performance of all baseline models, particularly the
MeanTeacher, where our method gains an average of 21.4%, 26.9% and 25.0%
improvement for three datasets. This demonstrates the potential of class-wise
uncertainty re-balancing in enhancing model performance for balanced datasets.

Ablation study on sampling strategies. To evaluate the effect of our sampling
strategy, we conduct a series of experiments by replacing the sampling
function. Results are summarized in Tab. 10. Random sampling only improves
performance slightly. Then, we split the class-balanced entropy-based sampling
function and find that the results drop on both datasets. Further, we replace
the effective number with the common number. Results indicate the effective
number more accurately measures the class distribution of datasets.

Ablation on the updating strategy of entropy threshold $\tau_{e}$. As shown in
Tab. 11, we perform experiments to validate the effect of the entropy
threshold $\tau_{e}$ updating strategy. When we filter the entropy mask with
fixed thresholds, the performance decreases and becomes unstable. Our EMA
updating strategy achieves the best result, indicating that it adaptively
adjusts the threshold following the training status of the model.

Ablation study on parameter $\alpha$. As shown in Tab. 13, we verify the
effect of $\alpha$ to balance the effective number and entropy in Eq. 7.
Results show the best $\alpha$ on CIFAR10-LT and STL10-LT are 0.5 and 0.3,
respectively. The visualization of sampling rate and class accuracy can be
seen in Appendix E.

Table 13: Ablation study on $\alpha$. | C10 | STL10
---|---|---
1.0 | 74.7 | 67.0
0.7 | 75.5 | 67.3
0.5 | 75.7 | 68.3
0.3 | 74.4 | 68.5
0 | 73.8 | 67.5

Ablation study on CAM threshold $\tau_{c}$. In Tab. 14, we study the effect of
CAM threshold $\tau_{c}$ on selected region. Results show that 0.8 is the best
threshold on both datasets. It indicates that the precise selection of
relevant regions is more advantageous for re-balancing long-tailed datasets.

Table 14: Ablation study on $\tau_{c}$. | C10 | STL10
---|---|---
0.9 | 73.0 | 65.8
0.8 | 75.7 | 68.3
0.6 | 74.4 | 67.3
0.4 | 71.5 | 64.6
0.2 | 69.3 | 61.3

Ablation study on the adding weight $\beta$. We conduct experiments to test
the impact of the adding weight $\beta$ in the equation $e_{c}=\beta
e_{c}^{u}+(1-\beta)e_{c}^{x}$. The results in Tab. 15 indicate that weight
addition has minimal impact. So we remove this parameter to simplify the
number of hyperparameters.

Table 15: Ablation study on the adding weight $\beta$. $\beta$ | C10 | STL10
---|---|---
0.7 | 75.1 | 67.7
0.5 | 75.7 | 68.3
0.3 | 75.8 | 68.1
0.1 | 74.9 | 67.9

More comparison with class-wise data mixing methods. We conduct additional
experiments to compare our BEM with other class-wise data mixing methods [78,
68]. The results in Tab. 16 show that BEM outperforms them. We infer that
these class-wise mixup methods are limited in not considering the uncertainty
issue in LTSSL.

Table 16: More comparison with class-wise data mixing methods. |  | C10 | STL10
---|---|---|---
FixMatch | 67.8 | 56.1
w/UniMix [68] | 72.9 | 66.0
w/MiSLAS [78] | 73.4 | 66.2
w/Ours | 75.7 | 68.3
Table 17: Comparison with AREA on supervised learning. | C10-LT | C100-LT
---|---|---
$\gamma$ | 200 | 50 | 200 | 50
CE | 65.7 | 74.8 | 34.8 | 43.9
AREA [13] | 75.0 | 82.7 | 43.9 | 51.8
Ours | 74.7 | 83.0 | 40.3 | 49.7

Comparison with AREA. We compare our BEM with AREA [13], which is a fully
supervised learning method in long-tailed learning. Our BEM is different from
AREA in three aspects: 1) Motivation: AREA does not consider class-wise
uncertainty. It optimizes the re-weighting strategy, which only focuses on
data quantity, by exploring the spanned space of each class and relations
between samples. While we propose to re-balance the class distribution of both
data quantity and uncertainty, which is more suitable for LTSSL. 2) Task: AREA
focuses only on the class imbalance issue in the supervised learning diagram.
While our method is specifically designed for LTSSL to further address the
issue of uncertainty in unlabeled sample predictions, which can not be
achieved by AREA. We also apply our BEM to supervised learning. Tab. 17 shows
that BEM is competitive with AREA, demonstrating its flexibility and
superiority. 3) Design: AREA is based on the re-weighting strategy, using the
effective area as class-wise weights in cross-entropy loss. While BEM is
primarily based on re-sampling, where we use class-wise data quantity and
uncertainty as sampling criteria for CamMix.

##  Appendix E Additional Visualization Analysis

In this section, we provide additional visualization analysis to better
understand our approach.

Visualization of confusion matrices on test set. We compare the confusion
matrices of the prediction from the test set. We conduct experiments on
CIFAR10-LT in the consistent scenario and apply our BEM to FixMatch and ACR,
respectively. As shown in Fig. 6, the prediction of FixMatch is significantly
biased towards the head classes, resulting in poor performance of the tail
classes. Our method greatly alleviates this bias, improving both the tail
performance and overall performance. ACR achieves good results in various
classes, and our method further improves the performance of the tail classes,
demonstrating the superiority and versatility of our method.

Figure 6: The confusion matrices of the test set on CIFAR10-LT under
$\gamma_{l}=\gamma_{u}$ setup.

Visualization of precision and recall on the test set. We analyze the
precision and recall on the test set to further verify the effect of our BEM.
As shown in Fig. 7, we apply our method to FixMatch and ACR. The results show
that the recall of tail classes achieves significant gains by combining our
BEM with both models.

(a) FixMatch

(b) FixMatch w/BEM

(c) ACR

(d) ACR w/BEM

Figure 7: The precision and recall of the test set on CIFAR10-LT under
$\gamma_{l}=\gamma_{u}$ setup.

Visualization of train curves and test accuracy class distribution. We further
assess the effect of BEM on FixMatch and ACR by plotting training curves and
class-wise test accuracy. As shown in Fig. 8(a), the low entropy ratio
increases, suggesting a large fraction of unlabeled data is used in the mixing
as the training state becomes stable. As shown in Fig. 8(b), our method
greatly improves the tail class performance of FixMatch and ACR.

(a)

(b)

Figure 8: (a): Train curves for tail low entropy ratio and tail class
accuracy. (b): Class distribution of test accuracy over different methods. C0
and C9 are the head and tail classes, respectively.

Visualization of the class distribution of sampling rate and accuracy under
different $\alpha$. We present the ablation study on $\alpha$ in Tab. 13. In
addition, we further visualize the class distribution of sampling rate and
accuracy under various $\alpha$. Fig. 9 (a) shows that as $\alpha$ increases,
the sampling rate of tail classes improves. When $\alpha$ is small, the
sampling function pays attention not only to tail classes but also to middle
classes with high uncertainty. In Fig. 9 (b), we can see that when
$\alpha=0.5$, both the tail class and the middle class with high uncertainty
have relatively high accuracy, indicating it achieves the balance of data
quantity and uncertainty.

(a)

(b)

Figure 9: Class distribution of sampling rate and test accuracy under various
$\alpha$ on CIFAR10-LT ($\gamma_{l}=\gamma_{u}=100$) using FixMatch. Figure
10: The visualization of data mixing process for CutMix, SaliencyMix, and
CamMix on CIFAR10-LT. The red box indicates the image area selected by data
mixing.

More visualization of data mixing. We provide the intermediate images of the
data mixing on STL10 in Fig. 5. To further illustrate the effectiveness of our
CamMix, we also present additional visualization results on CIFAR10 in Fig.
10. We select three images for each target size. Based on the results from the
two datasets, we can draw the following conclusions: 1) CutMix has a high
degree of randomness and often selects the context region. 2) The localization
ability of SaliencyMix needs to be optimized. The selection region is not
precise and tends to choose numerous redundant areas. 3) CamMix greatly
improves the localization ability due to the accuracy of CAM and excludes
irrelevant redundant areas as $\tau_{c}$ value decreases.

More visualization of t-SNE As displayed in Fig. 4, we show the t-SNE of
learning representations from the test data on CIFAR10-LT. We further conduct
experiments on STL10-LT to visualize the learning representations when
$\gamma_{l}\neq\gamma_{u}$. Results in Fig. 11 show that our method generates
clearer classification boundaries for representations when
$\gamma_{l}\neq\gamma_{u}$. Specially, the classification ability of FixMatch
is relatively poor, with most clusters gathered together. Our method greatly
enhances its classification ability.

(a)

(b)

(c)

(d)

Figure 11: Comparison of t-SNE visualization with combinations of FixMatch and
ACR on the test set of STL10-LT when $\gamma_{l}\neq\gamma_{u}$.

##  Appendix F Limitation and Future Work

A potential limitation is that the proposed BEM is restricted by only
exploring the data mixing for the LTSSL classification task, while ignoring
its further application for other vision tasks, such as object detection [11,
21, 81], semantic segmentation [77, 56, 24, 53] and others [42, 76, 41]. It is
worth noting that the application of semi-supervised learning for long-tailed
objection detection [71, 44] and semantic segmentation [25, 29] is not trivial
but much harder than the pure classification task, as it requires further
predict object location or semantic mask. In the future, we will extend our
BEM to more complex vision tasks to further demonstrate its effectiveness and
adaptability.