# Optimizing for ROC Curves on Class-Imbalanced Data by Training over a Family
of Loss Functions

Kelsey Lieberman  Shuai Yuan  Swarna Kamlam Ravindran  Carlo Tomasi

###### Abstract

Although binary classification is a well-studied problem in computer vision,
training reliable classifiers under severe class imbalance remains a
challenging problem. Recent work has proposed techniques that mitigate the
effects of training under imbalance by modifying the loss functions or
optimization methods. While this work has led to significant improvements in
the overall accuracy in the multi-class case, we observe that slight changes
in hyperparameter values of these methods can result in highly variable
performance in terms of Receiver Operating Characteristic (ROC) curves on
binary problems with severe imbalance. To reduce the sensitivity to
hyperparameter choices and train more general models, we propose training over
a family of loss functions, instead of a single loss function. We develop a
method for applying Loss Conditional Training (LCT) to an imbalanced
classification problem. Extensive experiment results, on both CIFAR and Kaggle
competition datasets, show that our method improves model performance and is
more robust to hyperparameter choices. Code will be made available here:
https://github.com/klieberman/roc_lct.

Class Imbalance, ROC Curves, Machine Learning

##  1 Introduction

Figure 1: Distribution of Area Under the ROC Curve (AUC) values obtained by
training the same model on the SIIM-ISIC Melanoma classification dataset with
48 different combinations of hyperparameters on VS loss. Results are shown at
three different imbalance ratios. As the imbalance becomes more severe, model
performance drops and the variance in performance drastically increases. LCT
addresses both of these issues by training over a family of loss functions,
instead of a single loss function.

Consider a classifier that takes images of skin lesions and predicts whether
the lesions are melanoma (Rotemberg et al., 2020). Such a system could be
especially valuable in underdeveloped countries where expert resources for
diagnosis are scarce (Cassidy et al., 2022). Classifying melanoma from images
is a problem with class imbalance since benign lesions are far more common
than melanomas. Furthermore, the accuracy on the melanoma (minority) class is
much more important than the accuracy on the benign (majority) class because
predicting a benign lesion as melanoma would result in the cost of a biopsy
while predicting a melanoma lesion as benign could result in the melanoma
spreading before the patient can receive appropriate treatment.

In this case, overall accuracy, even on a balanced test set, is clearly an
inadequate metric, as it implies that the accuracies of both classes are
equally important. Instead, Receiver Operating Characteristic (ROC) curves are
better suited for such problems (Egan, 1975). These curves plot the tradeoff
between the true positive rate (TPR) on the y-axis and the false positive rate
(FPR) on the x-axis over a range of classification thresholds. Unlike scalar
metrics (e.g., overall accuracy on a balanced test set or $F_{\beta}$), ROC
curves show model performance over a wide range of classification thresholds.
This allows practitioners to understand how the model’s performance changes
based on different classification thresholds and choose the best tradeoff for
their needs. ROC curves can also be summarized by their Area Under the Curve
(AUC) (Egan, 1975). Furthermore, both ROC curves and AUC have several
mathematical properties that make them preferred to alternative precision-
recall curves (Flach & Kull, 2015)111We provide definitions and visualizations
of commonly-used metrics for binary problems with imbalanced data in Appendix
A.

Although binary problems, like melanoma classification, are often cited as the
motivation for class imbalance problems and ROC curves are the de facto metric
of choice for such problems, the class imbalance literature largely focuses on
improving performance on longtailed multi-class datasets in terms of overall
accuracy on a balanced test set. We instead focus on binary problems with
severe imbalance and propose a method, which adapts existing techniques for
handling class imbalance, to optimize for ROC curves in these binary
scenarios.

In particular, we adapt models trained with Vector Scaling (VS) loss. This
loss is a general loss function that encompasses many other loss functions for
imbalanced data as special cases (Kini et al., 2021). It modifies Cross-
entropy loss by introducing additive and multiplicative factors to the logits,
which changes the inter-class margins. Although VS loss has shown strong
performance in the multi-class setting, it does have hyperparameters that
require tuning (e.g., the additive and multiplicative factors on the loss
function).

We find that, in the binary case, the effect of these hyperparameters is small
and reasonable at moderate imbalance ratios $\beta$ (where $\beta$ is defined
as the ratio of majority to minority samples); however, at large imbalance
ratios, small differences in these hyperparameters lead to very wide variance
in the results (Figure 1). This figure shows that increasing the imbalance
ratio not only decreases the AUCs (as expected) but also drastically increases
the variance in AUCs obtained by training models with slightly different
hyperparameter values. This suggests that traditional models trained using
losses with a fixed hyperparameter set are especially noisy, and extensive
hyperparameter tuning is necessary in the case of large imbalance ratios.

To address this problem and train a more robust model, we propose training a
single model on a wide range of hyperparameter values using Loss Conditional
Training (Dosovitskiy & Djolonga, 2020). We design a method for applying LCT
to classification problems under imbalance via VS loss. We find that this
method not only reduces the variance in model performance caused by
hyperparameter choices but also improves performance over the best
hyperparameter choices since it optimizes for many tradeoffs on the ROC curve.
We provide extensive results, both on CIFAR datasets and datasets of real
applications derived from Kaggle competitions, at multiple imbalance ratios
and across a wide range of hyperparameter choices.

In summary, our contributions are as follows.

  * •

We identify that higher levels of imbalance are not only associated with worse
model performance but also more variance.

  * •

We propose training over a family of loss functions as a way to reduce this
variance.

  * •

We modify Loss Conditional Training (LCT) for binary classification under
imbalance.

  * •

We show that this method consistently improves performance and reduces
variance at high imbalance ratios.

##  2 Related work

Many solutions have been proposed to address class imbalance, including
several specialized loss functions and optimization methods (Cao et al., 2019;
Rangwani et al., 2022; Buda et al., 2018; Kini et al., 2021; Shwartz-Ziv et
al., 2023). Perhaps the simplest of these is to change the class weights in
the loss function so that the minority and majority classes have “balanced”
class weights or class weights that are inversely proportional to the
frequency of the class in the training set (e.g., weighted Cross-entropy loss
(Xie & Manski, 1989)). Another popular loss function is Focal loss, which
down-weights “easy” samples (i.e., samples with high predictive confidence)
(Lin et al., 2017).

More recently, several loss functions have been proposed that add additive and
multiplicative factors to the logits before they are input to the softmax
function (Cao et al., 2019; Menon et al., 2021; Ye et al., 2020). These aim to
enforce larger margins on the minority class and/or calibrate the models. Kini
et al. (2021) recognized that many of the previous loss functions for
addressing class imbalance can be expressed by one general loss function:
Vector Scaling (VS) loss which gives strong performance on multi-class
datasets after hyperparameter tuning (Kini et al., 2021). Du et al. (2023) use
a global and local mixture consistency loss, contrastive learning, and a dual
head architecture. To gain insights into the effects of training over a family
of hyperparameters, we apply our method to standard networks and VS loss,
which has hyperparameters with well-known effects.

Additionally, Rangwani et al. (2022) proposed using Sharpness Aware
Minimization (SAM)(Foret et al., 2021) as an alternative optimizer and found
that this helped the model escape saddle points in multi-class problems with
imbalance. Similarly, Shwartz-Ziv et al. (2023) identify several tweaks that
can be made to models—including batch size, data augmentation, specialized
optimizers, and label smoothing—which can all improve training.

These methods are predominantly tested on multi-class datasets with overall
accuracy on a balanced test set as the primary metric. We instead propose to
optimize for the ROC curve in the binary case by training over a family of
loss functions. To this end, we use Loss Conditional Training (LCT), which was
proposed as a way to train one model to work over several rates in a variable-
rate problem (Dosovitskiy & Djolonga, 2020). LCT was proposed for applications
such as neural image compression and variational autoencoders and, to our
knowledge, we are the first to use it to improve ROC curves of a
classification problem.222We provide a more detailed related works section in
Appendix B

##  3 Problem setup

###  3.1 Data

In this paper, we focus on binary classification problems with class
imbalance. Specifically, let $D=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{n}$ be the
training set consisting of $n$ i.i.d. samples from a distribution on
$\mathcal{D}=\mathcal{X}\times\mathcal{Y}$ where $\mathcal{X}=\mathbb{R}^{d}$
and $\mathcal{Y}=\\{0,1\\}$. Then let $n_{c}$ be the number of samples with
label $c$. Without loss of generality, we assume that $n_{1}<n_{0}$ (i.e., 0
is the majority class and 1 is the minority class) and measure the amount of
imbalance in the dataset by $\beta=n_{0}/n_{1}>1$.

###  3.2 Predictor and predictions

Let $f$ be a predictor with weights $\boldsymbol{\theta}$ and
$\mathbf{z}=f(\mathbf{x},\boldsymbol{\theta})$ be $f$’s output for input
$\mathbf{x}$. We assume $\mathbf{z}$ are logits (i.e., a vector
$(z_{0},z_{1})$ of unnormalized scalars where $z_{c}$ is the logit
corresponding to class $c$) and $\mathbf{p}$ are the outputs of the softmax
function (i.e., a vector $(p_{0},p_{1})$ of normalized scalars such that
$p_{0}+p_{1}=1$). Then the model’s prediction is

$\displaystyle\hat{y}=\begin{cases}1&\text{if }p_{1}>t,\\\
0&\text{otherwise},\end{cases}$ (1)

where $t\in[0,1]$ is a threshold and $t=0.5$ by default. To find the ROC curve
of a classifier $f$, we compute the predictions over a range of $t$ values.

###  3.3 Loss function

For all experiments, we use the Vector Scaling (VS) loss as defined by Kini et
al. (2021). This loss is a modification of weighted cross entropy loss with
two hyperparameters $\Delta,\iota$ that specify an affine transformation
$\Delta z+\iota$ for each logit:

$\displaystyle\ell_{VS}(y,\mathbf{z})=-\omega_{y}\log\left(\frac{e^{\Delta_{y}z_{y}+\iota_{y}}}{\sum_{c\in\mathcal{Y}}e^{\Delta_{c}z_{c}+\iota_{c}}}\right).$
(2)

We follow parameterization by Kini et al. (2021) as follows:

$\displaystyle\iota_{c}=\tau\log\left(\frac{n_{c}}{n}\right)\quad\text{and}\quad\Delta_{c}=\left(\frac{n_{c}}{n_{0}}\right)^{\gamma}.$
(3)

where $\tau\geq 0$ and $\gamma\geq 0$ are hyperparameters set by the user. We
add an additional hyperparameter $\Omega$ to parameterize each class’ weight
$\omega_{c}$ as follows

$\displaystyle\omega_{c}=\begin{cases}\Omega&\text{if }c=1,\\\
1-\Omega&\text{otherwise},\end{cases}$ (4)

where $\Omega\in[0,1]$. In the binary case, we can simplify the loss as
follows (see Appendix G for details):

$\displaystyle\ell_{VS}(0,\mathbf{z})$
$\displaystyle=(1-\Omega)\log\left(1+e^{z_{1}/\beta^{\gamma}-(z_{0}+\tau\log\beta)}\right)$
(5) $\displaystyle\ell_{VS}(1,\mathbf{z})$
$\displaystyle=\Omega\log\left(1+e^{(z_{0}+\tau\log\beta)-z_{1}/\beta^{\gamma})}\right).$
(6)

Note that VS loss is equivalent to Cross-entropy loss when
$\Omega=0.5,\gamma=0$, and $\tau=0$. It is also equal to weighted cross
entropy loss when $\Omega=\frac{n_{0}}{n},\gamma=0$, and $\tau=0$.

##  4 Analysis on the effects of hyperparameters

Figure 2: Distribution of results obtained from training 512 models with
different hyperparameter values. Left: mean, min, max, and 95% confidence
interval of ROC curves. Right: boxplot of distribution of Area Under the ROC
Curve (AUC) values. Under class imbalance, differences in hyperparameter
values can lead to a large variance in results.

Metric $R^{2}$ AUC 0.30 Acc. 0.61 TPR 0.92

Figure 3: Effect of VS loss hyperparameters on AUC, overall accuracy, and TPR.
Results are shown for 512 models with different hyperparameter values. For
each metric and hyperparameter we plot a) all the values of the metric vs. the
hyperparameter (dots) and b) a fitted degree-2 polynomial between the metric
and hyperparameter (curves). In the table, we report the $R^{2}$ values of
polynomials fit with all three hyperparameters. All models were trained on
CIFAR10 cat vs. dog with $\beta=100$ with a ResNet32 model. Most of the
variance in AUC cannot be explained by the hyperparameter values.

###  4.1 Variance of results

As discussed in Section 3.3, our parameterization of VS loss has three
hyperparameters: $\Omega,\gamma$ and $\tau$. Although there are intuitive and
theoretical rationale for each of these hyperparameters (Kini et al., 2021),
it is not clear how sensitive a model’s performance is to their values,
especially in terms of ROC curves on a binary dataset.

To understand this better, we trained several models on the same dataset over
different combinations of hyperparameter values. Specifically, we trained 512
models on the same dataset (CIFAR10 cat vs. dog with $\beta=100$) and varied
only their values of $\Omega,\gamma$, and $\tau$. 333We used all unique
combinations of $\Omega\in\\{$0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99, 0.999$\\}$,
$\gamma\in\\{$0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7$\\}$, $\tau\in\\{$0, 0.5,
1, 1.5, 2, 2.5, 3, 3.5$\\}$.. Figure 2 shows that there is a wide variance in
the performance of these models. Specifically, these models have AUC values
which range from 0.58 to 0.76 with a standard deviation of 0.03. Additionally,
Figure 1 shows that this variance becomes more severe as the imbalance ratio
increases. This implies that models under severe class imbalance are
especially sensitive and variable in their results.

###  4.2 Correlation between hyperparameter values and scalar metrics

We next consider how much of this variance is explained by the hyperparameter
values (i.e., is the randomness coming from the hyperparameters themselves or
other randomness in training?). If the variance is mostly a result of
different hyperparameter choices, then we can reasonably expect the
practitioner to search for hyperparameters that generalize well. Otherwise,
tuning hyperparameters may not be an effective way to improve performance.

To test this, we fit polynomials between the three hyperparameters
($\Omega,\gamma,\tau$) and two scalar metrics: AUC and overall accuracy on a
balanced test set when $t=0.5$. Although TPR, equivalently recall, is not a
good indicator of a model’s overall performance by itself (because it is
trivial to increase TPR at the expense of FPR), we also fit a polynomial to
TPR to gain insights on the effect of the hyperparameters on a ROC curve. We
show results, including the $R^{2}$ of these regressions, in the three plots
in Figure 3. We also fit three (one for AUC, accuracy, and TPR) degree-2
polynomials using all three hyperparameters as features and report $R^{2}$
values of these polynomials in the table in Figure 3. We choose polynomials of
degree two instead of linear regression because we expect that an optimum
exists.

We see that no single hyperparameter is strongly correlated with AUC; however,
30% of the variance can be explained by the polynomial fit to all three
hyperparameters. This suggests that some variance can be removed by choosing
strong hyperparameters; however, much of the variance comes from other
sources. Additionally, only 61% of the variance in overall accuracy can be
explained by the polynomial with all three hyperparameters. While this is
significantly better than AUC’s $R^{2}$, it is still far from perfect. Thus,
hyperparameter values do not affect performance metrics in a very predictable
way. It would be preferable to reduce the variance of VS Loss over performing
extensive hyperparameter tuning.

###  4.3 Connection between hyperparameter values and softmax outputs

The third plot in Figure 3 shows that $\tau$ is strongly correlated with TPR
when the threshold $t=0.5$ ($R^{2}=0.92$). To understand this, recall that
$\tau$ affects the additive term on the logits in Equations 2 and 3.
Specifically, increasing $\tau$ enforces a larger margin on the minority
class. In general, this will lead to increasing the softmax scores of the
minority class at inference time, which is similar to post-hoc VS calibration
(Zhao et al., 2020). Thus, when $t$ is constant, increasing $\tau$ will likely
cause an increase in TPR.

We can also visualize the effect of $\tau$ by considering the set of “break-
even” logits, which are defined as the set of points of $\mathbf{z}$ where the
loss is equal whether the sample has label 0 or 1 (i.e., the points of
$\mathbf{z}$ such that $\ell_{VS}(1,\mathbf{z})=\ell_{VS}(0,\mathbf{z})$).
With regular Cross-entropy loss, this set is $z_{1}=z_{0}$. Assume instead
that we keep $\Omega=0.5,\gamma=0$, but have arbitrary values for $\tau$, then
the equation for the set of break-even points becomes
$z_{1}=z_{0}+\tau\log\beta$. In other words, varying $\tau$ is equivalent to
shifting the set of break-even points (and the rest of the loss landscape) by
$\tau\log(\beta)$. Figure 4 visualizes this.444Varying $\tau$ also shifts the
loss landscape in a similar way for any fixed $\Omega\in(0,1)$ and $\gamma>0$.
See Appendix I for details.

In addition, $\Omega$ and $\gamma$ also have well-understood effects on the
logit outputs. Appendices I and J include a general equation for the set of
break-even points and more loss contours. Just as we saw in the example with
$\tau$, different combinations of hyperparameters have different effects on
how $\mathbf{z}$ is scaled, which in turn affects their softmax scores. Thus,
training with different values of hyperparameters corresponds to optimizing
for different tradeoffs of TPR and FPR.

Figure 4: Effect of $\tau$ on loss landscape. Each plot shows
$\ell_{VS}(1,\mathbf{z})-\ell_{VS}(0,\mathbf{z})$ over $z_{0},z_{1}\in[-5,5]$
for $\beta=10$. White: “break-even” points. $\boldsymbol{\tau}$ shifts the
loss landscape.

##  5 Optimizing for ROC Curves via Loss Conditional Training

We saw in Section 4.3 that training with different combinations of
hyperparameters corresponds to optimizing for different tradeoffs of TPR and
FPR. Therefore, training one model over a range of hyperparameter values is a
proxy for optimizing over a range of the ROC curve. Using this intuition, we
design a system that optimizes one model over a range of VS loss
hyperparameter values via loss conditional training (LCT).

###  5.1 Loss Conditional Training (LCT)

Dosovitskiy & Djolonga (2020) observed the computational redundancy involved
in training separate models with slightly different loss functions, such as
neural image compression models with different compression rates. They
proposed loss conditional training (LCT) as a method to train one model to
work over a family of losses.

Let $\boldsymbol{\lambda}$ be a vector which parameterizes a loss function.
For example, we can parameterize VS loss by
$\boldsymbol{\lambda}=(\Omega,\gamma,\tau)$. Then let
$\mathcal{L}(\cdot,\cdot,\boldsymbol{\lambda})$ be the family of loss
functions parameterized by
$\boldsymbol{\lambda}\in\boldsymbol{\Lambda}\subseteq\mathbb{R}^{d_{\boldsymbol{\lambda}}}$
where $d_{\boldsymbol{\lambda}}$ is the length of $\boldsymbol{\lambda}$. In
our example, $\mathcal{L}(\cdot,\cdot,\boldsymbol{\lambda})$ is the set of all
possible VS loss functions obtained by different parameter values for
$\Omega,\gamma,\tau$.

Normally, training finds the weights $\boldsymbol{\theta}$ on a model $f$
which minimize a single loss function
$\mathcal{L}(\cdot,\cdot,\boldsymbol{\lambda})$ (i.e., we optimize for a
single combination of
$\boldsymbol{\lambda}_{0}=(\Omega_{0},\gamma_{0},\tau_{0})$ values in VS loss)
as shown below.

$\displaystyle\boldsymbol{\theta}^{*}_{\lambda_{0}}$
$\displaystyle=\operatorname*{arg\,min}_{\boldsymbol{\theta}}\mathbb{E}_{(\mathbf{x},y)\sim
D}\mathcal{L}(y,f(\mathbf{x},\boldsymbol{\theta}),\boldsymbol{\lambda}_{0})$
(7)

LCT instead optimizes over a distribution of $\boldsymbol{\lambda}$ values
$P_{\boldsymbol{\Lambda}}$ as shown below.

$\displaystyle\boldsymbol{\theta}^{*}$
$\displaystyle=\operatorname*{arg\,min}_{\boldsymbol{\theta}}\mathbb{E}_{\boldsymbol{\lambda}\sim
P_{\boldsymbol{\Lambda}}}\mathbb{E}_{(\mathbf{x},y)\sim
D}\mathcal{L}(y,f(\mathbf{x},\boldsymbol{\theta},\boldsymbol{\lambda}),\boldsymbol{\lambda})$
(8)

LCT is implemented on Deep Neural Network (DNN) predictors by augmenting the
network to take an additional input vector $\boldsymbol{\lambda}$ along with
each data sample $\mathbf{x}$. During training, $\boldsymbol{\lambda}$ is
sampled from $\boldsymbol{\Lambda}$ with each sample and is used in two ways:
1) as an additional input to the network and 2) in the loss function for the
sample. During inference, the model takes as input a $\boldsymbol{\lambda}$
along with the sample and outputs the appropriate prediction.

In order to condition the model on $\boldsymbol{\lambda}$, the DNN is
augmented with Feature-wise Linear Modulation (FiLM) layers (Perez et al.,
2018). These are small neural networks that take the conditioning parameter
$\boldsymbol{\lambda}$ as input and output a $\boldsymbol{\mu}$ and
$\boldsymbol{\sigma}$, used to modulate the activations channel-wise based on
the value of $\boldsymbol{\lambda}$. Specifically, suppose a layer in the DNN
has an activation map of size $H\times W\times C$. In LCT, we transform each
activation $\mathbf{f}\in\mathbb{R}^{C}$ by $\boldsymbol{\mu}$ and
$\boldsymbol{\sigma}$ as follows:
$\mathbf{\tilde{f}}=\boldsymbol{\sigma}*\mathbf{f}+\boldsymbol{\mu}$ where
both $\boldsymbol{\mu}$ and $\boldsymbol{\sigma}$ are vectors of size $C$, and
“$*$” stands for element-wise multiplication.

###  5.2 Details about implementing LCT for classification

We define our family of loss functions using VS loss. We try several choices
for $\boldsymbol{\lambda}$, including
$\boldsymbol{\lambda}=\Omega,\boldsymbol{\lambda}=\gamma,\boldsymbol{\lambda}=\tau$
and $\boldsymbol{\lambda}=(\Omega,\gamma,\tau)$. In the first three cases, we
set the hyperparameters excluded from $\boldsymbol{\lambda}$ to constants
(e.g., when $\boldsymbol{\lambda}=\tau$, $\Omega$ and $\gamma$ are set to
constants). We find that $\boldsymbol{\lambda}=\tau$ is especially strong at
improving the average performance and reducing the variance in performance
(Figure 8), which is supported by our insights in Figure 3 and Section 4.3.
Thus, for most of our experiments, we use $\boldsymbol{\lambda}=\tau$.

For each mini-batch, we draw one $\boldsymbol{\lambda}$. This is done by
independently sampling each hyperparameter in $\boldsymbol{\lambda}$ from a
distribution that has a linear density over range $[a,b]$. In this
distribution, the user specifies $a,b$, and the height of the probability
density function (pdf) at $b$, $h_{b}$. The height at $a$, $h_{a}$, is then
found to ensure the area under the pdf is 1. Unlike the triangular
distribution, this general linear distribution does not require the pdf to be
0 at either endpoint and, unlike the uniform function, it does not require the
slope of the pdf to be 0.555Appendix E contains more details about the linear
distribution. We use this general distribution as a way to experiment with
different distributions.

Although the value of $\boldsymbol{\lambda}$ at inference time affects the TPR
and FPR when $t=0.5$, we observe that the ROC curves are almost identical for
all values of $\boldsymbol{\lambda}$ in the range the model was trained on
(see Appendix D). Thus, we evaluate LCT models at one $\boldsymbol{\lambda}$
and find their ROC curve from this output.

##  6 Experiments

###  6.1 Experimental setup

Methods. For each dataset and imbalance ratio $\beta$, we train 48 models with
the regular VS loss, varying their hyperparameter values. We also train 48
models with LCT applied to VS loss where $\boldsymbol{\lambda}=\tau$ (VS loss
+ LCT). We use $\Omega=0.5,0.7,0.9,0.99$, $\gamma=0.0,0.2,0.4$ for both
methods (note that we set $\Omega,\gamma$ to constants when
$\boldsymbol{\lambda}=\tau$ in LCT). We use $\tau=0,1,2,3$ for VS loss, and
$[a,b]=[0,3]$ with $h_{b}=0.0,0.15,0.33,0.66$ for VS loss + LCT. We evaluate
with $\boldsymbol{\lambda}=\tau=3$. To apply LCT, we augment the networks with
one FiLM block after the final convolutional layer and before the linear
layer. This block is comprised of two linear layers with 128 hidden units.
Specifically, the first layer takes one input $\boldsymbol{\lambda}$ and
outputs 128 hidden values and the second layer outputs 64 values which are
used to modulate the 64 channels of convolutional activations (total of
$1*128+128*64=8320$ parameters if $\boldsymbol{\lambda}$ is a scalar).

Datasets. We experiment on both toy datasets derived from CIFAR10/100 and more
realistic datasets derived from Kaggle competitions. CIFAR10 cat/dog consists
of the cat and dog classes from the CIFAR10 dataset. We also show results for
all pairs of CIFAR10 classes in Section 6.3, but find the cat/dog pair is
particularly well-suited for additional experiments since it is a challenging
classification problem. CIFAR100 household electronics vs. furniture was
proposed as a binary dataset with imbalance by (Wang et al., 2016). Each class
contains 5 classes from CIFAR100: electronics contains clock, computer
keyboard, lamp, telephone, and television while furniture contains bed, chair,
couch, table, and wardrobe. For all CIFAR experiments, we split the data
according to their train and test splits. Kaggle Dogs vs. Cats contains 25,000
images of dogs and cats from the Petfinder website (Cukierski, 2013). We split
the data so that each class has 1,000 validation samples (a 92/8
train/validation split). Finally, SIIM-ISIC Melanoma is a classification
dataset, which was designed by the Society for Imaging Informatics in Medicine
(SIIM) and the International Skin Imaging Collaboration (ISIC) for a Kaggle
competition (Zawacki et al., 2020). This is a binary dataset where 8.8% of the
samples belong to the positive (melanoma) class (i.e., $\beta=12$). We follow
the procedure of (Fang et al., 2023) and combine the 33,126 and 25,331 images
from 2020 and 2019 respectively and split the data into an 80/20
train/validation split. Table 9 outlines the sizes of these. We subsample the
minority class to obtain various imbalance ratios $\beta$. Specifically, we
test the CIFAR datasets at $\beta=10,50,100,200$, the Kaggle datasets at
$\beta=100,200$.

Model architectures and training procedure. We use two model architectures and
training procedures, following the literature for each type of dataset. For
CIFAR10/100 data, we follow (Kini et al., 2021) and use a ResNet-32 model
architecture. We find that LCT takes longer to train, so we train for 500
epochs instead of 200 (note: training the baseline models longer does not
significantly alter performance and we do this for consistency). We train
these models from scratch using an initial learning rate of 0.1 and decrease
this to 0.01 and 0.001 at 400 and 450 epochs respectively. For the Kaggle
datasets, we follow (Shwartz-Ziv et al., 2023) and use ResNext50-32x4d with
weights pre-trained on ImageNet. We train the Melanoma dataset for 10 epochs,
following the learning rates and weight decays of Fang et al. (2023) and train
the Kaggle Dogs vs. cats models for 30 epochs, following the learning rates
and weight decays of other finetuning tasks in (Fang et al., 2023). For all
experiments, we use a batch size of 128 and gradient clipping with a maximum
norm equal to 0.5. Additionally, except for Figure 7, we use Stochastic
Gradient Descent (SGD) optimization with momentum=0.9. We run all experiments
on A5000 GPUs.

Figure 5: Distribution of ROC curves of models trained with (red) and without
(blue) LCT on CIFAR datasets at four different imbalance ratios $\beta$.
Solid, dashed, and dotted-dashed curves are the mean, minimum, and maximum of
the ROC curves respectively. The shaded region is one standard deviation away
from the mean. Datasets are tested with $\beta=10,50,100,200$. At high
imbalance ratios, LCT consistently improves the mean, max, and min of the ROC
curves. Figure 6: Distribution of ROC curves of models trained with (red) and
without (blue) LCT on Kaggle Dogs vs. Cats and SIIM-ISIC Melanoma datasets at
two different imbalance ratios $\beta$. Solid, dashed, and dotted-dashed
curves are the mean, minimum, and maximum of the ROC curves respectively. The
shaded region is one standard deviation away from the mean. Results are shown
on a log scale because all TPR values are compressed towards 1. LCT
significantly improves performance on Melanoma, especially at high TPRs.

###  6.2 VS loss with and without LCT

Figures 5 and 6 compare the performance of the VS loss and VS loss + LCT
methods on all four datasets, each at multiple imbalance ratios. For each
method, we find the mean, min, max, and standard deviation of the 48 models
trained with that method. We calculate these values across false positive
rates (FPRs). We also show the distribution of AUCs on the SSIM-ISIC Melanoma
dataset in Figure 1.

Figure 5 shows that at moderate imbalance ratios ($\beta=10$), there is very
little variation in the ROC curves obtained by models trained with different
hyperparameters. Here the two methods are essentially tied in their
performance. However, at high imbalance ratios $\beta=100,200$ (Figures 5 and
6), the variance in ROC curves becomes much more drastic. Here LCT
consistently outperforms the Baseline method. Specifically, LCT improves the
max and mean of the ROC curves in all cases, with a large gap in improvement
for Melanoma at $\beta=100,200$. Additionally, LCT significantly decreases the
variation in the results. Notice how the highlighted bands, representing the
standard deviation, are much narrower for LCT (red) than Baseline (blue).
Furthermore, the LCT’s minimum (red dashed) curves are much higher Baseline’s
(blue dashed). We also see this in the distribution of AUC values shown in
Figure 1. This reduction in variance indicates that LCT is more robust to
hyperparameter changes and hence requires less tuning.

###  6.3 Evaluation over all CIFAR10 pairs

For a more comprehensive evaluation, we compare the results of VS loss with
and without LCT on all 45 pairs of classes chosen from the 10 CIFAR10 classes.
We train each method on each of these binary datasets using 48 different
hyperparameter values. We then analyze the aggregated performance of these
models by AUC. For each binary dataset, we find the mean, maximum, minimum,
and standard deviation of the AUCs over the hyperparameter values. We then
compare Baseline (VS loss) and LCT (VS loss + LCT) for each of these metrics.
For example, let $a^{max}_{k,\text{Baseline}}$ and $a^{max}_{k,\text{LCT}}$ be
the maximum AUCs achieved on dataset $k$ by LCT and Baseline respectively.
Then we obtain 45 of these values for each method (one for each dataset $k$).
We then analyze these using a paired t-test and report results in Table 1. The
table shows that LCT improves performance, increasing the min and mean AUC in
all cases, as well as the max AUC in most cases. It also drastically reduces
the standard deviation of AUC values. This shows that LCT significantly
outperforms Baseline on a comprehensive set of datasets.

|  # LCT $>$ |  # Base. $>$ | Avg. diff | P-value
---|---|---|---|---
Max | 38 | 7 | 0.005 | 5.2e-05
Mean | 45 | 0 | 0.012 | 2.0e-14
Min | 45 | 0 | 0.036 | 1.6e-16
Std. dev. | 1 | 44 | -0.007 | 2.3e-16
Table 1: Differences between AUCs obtained via Baseline (VS loss) and LCT (VS
loss + LCT) over 45 binary subsets of CIFAR10. First and second columns show
the number of datasets where
$a^{max}_{k,\text{LCT}}>a^{max}_{k,\text{Baseline}}$ and vice versa. We also
show the average differences in the maximum AUC values over all datasets
(i.e., $\frac{1}{|K|}\sum_{k\in
K}(a^{\text{max}}_{k,\text{LCT}}-a^{\text{max}}_{k,\text{Baseline}})$) and the
P-value for the paired t-test.

###  6.4 LCT vs. SAM

Previous work has shown that Sharpness Aware Minimization (SAM) is effective
at optimizing imbalanced datasets in the multi-class setting (Foret et al.,
2021; Rangwani et al., 2022). SAM searches for parameters that lie in
neighborhoods of uniformly low loss and uses a hyperparameter $\rho$ to define
the size of these neighborhoods. Rangwani et al. (2022) shows that SAM can
effectively escape saddle points in imbalanced settings by using large values
for $\rho$ (i.e., $\rho=0.5,0.8$). We, however, find that using SAM in this
way does not translate to improved performance on a binary dataset. In Figure
7, we compare the performance of the models across several hyperparameter
values $\rho$. We find that, unlike the multi-class case, in the binary case
the best $\rho$ is quite small (i.e., $\rho=0.1$). While training with SAM and
$\rho=0.1$ can produce some very strong results, this also exacerbates the
amount of variance in the performance. LCT with SGD gives models that are
better on average and require less tuning. Note that larger $\rho$ values are
detrimental to training in this setting and lead to models that are
effectively naive classifiers. We also tried combining LCT and SAM with
$\rho=0.1$ and found this to be less effective than training with LCT and SGD.

Figure 7: Effects of using SAM optimization. Each boxplot contains results for
48 models trained with different hyperparameter values. Models were trained on
CIFAR10 cat/dog $\beta=100$. In the binary case, SAM fails with large $\rho$.
Although some models trained with SAM and small $\rho$ achieve strong
performance, SAM exacerbates the variance of the results.

###  6.5 Choice of $\lambda$

In this section, we compare different options for which hyperparameters in VS
loss to use as $\boldsymbol{\lambda}$ in LCT. We consider setting
$\boldsymbol{\lambda}$ to each of the three hyperparameters (using constant
values for the other two). We also consider setting
$\boldsymbol{\lambda}=(\Omega,\gamma,\tau)$. Figure 8 compares these options
and shows that $\boldsymbol{\lambda}=\tau$ achieves a good compromise between
improving the AUC of the models and reducing the variance of results. Appendix
F compares $\boldsymbol{\lambda}=\tau$ and
$\boldsymbol{\lambda}=(\Omega,\gamma,\tau)$ on the Melanoma dataset and finds
similar results. Furthermore, this choice of $\lambda$ follows our insights
from Section 4.

Figure 8: LCT with different $\boldsymbol{\lambda}$. Each boxplot contains
results for 27 models with different hyperparameter values. All models were
trained on CIFAR10 cat/dog with $\beta=100$.

##  7 Conclusion

In conclusion, we propose using Loss Conditional Training (LCT) to train
binary classification models under severe class imbalance. We find that this
consistently both improves the ROC curves and reduces the model’s sensitivity
to hyperparameter choices. This improvement comes from the fact that training
over a family of loss functions is a proxy for optimizing along different TPR
and FPR tradeoffs. Areas of future work include studying how to adapt this
method to work on multi-class classification problems under imbalance and
regression tasks.

## References

  * Alshammari et al. (2022) Alshammari, S., Wang, Y.-X., Ramanan, D., and Kong, S.  Long-tailed recognition via weight balancing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6897–6907, 2022.
  * Behnia et al. (2023) Behnia, T., Kini, G. R., Vakilian, V., and Thrampoulidis, C.  On the implicit geometry of cross-entropy parameterizations for label-imbalanced data.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 10815–10838. PMLR, 2023.
  * Bellinger et al. (2020) Bellinger, C., Corizzo, R., and Japkowicz, N.  ReMix: Calibrated resampling for class imbalance in deep learning.  _arXiv preprint arXiv:2012.02312_ , 2020.
  * Buda et al. (2018) Buda, M., Maki, A., and Mazurowski, M. A.  A systematic study of the class imbalance problem in convolutional neural networks.  _Neural Netw._ , 106:249–259, October 2018.
  * Cao et al. (2019) Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T.  Learning imbalanced datasets with label-distribution-aware margin loss.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Cassidy et al. (2022) Cassidy, B., Kendrick, C., Brodzicki, A., Jaworek-Korjakowska, J., and Yap, M. H.  Analysis of the ISIC image datasets: Usage, benchmarks and recommendations.  _Med. Image Anal._ , 75:102305, January 2022.
  * Chawla et al. (2011) Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P.  SMOTE: Synthetic minority over-sampling technique.  June 2011.
  * Cukierski (2013) Cukierski, W.  Dogs vs. cats, 2013.  URL https://kaggle.com/competitions/dogs-vs-cats.
  * Dong et al. (2022) Dong, B., Zhou, P., Yan, S., and Zuo, W.  Lpt: Long-tailed prompt tuning for image classification.  _arXiv preprint arXiv:2210.01033_ , 2022.
  * Dosovitskiy & Djolonga (2020) Dosovitskiy, A. and Djolonga, J.  You only train once: Loss-conditional training of deep networks.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=HyxY6JHKwr.
  * Du et al. (2023) Du, F., Yang, P., Jia, Q., Nan, F., Chen, X., and Yang, Y.  Global and local mixture consistency cumulative learning for long-tailed visual recognitions.  In _Conference on Computer Vision and Pattern Recognition 2023_ , 2023.  URL https://arxiv.org/abs/2305.08661.
  * Egan (1975) Egan, J. P.  _Signal Detection Theory and ROC Analysis. In Series in Cognition and Perception. Academic Press, New York_.  01 1975.
  * Fang et al. (2023) Fang, A., Kornblith, S., and Schmidt, L.  Does progress on ImageNet transfer to real-world datasets?  January 2023.
  * Feng et al. (2021) Feng, C., Zhong, Y., and Huang, W.  Exploring classification equilibrium in long-tailed object detection.  In _Proceedings of the IEEE/CVF International conference on computer vision_ , pp. 3417–3426, 2021.
  * Fernando & Tsokos (2021) Fernando, K. R. M. and Tsokos, C. P.  Dynamically weighted balanced loss: class imbalanced learning and confidence calibration of deep neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , 33(7):2940–2951, 2021.
  * Flach & Kull (2015) Flach, P. A. and Kull, M.  Precision-Recall-Gain curves: PR analysis done right.  _Advances in Neural Information Processing Systems_ , 28:838–846, 2015.
  * Foret et al. (2021) Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B.  Sharpness-aware minimization for efficiently improving generalization.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=6Tm1mposlrM.
  * Ghiasi et al. (2020) Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.-Y., Cubuk, E. D., Le, Q. V., and Zoph, B.  Simple Copy-Paste is a strong data augmentation method for instance segmentation.  December 2020.
  * Guo & Wang (2021) Guo, H. and Wang, S.  Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15089–15098, 2021.
  * Hong et al. (2021) Hong, M., Choi, J., and Kim, G.  StyleMix: Separating content and style for enhanced data augmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14862–14870, 2021.
  * Hou et al. (2023) Hou, C., Zhang, J., Wang, H., and Zhou, T.  Subclass-balancing contrastive learning for long-tailed recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 5395–5407, 2023.
  * Japkowicz (2000) Japkowicz, N.  The class imbalance problem: Significance and strategies.  In _In Proceedings of the 2000 International Conference on Artificial Intelligence (ICAI_ , pp. 111–117, 2000.
  * Jiang et al. (2023) Jiang, Z., Zhao, L., Lu, Y., Zhan, Y., and Mao, Q.  A semi-supervised resampling method for class-imbalanced learning.  _Expert Systems with Applications_ , 221:119733, 2023.
  * Johnson & Khoshgoftaar (2019) Johnson, J. M. and Khoshgoftaar, T. M.  Survey on deep learning with class imbalance.  _Journal of Big Data_ , 6(1):1–54, March 2019.
  * Kang et al. (2021) Kang, H., Vu, T., and Yoo, C. D.  Learning imbalanced datasets with maximum margin loss.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , pp. 1269–1273. IEEE, 2021.
  * Kim et al. (2020) Kim, J.-H., Choo, W., and Song, H. O.  Puzzle mix: Exploiting saliency and local statistics for optimal mixup.  In _International Conference on Machine Learning_ , pp. 5275–5285. PMLR, 2020.
  * Kini et al. (2021) Kini, G. R., Paraskevas, O., Oymak, S., and Thrampoulidis, C.  Label-imbalanced and group-sensitive classification under overparameterization.  In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=UZm2IQhgIyB.
  * Li et al. (2022a) Li, J., Tan, Z., Wan, J., Lei, Z., and Guo, G.  Nested collaborative learning for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6949–6958, 2022a.
  * Li et al. (2021a) Li, M., Zhang, X., Thrampoulidis, C., Chen, J., and Oymak, S.  Autobalance: Optimized loss functions for imbalanced data.  _Advances in Neural Information Processing Systems_ , 34:3163–3177, 2021a.
  * Li et al. (2022b) Li, M., Cheung, Y.-M., and Jiang, J.  Feature-balanced loss for long-tailed visual recognition.  In _2022 IEEE International Conference on Multimedia and Expo (ICME)_ , pp. 1–6. IEEE, 2022b.
  * Li et al. (2022c) Li, M., Cheung, Y.-m., and Lu, Y.  Long-tailed visual recognition via gaussian clouded logit adjustment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6929–6938, 2022c.
  * Li et al. (2021b) Li, S., Gong, K., Liu, C. H., Wang, Y., Qiao, F., and Cheng, X.  Metasaug: Meta semantic augmentation for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 5212–5221, 2021b.
  * Li et al. (2020) Li, Y., Wang, T., Kang, B., Tang, S., Wang, C., Li, J., and Feng, J.  Overcoming classifier imbalance for long-tail object detection with balanced group softmax.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10991–11000, 2020.
  * Lin et al. (2017) Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P.  Focal loss for dense object detection.  August 2017.
  * Liu et al. (2009) Liu, X.-Y., Wu, J., and Zhou, Z.-H.  Exploratory undersampling for class-imbalance learning.  _IEEE Trans. Syst. Man Cybern. B Cybern._ , 39(2):539–550, April 2009.
  * Menon et al. (2021) Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S.  Long-tail learning via logit adjustment.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=37nvvqkCo5.
  * Park et al. (2021) Park, S., Lim, J., Jeon, Y., and Choi, J. Y.  Influence-balanced loss for imbalanced visual classification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 735–744, 2021.
  * Perez et al. (2018) Perez, E., Strub, F., de Vries, H., Dumoulin, V., and Courville, A.  Film: Visual reasoning with a general conditioning layer.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), Apr. 2018.
  * Rangwani et al. (2022) Rangwani, H., Aithal, S. K., Mishra, M., and Radhakrishnan, V. B.  Escaping saddle points for effective generalization on class-imbalanced data.  In _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=9DYKrsFSU2.
  * Ren et al. (2020) Ren, J., Yu, C., Ma, X., Zhao, H., Yi, S., et al.  Balanced meta-softmax for long-tailed visual recognition.  _Advances in neural information processing systems_ , 33:4175–4186, 2020.
  * Ren et al. (2022) Ren, J., Zhang, M., Yu, C., and Liu, Z.  Balanced mse for imbalanced visual regression.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7926–7935, 2022.
  * Rotemberg et al. (2020) Rotemberg, V., Kurtansky, N., Betz-Stablein, B., Caffery, L., Chousakos, E., Codella, N., Combalia, M., Dusza, S., Guitera, P., Gutman, D., Halpern, A., Kittler, H., Kose, K., Langer, S., Lioprys, K., Malvehy, J., Musthaq, S., Nanda, J., Reiter, O., Shih, G., Stratigos, A., Tschandl, P., Weber, J., and Soyer, H. P.  A patient-centric dataset of images and metadata for identifying melanomas using clinical context, 2020.
  * Shwartz-Ziv et al. (2023) Shwartz-Ziv, R., Goldblum, M., Li, Y. L., Bruss, C. B., and Wilson, A. G.  Simplifying neural network training under class imbalance.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=iGmDQn4CRj.
  * Tan et al. (2021) Tan, J., Lu, X., Zhang, G., Yin, C., and Li, Q.  Equalization loss v2: A new gradient balance approach for long-tailed object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 1685–1694, 2021.
  * Tang et al. (2022) Tang, K., Tao, M., Qi, J., Liu, Z., and Zhang, H.  Invariant feature learning for generalized long-tailed classification.  In _European Conference on Computer Vision_ , pp. 709–726. Springer, 2022.
  * Wang et al. (2021a) Wang, J., Zhang, W., Zang, Y., Cao, Y., Pang, J., Gong, T., Chen, K., Liu, Z., Loy, C. C., and Lin, D.  Seesaw loss for long-tailed instance segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9695–9704, 2021a.
  * Wang et al. (2021b) Wang, P., Han, K., Wei, X.-S., Zhang, L., and Wang, L.  Contrastive learning based hybrid networks for long-tailed image classification.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 943–952, 2021b.
  * Wang et al. (2016) Wang, S., Liu, W., Wu, J., Cao, L., Meng, Q., and Kennedy, P. J.  Training deep neural networks on imbalanced data sets.  In _2016 International Joint Conference on Neural Networks (IJCNN)_ , pp. 4368–4374, 2016.  doi: 10.1109/IJCNN.2016.7727770.
  * Wang et al. (2020a) Wang, T., Li, Y., Kang, B., Li, J., Liew, J., Tang, S., Hoi, S., and Feng, J.  The devil is in classification: A simple framework for long-tail instance segmentation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16_ , pp. 728–744. Springer, 2020a.
  * Wang et al. (2021c) Wang, T., Zhu, Y., Zhao, C., Zeng, W., Wang, J., and Tang, M.  Adaptive class suppression loss for long-tail object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 3103–3112, 2021c.
  * Wang et al. (2020b) Wang, X., Lian, L., Miao, Z., Liu, Z., and Yu, S. X.  Long-tailed recognition by routing diverse distribution-aware experts.  _arXiv preprint arXiv:2010.01809_ , 2020b.
  * Wang et al. (2019) Wang, Y., Gan, W., Yang, J., Wu, W., and Yan, J.  Dynamic curriculum learning for imbalanced data classification.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 5017–5026, 2019.
  * Xie & Manski (1989) Xie, Y. and Manski, C. F.  The logit model and Response-Based samples.  _Sociol. Methods Res._ , 17(3):283–302, 1989.
  * Ye et al. (2020) Ye, H.-J., Chen, H.-Y., Zhan, D.-C., and Chao, W.-L.  Identifying and compensating for feature deviation in imbalanced deep learning.  January 2020.
  * Yun et al. (2019) Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 6023–6032, 2019.
  * Zada et al. (2022) Zada, S., Benou, I., and Irani, M.  Pure noise to the rescue of insufficient data: Improving imbalanced classification by training on random noise images.  In _International Conference on Machine Learning_ , pp. 25817–25833. PMLR, 2022.
  * Zang et al. (2021) Zang, Y., Huang, C., and Loy, C. C.  Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3457–3466, 2021.
  * Zawacki et al. (2020) Zawacki, A., Helba, B., George Shih, J. W., Elliott, J., Combalia, M., Kurtansky, N., NoelCodella, Culliton, P., and Rotemberg, V.  Siim-isic melanoma classification, 2020.  URL https://kaggle.com/competitions/siim-isic-melanoma-classification.
  * Zhang et al. (2018) Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=r1Ddp1-Rb.
  * Zhang et al. (2021) Zhang, S., Li, Z., Yan, S., He, X., and Sun, J.  Distribution alignment: A unified framework for long-tail visual recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 2361–2370, 2021.
  * Zhang et al. (2022) Zhang, Y., Hooi, B., Hong, L., and Feng, J.  Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition.  _Advances in Neural Information Processing Systems_ , 35:34077–34090, 2022.
  * Zhang et al. (2023) Zhang, Y., Kang, B., Hooi, B., Yan, S., and Feng, J.  Deep long-tailed learning: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Zhao et al. (2020) Zhao, Y., Chen, J., and Oymak, S.  On the role of dataset quality and heterogeneity in model confidence.  _arXiv preprint arXiv:2002.09831_ , 2020.
  * Zhong et al. (2021) Zhong, Z., Cui, J., Liu, S., and Jia, J.  Improving calibration for long-tailed recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 16489–16498, 2021.
  * Zhou et al. (2020) Zhou, B., Cui, Q., Wei, X.-S., and Chen, Z.-M.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9719–9728, 2020.
  * Zhu et al. (2022) Zhu, J., Wang, Z., Chen, J., Chen, Y.-P. P., and Jiang, Y.-G.  Balanced contrastive learning for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6908–6917, 2022.

##  Appendix A Binary classification metrics

In the binary case, we assume that the minority class is the positive class
(i.e., the class with label a label of one). For a given classifier, we can
categorize the samples in terms of their actual labels and the classifier’s
predictions as shown in Table 2. The remainder of this section defines several
metrics used for binary classification.

| Predicted Positive | Predicted Negative
---|---|---
Actually Positive |  green!20 True Positives (TP) |  red!20 False Negatives (FN)
Actually Negative |  orange!20 False Positives (FP) |  blue!20 True Negatives (TN)
Table 2: Categorization of samples in the binary case based on their actual
labels (rows) and predicted labels (columns).

###  A.1 True Positive Rate (TPR) = Minority-class accuracy = Recall

The true positive rate (TPR) is defined as the proportion of actual positive
samples which are predicted to be positive. Note that in the binary case, this
is equivalent to both the minority-class accuracy and the recall.

$\displaystyle\text{TPR}=\text{Minority-class
acc.}=\text{Recall}=\frac{TP}{TP+FN}$ (9)

| Predicted Positive | Predicted Negative
---|---|---
Actually Positive |  blue!40 True Positives (TP) |  blue!20 False Negatives (FN)
Actually Negative | False Positives (FP) | True Negatives (TN)
Table 3: Visualization of TPR calculation. All shaded cells are included in
calculation of metric. The numerator is highlighted in dark blue.

###  A.2 False Positive Rate (FPR) = 1 - Majority-class accuracy = 1 - TNR

The false positive rate (FPR) is defined as the proportion of actual negative
samples which are predicted to be positive. Note that in the binary case, this
is equivalent to 1 - the majority-class and 1 - the true negative rate (TNR).

$\displaystyle\text{FPR}=1-\text{Majority-class acc.}==\frac{FP}{TN+FP}$ (10)

| Predicted Positive | Predicted Negative
---|---|---
Actually Positive | True Positives (TP) | False Negatives (FN)
Actually Negative |  blue!40 False Positives (FP) |  blue!20 True Negatives (TN)
Table 4: Visualization of FPR metric. All shaded cells are included in
calculation of metric. The numerator is highlighted in dark blue.

###  A.3 Precision

The precision is defined as the proportion of predicted positive samples which
are actually positive.

$\displaystyle\text{Precision}=\frac{TP}{TP+FP}$ (11)

| Predicted Positive | Predicted Negative
---|---|---
Actually Positive |  blue!40 True Positives (TP) | False Negatives (FN)
Actually Negative |  blue!20 False Positives (FP) | True Negatives (TN)
Table 5: Visualization of precision metric. All shaded cells are included in
calculation of metric. The numerator is highlighted in dark blue.

###  A.4 Overall Accuracy

Perhaps the simplest metric is the overall accuracy of the classifier. This is
simply the proportion of samples which are correctly classified (regardless of
their class). If the test set is imbalanced, a trivial classifier which
predicts all samples as negative will achieve a high overall accuracy.
Specifically, the overall accuracy of this classifier will be the proportion
of negative samples or $\frac{\beta}{1+\beta}$. In class imbalance literature,
the overall accuracy is often reported on a balanced test set. In this case,
the accuracy is an average accuracy on the positive and negative classes.

$\displaystyle\text{Overall accuracy}=\frac{TP+TN}{TP+FN+FP+TN}$ (12)

| Predicted Positive | Predicted Negative
---|---|---
Actually Positive |  blue!40 True Positives (TP) |  blue!20 False Negatives (FN)
Actually Negative |  blue!20 False Positives (FP) |  blue!40 True Negatives (TN)
Table 6: Visualization of overall accuracy metric. All shaded cells are
included in calculation of metric. The numerator is highlighted in dark blue.

###  A.5 $F_{1}$ and $F_{\beta}$

In some problems, such as information retrieval, there is only one class of
interest (the positive class) and the true negatives can vastly outnumber the
other three categories. In this case, a method’s effectiveness is determined
by 1) how many positive samples it correctly predicted as positive (i.e., the
recall) and 2) how many samples are actually positive out of all the samples
it predicted as positive (i.e., the precision). The $F_{1}$ metric measures
how well a method can achieve both of these goals simultaneously.
Specifically, the $F_{1}$ measure is the harmonic mean between the precision
and recall and is defined as

$\displaystyle F_{1}$
$\displaystyle=\frac{2\cdot\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}}.$
(13)

The $F_{1}$ measure assumes that the precision and recall have equal weights;
however, sometimes problems have different costs for recall and precision.
These asymmetric costs can be addressed by the more general $F_{\beta}$
metric. Let $\beta$ be the ratio of importance between recall and precision,
then $F_{\beta}$ is defined as 666Note that this $\beta$ differs from the
$\beta$ which we defined in the main body.,

$\displaystyle F_{\beta}$
$\displaystyle=\frac{(1+\beta^{2})\cdot\text{precision}\cdot\text{recall}}{\beta^{2}\cdot\text{precision}+\text{recall}}.$
(14)

| Predicted Positive | Predicted Negative
---|---|---
Actually Positive |  blue!40 True Positives (TP) |  blue!20 False Negatives (FN)
Actually Negative |  blue!20 False Positives (FP) | True Negatives (TN)
Table 7: Visualization of $F_{\beta}$ metric. All shaded cells are included in
calculation of metric. The TP cell is highlighted in dark blue because it is
included in both the precision and recall calculation.

###  A.6 G-mean

The Geometric mean (G-mean or GM) is the geometric mean of the TPR (i.e.,
sensitivity) and TNR (i.e., specificity) and is defined as follows,

$\displaystyle GM$ $\displaystyle=\sqrt{\text{SE}*\text{SP}}$ (15)
$\displaystyle=\sqrt{\frac{\text{TP}}{\text{TP}+\text{FN}}*\frac{\text{TN}}{\text{TN}+\text{FP}}}.$
(16)

| Predicted Positive | Predicted Negative
---|---|---
Actually Positive |  blue!40 True Positives (TP) |  blue!20 False Negatives (FN)
Actually Negative |  blue!20 False Positives (FP) |  blue!40 True Negatives (TN)
Table 8: Visualization of G-mean metric. All shaded cells are included in
calculation of metric. Values in the numerator are highlighted in dark blue.

###  A.7 ROC curves and AUC

Of course, the number of true positives and true negatives are a trade-off and
any method can be modified to give a different combination of these metrics.
Specifically, the decision threshold can be modified to give any particular
recall. Receiver Operating Characteristic (ROC) curves take this in
consideration and show the trade-off of true positive rates and false positive
rates over all possible decision thresholds. The area under the ROC curve (ROC
AUC) is calculated using the trapezoid rule for integration over a sample of
values and is a commonly used metric under class imbalance (Buda et al.,
2018).

##  Appendix B Detailed related work

Training on imbalanced datasets with algorithms designed to work on balanced
datasets can be problematic because the gradients and losses are biased
towards the common classes, so the rare classes will not be learned well.
Current methods to mitigate the effects of training under imbalance include
methods that modify the loss functions, re-sample and augment training
samples, and improve the module via two-stage learning, ensembles, or
representation learning (Zhang et al., 2023).

###  B.1 Specialized loss functions

To balance the gradients from all classes, several papers adaptively change a
sample’s weight in the loss based on features such as the sample’s confidence
score, class frequency, and influence on model weights (Zhang et al., 2021;
Fernando & Tsokos, 2021; Park et al., 2021; Wang et al., 2021a; Li et al.,
2021a). Other work addresses the difference in the norms of features
associated with frequent (head) and rare (tail) classes and proposes to
balance this by utilizing a feature-based loss (Li et al., 2022b) or weight-
decay and gradient clipping (Alshammari et al., 2022).

Some work has focused on enforcing larger margins on the tail classes using
additive factors on the logits in Cross-entropy loss (Cao et al., 2019; Menon
et al., 2021; Li et al., 2022c). Other work proposed adding multiplicative
factors to the logits to adjust for the difference in the magnitude of
minority-class logits at training and test time or minimize a margin-based
generalization bound (Ye et al., 2020; Kang et al., 2021). Kini et al. (2021)
show that multiplicative factors are essential for the terminal phase of
training, but that these have negative effects early during training, so
additive factors are necessary to speed up convergence. They propose Vector
Scaling (VS) loss as a general loss function that includes both additive and
multiplicative factors on the logits. Behnia et al. (2023) study the implicit
geometry of classifiers trained on a special case of VS loss.

Beyond classification, many papers have focused on imbalance in instance
segmentation and object detection applications (Tan et al., 2021; Wang et al.,
2021c; Feng et al., 2021; Li et al., 2020). Ren et al. (2022) also propose a
balanced Mean Square Error (MSE) loss for regression problems, such as age and
depth estimation.

###  B.2 Data-level methods

Another way to balance the gradients of classes during training is resampling.
This could be done by sampling the minority class more often (random over-
sampling) or sampling the majority class less often (random under-sampling)
(Johnson & Khoshgoftaar, 2019; Japkowicz, 2000; Liu et al., 2009). Jiang et
al. (2023); Hou et al. (2023) use clustering to drive resampling;
specifically, they cluster head classes into multiple clusters and then
resample across clusters instead of classes. Meta-learning has been used to
estimate the optimal sampling rates of different classes (Ren et al., 2020),
while Wang et al. (2019) dynamically adapts both the loss function and
sampling procedure throughout training.

Additionally, data augmentation can be used alongside oversampling to increase
the size of the minority class samples and enable model generalization (Zhong
et al., 2021; Zang et al., 2021; Li et al., 2021b; Du et al., 2023). With
tabular data, small perturbations of random noise can be added to generate new
examples. Images lend themselves to more high-level augmentations. Methods
that copy and paste patches of images, such as CutMix (Yun et al., 2019) and
PuzzleMix (Kim et al., 2020) have been used to improve classification or
instance segmentation (Ghiasi et al., 2020) performance.

SMOTE (Synthetic Minority Over-sampling Technique) creates synthetic examples
by interpolating between samples in the same class of the training set (Chawla
et al., 2011). The interpolation is done in feature space instead of data
space. The mixup method also generates synthetic examples; however, unlike
SMOTE, it interpolates between samples of different classes (Zhang et al.,
2018). These synthetic examples are given a soft label that corresponds to the
proportion of input from each class. StyleMix adapts mixup to separately
manipulate an image’s content and style (Hong et al., 2021). ReMix advances
mixup to optimize for situations with class imbalance by combining mixup and
resampling (Bellinger et al., 2020). Li et al. (2021b) creates augmented data
samples by translating deep features along semantically meaningful directions.
Zada et al. (2022) adds pure noise images to the training set and introduces a
new type of distribution-aware batch normalization layer to facilitate
training with them.

###  B.3 Module improvements

Some work has shown that overly emphasizing the minority class can disrupt the
original data distribution and cause overfitting (Zhou et al., 2020; Du et
al., 2023) or reduce the accuracy of the majority class (Zhu et al., 2022). To
mitigate this issue, contrastive learning (Du et al., 2023; Zhu et al., 2022;
Wang et al., 2021b) has been used to learn a robust feature representation by
enforcing consistency between two differently augmented versions of the data.
Du et al. (2023) devise a single-stage approach, by combining the ideas of
contrastive representations with soft label class re-weighting to achieve
state-of-the-art performance on several benchmarks.

Multi-expert methods have also been explored (Zhang et al., 2022; Wang et al.,
2020b; Li et al., 2022a). Li et al. (2022a) collaboratively learn multiple
experts, by learning individual experts and transferring knowledge among them,
in a nested way. Methods have also used two branches or heads with one
classification branch along with another branch, either to re-balance (Guo &
Wang, 2021; Zhou et al., 2020) or calibrate (Wang et al., 2020a) the model.
Tang et al. (2022) adopt a generalized approach to learn attribute-invariant
features that first discovers sets of samples with diverse intra-class
distributions from the low confidence predictions, and then learns invariant
features across them. Dong et al. (2022) uses language prompts to improve both
general and group-specific features.

##  Appendix C Number of samples

| Train set | Test set
---|---|---
Dataset | # maj. | # min. | # min. | # min. | # maj. | # min.
($\beta=10$) |  ($\beta=100$) |  ($\beta=200$) |  |
CIFAR10 pair | 5,000 | 500 | 50 | 25 | 1,000 | 1,000
Household | 2,500 | 250 | 25 | 13 | 500 | 500
Dogs vs. Cats | 11,500 | 1,500 | 150 | 75 | 1,000 | 1,000
SIIM-ISIC Melanoma | 41,051 | 4,071* | 410 | 205 | 12,300 | 1,035
Table 9: Number of samples in majority class and minority class of the
datasets at different imbalance ratios $\beta$. $\beta$ only affects the
number of minority-class samples during training. *SIIM-ISIC Melanoma has an
imbalance ratio of 12 by default, so we study $\beta=12$ instead of $\beta=10$
in that case.

##  Appendix D Effect of evaluation $\lambda$

Figure 9: Effect of $\boldsymbol{\lambda}$ value at inference time. Results
are shown for one LCT model that has $\boldsymbol{\lambda}=\tau$. Model was
trained using the protocol described in Section 6 with
$\boldsymbol{\lambda}=\tau$ drawn from a linear distribution on $[0,3]$ with
$h_{b}=0$. Model is evaluated at four $\boldsymbol{\lambda}=\tau$ values:
$0.0,1.0,2.0$, and $3.0$. For each evaluation $\boldsymbol{\lambda}$, we plot
a) the model’s TPR/FPR when $t=0.5$ (circle) and b) the ROC curve of the model
(curve). The evaluation $\boldsymbol{\lambda}$ affects the TPR/FPR for a
constant $t$ (e.g., $t=0.5$); however, the ROC curves of models evaluated on
different $\boldsymbol{\lambda}$s are almost identical.

##  Appendix E Linear probability density function

We use a linear probability distribution to sample $\lambda$ from an interval
$[a,b]$. Unlike the triangular distribution, the probability distribution
function (PDF) of this distribution can be nonzero at both a and b. Figure 10
shows several examples of this distribution when $[a,b]=[0,3.0]$.

To implement this distribution, the user first selects the domain $[a,b]$ and
the height of PDF at $b$, $h_{b}$. The function then calculates $h_{a}$ so
that the area under the PDF equals one. To sample from this distribution, we
draw from a uniform(0,1) distribution and use the inverse cumulative
distribution function (CDF) of the linear distribution to find a value of
$\lambda$.

Note that this is the uniform distribution on $[a,b]$ when $h_{a}=h_{b}$.
Additionally, this is a triangular distribution when $h_{a}=0$ or $h_{b}=0$.

Figure 10: Example PDFs and CDFs of linear distribution with different $h_{b}$
when $[a,b]=[0,3]$.

##  Appendix F Using $\lambda=(\Omega,\gamma,\tau)$ on the SSIM Melanoma
dataset

Figure 11: Comparison of baseline, LCT with $\boldsymbol{\lambda}=\tau$, and
LCT with $\boldsymbol{\lambda}=(\Omega,\gamma,\tau$ on the Melanoma dataset
with $\beta=100.$ The two LCT methods are comparable with
$\boldsymbol{\lambda}=\tau$ having a slight edge over
$\boldsymbol{\lambda}=(\Omega,\gamma,\tau$ in terms of the best model at high
TPRs.

##  Appendix G Simplifying VS Loss

Recall the definition of VS loss.

$\displaystyle\ell_{VS}(y,\mathbf{z})=-\omega_{y}\log\left(\frac{e^{\Delta_{y}z_{y}+\iota_{y}}}{\sum_{c\in\mathcal{Y}}e^{\Delta_{c}z_{c}+\iota_{c}}}\right)$
(17)

Additionally, recall that $\Delta$ and $\iota$ are parameterized by $\gamma$
and $\tau$ as follows.

$\displaystyle\Delta_{c}$
$\displaystyle=\left(\frac{n_{c}}{n_{max}}\right)^{\gamma}$ (18)
$\displaystyle\iota_{c}$
$\displaystyle=\tau\log\left(\frac{n_{c}}{\sum_{c^{\prime}\in
C}n_{c^{\prime}}}\right)$ (19)

Consider a binary problem with an imbalance ratio $\beta$. Then $\Delta$’s and
$\iota$’s are defined as follows

$\displaystyle\Delta_{0}$ $\displaystyle=1$
$\displaystyle\iota_{0}=\tau\log\left(\frac{\beta}{\beta+1}\right)$ (20)
$\displaystyle\Delta_{1}$ $\displaystyle=\frac{1}{\beta^{\gamma}}$
$\displaystyle\iota_{1}=\tau\log\left(\frac{1}{\beta+1}\right)$ (21)

We can simplify $\ell_{VS}(0,\mathbf{z})$ as follows. First plug in the
$\Delta$’s and $\iota$’s:

$\displaystyle\ell_{VS}(0,\mathbf{z})$
$\displaystyle=-(1-\Omega)\log\left(\frac{e^{z_{0}+\tau\log(\frac{\beta}{\beta+1})}}{e^{z_{0}+\tau\log(\frac{\beta}{\beta+1})}+e^{\frac{z_{1}}{\beta^{\gamma}}+\tau\log(\frac{1}{\beta+1})}}\right)$
(23)

Then rewrite $\tau\log(\frac{a}{b})$ as $\tau\log(a)-\tau\log(b)$ and cancel
out $e^{\tau\log(\beta+1)}$ from the numerator and denominator.

$\displaystyle\ell_{VS}(0,\mathbf{z})$
$\displaystyle=-(1-\Omega)\log\left(\frac{e^{z_{0}+\tau\log(\beta)-\tau\log(\beta+1)}}{e^{z_{0}+\tau\log(\beta)-\tau\log(\beta+1)}+e^{\frac{z_{1}}{\beta^{\gamma}}+\tau\log(1)-\tau\log(\beta+1)}}\right)$
(24)
$\displaystyle=-(1-\Omega)\log\left(\frac{\frac{e^{z_{0}+\tau\log(\beta)}}{e^{\tau\log(\beta+1)}}}{\frac{e^{z_{0}+\tau\log(\beta)}}{e^{\tau\log(\beta+1)}}+\frac{e^{z_{1}/\beta^{\gamma}}}{e^{\tau\log(\beta+1)}}}\right)$
(25)
$\displaystyle=-(1-\Omega)\log\left(\frac{e^{z_{0}+\tau\log(\beta)}}{e^{z_{0}+\tau\log(\beta)}+e^{z_{1}/\beta^{\gamma}}}\right)$
(26)

Then use the following two facts to a) simplify the term inside the log and b)
rewrite the log term:

$\displaystyle\text{a)
}\frac{e^{a}}{e^{a}+e^{b}}=\frac{e^{a}}{e^{a}+e^{b}}\frac{\frac{1}{e^{a}}}{\frac{1}{e^{a}}}=\frac{1}{1+e^{b-a}}$
(27) $\displaystyle\text{b)
}\log\left(\frac{1}{1+e^{b-a}}\right)=\log(1)-\log(1+e^{b-a})=-\log(1+e^{b-a})$
(28)

$\displaystyle\ell_{VS}(0,\mathbf{z})$
$\displaystyle=(1-\Omega)\log\left(1+e^{z_{1}/\beta^{\gamma}-z_{0}-\tau\log(\beta)}\right)$
(29)

Finally, we move the $\tau\log(\beta)$ term outside of the exponent

$\displaystyle\ell_{VS}(0,\mathbf{z})$
$\displaystyle=(1-\Omega)\log\left(1+\frac{1}{\beta^{\tau}}e^{z_{1}/\beta^{\gamma}-z_{0}}\right).$
(30)

We follow a similar process for $\ell_{VS}(1,\mathbf{z})$ and get

$\displaystyle\ell_{VS}(1,\mathbf{z})$
$\displaystyle=\Omega\log\left(1+\beta^{\tau}e^{z_{0}-z_{1}/\beta^{\gamma}}\right).$
(31)

##  Appendix H Partial derivatives of VS-Loss with respect to
$\Omega,\tau,\gamma$

Then the partial derivatives of the loss when $y=1$ with respect to
$\Omega,\gamma$ and $\tau$ are as follows.

$\displaystyle\frac{\partial}{\partial\Omega}\ell_{VS}(1,\mathbf{z})=\log\left(1+\beta^{\tau}e^{z_{0}-z_{1}/\beta^{\gamma}}\right)$
(32)
$\displaystyle\frac{\partial}{\partial\gamma}\ell_{VS}(1,\mathbf{z})=\frac{\Omega
z_{1}e^{z_{0}-z_{1}/\beta^{\gamma}}\beta^{\tau-\gamma}\log(\beta)}{1+\beta^{\tau}e^{z_{0}-z_{1}/\beta^{\gamma}}}$
(33)
$\displaystyle\frac{\partial}{\partial\tau}\ell_{VS}(1,\mathbf{z})=\frac{\Omega
e^{z_{0}-z_{1}/\beta^{\gamma}}\beta^{\tau}\log(\beta)}{1+\beta^{\tau}e^{z_{0}-z_{1}/\beta^{\gamma}}}$
(34)

In particular,

$\frac{\partial}{\partial\gamma}\ell_{VS}(1,\mathbf{z})=\frac{z_{1}}{\beta^{\gamma}}\frac{\partial}{\partial\tau}\ell_{VS}(1,\mathbf{z})$
(35)

and for balanced classes

$\frac{\partial}{\partial\gamma}\ell_{VS}(1,\mathbf{z})=z_{1}\frac{\partial}{\partial\tau}\ell_{VS}(1,\mathbf{z})\;.$

##  Appendix I Full derivation for break-even points

With VS loss defined in Equations 5 and 6, denote
$\alpha=z_{1}/\beta^{\gamma}-(z_{0}+\tau\log\beta)$, setting
$\ell_{VS}(1,\mathbf{z})=\ell_{VS}(0,\mathbf{z})$ gives us

$\displaystyle(1+e^{-\alpha})^{\Omega}=(1+e^{\alpha})^{1-\Omega}.$ (36)

Consider the function
$f(\alpha)=(1+e^{-\alpha})^{\Omega}-(1+e^{\alpha})^{1-\Omega}$. Since
$0\leq\Omega\leq 1$, it is clear that $f(\alpha)$ is a continuous and
monotonically decreasing function. Also, $f(\alpha)\to+\infty$ as
$\alpha\to-\infty$, and $f(\alpha)\to-\infty$ as $\alpha\to+\infty$. These
imply that $f(\alpha)=0$ has exactly one unique solution, depended on
$\Omega$. Let $\alpha_{\Omega}$ be the solution such that
$f(\alpha_{\Omega})=0$.

Notice that when $\Omega=0.5$, the unique solution $\alpha_{\Omega}$ = 0. We
can also see that if $\Omega>0.5$, we have $f(0)=2^{\Omega}-2^{1-\Omega}>0$,
and since $f(\alpha)$ is monotonically decreasing, we know that the solution
$\alpha_{\Omega}$ should be strictly larger than 0. Otherwise, (i.e.,
$\Omega<0.5$), we have $\alpha_{\Omega}<0$.

Suppose we have found the $\alpha_{\Omega}$ such that $f(\alpha_{\Omega})=0$,
this means that the break-even point should have
$z_{1}/\beta^{\gamma}=z_{0}+\tau\log\beta+\alpha_{\Omega}$. Compared with the
regular cross-entropy break-even point, where $z_{1}=z_{0}$, we are adding
biases here by the terms $\beta^{-\gamma}$, $\tau\log\beta$, and
$\alpha_{\Omega}$, dependent on $\gamma,\tau,\Omega$, respectively. These
terms change the margin between $z_{0}$ and $z_{1}$, which poses a bias for
samples on the break-even point. Specifically, for a break-even point sample
(a sample that the trained model is totally confused with), the model still
prefers a positive output with $z_{1}>z_{0}$ if $\beta^{\gamma}>1$,
$\tau\log\beta>0$, and/or $\alpha_{\Omega}>0$, which translate to $\gamma>0$,
$\tau>0$, and/or $\Omega>0.5$, respectively.

###  I.1 Numerical Example

Since VS loss introduces a bias term $\tau\log(\beta)$ to $z_{0}$ during
training, the model will learn to consistently shrink its $z_{0}$ and/or
increase its $z_{1}$ outputs to match this bias. Specifically, in the case
$\Omega=0.5,\gamma=0$, but arbitrary $\tau$, the will re-calibrate to give
output $\mathbf{z}=(x,x+\tau\log(\beta))$ where $x$ is arbitrary to samples it
is completely unsure about. Consequently, the softmax output $p_{1}$
associated with the break-even points will be larger. For example, when
$\tau=0$ (regular cross-entropy), the softmax $p_{1}$ associated with the
break-even points is 0.5. When $\tau>0$, the softmax score associated with the
break-even points is $\frac{\beta^{\tau}}{1+\beta^{\tau}}$ (e.g., 0.99 when
$\beta=10$ and $\tau=2$).

##  Appendix J Contour plots of effect of $\Omega,\gamma,\tau$ on VS loss

In this section we consider the effect of hyperparameter values for
$\Omega,\gamma,\tau$ on the values of the loss functions over a set of
$\mathbf{z}$ for a binary problem. Specifically, we consider the difference of
the loss when the label is 1 and 0 (i.e.,
$\ell_{VS}(1,\mathbf{z})-\ell_{VS}(0,\mathbf{z})$). This gives insights into
how the model prioritizes different classes over the domain of $\mathbf{z}$.
We plot this for several combinations of hyperparameter values in Figure 12.
In particular, we are interested in the set of “break-even” points or the
points of $\mathbf{z}$ where $\ell_{VS}(1,\mathbf{z})=\ell_{VS}(0,\mathbf{z})$
because this gives insights into the thresholds which a model with that loss
function optimizes over.

Figure 12: Effect of hyperparameter values on
$\ell_{VS}(1,\mathbf{z})-\ell_{VS}(0,\mathbf{z})$ over $z_{0},z_{1}\in[-5,5]$
for $\beta=10$. Top, middle, and bottom rows show results for different values
of $\Omega$, $\gamma$, and $\tau$ respectively. A value of zero (white)
represents the “break even” points.

###  J.1 Regular cross-entropy

First consider regular cross-entropy loss (i.e.,
$\Omega=0.5,\gamma=0,\tau=0$). Here
$\ell_{VS}(1,\mathbf{z})=\ell_{VS}(0,\mathbf{z})$ when $z_{1}=z_{0}$. This is
shown on the top left column of Figure 12, where the line corresponding to
$z_{1}=z_{0}$ is white, indicating that the losses are equal along this line.
If $z_{1}>z_{0}$ (i.e., $\hat{y}=1$), then the difference of the losses is
negative (i.e., $\ell_{VS}(1,\mathbf{z})<\ell_{VS}(0,\mathbf{z})$). This makes
sense since the model should receive a bigger loss for an incorrect
prediction. Notice that this plot is symmetric along the line $z_{1}=z_{0}$.

###  J.2 Varying $\Omega$ (Weighted cross-entropy loss)

Next consider VS loss with arbitrary weights $\Omega$, but fixed
$\gamma=0,\tau=0$. Note that this is equivalent to weighted cross-entropy with
class 1 having weight $\Omega$ and class 0 having weight $1-\Omega$. We
include contour plots of this case in the top row of Figure 12 and simplify
the equation of equal losses below. Unfortunately there is no analytic
solution in this case, but we provide more insights about this case in
Subsection J.5.

$\displaystyle\ell_{VS}(1,\mathbf{z})$ $\displaystyle=\ell_{VS}(0,\mathbf{z})$
(37)
$\displaystyle-\Omega\log\left(\frac{e^{z_{1}}}{e^{z_{0}}+e^{z_{1}}}\right)$
$\displaystyle=-(1-\Omega)\log\left(\frac{e^{z_{0}}}{e^{z_{0}}+e^{z_{1}}}\right)$
(38)
$\displaystyle\left(\frac{e^{z_{1}}}{e^{z_{0}}+e^{z_{1}}}\right)^{\Omega}$
$\displaystyle=\left(\frac{e^{z_{0}}}{e^{z_{0}}+e^{z_{1}}}\right)^{1-\Omega}$
(39) $\displaystyle e^{\Omega z_{1}}$
$\displaystyle=e^{(1-\Omega)z_{0}}\left(e^{z_{0}}+e^{z_{1}}\right)^{2\Omega-1}$
(40) $\displaystyle z_{1}$
$\displaystyle=\left(\frac{1-\Omega}{\Omega}\right)z_{0}+\left(\frac{1}{\Omega}\right)\log(e^{z_{0}}+e^{z_{1}})^{2\Omega-1}$
(41)

###  J.3 Varying $\gamma$

Next consider VS loss with with arbitrary values of $\gamma$, but fixed
$\tau=0,\Omega=0.5$. Recall from the previous section that for a binary
problem with an imbalance ratio $\beta$, $\Delta_{0}=1$ and
$\Delta_{1}=1/\beta^{\gamma}$. Then

$\displaystyle\ell_{VS}(1,\mathbf{z})$ $\displaystyle=\ell_{VS}(0,\mathbf{z})$
(42)
$\displaystyle-0.5\log(\frac{e^{z_{1}/\beta^{\gamma}}}{e^{z_{0}}+e^{z_{1}/\beta^{\gamma}}})$
$\displaystyle=-0.5\log(\frac{e^{z_{0}}}{e^{z_{0}}+e^{z_{1}/\beta^{\gamma}}})$
(43) $\displaystyle z_{1}$ $\displaystyle=\beta^{\gamma}z_{0}$ (44)

Thus changing $\gamma$ equates to rotating the line of break-even points as
seen on the middle row of Figure 12.

###  J.4 Varying $\tau$

Finally consider VS loss with with arbitrary values of $\tau$, but fixed
$\gamma=0,\Omega=0.5$. Recall from the previous section that for a binary
problem with an imbalance ratio $\beta$,
$\iota_{0}=\tau\log\left(\frac{\beta}{1+\beta}\right)$ and
$\iota_{1}=\tau\log\left(\frac{1}{1+\beta}\right)$. Then

$\displaystyle\ell_{VS}(1,\mathbf{z})$ $\displaystyle=\ell_{VS}(0,\mathbf{z})$
(46)
$\displaystyle-0.5\log\left(\frac{e^{z_{1}+\tau\log\left(\frac{1}{1+\beta}\right)}}{e^{z_{0}+\tau\log\left(\frac{\beta}{1+\beta}\right)}+e^{z_{1}+\tau\log\left(\frac{1}{1+\beta}\right)}}\right)$
$\displaystyle=-0.5\log\left(\frac{e^{z_{0}+\tau\log\left(\frac{\beta}{1+\beta}\right)}}{e^{z_{0}+\tau\log\left(\frac{\beta}{1+\beta}\right)}+e^{z_{1}+\tau\log\left(\frac{1}{1+\beta}\right)}}\right)$
(47) $\displaystyle z_{1}+\tau\log\left(\frac{1}{1+\beta}\right)$
$\displaystyle=z_{0}+\tau\log\left(\frac{\beta}{1+\beta}\right)$ (48)
$\displaystyle z_{1}$
$\displaystyle=z_{0}+\tau\log\left(\frac{\frac{\beta}{1+\beta}}{\frac{1}{1+\beta}}\right)$
(49) $\displaystyle z_{1}$ $\displaystyle=z_{0}+\tau\log\left(\beta\right)$
(50)

Thus changing $\tau$ equates to shifting the line of break-even points as seen
on the bottom row of Figure 12.

###  J.5 $\Omega$ as a function of softmax scores

Recall that each model outputs a vector of logits $\mathbf{z}$ and that
$\hat{y}=\arg\max\mathbf{z}$. In the binary case, this is equivalent to
computing the softmax score for $z_{1}$ and thresholding this score at 0.5 as
shown below.

$\displaystyle p_{1}$
$\displaystyle=\frac{e^{\Delta_{1}z_{1}+\iota_{1}}}{e^{\Delta_{0}z_{0}+\iota_{0}}+e^{\Delta_{1}z_{1}+\iota_{1}}}$
(52) $\displaystyle\hat{y}$ $\displaystyle=\begin{cases}1&\text{if }p_{1}\geq
0.5\\\ 0&\text{otherwise}\end{cases}$ (53)

Because 0-1 loss is not differentiable, models are instead optimized for a
proxy function like cross-entropy. In traditional cross-entropy (i.e.,
$\Omega=0.5$), the loss function is symmetric such that
$\ell(0,p_{1})=\ell(1,-p_{1})$ which results in the DNN being optimal for a
threshold of 0.5 of $p_{1}$. Figure 13 shows how this threshold varies for
different values of $\Omega$. As $\Omega$ increases, the line $y=1$ becomes
steeper while $y=0$ becomes less steep and the value of $p_{1}$ where the
losses intersect increases. Note that the point of intersection is

$\displaystyle p_{1}^{\Omega}=(1-p_{1})^{1-\Omega}.$ (54)

Additionally, note that changing $\gamma$ or $\tau$ only affects the
calculation of $p_{1}$, but does not change the point $p_{1}$ where the losses
intersect.

Figure 13: Weighted cross entropy loss over different values of $\Omega$ and
softmax scores $p_{1}$, which correspond to class 1. Note that
$p_{0}=1-p_{1}$.