# An Unsupervised Cross-Modal Hashing Method Robust to Noisy Training Image-
Text Correspondences in Remote Sensing

###### Abstract

The development of accurate and scalable cross-modal image-text retrieval
methods, where queries from one modality (e.g., text) can be matched to
archive entries from another (e.g., remote sensing image) has attracted great
attention in remote sensing (RS). Most of the existing methods assume that a
reliable multi-modal training set with accurately matched text-image pairs is
existing. However, this assumption may not always hold since the multi-modal
training sets may include noisy pairs (i.e., textual descriptions/captions
associated to training images can be noisy), distorting the learning process
of the retrieval methods. To address this problem, we propose a novel
unsupervised cross-modal hashing method robust to the noisy image-text
correspondences (CHNR). CHNR consists of three modules: 1) feature extraction
module, which extracts feature representations of image-text pairs; 2) noise
detection module, which detects potential noisy correspondences; and 3)
hashing module that generates cross-modal binary hash codes. The proposed CHNR
includes two training phases: i) meta-learning phase that uses a small portion
of clean (i.e., reliable) data to train the noise detection module in an
adversarial fashion; and ii) the main training phase for which the trained
noise detection module is used to identify noisy correspondences while the
hashing module is trained on the noisy multi-modal training set. Experimental
results show that the proposed CHNR outperforms state-of-the-art methods.
††Our code is publicly available at https://git.tu-berlin.de/rsim/chnr

Index Terms—  cross-modal retrieval, unsupervised contrastive learning, remote
sensing, caption-noise.

##  1 Introduction

The fast-growing volume of multi-modal data (e.g., satellite images acquired
by different sensors and their textual descriptions) archives in remote
sensing (RS) has attracted great attention for the development of cross-modal
retrieval methods. Cross-modal retrieval (CMR), given a query from one
modality, aims at retrieving its counterpart from another modality. Among CMR
tasks in RS, the image–text retrieval is one of the most challenging tasks
because of the huge differences between the representations of RS image and
text modalities. The existing cross-modal image-text retrieval methods in RS
are defined based on supervised retrieval algorithms, which require the
availability of a multi-modal training set with accurately matched text-image
pairs. The quantity and the quality of the available image-text training pairs
are crucial for achieving accurate cross-modal retrieval. However, collecting
a sufficient number of reliable pairs is time-consuming and costly. Unlike RS,
in the computer vision (CV) community, unsupervised and self-supervised cross-
modal representation learning methods (which rely on the accurate matching
between the modalities) are widely studied [1, 2, 3, 4, 5, 6]. Su et al. [2]
introduce a deep joint-semantics reconstructing hashing (DJSRH) method to
learn binary codes that preserve the neighborhood structure in the original
data. To this end, DJSRH learns a mapping from different modalities into a
joint-semantics affinity matrix. The use of hashing allows mapping high-
dimensional feature vectors into compact binary hash codes, which are indexed
into a hash table that enables scalable search and retrieval. Liu et al. [3]
propose a joint-modal distribution-based similarity weighting (JDSH) method
based on DJSRH, exploiting an additional objective based on cross-modal
semantic similarities among samples. Unsupervised contrastive learning, which
aims to learn a metric space using the sample augmentations (different views
of the sample) is introduced in [5]. The feature space is learned in a way,
that views of one sample are pulled closer together and further from views of
other samples from the batch. The existing unsupervised contrastive learning
methods mainly rely on inter-modality contrastive objectives to obtain
consistent representations across different modalities, while the intra-modal
contrastive objectives are ignored. This may lead to learning an inefficient
embedding space, where the same semantic content can be mapped into different
points in the embedding space [7]. The success of the above-mentioned methods
also depends on the assumption that the multi-modal training data are
correctly matched between modalities (e.g., each training image is associated
with an accurate text sentence). However, manually collecting such accurate
training sets is costly and time-consuming, and the multi-modal
correspondences can be noisy (i.e., the text may not describe the
corresponding image content accurately), leading to a training set that
includes noisy correspondences. In detail, several factors can introduce noise
in the captions. For example, in automatic model-generated captions, noise can
occur due to noisy class labels assigned to the data. Manually-generated
captions through crowd-sourcing could be subject to noisy captions due to
human errors and/or subjectivity in describing the image content. In general,
noise in the captions can be due to: 1) the wrong (foil) words [8], where one
or several words in the caption may not be related to the image content; 2)
the missing words, when the caption does not represent all land-use and land-
cover class presented in the image; 3) the miscaptioning (wrong caption),
where the caption is semantically correct but does not correspond the image;
4) the redundancy [9, 10], where the image description contains too much
redundant information; 5) the typos and spell-check errors [11]. These issues
lead to the construction of a cross-modal training set with noisy image-text
correspondences, which may drastically reduce the CMR performance. To address
this problem, in this paper we introduce a novel unsupervised cross-modal
hashing method robust to the noisy image-text correspondences (CHNR). The
proposed CHNR: i) identifies noisy image-text pairs; ii) considers intra- and
inter-modal objectives for cross-modal representation learning; and iii)
allows high time-efficient search capability.

Fig. 1:  Block diagram of the proposed CHNR. In the feature extraction module
deep feature representations are extracted with the modality-specific encoders
$net_{img}$ and $net_{txt}$ for image and text modalities, respectively. The
discriminator $D_{N}$ of the noise detection module detects potential noisy
correspondences. The hashing module learns two hash functions $f$ and $g$ from
the input embedding.

##  2 Proposed Method

Let $\textbf{O}=\\{\textbf{X},\textbf{Y}\\}^{N}$ be a multi-modal training set
of $N$ image-text pairs, where $\textbf{X}=\\{\textrm{x}_{m}\\}_{m=1}^{N}$ and
$\textbf{Y}=\\{\textrm{y}_{m}\\}_{m=1}^{N}$ are associated to image and text
modalities, respectively. $\textrm{x}_{m}\in\mathbb{R}^{d_{i}}$ and
$\textrm{y}_{m}\in\mathbb{R}^{d_{t}}$ are image and text feature vectors,
respectively. $d_{i}$ and $d_{t}$ denote the size of image and text feature
dimensions. We assume that the training image-text pairs can be noisy, in
which an unknown number of pairs are mismatched, but a small subset
$\textbf{O}_{C}\in\textbf{O}$ of clean image-text pairs is available in the
training set as $\textbf{O}_{C}=\\{\textbf{X}_{C},\textbf{Y}_{C}\\}^{N_{C}}$,
where $\textbf{X}_{C}\in\textbf{X}$, $\textbf{Y}_{C}\in\textbf{Y}$ and
$N_{C}<N$. To reduce the adverse effect of the noisy correspondences, we
propose CHNR that aims at learning a noise discriminator $D_{N}$ and two hash
functions $f$ and $g$ for image and text modalities, respectively. Using the
clean subset $\textbf{O}_{C}$, the noise discriminator $D_{N}$ learns to
identify clean and noisy in the joint features
$\textbf{Z}=\\{\textrm{z}_{m}\\}_{m=1}^{N}$. $D_{N}$ assigns a noise
likelihood score (i.e., weight) to each pair
$\textbf{W}=\\{\textrm{w}_{m}\\}_{m=1}^{N}$, where,
$\textrm{z}_{m}=concat(\textrm{x}_{m},\textrm{y}_{m})$ and
$\textrm{w}_{m}=D_{N}(\textrm{x}_{m},\theta_{D_{N}})$, for which $concat(.)$
is a vector concatenation and $\theta_{D_{N}}$ are parameters of $D_{N}$.
Joint-features and corresponding weights of clean subset $\textbf{O}_{C}$, are
denoted as $\textbf{Z}_{C}$ and $\textbf{W}_{C}$, respectively. Using the
training set $\textbf{O}_{C}$, hash functions $f$ and $g$ learn to generate
binary hash codes $\textbf{B}_{i}=f(\textbf{X},\theta_{i})$ and
$\textbf{B}_{t}=g(\textbf{Y},\theta_{t})$, where
$\textbf{B}_{i}\in\\{0,1\\}^{N\times B}$ and
$\textbf{B}_{t}\in\\{0,1\\}^{N\times B}$ for image and text modalities,
respectively. $\theta_{i}$, $\theta_{t}$ are parameters of image and text
hashing networks and $B$ is the length of binary hash code. To learn the hash
functions $f(.)$ and $g(.)$ and noise discriminator $D_{N}$, the proposed CHNR
includes three main modules: i) the feature extraction module that produces
feature representations for image and text modalities; ii) the noise detection
module that aims to detect semantically incoherent feature pairs; and iii) the
hashing module that generates binary representations. The block diagram of the
CHNR is shown in Fig. 1. The training process of the proposed CHNR is
conducted in two phases: i) a meta-learning phase, where the noise
discriminator and the hashing module are trained on the clean subset
$\textbf{O}_{C}$ only; ii) the main training phase, where the weights of
$D_{N}$ are frozen and it is used to detect noisy correspondences while the
hashing module is trained on the training set O.

###  2.1 Feature extraction module

This module generates deep semantic representations for both image and text
modalities, and feed them into the noise detection and hashing modules. The
feature extraction module includes two modality-specific encoder networks: 1)
an image encoder network $net_{img}$; 2) a text (i.e., image captions) encoder
network $net_{txt}$. During the training of the noise detection and the
hashing modules, the weights of image and text encoders are frozen. Given the
training set O, the image, text and joint-features are denoted
$net_{img}(\textbf{X})$, $net_{txt}(\textbf{Y})$ and Z, respectively. For the
sake of simplicity we refer $net_{img}(\textbf{X})$ as X, and
$net_{txt}(\textbf{Y})$ as Y in the rest of this paper. For the unsupervised
contrastive representation learning of CHNR, we generate a corresponding
augmented set from O, which is defined as
$\textbf{O}^{\prime}=\\{\textbf{X}^{\prime},\textbf{Y}^{\prime}\\}^{N}$, where
$\textbf{X}^{\prime}=\\{\textrm{x}_{m}^{\prime}\\}_{m=1}^{N}$ and
$\textbf{Y}^{\prime}=\\{\textrm{y}_{m}^{\prime}\\}_{m=1}^{N}$ are augmented
image and caption where $\textrm{x}_{m}^{\prime}\in\mathbb{R}^{d_{i}}$ and
$\textrm{y}_{m}^{\prime}\in\mathbb{R}^{d_{t}}$.

The embeddings of augmented images and captions are extracted by $net_{img}$
and $net_{txt}$, respectively. For the sake of simplicity in the rest of this
paper we refer $net_{img}(\textbf{X}^{\prime})$ as $\textbf{X}^{\prime}$, and
$net_{txt}(\textbf{Y}^{\prime})$ as $\textbf{Y}^{\prime}$. Joint-features of
augmented image-text pairs and corresponding weights are denoted as
$\textbf{Z}^{\prime}$ and $\textbf{W}^{\prime}$ respectively. The same
notation principle applies to the augmented subset without the noisy
correspondences $\textbf{O}_{C}^{\prime}\in\textbf{O}^{\prime}$.
$\textbf{X}_{C}^{\prime}$, $\textbf{Y}_{C}^{\prime}$,
$\textbf{Z}_{C}^{\prime}$ and $\textbf{W}_{C}^{\prime}$ denote augmented clean
image features, text features, joint-features and corresponding feature pair
weights, respectively.

###  2.2 Noise detection module

This module aims at assigning weights to image-text pairs based on the
likelihood of being noisy. The noise discriminator $D_{N}$ is a fully-
connected network with single-neuron output that predicts if joint-feature is
clean. The noise discriminator assigns low weight values to semantically
incoherent (i.e., noisy) joint-features and high weights to clean pairs.
During the meta-learning stage the noise discriminator is trained as a binary
classifier, where original image-text feature pairs from the clean subset
$\textbf{O}_{C}$ are concatenated in ”clean” joint-features with label ”1”.
”Noisy” joint-features with label ”0” are generated with the feature mixer by
randomly shuffling image and text features. Noise discriminator training loss
is defined as:

$\displaystyle\small\min_{\theta_{D_{N}}}\mathcal{L}_{CE}\left(\mathbb{Z}_{C}\right)=-\frac{1}{N_{C}}\sum^{N_{C}}\Big{[}$
$\displaystyle\log\Big{(}D_{N}\big{(}mix(\mathbb{X}_{C},\mathbb{Y}_{C})\big{)}\Big{)}+$
(1)
$\displaystyle\log\Big{(}1-D_{N}\big{(}\mathbb{Z}_{C}\big{)}\Big{)}\Big{]},$

where $\mathbb{Z}_{C}=\\{\textbf{Z}_{C},\textbf{Z}_{C}^{\prime}\\}$,
$\mathbb{X}_{C}=\\{\textbf{X}_{C},\textbf{X}_{C}^{\prime}\\}$,
$\mathbb{Y}_{C}=\\{\textbf{Y}_{C},\textbf{Y}_{C}^{\prime}\\}$ and
$mix(X,Y)=concat(X,shuffle(Y))$ is the feature mixer function for the
generation of semantically incoherent joint-features, where $shuffle(.)$ is a
random shuffle function. During the main training phase the parameters
$\theta_{D_{N}}$ of the noise discriminator $D_{N}$ are frozen. The noise
discriminator $D_{N}$ discriminates image-text features Z and
$\textbf{Z}^{\prime}$ of the noisy dataset O to generate weights W and
$\textbf{W}^{\prime}$, which are passed into the hashing learning module. To
reduce the impact of noisy pairs we exclude them by thresholding W and use
discrete weights $\textbf{W}_{D}=threshold(\textbf{W})$, where
$threshold(x)=\left\\{\begin{matrix}1,&x>=0.5\\\
0,&x<0.5&\end{matrix}\right.$.

###  2.3 Hashing module

This module aims at learning two hash functions $f$ and $g$ for cross-modal
binary hash code B generation from the image features X, $\textbf{X}^{\prime}$
and text features Y, $\textbf{Y}^{\prime}$. Joint-feature weights W,
$\textbf{W}^{\prime}$ are generated by the noise discriminator $D_{N}$ and are
used to reduce the impact of noisy pairs on the learning process by reducing
the importance of pairs identified as noisy. The caption-noise-sensitive
contrastive loss is the main objective for unsupervised representation
learning in the proposed CHNR. We also employ quantization loss to improve the
approximation of generated continuous binary-like values to the discrete hash
code. We use both inter-modal and intra-modal contrastive losses for better
representation learning. The inter-modal term maps both modalities into a
common feature space, while intra-modal terms improve the mapping within
modalities. The normalized temperature scaled cross-entropy (NTXent) objective
function [12] is used for contrastive losses calculation. To obtain the
caption-noise-sensitive contrastive losses we introduce additional re-
weighting term to reduce the impact of noisy image-text pairs on the training.
The weighted inter-modal contrastive loss $\mathcal{L}_{C_{inter}}$ between
image $\textrm{x}_{j}$ and its paired caption $\textrm{y}_{j}$ with image-text
semantic coherence weight $\textrm{w}_{j}$ is computed as:

$\displaystyle\mathcal{L}_{C_{inter}}(\textrm{x}_{j},\textrm{y}_{j})=$ (2)
$\displaystyle-\textrm{w}_{j}\log\frac{S\left(f(\textrm{x}_{j}),g(\textrm{y}_{j})\right)}{\sum_{k=1,k\neq
j}^{M}S\left(f(\textrm{x}_{j}),f(\textrm{x}_{k})\right)+\sum_{k=1}^{M}S\left(f(\textrm{x}_{j}),g(\textrm{y}_{k})\right)},$

where
$S(\textrm{u},\textrm{v})=\exp\left(\cos\left(\textrm{u},\textrm{v}\right)/\tau\right)$,
and
$\cos\left(\textrm{u},\textrm{v}\right)=\textrm{u}^{T}\textrm{v}/\left\|\textrm{u}\right\|\left\|\textrm{v}\right\|$
is the cosine similarity, $\tau$ denotes a temperature, and $M$ is a batch
size. During the meta-learning phase weight of all pairs is set to $1$
($\textrm{w}_{j}=1$), while during the main training phase the weight values
are assigned by $D_{N}$. Image and text intra-modal contrastive losses are
defined as:

$\displaystyle\mathcal{L}_{C_{img}}(\textrm{x}_{j},\textrm{x}^{\prime}_{j})=$
(3)
$\displaystyle-\hat{\textrm{w}}\log\frac{S\left(f(\textrm{x}_{j}),f(\textrm{x}^{\prime}_{j})\right)}{\sum_{k=1,k\neq
j}^{M}S\left(f(\textrm{x}_{j}),f(\textrm{x}_{k})\right)+\sum_{k=1}^{M}S\left(f(\textrm{x}_{j}),f(\textrm{x}^{\prime}_{k})\right)},$

$\displaystyle\mathcal{L}_{C_{txt}}(\textrm{y}_{j},\textrm{y}^{\prime}_{j})=$
(4)
$\displaystyle-\hat{\textrm{w}}\log\frac{S\left(g(\textrm{y}_{j}),g(\textrm{y}^{\prime}_{j})\right)}{\sum_{k=1,k\neq
j}^{M}S\left(g(\textrm{y}_{j}),g(\textrm{y}_{k})\right)+\sum_{k=1}^{M}S\left(g(\textrm{y}_{j}),g(\textrm{y}^{\prime}_{k})\right)},$

where $\mathcal{L}_{C_{img}}$ is the contrastive loss between image
$\textrm{x}_{j}$ and its augmented view $\textrm{x}^{\prime}_{j}$ and
$\mathcal{L}_{C_{txt}}$ is the contrastive loss between caption
$\textrm{y}_{j}$ and its augmented view $\textrm{y}^{\prime}_{j}$. During the
meta-learning stage $\hat{\textrm{w}}=1$, for the full training the averaged
weight $\hat{\textrm{w}}=\frac{1}{M}\sum_{j=1}^{M}\textrm{w}_{j}$ is used to
avoid the skew towards intra-modal objectives in the total contrastive loss
$\mathcal{L}_{C}$ defined as:

$\small\mathcal{L}_{C}=\mathcal{L}_{C_{inter}}+\lambda_{1}\mathcal{L}_{C_{img}}+\lambda_{2}\mathcal{L}_{C_{txt}},$
(5)

where $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters for image and text
intra-modal contrastive losses, respectively.

The quantization loss $\mathcal{L}_{Q}$ optimizes the difference between
continuous and discrete hash values and calculated as:

$\small\mathcal{L}_{Q}=\left\|\textbf{B}-\textbf{H}_{i}\right\|^{2}_{F}+\left\|\textbf{B}-\textbf{H}_{i}^{\prime}\right\|^{2}_{F}+\left\|\textbf{B}-\textbf{H}_{t}\right\|^{2}_{F}+\left\|\textbf{B}-\textbf{H}_{t}^{\prime}\right\|^{2}_{F},$
(6)

where $\textbf{H}_{i}=f(\textbf{X})$,
$\textbf{H}_{i}^{\prime}=f(\textbf{X}^{\prime})$,
$\textbf{H}_{t}=g(\textbf{Y})$,
$\textbf{H}_{t}^{\prime}=g(\textbf{Y}^{\prime})$ are binary like codes for
images, augmented images, texts and augmented texts, respectively. The binary
code is updated by the following rule:

$\small\textbf{B}=sign\left(\frac{1}{2}\left(\frac{\textbf{H}_{i}+\textbf{H}_{i}^{\prime}}{2}+\frac{\textbf{H}_{t}+\textbf{H}_{t}^{\prime}}{2}\right)\right).$
(7)

The final loss function is the weighted sum of (5) and (6):

$\small\min_{\textbf{B},\theta_{i},\theta_{t},\theta_{D}}\mathcal{L}=\mathcal{L}_{C}+\alpha\mathcal{L}_{Q},$
(8)

where $\alpha$ is a hyperparameter for quantization loss. Finally, for the
retrieval of semantically similar captions to a query image $\textrm{x}_{q}$,
we compute the Hamming distance between $f(net_{img}(\textrm{x}_{q}))$ and
hash codes in the archive, and the most similar $K$ captions are retrieved.
Similarly, the most similar $K$ images are retrieved for a query caption
$\textrm{y}_{q}$ with regard to the Hamming distances estimated between
$g(net_{txt}(\textrm{y}_{q}))$ and hash codes of the images in the archive.

Table 1: The mAP@20 results for image-to-text ($I\to T$) and text-to-image ($T\to I$) retrieval for the RSICD dataset when $B=64$. Task | Metod | Injected noise rate
---|---|---
5% | 10% | 20% | 30% | 40% | 50%
$I\to T$ | CHNR | 0.786 | 0.770 | 0.739 | 0.732 | 0.717 | 0.708
CHNR-NW | 0.778 | 0.752 | 0.730 | 0.720 | 0.668 | 0.617
CHNR-PTC | 0.772 | 0.749 | 0.698 | 0.660 | 0.582 | 0.525
CHNR-WNR | 0.785 | 0.767 | 0.724 | 0.677 | 0.604 | 0.475
$T\to I$ | CHNR | 0.783 | 0.782 | 0.754 | 0.746 | 0.720 | 0.718
CHNR-NW | 0.778 | 0.767 | 0.745 | 0.737 | 0.704 | 0.664
CHNR-PTC | 0.776 | 0.766 | 0.727 | 0.709 | 0.648 | 0.614
CHNR-WNR | 0.784 | 0.783 | 0.751 | 0.706 | 0.647 | 0.540
Table 2: The mAP@20 results for $I\to T$ and $T\to I$ retrieval tasks for the RSICD and the UCM datasets when $B=64$. Task | Metod | Amount of the caption noise
---|---|---
RSICD | UCM
5% | 10% | 20% | 30% | 40% | 50% | 5% | 10% | 20% | 30% | 40% | 50%
$I\to T$ | CHNR | 0.786 | 0.770 | 0.739 | 0.732 | 0.717 | 0.708 | 0.843 | 0.832 | 0.844 | 0.823 | 0.821 | 0.796
JDSH[3] | 0.768 | 0.737 | 0.683 | 0.609 | 0.487 | 0.381 | 0.806 | 0.775 | 0.714 | 0.620 | 0.567 | 0.396
DJSRH[2] | 0.669 | 0.661 | 0.602 | 0.559 | 0.491 | 0.397 | 0.707 | 0.688 | 0.633 | 0.586 | 0.534 | 0.426
$T\to I$ | CHNR | 0.783 | 0.782 | 0.754 | 0.746 | 0.720 | 0.718 | 0.929 | 0.912 | 0.908 | 0.891 | 0.885 | 0.849
JDSH[3] | 0.769 | 0.754 | 0.699 | 0.639 | 0.540 | 0.415 | 0.882 | 0.860 | 0.764 | 0.699 | 0.603 | 0.442
DJSRH[2] | 0.666 | 0.647 | 0.600 | 0.547 | 0.472 | 0.373 | 0.751 | 0.742 | 0.685 | 0.621 | 0.521 | 0.468

##  3 Experimental Results

In the experiments, we used the RSICD [13] and the UC Merced Land Use (denoted
as UCM) [14] datasets. RSICD includes $10921$ aerial images, each of which is
a section of $224\times 224$ pixels and has $5$ corresponding captions. UCM
consists of $2100$ aerial images, each of which is a section of $256\times
256$ pixels and has $5$ captions per image. For both datasets, we used only
one randomly selected caption associated to each image for training. The
datasets were split randomly into train, query and retrieval sets. We applied
a Gaussian blur, random rotation, and center cropping for the image
augmentation, while the text augmentation was performed by the rule-based
replacement [15] of noun and verb tokens with semantically similar ones. The
original training sets are clean but in our experiments, we fixed 20% and 30%
of training set for RSICD and UCM, respectively as clean training set
$\textbf{O}_{C}$, while we inject noise to the rest.

For $net_{img}$, a pre-trained ResNet architecture [16] was used (the
classification layer was removed) and the image feature size $d_{i}$ is $512$.
For $net_{txt}$, a pre-trained BERT [17] language model was used and the text
feature size $d_{t}$ is $768$ (which was obtained by summing the last four
hidden states of each token). For the noise discriminator $D_{N}$, we used a
$5$-layer fully-connected network, which was trained only during the meta-
learning stage. Hashing networks $f$ and $g$ are fully connected $3$-layer
networks and a batch normalization layer after the second layer was included.
The quantization loss hyperparameter $\alpha$ was set to $\alpha=0.01$. Both
intra-modal weights $\lambda_{1}$ and $\lambda_{2}$ from (5) were set to $1$.
The total number of training epochs was set to $150$ ($75$ for meta-training
and $75$ for main training epochs).

To analyze the effect of the noise detection module and its training strategy,
we designed different configurations for the proposed CHNR as: 1) the noise
detection module is not included and training is achieved on the noisy
training set (denoted as CHNR-WNR); 2) the noise detection module is not
included and the model initially pretrained on the small subset of the clean
training set and then main training phase is applied using the noisy dataset
(denoted as CHNR-PTC); and 3) the noise detection module is included but the
model does not contain thresholding weights $W$ (CHNR-NW). The result of each
configuration is provided in terms of mean average precision assessed on
top-20 retrieved images (mAP@20) in Table 1 in the framework of image-to-text
($I\to T$) and text-to-image ($T\to I$) retrieval tasks when $B=64$ for the
RSICD dataset. The table shows that using the meta-training phase without the
noise detection module can reduce the performance except for the extreme noise
rate (e.g., 50%). As an example, when the injected noise rate is 20% CHNR-WNR
provides about 3% higher mAP@20 than CHNR-PTC for both $I\to T$ and $T\to I$
tasks. However, when the noise rate is 50% CHNR-PTC results in more than 5%
higher mAP@20 than CHNR-WNR for both $I\to T$ and $T\to I$ tasks. Training on
clean samples in the meta-learning phase prevents the model from fully
learning the data distribution in the main training phase. This can reduce the
performance when the noise rate is low since the model can not entirely learn
from the main training phase. However, when the training set is extremely
noisy, the model will not be distracted by the noise. Furthermore, the Table 1
shows the superiority of CHNR and CHNR-NW, particularly when the noise rate is
high. As an example, when the noise rate is 20%, CHNR leads to about 5% higher
mAP@20 than CHNR-WNR for both $I\to T$ and $T\to I$. This shows that using the
noise detection module makes the model robust to the noise in the training
set. For smaller noise rates (i.e., 5% and 10%) CHNR performs almost
comparable with CHNR-WNR. This is because of excluding some hard but
informative samples from the training via reweighing them through the noise
detection module.

We evaluated the effectiveness of the proposed CHNR method with respect to the
state of the art unsupervised cross-modal retrieval methods, which are: DJSRH
[2] and JDSH [3]. We trained all models under the same experimental setup for
a fair comparison. Results of each method were provided in terms of mAP@20.
The experiments were conducted over different injected noise rates (5%, 10%,
20%, 30%, 40%, and 50%) when $B=64$. Table 2 shows the retrieval performance
for the RSICD and UCM datasets. From the table, one can observe that the
proposed CHNR method sharply outperforms all the unsupervised baselines in
$I\to T$ and $T\to I$ tasks for all injected noise rates on both datasets. As
an example, for $I\to T$ retrieval task for the UCM dataset, when the injected
noise rate in 20% the proposed CHNR results in about 13% and 21% higher mAP@20
than JDSH and DJSRH, respectively. Similarly, for the RSICD dataset in $T\to
I$ retrieval task when the injected noise rate is 20%, the proposed CHNR
outperforms JDSH and DJSRH with 5% and 15% higher mAP@20, respectively. From
the table, one can observe that all the compared methods are robust to small
amounts of noise (i.e., $5\%$ and $10\%$), while the mAP@20 drops
significantly for all methods by increasing the injected noise rate (more than
20% ) for $I\to T$ and $T\to I$ tasks. However, the performance drop is
considerably smaller for the proposed CHNR than the other methods. As an
example, when the noise injection rate is $5\%$ for $I\to T$ task in RSICD
dataset, the performance of CHNR is $2\%$ and $9\%$ higher than JDSH and
DJSRH, respectively. However, when the injected noise rate increases to 50%
for the same task of the same dataset the performance of CHNR is $32\%$ and
$31\%$ higher than JDSH and DJSRH, respectively.

##  4 Conclusion

In this paper, we have proposed a novel unsupervised cross-modal hashing
method robust to the noisy image-text correspondences (CHNR). The proposed
CHNR uses a multi-term noise-robust contrastive loss function to learn cross-
modal hash codes in an unsupervised manner. In detail, the proposed loss
function has weighted intra- and inter-modal objectives, which take into
account the coherence of cross-modal correspondences represented by weights.
For the weight calculation, the cross-modal joint-feature noise discriminator
has been introduced. Furthermore, we have analyzed the proposed noise
detection module and demonstrated the effectiveness of the proposed CHNR
method through the experimental results. As future work, we plan to learn the
augmentations in the feature level to improve the unsupervised contrastive
learning process.

##  5 Acknowledgment

This work is funded by the European Research Council (ERC) through the
ERC-2017-STG BigEarth Project under Grant 759764.

## References

  * [1] H. Jung, Y. Oh, S. Jeong, C. Lee, and T. Jeon,  “Contrastive self-supervised learning with smoothed representation for remote sensing,”  IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2021\.
  * [2] S. Su, Z. Zhong, and C. Zhang,  “Deep joint-semantics reconstructing hashing for large-scale unsupervised cross-modal retrieval,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3027–3035.
  * [3] S. Liu, S. Qian, Y. Guan, J. Zhan, and L. Ying,  “Joint-modal distribution-based similarity hashing for large-scale unsupervised deep cross-modal retrieval,”  in Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020, pp. 1379–1388.
  * [4] X. Chen, H. Fan, R. Girshick, and K. He,  “Improved baselines with momentum contrastive learning,”  arXiv preprint arXiv:2003.04297, 2020.
  * [5] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton,  “A simple framework for contrastive learning of visual representations,”  in International Conference on Machine Learning. PMLR, 2020, pp. 1597–1607.
  * [6] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick,  “Momentum contrast for unsupervised visual representation learning,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9729–9738.
  * [7] M. Zolfaghari, Y. Zhu, P. Gehler, and T. Brox,  “Crossclr: Cross-modal contrastive learning for multi-modal video representations,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1450–1459.
  * [8] R. Shekhar, S. Pezzelle, Y. Klimovich, A. Herbelot, M. Nabi, E. Sangineto, and R. Bernardi,  “Foil it! find one mismatch between image and language caption,”  in Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2017, pp. 255–265.
  * [9] Y. Zhang, Y. Ding, R. Wu, and F. Xue,  “A denoising framework for image caption,”  in IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress, 2019, pp. 825–832.
  * [10] P. Qin, Y. Li, K. Deng, and Q. Wu,  “TVDIM: Enhancing image self-supervised pretraining via noisy text data,”  arXiv preprint arXiv:2106.01797, 2021.
  * [11] V. Malykh,  “Robust to noise models in natural language processing tasks,”  in Proceedings of the Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, 2019, pp. 10–16.
  * [12] Y. Chen, X. Lu, and S. Wang,  “Deep cross-modal image–voice retrieval in remote sensing,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 10, pp. 7049–7061, 2020.
  * [13] X. Lu, B. Wang, X. Zheng, and X. Li,  “Exploring models and data for remote sensing image caption generation,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 4, pp. 2183–2195, 2017.
  * [14] Y. Yang and S. Newsam,  “Bag-of-visual-words and spatial extensions for land-use classification,”  in Proceedings of the SIGSPATIAL International Conference on Advances in Geographic Information Systems, 2010, pp. 270–279.
  * [15] J. Wei and K. Zou,  “Eda: Easy data augmentation techniques for boosting performance on text classification tasks,”  in Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019, pp. 6382–6388.
  * [16] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  * [17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,”  in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, pp. 4171–4186.


# Margin Calibration for Long-Tailed Visual Recognition

Yidong Wang1, Bowen Zhang1, Wenxin Hou2, Zhen Wu3, Jindong Wang4, Takahiro
Shinozaki†1
Corresponding author.

###### Abstract

The long-tailed class distribution in visual recognition tasks poses great
challenges for neural networks on how to handle the biased predictions between
head and tail classes, i.e., the model tends to classify tail classes as head
classes. While existing research focused on data resampling and loss function
engineering, in this paper, we take a different perspective: the
classification margins. We study the relationship between the margins and
logits (classification scores) and empirically observe the biased margins and
the biased logits are positively correlated. We propose MARC, a simple yet
effective MARgin Calibration function to dynamically calibrate the biased
margins for unbiased logits. We validate MARC through extensive experiments on
common long-tailed benchmarks including CIFAR-LT, ImageNet-LT, Places-LT, and
iNaturalist-LT. Experimental results demonstrate that our MARC achieves
favorable results on these benchmarks. In addition, MARC is extremely easy to
implement with just three lines of code. We hope this simple method will
motivate people to rethink the biased margins and biased logits in long-tailed
visual recognition.

## Introduction

Despite the great success of neural networks in the visual recognition field
(Simonyan and Zisserman 2014; He et al. 2016), it is still challenging for
neural networks to deal with the ubiquitous long-tailed datasets in the real
world (Buda, Maki, and Mazurowski 2018; Kang et al. 2019; Zhou et al. 2020).
To be clear, in the long-tailed datasets, the high-frequency classes (head
classes) occupy most of the instances, whereas the low-frequency classes (tail
classes) involve a small amount of instances (Liu et al. 2019; Van Horn and
Perona 2017). Due to the imbalance of the training data, the model performs
well in head classes and its performance is much worse in tail classes (Buda,
Maki, and Mazurowski 2018; Zhang et al. 2021).

(a) Logits

(b) Margins

Figure 1: Logits and margins of CIFAR10-LT with imbalance factor 200, where
before and after refers to standard training and our method, respectively. The
class indexes are sorted by the number of samples (Head to tail).

Towards addressing the long-tailed recognition problem, there are several
strategies such as data re-sampling and loss function engineering. Data re-
sampling aims to ‘simulate’ a balanced training dataset by over-sampling the
tail class or under-sampling the head classes (Ando and Huang 2017; Buda,
Maki, and Mazurowski 2018; Pouyanfar et al. 2018; Shen, Lin, and Huang 2016),
while loss re-weighting is introduced to adjust the weights of losses for
different classes or different instances (Byrd and Lipton 2019; Khan et al.
2017; Wang, Ramanan, and Hebert 2017). For more balanced gradients between
classes, some class-balanced loss functions adjust the logits(classification
scores) instead of weighting the losses (Menon et al. 2020; Cao et al. 2019a;
Ren et al. 2020).

However, as pointed out by existing research (Ganganwar 2012; Zhou and Liu
2005; Cao et al. 2019a), data re-sampling strategies and loss re-weighting
schemes will possibly cause underfitting on the head class and overfitting on
the tail class. On the other hand, class-balanced loss functions or data re-
sampling will lead to worse data representations compared with the standard
training using the cross-entropy loss and the instance-balanced sampling
(i.e., each instance has the same probability of being sampled) (Kang et al.
2019; Ren et al. 2020). In addition, the recent research reveals that the
biased decision boundary given by the classifier head seems to be the
performance bottleneck of the long-tailed visual recognition (Kang et al.
2019; Zhang et al. 2021). To benefit from both good data representations and
the unbiased decision boundary, Decoupling, a heuristic two-stage strategy is
proposed to adjust the initially-learned classifier head (Kang et al. 2019)
after the standard training. Furthermore, distribution alignment (Zhang et al.
2021) is developed as an adaptive calibration function to adjust the initially
trained classification scores for each data point. However, the relation
between the biased margin and the biased logit is neglected in existing
research, where the margin is the distance from the data point to the decision
boundary.

In this paper, we study the relationship between margins and logits, which are
the fundamental factors that dominate the long-tailed performance. We
empirically find that the margin and the logit are correlated with the
cardinality of each class. To be concrete, as shown in Figure 1, head classes
tend to have much larger margins and logits than tail classes. Hence, to
obtain the unbiased logit, calibrating the biased margin is necessary.
Inspired by these evidence, we propose a simple yet effective MARgin
Calibration (MARC) for long-tailed recognition. In detail, after getting the
representations and the classifier head from the standard training, we propose
a simple class-specific margin calibration function with only $2K$ learnable
parameters to dynamically adjust the initially learned margins, where $K$ is
the number of classes. As demonstrated in Figure 1, the logits are more
balanced when using MARC.

We conduct experiments on CIFAR-10-LT (Krizhevsky, Hinton et al. 2009),
CIFAR-100-LT (Krizhevsky, Hinton et al. 2009), Places-LT (Zhou et al. 2017),
iNaturalist 2018 (Van Horn et al. 2018) and ImageNet-LT (Liu et al. 2019). The
results illustrate that our proposed MARC method performs remarkably well even
if MARC is very simple to implement. We hope our exploration will attract
attention to the biased margins in long-tailed recognition.

To conclude, our contributions are as follows:

  * •

For the first time, we study the biased predictions from a margin-based
perspective in long-tailed recognition. We find that biased margins will cause
biased predictions.

  * •

Based on our observations, we propose a simple yet effective margin
calibration (MARC) function with only $2K$ trainable parameters to adjust the
margin and get the unbiased prediction.

  * •

MARC performs notably well on various long-tailed visual benchmarks like
CIFAR-10-LT (2009), CIFAR-100-LT (2009), ImageNet-LT (2019), Places-LT (2017)
and iNaturalist2018 (2018). In addition, it is extremely easy to implement
with just three lines of code.

## Related Work

Long-tailed visual recognition has attracted much attention for its commonness
in the real world (He and Garcia 2009; Buda, Maki, and Mazurowski 2018; Kang
et al. 2019; Ren et al. 2020; Yang and Xu 2020; Hong et al. 2021). Existing
methods can be divided into four categories.

Data re-sampling. Data re-sampling technologies re-sample the imbalanced
training dataset to ‘simulate’ a balanced training dataset. These methods
include under-sampling, over-sampling, and classed-balanced sampling. Under-
sampling decreases the probability of the instance of head classes being
sampled (Drummond, Holte et al. 2003), whereas over-sampling makes instances
of tail classes more likely to be sampled (Chawla et al. 2002; Han, Wang, and
Mao 2005; Wang et al. 2021a). Class-aware sampling chooses instances of each
class with the same probabilities (Shen, Lin, and Huang 2016).

Loss function engineering. Loss function engineering is another direction to
obtain balanced gradients during the training. The typical methods can be
categorized as loss-reweighting and logits adjustment. Loss re-weighting
adjusts the weights of losses for different classes or different instances in
a more balanced manner, i.e. the instances in tail classes have larger weights
than those in head classes (Byrd and Lipton 2019; Khan et al. 2017; Wang,
Ramanan, and Hebert 2017). On the other hand, instead of re-weight losses,
some class-balanced loss functions adjust the logits to get balanced gradients
during training (Menon et al. 2020; Cao et al. 2019a; Ren et al. 2020; Yang,
Yang, and Wang 2009).

Decision boundary adjustment. Nevertheless, data re-sampling or loss function
engineering will influence the representations of data (Ren et al. 2020). Lots
of empirical observations show that we can acquire good representation when
using the standard training and the classifier head is the performance
bottleneck (Kang et al. 2019; Zhang et al. 2021; Yu et al. 2020; Kim and Kim
2020). To solve the above problem, decision boundary adjustment methods re-
adjust the classifier head after the standard training (Kang et al. 2019;
Zhang et al. 2021). However, they ignore the relationship between the biased
margins and the biased logits.

Other methods. There also exist other paradigms to deal with the long-tailed
recognition task, including task-specific neuron networks design (Wang et al.
2021c; Zhou et al. 2020; Wang et al. 2021a), transfer learning (Liu et al.
2019; Yin et al. 2019), domain adaptation (Jamal et al. 2020), semi supervised
learning and self supervised learning (Yang and Xu 2020). But these methods
either rely on the non-trivial network design or external data. In contrast,
our proposed MARC is very simple to implement and does not require external
data. The detailed comparison between MARC and similar methods is shown in
Table 1.

## Method

### Preliminaries

In the popular setting of long-tailed recognition (Kang et al. 2019; Cui et
al. 2019; Ren et al. 2020), the training data distribution is imbalanced while
the test data distribution is balanced. More formally, let
$\mathcal{D}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{n}$ be a training set, where
$y_{i}$ denotes the label of data point $\mathbf{x}_{i}$. Specifically,
$n=\sum_{j=1}^{K}n_{j}$ is the total number of training samples, where $n_{j}$
is the number of training samples in class $j$ and $K$ is the number of
classes. We assume $n_{1}>n_{2}>\cdots>n_{K}$ without loss of generality.
Normally, the prediction function is composed of two modules: the feature
representation learning function $f:\mathbf{x}\mapsto\mathbf{z}$ parameterized
by $\theta_{r}$ and the classifier $g:\mathbf{z}\mapsto y$ parameterized by
$\theta_{c}$, where $\mathbf{z}\in\mathbb{R}^{p}$ denotes the feature
representation and $p$ is the feature dimension. Typically, $g$ is a linear
classifier that gives the classification score of class $j$ as:
$\eta_{j}=g(\mathbf{z}):=\mathbf{W}_{j}\mathbf{z}+\mathbf{b}_{j}$ with
$\mathbf{W}_{j}$ and $\mathbf{b}_{j}$ its weight vector and bias for class
$j$. Finally, using the softmax function, the probability of $\mathbf{x}_{i}$
being classified as label $y_{i}$ is expressed as:

$p(y=y_{i}|\mathbf{x}_{i};\theta_{r},\theta_{c})=\frac{e^{\eta_{y_{i}}}}{\sum_{j=1}^{K}e^{\eta_{j}}},$
(1)

and its loss is computed as the cross-entropy loss:

$\ell(\mathbf{x}_{i},y_{i};\theta_{r},\theta_{c})=-\log(\frac{e^{\eta_{y_{i}}}}{\sum_{j=1}^{K}e^{\eta_{j}}}).$
(2)

### Motivation

The decision boundary is often biased in long-tailed recognition, which will
lead to biased predictions, i.e., the model tends to classify tail classes as
head classes. To alleviate this issue, data re-sampling and loss function
engineering are two directions to simulate a ‘balanced’ training dataset.
However, such techniques will do harm to the representation learning of the
model (Kang et al. 2019; Ren et al. 2020). To benefit from both the good
representation and the unbiased decision boundary, decision boundary adjusts
methods are developed (Kang et al. 2019; Zhang et al. 2021). However, existing
decision boundary adjusts methods ignore the bias in the margins, which we
think is essential to avoid biased predictions. Thus in this paper, we aim to
calibrate the biased margins to obtain unbiased predictions.

### Biased Margins and Biased Logits

In this paper, we find that the margins and logits are biased in long-tailed
recognition. We illustrate margins with Figure 2. We define an affine
hyperplane $H_{j}\in\mathbb{R}^{{p}-1}$ of class $j$ as
$\mathbf{W}_{j}\mathbf{z}+\mathbf{b}_{j}=0$, i.e. any representation point
falling on the positive side of $H_{j}$ can be attributed to class $j$. Assume
that $\mathbf{z}_{0}$ is a point satisfying
$\mathbf{W}_{j}\mathbf{z}_{0}+\mathbf{b}_{j}=0$, i.e., $\mathbf{z}_{0}$ is a
point on the hyperplane $H_{j}$. Suppose $\mathbf{z}_{1}$ is an arbitrary
point in the feature space. We construct the vector
$\mathbf{z}_{1}-\mathbf{z}_{0}$ pointing from $\mathbf{z}_{0}$ to
$\mathbf{z}_{1}$ and project it onto the normal vector $\mathbf{W}_{j}$. The
length of the projection vector
$proj_{\mathbf{W}_{j}}(\mathbf{z}_{1}-\mathbf{z}_{0})$ is the margin from
$\mathbf{z}_{1}$ to $H_{j}$. Specifically, the margin from $\mathbf{z}_{1}$ to
the hyperplane of class $j$ is calculated as:

$\begin{split}d_{j}&=\|proj_{\mathbf{W}_{j}}(\mathbf{z}_{1}-\mathbf{z}_{0})\|\\\
&=\|\frac{\mathbf{W}_{j}\cdot(\mathbf{z}_{1}-\mathbf{z}_{0})}{\mathbf{W}_{j}\cdot\mathbf{W}_{j}}\mathbf{W}_{j}\|\\\
&=\frac{\mathbf{W}_{j}\cdot\mathbf{z}_{1}-\mathbf{W}_{j}\cdot\mathbf{z}_{0}}{\|\mathbf{W}_{j}\|}\\\
&=\frac{\mathbf{W}_{j}\mathbf{z}_{1}+\mathbf{b}_{j}}{\|\mathbf{W}_{j}\|}\quad(\text{since}\
\mathbf{W}_{j}\mathbf{z}_{0}+\mathbf{b}_{j}=0),\end{split}$ (3)

where $\|\cdot\|$ denotes L2 norm. Thus, the logit
$\mathbf{W}_{j}\cdot\mathbf{z}_{1}+\mathbf{b}_{j}$ can also be expressed as
$\|\mathbf{W}_{j}\|d_{j}$. Based to this conclusion, we can rewrite Eq. (1)
as:

$p(y=y_{i}|\mathbf{x}_{i};\theta_{r},\theta_{c})=\frac{e^{\|\mathbf{W}_{y_{i}}\|d_{y_{i}}}}{\sum_{j=1}^{K}e^{\|\mathbf{W}_{j}\|d_{j}}}.$
(4)

We assume a data point is on the decision boundary of class $j$ and class $t$,
i.e., such data point has the same probability of being classified as class
$j$ or class $t$. Clearly, the assumed data point on the decision boundary
satisfies

$\eta_{j}=\eta_{t}=\|\mathbf{W}_{j}\|d_{j}=\|\mathbf{W}_{t}\|d_{t}.$ (5)

According to Eq. (5), data will be classified as class $t$ because
$d_{j}<d_{t}$ when $\|\mathbf{W}_{j}\|=\|\mathbf{W}_{t}\|$. And our empirical
observations show that head classes tend to have much larger margins and
logits than tail classes:

$\begin{split}&\bar{d_{1}}>\bar{d_{2}}>\cdots>\bar{d_{K}},\\\
&\bar{\eta_{1}}>\bar{\eta_{2}}>\cdots>\bar{\eta_{K}},\\\ &if\quad
n_{1}>n_{2}>\cdots>n_{K},\end{split}$ (6)

where $\bar{d_{j}}$ is the average margin of class $j$ and $\bar{\eta_{j}}$ is
the average logit of class $j$ after the standard training. In detail, on the
sub-dataset $\mathcal{D}_{j}=\\{(\mathbf{x}_{i},y_{i}=j)\\}_{i=1}^{n_{j}}$,
$\bar{\eta_{j}}=\frac{1}{n_{j}}\eta_{j}$,
$\bar{d_{j}}=\frac{\bar{\eta_{j}}}{\|\mathbf{W}_{j}\|}$.

Figure 2: Illustration of margins. The red and blue dots denote majority and
minority classes, respectively.

### Margin Calibration Function (MARC)

To get the unbiased logit, we propose to calibrate the margins after the
standard training. Concretely, we train a simple class-specific margin
calibration model with the original margin fixed:

$\begin{split}\hat{d_{j}}=\omega_{j}\cdot d_{j}+\beta_{j},\end{split}$ (7)

where $\omega_{j}$ and $\beta_{j}$ are learnable parameters for class $j$ and
$j\in[1,K]$. In other words, MARC only has $2K$ trainable parameters. Then the
calibrated logit is computed as:

$\begin{split}\|\mathbf{W}_{j}\|\hat{d_{j}}&=\|\mathbf{W}_{j}\|(\omega_{j}\cdot
d_{j}+\beta_{j})\\\
&=\omega_{j}\cdot\|\mathbf{W}_{j}\|d_{j}+\beta_{j}\cdot\|\mathbf{W}_{j}\|\\\
&=\omega_{j}\cdot\eta_{j}+\beta_{j}\cdot\|\mathbf{W}_{j}\|,\\\ \end{split}$
(8)

where $\eta_{j}$ is the initial fixed logit. Then we can get the calibrated
prediction distribution:

$p(y=y_{i}|\mathbf{x}_{i};\theta_{r},\theta_{c})=\frac{e^{\omega_{y_{i}}\cdot\eta_{y_{i}}+\beta_{y_{i}}\cdot\|\mathbf{W}_{y_{i}}\|}}{\sum_{j=1}^{K}e^{\omega_{j}\cdot\eta_{j}+\beta_{j}\cdot\|\mathbf{W}_{j}\|}}.$
(9)

The training process of the margin calibration function can be written with
_just three_ lines of Pytorch codes as shown in Line 4-6 of Algorithm 1.

Algorithm 1 The torch-like code for MARC.

1: Initialization of the margin calibration function:
omega=torch.nn.Parameter(torch.ones(1,K))beta=torch.nn.Parameter(torch.zeros(1,K))

2: Input: training data x, standard pre-trained neuron network model.

3: with torch.no_grad():

4: w_norm $=$ torch.norm(model.fc.weight, dim=1)

5: logit_before $=$ model(x)

6: logit_after $=$ omega * logit_before $\backslash$ \+ beta * w_norm

7: Compute loss and update parameters of omega and beta.

Furthermore, for more balanced gradients during training, we re-weight the
loss as the previous work does (Zhang et al. 2021). Finally, the loss for
training the margin calibration function is:

$\begin{split}&\ell(\mathbf{x}_{i},y_{i};\tilde{\theta_{r}},\tilde{\theta_{c}},\omega,\beta)=\\\
&-weight_{y_{i}}\cdot\log(\frac{e^{\omega_{y_{i}}\cdot\eta_{y_{i}}+\beta_{y_{i}}\cdot\|\mathbf{W}_{y_{i}}\|}}{\sum_{j=1}^{K}e^{\omega_{j}\cdot\eta_{j}+\beta_{j}\cdot\|\mathbf{W}_{j}\|}}),\end{split}$
(10)

where parameters with $\tilde{\cdot}$ are frozen during training and the
weight for class $y_{i}$ is calculated as:

$weight_{y_{i}}=K\cdot\frac{(1/n_{y_{i}})^{\gamma}}{\sum_{j=1}^{K}(1/n_{j})^{\gamma}},$
(11)

where $\gamma$ is a scale hyper-parameter. When $\gamma=0$, the weight for all
classes is $1$, which means no re-weighting at all.

To be more clear, the whole detailed training procedure including both
standard training and margin calibration function training is demonstrated in
Algorithm 2. Lines 2-6 include the training procedure of the standard training
using the instance-balanced sampling and the cross-entropy loss. Lines 7-11
contain the training process of our margin calibration function. It is well
noting that in the second stage, parameters $\theta_{r}$ and $\theta_{c}$ are
all fixed.

### Discussion

We clarify the differences between MARC and other learnable decision boundary
adjustment methods in detail. As shown in Table 1, Decouple-cRT (Kang et al.
2019) retrains the whole parameters of the classifier, while Decouple-LWS
(Kang et al. 2019) only adjusts the norm of weight vectors
$\|\mathbf{W}_{j}\|$. Instead of adjusting the classifier head, DisAlign
(Zhang et al. 2021) chooses to calibrate the logit for each data point. But
their calibration method is heuristic that simply adds the calibrated logit
and the original logit with a re-weighting scheme. To be more clear, the
weighted sum of logits for DisAlign is
$\sigma(\mathbf{z}_{j})(\omega_{j}\eta_{j}+\beta)+(1-\sigma(\mathbf{z}_{j}))\eta_{j}$,
where $\sigma(\cdot)$ is an instance-specific confidence function. However,
different from previous methods, our MARC focuses on calibrating the biased
margin which we believe is the performance bottleneck of the long-tailed
classifier.

Table 1: The differences between MARC and other decision boundary adjustment
methods. $j\in[1,K]$ is class index.

Methods | Calibration method
---|---
Decouple-cRT (Kang et al. 2019) | retrain $\mathbf{W}_{j},\mathbf{b}_{j}$
Decouple-LWS (Kang et al. 2019) | ${\|\mathbf{W}_{j}\|}^{1-\omega_{j}}$
DisAlign (Zhang et al. 2021) | $\sigma(\mathbf{z}_{j})(\omega_{j}\eta_{j}+\beta)+(1-\sigma(\mathbf{z}_{j}))\eta_{j}$
MARC | $\omega_{j}\cdot d_{j}+\beta_{j}$

Algorithm 2 The detailed training procedure including both standard training
and margin calibration function training.

1: Input: The training dataset
$\mathcal{D}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{n}$, the parameters of the
representation function $\theta_{r}$, the parameters of the classifier
$\theta_{c}$, the parameters of the margin calibration function $\omega$ and
$\beta$, the number of classes $K$ and the pre-defined scale hyper-parameter
$\gamma$.

2: First stage: the standard training use the instance-balanced sampling and
the cross entropy loss.

3: while not reach the maximum iteration do

4: Use instance-balanced sampling to sample a batch of data
$\mathcal{D}_{s}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{s}$ from the training
dataset $\mathcal{D}$, where $s$ is the batch size.

5: Compute the loss and update the model
parameters.$\ell(\mathcal{D}_{s};\theta_{r},\theta_{c})=\frac{1}{s}\sum_{i=1}^{s}(-\log(\frac{e^{\eta_{y_{i}}}}{\sum_{j=1}^{K}e^{\eta_{j}}})),$
where $\eta_{j}$ is the classification score of class $j$.

6: end while

7: Second stage: Calibrate the margins trained in the first stage.

8: while not reach the maximum iteration do

9: Use instance-balanced sampling to sample a batch of data
$\mathcal{D}_{s}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{s}$ from the training
dataset $\mathcal{D}$, where $s$ is the batch size.

10: Compute the loss and update the model
parameters.$\ell(\mathcal{D}_{s};\tilde{\theta_{r}},\tilde{\theta_{c}},\omega,\beta)=\frac{1}{s}\sum_{i=1}^{s}(-weight_{y_{i}}\cdot\log(\frac{e^{\omega_{y_{i}}\cdot\eta_{y_{i}}+\beta_{y_{i}}\cdot\|\mathbf{W}_{y_{i}}\|}}{\sum_{j=1}^{K}e^{\omega_{j}\cdot\eta_{j}+\beta_{j}\cdot\|\mathbf{W}_{j}\|}})),$
where parameters with $\tilde{\cdot}$ are fixed during training and
$weight_{j}$ is calculated as Eq. 11 shows.

11: end while

12: Return: Model parameters $\theta_{r},\theta_{c},\omega,\beta$.

## Experiments

In this section, we conduct extensive experiments compared with the state-of-
the-art methods to validate the effectiveness of MARC. Firstly, we report the
performance on common benchmarks like CIFAR-10-LT (2009), CIFAR-100-LT (2009),
ImageNet-LT (Liu et al. 2019), Places-LT (2017) and iNaturalist2018 (2018).
The results of MARC are competitive even though MARC is simple. Then we
conduct further analysis to explain the reason for the success of MARC.

#### Datasets

We follow the common evaluation protocol (Liu et al. 2019) and conduct
experiments on CIFAR-10-LT (2009), CIFAR-100-LT (2009), ImageNet-LT (2019),
Places-LT (2017) and iNaturalist2018 (2018). The imbalance factor used in
CIFAR datasets is defined as $N_{max}/N_{min}$ where $N_{max}$ is the number
of samples on the largest class and $N_{min}$ the smallest. We report CIFAR
results with two different imbalance ratios: 100 and 200. For ImageNet-LT and
Places-LT experiments, we further split classes into three sets: Many-shot
(with more than 100 images), Medium-shot (with 20 to 100 images), and Few-shot
(with less than 20 images).

Table 2: Accuracy on CIFAR-10-LT and CIFAR-100-LT datasets with different
imbalance ratios.

Dataset | CIFAR-10-LT | CIFAR-100-LT
---|---|---
Imbalance Factor |  100 |  200 |  100 |  200
Softmax | 78.7 | 74.4 | 45.3 | 41.0
Data Re-sampling |  |  |  |
Class Balanced Sampling (CBS) | 77.8 | 68.3 | 42.6 | 37.8
Loss Function Engineering |  |  |  |
Class Balanced Weighting (CBW) | 78.6 | 72.5 | 42.3 | 36.7
Class Balanced Loss (Cui et al. 2019) | 78.2 | 72.6 | 44.6 | 39.9
Focal Loss (Lin et al. 2017) | 77.1 | 71.8 | 43.8 | 40.2
LADE (Hong et al. 2021) | 81.8 | 76.9 | 45.4 | 43.6
LDAM (Cao et al. 2019b) | 78.9 | 73.6 | 46.1 | 41.3
Equalization Loss (Tan et al. 2020) | 78.5 | 74.6 | 47.4 | 43.3
Balanced Softmax (Ren et al. 2020) | 83.1 | 79.0 | 50.3 | 45.9
Decision Boundary Adjustment |  |  |  |
DisAlign (Zhang et al. 2021) | 78.0 | 71.2 | 49.1 | 43.6
Decouple-cRT (Kang et al. 2019) | 82.0 | 76.6 | 50.0 | 44.5
Decouple-LWS (Kang et al. 2019) | 83.7 | 78.1 | 50.5 | 45.3
Others |  |  |  |
BBN (Zhou et al. 2020) | 79.8 | - | 42.6 | -
Hybrid-SC (Wang et al. 2021c) | 81.4 | - | 46.7 | -
MARC | 85.3 | 81.1 | 50.8 | 47.4

#### Training Configuration

For a fair comparison, our experiments are conducted under the most commonly
used codebase of long-tailed studies: Open Long-Tailed Recognition (OLTR) (Liu
et al. 2019), using PyTorch (Paszke et al. 2019) framework. The model
structures used for CIFAR, ImageNet-LT, Places-LT and iNaturalist18 datasets
are ResNet32, ResNeXt50, ResNet152 and ResNet50, respectively. The model for
Places-LT is pre-trained on the full ImageNet-2012 dataset while models for
other datasets are trained from scratch. For ImageNet-LT, Places-LT, and
iNaturalist18, we train 90, 30, and 200 epochs in the first standard training
stage; and 10, 10, and 30 epochs in the second margin calibration stage, with
the batch size of 256, 128, and 256, respectively. For CIFAR-10-LT and
CIFAR-100-LT, the models are trained for 13,000 iterations with a batch size
of 512. We use the SGD optimizer with momentum 0.9 and weight decay $5e-4$ for
all datasets except for iNaturalist18 where the weight decay is $1e-4$. In the
standard training stage, we use a cosine learning rate schedule with an
initial value of 0.05 for CIFAR and 0.1 for other datasets, which gradually
decays to 0. In the margin calibration stage, we use a cosine learning rate
schedule with an initial learning rate starting from 0.05 to 0 for all
datasets. $\gamma$ is set to $1.2$ for all datasets. The hyper-parameters of
compared methods follow their paper. For fairness, we use the same pre-trained
model for decision boundary adjustment methods.

### Comparison with previous methods

In this section, we compare the performance of MARC to other recent works. We
select some recent methods from each of the following four categories for
comparison: data re-sampling, loss function engineering, decision boundary
adjustment, others. The standard training with the cross-entropy loss and
instance balance sampling is called Softmax in our results.

#### CIFAR

Table 2 presents results for CIFAR-10-LT and CIFAR-100-LT. MARC outperforms
all other methods in CIFAR-LT. Compared with other decision boundary
adjustment methods, MARC shows favorable results. The accuracy of MARC outruns
Decouple-LWS $1.6\%$, $3\%$, $0.3\%$ and $1.9\%$ on CIFAR-10-LT(100),
CIFAR-10-LT(200), CIFAR-100-LT(100) and CIFAR-100-LT(200) respectively, where
($\cdot$) denotes the imbalance factor. In addition, MARC outperforms all data
re-sampling and loss function engineering methods that my need laborious
hyper-parameter. The performance of well-designed networks such as BBN and
Hybrid-SC are also not as good as that of MARC.

Table 3: The performance on ImageNet-LT. Method | Many | Medium | Few | Overall
---|---|---|---|---
Softmax | 65.1 | 35.7 | 6.6 | 43.1
Loss Function Engineering |  |  |  |
Focal Loss (2017) | 64.3 | 37.1 | 8.2 | 43.7
Seesaw (2021b) | 67.1 | 45.2 | 21.4 | 50.4
Balanced Softmax (2020) | 62.2 | 48.8 | 29.8 | 51.4
LADE (2021) | 62.3 | 49.3 | 31.2 | 51.9
Decision Boundary Adjustment |  |  |  |
Decouple-$\pi$-norm (2019) | 59.1 | 46.9 | 30.7 | 49.4
Decouple-cRT (2019) | 61.8 | 46.2 | 27.4 | 49.6
Decouple-LWS (2019) | 60.2 | 47.2 | 30.3 | 49.9
DisAlign (2021) | 60.8 | 50.4 | 34.7 | 52.2
Others |  |  |  |
OLTR (2019) | 51.0 | 40.8 | 20.8 | 41.9
Causal Norm (2020) | 62.7 | 48.8 | 31.6 | 51.8
MARC | 60.4 | 50.3 | 36.6 | 52.3
Table 4: The performances on Places-LT, starting from an ImageNet pre-trained ResNet-152. Method | Many | Medium | Few | Overall
---|---|---|---|---
Softmax | 46.4 | 27.9 | 12.5 | 31.5
Loss Function Engineering |  |  |  |
Focal Loss (2017) | 41.1 | 34.8 | 22.4 | 34.6
Balanced Softmax (2020) | 42.0 | 38.0 | 17.2 | 35.4
LADE (2021) | 42.8 | 39.0 | 31.2 | 38.8
Decision Boundary Adjustment |  |  |  |
Decouple-LWS (2019) | 40.6 | 39.1 | 28.6 | 37.6
Decouple-$\pi$-norm (2019) | 37.8 | 40.7 | 31.8 | 37.9
DisAlign (2021) | 40.0 | 39.6 | 32.3 | 38.3
Others |  |  |  |
OLTR (2019) | 44.7 | 37.0 | 25.3 | 35.9
Causal Norm (2020) | 23.8 | 35.8 | 40.4 | 32.4
MARC | 39.9 | 39.8 | 32.6 | 38.4
Table 5: Top-1 accuracy on iNaturalist-LT. Method | Top-1 Accuracy
---|---
Softmax | 65.0
Loss Function Engineering |
Class Balanced Loss (2019) | 61.1
LDAM (2019a) | 64.6
Balanced Softmax (2020) | 69.8
LADE (2021) | 70.0
Decision Boundary Adjustment |
Decouple-$\pi$-norm (2019) | 69.3
Decouple-LWS (2019) | 69.5
DisAlign (2021) | 70.3
Others |
Casual Norm (2020) | 63.9
Hybrid-SC (2021c) | 68.1
MARC | 70.4

(a) Logits

(b) Margins

Figure 3: Logits and margins of CIFAR-100-LT(200), where ($\cdot$) is the
imbalance factor, before refers to the standard training results and after
refers to the results after our calibration method. The class indexes are
sorted by the number of samples(Head to tail).

(a) standard training.

(b) MARC.

Figure 4: Confusion matrix of the standard training and MARC on long-tailed
CIFAR-10-LT(200). The fading color of diagonal elements refers to the
disparity of the accuracy.

#### ImageNet-LT

We further evaluate MARC on the ImageNet-LT dataset. As Table 3 shows, MARC is
better than all loss function engineering methods. Compared with LADE,
although our overall accuracy is $0.4\%$ higher, the accuracy on the few-shot
classes is $5.4\%$ higher. The few-shot accuracy and overall accuracy of MARC
are $1.9\%$ and $0.1\%$ higher than DisAlign respectively. Our results are
quite surprising considering the simplicity of MARC.

#### Places-LT

For the Places-LT dataset, MARC achieves better performance than other
decision boundary adjustment methods. Though our overall accuracy is lower
than LADE, our few-shot accuracy is still $1.4\%$ higher than LADE. Though
MARC is not the best in Places-LT, the results of MARC are still competitive
compared with other methods.

#### iNaturalist-LT

Finally, we present the Top-1 accuracy results for iNaturalist-LT dataset in
Table 5. We can observe a similar trend that our proposed method wins all the
existing approaches and surpasses DisAlign by 0.1% absolute improvement.

(a) Margins of DisAlign.

(b) Logits of DisAlign.

(c) Margins of MARC.

(d) Logits of MARC.

Figure 5: The values of margins and logits for each class on the ImageNet-LT
dataset. The class indexes are sorted by the number of samples (Head to tail).
Before refers to the standard training results and after refers to the results
after using the decision boundary adjustment method.

### Further analysis

In this section, we conduct different experiments for further analysis. To be
concrete, we empirically show that MARC can achieve more balanced margins and
logits compared with DisAlign. Moreover, the class-wise accuracy of MARC is
much better than the standard training baseline model on CIFAR-LT, which
indicates that we can alleviate the biased prediction problem and reduce the
performance gap between the head classes and the tail classes.

#### Visualization of the margin and logit

In this subsection, we visualize the values of margins and logits for each
class to show the effect of MARC. As illustrated both in Figure 3, Figure 5(c)
and Figure 5(d), before margin calibration, the margins and the logits are
biased, i.e. the head classes tend to have much larger margins and logits than
tail classes. We believe the bias in margins and logits will lead to biased
predictions in long-tailed visual recognition. The biased margins and logits
become more balanced after the margin calibration. This result proves that we
can get more unbiased predictions by calibrating the margin. Moreover, as
shown in Figure 5(a) and Figure 5(b), MARC will obtain more balanced margins
and gradients than DisAlign. The instability of DisAlign may be caused by
their heuristic design of the combination of the calibrated logits and the
origin logits.

#### Class-wise performance on CIFAR-LT

As we can see in Figure 4, after our margin calibration method, the
performance on tail classes is improved while that on head classes is not
severely affected. More intuitively, Figure 6 shows the class-wise
performance. The accuracy of the tail class is much higher than that of the
head class. The performance degradation on head classes may be caused by the
false positive predictions on head classes, i.e., the standard training method
tends to classify tail classes as head classes, resulting in high accuracy on
the head classes. The bad performance on tail classes when using standard
training also proves this. In addition, the overall accuracy and F1-score show
that MARC alleviates the biased prediction problem.

(a) CIFAR-10-LT(200).

(b) CIFAR-100-LT(200).

Figure 6: The class-wise performance of CIFAR-10-LT(200) and
CIFAR-100-LT(200). Before refers to the standard training results and after
refers to the results after MARC.

(a) F1 scores on CIFAR-LT.

(b) Overall acc with different $\gamma$.

Figure 7: (a) F1 scores on CIFAR-10-LT(200) and on CIFAR-100-LT(200). (b) Overall accuracy on CIFAR-100-LT(200) when using different $\gamma$. Table 6: Top-1 accuracy on CIFAR-100-LT(200) with different standard pre-trained models. Standard pre-trained dataset | Top-1 Accuracy
---|---
CIFAR-100-LT(200) | 47.1
CIFAR-100-LT(100) | 50.7
CIFAR-100-LT(50) | 54.5

#### Effects of different $\gamma$

To explore the effect of different $\gamma$, we also conduct experiments and
visualize the performances on all CIFAR-100-LT(200). The results are shown in
Figure 7(b). We can observe that compared $1.2$ is the best compared with
other values. $\gamma$ can not be too large since in this way the weight for
head classes is too small. It is well noting that MARC also achieves $45.2\%$
accuracy when $\gamma$ is $0$. This means MARC still works even if we do not
use any loss re-weighting techniques in the second stage. For other datasets,
we directly use $1.2$ for $\gamma$.

#### Effects of different standard pre-trained model

We use different standard pre-trained models on CIFAR-100-LT(200) to explore
their effects. As Table 6 illustrates, as the pre-trained dataset gets more
balanced, the performance of our margin calibration method gets better. This
shows that when comparing decision boundary methods, using the same standard
training model is very important for fairness. And the codebase will also
affect the standard training. This is the reason why the result of DisAlign in
our paper is inconsistent with the original paper since we cannot get the same
standard pre-trained model they used. So instead, we use our own standard pre-
trained model for MARC and DisAlign for fairness. Table 6 also demonstrates
that the margin calibration method can achieve better performance when given
better representations. So our future works include how to get better
representations.

#### Comparison of the trainable parameters of decision boundary adjustment
methods

As shown in Table 7, MARC achieves the best performance among other compared
methods on CIFAR-100-LT(200) and ImageNet-LT. Even though our trainable
parameters are more than Decouple-LWS, our performance is better. Besides, it
is surprising that MARC obtains such a favorable performance with only so few
parameters.

Table 7: Comparison of the trainable parameters (#Param.) of the learnable
decision boundary adjustment methods. $p$ is the feature dimension and $K$ is
the number of classes (ResNeXt50 for ImageNet-LT: $p=2048,K=1000$).

Methods | CIFAR-100-LT | ImageNet-LT | #Param.
---|---|---|---
Decouple-cRT | 44.5 | 49.6 | $pK+K$
Decouple-LWS | 45.3 | 49.9 | $K$
DisAlign | 43.6 | 52.2 | $p+2K$
MARC | 47.4 | 52.3 | $2K$

## Conclusions

This paper studied the long-tailed visual recognition problem. Specifically,
we found that head classes tend to have much larger margins and logits than
tail classes. Motivated by our findings, we proposed a margin calibration
function with only $2K$ learnable parameters to obtain the unbiased logits in
long-tailed visual recognition. Even though our method is very simple to
implement, extensive experiments show that MARC achieves favorable results
compared with previous methods.

We hope that our study on logits and margins could provide experience for
future research. In the future, we plan to explore better representations for
long-tailed decision adjustment methods. At the same time, we plan to study
the theory behind the logits and margins to provide a more theoretical
analysis.

## References

  * Ando and Huang (2017) Ando, S.; and Huang, C. Y. 2017.  Deep over-sampling framework for classifying imbalanced data.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , 770–785. Springer.
  * Buda, Maki, and Mazurowski (2018) Buda, M.; Maki, A.; and Mazurowski, M. A. 2018.  A systematic study of the class imbalance problem in convolutional neural networks.  _Neural Networks_ , 106: 249–259.
  * Byrd and Lipton (2019) Byrd, J.; and Lipton, Z. 2019.  What is the effect of importance weighting in deep learning?  In _International Conference on Machine Learning_ , 872–881. PMLR.
  * Cao et al. (2019a) Cao, K.; Wei, C.; Gaidon, A.; Arechiga, N.; and Ma, T. 2019a.  Learning imbalanced datasets with label-distribution-aware margin loss.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , 1567–1578.
  * Cao et al. (2019b) Cao, K.; Wei, C.; Gaidon, A.; Arechiga, N.; and Ma, T. 2019b.  Learning imbalanced datasets with label-distribution-aware margin loss.  _arXiv preprint arXiv:1906.07413_.
  * Chawla et al. (2002) Chawla, N. V.; Bowyer, K. W.; Hall, L. O.; and Kegelmeyer, W. P. 2002.  SMOTE: synthetic minority over-sampling technique.  _Journal of artificial intelligence research_ , 16: 321–357.
  * Cui et al. (2019) Cui, Y.; Jia, M.; Lin, T.-Y.; Song, Y.; and Belongie, S. 2019.  Class-balanced loss based on effective number of samples.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 9268–9277.
  * Drummond, Holte et al. (2003) Drummond, C.; Holte, R. C.; et al. 2003.  C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling.  In _Workshop on learning from imbalanced datasets II_ , volume 11, 1–8. Citeseer.
  * Ganganwar (2012) Ganganwar, V. 2012.  An overview of classification algorithms for imbalanced datasets.  _International Journal of Emerging Technology and Advanced Engineering_ , 2(4): 42–47.
  * Han, Wang, and Mao (2005) Han, H.; Wang, W.-Y.; and Mao, B.-H. 2005.  Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning.  In _International conference on intelligent computing_ , 878–887. Springer.
  * He and Garcia (2009) He, H.; and Garcia, E. A. 2009.  Learning from imbalanced data.  _IEEE Transactions on knowledge and data engineering_ , 21(9): 1263–1284.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Hong et al. (2021) Hong, Y.; Han, S.; Choi, K.; Seo, S.; Kim, B.; and Chang, B. 2021.  Disentangling Label Distribution for Long-tailed Visual Recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6626–6636.
  * Jamal et al. (2020) Jamal, M. A.; Brown, M.; Yang, M.-H.; Wang, L.; and Gong, B. 2020.  Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7610–7619.
  * Kang et al. (2019) Kang, B.; Xie, S.; Rohrbach, M.; Yan, Z.; Gordo, A.; Feng, J.; and Kalantidis, Y. 2019.  Decoupling Representation and Classifier for Long-Tailed Recognition.  In _International Conference on Learning Representations_.
  * Khan et al. (2017) Khan, S. H.; Hayat, M.; Bennamoun, M.; Sohel, F. A.; and Togneri, R. 2017.  Cost-sensitive learning of deep feature representations from imbalanced data.  _IEEE transactions on neural networks and learning systems_ , 29(8): 3573–3587.
  * Kim and Kim (2020) Kim, B.; and Kim, J. 2020.  Adjusting decision boundary for class imbalanced learning.  _IEEE Access_ , 8: 81674–81685.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Lin et al. (2017) Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; and Dollár, P. 2017.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , 2980–2988.
  * Liu et al. (2019) Liu, Z.; Miao, Z.; Zhan, X.; Wang, J.; Gong, B.; and Yu, S. X. 2019.  Large-scale long-tailed recognition in an open world.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2537–2546.
  * Menon et al. (2020) Menon, A. K.; Jayasumana, S.; Rawat, A. S.; Jain, H.; Veit, A.; and Kumar, S. 2020\.  Long-tail learning via logit adjustment.  In _International Conference on Learning Representations_.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32: 8026–8037.
  * Pouyanfar et al. (2018) Pouyanfar, S.; Tao, Y.; Mohan, A.; Tian, H.; Kaseb, A. S.; Gauen, K.; Dailey, R.; Aghajanzadeh, S.; Lu, Y.-H.; Chen, S.-C.; et al. 2018.  Dynamic sampling in convolutional neural networks for imbalanced data classification.  In _2018 IEEE conference on multimedia information processing and retrieval (MIPR)_ , 112–117. IEEE.
  * Ren et al. (2020) Ren, J.; Yu, C.; Sheng, S.; Ma, X.; Zhao, H.; Yi, S.; and Li, H. 2020.  Balanced meta-softmax for long-tailed visual recognition.  _arXiv preprint arXiv:2007.10740_.
  * Shen, Lin, and Huang (2016) Shen, L.; Lin, Z.; and Huang, Q. 2016.  Relay backpropagation for effective learning of deep convolutional neural networks.  In _European conference on computer vision_ , 467–482. Springer.
  * Simonyan and Zisserman (2014) Simonyan, K.; and Zisserman, A. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Tan et al. (2020) Tan, J.; Wang, C.; Li, B.; Li, Q.; Ouyang, W.; Yin, C.; and Yan, J. 2020.  Equalization loss for long-tailed object recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 11662–11671.
  * Tang, Huang, and Zhang (2020) Tang, K.; Huang, J.; and Zhang, H. 2020.  Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect.  _Advances in Neural Information Processing Systems_ , 33.
  * Van Horn et al. (2018) Van Horn, G.; Mac Aodha, O.; Song, Y.; Cui, Y.; Sun, C.; Shepard, A.; Adam, H.; Perona, P.; and Belongie, S. 2018.  The inaturalist species classification and detection dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 8769–8778.
  * Van Horn and Perona (2017) Van Horn, G.; and Perona, P. 2017.  The devil is in the tails: Fine-grained classification in the wild.  _arXiv preprint arXiv:1709.01450_.
  * Wang et al. (2021a) Wang, J.; Lukasiewicz, T.; Hu, X.; Cai, J.; and Xu, Z. 2021a.  RSG: A Simple but Effective Module for Learning Imbalanced Datasets.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3784–3793.
  * Wang et al. (2021b) Wang, J.; Zhang, W.; Zang, Y.; Cao, Y.; Pang, J.; Gong, T.; Chen, K.; Liu, Z.; Loy, C. C.; and Lin, D. 2021b.  Seesaw loss for long-tailed instance segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9695–9704.
  * Wang et al. (2021c) Wang, P.; Han, K.; Wei, X.-S.; Zhang, L.; and Wang, L. 2021c.  Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 943–952.
  * Wang, Ramanan, and Hebert (2017) Wang, Y.-X.; Ramanan, D.; and Hebert, M. 2017.  Learning to model the tail.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , 7032–7042.
  * Yang, Yang, and Wang (2009) Yang, C.-Y.; Yang, J.-S.; and Wang, J.-J. 2009.  Margin calibration in SVM class-imbalanced learning.  _Neurocomputing_ , 73(1-3): 397–411.
  * Yang and Xu (2020) Yang, Y.; and Xu, Z. 2020.  Rethinking the Value of Labels for Improving Class-Imbalanced Learning.  In _NeurIPS_.
  * Yin et al. (2019) Yin, X.; Yu, X.; Sohn, K.; Liu, X.; and Chandraker, M. 2019.  Feature transfer learning for face recognition with under-represented data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5704–5713.
  * Yu et al. (2020) Yu, H.; Zhang, N.; Deng, S.; Yuan, Z.; Jia, Y.; and Chen, H. 2020.  The devil is the classifier: Investigating long tail relation classification with decoupling analysis.  _arXiv preprint arXiv:2009.07022_.
  * Zhang et al. (2021) Zhang, S.; Li, Z.; Yan, S.; He, X.; and Sun, J. 2021.  Distribution Alignment: A Unified Framework for Long-tail Visual Recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2361–2370.
  * Zhou et al. (2020) Zhou, B.; Cui, Q.; Wei, X.-S.; and Chen, Z.-M. 2020.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9719–9728.
  * Zhou et al. (2017) Zhou, B.; Lapedriza, A.; Khosla, A.; Oliva, A.; and Torralba, A. 2017.  Places: A 10 million image database for scene recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(6): 1452–1464.
  * Zhou and Liu (2005) Zhou, Z.-H.; and Liu, X.-Y. 2005.  Training cost-sensitive neural networks with methods addressing the class imbalance problem.  _IEEE Transactions on knowledge and data engineering_ , 18(1): 63–77.