# Towards Overcoming False Positives in Visual Relationship Detection

Daisheng Jin1,2 Equal contribution Xiao Ma311footnotemark: 1 Chongzhi
Zhang1,211footnotemark: 1 Yizhuo Zhou4 Jiashu Tao3
Mingyuan Zhang1 Corresponding author<EMAIL_ADDRESS>Haiyu Zhao1
Shuai Yi1 Zhoujun Li2 Xianglong Liu2 Hongsheng Li5

1 SenseTime Research  2 Beihang University  3 National University of Singapore
4 ByteDance AI Lab  5 Multimedia Laboratory  The Chinese University of Hong
Kong

###### Abstract

In this paper, we investigate the cause of the high false positive rate in
Visual Relationship Detection (VRD). We observe that during training, the
relationship proposal distribution is highly imbalanced: most of the negative
relationship proposals are easy to identify, e.g., the inaccurate object
detection, which leads to the under-fitting of low-frequency difficult
proposals. This paper presents Spatially-Aware Balanced negative pRoposal
sAmpling (SABRA), a robust VRD framework that alleviates the influence of
false positives. To effectively optimize the model under imbalanced
distribution, SABRA adopts Balanced Negative Proposal Sampling (BNPS) strategy
for mini-batch sampling. BNPS divides proposals into 5 well defined sub-
classes and generates a balanced training distribution according to the
inverse frequency. BNPS gives an easier optimization landscape and
significantly reduces the number of false positives. To further resolve the
low-frequency challenging false positive proposals with high spatial
ambiguity, we improve the spatial modeling ability of SABRA on two aspects: a
simple and efficient multi-head heterogeneous graph attention network (MH-GAT)
that models the global spatial interactions of objects, and a spatial mask
decoder that learns the local spatial configuration. SABRA outperforms SOTA
methods by a large margin on two human-object interaction (HOI) datasets and
one general VRD dataset.

##  1 Introduction

Visual Relationship Detection (VRD) is an important visual task that bridges
the gap between middle-level visual perception, e.g., object detection, and
high-level visual understanding, e.g., image captioning [42, 9], and visual
question answering [31]. General VRD aims to understand the interaction
between two arbitrary objects in the scene. Human-Object Interaction (HOI), as
a specific case of VRD, focuses on understanding the interaction between
humans and objects, e.g., woman-cut-cake.

Existing VRD methods focus on building powerful feature extractors for each
[_subject_ , _object_] pair, predicting the _predicate_ between the _subject_
and the _object_ , and outputting [_subject_ , _predicate_ , _object_] triplet
predictions. Some prior works model subject and object relationship
independently [8, 35], which loses the global context and is susceptible to
inaccurate detections. Some recent works incorporate the union bounding boxes
of the [_subject_ , _object_] as additional features to provide additional
spatial information [34, 43] or use the graph neural networks (GNNs) to better
extract global object relationships [52, 18, 33]. However, for any visual
relationship detector, 90% of its predictions are false positives because
according to our statistical result, an image normally contains less than 10
positive relationships, while detectors consider the top-$100$ predictions.

In addition to the high spatial ambiguity that prior works have been focusing
on, we observe that another critical reason for the high false positive rate
in VRD is the difficult optimization landscape caused by extremely imbalanced
relationship proposal distribution [36]. A typical visual relationship
detector takes as input a set of [_subject_ , _object_] proposals. However,
most of them are negative proposals, i.e., no relationship exists for given
[_subject_ , _object_] pairs. We observe that the negative proposals are
introduced mainly by two sources with different levels of difficulties:
inaccurate single object detections and the incorrect [_subject_ , _object_]
associations. Inaccurate detections, which contribute to more than 90% of the
negative proposals (first two columns in Fig. 1.a), can be easily identified
by the appearance feature of the single object detections. Incorrect
associations contribute to less than 10% of negative proposals (last three
columns in Fig. 1.a) but are significantly more difficult: the subject might
be related to other objects, but the selected [_subject_ , _object_] pair can
be unrelated, which requires careful inspection of the global context.

|
---|---
(a) Imbalanced negative proposal distribution | (b) Influence of ambiguous context
Figure 1: (a) VRD normally gives an extremely imbalanced negative proposal
distribution, where $S_{\textrm{neg}}^{1:5}$ denotes 5 different types of
negative proposals. Detailed definition of $S_{\textrm{neg}}^{1:5}$ is given
by Eqn. 6. (b) Contextual objects introduce ambiguous information to the
relationship classification, e.g., the model thinks the woman on the left is
cutting the cake by mistake.

We present Spatially-Aware Balanced negative pRoposal sAmpling (SABRA), a
robust and general VRD framework that alleviates the influence of false
positives for both HOI and general VRD tasks. According to the two sources of
negative proposals, i.e., inaccurate detections, and incorrect associations,
we introduce a division of negative proposals into 5 sub-classes
$S_{\textrm{neg}}^{1:5}$. For each single object detection, we consider (1) if
it is an accurate detection, and (2) if it is in any relationship with a
different object. Specifically, $S_{\textrm{neg}}^{1:2}$ cover the inaccurate
detections, and $S_{\textrm{neg}}^{3:5}$ discuss the incorrect associations.
From $S_{\textrm{neg}}^{1}$ to $S_{\textrm{neg}}^{5}$, the sample size
decreases, and the classification difficulty increases, because detecting the
false positives according to the accuracy of detection is no longer
sufficient, and careful understanding about object relationships becomes
necessary for the task. Detailed definitions can be found in Eqn. 6. As
visualized in Fig. 1.a, the sample sizes of 5 negative proposal sub-classes
give a highly imbalanced distribution, which degrades the performance of data-
driven VRD algorithms. Inspired by the learning under imbalanced distribution
literature [47, 3], we alleviate the optimization difficulty by Balanced
Negative Proposal Sampling scheme. BNPS computes the statistics of each class
and performs a simple yet effective Class Balanced Sampling [38] for balanced
data distribution. Balanced negative proposal sampling significantly reduces
the number of false positive occurrences. For low-frequency difficult classes
$S_{\textrm{neg}}^{3:5}$, e.g., Fig. 1.b, we further improve the spatial
modeling of SABRA on two aspects. For the global context understanding, SABRA
extends the existing GNN-based methods [50, 33] with a heterogeneous message
passing scheme which effectively addresses the distribution divergence between
different features. For local spatial configuration, SABRA learns a position-
aware embedding vector by predicting the locations in each [_subject_ ,
_object_] pair.

In our experiment, we evaluate SABRA on two commonly used HOI datasets,
V-COCO, and HICO-DET, and one general object relationship detection dataset,
VRD. We show that SABRA significantly outperforms SOTA methods by a large
margin. We also visualize the results and show that SABRA effectively reduces
the false positives in VRD and misclassification in terms of the spatial
ambiguity.

##  2 Related Works

###  2.1 Visual Relationship Detection

VRD is an important middle-level task bridging low-level visual recognition
with high-level visual understanding. Earlier works focus on Support Vector
Machines (SVMs) with discriminative features for relationship classification
[6, 7]. However, handcrafting features is difficult and harmful to the final
performance of VRD algorithms. With the advances of deep learning, data-driven
approaches are widely adopted for VRD. Specifically, Convolutional Neural
Networks (CNNs) are used for automatic feature extraction and information
fusion, which achieved great improvement in VRD [43, 49, 46]. Graph Neural
Networks (GNNs) further improve the feature extraction process by explicitly
modeling the instance-wise interactions between objects [50, 33]. Due to the
nature of VRD tasks, additional information has been introduced as auxiliary
training signals, such as language priors [29], prior interactiveness
knowledge of objects [23], and action co-occurrence knowledge [22]. In
comparison with the existing methods, SABRA is the first to identify the
importance of false positives in VRD tasks and has significantly outperformed
SOTA methods in our experiments.

###  2.2 Learning under Imbalanced Distribution

Real-world data is imbalanced by nature: a few high-frequency classes
contribute to most of the samples, while a large number of low-frequency
classes are under-represented in data. Standard imbalanced learning techniques
include data re-balancing [1, 2, 3], loss function engineering [4, 16] and
meta-learning [20]. VRD, as a common computer vision task, also suffers from
the imbalanced problem [27, 22]. Specifically, [22] considers the imbalance of
relationship imbalance, i.e., the imbalance of positive samples, and uses
action co-occurrence to provide additional labels. However, none of the
existing methods consider the imbalance of the negative [_subject_ , _object_]
proposals, which commonly exists in VRD settings. By re-balancing the
proposals, SABRA significantly improves the overall performance of VRD
algorithms.

Figure 2: SABRA classifies the false positive [subject, object] proposals into
5 sub-categories $S_{\textrm{neg}}^{1:5}$ and use Balanced-Sampling to re-
balance the categorical distribution
$\textrm{Cat}(S_{\textrm{pos}},S_{\textrm{neg}}^{1:5})$. To reduce the false
positive relationship predictions caused by spatial ambiguity, SABRA first
uses MH-GAT to capture the global context and then learns to predict the mask
of [subject, object].

###  2.3 Spatial Information

Spatial information is key to understanding the relationship between objects,
e.g., “woman sitting on the chair”. Prior methods fuse spatial information
with positional embedding, which normalizes the absolute or relative
coordinates of the subject, object, and union bounding boxes [57]. However,
simple positional embedding implicitly captures spatial information with
position coordinates as inputs to networks, which is unable to capture the
explicit spatial configuration in the feature space. [13] introduces binary
masks which explicitly specify the subject and object positions, implicitly
specifying the spatial configuration by concatenating with union features.
With the recent advances in graph neural networks, the relevant positional
information can be captured by message passing between instances [59]. The
implicit message passing, nevertheless, loses the contextual grounding of the
[_subject_ , _object_] pair. In contrast, the spatial mask decoder learns to
predict the positional information of the [_subject_ , _object_] , while
capturing the relevant features by end-to-end learning.

##  3 SABRA

We introduce Spatially-Aware Balanced negative pRoposal sAmpling (SABRA)
framework for effective visual relationship detection against false positives.
The SABRA framework is shown in Fig. 2. SABRA simplifies the difficult
optimization landscape of VRD tasks caused by imbalanced proposal distribution
by Balanced Negative Proposal Sampling (BNPS) scheme. BNPS divides the
imbalanced negative proposals into sub-classes and creates balanced batches
for model optimization. In addition, SABRA further improves the existing GNN-
based spatial feature extraction pipeline from two perspectives: (1) a
heterogeneous message passing scheme for global scene understanding; (2) a
spatial mask decoder for local spatial configuration learning. SABRA combines
these objectives and achieves the SOTA performance on multiple datasets.

###  3.1 Balanced Negative Proposal Sampling

####  3.1.1 Negative Relationship Proposals

Despite abundant researches in improving feature extraction for positive
[_subject_ , _predicate_ , _object_] pairs in VRD, effective learning under a
large number of negative proposals is never explored, which has a greater
impact on the performance of the VRD task.

Suppose we acquire two sets of bounding boxes
$B_{\textrm{subject}},B_{\textrm{object}}$ from the object detector, where
$B_{\textrm{subject}}$ contains all the bounding boxes for subjects, and
$B_{\textrm{object}}$ contains all the bounding boxes which probably contain
an object. A relationship proposal is defined as a tuple $(b_{1},b_{2})$. We
define the set of relationship proposals $S$ as follows:

$S=\\{(b_{1},b_{2})\mid b_{1}\in B_{\textrm{subject}},b_{2}\in
B_{\textrm{object}}\\}.$ (1)

$S$ can be divided into two disjoint subsets $S_{\textrm{pos}}$ and
$S_{\textrm{neg}}$,

$S=S_{\textrm{pos}}\cup S_{\textrm{neg}},\quad S_{\textrm{pos}}\cap
S_{\textrm{neg}}=\emptyset,$ (2)

where $S_{\textrm{pos}}$ denotes the set of positive proposals that correspond
to the ground truth, and $S_{\textrm{neg}}$ stands for the set of wrong
proposals, i.e., negative proposals. Specifically,

$\displaystyle S_{\textrm{pos}}=\\{(b_{1},$ $\displaystyle
b_{2})\mid\exists(g_{1},g_{2})\in\textrm{GT}_{\textrm{asso}},$
$\displaystyle\min(\textrm{IoU}(b_{1},g_{1}),\textrm{IoU}(b_{2},g_{2}))\geq
0.5\\},$ (3)

where $\textrm{GT}_{\textrm{asso}}$ is the set of ground truth [_subject_ ,
_object_] pairs and IoU stands for the intersection of the union.

Given that common VRD algorithms choose top-$K$ detections with highest
confidence to generate relationship proposals, we have $|S|=K^{2}$. However,
consider that there are normally less than 10 positive proposals for an image,
i.e., $|S_{\textrm{pos}}|<10$, when $K=100$, $|S_{\textrm{pos}}|/|S|<10^{-3}$.
Learning under such an extremely imbalanced data distribution is naturally
difficult.

One commonly used VRD mini-batch sampling strategy [28], inspired by object
detection, is to sample 25% of the mini-batch from $S_{\textrm{pos}}$ and
construct the rest 75% by $S_{\textrm{neg}}$. Although it ensures the presence
of positive proposals, it overlooks the complex distribution of negative
proposals $S_{\textrm{neg}}$. Different from object detection, negative
proposals in VRD are caused mainly by two reasons: inaccurate detections and
incorrect associations. Inaccurate detections cause negative proposals by
generating inaccurate bounding boxes with $\textrm{IoU}(b_{1},g_{1})<0.5$.
This type of negative proposals can be easily identified by the visual
appearance feature of single object detections alone. On the other hand,
incorrect associations cause negative proposals in a more complicated way. Due
to the limited number of accurate detections $D_{\textrm{pos}}$, proposals
which consist of both correct detections from $D_{\textrm{pos}}$ only
contribute a small portion of $S$. For detections with at least one positive
relationship, $D_{\textrm{rel}}=\\{b\mid\exists b^{\prime},(b,b^{\prime})\in
S_{\textrm{pos}}\vee(b^{\prime},b)\in S_{\textrm{pos}}\\}$, we have
$|D_{\textrm{rel}}|/|D_{\textrm{pos}}|<0.2$. Moreover, consider a proposal
$(b_{1},b_{2})$ where $b_{1}\in D_{\textrm{rel}}$, i.e., $\exists
b^{\prime}_{2},(b_{1},b^{\prime}_{2})\in S_{\textrm{pos}}$. The relationship
prediction of $(b_{1},b_{2})$ will be influenced by the positive proposal
$(b_{1},b^{\prime}_{2})$, which introduces extra confusion. Thus, the
proposals with bounding boxes from $D_{\textrm{rel}}$ are not only under-
represented but more confusing. In summary, an imbalanced proposal
distribution, where the difficulty of each proposal is negatively correlated
with its population size, commonly exists in VRD tasks and would degrade the
overall performance of VRD algorithms.

####  3.1.2 Balanced Negative Proposal Sampling

Our solution is motivated by learning with data imbalance [1, 2]. We propose a
balanced negative proposal sampling strategy considering 1 positive class and
5 negative classes. We first define two helper functions, $f_{\textrm{box}}$
and $f_{\textrm{rel}}$:

$\displaystyle f_{\textrm{box}}(b)=\begin{cases}1,&\textrm{if}\;\exists
g\in\textrm{GT}_{\textrm{box}},\textrm{IoU}(b,g)\geq 0.5,\\\
0,&\textrm{otherwise}.\end{cases}$ (4) $\displaystyle
f_{\textrm{rel}}(b)=\begin{cases}1,&\textrm{if}\;\exists(g_{1},g_{2})\in\textrm{GT}_{\textrm{rel}},\textrm{maxIoU}(b,(g_{1},g_{2}))\geq
0.5,\\\ 0,&\textrm{otherwise},\end{cases}$ (5)

where $\textrm{GT}_{\textrm{box}}$ denotes the set of ground truth bounding
boxes, $\textrm{GT}_{\textrm{rel}}$ denotes the set of ground truth
relationships, and
$\textrm{maxIoU}(b,(g_{1},g_{2}))=\max(\textrm{IoU}(b,g_{1}),\textrm{IoU}(b,g_{2}))$.
Intuitively, $f_{\textrm{box}}(b)$ indicates if a bounding box $b$ is a
positive bounding box, and $f_{\textrm{rel}}(b)$ denotes if bounding box $b$
belongs to a positive relationship. Next, we divide $S_{\textrm{neg}}$
following these two principles: (1) simple proposals are caused by inaccurate
bounding boxes; (2) difficult proposals are introduced by incorrect [_subject_
, _object_] associations. We formulate the 5 sub-classes:

$\displaystyle S_{\textrm{neg}}^{1}=$ $\displaystyle\\{(b_{1},b_{2})|\neg
f_{\textrm{box}}(b_{1})\land\neg f_{\textrm{box}}(b_{2})\\},$ $\displaystyle
S_{\textrm{neg}}^{2}=$ $\displaystyle\\{(b_{1},b_{2})|(\neg
f_{\textrm{box}}(b_{1})\land f_{\textrm{box}}(b_{2}))$
$\displaystyle\vee(f_{\textrm{box}}(b_{1})\land\neg
f_{\textrm{box}}(b_{2}))\\},$ $\displaystyle S_{\textrm{neg}}^{3}=$
$\displaystyle\\{(b_{1},b_{2})|f_{\textrm{box}}(b_{1})\land\neg
f_{\textrm{rel}}(b_{1})\land f_{\textrm{box}}(b_{2})\land\neg
f_{\textrm{rel}}(b_{2})\\},$ $\displaystyle S_{\textrm{neg}}^{4}=$
$\displaystyle\\{(b_{1},b_{2})|(f_{\textrm{rel}}(b_{1})\land
f_{\textrm{box}}(b_{2})\land\neg f_{\textrm{rel}}(b_{2}))\vee$
$\displaystyle(f_{\textrm{box}}(b_{1})\land\neg f_{\textrm{rel}}(b_{1})\land
f_{\textrm{rel}}(b_{2}))\\},$ $\displaystyle S_{\textrm{neg}}^{5}=$
$\displaystyle\\{(b_{1},b_{2})|f_{\textrm{rel}}(b_{1})\land
f_{\textrm{rel}}(b_{2})\land(b_{1},b_{2})\notin S_{\textrm{pos}}\\}.$ (6)

This divides the negative proposals into 5 sub-classes: (1) both detections
are incorrect; (2) one detection is incorrect; (3) both detections are
correct, but they belong to no [_subject_ , _predicate_ , _object_] triplet;
(4) both detections are correct, but only one of them appears in [_subject_ ,
_predicate_ , _object_] triplets; (5) both detections are correct, but they
appear in two disjoint sets of [_subject_ , _predicate_ , _object_] triplets,
which is the minority of the negative proposals but is the most difficult to
learn. Together with the $S_{\textrm{pos}}$, we divide $S$ into 6 classes,
$S=\bigcup\limits_{i=1}^{5}S_{\textrm{neg}}^{i}\cup S_{\textrm{pos}}$, which
better describe the underlying distribution.

We adopt the most standard Balanced Sampling scheme [38] in learning with
imbalance literature to address the imbalance of negative proposals
$S_{\textrm{neg}}$, and we introduce Balanced Negative Proposal Sampling
(BNPS). During training time, for each image, we find the top-$K$ object
detections, construct $K\times K$ relationship proposals, and count the number
of samples for each class. For each proposal $s_{i}$, we assign a weight
$w_{i}$:

$w_{i}=\begin{cases}0.25/|S_{\textrm{pos}}|,&\textrm{if}\;s_{i}\in
S_{\textrm{pos}},\\\ 0.15/|S_{\textrm{neg}}^{j}|,&\textrm{if}\;s_{i}\in
S_{\textrm{neg}}^{j},j=1,\dots,5\end{cases}$ (7)

where we keep the weight, 0.25, of positive proposals, while re-balancing the
weight of negative proposals, which improves the prediction of low-frequency
difficult classes.

###  3.2 Spatially-Aware Embedding Learning

For low-frequency difficult false positives, GNNs generally improve the global
scene understanding [50, 33, 23] and learning the spatial position of objects
improves the local feature representation of [_subject_ , _object_] pairs [57,
24, 59]. For global features, SABRA improves the graph representation learning
in VRD by Multi-head Heterogeneous Graph Attention networks (MH-GAT) for
message passing. For local features, SABRA uses a Spatial Mask Decoder (SMD)
to explicitly model the positional configuration of a [_subject_ , _object_]
pair.

####  3.2.1 Multi-head Heterogeneous Graph Attention

We introduce Multi-head Heterogeneous Graph Attention Networks (MH-GAT) for
effective learning of the global context. One limitation of widely used Graph
Attention networks is that it assumes a homogeneous node feature distribution
and uses the same set of parameters for learning embeddings for all nodes
[41]. However, the features of subjects, objects, and union bounding boxes
often follow different distributions. Directly using GAT might degrade the
final performance. MH-GAT extends GAT by performing message passing on
heterogeneous nodes, i.e., subject, object, and union feature, in VRD. We
treat each object in the image as a node $i$ with feature $x_{i}$, and
construct a fully connected graph $G=(V,E)$, where $|V|=N$ is the set of nodes
and $E=\\{(i,j)\mid i,j=1,2,\dots,N\\}$. We treat the feature for edge $(i,j)$
as the union feature $e_{ij}$. The MH-GAT message passing mechanism is as
follows:

$\displaystyle x^{\prime}_{i}$ $\displaystyle=x_{i}+\sum\limits_{j\in
N(i)\cup\\{i\\}}\alpha_{ij}f_{m}([x_{i},x_{j},e_{ij}]),$ (8)
$\displaystyle\alpha_{ij}$
$\displaystyle=\frac{\exp(g_{ij})}{\sum\limits_{k\in
N(i)\cup\\{i\\}}\exp(g_{ik})},$ (9) $\displaystyle g_{ij}$
$\displaystyle=[f_{s}(x_{i}),f_{t}(x_{j}),f_{e}(e_{ij})],$ (10)

where $\alpha_{ij}$ is the attention weight for edge $(i,j)$, $f_{m}$,
$f_{s}$, $f_{t}$ and $f_{e}$ are functions for message embedding, source node
feature embedding, target node feature embedding, and edge feature embedding
respectively. In contrast to most of the GNNs, when computing the attention
values, we process source features, target features, edge features with
different functions. This allows modeling the heterogeneous distribution
between different nodes. This formulation also enables our model to pass
diverse features from a single source node to different target nodes. Besides,
we apply a multi-head attention mechanism, which provides a better modeling
ability to capture the complex attentions among objects.

| V-COCO | HICO-DET
---|---|---
|  |  | Default | Known Objects
Method | Backbone | AProle | Full | Rare | Non-Rare | Full | Rare | Non-Rare
iCAN [11] | ResNet50 | 45.30 | 14.84 | 10.45 | 16.15 | 16.26 | 11.33 | 17.73
Contextual Attention [45] | ResNet50 | 47.30 | 16.24 | 11.16 | 17.75 | 17.73 | 12.78 | 19.21
In-GraphNet [51] | ResNet50 | 48.90 | 17.72 | 12.93 | 13.91 | - | - | -
VCL [17] | ResNet50 | 48.30 | 19.43 | 16.55 | 20.29 | 22.00 | 19.09 | 22.87
PD-Net [58] | ResNet50 | 52.30 | 20.76 | 15.68 | 22.28 | 25.58 | 19.93 | 27.28
SABRA(Ours) | ResNet50 | 53.57 | 23.48 | 16.39 | 25.59 | 28.79 | 22.75 | 30.54
InteractNet [14] | ResNet50-FPN | 40.00 | 9.94 | 7.16 | 10.77 | - | - | -
PMFNet [43] | ResNet50-FPN | 52.00 | 17.46 | 15.65 | 18.00 | 20.34 | 17.47 | 21.20
DRG [10] | ResNet50-FPN | 51.00 | 19.26 | 17.74 | 19.71 | 23.40 | 21.75 | 23.89
IP-Net [46] | ResNet50-FPN | 51.00 | 19.56 | 12.79 | 21.58 | 22.05 | 15.77 | 23.92
Contextual HGNN [44] | ResNet50-FPN | 52.70 | 17.57 | 16.85 | 17.78 | 21.00 | 20.74 | 21.08
SABRA(Ours) | ResNet50-FPN | 54.69 | 24.12 | 15.91 | 26.57 | 29.65 | 22.92 | 31.65
VSGNet [40] | ResNet152 | 51.76 | 19.80 | 16.05 | 20.91 | - | - | -
PD-Net [58] | ResNet152 | 52.20 | 22.37 | 17.61 | 23.79 | 26.89 | 21.70 | 28.44
ACP [21] | ResNet152 | 52.98 | 20.59 | 15.92 | 21.98 | - | - | -
SABRA(Ours) | ResNet152 | 56.62 | 26.09 | 16.29 | 29.02 | 31.08 | 23.44 | 33.37
Table 1: Results on V-COCO and HICO-DET datasets. In most of the cases, SABRA significantly outperforms SOTA methods. | Relationship Detection | Phrase Detection
---|---|---
| k=1 | k=70 | k=1 | k=70
Method | Backbone | R@50 | R@100 | R@50 | R@100 | R@50 | R@100 | R@50 | R@100
VRD [30] | VGG16 | 17.03 | 16.17 | 24.90 | 20.04 | 14.70 | 13.86 | 21.51 | 17.35
KL distilation [55] | VGG16 | 19.17 | 21.34 | 22.68 | 31.89 | 23.14 | 24.03 | 26.32 | 29.43
Zoom-Net [53] | VGG16 | 18.92 | 21.41 | 21.37 | 27.30 | 24.82 | 28.09 | 29.05 | 37.34
CAI + SCA-M [53] | VGG16 | 19.54 | 22.39 | 22.34 | 28.52 | 25.21 | 28.89 | 29.64 | 38.39
Hose-Net [48] | VGG16 | 20.46 | 23.57 | 22.13 | 27.36 | 27.04 | 31.71 | 28.89 | 36.16
RelDN [57] | VGG16 | 18.92 | 22.96 | 21.52 | 26.38 | 26.37 | 31.42 | 28.24 | 35.44
AVR [32] | VGG16 | 22.83 | 25.41 | 27.35 | 32.96 | 29.33 | 33.27 | 34.51 | 41.36
SABRA(Ours) | VGG16 | 24.47 | 29.16 | 27.27 | 33.99 | 30.57 | 36.80 | 33.39 | 41.79
GPS-Net [28] | VGG16 (MS COCO) | 21.50 | 24.30 | - | - | 28.90 | 34.00 | - | -
MCN [56] | VGG16 (MS COCO) | 24.50 | 28.00 | - | - | 31.80 | 37.10 | - | -
SABRA(Ours) | VGG16 (MS COCO) | 26.29 | 31.08 | 29.44 | 36.44 | 32.01 | 38.48 | 35.45 | 44.07
UVTransE [19] | VGG16 (VG) | 25.66 | 29.71 | 27.32 | 34.11 | 30.01 | 36.18 | 31.51 | 39.79
SABRA(Ours) | VGG16 (VG) | 27.87 | 32.48 | 30.71 | 37.71 | 33.56 | 39.62 | 36.62 | 45.29
ATR-Net [12] | ResNet101 | - | - | - | - | 31.96 | 36.54 | 36.06 | 43.45
SABRA(Ours) | ResNet101 | 26.73 | 31.11 | 29.92 | 37.43 | 32.81 | 38.68 | 36.24 | 45.26
Table 2: Results on the VRD dataset. SABRA outperforms all SOTA methods by a
large margin in all cases. By default, all backbones are pretrained with
ImageNet. To align with the setups of some prior works [28, 56, 19], we use MS
COCO or Visual Genome as additional data for training, denoted by VGG (MS
COCO) or VGG (VG).

####  3.2.2 Spatial Mask Decoder

To learn the local spatial configuration, we present Spatial Mask Decoder
(SMD) which predicts the locations of [_subject_ , _object_] . The
architecture of SMD is given in Fig. 3. SMD predicts a $2\times l_{p}\times
l_{p}$ mask using the VRD feature vectors, where $l_{p}$ represents the
pooling size and the two channels represent the spatial location of subjects
and objects in the pooled feature map with size $l_{p}$. This guarantees that
the local spatial configuration is tightly embedded in the feature vector for
relationship prediction. We scale the absolute coordinates of the [_subject_ ,
_object_] pair to get the relative position in the pooled feature map.
Compared to reconstruction in the image space, the ROI feature space preserves
the locality of the union bounding box but requires fewer parameters.
Different from the standard positional embedding [57] which implicitly
utilizes the spatial information with the absolute positions as an input, SMD
explicitly learns a structured embedding. Compared to [13] which concatenates
binary mask as position feature, SMD explicitly imposes the spatial structure
in the embedding vector and gives better spatially-aware embeddings. We
empirically show SMD outperforms these variants in our experiments.

Figure 3: Structure of Spaital Mask Decoder

###  3.3 SABRA Framework

The pipeline of SABRA is shown in Fig. 2. The input image is fed into an
object detector to predict all bounding boxes, which are exhaustively paired
to generate all relationship proposals. In the proposal sampling stage, given
the imbalanced relationship proposals, SABRA constructs a balanced pair mini-
batch by BNPS that samples data point $i$ proportionally to its weight
$w_{i}$. The features of subjects, objects, and union bounding boxes are fed
into MH-GAT with heterogeneous message passing to extract spatial object
interactions. We use only one layer of MH-GAT to avoid over-smoothing in GNNs
[54]. The mask decoder discovers the spatial configuration of [_subject_ ,
_object_] pairs in the learned embeddings. SABRA is robust to the imbalanced
negative proposal distribution and reduces the number of false positive
predictions caused by spatial ambiguity. More details are available in the
appendix.

During training, we jointly optimize the object detector and relationship
classifier by

$L=L_{\textrm{RPN}}^{\textrm{cls}}+L_{\textrm{RPN}}^{\textrm{loc}}+L_{\textrm{RCNN}}^{\textrm{cls}}+L_{\textrm{RCNN}}^{\textrm{loc}}+L_{\textrm{VRD}}^{\textrm{cls}}+L_{\textrm{VRD}}^{\textrm{mask}},$
(11)

where $L_{\textrm{RPN}}^{\textrm{cls}}$, $L_{\textrm{RPN}}^{\textrm{loc}}$,
$L_{\textrm{RCNN}}^{\textrm{cls}}$, $L_{\textrm{RCNN}}^{\textrm{loc}}$ are
losses for object detection. Their definitions are the same as
[37].$L_{\textrm{VRD}}^{\textrm{cls}}$ is a classification loss of each
relation proposal. $L_{\textrm{VRD}}^{\textrm{mask}}$ is an auxiliary loss
used in our SMD model. These two losses are designed for the VRD branch. We
opt for Binary Cross Entropy loss for both items. During inference, SABRA
predicts all given relationship proposals without sampling.

| Sampling | Spatial Learning | GNN | AProle
---|---|---|---|---
1 | - | - | - | 50.20
2 | BNPS | - | - | 52.24
3 | - | SMD | - | 50.84
4 | - | - | MH-GAT | 51.65
5 | - | SMD | MH-GAT | 52.82
6 | BNPS-2cls | SMD | MH-GAT | 53.74
7 | BNPS-3cls | SMD | MH-GAT | 53.93
8 | BNPS | - | MH-GAT | 53.67
9 | BNPS | Binary [13] | MH-GAT | 53.90
10 | BNPS | PE [57] | MH-GAT | 54.29
11 | BNPS | SMD | - | 52.53
12 | BNPS | SMD | M-GAT | 51.65
13 | BNPS | SMD | MH-GAT-no-edge | 53.80
14 | BNPS | SMD | MH-GAT | 54.69
Table 3: Ablation Study on V-COCO dataset. We compare each model variant with
the baseline (row 1) and the SABRA (row 14).

##  4 Experiment

We evaluate SABRA on three commonly used datasets: V-COCO [15], HICO-DET [5],
and VRD [30], covering both human-object interaction (V-COCO and HICO-DET) and
general object relationship detection (VRD). We compared SABRA over 20 SOTA
methods. Specifically, we trained SABRA with different backbones and perform a
comprehensive and fair comparison with existing methods. We incrementally add
our proposed components to a baseline model. The baseline directly
concatenates appearance features of subjects, objects, and their union
features, which are fed into a 3-layer Multi-Layer Perceptron (MLP) for
classification.

As a brief conclusion, we show that: 1) SABRA generally outperforms other
methods by a large margin; 2) the balanced negative proposal sampling strategy
can reduce the number of false positive predictions; 3) spatial mask decoder
successfully reduces the number of false positives caused by spatial
ambiguity.

###  4.1 Datasets

V-COCO is based on the 80-class object detection annotations of COCO [25]. It
has 10,346 images (2,533 for training, 2,867 for validating and 4,946 for
testing). HICO-DET has a total of 47,774 images, covering 600 categories of
human-object interactions over 117 common actions on 80 common objects. VRD
dataset contains 4,000 images in the train split and 1,000 in the test split.
It has 100 different types of objects and 70 types of relationships.

###  4.2 Evaluation Metrics

We followed the convention in prior literature and used three different
evaluation metrics for these datasets.

Following [30], we use Recall@$N$ as our evaluation metric on VRD. Recall@$N$
computes the _recall_ rates using the top $N$ predictions per image. To be
consistent with prior works, we report Recall@50 and Recall@100 in our
experiments. We evaluate two tasks: relationship detection that outputs triple
labels and evaluates bounding boxes of the subject and object separately;
phrase recognition that takes a triplet as a union bounding box and predicts
the triple labels. Besides, we report the top-$k$ predictions as [55], where
$k=1$ or $k=70$.

For V-COCO and HICO-DET, mean average precision (mAP) is used to estimate the
performance. A triplet [human, verb, object] is considered a positive
prediction if and only if there exists a triplet [human’, verb’, object’] in
the ground truth satisfying: (1) IoU(human, human’) $\geq 0.5$, (2) verb =
verb’, (3) IoU(object, object’) $\geq 0.5$

In HICO-DET, we calculate the mAP among all pairs of [verb, object]. In
_Default_ mode, we calculate AP on all images. In _Known Objects_ mode, the
category of the bounding box is known and we only calculate AP between humans
and objects from a specific category. In V-COCO, we calculate the mAP for all
categories of verbs, which is called AProle. More details are available in the
appendix.

###  4.3 Quantitative Results

We present our results in Table 1 for HOI datasets (V-COCO and HICO-DET) and
Table 2 for the VRD dataset. For HOI, we cluster our results according to the
backbones, including ResNet50, ResNet50-FPN, and ResNet152, with increasing
feature extraction ability. For VRD datasets, VGG-16 pretrained on ImageNet is
used for all methods. To align with the setup of some prior works [28, 56,
19], we use MS COCO and VG as additional datasets for VGG16.

We observe that SABRA generally improves the SOTA methods significantly on all
datasets for both HOI and VRD. For example, on V-COCO with ResNet152 backbone,
SABRA achieves 56.62 mAP while the SOTA model, ACP [22] gives an mAP of 52.98.
Specifically, we want to highlight the performance gain of SABRA on the V-COCO
dataset. V-COCO uses the object annotations of the COCO dataset, which are
accurate and include all objects, regardless of the relationships. This gives
a more imbalanced distribution than HICO-DET, where only the objects in
relationships are considered. Furthermore, accurate annotations also give rise
to better object detection, which amplifies the significance of spatial
information for a good performance. Specifically, we noticed that for PD-Net
[58], using a more powerful backbone (ResNet152) gives no performance gain
than the smaller one (ResNet50). SABRA, on the contrary, can fully exploit the
feature extraction power of ResNet152 and give a large performance improvement
compared with ResNet50 (56.62 V.S. 53.57).

|  |
---|---|---
(a) False positive comparison | (b) False positive reduction example of SABRA
Figure 4: Qualitative analysis of SABRA on V-COCO dataset. (a) The number of
false positive predictions under each class by Baseline model, SABRA w/o BNPS,
SABRA w/o spatial module (MH-GAT + SMD), and SABRA. BNPS and spatial module
significantly reduce the number of false positive predictions. (b) Predictions
of SABRA. In $\textrm{sub-predicate-obj}(v_{1},v_{2})$, $v_{1}$ denotes the
prediction confidence of the baseline model and $v_{2}$ denotes the confidence
of SABRA. SABRA successfully reduced the low-frequency difficult false
positives.

###  4.4 Ablation Studies

We conduct a comprehensive ablation study on the V-COCO dataset to understand
each proposed component: Balanced Negative Proposal Sampling (BNPS), Multi-
head Heterogeneous Graph Attention (MH-GAT), and Spatial Mask Decoder (SMD).
We present the results in Table 3, where row 1 denotes the baseline model and
row 14 denotes SABRA with all proposed components.

Each proposed component improves the baseline. We perform incremental analysis
in rows 1-4. Specifically, we observe that by simply improving the
optimization process with BNPS, baseline + BNPS (row 2) gains 2.04 improvement
on AProle. This suggests that imbalanced proposal distribution significantly
hinders the model performance, and BNPS addresses this issue effectively.

Balancing negative proposal distribution generally improves the VRD
performance. In rows 5-7 and row 14, we compare BNPS with 3 other
alternatives. (1) BNPS-3cls (row 7): balanced sampling over
$\\{S_{\textrm{neg}}^{1},S_{\textrm{neg}}^{2},S_{\textrm{neg}}^{3}\cup
S_{\textrm{neg}}^{4}\cup S_{\textrm{neg}}^{5}\\}$, i.e., ignoring the
difference between negative proposals when detections are correct; (2)
BNPS-2cls (row 6): balanced sampling over $\\{S_{\textrm{neg}}^{1}\cup
S_{\textrm{neg}}^{2},S_{\textrm{neg}}^{3}\cup S_{\textrm{neg}}^{4}\cup
S_{\textrm{neg}}^{5}\\}$, i.e., further ignoring the differences of negative
proposals when detections are incorrect; (3) None (row 5): we remove the BNPS.
We fix the positive sample rate to be 25% and balance the rest 75% samples
over the given distributions. Comparing BNPS with BNPS-2cls and BNPS-3cls, we
conclude BNPS improves prediction accuracy for both inaccurate detections and
incorrect associations.

Understanding spatial information is crucial to VRD. In rows 8-10 and row 14,
we compare the proposed Spatial Mask Decoder (SMD) with 3 alternatives. (1)
Binary (row 9) [13]: a binary mask over the union feature to specify the
position of [_subject_ , _object_] ; (2) PE (row 10) [57]: naive positional
embedding; (3) None (row 8): no spatial learning module. We conclude that: (1)
adding spatial information learning generally improves the performance; (2)
Implicitly imposing spatial information with PE or binary masks gives worse
performance than SMD which enforces an explicit constraint over the feature
space.

MH-GAT is more effective in heterogeneous VRD graphs In rows 11-13 and row 14,
we compare MH-GAT with other 4 alternatives: (1) Multi-head GAT [41] (row 12),
standard homogeneous message passing scheme which has the same number of heads
and edge features used in MH-GAT, (2) MH-GAT without edge feature (row 13),
and (3) None (row 11). We observe that: (1) GNNs generally improve the VRD
performance; (2) by separating the parameters for objects, subjects, and union
features, the heterogeneous message passing scheme in MH-GAT improves the
M-GAT; (3) edge feature gives extra information and is important to MH-GAT.

###  4.5 Qualitative Analysis

Qualitatively, we provide extra statistics and visualizations on the V-COCO
dataset to better understand the performance improvement of SABRA in Fig. 4.

To verify the ability of SABRA on reducing the number of false positives, we
compute the per-image false positive predictions for $S_{\textrm{neg}}^{1:5}$
by thresholding the prediction confidence at 0.5 in Fig. 4. Compared with the
baseline, SABRA w/o BNPS, SABRA w/o Spatial, and SABRA have reduced the total
number of low-frequency difficult $S_{\textrm{neg}}^{3:5}$ by 14.7%, 58.5%,
and 71.1%. BNPS gave a sharp decrease on $S_{\textrm{neg}}^{3:5}$, which
suggests the necessity of learning from a balanced proposal distribution. The
spatial module successfully filtered the irrelevant objects in
$S_{\textrm{neg}}^{3}$ and significantly reduced the number of false positives
in total. Combining both of them, SABRA gives the best performance among all
alternatives. However, we also noticed that on $S_{\textrm{neg}}^{4:5}$ and
$S_{\textrm{neg}}^{1:2}$, the spatial module has no clear improvement, because
the spatial module potentially overexploited the correct detection of the
positive [_subject_ , _predicate_ , _object_] triplet. Auxiliary constraints
on the relationship assignment could be considered to address this issue. We
leave it for future study.

We visualize two examples of SABRA successfully reducing the low-frequency
difficult false positives where both detections are correct in Fig. 4.b.
Without the spatial module, the VRD algorithm assigns high confidence (0.92)
to the [_subject_ , _object_] pair that the woman is cutting the cake. This
confidence was reduced to 0.13 by SABRA.

##  5 Conclusion

We present SABRA for alleviating false positives in VRD. We divided the
negative [_subject_ , _object_] proposals into 5 sub-classes with imbalanced
data distribution, and addressed the data imbalance by Balanced Negative
Proposal Sampling. SABRA incorporates the global contextual information with
MH-GAT and local spatial configuration by SMD. SABRA significantly outperforms
the SOTA methods on V-COCO, HICO-DET, and VRD datasets.

As the first paper to consider the data imbalance in the negative proposal
distribution, SABRA used a relatively simple strategy, balanced sampling. More
advanced techniques, e.g., Meta-Learning [36], could be considered.

## References

  * [1] Ricardo Barandela, E Rangel, José Salvador Sánchez, and Francesc J Ferri.  Restricted decontamination for the imbalanced training sample problem.  In Iberoamerican congress on pattern recognition, pages 424–431. Springer, 2003.
  * [2] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski.  A systematic study of the class imbalance problem in convolutional neural networks.  Neural Networks, 106:249–259, 2018.
  * [3] Jonathon Byrd and Zachary Lipton.  What is the effect of importance weighting in deep learning?  In International Conference on Machine Learning, pages 872–881, 2019.
  * [4] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  In Advances in Neural Information Processing Systems, pages 1567–1578, 2019.
  * [5] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng.  Learning to detect human-object interactions.  In 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, NV, USA, March 12-15, 2018. IEEE Computer Society, 2018.
  * [6] Vincent Delaitre, Ivan Laptev, and Josef Sivic.  Recognizing human actions in still images: a study of bag-of-features and part-based representations.  In Proceedings of the British Machine Vision Conference, 2010.
  * [7] Vincent Delaitre, Josef Sivic, and Ivan Laptev.  Learning person-object interactions for action recognition in still images.  In Advances in neural information processing systems, pages 1503–1511, 2011.
  * [8] Carolina Galleguillos, Andrew Rabinovich, and Serge Belongie.  Object categorization using co-occurrence, location and appearance.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [9] Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng.  Stylenet: Generating attractive visual captions with styles.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3137–3146, 2017.
  * [10] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang.  DRG: dual relation graph for human-object interaction detection.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII, volume 12357 of Lecture Notes in Computer Science, pages 696–712. Springer, 2020\.
  * [11] Chen Gao, Yuliang Zou, and Jia-Bin Huang.  ican: Instance-centric attention network for human-object interaction detection.  In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018. BMVA Press, 2018.
  * [12] Nikolaos Gkanatsios, Vassilis Pitsikalis, Petros Koutras, and Petros Maragos.  Attention-translation-relation network for scalable scene graph generation.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019.
  * [13] Nikolaos Gkanatsios, Vassilis Pitsikalis, Petros Koutras, Athanasia Zlatintsi, and Petros Maragos.  Deeply supervised multimodal attentional translation embeddings for visual relationship detection.  In 2019 IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, September 22-25, 2019, pages 1840–1844. IEEE, 2019.
  * [14] Georgia Gkioxari, Ross B. Girshick, Piotr Dollár, and Kaiming He.  Detecting and recognizing human-object interactions.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, 2018.
  * [15] Saurabh Gupta and Jitendra Malik.  Visual semantic role labeling.  CoRR, 2015.
  * [16] Munawar Hayat, Salman Khan, Syed Waqas Zamir, Jianbing Shen, and Ling Shao.  Gaussian affinity for max-margin class imbalanced learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6469–6479, 2019.
  * [17] Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao.  Visual compositional learning for human-object interaction detection.  CoRR, abs/2007.12407, 2020.
  * [18] Yue Hu, Siheng Chen, Xu Chen, Ya Zhang, and Xiao Gu.  Neural message passing for visual relationship detection.  In ICML Workshop on Learning and Reasoning with Graph-Structured Representations, Long Beach, CA, 2019.
  * [19] Zih-Siou Hung, Arun Mallya, and Svetlana Lazebnik.  Contextual translation embedding for visual relationship detection and scene graph generation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [20] Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong.  Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7610–7619, 2020.
  * [21] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon.  Detecting human-object interactions with action co-occurrence priors.  CoRR, 2020.
  * [22] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon.  Detecting human-object interactions with action co-occurrence priors.  arXiv preprint arXiv:2007.08728, 2020.
  * [23] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu.  Transferable interactiveness knowledge for human-object interaction detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3585–3594, 2019.
  * [24] Xiaodan Liang, Lisa Lee, and Eric P. Xing.  Deep variation-structured reinforcement learning for visual relationship and attribute detection.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, 2017.
  * [25] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, 2014.
  * [26] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar.  Focal loss for dense object detection.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [27] Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao.  Gps-net: Graph property sensing network for scene graph generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3746–3753, 2020.
  * [28] Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao.  Gps-net: Graph property sensing network for scene graph generation.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, 2020.
  * [29] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei.  Visual relationship detection with language priors.  In European conference on computer vision, pages 852–869. Springer, 2016.
  * [30] Cewu Lu, Ranjay Krishna, Michael S. Bernstein, and Fei-Fei Li.  Visual relationship detection with language priors.  In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I, 2016\.
  * [31] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh.  Hierarchical question-image co-attention for visual question answering.  In Advances in neural information processing systems, pages 289–297, 2016.
  * [32] Jianming Lv, Qinzhe Xiao, and Jiajie Zhong.  Avr: Attention based salient visual relationship detection.  arXiv preprint arXiv:2003.07012, 2020.
  * [33] Li Mi and Zhenzhong Chen.  Hierarchical graph attention network for visual relationship detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13886–13895, 2020.
  * [34] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu.  Learning human-object interactions by graph parsing neural networks.  In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IX, 2018.
  * [35] Vignesh Ramanathan, Congcong Li, Jia Deng, Wei Han, Zhen Li, Kunlong Gu, Yang Song, Samy Bengio, Charles Rosenberg, and Li Fei-Fei.  Learning semantic relationships for better action retrieval in images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1100–1109, 2015.
  * [36] Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li.  Balanced meta-softmax for long-tailed visual recognition.  arXiv preprint arXiv:2007.10740, 2020.
  * [37] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, 2015.
  * [38] Li Shen, Zhouchen Lin, and Qingming Huang.  Relay backpropagation for effective learning of deep convolutional neural networks.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, European Conference on Computer Vision ECCV, 2016.
  * [39] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick.  Training region-based object detectors with online hard example mining.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [40] Oytun Ulutan, A. S. M. Iftekhar, and B. S. Manjunath.  Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13614–13623. IEEE, 2020.
  * [41] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph attention networks.  In ICLR, 2018.
  * [42] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: Lessons learned from the 2015 mscoco image captioning challenge.  IEEE transactions on pattern analysis and machine intelligence, 39(4):652–663, 2016.
  * [43] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He.  Pose-aware multi-level feature network for human object interaction detection.  In International Conference on Computer Vision, ICCV, 2019.
  * [44] Hai Wang, Wei-Shi Zheng, and Yingbiao Ling.  Contextual heterogeneous graph network for human-object interaction detection.  CoRR, abs/2010.10001, 2020.
  * [45] Tiancai Wang, Rao Muhammad Anwer, Muhammad Haris Khan, Fahad Shahbaz Khan, Yanwei Pang, Ling Shao, and Jorma Laaksonen.  Deep contextual attention for human-object interaction detection.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 5693–5701. IEEE, 2019.
  * [46] Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun.  Learning human-object interaction detection using interaction points.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4116–4125, 2020.
  * [47] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Learning to model the tail.  In Advances in Neural Information Processing Systems, pages 7029–7039, 2017.
  * [48] Meng Wei, Chun Yuan, Xiaoyu Yue, and Kuo Zhong.  Hose-net: Higher order structure embedded network for scene graph generation.  CoRR, 2020.
  * [49] Bingjie Xu, Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli.  Interact as you intend: Intention-driven human-object interaction detection.  IEEE Transactions on Multimedia, 22(6):1423–1432, 2019.
  * [50] Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mohan S Kankanhalli.  Learning to detect human-object interactions with knowledge.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [51] Dongming Yang and Yuexian Zou.  A graph-based interactive reasoning for human-object interaction detection.  In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020. ijcai.org, 2020.
  * [52] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei.  Exploring visual relationship for image captioning.  In Proceedings of the European conference on computer vision (ECCV), pages 684–699, 2018.
  * [53] Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, Jing Shao, and Chen Change Loy.  Zoom-net: Mining deep feature interactions for visual relationship recognition.  In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part III, 2018.
  * [54] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi.  Spatio-temporal graph transformer networks for pedestrian trajectory prediction.  arXiv preprint arXiv:2005.08514, 2020.
  * [55] Ruichi Yu, Ang Li, Vlad I. Morariu, and Larry S. Davis.  Visual relationship detection with internal and external linguistic knowledge distillation.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, 2017.
  * [56] Yibing Zhan, Jun Yu, Ting Yu, and Dacheng Tao.  Multi-task compositional network for visual relationship detection.  International Journal of Computer Vision, 128(8):2146–2165, 2020\.
  * [57] Ji Zhang, Kevin J. Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro.  Graphical contrastive losses for scene graph parsing.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, 2019.
  * [58] Xubin Zhong, Changxing Ding, Xian Qu, and Dacheng Tao.  Polysemy deciphering network for robust human-object interaction detection.  CoRR, 2020.
  * [59] Hao Zhou, Chongyang Zhang, and Chuanping Hu.  Visual relationship detection with relative location mining.  In Laurent Amsaleg, Benoit Huet, Martha A. Larson, Guillaume Gravier, Hayley Hung, Chong-Wah Ngo, and Wei Tsang Ooi, editors, Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019. ACM, 2019.

##  Appendix A Implementation Details

###  A.1 Overall Pipeline

The overall pipeline of SABRA consists of three major components: backbone,
detector, and VRD classifier. Backbone is a feature extractor for object
detection and relationship detection. Detector predicts all potential bounding
boxes and produces a set of detections:

$\displaystyle B=\\{(x_{1},y_{1},x_{2},y_{2},\textit{cls},\textit{score})\\},$
(12)

where $(x_{1},y_{1})$ are the coordinates of the upper left corner,
$(x_{2},y_{2})$ are the coordinates of the bottom right corner, cls is the
category of this bounding box, and score is the confidence of this prediction.
VRD classifier uses this set of detections to generate all relationship
proposals $S$ and categorize each proposal. In this way, we obtain a set of
triplets associated with a score from VRD classifier:

$\displaystyle T=\\{(b_{1},b_{2},\textit{cls},\textit{score})\\}.$ (13)

###  A.2 Backbone

To fully compare with other methods, we use several backbone setups in
experiments including VGG-16, ResNet50, ResNet50-FPN, ResNet101, and
ResNet152. In particular, when we use ResNet50-FPN for both object detection
and relationship detection following [14], we use ResNet50-FPN to denote this
setting. Meanwhile, some other works like [58] use ResNet50-FPN for object
detection, but use only ResNet50 for relationship detection. Therefore, we
build up this setting named ResNet50 that a ResNet50 module is shared for both
object detection and relationship detection, and an extra FPN is used only for
object detection. In addition, FPN is used neither in object detection nor
relationship detection process when we use VGG-16, ResNet101, and ResNet152 as
the backbone.

Specifically, for V-COCO and HICO-DET datasets, we use ImageNet + MS COCO pre-
trained ResNet50, ResNet50-FPN, and ResNet152. For VRD dataset, we
respectively use VGG-16 pre-trained on ImageNet, ImageNet + MS COCO, and
ImageNet + Visual Genome. We also use ImageNet pre-trained ResNet101 for VRD
task.

###  A.3 VRD Classifier

The overall structure of the VRD classifier is shown in Fig. 2 in the main
text. In this section, we will describe the training strategy and inference
procedure of our VRD classifier.

Training Strategy VRD classifier receives the detection set $B$ and further
divides it into two sets $B_{\textrm{subject}}$ and $B_{\textrm{object}}$. In
HOI task, $B_{\textrm{subject}}$ contains all human bounding boxes and
$B_{\textrm{object}}$ contains all bounding boxes from detector. In general
VRD task, both $B_{\textrm{subject}}$ and $B_{\textrm{object}}$ contain all
detection results. Under this definition, we generate a proposal set $S$. For
each image, we sample 64 relationship proposals from the set $S$. This hyper-
parameter holds across all our model configurations, no matter whether we use
BNPS. Moreover, we also keep the ratio of positive proposals as 0.25 for all
experiments. Sigmoid activation is used to predict both the confidence of each
category and the spatial mask. Correspondingly, binary cross-entropy loss is
used for both supervisions.

Inference Procedure For each relationship proposal $(b_{1},b_{2})$, we predict
the score $s_{\textit{cls}}$ of each relationship category cls. The final
score of triplet $(b_{1},b_{2},\textit{cls})$ is calculated as below:

$\displaystyle\textit{score}=s_{1}\times s_{2}\times s_{\textit{cls}},$ (14)

where $s_{1}$ and $s_{2}$ are the scores of bounding box $b_{1}$ and $b_{2}$
respectively.

##  Appendix B Experiment Setup

###  B.1 Data Preprocessing

For V-COCO, the ground truth detection is obtained from COCO. As some
relationships containing invisible objects, we fill the region of the object
with the coordinates as the subject. In other words, we generate a ground
truth triplet $(b_{1},b_{1},\textit{cls})$ when the object is invisible.

For HICO-DET, we generate ground truth detection from all bounding boxes of
triplets. It should be noted that, in HICO-DET, the annotation of each
relationship is independent. Therefore, there may exist multiple bounding
boxes for a single person or object. To generate a unique bounding box
annotation for each instance, we merge those bounding boxes where the IoU
values are no less than 0.5.

For general VRD, we have no extra pre-processing.

###  B.2 Training Procedure

For V-COCO, we first train the object detection on the COCO dataset. Then we
freeze the backbone and jointly train the detector and VRD classifier on the
corresponding dataset.

For HICO-DET, we first train the object detection on the COCO dataset and then
finetune on the detection results from HICO-DET. After that, we jointly train
the Detector and VRD classifier with a frozen backbone.

For general VRD, we use ImageNet pre-trained VGG-16 backbone to initialize our
model and then train object detection on the general VRD dataset. Finally, we
jointly train the detector and VRD classifier with the frozen backbone.

In each training process, we use 16 GPUs (1080TI) and train 25 epochs with the
initial learning rate being 0.00125. The learning rate is decreased in the
17th and 23rd epoch with a 0.1 decay rate. When training solely the object
detection, each batch contains four different images. For the joint training
of detection and relationship detection, each batch contains two different
images.

##  Appendix C Additional Analysis for BNPS

###  C.1 Other reasons for inaccurate detections

Detection Top-$N$ | RS | BNPS
---|---|---
100 | 51.95 | 54.29
50 | 52.82 | 54.69
40 | 52.53 | 54.43
30 | 53.05 | 54.52
20 | 52.92 | 53.68
Table 4: AProle performance of the different numbers of detections from the
detector on the V-COCO test set. We use ResNet50-FPN as our backbone and
modify the sampling strategy and selection of the detector’s top-$N$ while
keeping others unchanged.

Although inaccurate bounding boxes lead to a large number of easy negative
proposals, we cannot simply ignore these detection results. We observe that
decreasing the number of top-$N$ detections may have a negative influence on
VRD algorithms. Table 4 is a thorough comparison of the top-$N$ detections we
keep from the detector and whether we use BNPS or random sampling (RS) against
the model performance. The results suggest that decreasing the number of
top-$N$ has no evident improvement in the final performance (Top-50 w/o BNPS
V.S. Top-30 w/o BNPS, 52.82 V.S. 53.05). The major reason is that we still
need many inaccurate bounding boxes in inference for higher recall values.
Besides, we find that our proposed Balanced Negative Proposal Sampling has
remarkable improvement no matter which top-$N$ we select. These results
strongly prove the effectiveness and robustness of our proposed method.

###  C.2 Analysis of the improvement

The improvement of BNPS comes from two major sources. Firstly, BNPS reduces
the easy negative proposals caused by inaccurate detections; secondly, BNPS
balances the difficult negative proposals, considering whether the subject and
the object are involved in any triplet from the ground truth. These two parts
are both significant and essential, which is partially proven in the ablation
study. In this section, we add some extra quantitative and qualitative
results.

We include an additional experiment to test if the performance improvement of
SABRA comes only from the increase in the number of difficult negative
proposals $S_{\textrm{neg}}^{3:5}$. We compare a BNPS-3cls variant, BNPS-3cls-
HN with BNPS. In the original BNPS-3cls, we use a sample rate [0.25, 0.25,
0.25] for $S_{\textrm{neg}}^{1}$, $S_{\textrm{neg}}^{2}$ and
$S_{\textrm{neg}}^{3:5}$. However, in BNPS, each class receives a sample rate
of 0.15, which gives a $0.45$ sample rate for the negative classes
$S_{\textrm{neg}}^{3:5}$. We design BNPS-3cls-HN such that it assigns the same
sample rate to the hard negative classes, [0.15, 0.15, 0.45] for
$S_{\textrm{neg}}^{1}$, $S_{\textrm{neg}}^{2}$ and $S_{\textrm{neg}}^{3:5}$.
We train both models on V-COCO and report the results briefly here: BNPS-3cls-
HN gives 54.20 AProle while SABRA achieves 54.69. This suggests that the
improvement from BNPS-3cls to SABRA is not simply because of the increased
sample rate of hard negative proposals. The balance among
$S_{\textrm{neg}}^{3:5}$ also plays a critical role in this improvement.

Figure 5: Qualitative analysis of different sampling strategies.

We also visualize the number of negative predictions of each negative type in
Fig. 5. We observe that: (1) the total number of false positives in low-
frequency difficult classes, $S_{\textrm{neg}}^{3:5}$, of SABRA is lower than
BNPS-3cls because we increase the total ratio of difficult negative proposals.
(2) the reduction ratio of $S_{\textrm{neg}}^{5}$ is higher than that of
$S_{\textrm{neg}}^{4}$. Meanwhile, the reduction ratio of
$S_{\textrm{neg}}^{4}$ is higher than that of $S_{\textrm{neg}}^{3}$, which
suggests that our balance among
$S_{\textrm{neg}}^{3},S_{\textrm{neg}}^{4},S_{\textrm{neg}}^{5}$ clearly
improves the ability to identify the negative proposals, especially on the
difficult proposals.

###  C.3 BNPS compared with other sampling methods

We have proved the rationality and effectiveness of our proposed BNPS in
sampling ideal relationship pairs among a vast number of proposals. To further
examine the superiority of BNPS, we implement several alternative sampling
methods including online hard example mining (OHEM) [39] and focal loss [26]
and compare their capacities.

OHEM was first proposed in the object detection area and it prefers to sample
harder examples than easier ones [39]. Typically, for a mini-batch of samples,
only the top-$k$ most difficult proposals, i.e. proposals with top-$k$ highest
loss values, will contribute to model optimization. In practice, the mini-
batch is constructed to enforce a 1:3 ratio between positives and negatives,
i.e. proposals in $S_{pos}$ and $S_{neg}$, to help ensure each mini-batch has
enough positives. Since positive proposals are usually insufficient, we only
apply OHEM on $S_{neg}$ to ensure this claim. We set mini-batch size
$B\in\\{48,64,96\\}$ in our experiment.

The focal loss modifies the standard cross-entropy loss to dynamically down-
weight the contribution of easy examples:

$\displaystyle\text{FL}(p_{t})=-\alpha(1-p_{t})^{\gamma}\log(p_{t}),$ (15)
$\displaystyle p_{t}=\left\\{\begin{array}[]{ll}p&\text{if}\ y=1\\\
1-p&\text{otherwise}.\end{array}\right.$

In the above $y\in\\{\pm 1\\}$ specifies the ground-truth class and
$p\in[0,1]$ is the model’s estimated probability for the class with label
$y=1$. $\alpha$ and $\gamma$ are the balance coefficient and the tunable
focusing parameter respectively. In our experiment, we use $\alpha=0.25$ with
$\gamma\in\\{1,2\\}$.

Besides the above methods, Random Sampling and BNPS are also implemented as
relationship pair sampling methods for comparison. We train models using these
methods on V-COCO and keep all other settings consistent with setups in
Section B. Additionally, we assign $N=50$ for top-$N$ detections.

As results shown in Table 5, we observe that BNPS achieves the best
performance, which reconfirms its superiority. Meanwhile, focal loss and OHEM
in all settings behave worse than the basic random sampling method, which
draws our attention. Focal loss and OHEM both emphasize the importance of
samples with high loss values. However, in the scenario of relationship
detection, these methods may focus too much on the hard samples and overlook
the role of easy ones. Compared with them, our proposed BNPS uses the
predefined sample classification method and pay equal attention to 5 classes
of negative proposals, which makes our sampling strategy more robust towards
the complicated distribution of samples in this scenario and less likely to be
influenced by outliers.

Method | Setup | AProle
---|---|---
RS | - | 52.82
OHEM | $B=48$ | 45.81
$B=64$ | 45.75
$B=96$ | 45.73
Focal Loss | $\gamma=1$ | 52.55
$\gamma=2$ | 51.10
BNPS | - | 54.69
Table 5: Performance of the different relationship pair sampling methods. We
use ResNet50-FPN as our backbone and modify the sampling strategy while
keeping others unchanged.


# The Bad Batches: Enhancing Self-Supervised Learning in Image Classification
Through Representative Batch Curation

Özgü Göksu
School of Computing Science
University of Glasgow
<EMAIL_ADDRESS>Nicolas Pugeault
School of Computing Science
University of Glasgow
<EMAIL_ADDRESS>
###### Abstract

The pursuit of learning robust representations without human supervision is a
longstanding challenge. The recent advancements in self-supervised contrastive
learning approaches have demonstrated high performance across various
representation learning challenges. However, current methods depend on the
random transformation of training examples, resulting in some cases of
unrepresentative positive pairs that can have a large impact on learning. This
limitation not only impedes the convergence of the learning process but the
robustness of the learnt representation as well as requiring larger batch
sizes to improve robustness to such bad batches. This paper attempts to
alleviate the influence of false positive and false negative pairs by
employing pairwise similarity calculations through the Fréchet ResNet Distance
(FRD), thereby obtaining robust representations from unlabelled data. The
effectiveness of the proposed method is substantiated by empirical results,
where a linear classifier trained on self-supervised contrastive
representations achieved an impressive 87.74% top-1 accuracy on STL10 and
99.31% on the Flower102 dataset. These results emphasize the potential of the
proposed approach in pushing the boundaries of the state-of-the-art in self-
supervised contrastive learning, particularly for image classification tasks.

###### Index Terms:

self-supervised learning, contrastive learning, batch curation, representation
learning, Huber loss

##  I Introduction

00footnotetext: This paper is published in the Proceedings of the
International Joint Conference on Neural Networks (IJCNN), with the World
Congress in Computational Intelligence (WCCI) 2024.

Despite the rapid progress of supervised deep learning, the availability of
labelled data remains a limitation: transferring the success of deep learning
approaches to applications and domains where data and annotations are scarce
is a challenge. This has led to the popularity of approaches for unsupervised
and self-supervised pre-training: training a deep network to solve a pretext
task on a large dataset of usually unlabelled examples, before fine-tuning it
for the desired downstream task on a typically much smaller, but correctly
labelled dataset. This approach has been successful in allowing the use of
deep models for tasks where limited labelled data is available. One successful
example of self-supervised learning is contrastive learning, such as SimCLR
[simclrv2], where the pretext task is to optimize a latent space such that
augmented versions of the same training example are more similar to each other
than to other training examples. Contrastive learning has shown excellent
performance [dino], [adco], [chen2020simple] but requires large datasets,
large batch sizes and long training times.

Figure 1: Existing self-supervised contrastive methods mainly rely on various
data augmentations to increase diversity, however, it causes weak transformed
views of original images. Our method aims to eliminate weak augmented views
such as darker images as a similar pair, and insufficient colour changes.

In this article, we propose the hypothesis that contrastive learning is
negatively impacted by a relatively small proportion of training examples and
augmentations. This is illustrated in Figure 1, which shows examples of
positive pairs produced by existing methods. In those examples, we observe
that data augmentation has degraded essential information in the image,
leading to the creation of false positive, which could hinder the convergence
of the contrastive objective. We argue that discarding such “bad batches”
would converge more effectively and efficiently, thereby necessitating smaller
batch sizes. While self-supervised contrastive learning has shown promise in
representation learning, its reliance on randomly-formed batches containing
numerous false positives and negatives presents a major obstacle. In this
work, we present a simple yet effective approach, which involves the
evaluation of each batch’s representations by measuring their FRD (Fréchet
ResNet Distance). FRD can provide a criterion to detect and reject “bad
batches” in unsupervised representation learning. We assume that bad batches
are characterized by an abundance of false positives and negatives, whereas
good batches consist of images with semantically similar views. This study
focuses on learning robust representations through the utilization of good
batches, which consist of appropriate views of the original images.
Contrarily, darker, blurry, or randomly cropped and inappropriate views are
intentionally excluded to enhance the quality of the learned representations.
In summary, our major contributions to this work are

  * •

a criterion to evaluate augmented batches by Fréchet ResNet Distance (FRD)
calculation,

  * •

a regularized loss function to reduce the required batch size for training
self-supervised contrastive models,

  * •

a robust representation learning method by avoiding false positives in
batches.

This work breaks new ground by proposing a novel approach for evaluating batch
quality in self-supervised contrastive learning (SSCL). In traditional SSCL,
batches are typically randomly formed, which can lead to the inclusion of
distorted or irrelevant image views, thereby introducing false positives and
negatives. By implementing a method based on (FRD), we can identify and remove
batches that are likely to contain such misleading samples. Our process
ensures that only batches with semantically similar views of the original
images, referred to as “good batches” are utilized for representation
learning. By excluding darker, blurry, or inappropriately cropped views, which
are considered irrelevant, the quality of the learned representations is
significantly enhanced. This breakthrough has the potential to significantly
accelerate the development of efficient and generalizable self-supervised
models.

###  I-A Related Work

Learning representations without human supervision has long been an ongoing
challenge due to enhancing generalization and extracting relevant features
from raw data. Self-supervised learning, as opposed to traditional supervised
approaches, learns features from unlabelled data and captures representative
features as well as the underlying data distribution. Self-supervised learning
approaches are divided into two subcategories: generative and discriminative
[ozbulak2023know]. Generative models such as Generative Adversarial Networks
(GANs) and autoencoders are capable of generating realistic, high-quality
images from the image dataset. However, they might be insufficient to learn
representations, since achieving interpretable representations in the latent
space and hard-to-learn informative features are challenging [chen2020simple].
Unlike generative algorithms, discriminative models learn representations
through pretext tasks, which focus on understanding the content of images to
learn representations instead of generating an image. There are many pretext
tasks such as image rotation prediction [johnson2020rotnet], jigsaw puzzle
solving [noroozi2016unsupervised], image colourization
[larsson2017colorization] and contrastive learning [chen2020simple],
[jaiswal2020survey], which are designed to replace supervision and extract
meaningful and informative representations from raw data, which can later be
used for downstream tasks. Contrastive learning (CL) is a self-supervised
learning methodology that is designed to determine a representation space
wherein similar instances (positive) are drawn into closeness while dissimilar
instances (negative) are pushed apart. Contrastive learning (CL) facilitates
mutual information (MI) among views while preserving task-relevant
information. MI evaluates the statistical dependency of two random variables.
Regarding CL, the two views of the same image are utilized as random
variables. CL attempts to decrease the MI between the two views by maximizing
the similarity (and minimizing the dissimilarity) between positive pairs while
minimizing the similarity between negative pairs. The fundamental assumption
underlying CL is that obliging the model to obtain representations emphasizing
task-relevant information enhances the similarity among positive pairs,
facilitating a more straightforward differentiation between positive and
negative pairs. This objective is realized by being able to consistently map
positive pairs onto proximate points within the feature space, thereby
enhancing their ability to distinguish from negative pairs. Positive pairs
encompass images featuring similar views of the original raw data, including
various transformations such as rotations or cropping applied to the same
image. Conversely, negative pairs are deliberately chosen to be dissimilar,
typically involving random combinations from the dataset. Without labels,
recent improvements rely on instance discrimination problems in which positive
pairs are treated as two augmented versions of the same images while
considering the remainder as negatives [chen2020simple], [moco]. As an
illustration, the initial row of the figure 1 shows two transformed views of
the same image, captured from distinct angles. CL aims to extract features
that are present in both views, like the object’s shape and colour. This
shared information is generally beneficial for recognizing the object in
different contexts. However, some details specific to one angle, like a unique
marking or texture, might be discarded as noise during the simplification
process in figure 1. This discarded information could potentially be relevant
for a specific task like identifying individual instances of the object. This
process ensures that the positive pairs share the same semantic content or
meaning. The model can generalize well to downstream tasks by learning to
recognize these shared features. Existing self-supervised contrastive learning
methodologies predominantly generate batches through random transformations
like cropping and colour jittering. The randomness in batches introduces many
weak transformed views (false positives or negatives) and hinders the model’s
ability to learn from relevant image pairs in figure 1. Moreover, the
requirement to fine-tune transformations for each dataset introduces
unnecessary complexity and undermines performance on subsequent tasks. Several
methods eliminate false negative impact on representation, they require larger
batch sizes such as 4096 [chen2020simple], and 8192 [moco], [dino]. A larger
batch provides a more diverse set of negative examples, making it harder for
the model to distinguish them based on spurious correlations or artefacts of
the data augmentation process. This encourages the model to focus on learning
more generalizable features. However, it’s important to note that training
with large batches requires more memory, which can be a limiting factor for
smaller machines or datasets. Many approaches to tackle this problem have been
offered to eliminate weak data augmentations in batches.
GeoDA[cosentino2022geometry] method focuses on shape-preserving geometric
augmentations on images and [mishra2021object] proposes context-aware
augmentation. [fawzi2016adaptive] aims for effective representation learning
by adaptive augmentations based on model uncertainty. The Patch Curation
method, as outlined in [welle2021batch], introduces a batch curation algorithm
through the transformation of images into patches. This approach determines
batches by assessing their Euclidean distance. Not only, new data augmentation
is more specific to use datasets, but also it is not representative. Previous
investigations into batch behaviours have predominantly concentrated on novel
data augmentation methods [kurtulucs2023tied], hard negative mining
[zhu2023hnssl], [robinson2020contrastive], or the utilization of larger batch
sizes for self-supervised contrastive learning. However, batch evaluation by
measuring pairwise similarity is not only effective in avoiding weak
representation learning on downstream tasks but also benefits from smaller
batch size, and sufficient data augmentation tunings. Traditional self-
supervised learning leverages large-scale data and pretext tasks like image
inpainting or contrastive learning to induce informative data representations.
While effective, these methods often require significant computational
resources. Also, models learn less discriminative features. We propose a novel
batch curation approach that significantly improves representation learning
efficiency, requiring neither extensive datasets nor large batch sizes. This
enables effective self-supervised learning in resource-constrained scenarios.
Larger batches can mitigate random noise in gradients, leading to smoother
optimization. However, excessively large sizes can worsen vanishing gradients,
hindering efficient network updates. Increased batch size introduces a wider
variety of negative examples for contrasting, potentially enriching
representations. However, diminishing marginal returns and potential
overfitting to sincere correlations within large batches must be balanced.
Certain loss functions, like Normalized Temperature-scaled Cross-Entropy (NT-
Xent), rely on statistical properties of the data distribution, benefiting
from larger batches for accurate estimation. However, over-reliance on larger
batches can make performance sensitive to specific data characteristics.
SwAV[caron2020unsupervised] is a self-supervised contrastive learning method,
that aims to enhance representation learning by leveraging massive web-based
images for data augmentation. BYOL [byol] utilizes a two-network architecture
with a student and teacher network without a memory bank or momentum contrast.
These models benefit from representation learning on many downstream tasks,
however, these models have larger batch size usage. Like SimCLR
[chen2020simple] and SimCLR-v2 [simclrv2] train with large batch sizes from
4096 to 16384, SwAV uses even larger batch sizes exceeding 32768. Even BYOL
has a simple framework, from 256 to 1024 batch size. However, training larger
batches with these models may not consistently be the optimal strategy, and
striking the right balance is essential for effective and high-quality
representation learning. SimCLR-v2 aims to improve representation learning by
using larger ResNet and inspired memory mechanisms like MoCo [moco] which is a
dynamic dictionary that stores past data augmentations or representations.
However, they have limitations such as memory management, outdated negative
samples in batches, and duplicate samples. Current methods rely on a pairing
strategy that designates two augmented views of the original raw data as
positive pairs and randomly selects the remaining transformed views as
negative pairs. However, the indiscriminate generation of random images has
led to numerous inaccurate augmentations. This has resulted in less
discriminative representations and a diminished generalization capacity for
self-supervised contrastive models. These methods focus on representation
learning in self-supervised contrastive learning; however, they cannot extract
robust representations and have poor performance on downstream tasks. In
addition, they might not be efficient in reducing gradient noise, eliminating
false positive pairs in batches, and loss function behaviour, which requires a
larger batch size due to their reliance on data distributions.

##  II Methodology

The self-supervised model employs a predetermined method for curating random
batches without evaluating individual batches. Fréchet ResNet Distance (FRD)
scores for each batch are computed at the nth epoch, before the activation of
the batch curation algorithm. The rationale behind applying the batch curation
method in the early stages of learning is to leverage the advantages of
features acquired during the initial epochs. As a subsequent step, we seek to
enhance the model’s robustness to false positives or noise within the batch.
To achieve this, we introduce regularization to the contrastive loss by
incorporating the Huber loss [barron2019general] with an associated
coefficient.

Figure 2:  Our presented framework for batch curation in self-supervised
contrastive learning. Task 1 illustrates image classification as a downstream
task. The batch curation part mainly decides which batches are used to update
gradients.

###  II-A Contrastive Learning

Following the contrastive learning framework (as proposed by, e.g.,
[chen2020simple]), we have a training set of images $x\in\mathcal{D}$, a
family of transformations $\mathcal{T}$, a backbone network $f$ such that
$h=f(x)$ is a latent encoding of $x$ and a projection head $g$ such that
$z=g(z)$ is a projection of $h$, as illustrated in Figure 2. Considering a
batch of $N$ training examples
$B=\left\\{x_{1},x_{2},\ldots,x_{N}\right\\}\subset\mathcal{D}$, by applying
randomly sampled transformations $t\in\mathcal{T}$, we generate two augmented
batches $B_{1}$ and $B_{2}$, for a total of $2N$ data points. We say that two
projection vectors $z_{i}$ and $z_{j}$ form a positive pair if they are
transformations of the same training image $I$, a negative pair if they are
transformations from two distinct images. Contrastive learning approaches in
the literature predominantly employ the Normalized Temperature-scaled Cross-
Entropy (NT-Xent) loss[chen2020simple], [SiameseIM]. It is typical to
concatenate the projection vectors from $B_{1}$ and $B_{2}$ in a single tensor
of $2N$ vectors $z_{i}$. Then if $z_{i},z_{j}$ is a positive pair of
projections, the NT-Xent loss over those $2N$ projections is:

$\mathcal{L}_{i,j}=-\log\frac{exp(sim(z_{i},z_{j})/\tau)}{\sum_{k=1}^{2N}{1}_{[k\neq
i]}exp(sim(z_{i},z_{k})/\tau)}$ (1)

The function $sim(.,.)$ represents the cosine similarity, and $2N-1$ denotes
the number of negative (dissimilar) samples since $z_{i}$, $z_{j}$ are
positive (similar) samples transformed in the same image and $\tau$ is the
temperature value. Because Equation LABEL:eq:eq4 is asymmetric, we calculate
the final loss by the loss of a batch $B$ is defined as

$\mathcal{L}_{nt\mbox{-}xent}=\frac{1}{2N}\sum_{k=1}^{N}\left[\mathcal{L}_{2k-1,2k}+\mathcal{L}_{2k,2k-1}\right]$
(2)

The contrastive learning literature has shown that minimizing this loss
produces a latent space $\mathcal{H}$ that can allow solving many downstream
tasks by just training a new projection head, requiring limited sets of
labelled examples. In this article, we argue that contrastive learning can be
severely affected by just a few instances of poor data augmentations. In the
following, we describe two approaches to reduce the impact of those: Huber
loss regularization and Fréchet batch curation.

###  II-B Huber Loss Regularization

Contrastive learning ensures that projections of the same image (positive
pairs) are more similar than projections of other samples. We argue that
convergence can be improved by ensuring that the projections $z_{i}$ and
$z_{j}$ for two different data augmentations $t_{i},t_{j}$ of the same image
$I$ should be similar. This can be enforced by adding a regularization term to
the loss. When there is considerable dissimilarity between positive pairs, the
Huber loss imposes a penalty on the loss value, encouraging the model to bring
the representations closer together in the latent space. Due to its proven
effectiveness in handling outliers, the Huber loss ensures enhanced resilience
and stability throughout the learning process. In practice, for a positive
pair $z_{i}$, $z_{j}$, we have

$\footnotesize l_{i,j}=\begin{cases}0.5\left(z_{i}-z_{j}\right)^{2},&\text{ if
}\left|z_{i}-z_{j}\right|<\text{$\delta$ }\\\ \text{ $\delta$
}\left(\left|z_{i}-z_{j}\right|-0.5\text{ $\delta$ }\right),&\text{ otherwise
}\end{cases}.$ (3)

where the parameter $\delta$ determines the point at which the loss function
transitions from quadratic to absolute, set to 1.0 in our experiments. Over
the two transformed versions $B_{1}$ and $B_{2}$ of a batch $B$ of $N$ images,
the Huber loss is defined as the average loss over all $N$ positive pairs in
$B_{1}$ and $B_{2}$:

$\mathcal{L}_{huber}=\operatorname{mean}(l_{1},...,l_{N}).$ (4)

Including this term, the new regularized objective function becomes

$\mathcal{L}=\mathcal{L}_{nt\mbox{-}xent}+\lambda\mathcal{L}_{huber}$ (5)

Notably, our approach distinguishes itself by not necessitating the
utilization of a memory bank, a larger batch size, extended training periods
for effective generalization, or a huge dataset like ImageNet with a
substantial number of samples.

###  II-C Fréchet Batch Curation

In this paper, we mainly focus on two problems; avoiding learning false
positive and negative views, and updating the gradient with each randomly
curated batch. In our reasonable non-stochastic method; the FRD score measures
the similarity between two distributions of augmented images which is less
than the threshold value, the batch updates the gradients otherwise the batch
is reconstructed. The threshold value is calculated as the average FRD scores
of batches at epoch 5. We calculate the threshold value of batches after a few
epochs of training since random initialization of model weights can be harmful
to FRD scores. During the initial five epochs, we employ algorithms without
batch curation to facilitate the learning of representations. In our
experiments, empirical testing of various starting epochs reveals that the 5th
epoch is optimal as the starting point for curating the batch algorithm. We
hypothesize that some unusual combination of image transformation and training
images can produce misleading examples (as illustrated in Figure—1) that
hampers the convergence and performance of contrastive learning. This section
proposes to discard “bad batches” $B$ by considering the average FRD between
the two augmented batches $B_{1}$ and $B_{2}$. The Fréchet distance calculates
the difference between two Normal distributions. In the generative learning
field, the Fréchet Inception Distance (FID) is commonly employed to measure
the similarity between real and fake images in the generative learning
literature. We extract representations in the latent space including the
projection head, as depicted in figure 2 ($Z,Z^{\prime}$), for (FRD) scores.
The Fréchet ResNet Distance $FRD(\cdot)$, inspired by FID [heusel2017gans],
calculates the difference between the augmented views of an image batch as
distributions in the latent space of ResNet backbone network. Given a
candidate batch $B$, and two augmented batches $B_{1},B_{2}$ produced by the
application of random transformations drawn from $\mathcal{T}$, we can
estimate the mean and covariance $\mu_{1}$, $\mu_{2}$ and $\Sigma_{1}$,
$\Sigma_{2}$ of their latent ResNet representations, and calculate the FRD:

(7)