# Learning from Multiple Expert Annotators for Enhancing Anomaly Detection in
Medical Image Analysis

Khiem H. Le1,†, Tuan V. Tran1,†, Hieu H. Pham1,2,3,∗
Hieu T. Nguyen1, Tung T. Le1, Ha Q. Nguyen1,2 1Smart Health Center, VinBigData
JSC, Hanoi, Vietnam
2College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam
3VinUni-Illinois Smart Health Center, Hanoi, Vietnam

###### Abstract

Building an accurate computer-aided diagnosis system based on data-driven
approaches requires a large amount of high-quality labeled data. In medical
imaging analysis, multiple expert annotators often produce subjective
estimates about “ground truth labels” during the annotation process, depending
on their expertise and experience. As a result, the labeled data may contain a
variety of human biases with a high rate of disagreement among annotators,
which significantly affect the performance of supervised machine learning
algorithms. To tackle this challenge, we propose a simple yet effective
approach to combine annotations from multiple radiology experts for training a
deep learning-based detector that aims to detect abnormalities on medical
scans. The proposed method first estimates the ground truth annotations and
confidence scores of training examples. The estimated annotations and their
scores are then used to train a deep learning detector with a re-weighted loss
function to localize abnormal findings. We conduct an extensive experimental
evaluation of the proposed approach on both simulated and real-world medical
imaging datasets. The experimental results show that our approach
significantly outperforms baseline approaches that do not consider the
disagreements among annotators, including methods in which all of the noisy
annotations are treated equally as ground truth and the ensemble of different
models trained on different label sets provided separately by annotators.

###### keywords:

Supervised learning , multiple annotators , object detection.

††journal: Neurocomputing

##  1 Introduction

Computer-aided diagnosis (CAD) systems for medical imaging analysis are
getting more and more successful thanks to the availability of large-scale
labeled datasets and the advances of supervised learning algorithms [1, 2]. To
reach expert-level performance, those algorithms usually require high-quality
label sets, commonly scarce because of the costly and intensive labeling
procedures. A typical label collection process in medical imaging is
“repeated-labeling”, where multiple clinical experts annotate each data
instance to overcome human biases [3, 4, 5]. However, because of the
differences from annotator biases and proficiency, annotations from the
repeated-labeling process often suffer from high inter-reader variability [6,
7, 8], which could reduce leaning performance if we treat them as ground-
truth.

Many prior works have been done to mitigate inter-reader variations in
annotations, which can be categorized into two main groups: (i) one-stage
approach and (ii) two-stage approach. The first group learns the model,
annotators’ proficiency, and latent true labels jointly. Meanwhile, the second
group first estimates the true label of each instance from its multiple label
sets [9]. This process is known as “truth inference”. After that, a supervised
learning model is trained on the estimated true labels. All of those
approaches show impressive results on both classification and segmentation
problems [10, 11].

This work aims at addressing a fundamental question “How to train a deep
learning-based detector effectively from a set of possibly noisy labeled data
provided by multiple annotators?” [12]. To this end, we introduce a novel
approach that learns from multiple expert annotators to improve the
performance of a deep neural network in detecting abnormalities from chest
X-ray images. The proposed approach, as visualized in Figure 1, consists of
two stages. The first one is truth inference using Weighted Boxes Fusion (WBF)
algorithm [13] to estimate the true labels and their confidence scores. The
second stage is to train an object detector on estimated labels with a re-
weighted loss function using implicit annotators’ agreement, which is
represented by the estimated confidence scores. For evaluation, we first
simulate and test the proposed approach on a multiple-experts-detection
dataset from MNIST [14] called MED-MNIST. We then validate our approach on a
real-world chest X-ray dataset with radiologist’s annotations. Experiments on
those scenarios demonstrate that the proposed approach provides better
detection performance in terms of mAP scores than the baseline of treating
multiple annotations as ground truth and the ensemble of models supervised by
individual expert annotations.

In summary, our main contributions in this work are two-folds:

  * 1.

First, we introduce a simple yet effective method that allows a deep learning
network to learn from multiple annotators to improve its performance in
detecting abnormalities from medical images. The proposed approach aims at
estimating the true annotations from multiple experts with confidence scores
and uses these annotations to train a deep learning-based detector. This helps
remove uncertainty in the learning process and provides higher label quality
to train predictive models.

  * 2.

Second, the proposed approach demonstrates its effectiveness on both simulated
and real medical imaging datasets by surpassing current state-of-the-art
methods on the context of learning with multiple annotators. In particular,
our method is simple and can be applied for a wide range of applications in
medical imaging and object detection in general. The codes used in the
experiments are available on our Github page at
https://github.com/huyhieupham/learning-from-multiple-annotators. We also have
made the dataset used in this study available for public access on our
project’s webpage at https://vindr.ai/datasets/cxr.

The rest of the paper is organized as follows. Related works on learning from
multiple annotators and weighted training techniques are reviewed in Section
2. Section 3 presents the details of the proposed method with a focus on how
to estimate the ground truth annotations from multiple experts. Section 4
provides comprehensive experiments on a simulated object detection dataset and
a real-world chest X-ray dataset. Section 5 discusses the experimental
results, some key findings, and limitations of this work. Finally, Section 6
concludes the paper.

##  2 Related works

Learning from multiple annotators. There are two major lines of research on
learning from multiple annotators: two-stage approaches [15, 9, 16] and one-
stage approaches [10, 17, 18]. Two-stage approaches infer the true labels
first, then train a model using the estimated ones. The most simple solution
for label aggregation is majority voting, in which the choice of majority
annotators regards as the truth [19]. However, when the skill levels of the
annotators differ, the majority voting strategy may not work well. This is a
common occurrence in the general “learning from crowds” problem when
“spammers” are present. Later approaches typically incorporate other
information into the truth inference procedure, such as the annotators’
proficiency [20], annotators’ confusion matrix [21, 22], or the difficulty of
each sample [23]. While two-stage approaches have the advantage of simplicity
in both implementing and debugging, they do not make use of the raw
annotations in model learning. One-stage approaches address this issue by
simultaneously estimating the hidden true labels and learning the desired
model from noisy labels of multiple annotators. Earlier works in this group
use Expectation Maximization (EM) algorithm [24] for jointly modeling the
annotators’ ability and the latent ground-truth. More recent approaches employ
end-to-end frameworks which enable the neural networks to learn directly from
the noisy labels [12], and further developed by incorporating annotators’
confusion matrix [11, 10], or instance features [17].

Weighted training examples. In this paper, we propose a new re-weighted loss
function in which we assign more weights to examples that we consider be more
confident. Previous works on the use of weighted training examples can be
briefly categorized into two groups: (i) emphasize hard examples and (ii)
emphasize easy examples. Methods in the group (i) include hard-example mining
[25, 26], which is a bootstrapping technique over the difficult examples;
boosting algorithms [27], where the misclassified examples in preceding weak
classifiers are assigned with higher weights; and focal loss [28] that
addresses class imbalance problems by adding a regulator to the cross-entropy
loss for focusing on hard negative examples. Works in the group (ii) are
instances of broader topics such as curriculum learning [29], which is
biologically inspired by human gradual learning, with easier examples are
preferred in early training stages; learning with noisy labels [30, 31], which
prefers examples with smaller training losses as they are more likely to be
clean.

Unlike any approaches above, we propose in this paper a new loss function that
assigns more weights to more confident examples that determine by the
consensus of multiple annotators. Our experimental results validate the
correctness of this hypothesis.

##  3 Proposed Method

This section presents details of the proposed method. We first give a
formulation on learning from multiple annotators (Section 3.1). We then
introduce a simple way to estimate the true labels from multiple annotators
(Section 3.2). Next, our network architecture and training methodology with a
new re-weighted loss function are described (Section 3.3).

###  3.1 Problem formulation

Given a set of $N$ training images $\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N}$
with corresponding bounding box annotations
$\left\\{\tilde{y}_{i}^{(r)}\right\\}_{i=1}^{N}$ from multiple annotators
where $\tilde{y}_{i}^{(r)}$ denotes the label for example $\mathbf{x}_{i}$
given by annotator $r\in\mathbb{S}(R)$, which ${\mathbb{S}}(R)$ is a set of
$R$ different expert annotators. In this study, we make use of those expert
annotations $\left\\{\tilde{y}_{i}^{(r)}\right\\}_{i=1}^{N}$ to estimate a
single set of true labels with confidence scores
$\left\\{\mathbf{y}_{i};c_{i}\right\\}_{i=1}^{N}$. We then train a supervised
object detector with the estimated labels using the proposed re-weighted loss
function. In order to evaluate the effectiveness of the proposed method, we
use a gold-standard test set
$\mathcal{T}=\left\\{\left(\mathbf{x}^{(j)},\mathbf{y}^{(j)}\right)\right\\}_{j=1}^{M}$
containing $M$ examples. In medical imaging scenarios, where the true labels
are not available, we obtain the gold-standard test labels $\mathbf{y}^{(j)}$
from the consensus of a group of experiences radiologists. Figure 1 below
shows an overview of the proposed method.

Figure 1: Illustration of the proposed approach that aims to build a deep
learning system for abnormal detection on medical scans from multiple expert
annotators. The training process contains two stages. The first stage focuses
on truth inference, in which it estimates the true labels using the WBF
algorithm [13] with the implicit annotator’s agreement as confidence scores.
The second uses the estimated confidence scores to train a deep learning-based
detector using a re-weighted object detection loss function. To provide
abnormality analysis during the testing phase, only the fully trained image
detector is required.

###  3.2 Estimating the true labels from multiple expert annotators

We firstly estimate the true labels using Weighted Boxes Fusion (WBF)
algorithm [13]. This technique is used for combining predictions from multiple
sources, i.e., using ensemble to achieve better prediction results or
combining labels of different expert annotators. We describe the WBF algorithm
in more detail in Algorithm 1. The final examples used to train deep learning
detectors contain merged boxes with confidence scores. The visualization of
fused boxes and the corresponding confidence scores are shown in Figure 2. Our
fusion box algorithm emphasizes that the greater agreement between bounding
boxes (e.g., two or three annotators have the same diagnosis for an abnormal
finding on the image), the more likely the box annotation is correct.

(a) The original annotations provided by multiple radiology experts. The same
abnormal finding is represented by the sample color.

(b) Fused boxes with corresponding confidence scores after applied the WBF
algorithm.

Figure 2: (a) Visualization of multiple expert annotations on a chest X-ray
example from the VinDr-CXR dataset [5] and (b) the fused boxes with confidence
scores obtained by the WBF algorithm.

###  3.3 Network architecture and training methodology

1

Input: An image $\mathbf{x}$ with a list of annotations $\tilde{y}$ given by a
set $\mathbb{S}(R)$ of $R$ experts. The expert $r\in\mathbb{S}(R)$ with
proficiency $p_{r}$ provides the annotations including $r_{x}$ boxes,
$A_{r}=\left[\text{box}_{1},\ldots,\text{box}_{r_{x}}\right]$. All of the
experts’ annotations being merged into a list $A$.

2

Output: A list of $k$ fused boxes
$F=\left[\text{box}_{1},\ldots,\text{box}_{k}\right]$.

3

Declare empty lists $L$ and $F$ for boxes clusters and fused boxes,
respectively. Each position in the list $L$ can have a cluster of boxes or a
single box. Each position in $F$ has only one box, which is the fused box from
the corresponding cluster in $L$. Iterate through all boxes in $A$ in a cycle
and attempt to find a matching box in the list $F$. Two boxes are defined
matched if they have a high degree of overlap $(\text{e.g. IoU}>0.4)$. If
there are more than one matching boxes in $F$, the one with the highest IoU
will be chosen. If the matching box is not found in step 1, add the current
box to $L$ and $F$ as new entry for the new cluster before moving on to the
next box in the list $A$. If the match is found in step 1, add this box to the
list $L$ at the position pos which corresponds to the matching box in the list
$F$ Set the fused box’s coordinates $F[\textit{pos}]$ to be the weighted
average of $T$ boxes accumulated in cluster $L[\textit{pos}]$ with the
following formulas:
$x_{1,2}\coloneqq\frac{\sum_{i=1}^{T}p_{i}x_{1,2}}{\sum_{i=1}^{T}p_{i}}$
$y_{1,2}\coloneqq\frac{\sum_{i=1}^{T}p_{i}y_{1,2}}{\sum_{i=1}^{T}p_{i}}$ Set
the the fused boxes’ confidence scores in $F$ to the number of boxes in the
corresponding cluster in $L$ once all boxes in $A$ have been processed.
$c\coloneqq c\min{(T,N)}$ The fused boxes with confidence scores now represent
the annotators’ level of agreement.

Algorithm 1 The WBF algorithm applied for multiple expert annotations

Object detection is a multi-task problem, in which the loss function consists
of two parts: (1) the localization loss $\mathcal{L}_{\text{loc}}$ for
predicting bounding box offsets and (2) the classification loss
$\mathcal{L}_{\text{cls}}$ for predicting conditional class probabilities. In
this work, we focus on one-stage anchor-based detectors. A general form of the
loss function for those detectors can be written as

$\displaystyle\begin{aligned}
\mathcal{L}\left(p,p^{*},t,t^{*}\right)&=\mathcal{L}_{cls}\left(p,p^{*}\right)+\beta
I(t)\mathcal{L}_{loc}\left(t,t^{*}\right)\\\
I(t)&=\left\\{\begin{array}[]{ll}1&\text{if}\operatorname{IoU}\left\\{a,a^{*}\right\\}>\eta\\\
0&\text{ otherwise},\end{array}\right.\end{aligned}$ (1)

where $t$ and $t^{*}$ are the predicted and ground truth box coordinates, $p$
and $p^{*}$ are the class category probabilities, respectively;
$\operatorname{IoU}\left\\{a,a^{*}\right\\}$ denotes the Intersection over
Union (IoU) between the anchor $a$ and its ground truth $a^{*}$; $\eta$ is an
IoU threshold for objectness, i.e. the confidence score of whether there is an
object or not; $\beta$ is a constant for balancing two loss terms
$\mathcal{L}_{cls}$ and $\mathcal{L}_{loc}$ [32].

We use fused boxes confidence scores $c^{i}_{k}$ obtained from Algorithm 1 to
get a re-weighted loss function that emphasizes boxes with high annotators
agreement. The new loss function, which we name it as Experts Agreement Re-
weighted Loss (EARL) can now be written as

$\mathcal{L}\left(p,p^{*},t,t^{*}\right)=c\mathcal{L}_{cls}\left(p,p^{*}\right)+c\beta
I(t)\mathcal{L}_{loc}\left(t,t^{*}\right),$ (2)

where $c$ is the fused box confidence score.

##  4 Experiments

We validate the proposed method in both synthetic and real-world scenarios:
(1) the MED-MNIST, an object detection dataset, which was simulated from MNIST
[14] with multiple expert annotations; (2) VinDr-CXR [5], a chest X-ray
dataset with labels provided by multiple radiologists. In the following
sections, we describe those two datasets and our experiment setup, as well as
the experimental results.

###  4.1 Datasets

####  4.1.1 MED-MNIST Dataset

Based on MNIST [14] – a database of handwritten digits, we synthesize a
multiple-experts-detection dataset so-called MED-MNIST by two steps: (1) we
construct the detection task by copying and pasting digits from MNIST into a
black background with digit sizes are randomly chosen from a predefined range,
the bounding box annotations would be the smallest rectangle that contains
digits as visualized in Figure 4a; (2) we simulate $R$ different expert
opinions for each sample, assuming those $R$ experts have the same proficiency
$p$. The expert annotations are generated by varying two key factors that
influence detection annotations: (i) class labels and (ii) object coordinates.
To synthesize the expert annotations on class labels, we use an unique
transition matrix $A_{k}(k\in\\{1,\ldots,R\\})$ for each expert $E_{k}$ to
compute probability distributions that represent the expert mis-
classification. The proposed transition matrix is shown in Figure 3. About the
object coordinates, we simulate bounding box annotations that are highly
overlapping with the true bounding box. Both factors (i) and (ii) are
controlled by proficiency $p$. More specifically, $A_{k}$ are diagonally
dominant $(a_{ii}>a_{ij}\text{ for all }i\neq j)$, and
$a_{ii}=\text{min}(\text{max}(0.5,\alpha),1)\text{ with
}\alpha\sim\mathcal{N}(p,0.05)\ $. The simulated bounding boxes are subject to
have IoU with the true bounding box being larger than $p$. Here we set the
number of expert annotations per sample $R$ to 3, and the proficiency $p$ to
$0.8$. The simulated MED-MNIST dataset consists of 5,000 samples for training,
1,000 for hold-out validation and 1,000 for testing.

(a) Original transition matrix

(b) Simulated expert transition matrix

Figure 3: Visualization of the original and synthesized transition matrices.
To simulate the false negative scenario, we use an additional class called
no_obj.

(a) MNIST Detection

(b) Simulated expert annotations

Figure 4: The MED-MNIST dataset with multiple expert annotations, obtained by
perturbing boxes and classes from the MNIST dataset [14].

####  4.1.2 VinDr-CXR Dataset

VinDr-CXR [5], by far the largest public chest X-ray database with
radiologist-generated annotations. It consists of 18,000 chest X-ray scans
that come with both the localization of critical findings and the
classification of common thoracic diseases. The dataset includes 15,000 scans
for training and 3,000 scans for testing. In particular, the annotations were
obtained by a group of 17 radiologists with at least eight years of
experience. Each image in the training set was independently labeled by three
radiologists, while the annotations in the test set were carefully treated and
obtained by the consensus of 5 radiologists. Several examples from the VinDr-
CXR dataset are shown in Figure 5.

(a)

(b)

Figure 5: Visualization of abnormal findings (different bounding box colors
represent different findings) from the VinDr-CXR dataset: (top) Each scan in
the training set was annotated by three different radiologists; (bottom) Test
set annotations were obtained from the consensus of five radiologists.

####  4.1.3 Rads-VinDr-CXR Dataset

One intriguing characteristic of the VinDr-CXR dataset [5] is that 94.28% of
the abnormal scans in the training set (3,315 out of 3,516) were annotated by
a group of three radiologists with their correspondence IDs being R8, R9 and
R10. As a result, we create Rads-VinDr-CXR, a sub-dataset that is only
annotated by those three radiologists. The Rads-VinDr-CXR serves as a suitable
multiple annotators dataset to validate the proposed approach.

###  4.2 Experimental Setup

####  4.2.1 Evaluation metric

For all experiments, we report the detection performance using the standard
mean average precision metric at a threshold of 0.4<EMAIL_ADDRESS>[33].
Specifically, a predicted object is a true positive if it has an IoU of at
least 0.4 with a ground truth bounding box. The average precision (AP) is the
mean of 101 precision values, corresponding to recall values ranging from 0 to
1 with a step size of 0.01. The final metric is the mean of AP over all lesion
categories. We also employ mAP@[0.5:0.95:0.05] as an additional metric to
assess the model’s performance on different IoU thresholds ranging from 0.5 to
0.95 with a step size of 0.05.

####  4.2.2 Implementation Details

The main detector used in our experiments is YOLOv5-S [34]. The network is
built with PyTorch 1.7.1 (https://pytorch.org/) and trained on two NVIDIA RTX
2080 Ti GPUs. All training and test images are resized to the dimension of
$640\times 640$ pixels. The detector is trained for 50 epochs with 1cycle
learning rate decay [35] using the SGD optimizer [36]. The initial learning
rate is set to 1e-3.

To validate the robustness of the proposed approach across different deep
learning detectors, we further train and evaluate EfficientDet [37] with sizes
D3 and D4. Specifically, all images are resized to 640×640 pixels and the
model is trained for 30 epochs with constant learning rate 3e-4 using the
AdamW optimizer [38].

####  4.2.3 Comparison with the state-of-the-art

To the best of our knowledge, there is no existing multiple-annotators model
for object detection tasks in the literature. Hence, we compare the
performance of the proposed method against the baseline, which uses all
experts’ annotations per example without taking into account the disagreement
among annotators. On the Rads-VinDr-CXR dataset, we further compare our method
with the Rads-ensemble, which is the ensemble of independent models trained on
separate radiologists’ annotation sets. In this case, the WBF algorithm is
used to combine the predictions of those models.

###  4.3 Experimental Results

Table 1 and Table 3 report the experimental results of the YOLOv5-S detector
on MED-MNIST and VinDr-CXR datasets, respectively. On both synthetic and real-
world datasets, the proposed approach outperforms the chosen baselines, even
with the ensemble of individual experts’ models. Specifically, on the test set
of the MED-MNIST dataset, our method reports an overall<EMAIL_ADDRESS>of 0.980 and
an overall mAP@[0.5:0.95:0.05] of 0.849. These results are much higher the
performance of the baseline with<EMAIL_ADDRESS>= 0.975 and mAP@[0.5:0.95:0.05] =
0.815, boosting the mAP scores of the baseline by 0.51% and 4.2%,
respectively. Experimental results on the VinDr-CXR and Rads-VinDr-CXR
datasets also validate the effectiveness of the proposed method. We achieve an
overall<EMAIL_ADDRESS>of 0.200 on the VinDr-CXR dataset and an overall<EMAIL_ADDRESS>of
0.158 on the Rads-VinDr-CXR dataset. We emphasize that these results
outperform both the baseline model, individual model trained on label provided
by individual annotator (i.e. R8, R9, R10), as well as the ensemble model.

The experimental results with EfficientDet detector are provided in Table 3.
We found that better detection performances compared to the baseline have been
reported. This evidence confirm the robustness of the proposed approach across
deep learning detectors.

Table 1:  Experimental results on the MED-MNIST dataset. The highest scores are highlighted in red. Method |<EMAIL_ADDRESS>| mAP@[0.5:0.95:0.05]
---|---|---
Baseline | 0.975 | 0.815
WBF+EARL (ours) | 0.980 | 0.849
Table 2:  Experimental results on the VinDr-CXR and Rads-VinDr-CXR datasets with the YOLOv5-S detector. The highest scores are highlighted in red. Dataset | Method |<EMAIL_ADDRESS>
---|---|---
VinDr-CXR | Baseline | 0.190
WBF+EARL (ours) | 0.200
Rads-VinDr-CXR | Baseline | 0.148
R8 | 0.121
R9 | 0.132
R10 | 0.124
Rads-ensemble | 0.154
WBF+EARL (ours) | 0.158
Table 3:  Experimental results on the VinDr-CXR dataset while EfficientDet is used as the detector. The scores are measured in mAP@[0.5:0.95:0.05], with highest values highlighted in red. | Baseline | WBF+EARL
---|---|---
EfficientDet-D3 | 0.1142 | 0.1353
EfficientDet-D4 | 0.1223 | 0.1431

##  5 Discussions

###  5.1 Key findings and meaning

To the best of our knowledge, the proposed method is the first effort to train
an image detector from labels provided by multiple annotators, which is
crucial in constructing high-quality CAD systems for medical imaging analysis.
In particular, we empirically showed a notable improvement in terms of mAP
scores by estimating the true labels and then integrating the implicit
annotators’ agreement into the loss function to emphasize the clean bounding
boxes over the noisy ones. The idea is simple but effective, allowing the
overall framework can be applied in training any image machine learning-based
detectors.

###  5.2 Limitations

Despite the higher predictive performance over the relevant baselines, we
acknowledge that the proposed method has some limitations. First, the overall
architecture is not end-to-end. It may not fully exploit the benefits of
combining truth inference and training the desired image detector. Second,
applying the WBF algorithm to annotation sets with a high level of noise may
produce low-quality training data. This case is quite impractical in the
medical imaging field when the annotators are experienced clinical experts,
but it frequently occurs in the general learning from crowds problems.

##  6 Conclusion

This paper concentrates on the use of annotations from multiple experts to
build a robust deep learning system for abnormality detection on medical
images. We propose using Weighted Boxes Fusion (WBF) algorithm to obtain the
aggregated annotations with the implicit annotators’ agreement as confidence
scores. The estimated annotations are then used to train a deep learning
detector with a re-weighted loss function that incorporates the confidence
scores to localize abnormal findings. We empirically demonstrate that the
proposed approach outperforms current state-of-the-art baseline approaches in
both synthetic and real-world scenarios. To the best of our knowledge, we
introduce for the first time an effective method that trains an object
detector from multiple annotators. We believe our method is simple and can be
applied widely in medical imaging.

##  7 Acknowledgements

This work was supported by Smart Health Center at VinBigData JSC. The authors
gratefully acknowledge all anonymous reviewers for their valuable comments and
suggestions.

## References

  * [1] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, C. I. Sánchez, A survey on deep learning in medical image analysis, Medical Image Analysis 42 (2017) 60–88.
  * [2] H.-P. Chan, L. M. Hadjiiski, R. K. Samala, Computer-aided diagnosis in the era of deep learning, Medical Physics 47 (5) (2020) e218–e227.
  * [3] M. I. Razzak, S. Naz, A. Zaib, Deep learning for medical image processing: Overview, challenges and the future, Classification in BioApps (2018) 323–350.
  * [4] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al., CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp. 590–597.
  * [5] H. Q. Nguyen, K. Lam, L. T. Le, H. H. Pham, D. Q. Tran, D. B. Nguyen, D. D. Le, C. M. Pham, H. T. Tong, D. H. Dinh, et al., VinDr-CXR: An open dataset of chest X-rays with radiologist’s annotations, arXiv preprint arXiv:2012.15029.
  * [6] T. Watadani, F. Sakai, T. Johkoh, S. Noma, M. Akira, K. Fujimoto, A. A. Bankier, K. S. Lee, N. L. Müller, J. W. Song, J. S. Park, D. A. Lynch, D. M. Hansell, M. Remy-Jardin, T. Franquet, Y. Sugiyama, Interobserver variability in the CT assessment of honeycombing in the lungs, Radiology 266 (3) (2013) 936–944.
  * [7] A. B. Rosenkrantz, R. P. Lim, M. Haghighi, M. B. Somberg, J. S. Babb, S. S. Taneja, Comparison of interreader reproducibility of the prostate imaging reporting and data system and likert scales for evaluation of multiparametric prostate MRI, AJR Am J Roentgenol 201 (4) (2013) W612–618.
  * [8] E. Lazarus, M. B. Mainiero, B. Schepps, S. L. Koelliker, L. S. Livingston, BI-RADS lexicon for US and mammography: interobserver variability and positive predictive value, Radiology 239 (2) (2006) 385–391.
  * [9] V. S. Sheng, J. Zhang, Machine learning with crowdsourcing: A brief summary of the past research and future directions, Proceedings of the AAAI Conference on Artificial Intelligence 33 (01) (2019) 9837–9843.
  * [10] R. Tanno, A. Saeedi, S. Sankaranarayanan, D. C. Alexander, N. Silberman, Learning from noisy labels by regularized estimation of annotator confusion, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 11244–11253.
  * [11] L. Zhang, R. Tanno, M.-C. Xu, C. Jin, J. Jacob, O. Cicarrelli, F. Barkhof, D. Alexander, Disentangling human error from ground truth in segmentation of medical images, in: H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, Vol. 33, Curran Associates, Inc., 2020, pp. 15750–15762.
  * [12] F. Rodrigues, F. Pereira, Deep learning from crowds, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, 2018.
  * [13] R. Solovyev, W. Wang, T. Gabruseva, Weighted boxes fusion: Ensembling boxes from different object detection models, Image and Vision Computing 107 (2021) 104117.
  * [14] Y. LeCun, C. Cortes, C. Burges, MNIST handwritten digit database, ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist 2.
  * [15] Y. Zheng, G. Li, Y. Li, C. Shan, R. Cheng, Truth inference in crowdsourcing: Is the problem solved?, Proc. VLDB Endow. 10 (5) (2017) 541–552.
  * [16] Y. Jin, M. Carman, Y. Zhu, Y. Xiang, A technical survey on statistical modelling and design methods for crowdsourcing quality control, Artificial Intelligence (2020) 103351.
  * [17] J. Li, H. Sun, J. Li, Z. Chen, R. Tao, Y. Ge, Learning from multiple annotators by incorporating instance features (2021).  arXiv:2106.15146.
  * [18] L. Zhang, R. Tanno, M.-C. Xu, C. Jin, J. Jacob, O. Cicarrelli, F. Barkhof, D. Alexander, Disentangling human error from ground truth in segmentation of medical images, in: H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, Vol. 33, Curran Associates, Inc., 2020, pp. 15750–15762.
  * [19] V. S. Sheng, F. Provost, P. G. Ipeirotis, Get another label? Improving data quality and data mining using multiple, noisy labelers, in: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’08, Association for Computing Machinery, New York, NY, USA, 2008, p. 614–622.
  * [20] D. Karger, S. Oh, D. Shah, Iterative learning for reliable crowdsourcing systems, Advances in Neural Information Processing Systems 24.
  * [21] A. Dawid, A. Skene, Maximum likelihood estimation of observer error-rates using the em algorithm, Journal of The Royal Statistical Society Series C-applied Statistics 28 (1979) 20–28.
  * [22] P. Smyth, U. Fayyad, M. Burl, P. Perona, P. Baldi, Inferring ground truth from subjective labelling of venus images, in: Proceedings of the 7th International Conference on Neural Information Processing Systems, NIPS’94, MIT Press, Cambridge, MA, USA, 1994, p. 1085–1092.
  * [23] J. Whitehill, T.-f. Wu, J. Bergsma, J. Movellan, P. Ruvolo, Whose vote should count more: Optimal integration of labels from labelers of unknown expertise, Advances in Neural Information Processing Systems 22 (2009) 2035–2043.
  * [24] V. C. Raykar, S. Yu, L. H. Zhao, A. Jerebko, C. Florin, G. H. Valadez, L. Bogoni, L. Moy, Supervised learning from multiple experts: Whom to trust when everyone lies a bit, in: Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, Association for Computing Machinery, New York, NY, USA, 2009, p. 889–896.
  * [25] N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, 2005, pp. 886–893 vol. 1.
  * [26] A. Shrivastava, A. Gupta, R. Girshick, Training region-based object detectors with online hard example mining, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 761–769.
  * [27] R. E. Schapire, Explaining AdaBoost, in: Empirical Inference, Springer, 2013, pp. 37–52.
  * [28] T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object detection, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2980–2988.
  * [29] Y. Bengio, J. Louradour, R. Collobert, J. Weston, Curriculum learning, in: Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, Association for Computing Machinery, New York, NY, USA, 2009, p. 41–48.
  * [30] C. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals, Understanding deep learning (still) requires rethinking generalization, Communications of the ACM 64 (3) (2021) 107–115.
  * [31] D. Arpit, S. Jastrzundefinedbski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, S. Lacoste-Julien, A closer look at memorization in deep networks, in: Proceedings of the 34th International Conference on Machine Learning, ICML’17, JMLR.org, 2017, p. 233–242.
  * [32] Z. Zou, Z. Shi, Y. Guo, J. Ye, Object detection in 20 years: A survey, arXiv preprint arXiv:1905.05055.
  * [33] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, A. Zisserman, The Pascal Visual Object Classes (VOC) challenge, International Journal of Computer Vision 88 (2) (2010) 303–338.
  * [34] G. Jocher, A. Stoken, J. Borovec, NanoCode012, A. Chaurasia, TaoXie, L. Changyu, A. V, Laughing, tkianai, yxNONG, A. Hogan, lorenzomammana, AlexWang1900, J. Hajek, L. Diaconu, Marc, Y. Kwon, oleg, wanghaoyang0106, Y. Defretin, A. Lohia, ml5ah, B. Milanko, B. Fineran, D. Khromov, D. Yiwei, Doug, Durgesh, F. Ingham, ultralytics/yolov5: v5.0 - YOLOv5-P6 1280 models, AWS, Supervise.ly and YouTube integrations (Apr. 2021).
  * [35] L. N. Smith, Cyclical learning rates for training neural networks, in: IEEE Winter Conference on Applications of Computer Vision (WACV), 2017, pp. 464–472.
  * [36] S. Ruder, An overview of gradient descent optimization algorithms, arXiv preprint arXiv:1609.04747.
  * [37] M. Tan, R. Pang, Q. V. Le, Efficientdet: Scalable and efficient object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10781–10790.
  * [38] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101.


# Region-Aware Image-Based Human Action Retrieval with Transformers

Hongsong Wang<EMAIL_ADDRESS>Jianhua Zhao  Jie Gui
<EMAIL_ADDRESS>Department of Computer Science and Engineering, Southeast
University, Key Laboratory of New Generation Artificial Intelligence
Technology and Its Interdisciplinary Applications Nanjing, China  School of
Cyber Science and Engineering, Southeast University, Purple Mountain
Laboratories, Engineering Research Center of Blockchain Application,
Supervision And Management Nanjing, China

###### Abstract

Human action understanding is a fundamental and challenging task in computer
vision. Although there exists tremendous research on this area, most works
focus on action recognition, while action retrieval has received less
attention. In this paper, we focus on the neglected but important task of
image-based action retrieval which aims to find images that depict the same
action as a query image. We establish benchmarks for this task and set up
important baseline methods for fair comparison. We present a Transformer-based
model that learns rich action representations from three aspects: the anchored
person, contextual regions, and the global image. A fusion transformer is
designed to model the relationships among different features and effectively
fuse them into an action representation. Experiments on both the Stanford-40
and PASCAL VOC 2012 Action datasets show that the proposed method
significantly outperforms previous approaches for image-based action
retrieval.

###### keywords:

###### MSC:

41A05, 41A10, 65D05, 65D17 Human action retrieval , Instance-level image
retrieval , Image-based action understanding

††journal: Computer Vision and Image Understanding

##  1 Introduction

Understanding human actions from visual data is a crucial task in the computer
vision community, with tremendous applications in video surveillance,
autonomous driving, human–robot interaction, health monitoring, etc. Action
recognition and action retrieval are two important tasks for human action
understanding. The former aims to classify human action in a given video or
image, while the latter strives to find videos or images that depict the same
action as a query video or image.

Despite great progress has been made, most works focus on video-based human
action understanding and particularly on the task of video-based action
recognition Kong and Fu (2022). Action retrieval has received less attention
than action recognition in the academic community. Limited existing works of
human action retrieval are primarily targeted at actions in the video Wray et
al. (2019); Kico et al. (2022a). Although videos contain more information that
serves better action recognition, understanding actions in videos requires
more computational resources.

Fig. 1: The illustration of image-based human action retrieval. The red box
indicates the anchored region of the person who performs the action.

The understanding of human actions in images, which lack temporal information,
is a challenging task since the semantic information contained in images is
sparser and more abstract than that in videos. Image-based human action
recognition aims to distinguish the static action categories in images,
usually the bound box of the person who performs the target action is provided
to recognize the corresponding action. Previous approaches can be roughly
categorized into body part-based Zhao et al. (2017); Li et al. (2020) and
human-object interaction-based Wu and Yu (2021); Girish et al. (2020). The
part-based approaches require semantic labeling of body parts to allow precise
analysis of human actions, and pose additional challenges in terms of
annotation efforts and computational complexity. The human-object interaction-
based approaches require additional annotation of objects and are limited in
their ability to handle actions involving a person manipulating an object. The
existing works on image-based human action recognition primarily focus on
closed-set scenarios, while research on open-set human action understanding
remains scarce.

Image-based visual retrieval aims to search and retrieve relevant images by
providing a sample image describing visual features. Typical tasks involves
content-based image retrieval Dubey (2021), person re-identification Ye et al.
(2021); Zhang et al. (2024), sketch-based image retrieval Bhunia et al. (2020)
and image-based patent retrieval Wang and Zhang (2023). However, these works
primarily concentrate on retrieving contents such as objects or landmarks
within images, whereas retrieving human actions from images remains an
underexplored problem.

Although image-based action retrieval can be seen as a particular application
of instance-level image retrieval, it is more challenging than retrieving the
landmark or product image. This is due to the fact that understanding human
actions involves a semantically higher level of perception than object
recognition, and there are often more significant appearance differences among
images depicting the same action. Fig. 1 shows the example of image-based
action retrieval. Considering the action of “repairing the bike,” the query
image can easily be confused with the action of “riding the bike,”
particularly when the appearances or persons depicted in the image are very
similar. In addition, image-based human action retrieval not only promotes
deeper research into visual human action understanding, but also demonstrates
significant practical potential in real-time and efficient action
recognition/retrieval scenarios.

In this paper, we study the neglected but important task of image-based action
retrieval. As the image may contain multiple persons, an anchored region is
required to indicate the person who performs the action. Since the action
involves interactions between the person and contextual objects, we
hypothesize that the possible regions located close to the human in the image
also play a more important role in understanding human actions. Considering
that the Transformer Vaswani et al. (2017) is capable of learning a joint
representation that integrates information from multiple modalities, rendering
it highly suitable for fusion tasks, we employ the Transformer to fuse the
representations of the person, contextual objects, and their interactions. In
the query example of repairing the bike in Fig. 1, simple feature fusion
strategies like averaging pooling or concatenation are unable to capture
intricate interactions and relationships between the person and related
objects. In contrast, Transformer can learn the joint representations that
encapsulates the complementary information between them.

Accordingly, we propose an end-to-end model that learns multi-level
representations of different granularities for image-based human action
retrieval. Multi-level representations are representation of anchored person,
representations of contextual regions and the global representation. The
representation of anchored person and the global representation capture the
discriminative information of the human performing the action and the whole
image, respectively. To obtain reliable contextual regions, we first employ
proposal generation to generate bounding boxes encompassing potential object
regions, then a simple contextual regions ranking module is designed to select
proposals. Finally, a fusion transformer module is proposed to fuses features
from different levels into an efficient representation using type embedding
and positional embedding.

In summary, the contributions of this paper are as follows:

  * 1.

We empirically study the neglected task of image-based human action retrieval,
and establish new benchmarks and important baselines to promote research in
this field.

  * 2.

We introduce an efficient Region-aware Image-based human Action Retrieval with
Transformers (RIART), which leverages both person-related and contextual
object cues, and employs a fusion transformer module for human action
retrieval.

  * 3.

The proposed method has significant performance improvement than other
recently proposed instance-level image retrieval methods on several important
benchmarks.

##  2 Related Work

###  2.1 Image-Based Action Recognition

Image-based action recognition is a complex and challenging endeavor that
requires a deep comprehension of the human actions depicted within the image.
Deep learning has been widely applied to this task and achieved remarkable
results. We categorize recent works on this topic into three groups: body
part-based method, human-object interaction-based method and transfer
learning-based method.

Body part-based method focuses on extracting features from different parts of
the human body and learning their semantic meanings. For instance, Zhao et al.
(2017) divide the human body into several parts and assign each part a
semantic part action label. Li et al. (2020) propose a body structure
exploration sub-network that captures the global and local structure
information of human bodies. These methods aim to exploit the fine-grained
details of human poses for action recognition.

Human-object interaction-based method studies the relationship between humans
and objects in the scene and use it as a cue for action recognition. For
example, Wu and Yu (2021) introduce an efficient relation module that
integrates both human-object and scene-object relationships for the purpose of
enhancing action recognition capabilities. Girish et al. (2020) develop an
action recognition method based on target detection and pose estimation that
leverages the spatial relationship between humans and objects and the
hierarchical relationship between action categories. These methods aim to
exploit the contextual information of objects for action recognition.

Transfer learning-based method uses pre-trained models on large-scale datasets
and fine-tune them on specific action recognition datasets. For example,
Mohammadi et al. (2019) employ the transfer learning technique to address the
problem of data scarcity in action recognition datasets. Chakraborty et al.
(2021) use a pre-trained model on ImageNet and fine-tune it on a small-scale
action recognition dataset using a novel loss function. These methods aim to
exploit the generalization ability of pre-trained models for action
recognition. Wang et al. (2018) aim to hallucinate temporal characteristics
from static images, leveraging video memory techniques, in order to enhance
action recognition capabilities with few still images.

To the best of our knowledge, only a very limited number of works Ramanathan
et al. (2015); Li et al. (2011) have studied image-based action retrieval,
primarily owing to the scarcity of benchmarks. The work in Ramanathan et al.
(2015) follows the traditional Bag-of-Words pipeline, which is based on
manually designed local feature descriptors. The work Li et al. (2011) aims to
learn global image-level action representations by leveraging the semantic
relationship between different actions. However, both models are only suitable
for simple images that contain a single action and are not scalable enough for
large-scale applications. In contrast, we comprehensively study image-based
action retrieval and introduce an efficient, deep learning-based pipeline.

###  2.2 Human Action Retrieval

Content-based video retrieval Spolaôr et al. (2020) aims to search for
relevant videos based on a query video. Given that videos contain vast amounts
of raw data, techniques such as video segmentation, feature extraction, and
dimensionality reduction can be employed to enhance efficiency. The challenge
lies in achieving high accuracy while ensuring efficient computation and low
storage consumption Jiang et al. (2021).

Human action retrieval is a new and challenging topic that aims to find videos
or images that contain specific human actions. Early works rely on efficient
feature matching methods based on local features Tang et al. (2013). Most
existing methods retrieve human actions from videos by describing local
features of the motion patterns Veinidis et al. (2019). Kico et al. (2022a)
present a metric learning-based method to learn a feature space that preserves
the similarity of human actions and enables efficient content-based retrieval.
Some other methods Veinidis et al. (2019) use unsupervised approaches to
cluster and retrieve action videos based on motion features. There are also
works Kico et al. (2022b) that focus on content-based action retrieval from
human skeleton sequences. It should be noted that human action retrieval is
also closely related to zero-shot action recognition C et al. (2021), which
aims to recognize novel action categories without training data. However,
zero-shot action recognition emphasizes generalization to unseen classes,
while action retrieval emphasizes accuracy and efficiency of retrieval.

###  2.3 Instance-level Image Retrieval

Instance-level image retrieval involves searching a vast database to retrieve
images that contain an object identical to the one specified in a query image.
This task requires effective and efficient methods to represent and compare
images. Some works Siméoni et al. (2019); Cao et al. (2020) adopt a geometric
verification approach to rerank images based on local features. Recently, some
methods Radenović et al. (2018); Tan et al. (2021); Wu et al. (2022) have
aimed at learning more powerful representations of images using global
features.

Siméoni et al. (2019) propose a new Deep Spatial Matching (DSM) method for
image retrieval that uses an initial ordering of image descriptors extracted
from the activation layer of a convolutional neural network, followed by a set
of local features approximating the same sparse 3D activation tensor. Cao et
al. (2020) integrate global and local features within a unified deep model,
facilitating precise retrieval through efficient feature extraction. This
integration is achieved by leveraging generalized mean pooling for capturing
holistic global characteristics and attentive selection mechanisms for
pinpointing salient local details. Radenović et al. (2018) fine-tune CNNs for
image retrieval on extensive collections of unordered images. They use hard
negative mining and feature normalization to improve the performance of global
descriptors. Tan et al. (2021) introduce Reranking Transformers as a versatile
model that incorporates both local and global features to refine the ranking
of matched images in a supervised manner, thereby streamlining the typically
resource-intensive process of geometric verification. Wu et al. (2022) propose
a tokenizer module to aggregate deep local features of images into a few
visual tokens, then enhance these tokens by a refinement block, and finally
generate the global representation by concatenating these visual tokens.

These methods effectively retrieve landmarks and products, yet they are not
fit for retrieving action images which contain richer semantic information
than landmarks. Our approach retrieves action images more efficiently by
better understanding the semantics in images.

##  3 Method

To enhance understanding of human actions within images, we aim to harness
representations that span various levels of granularity: the anchored person,
contextual regions, and the entire image. The anchored person specifically
denotes the human body, whereas contextual regions represent objects proximate
to the human that are pertinent to the action being depicted. We propose an
efficient Region-aware Image-based human Action Retrieval with Transformer
(RIART). The overall architecture of the proposed method is illustrated in
Fig. 2, and the details of its key components are described as follows.

Fig. 2: The overall architecture of the proposed Region-aware Image-based
human Action Retrieval with Transformer (RIART). The RIART takes an image and
the corresponding anchored region of the human bounding box as input,
generating a human action representation that can be utilized for action
retrieval or recognition. To acquire contextual regions associated with the
action, an off-the-shelf proposal generation module is employed to generate
object proposals, which are then filtered and ranked by a contextual regions
ranking module. Multi-level action representations are obtained, specifically,
the representation of the anchored person $F_{a}$, representations of
contextual regions $\mathbf{F_{i}}$, and the global image-level representation
$F_{g}$. Subsequently, these representations are fused by a fusion transformer
module to yield the final representation $F_{f}$.

###  3.1 Problem Formulation

Fig. 3: Examples query image for action retrieval. Human bounding boxes
indicate the person performing a certain action. (a) Simple image that
contains only one human action. (b) Complex image that contains multiple
actions from different persons. Note that actions of different persons in the
same image can be different.

Image-based human action retrieval refers to the process of retrieving images
from a database that depict specific human actions. Unlike video-based human
action retrieval, the actions of interest for this task are typically static
actions that can be exemplified in a single image. Examples of query image are
shown in Fig. 3. The query image may contain multiple action classes within
the same image. We assume that a given human bounding box in an image only
encompasses one action class. If the query image depicts multiple persons
performing different actions, each image with distinct human bounding boxes is
regarded as a separate query example.

###  3.2 Representation of Anchored Person

The input image may contain multiple persons who perform different actions. To
understand the action performed by a particular person in the image, an
anchored person should be used to identify the position of the human.

To facilitate image-based action understanding, it is crucial to learn the
representation of the anchored person. To achieve this, we utilize ROI pooling
to extract this representation based on the provided bounding box. Let
$\mathbf{F}$ be the output feature map of the backbone network, $B_{a}$ be the
bounding box of the person, representation of the anchored person $F_{a}$ is

${F_{a}}=\mathrm{ROI\\_Pooling}(\mathbf{F},B_{a}).$ (1)

where $\mathrm{ROI\\_Pooling}(\cdot)$ represents the Region of Interest
Pooling function Ren et al. (2015), which takes as input a feature map from
and a bounding box defining a Region of Interest (ROI), and outputs a fixed-
size feature map from the specified ROI.

###  3.3 Representations of Contextual Regions

Many human actions involve interactions with surrounding objects and
environments, therefore, learning representations of contextual regions around
the individual performing the action is crucial for action understanding. We
employ a proposal generation module to generate bounding boxes encompassing
potential object regions. The proposal generation process often generates
thousands of proposals, most of which are unrelated to the concerned action.
Therefore, we design a contextual regions ranking module to select a few
useful proposals of contextual regions.

The contextual regions ranking module selects the proposals nearest to the
anchored person, with confidence scores exceeding a predefined threshold. The
Intersection-over-Union (IoU) metric is employed to measure the similarity
between two boxes. We select the top $k$ bounding boxes with the highest IoU
scores with the bounding box of anchored person. This process is formulated as

$\displaystyle\widetilde{\mathbb{P}}=$ $\displaystyle\
\mathrm{Rank\\_Select}(\mathrm{IoU}(\mathbb{P},B_{a})),$ (2)
$\displaystyle\mathbf{F_{l}}=$ $\displaystyle\
\mathrm{ROI\\_Pooling}(\mathbf{F},\widetilde{\mathbb{P}})),$

where $\mathbb{P}$ denotes the proposals with the confidence scores above the
threshold, $\widetilde{\mathbb{P}}$ denotes the selected top-k proposals about
contextual regions, $\mathbf{F_{l}}$ is the set of representations of
contextual regions (for symbols’ fonts, italic and bold styles are used to
denote feature vectors and sets of features, respectively).

The whole image contains rich contextual information about the action. To
utilize the overall contextual information for the whole image, we also use
the global representation of the image for action understanding. The global
image-level feature is

$F_{g}=\ \mathrm{Max\\_Pooling}(\mathbf{F}),$ (3)

where $\mathrm{Max\\_Pooling}$ denotes the max pooling operation.

###  3.4 Fusion Transformer

Considering the advantages of the Transformer in global dependency modeling,
we design a Transformer-based fusion module to merge multi-level
representations from different aspects. The fusion transformer, shown in
Fig.4, consists of a cascade of $N$ Transformer blocks. The input of
Transformer-based fusion is a set of multi-level representations, i.e.,
$\\{F_{a},F_{g},\mathbf{F_{l}}\\}$. We flatten these features and convert them
into a sequence of tokens. The number of input tokens is $(k+2)$, where $k$ is
the number of selected contextual regions. The dimension of the concatenated
features is ${(k+2)\times D}$, where $D$ is the dimension of representation.

Subsequently, we add positional embedding and type embedding before feeding
concatenated features into the transformer block. The type embedding $typ$
identifies different aspects of features in the concatenated sequence of
tokens. The positional embedding $pos$ provide information about the position
of the feature in the original feature map after being flattened into tokens.
The calculation of positional embedding is

$pos_{t}^{(j)}=\left\\{\begin{array}[]{l}\sin\left(w_{i}t\right),\quad\text{
if }j=2i\\\ \cos\left(w_{i}t\right),\quad\text{ if }j=2i+1\end{array}\right.$
(4)

where $t$ represents the position in the sequence of different features, $j$
indexes the dimension of the feature and $w_{i}$ is a frequency term that
varies with the iterating index $i$. The $w_{i}$ is

$w_{i}=\frac{1}{10000^{2i/D}},i=0,1,\ldots,\frac{D}{2}-1.$

This formulation captures both fine-grained and coarse-grained positional
information across different dimensions of features.

The output sequence also has $(k+2)$ tokens, and we simply average them to get
final features. The above processes could be summarized as follows

$\displaystyle emb_{i}=pos_{i}+typ_{i},i=(1,2,3),$ (5) $\displaystyle
F_{c}=Concat(F_{a}+emb_{1},F_{g}+emb_{2},\mathbf{F_{l}}+$ $\displaystyle
emb_{3}),$ $\displaystyle
F_{T}=\mathrm{Avg}(\mathrm{Fusion\\_Transformer}(F_{c})),$

where the $pos_{i}$ denotes the positional embedding while the $typ_{i}$
denotes the type embedding, $\mathrm{Avg}$ is the average operation. The
output representation $F_{T}$ is used to recognize or retrieve human action in
the image.

It should be noted that Transformer-based fusion has also appeared in recent
works Liu et al. (2022) for multi-modal human action recognition. For
instance, a cross-modality fusion module is applied to learn better multi-
modal representations Liu et al. (2022). Unlike the fusion modules of these
approaches, the proposed Fusion Transformer aims to learn efficient
representations from a single image modality by fusing multi-level or multi-
grained features. Additionally, type embedding and position embedding are
designed to facilitate the feature learning.

Fig. 4: The architecture of Fusion Transformer module. The input $F_{c}$
denotes multi-level action representations, i.e., representation of the
anchored person $F_{a}$, representations of contextual regions
$\mathbf{F_{i}}$, and the global image-level representation $F_{g}$. The
Fusion Transformer aims to capture the complementary information among these
diverse input representations and output a learned joint representation. To
facilitate the learning of this complementary action representation, type
embeddings and position embeddings are employed. Similar to the standard
Transformer architecture, the Fusion Transformer module consists of $N$
blocks.

###  3.5 Training Strategy

We also add a batch norm layer on top of $F_{T}$ for better performance. We
train our model through the action classification task to learn the
representation $F_{T}$. We use a two layers MLP with ReLU Glorot et al. (2011)
activation function as the classifier to classify human actions in images. The
input of the classifier is $F_{T}$ which is the output feature of the
Transformer module. We adopt cross-entropy loss as the loss function of our
classifier:

$Loss=-\sum_{i=1}^{n}{p(x_{i})\log{q(x_{i})}},$ (6)

where the $p(x_{i})$ is the probability of the ground truth label, $q(x_{i})$
is the probability of the label predicted by the classifier and n is the
number of classes.

The action classification task is also the pretext task for the action
retrieval task. The representation $F_{T}$ learned by the action
classification task is used to represent the action image at retrieval time.

##  4 Experiments

###  4.1 Datasets and Implementation Details

Stanford-40. The Stanford 40 Action dataset Yao et al. (2011) is an image-
based static action recognition dataset with a diverse range of 40 actions.
Each image is annotated with a bounding box which indicates the person who is
performing the specific action. This dataset comprises a total of 9532 images,
with each action class containing a balanced and diverse representation of the
respective action, ranging from 180 to 300 images.

PASCAL VOC. The PASCAL VOC 2012 Action Classification Everingham et al. (2015)
dataset comprises a comprehensive collection of annotated images, specifically
consisting of 2296 training images and 2292 validation images. There are 10
actions which spans a range of categories. For each image, annotations are
provided to indicate the presence of an action and its corresponding class
label.

Implementation Details. For proposal generation, we use FasterRCNN Ren et al.
(2015) with pre-trained parameters. The default backbone for the
representation learning module is ResNet-50 He et al. (2016). Additionally, a
Transformer-based backbone, specifically Swin-T Liu et al. (2021), is also
employed due to its comparable model size to ResNet-50. The fusion transformer
has 3 Transformer blocks with 8 heads each and is trained from scratch. The
input size of images is $224\times 224$, with data augmentation including
horizontal flipping, random crop, random color jitter, and mix-up during
training. We use the batch size of 128 to train our model on a single Nvidia
RTX 3090 GPU. The early-stop strategy is used and the max training epochs is
set to 150. AdamW is used to optimize the model with an initial learning rate
is 0.00003. The number of selected bounding boxes $k$ is set to 10. The
feature dimension $D$ is set to 2048.

###  4.2 Experimental Setup

Image-based Action Retrieval. As there are no benchmarks for image-based
action retrieval, we establish experimental settings and conduct baselines for
subsequent analysis and comparison. The experimental settings for image-based
action retrieval are constructed based on those of action recognition on both
the Stanford-40 and PASCAL VOC datasets. Specifically, we train the action
representation model on the training set, and consider the validation set as
both the query and gallery sets for action retrieval. As the query set and
gallery set are the same set, we remove the query sample from the gallery set
while retrieving. The query set refers to the collection of query images that
are used as input to the image-based retrieval model, while the gallery set is
a collection of images that the image retrieval system searches through to
find matches for the query images.

Baselines. To demonstrate the effectiveness of our approach, we select four
representative baseline methods: Reid strong baseline (Strong baseline) Luo et
al. (2019), DELG Cao et al. (2020), Reranking Transformer (RRT) Tan et al.
(2021), and Token Wu et al. (2022). Reid strong baseline Luo et al. (2019) is
a method for retrieving the same person that appears in photos taken from
different cameras. We use this method for the action retrieval task where we
set the query images and the gallery images as coming from different cameras
and treat the same class of actions as the same person in the original setting
of Reid strong baseline. DELG Cao et al. (2020) is an instance-level retrieval
method for identifying and retrieving different views of a specific instance
in a database. We mainly use the global feature in DELG to conduct the action
retrieval task. Reranking Transformer Tan et al. (2021) is another instance-
level image retrieval method that learns features contrastively between
samples using a Transformer. Token Wu et al. (2022) is also an instance-level
method for image retrieval, it designs a tokenizer module to aggregate
different features into a few vision tokens.

Evaluation Metrics. For action retrieval, we adopt the widely used mAP,
Rank-1, Rank-5 as the evaluation metrics. Reranking Luo et al. (2019); Zhang
et al. (2024) is a common post-processing step of reordering the initial
retrieved results based on additional criteria, with the goal of improving the
retrieved results. Unless otherwise specified, we use the reranking method in
Luo et al. (2019), and set the parameters $k1=200$, $k2=20$, and $lambda=0.3$.

Table 1: Results of human action retrieval on the Stanford-40 dataset. In
order to keep the model lightweight, we employ not only ResNet50 but also
Swin-T Liu et al. (2021) as the backbone. The choice of Swin-T stems from the
fact that its model size is comparable to that of ResNet50.

Method | w/o reranking | w/ reranking
---|---|---
mAP | Rank-1 | Rank-5 | mAP | Rank-1 | Rank-5
Strong baseline Luo et al. (2019) | 59.1 | 73.6 | 83.9 | 63.7 | 70.7 | 81.5
DELG global Cao et al. (2020) | 28.1 | 57.8 | 81.1 | 42.9 | 60.6 | 79.1
RRT Tan et al. (2021) | – | – | – | 53.9 | 53.4 | 55.9
Swin-T Liu et al. (2021) | 78.2 | 85.2 | 93.1 | 81.1 | 84.3 | 92.2
R50-Token Wu et al. (2022) | 62.9 | 78.7 | 89.2 | 71.8 | 77.8 | 87.1
RIART (ResNet-50) | 73.2 | 83.1 | 91.3 | 77.5 | 81.5 | 90.1
RIART (Swin-T) | 80.7 | 85.6 | 92.5 | 81.9 | 84.5 | 91.7

Table 2: Results of static action retrieval on the PASCAL VOC dataset.

Method | w/o reranking | w/ reranking
---|---|---
mAP | Rank-1 | Rank-5 | mAP | Rank-1 | Rank-5
Strong baseline Luo et al. (2019) | 26.0 | 51.3 | 82.6 | 32.0 | 49.6 | 78.2
DELG global Cao et al. (2020) | 31.2 | 58.5 | 86.1 | 43.6 | 60.1 | 84.2
RRT Tan et al. (2021) | – | – | – | 50.7 | 50.1 | 53.7
R50-Token Wu et al. (2022) | 46.3 | 65.5 | 84.5 | 52.2 | 62.9 | 83.4
RIART (ResNet-50) | 56.3 | 73.1 | 88.4 | 60.0 | 73.4 | 87.9

###  4.3 Results of Action Retrieval

Table 1 shows the results of action retrieval on the Stanford-40 dataset. Our
approach outperforms the baseline methods across all metrics. Specifically,
without reranking, our method surpasses the R50-Token Wu et al. (2022) by
10.3% in terms of mAP and 4.4% in Rank-1. Furthermore, when reranking is
employed, our approach beats the R50-Token Wu et al. (2022) by 5.7% and 3.7%
for the mAP and Rank-1, respectively. The Reid strong baseline Luo et al.
(2019) is unsuitable for action retrieval, exhibiting a 14.1% lower
performance than ours in terms of mAP when reranking is not used.

The results on the PASCAL VOC dataset are summarized in Table 2. Similar to
results on the Stanford-40 dataset, the worst-performing method is no longer
DELG Cao et al. (2020), followed by the Reid Strong Baseline Luo et al.
(2019). Specifically, without reranking, our method outperforms the R50-Token
Wu et al. (2022) by 10.0% in terms of mAP and 7.6% in Rank-1. Additionally,
when reranking is applied, our approach surpasses the R50-Token Wu et al.
(2022) by 7.8% for mAP and 10.5% for Rank-1. These experimental results
confirm the effectiveness of our method for image-based action retrieval.

###  4.4 Results of Action Recognition

As our approach is trained utilizing the classification loss, we also present
the results of image-based action recognition. Consistent with prior works
Zhao et al. (2017); Wu and Yu (2020), we report the mean Average Precision
(mAP) metric for action classification. We have chosen four representative
methods: Action Mask Zhang et al. (2016), Single Image Action Recognition Zhao
et al. (2017), the fine-tuned FasterRCNN Bas and Ikizler-Cinbis (2022), and
ResNet-50 He et al. (2016). The baseline single image action recognition Zhao
et al. (2017) uses features of the whole image and the human box to predict
the action in the image.

Table 3 shows results of action recognition on the Stanford-40 dataset. The
widely used ResNet-50 He et al. (2016) only has an mAP of 81.2%. The part-
based method Action Mask Zhang et al. (2016) and the baseline Zhao et al.
(2017) achieve the mAP of 82.6% and 84.2%, respectively. Our proposed achieves
89.0% mAP on action classification, and outperforms the baseline by 4.8%.

Table 3: Results of image-based action recognition on the Stanford-40. The
symbol “*” signifies human pose-based methods that utilize human pose as
auxiliary information, whereas the alternative approach solely necessitates
the global human bounding box.

Method | mAP
---|---
ResNet-50 He et al. (2016) | 81.2
Action Mask Zhang et al. (2016) | 82.6
Single image action recognition (Baseline) Zhao et al. (2017) | 84.2
Body Structure Cues Li et al. (2020)* | 93.8
Swin-T Liu et al. (2021) | 91.1
FasterRCNN finetune Bas and Ikizler-Cinbis (2022) | 78.0
MASPP Ashrafi et al. (2023)* | 94.8
RIART (ResNet-50) | 89.0
RIART (Swin-T) | 91.8

###  4.5 Ablation Studies

We perform ablation experiments to assess the effectiveness of each component
of our approach. First, we validate the three aspects of representations.
Subsequently, we investigate the significance of the fusion transformer
module. Furthermore, we evaluate the impact of the number of Transformer
blocks in the fusion transformer module. Lastly, we evaluate the efficacy of
positional and segment embedding. The ablation studies are conducted on the
Stanford-40 dataset.

Multi-Level Representations. There are three aspects of features: the anchored
representation $F_{a}$, the global representation $F_{g}$, and contextual
representations $\mathbf{F_{l}}$. Fig. 5 shows the results of using different
feature combinations on the Stanford-40 dataset. Using all three kinds of
features, our approach achieves the best performance in terms of mAP and
rank-5 accuracy compared to any combination of two kinds of features. When
removing the $F_{a}$, the mAP decreases by nearly 0.8%. Similarly, the mAP
falls by approximately 1.9% upon removing $F_{g}$. Furthermore, our approach
outperforms the variant that excludes the feature $\mathbf{F_{l}}$ by 0.9%.
This ablation study demonstrates that all three aspects of features contribute
significantly to learning an effective representation of the action image.

The Fusion Transformer Module. We remove the fusion transformer module and
instead use the average value of the input token sequences as a representation
of the action image. The results on the Stanford-40 dataset are also shown in
Fig. 5. We observe that the mAP and rank-5 accuracy decline significantly by
4.5% and 1.3%, respectively, upon removing the fusion transformer module. This
demonstrates that the fusion transformer effectively merges the features into
a superior representation of the action image.

The Positional and Type Embedding. We also investigate the significance of
positional embedding $pos$ and type embedding $typ$ utilized in our model.
Fig. 5 presents the results obtained using different embeddings on the
Stanford-40 dataset. Our model attains optimal performance when utilizing both
embeddings. Conversely, excluding both results in the poorest performance,
leading to a decrease of 0.4% in mAP and 0.7% in rank-5. Evidently, the final
representation benefits from both positional and type embeddings.

Fig. 5: (a) Ablation studies of features. (b) Ablation studies of embeddings in the fusion transformer. Table 4: Analyses of different number of blocks in the fusion transformer. No. of Blocks | mAP | Rank-1 | Rank-5
---|---|---|---
N=1 | 76.6 | 81.0 | 89.6
N=2 | 76.8 | 80.8 | 89.2
N=3 | 77.5 | 81.5 | 90.1
N=4 | 77.0 | 81.2 | 89.3
N=6 | 76.6 | 80.6 | 89.4
Fig. 6: Visualization of retrieval results. For each example, the query image
is on the left, and the top-$8$ retrieved images are displayed from left to
right. The red box indicates the correctly retrieved sample, whereas the other
images that do not have red boxes represent wrongly retrieved samples. DELG,
Baseline, and Token denote DELG global Cao et al. (2020), Strong baseline Luo
et al. (2019), Token Wu et al. (2022), respectively, which are representative
instance-level image retrieval methods. The anchored human box in the query
image is omitted for simplicity. The first four, i.e., “blowing bubbles”,
“drinking”, “fixing a bike” and “reading”, and our retrieved results are
mostly accurate. The middle two, i.e., “pouring liquid” and “texting message”
pose a bit of a challenge, and our approach can only correctly retrieve two
out of the top-8 results. The last four, i.e., “playing guitar”, “writing on a
book”, “fishing” and “phoning”, are hard examples, and most approaches almost
fail.

The Number of Transformer Blocks. To investigate the impact of varying the
number of Transformer blocks, we experimented with settings of 1, 2, 3, 4, and
6 blocks, respectively. As shown in Table 4, the performance initially
improves and then deteriorates as the number of blocks increases. Having
either too many or too few blocks can adversely affect performance. Our model
attains the best performance when using three blocks.

###  4.6 Visualizations

Fig. 6 shows the retrieval results for different methods. The image on the
left is the query, and the red boxes note the correct retrieved samples from
the gallery. Our proposed method outperforms other methods in terms of
retrieval performance. For the action “repair bicycle”, our method correctly
retrieves 7 out of the top-8 results. In a challenging case for the action
“Drinking”, our method retrieves only 4 correct samples out of the top-8
results. Our method tends to misclassify some error samples as “smoking”
actions, as they share significant similarities with the query action. The
last four, i.e., “playing guitar”, “writing on a book”, “fishing” and
“phoning”, represent challenging examples for image-based action retrieval,
and most approaches only achieve unsuccessful retrieval results. Nevertheless,
in comparison to other methods, the error samples retrieved by our method are
semantically more reasonable.

##  5 Conclusion

In this paper, we empirically investigate the overlooked task of image-based
human action retrieval, and establish new benchmarks and important baselines
for this task. We introduce an efficient Region-aware Image-based human Action
Retrieval with Transformers (RIART), which learns multi-level action
representations from three perspectives: the anchored person, contextual
regions, and the global image. A Transformer-based fusion module is also
proposed to integrate these features into an enhanced representation. Our
approach achieves state-of-the-art performance on both action recognition and
retrieval tasks. Furthermore, it demonstrates the capability to retrieve
images depicting the same or similar actions as a given query image using a
single image. Ablation studies highlight the effectiveness and complementarity
of the three types of representations. We hope that this work and the
established benchmarks can promote future research on image-based action
retrieval.

## Acknowledgments

This work is supported by the National Natural Science Foundation of China
(62302093, 62172090, 62202438), Jiangsu Province Natural Science Fund
(BK20230833), CAAI-Huawei MindSpore Open Fund, and the Southeast University
Start-Up Grant for New Faculty (RF1028623063, RF1028623097). This work is also
supported by the Big Data Computing Center of Southeast University.

## References

  * Ashrafi et al. (2023) Ashrafi, S.S., Shokouhi, S.B., Ayatollahi, A., 2023.  Still image action recognition based on interactions between joints and objects.  Multimedia Tools and Applications 82, 25945–25971.
  * Bas and Ikizler-Cinbis (2022) Bas, C., Ikizler-Cinbis, N., 2022\.  Top-down and bottom-up attentional multiple instance learning for still image action recognition.  Signal Processing: Image Communication 104, 116664.
  * Bhunia et al. (2020) Bhunia, A.K., Yang, Y., Hospedales, T.M., Xiang, T., Song, Y.Z., 2020.  Sketch less for more: On-the-fly fine-grained sketch-based image retrieval, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9779–9788.
  * C et al. (2021) C, V.E.A., B, H.P., C, D.M., 2021.  Zero-shot action recognition in videos: A survey.  Neurocomputing 439, 159–175.
  * Cao et al. (2020) Cao, B., Araujo, A., Sim, J., 2020.  Unifying deep local and global features for image search, in: Proceedings of the European Conference on Computer Vision, Springer. pp. 726–743.
  * Chakraborty et al. (2021) Chakraborty, S., Mondal, R., Singh, P.K., Sarkar, R., Bhattacharjee, D., 2021.  Transfer learning with fine tuning for human action recognition from still images.  Multimedia Tools and Applications 80, 20547–20578.
  * Dubey (2021) Dubey, S.R., 2021.  A decade survey of content based image retrieval using deep learning.  IEEE Transactions on Circuits and Systems for Video Technology 32, 2687–2704.
  * Everingham et al. (2015) Everingham, M., Eslami, S.A., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A., 2015\.  The pascal visual object classes challenge: A retrospective.  International Journal of Computer Vision 111, 98–136.
  * Girish et al. (2020) Girish, D., Singh, V., Ralescu, A., 2020.  Understanding action recognition in still images, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 370–371.
  * Glorot et al. (2011) Glorot, X., Bordes, A., Bengio, Y., 2011.  Deep sparse rectifier neural networks, in: Proceedings of International Conference on Artificial Intelligence and Statistics, pp. 315–323.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.
  * Jiang et al. (2021) Jiang, C., Huang, K., He, S., Yang, X., Zhang, W., Zhang, X., Cheng, Y., Yang, L., Wang, Q., Xu, F., et al., 2021.  Learning segment similarity and alignment in large-scale content based video retrieval, in: Proceedings of the ACM International Conference on Multimedia, pp. 1618–1626.
  * Kico et al. (2022a) Kico, I., Sedmidubsky, J., Zezula, P., 2022a.  Towards efficient human action retrieval based on triplet-loss metric learning, in: International Conference on Database and Expert Systems Applications, Springer. pp. 234–247.
  * Kico et al. (2022b) Kico, I., Sedmidubsky, J., Zezula, P., 2022b.  Towards efficient human action retrieval based on triplet-loss metric learning, in: International Conference on Database and Expert Systems Applications, Springer-Verlag. pp. 234––247.
  * Kong and Fu (2022) Kong, Y., Fu, Y., 2022.  Human action recognition and prediction: A survey.  International Journal of Computer Vision 130, 1366–1401.
  * Li et al. (2011) Li, P., Ma, J., Gao, S., 2011\.  Actions in still web images: visualization, detection and retrieval, in: International Conference on Web-Age Information Management, Springer. pp. 302–313.
  * Li et al. (2020) Li, Y., Li, K., Wang, X., 2020\.  Recognizing actions in images by fusing multiple body structure cues.  Pattern Recognition 104, 107341\.
  * Liu et al. (2022) Liu, Z., Cheng, J., Liu, L., Ren, Z., Zhang, Q., Song, C., 2022.  Dual-stream cross-modality fusion transformer for rgb-d action recognition.  Knowledge-based systems 255, 0950–7051.
  * Liu et al. (2021) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B., 2021.  Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022.
  * Luo et al. (2019) Luo, H., Gu, Y., Liao, X., Lai, S., Jiang, W., 2019\.  Bag of tricks and a strong baseline for deep person re-identification, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–0.
  * Mohammadi et al. (2019) Mohammadi, S., Majelan, S.G., Shokouhi, S.B., 2019.  Ensembles of deep neural networks for action recognition in still images, in: International Conference on Computer and Knowledge Engineering, IEEE. pp. 315–318.
  * Radenović et al. (2018) Radenović, F., Tolias, G., Chum, O., 2018.  Fine-tuning cnn image retrieval with no human annotation.  IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 1655–1668.
  * Ramanathan et al. (2015) Ramanathan, V., Li, C., Deng, J., Han, W., Li, Z., Gu, K., Song, Y., Bengio, S., Rosenberg, C., Fei-Fei, L., 2015.  Learning semantic relationships for better action retrieval in images, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1100–1109.
  * Ren et al. (2015) Ren, S., He, K., Girshick, R., Sun, J., 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in Neural Information Processing Systems 28\.
  * Siméoni et al. (2019) Siméoni, O., Avrithis, Y., Chum, O., 2019.  Local features and visual words emerge in activations, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11651–11660.
  * Spolaôr et al. (2020) Spolaôr, N., Lee, H.D., Takaki, W.S.R., Ensina, L.A., Coy, C.S.R., Wu, F.C., 2020\.  A systematic review on content-based video retrieval.  Engineering Applications of Artificial Intelligence 90, 103557.
  * Tan et al. (2021) Tan, F., Yuan, J., Ordonez, V., 2021.  Instance-level image retrieval using reranking transformers, in: proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12105–12115.
  * Tang et al. (2013) Tang, J., Shao, L., Zhen, X., 2013.  Human action retrieval via efficient feature matching, in: IEEE International Conference on Advanced Video Signal Based Surveillance.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017\.  Attention is all you need.  International Conference on Neural Information Processing Systems , 6000–6010.
  * Veinidis et al. (2019) Veinidis, C., Danelakis, A., Pratikakis, I., Theoharis, T., 2019\.  Effective descriptors for human action retrieval from 3d mesh sequences.  International Journal of Image and Graphics 19, 1950018.
  * Wang and Zhang (2023) Wang, H., Zhang, Y., 2023.  Learning efficient representations for image-based patent retrieval, in: Chinese Conference on Pattern Recognition and Computer Vision, Springer. pp. 15–26.
  * Wang et al. (2018) Wang, Y., Zhou, L., Qiao, Y., 2018.  Temporal hallucinating for action recognition with few still images, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5314–5322.
  * Wray et al. (2019) Wray, M., Larlus, D., Csurka, G., Damen, D., 2019\.  Fine-grained action retrieval through multiple parts-of-speech embeddings, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 450–459.
  * Wu et al. (2022) Wu, H., Wang, M., Zhou, W., Hu, Y., Li, H., 2022\.  Learning token-based representation for image retrieval, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 2703–2711.
  * Wu and Yu (2020) Wu, W., Yu, J., 2020.  A part fusion model for action recognition in still images, in: International Conference on Neural Information Processing, Springer. pp. 101–112.
  * Wu and Yu (2021) Wu, W., Yu, J., 2021.  An improved deep relation network for action recognition in still images, in: International Conference on Acoustics, Speech and Signal Processing, IEEE. pp. 2450–2454.
  * Yao et al. (2011) Yao, B., Jiang, X., Khosla, A., Lin, A.L., Guibas, L., Fei-Fei, L., 2011\.  Human action recognition by learning bases of action attributes and parts, in: International Conference on Computer Vision, IEEE. pp. 1331–1338.
  * Ye et al. (2021) Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., Hoi, S.C., 2021.  Deep learning for person re-identification: A survey and outlook.  IEEE transactions on pattern analysis and machine intelligence 44, 2872–2893.
  * Zhang et al. (2016) Zhang, Y., Cheng, L., Wu, J., Cai, J., Do, M.N., Lu, J., 2016.  Action recognition in still images with minimum annotation efforts.  IEEE Transactions on Image Processing 25, 5479–5490.
  * Zhang et al. (2024) Zhang, Y., Qian, Q., Wang, H., Liu, C., Chen, W., Wan, F., 2024.  Graph convolution based efficient re-ranking for visual retrieval.  IEEE Transactions on Multimedia 26, 1089–1101.
  * Zhao et al. (2017) Zhao, Z., Ma, H., You, S., 2017.  Single image action recognition using semantic body part actions, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 3391–3399.