# SniffyArt: The Dataset of Smelling Persons

Mathias Zinnen  0000-0003-4366-5216 Pattern Recognition Lab, Friedrich-
Alexander-UniversitätErlangenGermany ,  Azhar Hussian  0009-0008-8125-7081
Pattern Recognition Lab, Friedrich-Alexander-UniversitätErlangenGermany ,
Hang Tran  0009-0004-0812-9316 Pattern Recognition Lab, Friedrich-Alexander-
UniversitätErlangenGermany ,  Prathmesh Madhu  0000-0003-2707-415X Pattern
Recognition Lab, Friedrich-Alexander-UniversitätErlangenGermany ,  Andreas
Maier  0000-0002-9550-5284 Pattern Recognition Lab, Friedrich-Alexander-
UniversitätErlangenGermany and  Vincent Christlein  0000-0003-0455-3799
Pattern Recognition Lab, Friedrich-Alexander-UniversitätErlangenGermany

###### Abstract.

Smell gestures play a crucial role in the investigation of past smells in the
visual arts yet their automated recognition poses significant challenges. This
paper introduces the SniffyArt dataset, consisting of 1941 individuals
represented in 441 historical artworks. Each person is annotated with a
tightly fitting bounding box, 17 pose keypoints, and a gesture label. By
integrating these annotations, the dataset enables the development of hybrid
classification approaches for smell gesture recognition. The dataset’s high-
quality human pose estimation keypoints are achieved through the merging of
five separate sets of keypoint annotations per person. The paper also presents
a baseline analysis, evaluating the performance of representative algorithms
for detection, keypoint estimation, and classification tasks, showcasing the
potential of combining keypoint estimation with smell gesture classification.
The SniffyArt dataset lays a solid foundation for future research and the
exploration of multi-task approaches leveraging pose keypoints and person
boxes to advance human gesture and olfactory dimension analysis in historical
artworks.

Figure 1. Samples from the dataset displaying various smell
gestures.22footnotemark: 2

Six samples from the SniffyArt dataset, displaying a variety of smell
gestures.

##  1\. Introduction

Smells play a crucial role in shaping human everyday experience, influencing
emotions, memories and behaviour. Despite their ubiquitousness, they rarely
cross the threshold of our consciousness. Recently, the significance of smell
has increasingly been acknowledged in the field of cultural heritage (Verbeek
and Van Campen, 2013; Bembibre and Strlič, 2017) and the humanities (Tullett,
2021; Leemans et al., 2022; Jenner, 2011). Specifically in digital heritage
and computational humanities, the role of smells is gaining more and more
prominence (van Erp et al., 2023; Lisena et al., 2022; Menini et al., 2023).
Tracing past smells and their societal roles can be achieved through the
identification of olfactory references in artworks and visual media. However,
the inherent invisibility of smells poses a significant challenge in this
endeavour. Recognising olfactory references requires the detection of proxies
such as smell-active objects, fragrant spaces, or olfactory iconography which
indirectly indicate the presence of smells (Zinnen, 2021). Among these
proxies, smell gestures, such as reactions to smell or smell-producing
actions, provide the most explicit gateway to the olfactory dimensions of a
painting. However, recognizing smell gestures is a particularly challenging
task, as they exhibit high intra-class variance, are difficult to precisely
localize, and their identification involves a higher degree of subjectivity.
As a first step towards recognizing smell gestures, we present the SniffyArt
dataset, annotated with person boxes, pose estimation keypoints, and smell
gesture labels. By combining these three types of annotations, we aim to
facilitate the development of novel gesture recognition methods that leverage
all three label types. Furthermore, we evaluate various baseline approaches
for person detection, keypoint estimation, and smell gesture classification
using this dataset. Our contributions are as follows:

  * •

We introduce the SniffyArt dataset, featuring artworks annotated with bounding
boxes, pose estimation keypoints, and smell gesture annotations for nearly
2000 persons.

  * •

We evaluate initial baseline methods for person detection, keypoint
estimation, and smell gesture recognition on the SniffyArt dataset.

Through this work, we hope to advance research in the domain of smell gesture
classification and pave the way for a deeper understanding of olfactory
dimensions in visual art and cultural heritage.

##  2\. Related Work

#### Computer Vision and the Humanities

Many computer vision tasks like object detection, human pose estimation, or
image segmentation have had their main research focus on real-world images.
The availability of large-scale photographic datasets like ImageNet
(Russakovsky et al., 2015), COCO (Lin et al., 2014), OpenImages (Kuznetsova et
al., 2020), or Objects365 (Shao et al., 2019) has enabled computer vision
methods to achieve impressive performance on natural images. Applying those
methods to digital humanities and cultural heritage can provide a valuable
addition to traditional methods of the humanities (Bell and Ommer, 2016,
2018). It enables humanities scholars to complement their analysis with a
data-driven perspective, thus broadening their view and enabling them to
perform distant viewing (Arnold and Tilton, 2019). Unfortunately, when
applying standard architectures on artwork images, we observe a significant
performance drop, which has been attributed to the domain shift problem (Cai
et al., 2015; Hall et al., 2015; Cetinic and She, 2022). This domain mismatch
can be tackled by applying domain adaptation techniques (Farahani et al.,
2021) to overcome the representational gap between artworks and real-world
images. Various researchers have proposed the application of style transfer
(Kadish et al., 2021; Lu et al., 2022; Madhu et al., 2022b), transfer learning
(Sabatelli et al., 2019; Gonthier et al., 2021; Zinnen et al., 2022a; Zhao et
al., 2022b), or the combination of multiple modalities (Radford et al., 2021;
Ali et al., 2022; Kiymet et al., 2022; Garcia and Vogiatzis, 2018; Gupta et
al., [n. d.]).

#### Person Detection

The task to detect persons can be considered a special case of the more
generic object detection task. Object detection algorithms are usually
categorised as one-stage, two-stage, and more recently transformer-based
approaches (Jiao et al., 2019). Two-stage algorithms propose candidate regions
of interest in the first step and refine and classify those regions in the
second step. The most prominent two-stage algorithms are representatives of
the R-CNN (Girshick et al., 2014) family. With various tweaks and refinements
(Girshick, 2015; Ren et al., 2015; Lin et al., 2017; Cai and Vasconcelos,
2018), R-CNN-based algorithms still provide competitive results today. Due to
its canonical role, we will apply the R-CNN based detector Faster R-CNN (Ren
et al., 2015) to generate baseline results for our experiments. One-stage
algorithms, on the other hand, merge the two stages and operate on a
predefined grid. On this grid, candidate objects are simultaneously predicted
and classified in a single step, thus achieving a higher inference speed. The
best-known examples of one-stage algorithms are You Only Look Once (YOLO)
(Redmon et al., 2016) architecture and descendants (Redmon and Farhadi, 2017,
2018; Jocher et al., 2020; Terven and Cordova-Esparza, 2023). In contrast to
these paradigms, our approach is based on transformer detection heads as
proposed by Carion _et al. missing_ (Carion et al., 2020) in their Detection
Transfomer (DETR) architecture. In DETR and derivatives (Zhu et al., 2020;
Zhang et al., 2022) a set of predicted candidate boxes are assigned to ground
truth boxes by solving a set assignment problem using the Hungarian Algorithm
(Kuhn, 1955). DETR-based algorithms, most notably DINO (Zhang et al., 2022),
set the current state of the art in object detection in natural images. DETR-
based algorithms, most notably DINO (Zhang et al., 2022), set the current
state of the art in object detection in natural images.

In the artistic domain, pioneering work by (Crowley and Zisserman, 2014, 2015,
2016) has opened the field of object recognition in the visual arts. Gonthier
_et al. missing_ (Gonthier et al., 2019) proposed a weakly supervised approach
to cope with the shortage of object-level labels in artworks and published the
IconArt dataset consisting of about 5000 instances within 10 iconography-
related classes (Gonthier, 2018). Going in the same direction, Madhu _et al.
missing_ (Madhu et al., 2022a) propose a one-shot algorithm that enables the
detection of unseen objects in artworks. Specifically for person detection,
Westlake _et al. missing_ (Westlake et al., 2016) provide the PeopleArt
dataset and evaluate a Fast-RCNN on the dataset. In the ODOR challenge (Zinnen
et al., 2022b), participants were given the task of detecting a set of 87
smell-related objects depicted in historical artworks. The recent introduction
of the DeART dataset (Reshetnikov et al., 2022) promises to advance the field
further by providing more than 15,000 artworks annotated with object-level
annotations across 70 categories.

#### Human Pose Estimation (HPE)

The estimation of body poses is achieved via the regression of a set of
keypoints corresponding to body joints that define a person’s pose. In
practice, many modern pose estimation algorithms do not directly regress the
exact keypoints but operate on heatmaps indicating the probability
distribution for keypoint existence in a region. The set of keypoints defining
the body pose can be defined in multiple ways. In this work, we consider the
definition of body joints defined by Lin _et al. missing_ (Lin et al., 2014).
Pose estimation algorithms can be roughly grouped into bottom-up or top-down
approaches (Zheng et al., 2020). In bottom-up algorithms (Cao et al., 2017;
Kreiss et al., 2019; Cheng et al., 2020; Geng et al., 2021), keypoints are
detected first and assigned to specific persons afterwards whereas top-down
algorithms (Cai et al., 2020; Cai and Vasconcelos, 2018; Sun et al., 2019; Xu
et al., 2022; Xiao et al., 2018) require a person-detection stage before
estimating the pose keypoints. Recent state-of-the-art pose estimation
networks combine a two-stage pipeline with transformer-based keypoint
regression heads. Zhang _et al. missing_ (Zhang et al., 2021) demonstrated
that an additional skeleton refinement step can further increase the
estimation accuracy.

Applications in the artistic domain suffer from a lack of large-scale
annotated datasets, which is even worse than in the case of object detection.
Springstein _et al. missing_ (Springstein et al., 2022) tackle this lack of
annotated data by training on stylised versions of the COCO dataset and
applying the semi-supervised soft-teacher approach (Xu et al., 2021). A recent
application of HPE for artwork analysis has been presented by Zhao _et al.
missing_ (Zhao et al., 2022a) who combine body segmentation, HPE, and
hierarchical clustering to analyze body poses in a dataset of c. 100k
artworks.

Apart from the yet unpublished PoPArt (Schneider and Vollmer, 2023) dataset,
the proposed SniffyArt dataset constitutes the first artwork dataset with
keypoint-level annotations.

##  3\. SniffyArt Dataset

###  3.1. Data Collection

The data was collected and annotated in three phases: preselection, person
annotation, and keypoint annotation.

In the preselection phase, we automatically annotated a large set of candidate
artworks from various digital museum collections with 139 smell-active
objects. From this annotated set of artworks, we selected about 2000 images
containing depictions of smell gestures. The object annotations served as cues
to facilitate the search; e. g., by filtering for images containing pipes when
looking for “smoking” gestures. Filtering and tagging in this phase was
achieved using the dataset management tool FiftyOne (Moore and Corso, 2020).

In the person annotation phase, we annotated each person with tightly fitting
bounding boxes and (possibly multiple) gesture labels. Depending on the
artwork style and reproduction quality, it can sometimes be difficult to
distinguish between background and depicted persons. To handle these edge
cases, we defined multiple requirements for image regions to be considered
persons:  (1) The head of the person must be visible.  (2) Apart from the
head, at least two additional pose keypoints must be visible.  (3) It must be
possible to assign this minimal set of keypoints to the person in question (in
contrast to different, overlapping persons).  (4) It must be possible to
clearly distinguish the persons from the background.  Figure 2 shows an
example image where some of the depicted persons meet the criteria and are
annotated and some others are not annotated.

These criteria, especially the third one, can be quite subjective. There will
always be instances where one annotator perceives a person as clearly
distinguishable from the background, while another may not. The person on the
right corner of Fig. 2 provides an example of such an edge case. Given the
diverse stylistic variations and artistic abstractions, we believe that
encountering such edge cases is inevitable. We aim to address this issue by
explicitly outlining the (unavoidably somewhat subjective) criteria in the
annotation guidelines, yet we acknowledge that avoiding these ambiguities
completely is not achievable.

Figure 2. Example from the person annotation phase. While the four persons in
the foreground were annotated with bounding boxes, the three persons in the
background are hardly visible and were not annotated. Image credits: Company
drinking and smoking in an interior. David Rijckaert (III). 1627 – 1661. Oil
on canvas. RKD – Netherlands Institute for Art History, RKDimages (301815).
Public Domain.

Finally, in the keypoint annotation phase, we applied the crowd-working
platform AMT to annotate the cropped person boxes obtained in the second step
with 17 keypoints. Those points define the body pose as exemplified in Fig. 3.
To ensure annotation quality, we gathered five annotations for each of the
person boxes and merged them by averaging over the set of defined keypoint
annotations. The annotation merging process can be defined as follows:

Let $\mathbf{k}^{n}_{i}=(x^{n}_{i},y^{n}_{i},v^{n}_{i})$ denote the i-th
keypoint annotated by the n-th annotator, where
$(x^{n}_{i},y^{n}_{i},v^{n}_{i})$ define the respective keypoint coordinates
and visibility. We encode the absence of the i-th keypoint annotation for the
n-th annotator as $v^{n}_{i}=0$ and define the set of present annotations
indices $N=\\{n_{i}|v^{n}_{i}\neq 0\\}$. The merged keypoint coordinates
$\mathbf{k}^{*}_{i}$ are then given by:

(1) $\mathbf{k}^{*}_{i}=(x^{*}_{i},y^{*}_{i})$

with

(2) $(x^{*}_{i},y^{*}_{i})=\Big{(}\frac{1}{\lvert N\rvert}\sum_{n\in
N}x^{n}_{i},\frac{1}{\lvert N\rvert}\sum_{n\in N}y^{n}_{i}\Big{)}\;.$

In simpler terms, we construct the centroid of all annotated coordinates for
each of the keypoints defining the body pose. Figure 4 illustrates how using
this process multiple imperfect annotations can be merged into an accurate
pose skeleton.

Figure 3. Illustration of Pose Estimation Keypoints. Image Credits: Detail
from Ein ruhiges Stündchen. Ludwig Noster. 1895. Oil on Canvas. Alte
Nationalgalerie, Staatliche Museen zu Berlin / Andreas Kilger. Public Domain.
Figure 4. Example of an increase in annotation quality by merging multiple
flawed annotations. Image credits: Detail from Die Auferweckung des Lazarus.
Bonifazio Veronese. ca. 1487 – 1553. Deutsche Fotothek / Walter Möbius.

###  3.2. Dataset Statistics

The SniffyArt dataset consists of 1941 persons annotated with tightly fitting
bounding boxes, 17 pose estimation keypoints, and gesture labels. The
annotations are spread over 441 historical artworks with diverse styles. Note
that the relatively low number of artworks is due to the difficulty of finding
smell gestures in digital collections and we plan to extend the dataset in the
future. To the best of our knowledge, the current state of the SniffyArt
dataset already constitutes the second-largest keypoint-level dataset in arts
after the yet unpublished PoPArt (Schneider and Vollmer, 2023). We provide
predefined train, validation, and test splits, containing 307, 83, and 89
images, respectively (cf. Table 1) to facilitate training and enable a
consistent baseline evaluation. The splits were generated image-wise and based
on the gesture labels, i. e., person crops from the same image are always
assigned to the same split, and the splits are used unmodified for all tasks.

Table 1. Overview of image, box, and gesture distribution for the dataset splits. Background class (i. e., person not performing any smell gesture) is not listed. Note that the splits were used unchanged for detection, keypoint estimation, and gesture classification and person boxes from one image were always assigned to the same split.  | Train | Validation | Test
---|---|---|---
# Images |  0307 ($64.1\text{\,}\mathrm{\char 37\relax}$) |  083 ($17.3\text{\,}\mathrm{\char 37\relax}$) |  089 ($18.5\text{\,}\mathrm{\char 37\relax}$)
# Persons | 1245 ($64.1\text{\,}\mathrm{\char 37\relax}$) | 332 ($17.1\text{\,}\mathrm{\char 37\relax}$) | 364 ($18.8\text{\,}\mathrm{\char 37\relax}$)
# Gestures |  0434 ($60.4\text{\,}\mathrm{\char 37\relax}$) | 127 ($17.7\text{\,}\mathrm{\char 37\relax}$) | 130 ($18.1\text{\,}\mathrm{\char 37\relax}$)

Due to our choice to annotate all persons meeting the requirements defined in
Section 3.1 irrespective of whether they perform a smell gesture, we observe a
large class imbalance with background persons (i. e., performing no smell
gesture) being vastly overrepresented (cf. Fig. 6(b). Figure 5 shows an
example from the dataset where only three of the twelve annotated persons
perform a smell gesture while the remaining nine are labelled as background
persons.

Figure 5. Example of a large group of persons where only three out of twelve
depicted persons perform a smell gesture. Image credits: Carousing peasant
company in an inn. Joachim van den Heuvel. Oil on panel. RKD – Netherlands
Institute for Art History, RKDimages (284006). Public Domain.

While the resulting imbalance negatively affects the performance of gesture
classification, it was necessary to enable complete annotations for person and
keypoint detection algorithms. However, without considering the background
class, the class imbalance is reduced considerably as illustrated in Fig.
6(a).

(a) Excluding background class.smokingdrinkingholding the
nosecookingsniffing$0$$50$$100$$150$Number of InstancesTrainValidTest

(b) Including background class and multi-labels. Note that multi-class labels (smoking, drinking) are not mutually exclusive with their single-class constituents, i. e., a person annotated as smoking and drinking is counted three times in this distribution. Gesture | Train | Valid | Test
---|---|---|---
Background | 795 | 198 | 230
Smoking | 160 |  043 |  052
Drinking | 123 |  045 |  045
Holding the Nose | 101 |  021 |  024
Cooking |  032 |  016 |  010
Sniffing |  031 |  013 |  012
Smoking, Drinking |  013 |  011 |  010

Figure 6. Distribution of gesture class labels in train and test set.

We allowed persons to be annotated with multiple gestures, effectively
rendering the classification problem as multi-label classification. In
practice, we encountered more than thirty examples of persons smoking and
drinking at the same time (cf. Fig. 6(b)) but no other combinations. However,
for future extensions of the dataset, different label combinations are to be
expected.

The distribution of the number of depicted persons per image (cf. Fig. 7)
reflects the remarks about the high number of background persons. While
$53\text{\,}\mathrm{\char 37\relax}$ of the images contain only one or two
persons, a considerable amount of images depict 10 or more persons.

12345678910$>$10$0$$50$$100$$150$$200$Number of PersonsNumber of Images Figure
7. Distribution of annotated persons per image.

Regarding the distribution of annotated keypoints per person we observe that
the majority of person boxes $46\text{\,}\mathrm{\char 37\relax}$ have
annotations for each of the 17 possible keypoints, while only
$6\text{\,}\mathrm{\char 37\relax}$ have annotations for less than 10
keypoints (cf. Fig. 8).

171615141312111098765$0$$200$$400$$600$$800$$1{,}000$Number of KeypointsNumber
of Instances Figure 8. Distribution of annotated keypoints per person.

###  3.3. Distribution Format

The annotations are provided in a JSON file following the COCO standard for
object detection and keypoint annotations. Extending the default COCO format,
we enrich each entry in the annotations array of the COCO JSON with a gestures
key that contains a (possibly empty) list of smell gestures the annotated
person is performing. To facilitate label transformation for single-label
classification, we add a derived gesture key, which contains the list of
gesture labels as a single, comma-separated string. Additionally, we provide a
CSV file with image-level metadata, which includes content related-fields such
as Iconclass codes or image descriptions, as well as formal annotations, such
as artist, license or creation year. For license compliance, we do not publish
the images directly. Instead, we provide links to their source collections in
the metadata file and a Python script to download the artwork images. The
dataset is available for download on
Zenodo.333https://doi.org/10.5281/zenodo.8273616

##  4\. Baseline Experiments

To showcase the applicability of our dataset and provide initial baselines, we
conduct experiments for person detection, human pose estimation, and gesture
classification.

###  4.1. Detection

Detecting depicted persons is a prerequisite for both, top-down approaches in
keypoint estimation, and person-level gesture classification. Here, we
evaluate three representative object detection configurations:  (1) Faster
R-CNN (Ren et al., 2015) with a ResNet-50 (He et al., 2016) serves as a
default baseline as it is still the most widely used object detection system.
(2) To assess the effect of scaling up the backbone, we evaluate a Faster
R-CNN with the larger ResNet-101 (He et al., 2016) backbone.  (3) To
understand the effects of more modern detection heads, we evaluate the state-
of-the-art transformer-based DINO (Zhang et al., 2022) architecture with a
ResNet-50 backbone.

All models are trained for 50 epochs using the MMDetection (Chen et al., 2019)
framework, applying the respective default training parameters. Please refer
to Table 2 for a detailed list of hyperparameters.

Table 2. Fine-tuning settings of detection and pose estimation experiments. Parameter |  Faster R-CNN RN-50/RN-101 |  DINO RN-50 |  Pose HRNet HRNet-W32 |  DEKR HRNet-W32
---|---|---|---|---
task | detection | detection | pose estimation | pose estimation
pre-training dataset | ImageNet-1k | ImageNet-1k | ImageNet-1k | ImageNet-1k
optimizer | SGD | AdamW | Adam | Adam
base lr | 0.02 | 0.0001 | 0.001 | 0.0005
weight decay | 0.0001 | 0.0001 | - | -
optim. momentum | 0.9 | - | - | -
batch size | 2 | 2 | 10 | 64
num_gpus | 2 | 2 | 2 | 2
training epochs. | 50 | 50 | 210 | 210
warmup iterations | 500 | - | 500 | 500
warmup scheduler | linear | - | linear | linear
lr scheduler | step (30,40,48) | step (11) | step (170, 200) | step (170, 200)
lr gamma | 0.1 | 0.1 | 0.1 | 0.1

In Table 3, we report the model performances, following the standard COCO
evaluation protocol444https://cocodataset.org/#detection-eval for object
detection. For each configuration, we fine-tune five models on the SniffyArt
training set and report their respective mean and standard deviation of test
set performance.

Table 3. COCO detection performance of representative detection algorithms fine-tuned on SniffyArt-train and evaluated on SniffyArt-test, averaged over five runs. The standard deviation is reported in brackets. Model | Backbone | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{M}$ | $AP_{L}$ | $AR$
---|---|---|---|---|---|---|---
Faster R-CNN (Ren et al., 2015) | ResNet-50 (He et al., 2016) | 34.2($\pm$0.02) | 75.5($\pm$0.10) | 24.6($\pm$0.17) | 22.6($\pm$0.24) | 35.9($\pm$0.05) | 43.0($\pm$0.04)
Faster R-CNN (Ren et al., 2015) | ResNet-101 (He et al., 2016) | 35.7($\pm$0.08) | 76.0($\pm$0.13) | 28.5($\pm$0.21) | 24.3($\pm$0.10) | 37.4($\pm$0.09) | 44.0($\pm$0.08)
DINO (Zhang et al., 2022) | ResNet-50 (He et al., 2016) | 28.4($\pm$0.09) | 53.7($\pm$0.31) | 27.0($\pm$0.09) | 15.4($\pm$0.16) | 30.1($\pm$0.15) | 61.7($\pm$0.04)

Despite the relatively small size of the dataset, we observe an increase of
$1.5\text{\,}\mathrm{\char 37\relax}$ mAP in detection accuracy when scaling
up the feature extraction backbone. While the configuration equipped with a
ResNet-101 outperforms its ResNet-50 counterpart in the stricter $AP_{75}$
metric considerably ($3.9\text{\,}\mathrm{\char 37\relax}$), the difference in
$AP_{50}$ amounts to only $0.5\text{\,}\mathrm{\char 37\relax}$.

This suggests that the larger backbone mostly increases the model’s capacity
to localize persons very precisely. Surprisingly, the modern DINO architecture
performs considerably worse ($-5.8\text{\,}\mathrm{\char 37\relax}$) than its
Faster R-CNN counterpart. Noticeable is the high recall of the DINO models,
which is $18\text{\,}\mathrm{\char 37\relax}$ higher than that of the Faster
R-CNN counterpart with the same backbone. We hypothesize that the DINO models
generate too many box predictions for the images in our datasets and that the
performance can significantly be increased by filtering out weak predictions
or reducing the number of object queries.

We conclude that standard architectures with relatively weak backbones can
already produce sufficient person predictions based on the size of the
SniffyArt training set. If required, more accurate boxes can likely be
obtained by pre-training using external data (e. g., DeArt (Reshetnikov et
al., 2022), COCO (Lin et al., 2014), or (Westlake et al., 2016)) or scaling up
model capacity.

###  4.2. Pose Estimation

To understand how different human pose estimation paradigms work for our
dataset, we analyse one top-down method (DEKR) and one bottom-up method (Pose
HRNet). In the top-down scenario, the pose estimation model gets the box
predictions from our strongest detection model as an auxiliary input at
validation and test time. We use the MMPose (Contributors, 2020) framework for
model training, initialize the backbones with ImageNet-1k weights and train
for 210 epochs using the default hyperparameters. For more details on training
settings, please refer to the two rightmost columns of Table 2. Again, we
fine-tune five models on the SniffyArt training set and report the mean and
standard deviation on the SniffyArt test set in Table 4. As the evaluation
metric, we apply COCO’s object keypoint similarity
(OKS)555https://cocodataset.org/#keypoint-eval.

Table 4. Performance of representative human pose estimation (HPE) algorithms fine-tuned on SniffyArt-train and evaluated on SniffyArt-test, averaged over five runs. The standard deviation is reported in brackets. Model | Backbone | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{M}$ | $AP_{L}$ | $AR$
---|---|---|---|---|---|---|---
Pose HRNet (Sun et al., 2019) | HRNet-W32 (Wang et al., 2020) | 53.3($\pm$0.05) | 79.2($\pm$0.07) | 58.3($\pm$0.17) | 30.7($\pm$0.13) | 56.3($\pm$0.07) | 58.8($\pm$0.05)
DEKR (Geng et al., 2021) | HRNet-W32 (Wang et al., 2020) | 36.7($\pm$0.10) | 70.0($\pm$0.08) | 35.6($\pm$0.19) | 13.7($\pm$0.07) | 43.4($\pm$0.07) | 45.8($\pm$0.06)

The results show that while keeping the backbone invariant, the bottom-up
pipeline DEKR is significantly outperformed by the top-down approach Pose
HRNet in all metrics.

###  4.3. Gesture Classification

We analyze the performance of various representative networks for the
classification of smell gestures. Experiments are conducted per-person,
meaning that each person is cropped and classified separately. To simplify our
models, we transform the multi-label problem into a single-label
classification by introducing new labels representing combinations of single
labels. Effectively, this required the introduction of only a single new
class, since drinking and smoking is the only combination of smell gestures
present in the dataset. We apply cross-entropy loss and handle the class
imbalance by weighing it with normalised inverse class frequencies.
Experiments are conducted using the MMPretrain (Contributors, 2023) framework
keeping the default parameters for the classification algorithms. As for
detection and keypoint estimation, we fine-tune five models and report the
average top 1 accuracy, precision, and $F_{1}$ scores together with the
standard deviations in Table 5. Additionally, we report the metrics of a naive
classifier that always predicts the majority class.

Table 5. Classification results on the SniffyArt test set for three classification networks pre-trained on ImageNet-1k. _Majority Class_ denotes the trivial solution of always predicting the most frequent class (i. e., _no gesture_). We report the mean over five experiments per configuration with standard deviation in brackets. Model | Acc./top1 | Prec. | $F_{1}$
---|---|---|---
Majority Class | 9.1 | 14.3 | 11.1
ResNet-50 (He et al., 2016) | 31.8($\pm$3.7) | 31.8($\pm$1.4) | 31.1($\pm$2.0)
ResNet-101 (He et al., 2016) | 36.7($\pm$3.8) | 34.1($\pm$1.2) | 34.2($\pm$1.8)
HRNet-W32 (Wang et al., 2020) | 15.7($\pm$1.5) | 19.8($\pm$2.1) | 17.3($\pm$1.8)

The evaluation highlights how challenging the classification of odor gestures
on historical artworks is. While we do see an increase in the metrics when
increasing the number of model parameters, the overall $F_{1}$ score stays
quite low with $34\text{\,}\mathrm{\char 37\relax}$. Surprisingly, the
performance of the modern HRNet falls significantly behind that of the two
ResNet models. We note that this performance gap is consistent over the
evaluations of all trained models which is reflected in the relatively low
standard deviations in all metrics.

To assess how well feature representations learned from person detection and
keypoint estimation generalise to the gesture classification task, we
initialize the networks with weights obtained from the feature extraction
backbones of the person detection and keypoint estimation tasks discussed
above. With similar experimental settings, we train five models for each
configuration and report the results in Table 6.

Table 6. Classification performance when initializing the feature extraction backends with weights obtained by person detection (for ResNet-50 & ResNet-101), or keypoint estimation (for HRNet-W32) on the SniffyArt dataset. Model | Acc./top1 | Prec. | $F_{1}$
---|---|---|---
ResNet-50 (He et al., 2016) | 12.6($\pm$1.2) | 16.5($\pm$0.7) | 14.1($\pm$0.9)
ResNet-101 (He et al., 2016) | 15.0($\pm$2.0) | 18.4($\pm$2.2) | 16.1($\pm$2.0)
HRNet-W32 (Wang et al., 2020) | 37.7($\pm$4.1) | 33.3($\pm$2.3) | 33.7($\pm$2.8)

When comparing the ResNets pre-trained for person detection with their
ImageNet-pretrained counterparts, we observe a significant performance drop,
with $F_{1}$ scores decreasing by over half. This suggests that feature
representations learned from person detection are not suited for smell gesture
classification. The HRNet models, on the other hand, seem to benefit greatly
from initializing them with weights obtained by keypoint estimation. We find
that the weak performance metrics of the ImageNet-pretrained models are more
than doubled when keypoint estimation pre-training is used. This finding
demonstrates the large potential of combining the representational space from
the two tasks of gesture classification and keypoint estimation.

##  5\. Limitations

#### Dataset Size

With 400 images, the number of annotated artworks is relatively low. This is
due to the difficulties in finding a sufficient number of smell gestures in
artworks which can partly be explained by a lack of olfaction-related metadata
in digital museum collections (Ehrich et al., 2022). We plan to extend the
dataset in the future, alleviating this issue by applying semi-automated
approaches based on the set of existing images.

#### Annotation Quality

During the test runs of the keypoint annotation phase, we observed that
annotators often incorrectly left out occluded keypoints, even if they were
inside of the image boundaries. To alleviate this problem, we incorporated
pose keypoints, even if they were annotated by only one of the five
annotators. However, this approach may lead to incorrect annotations if one
annotator misunderstands the task or provided incorrect keypoints
deliberately. To prevent such cases, a more advanced outlier detection
algorithm could be implemented to filter out annotations from obstructive
annotators.

#### Experimental Evidence

To confirm and strengthen the hypothesis that leveraging pose estimation
keypoints is beneficial for smell gesture classification, more experiments
would be needed. A deeper analysis is out of the scope of this paper but it
would certainly be a valuable line of future research to investigate the
combination potential further.

#### Image Properties

The degree of artistic abstraction and low quality of some of the images might
set an upper bound to algorithmic gesture recognition capabilities. While
extensions with regard to dataset size and the incorporation of different
digital collections might alleviate this issue to some degree, it is a general
problem of computer vision algorithms in the artistic domain. From another
angle, it might as well be viewed as a strength as it enforces algorithm
robustness towards diverse stylistic representations.

##  6\. Conclusion

We introduced the SniffyArt dataset consisting of 1941 persons on 441
historical artworks, annotated with tightly fitting bounding boxes, 17 pose
estimation keypoints and gesture labels. By combining detection, pose
estimation, and gesture labels, we pave the way for innovative classification
approaches connecting these annotations. Our dataset features high-quality
human pose estimation keypoints, which are achieved through merging five
distinct sets of keypoint annotations per person. In addition, we have
conducted a comprehensive baseline analysis to evaluate the performance of
various representative algorithms for detection, keypoint estimation, and
classification tasks. Preliminary experiments demonstrate that there is a
large potential in combining keypoint estimation and smell gesture
classification tasks. Looking ahead, we plan to extend the dataset and address
the relatively low number of samples. Given the scarcity of metadata related
to olfactory dimensions in digital museum collections, we intent to apply
semi-automated approaches to identify candidate images containing smell
gestures. Even in its current state, the SniffyArt dataset provides a solid
foundation for the development of novel algorithms focused on smell gesture
classification. We are particularly interested in exploring multi-task
approaches that leverage both pose keypoints and person boxes. As we move
forward, we envision that this dataset will stimulate significant advancements
in the field, ultimately enhancing our understanding of human gestures and
olfactory dimensions in historical artworks.

##  7\. Acknowledgements

This paper has received funding from the Odeuropa EU H2020 project under grant
agreement No. 101004469. We gratefully acknowledge the donation of the NVIDIA
corporation of two Quadro RTX 8000 that we used for the experiments.

## References

  * (1)
  * Ali et al. (2022) Hürriyetoğlu Ali, Teresa Paccosi, Stefano Menini, Zinnen Mathias, Lisena Pasquale, Akdemir Kiymet, Troncy Raphaël, and Marieke van Erp. 2022.  MUSTI-Multimodal Understanding of Smells in Texts and Images at MediaEval 2022. In _Proceedings of MediaEval 2022 CEUR Workshop_.
  * Arnold and Tilton (2019) Taylor Arnold and Lauren Tilton. 2019.  Distant viewing: analyzing large visual corpora.  _Digital Scholarship in the Humanities_ 34, Supplement_1 (2019), i3–i16.
  * Bell and Ommer (2016) Peter Bell and Björn Ommer. 2016.  Visuelle Erschliessung (Computer Vision als Arbeits-und Vermittlungstool).  _Elektronische Medien & Kunst, Kultur und Historie_ 23 (2016), 67–73.
  * Bell and Ommer (2018) Peter Bell and Björn Ommer. 2018.  Computer Vision und Kunstgeschichte–Dialog zweier Bildwissenschaften.  _Computing Art Reader: Einführung in die digitale Kunstgeschichte_ 1 (2018), 61–75.
  * Bembibre and Strlič (2017) Cecilia Bembibre and Matija Strlič. 2017.  Smell of heritage: a framework for the identification, analysis and archival of historic odours.  _Heritage Science_ 5, 1 (2017), 1–11.
  * Cai et al. (2015) Hongping Cai, Qi Wu, Tadeo Corradi, and Peter Hall. 2015\.  The cross-depiction problem: Computer vision algorithms for recognising objects in artwork and in photographs.  _arXiv preprint arXiv:1505.00110_ (2015).
  * Cai et al. (2020) Yuanhao Cai, Zhicheng Wang, Zhengxiong Luo, Binyi Yin, Angang Du, Haoqian Wang, Xiangyu Zhang, Xinyu Zhou, Erjin Zhou, and Jian Sun. 2020\.  Learning delicate local representations for multi-person pose estimation. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_. Springer, 455–472.
  * Cai and Vasconcelos (2018) Zhaowei Cai and Nuno Vasconcelos. 2018.  Cascade r-cnn: Delving into high quality object detection. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6154–6162.
  * Cao et al. (2017) Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017\.  Realtime multi-person 2d pose estimation using part affinity fields. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7291–7299.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-end object detection with transformers. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_. Springer, 213–229.
  * Cetinic and She (2022) Eva Cetinic and James She. 2022.  Understanding and creating art with AI: review and outlook.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ 18, 2 (2022), 1–22.
  * Chen et al. (2019) Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. 2019\.  MMDetection: Open MMLab Detection Toolbox and Benchmark.  _arXiv preprint arXiv:1906.07155_ (2019).
  * Cheng et al. (2020) Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang. 2020\.  Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 5386–5395.
  * Contributors (2020) MMPose Contributors. 2020\.  OpenMMLab Pose Estimation Toolbox and Benchmark.  https://github.com/open-mmlab/mmpose.
  * Contributors (2023) MMPreTrain Contributors. 2023\.  OpenMMLab’s Pre-training Toolbox and Benchmark.  https://github.com/open-mmlab/mmpretrain.
  * Crowley and Zisserman (2014) Elliot Crowley and Andrew Zisserman. 2014.  The State of the Art: Object Retrieval in Paintings using Discriminative Regions. In _Proceedings of the British Machine Vision Conference. BMVA Press_.
  * Crowley and Zisserman (2015) Elliot J Crowley and Andrew Zisserman. 2015.  In search of art. In _Computer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6-7 and 12, 2014, Proceedings, Part I 13_. Springer, 54–70.
  * Crowley and Zisserman (2016) Elliot J Crowley and Andrew Zisserman. 2016.  The art of detection. In _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part I 14_. Springer, 721–737.
  * Ehrich et al. (2022) Sofia Collette Ehrich, Caro Verbeek, Mathias Zinnen, Lizzie Marx, Cecilia Bembibre, and Inger Leemans. 2022.  Nose-First. Towards an Olfactory Gaze for Digital Art History. In _2021 Workshops and Tutorials-Language Data and Knowledge, LDK 2021_. CEUR-WS. org, 1–17.
  * Farahani et al. (2021) Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. 2021.  A brief review of domain adaptation.  _Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020_ (2021), 877–894.
  * Garcia and Vogiatzis (2018) Noa Garcia and George Vogiatzis. 2018.  How to read paintings: semantic art understanding with multi-modal retrieval. In _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_. 0–0.
  * Geng et al. (2021) Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang. 2021.  Bottom-up human pose estimation via disentangled keypoint regression. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 14676–14686.
  * Girshick (2015) Ross Girshick. 2015\.  Fast r-cnn. In _Proceedings of the IEEE international conference on computer vision_. 1440–1448.
  * Girshick et al. (2014) Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014.  Rich feature hierarchies for accurate object detection and semantic segmentation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 580–587.
  * Gonthier (2018) Nicolas Gonthier. 2018\.  IconArt Dataset.  https://doi.org/10.5281/zenodo.4737435 https://doi.org/10.5281/zenodo.4737435.
  * Gonthier et al. (2021) Nicolas Gonthier, Yann Gousseau, and Saïd Ladjal. 2021\.  An analysis of the transfer learning of convolutional neural networks for artistic images. In _Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10–15, 2021, Proceedings, Part III_. Springer, 546–561.
  * Gonthier et al. (2019) Nicolas Gonthier, Yann Gousseau, Said Ladjal, and Olivier Bonfait. 2019.  Weakly Supervised Object Detection in Artworks. In _Computer Vision – ECCV 2018 Workshops_ , Laura Leal-Taixé and Stefan Roth (Eds.). Springer International Publishing, Cham, 692–709.
  * Gupta et al. ([n. d.]) Jahnvi Gupta, Prathmesh Madhu, Ronak Kosti, Peter Bell, Andreas Maier, and Vincent Christlein. [n. d.].  Towards image caption generation for art historical data.
  * Hall et al. (2015) Peter Hall, Hongping Cai, Qi Wu, and Tadeo Corradi. 2015\.  Cross-depiction problem: Recognition and synthesis of photographs and artwork.  _Computational Visual Media_ 1 (2015), 91–103.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Jenner (2011) Mark SR Jenner. 2011\.  Follow your nose? Smell, smelling, and their histories.  _The American Historical Review_ 116, 2 (2011), 335–351.
  * Jiao et al. (2019) Licheng Jiao, Fan Zhang, Fang Liu, Shuyuan Yang, Lingling Li, Zhixi Feng, and Rong Qu. 2019.  A survey of deep learning-based object detection.  _IEEE access_ 7 (2019), 128837–128868.
  * Jocher et al. (2020) Glenn Jocher, Alex Stoken, Jirka Borovec, Liu Changyu, Adam Hogan, Laurentiu Diaconu, Jake Poznanski, Lijun Yu, Prashant Rai, Russ Ferriday, et al. 2020\.  ultralytics/yolov5: v3. 0.  _Zenodo_ (2020).
  * Kadish et al. (2021) David Kadish, Sebastian Risi, and Anders Sundnes Løvlie. 2021\.  Improving object detection in art images using only style transfer. In _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–8.
  * Kiymet et al. (2022) Akdemir Kiymet, Hürriyetoğlu Ali, Troncy Raphaël, Teresa Paccosi, Stefano Menini, Zinnen Mathias, and Christlein Vincent. 2022.  Multimodal and Multilingual Understanding of Smells using VilBERT and mUNITER. In _Proceedings of MediaEval 2022 CEUR Workshop_.
  * Kreiss et al. (2019) Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. 2019\.  Pifpaf: Composite fields for human pose estimation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 11977–11986.
  * Kuhn (1955) Harold W Kuhn. 1955\.  The Hungarian method for the assignment problem.  _Naval research logistics quarterly_ 2, 1-2 (1955), 83–97.
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020\.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  _International Journal of Computer Vision_ 128, 7 (2020), 1956–1981.
  * Leemans et al. (2022) Inger Leemans, William Tullett, Cecilia Bembibre, and Lizzie Marx. 2022.  Whiffstory: Using Multidisciplinary Methods to Represent the Olfactory Past.  _The American Historical Review_ 127, 2 (2022), 849–879.
  * Lin et al. (2017) Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017\.  Feature pyramid networks for object detection. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2117–2125.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 740–755.
  * Lisena et al. (2022) Pasquale Lisena, Daniel Schwabe, Marieke van Erp, Raphaël Troncy, William Tullett, Inger Leemans, Lizzie Marx, and Sofia Colette Ehrich. 2022\.  Capturing the Semantics of Smell: The Odeuropa Data Model for Olfactory Heritage Information. In _The Semantic Web: 19th International Conference, ESWC 2022, Hersonissos, Crete, Greece, May 29–June 2, 2022, Proceedings_. Springer, 387–405.
  * Lu et al. (2022) Yue Lu, Chao Guo, Xingyuan Dai, and Fei-Yue Wang. 2022\.  Data-efficient image captioning of fine art paintings via virtual-real semantic alignment training.  _Neurocomputing_ 490 (2022), 163–180.
  * Madhu et al. (2022a) Prathmesh Madhu, Anna Meyer, Mathias Zinnen, Lara Mührenberg, Dirk Suckow, Torsten Bendschus, Corinna Reinhardt, Peter Bell, Ute Verstegen, Ronak Kosti, et al. 2022a.  One-shot object detection in heterogeneous artwork datasets. In _2022 Eleventh International Conference on Image Processing Theory, Tools and Applications (IPTA)_. IEEE, 1–6.
  * Madhu et al. (2022b) Prathmesh Madhu, Angel Villar-Corrales, Ronak Kosti, Torsten Bendschus, Corinna Reinhardt, Peter Bell, Andreas Maier, and Vincent Christlein. 2022b.  Enhancing human pose estimation in ancient vase paintings via perceptually-grounded style transfer learning.  _ACM Journal on Computing and Cultural Heritage_ 16, 1 (2022), 1–17.
  * Menini et al. (2023) Stefano Menini, Teresa Paccosi, Serra Sinem Tekiroğlu, and Sara Tonelli. 2023.  Scent Mining: Extracting Olfactory Events, Smell Sources and Qualities. In _Proceedings of the 7th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature_. Association for Computational Linguistics, Dubrovnik, Croatia, 135–140.  https://aclanthology.org/2023.latechclfl-1.15
  * Moore and Corso (2020) B. E. Moore and J. J. Corso. 2020.  FiftyOne.  _GitHub. Note: https://github.com/voxel51/fiftyone_ (2020).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Redmon et al. (2016) Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016.  You only look once: Unified, real-time object detection. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 779–788.
  * Redmon and Farhadi (2017) Joseph Redmon and Ali Farhadi. 2017.  YOLO9000: better, faster, stronger. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7263–7271.
  * Redmon and Farhadi (2018) Joseph Redmon and Ali Farhadi. 2018.  Yolov3: An incremental improvement.  _arXiv preprint arXiv:1804.02767_ (2018).
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015\.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ 28 (2015).
  * Reshetnikov et al. (2022) Artem Reshetnikov, Maria-Cristina Marinescu, and Joaquim More Lopez. 2022.  DEArt: Dataset of European Art.  _arXiv preprint arXiv:2211.01226_ (2022).
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015\.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ 115 (2015), 211–252.
  * Sabatelli et al. (2019) Matthia Sabatelli, Mike Kestemont, Walter Daelemans, and Pierre Geurts. 2019.  Deep Transfer Learning for Art Classification Problems. In _Computer Vision – ECCV 2018 Workshops_ , Laura Leal-Taixé and Stefan Roth (Eds.). Springer International Publishing, Cham, 631–646.
  * Schneider and Vollmer (2023) Stefanie Schneider and Ricarda Vollmer. 2023.  Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art History.  _arXiv preprint arXiv:2301.05124_ (2023).
  * Shao et al. (2019) Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019\.  Objects365: A large-scale, high-quality dataset for object detection. In _Proceedings of the IEEE/CVF international conference on computer vision_. 8430–8439.
  * Springstein et al. (2022) Matthias Springstein, Stefanie Schneider, Christian Althaus, and Ralph Ewerth. 2022.  Semi-supervised Human Pose Estimation in Art-historical Images.  _arXiv preprint arXiv:2207.02976_ (2022).
  * Sun et al. (2019) Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. 2019\.  Deep high-resolution representation learning for human pose estimation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 5693–5703.
  * Terven and Cordova-Esparza (2023) Juan Terven and Diana Cordova-Esparza. 2023.  A comprehensive review of YOLO: From YOLOv1 to YOLOv8 and beyond.  _arXiv preprint arXiv:2304.00501_ (2023).
  * Tullett (2021) William Tullett. 2021\.  State of the field: sensory history.  _History_ 106, 373 (2021), 804–820.
  * van Erp et al. (2023) Marieke van Erp, William Tullett, Vincent Christlein, Thibault Ehrhart, Ali Hürriyetoğlu, Inger Leemans, Pasquale Lisena, Stefano Menini, Daniel Schwabe, Sara Tonelli, et al. 2023\.  More than the Name of the Rose: How to Make Computers Read, See, and Organize Smells.  _The American Historical Review_ 128, 1 (2023), 335–369.
  * Verbeek and Van Campen (2013) Caro Verbeek and Cretien Van Campen. 2013.  Inhaling memories: Smell and taste memories in art, science, and practice.  _The Senses and Society_ 8, 2 (2013), 133–148.
  * Wang et al. (2020) Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. 2020\.  Deep high-resolution representation learning for visual recognition.  _IEEE transactions on pattern analysis and machine intelligence_ 43, 10 (2020), 3349–3364.
  * Westlake et al. (2016) Nicholas Westlake, Hongping Cai, and Peter Hall. 2016\.  Detecting people in artwork with CNNs. In _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part I 14_. Springer, 825–841.
  * Xiao et al. (2018) Bin Xiao, Haiping Wu, and Yichen Wei. 2018.  Simple baselines for human pose estimation and tracking. In _Proceedings of the European conference on computer vision (ECCV)_. 466–481.
  * Xu et al. (2021) Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. 2021\.  End-to-end semi-supervised object detection with soft teacher. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 3060–3069.
  * Xu et al. (2022) Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. 2022\.  Vitpose: Simple vision transformer baselines for human pose estimation.  _Advances in Neural Information Processing Systems_ 35 (2022), 38571–38584.
  * Zhang et al. (2022) Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Harry Shum. 2022.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In _International Conference on Learning Representations_.
  * Zhang et al. (2021) Jing Zhang, Zhe Chen, and Dacheng Tao. 2021.  Towards high performance human keypoint detection.  _International Journal of Computer Vision_ 129, 9 (2021), 2639–2662.
  * Zhao et al. (2022a) Shu Zhao, Almila Akdağ Salah, and Albert Ali Salah. 2022a.  Automatic Analysis of Human Body Representations in Western Art. In _European Conference on Computer Vision_. Springer, 282–297.
  * Zhao et al. (2022b) Wentao Zhao, Wei Jiang, and Xinguo Qiu. 2022b.  Big transfer learning for fine art classification.  _Computational Intelligence and Neuroscience_ 2022 (2022).
  * Zheng et al. (2020) Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. 2020\.  Deep learning-based human pose estimation: A survey.  _Comput. Surveys_ (2020).
  * Zhu et al. (2020) Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2020\.  Deformable detr: Deformable transformers for end-to-end object detection.  _arXiv preprint arXiv:2010.04159_ (2020).
  * Zinnen (2021) Mathias Zinnen. 2021\.  How to See Smells: Extracting Olfactory References from Artworks. In _Companion Proceedings of the Web Conference 2021_. 725–726.
  * Zinnen et al. (2022a) Mathias Zinnen, Prathmesh Madhu, Peter Bell, Andreas Maier, and Vincent Christlein. 2022a.  Transfer Learning for Olfactory Object Detection. In _Digital Humanities Conference, 2022_. Alliance of Digital Humanities Organizations, 409–413.  https://arxiv.org/abs/2301.09906.
  * Zinnen et al. (2022b) Mathias Zinnen, Prathmesh Madhu, Ronak Kosti, Peter Bell, Andreas Maier, and Vincent Christlein. 2022b.  Odor: The icpr2022 odeuropa challenge on olfactory object recognition. In _2022 26th International Conference on Pattern Recognition (ICPR)_. IEEE, 4989–4994.