# SketchEdit: Image Editing with Partial Sketches

Yu Zeng
Johns Hopkins University
<EMAIL_ADDRESS>Zhe Lin
Adobe Research
<EMAIL_ADDRESS>Vishal M. Patel
Johns Hopkins University
<EMAIL_ADDRESS>
# SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches

Yu Zeng
Johns Hopkins University
<EMAIL_ADDRESS>Zhe Lin
Adobe Research
<EMAIL_ADDRESS>Vishal M. Patel
Johns Hopkins University
<EMAIL_ADDRESS>
###### Abstract

Sketch-based image manipulation is an interactive image editing task to modify
an image based on input sketches from users. Existing methods typically
formulate this task as a conditional inpainting problem, which requires users
to draw an extra mask indicating the region to modify in addition to sketches.
The masked regions are regarded as holes and filled by an inpainting model
conditioned on the sketch. With this formulation, paired training data can be
easily obtained by randomly creating masks and extracting edges or contours .
Although this setup simplifies data preparation and model design, it
complicates user interaction and discards useful information in masked
regions. To this end, we investigate a new paradigm of sketch-based image
manipulation: mask-free local image manipulation, which only requires sketch
inputs from users and utilizes the entire original image. Given an image and
sketch, our model automatically predicts the target modification region and
encodes it into a structure agnostic style vector. A generator then
synthesizes the new image content based on the style vector and sketch. The
manipulated image is finally produced by blending the generator output into
the modification region of the original image. Our model can be trained in a
self-supervised fashion by learning the reconstruction of an image region from
the style vector and sketch. The proposed method offers simpler and more
intuitive user workflows for sketch-based image manipulation and provides
better results than previous approaches. More results, code and interactive
demo will be available at https://zengxianyu.github.io/sketchedit.

Images Sketches Results Images Sketches Results

Figure 1: Our system allows users to interactively manipulate an image by
sketching desired contours directly on top of the image.

##  1 Introduction

Recently there have been increasing efforts and demand for building
interactive photo editing tools on devices with touch interfaces. Being
expressive and easily editable, sketching is one of the most straightforward
ways that people illustrate their creative ideas and interact with the apps
[2, 41, 10, 23]. Sketch-based image editing is an emerging research topic
where the goal is to build models which can manipulate holistic or local
structures of an image according to the user-drawn sketches. It has made a
significant progress in the last few years with advancements in deep learning
and generative models, , GANs [11].

There are two challenges in sketch-based image manipulation: (1) The input
sketch roughly indicates where to modify, but the precise modification region
is unknown, and (2) it is difficult to collect large-scale image pair data
(images before and after manipulation) for training. Previous approaches avoid
these obstacles by converting sketch-based image manipulation into a
conditional inpainting problem. Users are required to draw an extra mask to
indicate the modification regions in addition to the sketch. The masked
regions are regarded as holes and filled by an inpainting model conditioned on
the sketch, as illustrated in Fig. 2 (a). Under this inpainting framework,
paired training data can be easily obtained by randomly generating masks as in
general image inpainting [19, 39, 59, 53, 30, 15, 24, 48, 35, 22, 45, 56, 51,
50, 44, 32, 37, 35] and extracting edges/contours in the masked regions as
surrogate sketches using edge detection algorithms [1, 13, 47, 6, 5, 38, 49,
26, 25, 21, 43]. Then the problem can be solved by training an inpainting
model to predict the original image given the masked image, mask, and sketch
as input. This clever formulation simplifies model design and training setup,
and has been explored extensively in the recent literature [27, 52, 17, 33,
55].

However, the inpainting framework is sub-optimal for sketch-based image
manipulation. First, although it eases model design and training, it leaves
extra work to users and results in a complicated user interface. After users
sketch in an image region, they have to draw a mask again around the same
location. This procedure is tedious and redundant. An untrained user may not
be able to draw a good mask that exactly covers the desired modification area,
making the model difficult to produce reasonable results. Second, with this
framework, masked regions in an input image has to be removed to match the
training condition of the inpainting model. However, since the original
content in a modification region is usually highly correlated with the desired
result, ignoring this information will degrade the quality of the manipulated
image and lead to unwanted changes in the modification region.

Figure 2: Illustration of the previous inpainting-based image manipulation
systems and the proposed one with only sketch input.

To this end, we investigate a new paradigm of sketch-based image manipulation,
which requires only sketch inputs from users while leveraging the entire
original image. Our system provides a more straightforward and user friendly
interface for image manipulation: to solely sketch directly on top of the
original image, as illustrated in Fig. 2 (b). Moreover, as the system takes
the entire image as input, information in the modification region can be
reserved, resulting in more consistent appearance (color, texture) with the
original content in the modification region.

For local editing, it is desired to preserve most of the original image
content and only modify the relevant image region surrounding the sketch
input. Therefore, we first predict the modification region with a mask
estimator and then synthesize new content inside with a generator. The
manipulated image is produced by blending the generator output into the
original image using the predicted mask. To encourage the structure of the
synthesized content to follow the sketch while only keeping the style of the
original content, we encode the modification region of an input image into a
structure agnostic style vector with a style encoder. The system can be
trained in a self-supervised fashion by learning to reconstruct the target
modification region based on the style vectors and sketches.

We evaluate our method on multiple datasets, including CelebAHQ, Places2, and
newly constructed datasets SketchFace and SketchImg containing sketches and
masks drawn by users. Extensive experiments demonstrate that the proposed
method outperforms the state-of-the-art approaches. To show the advantage of
our new framework in terms of user interaction, we include an interactive demo
in the supplementary material. Our contributions are as follow,

  * •

Investigation of a new paradigm of sketch-based image manipulation: mask-free
local image manipulation that requires only partial sketch inputs from users.

  * •

The first system for mask-free sketch-based local image manipulation,
including the network architecture, data acquisition, and training strategy.

  * •

Extensive experiments are conducted on multiple datasets to demonstrate the
superiority of our approach over the related methods.

##  2 Related Work

Sketch-guided image completion.  Previous research on sketch-based image
manipulation is studied mainly under a conditional image inpainting framework.
Yu et al. [55] propose DeepFill-v2 which can perform sketch-guided image
inpainting of general images as well as face images. The gated convolution
layer is introduced to select useful features from the incomplete input
dynamically. Portenier et al. [33] propose FaceShop for sketch-based face
images manipulation through conditional image completion. FaceShop allows
users to modify the local shape and color in a face image by drawing the mask,
sketch, and a few color strokes. Jo and Park [17] explore face manipulation
with sketch and color strokes through image completion. They use free-form
masks and style loss to train the image completion model to handle irregular
and possibly large missing areas. Yang et al. [52] focus on the adaptation of
the face manipulation model trained on sketches generated by edge detection to
the human-drawn sketches. A sketch refinement strategy is proposed, which
first dilates then refines a user-drawn sketch to close to an edge detection
result. Our work is inspired by these methods but focuses on the manipulation
of complete images. We do not require the users to draw an extra mask to
construct an incomplete image when editing a photo. Instead, the region to
edit is automatically discovered and modified based on where and what they
draw.

Image-to-image translation.  Image translation aims to learn the mapping
between different image domains, to generate images from semantic label maps,
edges, . Isola et al. [16] propose an image-to-image translation framework,
called Pix2Pix, to generate images from label maps or edge maps. Zhu et al.
[60] propose CycleGAN, which allows training an image translation model on
unpaired data with a cycle consistency constraint. Park et al. [31] propose
spatially-adaptive normalization for image generation conditioned on semantic
layout, which modulates the activations using input semantic layouts to
propagate semantic information throughout the network better. Zhang et al.
[57] propose a framework for exemplar-based image translation with cross-
domain correspondence by jointly learning cross-domain correspondence and
image translation. When dealing with the edge input, image translation methods
usually require a complete edge map and generate a brand new image. In
contrast, our method focuses on local image manipulation with partial sketches
as input.

Contour-based image manipulation is another line of research where an image is
edited in the contour domain. Approaches in this category typically encode an
image into an edge-based representation and decode after users modify the
edges. Elder et al. [8] encode images with the brightness on edges and perform
image editing operations (crop, paste, delete) in the contour domain. Dekel et
al.[3] propose to encode images with learned features at contour points to
perform complex changes in the image domain by simple edits of contours.
Vinker et al. [40] explore deep image manipulation with a single training
sample by extensive augmentation of the training sample through thin-plate
splines (TPS) warping. A conditional generator trained on the augmented
samples can map the edited edges and segmentation to the modified image. These
methods require the contour representation to be complete from which the image
can be faithfully reconstructed and assume that the users have basic photo
editing skills to perform certain editing operations on a pre-computed edge
map. In comparison, our method allows a broader range of users to manipulate
an image more easily, by directly sketching on top of the input image.

##  3 Proposed Method

We focus on local image manipulation based on free-form sketches. Given an
input image $x$ and partial sketch $c$ indicating the target contour, our
model modifies a local region of $x$ to match the sketch, resulting in a
manipulated image $y$. Due to the lack of data for supervised training, we
propose a self-supervised learning approach to jointly learn mask estimation
and masked region reconstruction.

###  3.1 Model

Our model consists of three components: mask estimator, structure agnostic
style encoder, and generator. Fig. 3 shows the overall structure of our model.
Given an input image $x$ and sketch $c$, the generator synthesizes new content
based on the image, sketch, and a structure agnostic style vector produced by
the style encoder. The mask estimator predicts a mask to blend the synthesized
content into the original image at a proper location. In what follows, we
describe each of these components in detail.

Mask estimator. The mask estimator $M$ is an encoder-decoder style network
that maps an image $x$ and a partial sketch map $c$ to a mask $m$, $m=M(x,c)$.
The mask is produced by applying the sigmoid function on the output of a
convolutional layer, which is a gray-scale map of range $[0,1]$. We use the
term mask for consistency with previous inpainting-based approaches. Each
element of the mask represents how likely the corresponding pixel of the input
image should be modified to match the sketch. Thus, we can separate the
modification region and non-modification static region by multiplying an input
image $x$ with $m$ and $1-m$ element-wise, respectively. Since we are
interested in the style information in the modification region, we define a
style partial image as $x^{sty}=m\odot x$, where $\odot$ represents element-
wise multiplication. We define a static partial image $x^{sta}=(m-1)\odot x$
as the non-modification region of the input image.

Figure 3: Model structure of the proposed method. Given an input image $x$ and
sketch $c$, the mask estimator predicts a mask $m$ indicating the region to
edit. The style partial image $x^{sty}$ and the static partial image $x^{sta}$
can be created by multiplying $x$ with $m$ and $1-m$ element-wise,
respectively. Then the generator synthesizes new content in the modification
region based on the static image $x^{sta}$, sketch $c$, and a structure
agnostic style feature extracted from $x^{sty}$ by the style encoder.

Structure agnostic style encoder. The goal of the style encoder $S$ is to
extract the style information from the modification region of the input image
but get rid of the structure information. It is composed of stacked
convolutional layers followed by a global max pooling, which produces a
$d$-dimensional style vector sty given the style partial image $x^{sty}$ and
the mask $m$, $v^{sty}=S(x^{sty},m)$. By repeating $v^{sty}$ by $h\times w$
times, we can obtain a style feature $\hat{v}^{sty}_{(d\times h\times w)}$ of
arbitrary spatial size $h\times w$ while keeping position invariant. However,
it may still be sensitive to spatial transformation due to the local
connectivity of convolutional layers. To further encourage the extracted style
features to be structure agnostic, we apply random warping and deterioration
to $x^{sty}$ in training, which will be introduced in Sec. 3.2.

Generator. The proposed framework is general and does not depend on the
specific architecture of the generator. We can combine it with most existing
inpainting-based models by using the estimated modification regions as missing
regions and tiling the style vector into style feature maps of the suitable
size as a generator’s input. In our implementation, we incorporate the network
architecture of DeepFill-v2 [55] due to its simplicity and lightweight. The
generator $G$ consists of a coarse stage $G_{0}$ and refinement stage network
$G_{1}$. The coarse stage takes the static partial image, estimated mask, and
sketch as input. We use the style feature maps $\hat{v}^{sty}$ as the side
input at the bottleneck of the coarse stage network. The output of the coarse
stage is denoted as $y_{0}$, which is passed through the refinement stage to
produce a refined output $y_{1}$:

$\begin{split}y_{0}&=G_{0}(x^{sta},m,\hat{v}^{sty})\\\
y_{1}&=G(x^{sta},m,\hat{v}^{sty})=G_{1}(y_{0}).\\\ \end{split}$ (1)

The final result $y$ is produced by blending the refinement stage output into
the original image using the estimated mask

$y=y_{1}\odot m+x\odot(1-m).$ (2)

To train the generator, we minimize the L1 error between the output
$y_{0},y_{1},y$ and the original image $x$. We also use adversarial training
with a discriminator to encourage visual realism of the blended image $y$. The
detailed training strategy and loss functions will be described in the next
section.

###  3.2 Learning by reconstruction

Figure 4: Illustration of a training iteration. The mask estimator, style
encoder, and generator are trained jointly by learning to reconstruct the
original images. To avoid the trivial solution of generating all-zero masks,
we warp the input image in a local areal before passing them through the mask
estimator. A bi-directional mask regularization loss (BMR) is applied to help
the mask estimator produce reasonable masks rather than simply detecting
warping artifacts.

Due to the lack of training data, previous inpainting-based approaches resort
to self-supervised learning, specifically, by learning to reconstruct the
original image from a masked image and sketch. In this work, we also seek to
train our model using self-supervision. However, as our model infers the mask,
directly learning by reconstructing the original image will lead to a trivial
solution. The mask estimator will constantly predict an all-zero mask; the
generator becomes an identity mapping; the style encoder does not need to
learn any meaningful representation. We solve this issue by deteriorating the
input images with random local warping and regional dropout before passing
them through the model. Fig. 4 shows a training iteration of the proposed
method.

Local warping. Before passing an input image through the model in training, we
warp $x$ in a random area using triangular local warping. We construct a
triangular mesh by placing vertices on the image boundary, boundary of a
randomly sampled region, and inside the region. Then a warping flow can be
created by moving interior vertices while keeping other vertices unmoved. For
general images, we place and move the interior vertices randomly. For face
images, to make the warped faces remain visually plausible, we use facial
landmarks and blendshapes to guide the selection and movements of the interior
vertices. The detailed warping algorithm can be found in the supplementary
material. Fig. 5 shows samples augmented by warping. We can see that the
warped images have the same color and texture as the original ones but with
different structures in warped regions. Therefore, we can construct the
original image with structure information from the sketch and style
information extracted from a warped image. Also, as a warped image is
identical to the original image in non-warped regions, the mask estimator will
be encouraged to predict zero where no sketch appears. Thus the regions
irrelevant to sketches can be preserved.

Figure 5: Example images augmented by local warping. Top: original images,
bottom: warped images.

Bi-directional mask regularization. Since we warp the input images in
training, the mask predictor may end up producing masks by simply detecting
warping artifacts and fail in the inference stage, as shown in thee third
column of Fig. 6. To avoid this behavior, we train the mask estimator with a
bi-directional image reconstruction loss as regularization. Let $f(x)$ denote
the image produced by distorting $x$ with the warping flow $f$. By applying
the same warping flow to the sketch $c$, we obtain a sketch $f(c)$
corresponding to the warped image. During training, we add an extra decoder to
the mask estimator to predict an image, with the goal of estimating the
original image $x$ from the warped image $f(x)$ and the original sketch $c$,
and the opposite, to estimate $f(x)$ from $x$ and $f(c)$. Thus the mask
estimator $M$ has an extra output in addition to the mask output $m=M(x,c)$.
For input image $x$ and sketch $c$, we let $\bar{M}(x,c)$ represent the
estimated image and $\hat{M}(x,c)$ represent the image blended using the mask,
$\hat{M}(x,c)=\bar{M}(x,c)\odot M(x,c)+x\odot(1-M(x,c))$. The bi-directional
mask regularization (BMR) loss is defined as follows:

$\begin{split}L_{BMR}&=\norm{\bar{M}(f(x),c)-x}_{1}+\norm{\bar{M}(x,f(c))-f(x)}_{1}\\\
&+\norm{\hat{M}(f(x),c)-x}_{1}+\norm{\hat{M}(x,f(c))-f(x)}_{1}.\end{split}$

Since $L_{BMR}$ is computed for both the warped and the original images, the
mask estimator can be adapted to natural image input and thus can predict
reasonable masks in the inference stage, as shown in the forth column of Fig.
6.

Image Sketch w/o BMR w/ BMR

Figure 6: The predicted masks with and without the bi-directional mask
regularization (BMR). Top: warped images, bottom: real images. BMR helps the
mask estimator generalize better to real image inputs.

Regional dropout. Warping helps the style encoder learn structure agnostic
representation by creating structural variation while preserving the style in
a local region. However, the warped image and the original image usually have
identical visual elements, while real cases might involve the generation of
new elements. Therefore, during training, we randomly dropout sampled regions
in the style image $x^{sty}$ before passing it through the style encoder to
reduce the correlation between $v^{sty}$ and $x$.

Loss functions. For a pair of image $x$ and sketch map $c$ in the training
set, we compute the following loss functions to jointly train the mask
estimator, style encoder, and the generator:

$L_{R}=\norm{y_{0}-x}_{1}+\norm{y_{1}-x}_{1}+\norm{y-x}_{1}.$ (3)

Eqn. 3 measures the error of the coarse stage output $y_{0}$ and refinement
stage output $y_{1}$, as well as the error of the manipulated image $y$, which
is the combination of $x$ and $y_{1}$ blended using the estimated mask. In
addition,

$L_{G}=\mbox{ReLU}[\mathbbm{1}-D(y,c)],$ (4)

where $D$ represents the discriminator. For the discriminator, we use the
hinge loss and spectral normalization [29] in [54, 55]. Eqn. 4 defines the
adversarial loss inspired by GANs. To minimize $L_{G}$, the generator and mask
estimator have to cooperate reasonably to fool the discriminator. It requires
the generator to synthesize visually realistic content and the mask estimator
to provide the mask that blends the synthesized content into a proper region
in the original image seamlessly. The overall loss function is the sum of
$L_{G}$, $L_{R}$ and the mask regularization term $L_{BMR}$:

$L_{total}=L_{R}+L_{G}+L_{BMR}.$ (5)

##  4 Experiments

As introduced in Sec. 3.1, our framework can be combined with most inpainting-
based models to enable mask-free manipulation. In our implementation, we plug
the DeepFill-v2 generator into our framework due to its simplicity and
efficiency. To verify the effectiveness of the proposed method, we compare the
results under our mask-free framework to the original DeepFill-v2 as a
baseline. We also compare with other state-of-the-art inpainting-based face
manipulation methods including SC-FEGAN [17] and DeepPS [52] to demonstrate
the superiority of our approach.

###  4.1 Datasets & Evaluation Metric

We train our general image manipulation model on the training split of Places
[36]. For face manipulation, we train the model on CelebHQ [18] for
consistency with the inpainting-based approaches. We randomly sample 2,000
images as the validation set and use the remaining data for training.

For evaluation, we use edge maps extracted by [13] as surrogate sketches for
Places and obtain the sketches in face regions by connecting detected facial
landmarks for the CelebAHQ dataset. We randomly sample modification regions
for each test image and obtain partial sketches by discarding all edges
outside this region. For evaluation of our method, we apply local warping in
modification regions. For inpainting-based methods, we set pixel values in
modification regions equal to zero according to their training condition. For
synthetic samples, the original images can be seen as the ground-truth. Hence,
we evaluate the performance using PSNR, L1 error, SSIM, and FID [14] between
the results and the original images. PSNR, L1 error and SSIM are based on
pixel-wise errors, often used in image inpainting and restoration tasks [54,
7, 58, 12, 46, 42, 34, 20]. FID measures the distance between the distribution
of the deep features of generated images and real images. It is the current
standard metric for assessing the quality of generative models [28].

Apart from using synthetic samples for evaluation, to reflect the performance
in real cases, we construct two test sets SketchImg and SketchFace for face
and general image manipulation, respectively. These two datasets contain 100
natural images and 400 face images as well as the corresponding sketches and
masks drawn by human users. The samples are created under the conditional
inpainting framework, where the users draw a sketch indicating the target
contour and the mask that marks the region to modify. To evaluate our method,
we simply ignore the masks and only use the images and sketches as input. For
the real cases, the results are supposed to have a similar style to the input
images. Therefore, we evaluate the performance using style loss [9], the mean
squared error between the Gram matrices of the deep features of the input
image and results. We use the feature maps from the relu_1 and relu_2 of a VGG
[36] network pretrained on ImageNet [4]. We denote the obtained style loss as
SL_1 and SL_1, respectively. We also report FID between the results and input
images as a reflection of the image quality.

###  4.2 Qualitative Evaluation

Fig. 7 presents the face manipulation results of our method and the state-of-
the-art methods. As our model leverages entire images as input, it can better
reserve the original appearance in the modification region of the input image,
the eyebrows in the firs row and the lip color in the second row. Our method
can also produces visually realistic results for more challenging general
image manipulation. Fig. 8 shows the general image manipulation results of our
method and DeepFill-v2. It can be seen that our method well preserves the
colors and textures of the original images. In contrast, DeepFill-v2 often
adds extra changes in the modification regions, it changes the colors of the
island, arch, and the beak in the first, second, and forth row, respectively.
In addition, as our framework prevents the information loss caused by masking
out the modification area, our results are less blurry and have more details.

aaaImageaa aaInput*aa DeepFill-v2 aSC-FEGANa aaDeepPSaa aaaInputaa aaaOursaaa

Figure 7: Visual comparison of face manipulation results on CelebAHQ dataset.
Input*: input sketch and masks for inpainting-based methods. Input: input
sketch for our method.

a aaImageaa aaaInput*aa DeepFill-v2 aaaInputaa aaaOursaaa

Figure 8: Visual comparison of general image manipulation results on the
Places dataset. Input*: input sketch and masks for inpainting-based methods.
Input: input sketch for our method.

###  4.3 Quantitative Evaluation

Table 1 reports the quantitative evaluation results on the synthetic samples
from the CelebAHQ and Places validation sets. It can be seen that our results
have larger PSNR, SSIM and smaller L1 error than previous approaches, which
imply a smaller difference between the generated images and the corresponding
ground-truth. Our results also have smaller FID, indicating a closer distance
to the distribution of original images. Table 2 reports the quantitative
evaluation results on real samples. It can be seen that the style loss of our
results are significantly smaller than previous approaches, as our mask-free
framework effectively keeps the style of the input images. Our method also
outperforms previous approaches in terms of FID on real samples.

Table 1: Quantitative comparison on synthetic samples from CelebAHQ (top) and
Places (bottom) validation sets.

CelebAHQ
---
Method |  L1 error$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  FID$\downarrow$
SC-FEGAN | 0.0110 | 30.39 | 0.9408 | 2.693
DeepPS | 0.0125 | 30.54 | 0.9340 | 2.460
DeepFill-v2 | 0.0115 | 30.59 | 0.9370 | 2.345
Ours | 0.0071 | 34.21 | 0.9623 | 0.806

Places
---
Method |  L1 error$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  FID$\downarrow$
DeepFill-v2 | 0.0293 | 24.06 | 0.8442 | 2.238
Ours | 0.0223 | 26.35 | 0.8527 | 1.500

Table 2: Quantitative comparison on SketchFace and SketchIMG containing
sketches and masks drawn by human users.

SketchFace | SketchImg
---|---
Method | SC-FEGAN | DeepPS | DeepFill-v2 | Ours | DeepFill-v2 | Ours
FID$\downarrow$ | 4.092 | 5.285 | 4.382 | 3.31 | 27.11 | 24.27
SL_1$\downarrow$ | 2484 | 3797 | 2859 | 2211 | 5314 | 4374
SL_2$\downarrow$ | 13145 | 15585 | 13970 | 10819 | 24169 | 21575

###  4.4 Ablation Study

Mask estimator. The mask estimator is an important component in our framework
to enable local manipulation and encourage the model to learn better
generative capability. Without the mask predictor, we may train a common
conditional GAN on training samples synthesized by warping. However, a model
trained to mimic warping often fails in challenging cases involving generating
new elements or deleting existing elements. For example, as shown in Fig. 9,
the generator jointly trained with the mask estimator correctly deletes and
adds teeth in manipulation of mouths. In contrast, a common conditional GAN
only squeezes or stretches the mouth.

Sketch Image w/o ME with ME

Figure 9: Top: results of the model without the mask estimator (ME), bottom:
results with ME.

The mask estimator also helps to better preserve the non-modification regions.
Without the mask estimator, slight changes in the static region is almost
inevitable. This is reflected by the larger L1 error and smaller PSNR in the
third row of Table 3. Another baseline without the mask estimator is to use
masks generated based on rules. The third row reports the results with the
rule-based masks given by the minimum bounding boxes enclosing sketches. By
comparing the first row and the third row, we can see the significant
improvements brought by the mask estimator.

Table 3: Effect of each components. Rule-based Mask: masks given by the minimum bounding boxes enclosing sketches, ME: mask estimator, SE: style encoder. The first row corresponds to the full model with both ME and SE.  w/o Mask | Rule-based Mask | ME | SE |  L1 error$\downarrow$ |  PSNR$\uparrow$
---|---|---|---|---|---
|  | ✓ | ✓ | 0.0071 | 34.21
✓ |  |  | ✓ | 0.0127 | 32.50
| ✓ |  | ✓ | 0.0102 | 31.87
|  | ✓ |  | 0.0074 | 33.98

Style encoder. The style encoder is essential to keep the appearance of the
modification region. The model without the style encoder has no access to
information in the modification region and generates new content based on
prior, resulting in unexpected changes. Fig. 10 shows example results for face
manipulation. It can be seen that the model with the style encoder reserves
the colors of the lips and eyes, while the one without the style encoder
changes the colors.

Sketch Image w/o SE with SE

Figure 10: Top: results of the model without the style encoder (SE), bottom:
results with SE.

The quantitative evaluation results also reflect the effect of the style
encoder. The first and the forth row of Table 3 report the quantitative
evaluation results with and without the style encoder, respectively. It can be
seen that the model with the style encoder results in smaller L1 error and
larger PSNR.

##  5 Conclusion, Limitations and Future Work

This paper presents a new paradigm of sketch-based image manipulation, mask-
free local image manipulation, and propose a complete system implementation
for this task including the network architecture, data acquisition and
training strategy. The proposed method offers simpler and more robust user
workflows for sketch-based image manipulation and provides consistently better
results than the related approaches. A limitation is that the current
framework only supports structure editing based on monochrome sketches. Color
editing based on colored sketches under the proposed new mask-free framework
is an interesting topic for future work. Potential negative social impact of
this work might include: fake news can be created by manipulating photos; The
spreading of manipulated images may distort of people’s perception of body
images and beauty. The negative impact can be mitigated by embedding digital
watermarks in manipulated images to make them easy to identify.

## References

  * [1] John Canny.  A computational approach to edge detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, (6):679–698, 1986.
  * [2] Wengling Chen and James Hays.  Sketchygan: Towards diverse and realistic sketch to image synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 9416–9425, 2018.
  * [3] Tali Dekel, Chuang Gan, Dilip Krishnan, Ce Liu, and William T Freeman.  Sparse, smart contours to represent and edit images.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image batabase.  In IEEE Conference on Computer Vision and Pattern Recognition, 2009\.
  * [5] Ruoxi Deng and Shengjun Liu.  Deep structural contour detection.  In ACM Int. Conf. Multimedia, 2020.
  * [6] Ruoxi Deng, Chunhua Shen, Shengjun Liu, Huibing Wang, and Xinru Liu.  Learning to predict crisp boundaries.  In European Conference on Computer Vision, 2018.
  * [7] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295–307, 2015.
  * [8] James H Elder and Richard M Goldberg.  Image editing in the contour domain.  In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1998.
  * [9] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  A neural algorithm of artistic style.  arXiv preprint arXiv:1508.06576, 2015.
  * [10] Arnab Ghosh, Richard Zhang, Puneet K Dokania, Oliver Wang, Alexei A Efros, Philip HS Torr, and Eli Shechtman.  Interactive sketch & fill: Multiclass sketch-to-image translation.  In International Conference on Computer Vision, 2019.
  * [11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Conference on Neural Information Processing Systems, 27, 2014.
  * [12] Wei Han, Shiyu Chang, Ding Liu, Mo Yu, Michael Witbrock, and Thomas S Huang.  Image super-resolution via dual-state recurrent networks.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [13] Jianzhong He, Shiliang Zhang, Ming Yang, Yanhu Shan, and Tiejun Huang.  Bi-directional cascade network for perceptual edge detection.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3828–3837, 2019.
  * [14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Conference on Neural Information Processing Systems, 30, 2017.
  * [15] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics, 36(4):1–14, 2017.
  * [16] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1125–1134, 2017.
  * [17] Youngjoo Jo and Jongyoul Park.  Sc-fegan: Face editing generative adversarial network with user’s sketch and color.  In International Conference on Computer Vision, pages 1745–1753, 2019.
  * [18] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [19] Avisek Lahiri, Arnav Kumar Jain, Sanskar Agrawal, Pabitra Mitra, and Prabir Kumar Biswas.  Prior guided gan based semantic inpainting.  In IEEE Conference on Computer Vision and Pattern Recognition, 2020\.
  * [20] Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha.  Recurrent squeeze-and-excitation context aggregation net for single image deraining.  In European Conference on Computer Vision, 2018.
  * [21] Liang Liao, Jing Xiao, Zheng Wang, Chia-Wen Lin, and Shin’ichi Satoh.  Guidance and evaluation: Semantic-aware image inpainting for mixed scenes.  In European Conference on Computer Vision, 2020.
  * [22] Liang Liao, Jing Xiao, Zheng Wang, Chia-Wen Lin, and Shin’ichi Satoh.  Image inpainting guided by coherence priors of semantics and textures.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [23] Fang Liu, Xiaoming Deng, Yu-Kun Lai, Yong-Jin Liu, Cuixia Ma, and Hongan Wang.  Sketchgan: Joint sketch completion and recognition with generative adversarial network.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [24] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro.  Image inpainting for irregular holes using partial convolutions.  In European Conference on Computer Vision, 2018.
  * [25] Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang.  Rethinking image inpainting via a mutual encoder-decoder with feature equalizations.  In European Conference on Computer Vision, 2020.
  * [26] Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, and Jing Liao.  Pd-gan: Probabilistic diverse gan for image inpainting.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [27] Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, Jing Liao, Bin Jiang, and Wei Liu.  Deflocnet: Deep image editing via flexible low-level controls.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [28] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet.  Are gans created equal? a large-scale study.  In Conference on Neural Information Processing Systems, 2018.
  * [29] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  arXiv preprint arXiv:1802.05957, 2018.
  * [30] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi.  Edgeconnect: Generative image inpainting with adversarial edge learning.  arXiv preprint arXiv:1901.00212, 2019.
  * [31] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2337–2346, 2019.
  * [32] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In IEEE Conference on Computer Vision and Pattern Recognition, 2016\.
  * [33] Tiziano Portenier, Qiyang Hu, Attila Szabo, Siavash Arjomand Bigdeli, Paolo Favaro, and Matthias Zwicker.  Faceshop: Deep sketch-based face image editing.  ACM Transactions on Graphics, 2018.
  * [34] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng.  Progressive image deraining networks: A better and simpler baseline.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [35] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li.  Structureflow: Image inpainting via structure-aware appearance flow.  In International Conference on Computer Vision, 2019.
  * [36] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [37] Yuhang Song, Chao Yang, Zhe Lin, Xiaofeng Liu, Qin Huang, Hao Li, and C-C Jay Kuo.  Contextual-based image inpainting: Infer, match, and translate.  In European Conference on Computer Vision, pages 3–19, 2018.
  * [38] Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikainen, and Li Liu.  Pixel difference networks for efficient edge detection.  In International Conference on Computer Vision, 2021.
  * [39] Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan.  Distillation-guided image inpainting.  In International Conference on Computer Vision, pages 2481–2490, 2021.
  * [40] Yael Vinker, Eliahu Horwitz, Nir Zabari, and Yedid Hoshen.  Deep single image manipulation.  arXiv preprint arXiv:2007.01289, 2020.
  * [41] Sheng-Yu Wang, David Bau, and Jun-Yan Zhu.  Sketch your own gan.  In International Conference on Computer Vision, pages 14050–14060, 2021.
  * [42] Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson WH Lau.  Spatial attentive single-image deraining with a high quality real rain dataset.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [43] Yi Wang, Ying-Cong Chen, Xin Tao, and Jiaya Jia.  Vcnet: A robust approach to blind image inpainting.  In European Conference on Computer Vision, 2020.
  * [44] Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia.  Image inpainting via generative multi-column convolutional neural networks.  In Conference on Neural Information Processing Systems, 2018.
  * [45] Jing Xiao, Liang Liao, Qiegen Liu, and Ruimin Hu.  Cisi-net: Explicit latent content inference and imitated style rendering for image inpainting.  In he AAAI Conference on Artificial Intelligence, 2019.
  * [46] Minshan Xie, Menghan Xia, and Tien-Tsin Wong.  Exploiting aliasing for manga restoration.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021\.
  * [47] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In International Conference on Computer Vision, pages 1395–1403, 2015.
  * [48] Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo.  Foreground-aware image inpainting.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [49] Dan Xu, Wanli Ouyang, Xavier Alameda-Pineda, Elisa Ricci, Xiaogang Wang, and Nicu Sebe.  Learning deep structured multi-scale features using attention-gated crfs for contour prediction.  2018\.
  * [50] Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li.  High-resolution image inpainting using multi-scale neural patch synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition, 2017\.
  * [51] Jie Yang, Zhiquan Qi, and Yong Shi.  Learning to incorporate structure knowledge for image inpainting.  In he AAAI Conference on Artificial Intelligence, 2020.
  * [52] Shuai Yang, Zhangyang Wang, Jiaying Liu, and Zongming Guo.  Deep plastic surgery: Robust and controllable image editing with human-drawn sketches.  In European Conference on Computer Vision, 2020.
  * [53] Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, and Zhan Xu.  Contextual residual aggregation for ultra high-resolution image inpainting.  In IEEE Conference on Computer Vision and Pattern Recognition, 2020\.
  * [54] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [55] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Free-form image inpainting with gated convolution.  In International Conference on Computer Vision, 2019.
  * [56] Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, and Sen Liu.  Region normalization for image inpainting.  In he AAAI Conference on Artificial Intelligence, 2020.
  * [57] Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen.  Cross-domain correspondence learning for exemplar-based image translation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 5143–5153, 2020.
  * [58] Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu.  Residual non-local attention networks for image restoration.  arXiv preprint arXiv:1903.10082, 2019.
  * [59] Yuqian Zhou, Connelly Barnes, Eli Shechtman, and Sohrab Amirghodsi.  Transfill: Reference-guided image inpainting by merging multiple color and spatial transformations.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2266–2276, 2021.
  * [60] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In International Conference on Computer Vision, pages 2223–2232, 2017.


11institutetext: City, University of London, United Kingdom
11email: {david.herron, ernesto.jimenez-ruiz, giacomo.tarroni,
<EMAIL_ADDRESS>22institutetext: SIRIUS, University of Oslo, Norway
33institutetext: Imperial College London, United Kingdom

# NeSy4VRD: A Multifaceted Resource for Neurosymbolic AI Research using
Knowledge Graphs in Visual Relationship Detection

David Herron 11 0009-0008-2736-6789 Ernesto Jiménez-Ruiz 1122
0000-0002-9083-4599 Giacomo Tarroni 1133 0000-0002-0341-6138 Tillman Weyde 11
0000-0001-8028-9905

###### Abstract

NeSy4VRD is a multifaceted resource designed to support the development of
neurosymbolic AI (NeSy) research. NeSy4VRD re-establishes public access to the
images of the VRD dataset and couples them with an extensively revised,
quality-improved version of the VRD visual relationship annotations.
Crucially, NeSy4VRD provides a well-aligned, companion OWL ontology that
describes the dataset domain. It comes with open source infrastructure that
provides comprehensive support for extensibility of the annotations (which, in
turn, facilitates extensibility of the ontology), and open source code for
loading the annotations to/from a knowledge graph. We are contributing
NeSy4VRD to the computer vision, NeSy and Semantic Web communities to help
foster more NeSy research using OWL-based knowledge graphs.

###### Keywords:

Neurosymbolic AI Deep Learning OWL Ontologies Knowledge Graphs Reasoning
Visual Relationship Detection.

Resource type: Image Dataset with Ontology and Infrastructure for
Extensibility
Licenses: CC BY 4.0 (Dataset/Ontology), MIT/Expat (Infrastructure)
Dataset and ontology: https://doi.org/10.5281/zenodo.7916355
Infrastructure: https://github.com/djherron/NeSy4VRD

##  1 Introduction

Neurosymbolic AI (NeSy) explores combining neural learning with symbolic
background knowledge and reasoning [1]. Interest in NeSy is accelerating [3,
7]. The symbolic background knowledge representation and deductive reasoning
capabilities of OWL ontologies and knowledge graphs (KGs) mark them as
compelling candidates for participating as symbolic components in hybrid NeSy
systems.

The Semantic Web has a strong association with the symbolic tradition of AI
[8]. OWL-based knowledge graphs are exemplars of the explicit symbolic
knowledge and symbol manipulation and reasoning machinery that Marcus [15]
advocates be included in hybrid NeSy systems. The many synergies between
Semantic Web technologies and deep learning promise improved predictive
performance [9]. Additionally, there are many benefits of OWL-based KGs for
NeSy systems [6].

However, the amount of NeSy research using OWL-based KGs, particularly
leveraging their reasoning capabilities, is low. A recent study of approaches
to combining Semantic Web technologies with machine learning [2], involving
nearly 500 papers, finds that only 29 of the papers report using ‘deductive
components’ of any kind, and of these only 20 report actually using ‘reasoning
capabilities’. Thus, the potential for OWL-based KGs to contribute to NeSy
research is under-explored and, as we argue in [6], they merit a larger more
prominent role in NeSy research.

An important factor in explaining the low volume of NeSy research using OWL-
based KGs, in addition to the cross-disciplinary nature of the endeavour, is
surely that it has highly specialised prerequisites: a dataset for neural
learning, plus a companion OWL ontology describing the domain of the data over
which a knowledge graph enables pertinent reasoning. Such specialised
resources are scarce, especially in combination, and, for many application
areas, may not exist. The scarcity of appropriate dataset/ontology resources
is hindering promising NeSy research that could use OWL-based KGs.

In this paper we present NeSy4VRD, a multifaceted resource designed to foster
NeSy research using OWL-based KGs within the application area of visual
relationship detection in images. The paramount purpose of NeSy4VRD is to
lower the barriers to entry for conducting NeSy research using OWL-based KGs.
NeSy4VRD builds on top of the Visual Relationship Detection (VRD) dataset [13]
and provides an extensively revised, quality-improved version of its visual
relationship annotations. The main components and contributions of NeSy4VRD
are as follows:

  1. (i)

an image dataset with high-quality visual relationship annotations that
reference a large number of object classes and predicates (relations);

  2. (ii)

a well-aligned, companion OWL ontology, called VRD-World, that describes the
domain of the images and visual relationships;

  3. (iii)

sample Python code for loading the annotated visual relationships into a
knowledge graph hosting the VRD-World ontology, and for extracting them from a
knowledge graph and restoring them to their native format;

  4. (iv)

support for extensibility of the annotations (and, thereby, the ontology) in
the form of (a) comprehensive Python code enabling deep but easy analysis of
the images and their annotations, (b) a custom, text-based protocol for
specifying annotation customisation instructions declaratively, and (c) a
configurable, managed Python workflow for customising annotations in an
automated, repeatable process.

The remainder of this paper is structured as follows. Section 2 outlines the
background motivation for NeSy4VRD. Sections 3-6 describe the four components
of NeSy4VRD mentioned above. Section 7 highlights intended and potential
NeSy4VRD user groups and use cases. Finally, Section 8 concludes with summary
remarks.

##  2 Background to NeSy4VRD

Our ambition was to conduct vision-based NeSy research using OWL-based KGs,
but we were unable to find an image dataset with a companion OWL ontology
appropriate for enabling our research vision. During our search for an
appropriate resource, the application task of visual relationship detection
caught our attention. A number of image datasets exist that are accompanied by
annotations (in various formats) describing visual relationships between
ordered pairs of objects in the images, such as [11, 12, 13, 16] and various
dataset versions derived from these (as mentioned in [17]). But none of these
has a companion OWL ontology describing the domain of the images.

###  2.1 The VRD dataset

To advance our research, we selected the Visual Relationship Detection (VRD)
image dataset [13] and resolved to engineer our own custom, companion OWL
ontology. The visual relationships annotated for the VRD images are 5-tuples

     (subj_bbox, subj_class, predicate, obj_bbox, obj_class)

that, for simplicity, can be thought of as 3-tuples, where the subject and
object of each relationship are understood to be specified in terms of both a
box (i.e., subj_bbox and obj_bbox) and class (i.e., subj_class and obj_class).
The predicate describes some relation between the ordered pair of objects.
Figure 1 shows two example VRD images with their objects and some of their
annotated visual relationships.

Figure 1: Two representative images from the VRD dataset with samples of
representative annotated visual relationships as 3-tuples.

###  2.2 Attractive characteristics of the VRD dataset

Several characteristics make the VRD image dataset attractive, especially for
NeSy research. The first is its small size: 4,000 training images and 1,000
test images. Since deep learning is known to be data hungry, the small amount
of training data (in theory) leaves greater space for symbolic components
(like OWL-based KGs) to enhancing deep learning’s predictive performance.
Second, relative to its size, the number of distinct (common) object classes
and predicates (mostly common spatial relations, verbs and prepositions)
referenced in its visual relationships is relatively large: 100 and 70,
respectively. Third, the distribution of the types of annotated visual
relationships,

$(s_{i},p_{k},o_{j})\quad i,j=1,\dots,100\quad k=1,\dots,70,$ (1)

has a long tail that provides plentiful zero/few-shot learning scenarios.
These scenarios represent ideal opportunities for NeSy researchers to explore
ways by which symbolic components (like OWL-based KGs) can improve deep
learning’s ability to generalise.

The survey [17] provides evidence that many researchers have been attracted to
the VRD dataset. It also conveys the fact that the VRD dataset has supported
research that has focused not only on the task of visual relationship
detection in isolation but also on the broader, more general task of scene
graph generation, which relies on visual relationship detection as a building
block capability. As explained in [10, 17], scene graphs are commonly used as
precursors to a variety of enriched, downstream visual understanding and
reasoning application tasks, such as image captioning, visual question
answering, image retrieval, image generation and multimedia event processing.

###  2.3 Unattractive characteristics of the VRD dataset

Motivated by a determination to engineer a companion OWL ontology for the VRD
dataset that we knew to be robust, we embarked upon a deep analysis of the VRD
features. Despite its attractive characteristics and popularity, our analysis
revealed that its visual relationship annotations contain many serious
shortcomings. Our primary concern was to ensure that we fully understood the
semantics of the 100 object class names and 70 predicate names so that we
could make reliable, informed decisions (a) when constructing an OWL class
hierarchy and (b) when specifying object property characteristics and
relationships, and when specifying classes for property domain and range
restrictions. For example, the object class name plate suggests dishware, but
does it really mean that, and is that all it means? Similarly, the predicate
across feels familiar at first glance, but how is it really used, and is it
used in one way consistently? Unfortunately, the more our analysis progressed,
the more the quantity and seriousness of the quality issues we uncovered
accumulated.

The main categories of quality problems that exist within the original VRD
visual relationship annotations are as follows:

  1. (1)

stark variability in the types of objects sharing certain object class names;

  2. (2)

different object class names for objects clearly drawn from the same
distribution and otherwise indistinguishable from one another;

  3. (3)

diverse semantics (usage) of single predicate names;

  4. (4)

outright errors in visual relationship construction;

  5. (5)

multiple near duplicate bounding boxes for the same object in a single image;

  6. (6)

multiple near and/or exact duplicate visual relationships annotated for an
image;

  7. (7)

poor quality bounding boxes.

Examples of category (1) problems are:

  * •

object class bear pertains to real bears and teddy bears;

  * •

object class plate pertains to dishware plates, vehicle license plates,
baseball ‘home’ plates and bases, plaques on walls, etc.;

  * •

object class glasses pertains to eyeglasses, drinking glasses, miscellaneous
things made of glass (e.g. table tops, glass barriers, etc.);

  * •

object class person pertains to people and to stereo (audio) speakers.

Examples of category (2) problems are:

  * •

object classes plane and airplane label objects that are indistinguishable;

  * •

object classes coat and jacket do the same;

  * •

object classes road and street do the same.

Examples of category (3) problems are:

  * •

predicate across is used in the sense of ‘along side of’, ‘is crossing the’,
‘is across from’, etc.;

  * •

predicate fly is used in the sense of ‘is flying a’, ‘is flown by’, ‘is flying
in‘, ‘is flying above’, etc.;

  * •

predicate walk is used in the sense of ‘walk on’, ‘walk next to’, ‘walk
towards each other’, etc..

Examples of category (4) problems are:

  * •

in possessive relationship patterns such as (person, wear, X) or (person,
hold, X), frequently the bounding box for X places it on/with a different
person;

  * •

in positional relationship patterns such as (X, behind, Y) or (X, above, Y),
frequently either X and Y need to be swapped, or the predicate changed to its
inverse.

####  2.3.1 Impact of the VRD shortcomings.

The VRD annotation quality problem categories just discussed have real
consequences for the application task of visual relationship detection and
will have (silently) negatively impacted all of the research that has been
undertaken using the VRD dataset. Problem categories (1), (2), (5) and (7)
hamper reliable object detection, while categories (3), (4) and (6) have a
negative impact on the detection of relationships.

The problem categories (1) and (3) also had real consequence for our objective
of engineering a robust and credible OWL ontology to act as companion to the
VRD dataset to enable our NeSy research using knowledge graphs. Together,
problem categories (1) and (3) were serious enough that they effectively made
the modeling of a credible ontology impossible.

We faced a tough choice between two awkward options. Option (i) was to abandon
the VRD dataset and continue searching for an image dataset with a matching
ontology, whilst being uncertain as to our prospects for success at finding
such a resource. Option (ii) was to commit to investing the effort required to
customise the VRD annotations so as to resolve the many issues uncovered by
our deep analysis. At a minimum, this effort would need to be sufficient to
resolve problem categories (1) and (3) to enable us to engineer a robust
companion ontology. This amount of effort might, however, also open the way to
addressing the issues associated with the other problem categories as well
with marginal incremental energy. We selected option (ii). This decision led
to the evolution of the resource that we call NeSy4VRD that is the subject of
this paper.

##  3 The NeSy4VRD dataset

The NeSy4VRD image dataset is a substantially-improved version of the VRD
image dataset [13] introduced in 2016. The images of NeSy4VRD are the same as
those of the original VRD dataset, but the NeSy4VRD visual relationship
annotations for these images are a massively quality-improved version of the
original VRD annotations.

###  3.1 The images of the NeSy4VRD dataset

The full VRD dataset stopped being publicly available around late 2021. Given
the popularity it has enjoyed in the past, it is very likely to be missed,
despite its many serious (but, likely, largely unknown) shortcomings
(discussed previously). The original (problematic) VRD visual relationship
annotations are still publicly available (through the website associated with
[13]), but the VRD images themselves are no longer accessible from the URL
specified in the documentation (readme.txt file) accompanying those
annotations.

We recently contacted Dr. Ranjay Krishna, one of the principal originators of
the VRD dataset and principal authors of [13], about this matter. Dr. Krishna
has graciously granted us permission to host the VRD images ourselves so as to
re-establish their public availability as part of NeSy4VRD. Appropriate
attribution for the VRD images is given as part of our NeSy4VRD dataset which
is registered on Zenodo, at https://doi.org/10.5281/zenodo.7916355.

###  3.2 The NeSy4VRD visual relationship annotations

NeSy4VRD has comprehensively addressed and resolved the many serious
shortcomings of the original VRD visual relationship annotations, outlined
above in Section 2.3. As already mentioned, the prime motivation for resolving
these shortcomings was to open the way for us to engineer a robust and
credible OWL ontology as a well-aligned companion to the NeSy4VRD dataset.
This objective was achieved, and the NeSy4VRD VRD-World OWL ontology is the
outcome of that effort.

To enable us to undertake this comprehensive customisation and quality-
improvement exercise in a responsible fashion, we developed the NeSy4VRD
protocol (for specifying visual relationship annotation customisations
declaratively, in text files) and the NeSy4VRD workflow (for applying these
customisations in a managed, automated and repeatable process). These
components, the NeSy4VRD protocol and workflow, are discussed shortly, in
Section 6, as part of the explanation of the support that exists within
NeSy4VRD for extensibility of the visual relationship annotations and,
thereby, of the VRD-World ontology as well.

As will become clear in Section 6, the NeSy4VRD protocol allows one to specify
instructions to change existing annotations, remove unwanted annotations, and
add new visual relationship annotations for any VRD image. The visual
relationship annotation customisation instructions we specified using the
NeSy4VRD protocol applied adjustments (changes, removals and additions) to the
annotations of 1,715 of the 4,000 training images and to 828 of the 1,000 test
images, for a total of 2,543 (just over half) of the 5,000 VRD images.

Table 1 compares the NeSy4VRD visual relationship annotations with the
original VRD annotations in terms of various salient statistics. It shows the
effect of the extensive annotation customisation exercise we undertook that
can be conveyed quantitatively. The table indicates that the NeSy4VRD
annotations have more object classes, more predicates, more overall visual
relationship annotations, and greater average annotations per image. Most of
the (arguably, more valuable) quality improvements implemented as a result of
our comprehensive annotation customisation exercise cannot be conveyed
quantitatively in a table and remain implicit, but should not go
unappreciated.

Table 1: Quantitative comparison of the NeSy4VRD and VRD visual relationship annotations. ‘Visual relationship’ has been abbreviated with the acronym VR. Metric | NeSy4VRD | VRD
---|---|---
Object classes | 109 | 100
Predicates | 71 | 70
Number of training set VR annotations | 29,333 | 30,355
Number of test set VR annotations | 9,201 | 7,638
Total number of VR annotations | 38,534 | 37,993
Average VR annotations per training image | 7.8 | 7.6
Average VR annotations per test image | 9.9 | 7.6
Number of training images with duplicate VRs | 0 | 323
Number of test images with duplicate VRs | 0 | 91

##  4 The NeSy4VRD OWL Ontology: VRD-World

In this Section we describe the OWL ontology that we engineered to be a well-
aligned companion of the NeSy4VRD dataset. We call this ontology VRD-World
because it directly describes the domain of the NeSy4VRD dataset as reflected
in the 109 object classes and 71 predicates referenced in the NeSy4VRD visual
relationship annotations of the VRD images. The number and diversity of the
object classes and predicates made it feasible to design the VRD-World
ontology so as to have a reasonably rich class and property hierarchies which
offer good opportunity for OWL reasoning capabilities to be meaningfully
exercised. Table 2 gives a summary, quantitative view of the VRD-World
ontology in terms of key metrics.

Table 2: Summary metrics for the VRD-World ontology

Summary Metric | Count
---|---
Axiom | 815
Logical axioms | 433
Declaration axioms | 322
Classes | 239
Object properties | 74
Data properties | 4
Annotation properties | 8

Class axioms | Count
---|---
SubClassOf | 242
EquivalentClasses | 21
Object property axioms |
SubObjectPropertyOf | 46
EquivalentObjectProperties | 7
InverseObjectProperties | 6
TransitiveObjectProperties | 17
SymmetricObjectProperties | 8

Figure 2: A portion of the VRD-World class hierarchy.

Class hierarchy. Figure 2 gives a pictorial representation of a small portion
of the VRD-World class hierarchy. Names situated at the top edge of a bubble
(like Personal Accessory) represent class names that we introduced as part of
organising the NeSy4VRD object classes into a reasonable hierarchy. Names
situated inside bubbles (like sunglasses and watch) are NeSy4VRD object
classes, and the fact of being ‘inside’ represents an implicit rdfs:subClassOf
relationship. All dotted lines represent explicit rdfs:subClassOf
relationships.

Property hierarchy. Figure 3 gives a pictorial representation of two small
portions of the VRD-World object property hierarchy. All names surrounded in
dark blue are NeSy4VRD predicate names that have directly corresponding object
properties in the VRD-World ontology. Dotted lines represent
rdfs:subPropertyOf relationships and dark, double-headed arrows represent
owl:equivalentClass relationships. Several owl:inverseOf relationships exist
but are not represented in the diagram to minimise clutter. For example,
properties above and below are inverses, as are over and under.

Figure 3: Two portions of the VRD-World object property hierarchy. Figure 4:
Two object properties of VRD-World.

Property descriptions and characteristics. Figure 4 shows two object
properties of VRD-World as they appear in the Protégé ontology editor. Most of
the characteristics they possess are reflected in the summary metrics
displayed in Table 2. These examples help to justify the claim made earlier
that VRD-World is a rich ontology capable of exercising OWL reasoning
capabilities meaningfully.

##  5 Infrastructure to load and extract annotations

The VRD-World ontology describes more of the NeSy4VRD domain than just the
object classes and predicates referenced in the annotations: it describes
conceptual infrastructure sufficient to capture the NeSy4VRD visual
relationship annotations in their entirety, including representations of the
images (labelled with their unique filenames) and the bounding box
specifications of the objects. As in the annotations, in the ontology each
object (with its bounding box) is anchored to a unique image (via an inverse-
functional :hasObject object property). So, in a knowledge graph hosting the
VRD-World ontology, each data instance representing a VRD image is effectively
the root of a distinct subgraph and can be leveraged in that capacity.

This feature of VRD-World permits users to do things such as (i) load the
NeSy4VRD visual relationship annotations into a knowledge graph (in their
entirety, or for a single image), (ii) materialise the knowledge graph (by
appropriate means) to infer new visual relationships, and (iii) extract an
augmented set of visual relationship annotations, in their native format.
NeSy4VRD provides sample code for performing these load and extract tasks
using the well-known Python package RDFLib. The rationale for this is to
further lower the barrier to entry to NeSy research using OWL-based knowledge
graphs for researchers who may be tempted to pursue it.

##  6 NeSy4VRD support for extensibility

In this Section we describe the components of NeSy4VRD that together provide
comprehensive support for the extensibility of the NeSy4VRD visual
relationship annotations and, thereby, of the VRD-World ontology as well. The
NeSy4VRD visual relationship annotations and companion VRD-World OWL ontology
can of course be used as is. This is what we do in our NeSy research using
OWL-based knowledge graphs. But these two aspects of NeSy4VRD have, from the
outset, been regarded by us as defaults. We anticipate that, oftentimes,
NeSy4VRD may be perceived by researchers as being close to what they are
looking for in a research resource, but that their particular research needs
and interests are such that the NeSy4VRD visual relationship annotations and
companion OWL ontology, VRD-World, require some amount of tailoring in order
that they suit the particularities of their research needs sufficiently well.
The rationale for NeSy4VRD providing the extensibility-supporting components
that we describe here is to lower the barrier to performing such tailoring,
should it be required.

###  6.1 Comprehensive code for dataset analysis

A prerequisite for customising the NeSy4VRD visual relationship annotations in
a systematic way is being able to thoroughly analyse them, and in conjunction
with their associated VRD images. NeSy4VRD provides comprehensive Python code
for doing just this. This is the same code we developed for undertaking our
own deep analysis of the VRD images and original VRD visual relationship
annotations, and which led to the discovery of the many issues and
shortcomings latent in those annotations, as detailed in Section 2.3.

We list a small sampling of the basic functionality available in order to give
a flavour of the analytic facilities NeSy4VRD provides:

  * •

display an image, showing the objects annotated for: (i) one particular visual
relationship (VR), (ii) a particular subset of VRs, (iii) all VRs;

  * •

print all VRs for an image: (i) in readable (s, p, o) form, (ii) in raw form;

  * •

find all images having VRs for: (i) object class X or object classes
[X,Y,...], (ii) a particular predicate, (iii) a particular VR pattern, such as
(s, p, X), (X, p, o), (s, X, o), (s, p, o), and list the distinct values for
X;

  * •

find all images with a target number of VRs

  * •

distributional analyses: (i) number of VRs per image, (ii) number of distinct
object classes per image, (iii) number of distinct predicates per image, etc.;

  * •

quality verification analyses: find images with (i) duplicate VRs, (ii)
degenerate bounding boxes, (iii) bounding boxes assigned multiple object
classes, etc.

###  6.2 The NeSy4VRD protocol

To enable large volumes of visual relationship annotation customisations to be
applied safely and using an automated, repeatable mechanism, NeSy4VRD uses a
custom protocol so that they can be specified declaratively, in text files. We
call this the NeSy4VRD protocol. Listing 1 shows some representative
customisation instructions associated to five images and the different
instruction types defined for the protocol.

Listing 1: Example NeSy4VRD protocol annotation customisation instructions.

⬇

imname; 3223670633_7d3d72dfe8_b.jpg

cvrsoc; 4; (‘person’, ‘on’, ‘shelf’); speaker

cvrsbb; 4; (‘speaker’, ‘on’, ‘shelf’); [161,234,231,270]

imname; 8934043045_251b42d19a_b.jpg

cvrooc; 7; (‘bus’, ‘beside’, ‘car’); truck

cvrobb; 7; (‘bus’, ‘beside’, ‘truck’); [334,557,99,403]

imname; 1426904233_ee344879b6_b.jpg

cvrsoc; 5; (‘bear’, ‘sit on’, ‘basket’); teddy bear

cvrpxx; 5; (‘teddy bear’, ‘sit on’, ‘basket’); in

imname; 4929276486_ca06aedbb9_b.jpg

rvrxxx; 4; (‘person’, ‘wear’, ‘jacket’);

avrxxx; boat; [477,594,319,746]; has; dog; [478,529,587,618]

avrxxx; boat; [477,594,319,746]; carry; dog; [478,529,587,618]

imname; 7171463996_900cb4ce33_b.jpg; rimxxx

For the first image in the listing, an improvement to the visual relationship
annotation at index 4 (in the image’s list of annotations) is being specified.
The cvrsoc instruction declares an intention to change the subject’s object
class (soc) from person to speaker, (as in ‘stereo speaker’). The cvrsbb
instruction declares an intention to change the subject’s bounding box (sbb)
to a more accurate localisation of the object.

For the second image, similar types of changes are being specified, but this
time for the visual relationship at index 7 and in relation to the object’s
object class and bounding box (cvrooc and cvrobb).

For the third image, an improvement to the visual relationship at index 5 is
being specified. After a cvrsoc instruction specifying that the subject’s
object class be changed from bear to teddy bear, a cvrpxx instruction declares
an intention to change the predicate (pxx) from sit on to in.

For the fourth image, the rvrxxx instruction declares an intention to remove
the visual relationship at index 4 (because, say, it was found to be too badly
broken or to be a near or exact duplicate of another annotation for the same
image). The two avrxxx instructions declare intentions to add two new visual
relationships to the set of annotations for the image.

Finally, the rimxxx instruction following the name of the fifth image in the
listing declares an intention to have the entry for that image removed from
the annotations dictionary (because the image and its annotations were found
to be highly problematic and not recoverable).

A dedicated Python script processes an annotation customisation instruction
text file sequentially, interprets the protocol, validates the construction of
the annotation customisation instructions, and executes them. If an error is
detected, execution is aborted and the cause of the problem and the line
number of the offending instruction are reported to the user. We used GIMP
(gimp.org), the free and open source image editor, to determine all bounding
boxes specified in our annotation customisation instructions. But users of
NeSy4VRD can of course select whichever tool they prefer for this subtask.

###  6.3 The NeSy4VRD workflow

The NeSy4VRD workflow is a set of Python modules and scripts that implement a
configurable, multi-step sequential process for applying one’s planned and
pre-specified visual relationship annotation customisations in a configurable,
automated and repeatable manner. Each sequential step of the workflow relates
to discrete category of annotation customisation and is performed by a
dedicated Python script designed for that task. The data governing the precise
annotation customisations applied by each sequential step in a particular
execution of the workflow are defined in a Python configuration module using
step-specific, predetermined variable names and formats. Each step (script) of
the workflow imports this configuration module to access the variables it
needs in order to execute properly. There are two such configuration modules:
one for managing the running of the workflow to apply customisations to the
annotations of the VRD training images, and one for the test images. The
precise set of workflow steps needed when running the workflow against the
training annotations and test annotations need not be identical. The
responsibilities of the successive steps of the NeSy4VRD worflow that we used
for applying our NeSy4VRD customisations to the original VRD annotations are
as follows:

  1. 1.

change existing and/or add new object class or predicate names to the
respective master lists that define these names;

  2. 2.

apply the annotation customisation instructions that have been specified in a
particular text file using the NeSy4VRD protocol;

  3. 3.

for a specified set of images, change all instances of object class X to
object class Y; (more than one set of images can be processed in this way);

  4. 4.

merge all instances of object class X into object class Y; merge all instances
of predicate X into predicate Y; (multiple such pairs can be processed);

  5. 5.

remove all instances of specified VR types, globally;

  6. 6.

remove all image entries from the annotations dictionary with zero VRs;

  7. 7.

apply the annotation customisation instructions that have been specified in a
particular text file using the NeSy4VRD protocol;

  8. 8.

apply the annotation customisation instructions that have been specified in a
particular text file using the NeSy4VRD protocol;

  9. 9.

change VR type X to type Y, globally; (multiple such pairs can be processed;
restrictive conditions apply);

  10. 10.

find images with duplicate VRs and remove the duplicates, globally;

  11. 11.

apply the annotation customisation instructions that have been specified in a
particular text file using the NeSy4VRD protocol.

##  7 NeSy4VRD beneficiaries and use cases

In this Section we briefly highlight a few intended and potential NeSy4VRD
beneficiaries and use cases. NeSy4VRD has a real prospect of being attractive
to diverse user groups. The primary users of NeSy4VRD are expected to be:

  * •

computer vision researchers interested in using NeSy4VRD as a quality-improved
version of the (now partly unavailable) VRD dataset, but for deep learning
alone, ignoring the companion VRD-World OWL ontology;

  * •

NeSy researchers not using Semantic Web technologies but who nonetheless find
NeSy4VRD attractive for NeSy research purposes, for reasons such as those
outlined in Section 2.2 (small size, and zero/few-shot learning);

  * •

NeSy researchers using Semantic Web technologies such as OWL ontologies and
OWL-based knowledge graphs who are looking for a dataset with a well-aligned,
companion OWL ontology to enable them to pursue their research vision.

The paramount intended use case for NeSy4VRD is vision-based NeSy research
that relies on leveraging OWL ontologies in some way. The manner in which the
NeSy4VRD dataset and its companion VRD-World OWL ontology might be leveraged
in this context is limited only by the imaginations of NeSy researchers. The
literature of NeSy research using Semantic Web technologies has examples of
OWL ontologies being leveraged from a structural perspective (irrespective of
their ability to drive deductive reasoning) to enhance image classification.
One instance of this is [4], which uses embeddings of ontologies to enhance
zero-shot learning in image classification. Our own research has a particular
focus on exploring how OWL and Datalog knowledge graph reasoning, driven and
governed by our VRD-World ontology, can be leveraged to enhance deep learning
for visual relationship detection [5]. NeSy4VRD and its VRD-World ontology are
also well positioned to participate in the emerging NeSy subfield of symbolic
knowledge injection/infusion [10, 14]

Another potential category of use case for NeSy4VRD is as a standard or
benchmark resource for vision-based research, especially for NeSy research
using OWL ontologies and knowledge graphs. The popularity of the VRD dataset
suggests it became something of a standard or benchmark dataset for the
application tasks of visual relationship detection and scene graph generation.
Given that NeSy4VRD is a quality-improved version of the (no longer fully
available) VRD dataset, it is entirely plausible that NeSy4VRD inherits that
role. And given the scarcity of research resources like NeSy4VRD that target
the specialised needs of NeSy researchers wishing to use OWL ontologies and
knowledge graphs, NeSy4VRD has strong potential for becoming a standard or
benchmark resource for that user group in particular. Finally, given its
comprehensive support for extensibility, it is conceivable that the default
NeSy4VRD visual relationship annotations and companion VRD-World OWL ontology
give rise to an ever-growing family of unique but strongly related standard
dataset resources.

##  8 Concluding remarks

NeSy4VRD is a multifaceted, multipurpose research resource. It has an image
dataset for visual relationship detection accompanied by a well-aligned OWL
ontology describing the dataset domain. It provides comprehensive support for
extensibility of the image annotations and, thereby, of the OWL ontology. And
it provides sample code for loading the annotations to/from a knowledge graph.

NeSy4VRD makes the VRD images available again and in conjunction with a
massively quality-improved version of the VRD visual relationship annotations.
Like the VRD dataset, NeSy4VRD has characteristics (small size, and plentiful
zero/few-shot learning data conditions) that are likely to be especially
attractive for NeSy researchers. It specifically addresses the special needs
of NeSy researchers wishing to use OWL ontologies and knowledge graphs as
symbolic background knowledge and reasoning components for whom appropriate
research resources are particularly scarce. NeSy4VRD has strong prospects for
becoming a standard resource for diverse user groups and we are pleased to
contribute it to the computer vision, NeSy and Semantic Web communities. In so
doing, we particularly hope to foster more NeSy research using OWL-based
knowledge graphs.

### Acknowledgements

We would like to thank Dr. Ranjay Krishna for granting us permission to re-
establish the public availability of the VRD images as part of NeSy4VRD. This
research was partially funded by the SIRIUS Centre for Scalable Data Access
(Research Council of Norway, project 237889).

## References

  * [1] Besold, T.R., d’Avila Garcez, A.S., Bader, S., Bowman, H., Domingos, P.M., Hitzler, P., Kühnberger, K.U., Lamb, L., Lowd, D., Lima, P.M.V., de Penning, L., Pinkas, G., Poon, H., Zaverucha, G.: Neural-Symbolic Learning and Reasoning: A Survey and Interpretation. CoRR (2017), https://arxiv.org/abs/1711.03902
  * [2] Breit, A., Waltersdorfer, L., Ekaputra, F.J., Sabou, M., Ekelhart, A., Iana, A., Paulheim, H., Portisch, J., Revenko, A., Teije, A.t., van Harmelen, F.: Combining Machine Learning and Semantic Web: A Systematic Mapping Study. ACM Computing Surveys (mar 2023), https://doi.org/10.1145/3586163, just accepted
  * [3] d’Avila Garcez, A., Lamb, L.C.: Neurosymbolic AI: The 3rd Wave. CoRR abs/2012.05876 (2020), https://arxiv.org/abs/2012.05876
  * [4] Geng, Y., Chen, J., Chen, Z., Pan, J.Z., Ye, Z., Yuan, Z., Jia, Y., Chen, H.: OntoZSL: Ontology-enhanced Zero-shot Learning. In: Leskovec, J., Grobelnik, M., Najork, M., Tang, J., Zia, L. (eds.) WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021. pp. 3325–3336. ACM / IW3C2 (2021), https://doi.org/10.1145/3442381.3450042
  * [5] Herron, D.: Visual Relationship Detection using Knowledge Graphs for Neural-Symbolic AI. In: Proceedings of the Doctoral Consortium at ISWC. vol. 3165. CEUR Workshop Proceedings (2022), https://ceur-ws.org/Vol-3165/
  * [6] Herron, D., Jiménez-Ruiz, E., Weyde, T.: On the Benefits of OWL-based Knowledge Graphs for Neural-Symbolic Systems. In: NeSy 2023: 17th International Workshop on Neural-Symbolic Learning and Reasoning, July 3–5, 2023, Siena, Italy. CEUR Workshop Proceedings, CEUR-WS.org (2023), (Accepted paper)
  * [7] Hitzler, P.: Knowledge Graphs in Neuro-Symbolic AI. Virtual Workshop at the 3rd Conference on Automated Knowledge Base Construction (AKBC 2021), October 8 (2021), https://people.cs.ksu.edu/~hitzler/pub2/2021-10-AKBC.pdf, (Presentation slides)
  * [8] Hitzler, P.: A Review of the Semantic Web Field. Commun. ACM 64(2), 76–83 (2021), https://doi.org/10.1145/3397512
  * [9] Hitzler, P., Bianchi, F., Ebrahimi, M., Sarker, M.K.: Neural-Symbolic Integration and the Semantic Web. Semantic Web 11(1), 3–11 (2020)
  * [10] Khan, M.J., Breslin, J.G., Curry, E.: Common Sense Knowledge Infusion for Visual Understanding and Reasoning: Approaches, Challenges, and Applications. IEEE Internet Comput. 26(4), 21–27 (2022), https://doi.org/10.1109/MIC.2022.3176500
  * [11] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L., Shamma, D.A., Bernstein, M.S., Fei-Fei, L.: Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. CoRR (2016), http://arxiv.org/abs/1602.07332
  * [12] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J.R.R., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., Duerig, T., Ferrari, V.: The Open Images Dataset V4. Int. J. Comput. Vis. 128(7), 1956–1981 (2020), https://doi.org/10.1007/s11263-020-01316-z
  * [13] Lu, C., Krishna, R., Bernstein, M., Fei-Fei, L.: Visual Relationship Detection with Language Priors. In: European Conference on Computer Vision. pp. 852–869. Springer (2016), https://cs.stanford.edu/people/ranjaykrishna/vrd/
  * [14] Magnini, M., Ciatto, G., Omicini, A.: On the Design of PSyKI: a Platform for Symbolic Knowledge Injection into Sub-Symbolic Predictors. In: Proceedings of the 4th International Workshop on EXplainable and TRAnsparent AI and Multi-Agent Systems. Springer (2022)
  * [15] Marcus, G.: The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence. CoRR (2020), https://arxiv.org/abs/2002.06177
  * [16] Ronchi, M.R., Perona, P.: Describing Common Human Visual Actions in Images. In: Xie, X., Jones, M.W., Tam, G.K.L. (eds.) Proceedings of the British Machine Vision Conference (BMVC). pp. 52.1–52.12. BMVA Press (2015), http://www.vision.caltech.edu/~mronchi/projects/Cocoa/, (COCO-a)
  * [17] Zhu, G., Zhang, L., Jiang, Y., Dang, Y., Hou, H., Shen, P., Feng, M., Zhao, X., Miao, Q., Shah, S.A.A., Bennamoun, M.: Scene Graph Generation: A Comprehensive Survey. CoRR (2022), https://arxiv.org/abs/2201.00443