# Wider Vision: Enriching Convolutional Neural Networks via Alignment to
External Knowledge Bases

Xuehao Liu  Technological University DublinDublinIreland
<EMAIL_ADDRESS>,  Sarah Jane Delany  Technological University
DublinDublinIreland<EMAIL_ADDRESS>and  Susan McKeever
Technological University DublinDublinIreland<EMAIL_ADDRESS>
(2021)

###### Abstract.

Deep learning models suffer from opaqueness. For Convolutional Neural Networks
(CNNs), current research strategies for explaining models focus on the target
classes within the associated training dataset. As a result, the understanding
of hidden feature map activations is limited by the discriminative knowledge
gleaned during training. The aim of our work is to explain and expand CNNs
models via the mirroring or alignment of the network to an external knowledge
base. This will allow us to give a semantic context or label for each visual
feature. Using the resultant aligned embedding space, we can match CNN feature
activations to nodes in our external knowledge base. This supports knowledge-
based interpretation of the features associated with model decisions.

To demonstrate our approach, we build two separate graphs from: (1) ConceptNet
knowledge base and (2) a public CNN. We use an entity alignment method to
align the feature nodes in the CNN with the nodes in the ConceptNet based
knowledge graph. We then measure the proximity of CNN graph nodes to
semantically meaningful knowledge base nodes. Our results show that in the
aligned embedding space, nodes from the knowledge graph are close to the CNN
feature nodes that have similar meanings, indicating that nodes from an
external knowledge base can act as explanatory semantic references for
features in the model. We analyse a variety of graph building methods in order
to improve the results from our embedding space. We further demonstrate that
by using hierarchical relationships from our external knowledge base, we can
locate new unseen classes outside the CNN training set in our embeddings space
based on visual feature activations. This suggests that we can adapt our
approach to identify unseen classes based on CNN feature activations. Our
demonstrated approach of aligning a CNN with an external knowledge base paves
the way to reason about and beyond the trained model, with future adaptations
to explainable models and zero-shot learning.

Knowledge-enhanced Reasoning, Explainable AI, Deep learning, Knowledge graphs

††journalyear: 2021††copyright: rightsretained††conference: The 1st
International Workshop on Machine Reasoning; March 12, 2021; Jerusalem,
Israel††booktitle: Proceedings of the 1st International Workshop on Machine
Reasoning (MRC’21), March 12, 2021, Jerusalem, Israel

##  1\. Introduction

Deep Learning models are considered to be black boxes, with a lack of
visibility of layers and decisions paths (Castelvecchi, 2016). With the
increasing complexity of models, research has focused on providing
explanations to explain the relationship between model inputs and outputs
(Arrieta et al., 2020; Gilpin et al., 2018). Explanations can be used for
several purposes. One purpose is to underpin the fine-tuning and achievement
of higher accuracies (Kim et al., 2016; Koh and Liang, 2017). Another is to
help researchers identify why an image input has been incorrectly interpreted
(Selvaraju et al., 2017). Other work in the field has shown that different
layers of the CNNs interpret different levels of visual features within the
image (Simonyan et al., 2013; Zeiler and Fergus, 2014; Springenberg et al.,
2014; Gatys et al., 2016). Current explanations of CNNs that interpret
predictions are limited to interpreting the relationship between inputs,
feature activation and outputs generated from the training dataset. There is
no additional insight or knowledge available outside the knowledge embedded in
the training dataset. Selvaraju et al. (2017) highlighted the areas of
features that make most contribution to the output label. However, this work
was unable to show the reason why these areas are combined together. Bau et
al. (2017) involved more concepts to explain features that CNNs are detecting,
such as textures, colours, and scenes. They showed that specific hidden units
in the network are detecting particular semantically meaningful concepts.
However, a richer explanation for features from a wider pool of knowledge
beyond the training set is still missing. CNN explanations with no external
knowledge are able to indicate which individual features are activated for
outputs, but will be unaware of the semantic concepts behind these features,
or indeed, how they relate to other outputs. A richer knowledge driven
explanation (akin to a human who has real world knowledge) could show, for
example, that ear, paw and tail features were appearing in a particular dog
image; that dogs generally have such features, and indeed further reason that
these features may appear for other previous unseen animal types to help
identify them. Mitsuhara et al. (2019) use attention maps and manual image
labelling in the direction of such fine-tuned explanations. However, manual
labelling is not practical for large image datasets.

Inspired by the work of Lecue (2019), our approach is based upon aligning a
CNN to an external knowledge base. We hypothesise that by aligning a trained
CNN model to an external knowledge base, we can discover semantically
meaningful labels for activated CNN features. We also hypothesise that we can
use the alignment process to identify an unseen output (i.e. a class not
included in the training data) based on CNN feature activations.

We represent the CNN and the external knowledge base as two separate graph
structures. The premise is that outputs in the CNN (e.g. dog) and their
associated activated features align with concepts in the knowledge graph (i.e.
dog) and its associated connected nodes (e.g. tail, paws, ears). We then align
these two graphs, to create a shared embedding space. We can then locate
labels for CNN feature activations in the external knowledge base, via
distance measurement in the embedding space. This labelling supports an
explanation of a CNN decision. It also paves the way for detecting unseen
classes in our CNN, via identification from activated features mapped to the
external knowledge base.

We demonstrate and test our approach using the external knowledge base,
ConceptNet (Speer et al., 2017). For the CNN as a graph, we use VGG-16
(Simonyan and Zisserman, 2014; Zeiler and Fergus, 2014) where the feature
layers represent visual features. We align features of the CNN and entities in
the knowledge graph using an entity alignment method, using the GCN-Align
method (Wang et al., 2018a).

Our evaluation focuses on two parts (1) whether we have correctly aligned the
CNN and knowledge graph and (2) the effectiveness of our embedding space for
labelling CNN features. The remainder of the paper is structured into Related
Work, our Approach, explained in detail, and our Experimental evaluation and
results. In our conclusion, we cover possible future work directions.

##  2\. Related work

The closest domain to our work is Explainable AI, as we are aiming to discover
the meanings of CNN features. For CNNs, explainable AI techniques started with
intuiting what features are being detected in an image. One way of visualizing
features is to trace the gradients back to the input image (Zeiler and Fergus,
2014; Springenberg et al., 2014). The resultant highlighting contour on the
input image shows the pattern that the filter is detecting. It also shows that
later layers closer to the output will detect higher object level patterns,
while earlier network layers detect features that are more basic, such as
textures. Tracing back the gradients not only provides a visualization of
features, but also highlights the part of an image that leads to the CNN
prediction, such as Deconvolution (Zeiler and Fergus, 2014; Simonyan et al.,
2013), Layer-Wise Relevance Propagation (LRP) (Bach et al., 2015),
DeepSHAP(Lundberg and Lee, 2017), and Grad-CAM (Selvaraju et al., 2017).
Another explanation approach is to generate an image from noise, and the image
maximizes the prediction score (Mordvintsev et al., 2015; Simonyan et al.,
2013). Although such images may not resemble any meaningful object, the
patterns found by the CNNs are informative. Generally, while the visualisation
of detected features can highlight what and where CNNs have focused in an
image, the meaning of the feature is still unknown until it is interpreted
further. Bau et al. (2017) advanced this interpretation via labelled bounding
boxes around concepts within the dataset images. By comparing the activated
area of a feature with the ground truth areas, activated features can be
identified as specific concepts within an image.

These explanation approaches can only find explanations based on knowledge
gleaned from the training dataset. To find an explanation in a broader
knowledge space, we combine the CNN with a knowledge graph in order to assign
and enhance feature meanings. We note that this approach of bringing knowledge
graphs into machine learning systems in order to expand knowledge beyond the
training set is gaining traction in the machine learning research domain - and
is applied to both image classification and zero-shot learning. For image
classification, Marino et al. (2016) used labels of the COCO dataset (Lin et
al., 2014) to build a knowledge graph. Both images and concepts of the COCO
dataset are from the lexical database, WordNet (Miller, 1995). They applied a
Graph Search Neural Network(GSNN) to do reasoning through the graph, and thus
improved mean average precision. In zero-shot learning, Lee et al. (2018) also
built a knowledge graph using WordNet. Using the features of ResNet (He et
al., 2016) as input, they used a GSNN to infer unknown classes. Wang et al.
(2018b) used WordNet, Never-Ever-Language-Learning (NELL) (Carlson et al.,
2010) and Never-Ever-Image-Learning (NEIL) (Chen et al., 2013) to build a
knowledge graph, where this graph takes Glove (Pennington et al., 2014) word
embeddings as input. They applied a Graph Convolutional Network(GCN) on the
graph to generate the weight of the last layer of ResNet, so that this updated
ResNet model can then recognise the nodes in the knowledge graph. In Zero-shot
Action Recognition, Gao et al. (2019) used ConceptNet (Speer et al., 2017) to
build a knowledge graph. In their approach, both Word2vec (Mikolov et al.,
2013) and GoogleNet (Szegedy et al., 2015) features are used in parallel as
input. Two-Stream Graph Convolutional Network (TS-GCN) is applied on the graph
to recognize actions.
The above research indicates that using knowledge graphs as external
assistants can benefit and expand the knowledge of current deep learning
systems. In our case, where we are linking a CNN to an external knowledge base
for feature labelling and network insights, We require  entity alignment
methods to correctly map our CNN to the knowledge base. Entity alignment is
about aligning nodes in two separate graphs that have the same semantic
meaning. The most intuitive way of matching two entities is to compare the
traditional language features (Akbari et al., 2009), such as n-grams and TF-
IDF(Sun et al., 2015). Recently, with the help of node embedding (Bordes et
al., 2013), entity alignment has been done by minimizing the distance between
embeddings of labelled nodes with same meanings from two separate graphs (Hao
et al., 2016). In addition, embeddings of the attributes for each node (from
the graph) can be also used for minimizing distance between two nodes
(Trisedya et al., 2019; Sun et al., 2017). Sun et al. (2018) introduced a
bootstrapping strategy to consider nodes that have a distance small enough as
the labelled nodes when more training steps are made. The bootstrapping
strategy provided a better alignment accuracy on Hit@1 score. All of these
methods can align nodes in two graphs. We apply GCN-Align(Wang et al., 2018a)
as our Entity Alignment method as it could do alignment only using structure
information such as neighbours and relationships.

##  3\. Approach

imageVGGoutput(a) Starting point Knowledge
---
base
CNN(b) Graph Building Knowledge
---
graph
CNN
---
graph
(c) Entity Alignment Knowledge
---
graph
CNN
---
graph
(d) Nearest neighbour findingEmbedding space Figure 1. Overall diagram for our
approach

An overview of our approach is shown in Fig.1. First, we build two graphs from
the knowledge base and the CNN(Fig.1(a-b)). We then apply entity alignment by
aligning the labelled nodes across these two graphs (Fig.1(c)) that have the
same meaning. Finally, in the embedding space generated by the entity
alignment stage, we explore the knowledge graph nodes which are the nearest
neighbours of the CNN feature nodes with the goal of labelling the CNN feature
nodes, based on equivalent nodes aligned from the knowledge graph. The
following sections will introduce the details of each step.

###  3.1. Building a graph from a knowledge base

Knowledge bases are mainly stored as two kinds of triples: (head,
relationship, tail) and (entity, property, value). The first one represents
the relationships between entities, such as (leopard, IsA, cat) (Speer et al.,
2017). An example of the second kind is (Texas, areaTotal, “696241.0”) (Sun et
al., 2017). For our purposes, we use the first kind of triple, which describes
relationships explicitly. Triples can have associated weights which represents
the strength of the relationship between the head and the tail entities. To
build a graph from the knowledge base, entities in the knowledge base become
nodes in the graph. Since the knowledge base is very large as we are using
ConceptNet, we will use a subset. The starting points of building the
knowledge graph are the entities from the knowledge base that have the same
meaning as the CNN outputs (i.e. classes). Using a breadth-first search
strategy through the knowledge base, we then add in other entities that are
directly connected to these entities(i.e. one-hop) that have a specific
relationship and weight. These knowledge graph outputs nodes are connected to
the nodes from the knowledge base that had the largest triple relationship
weights. There are many kinds of triple relationships (such as IsA, HasA). In
an ideal knowledge base, we would aim to include only those triples that are
relevant to visual features. As we do not have this kind of knowledge base, we
use ConceptNet (Speer et al., 2017), which is an existing knowledge base and
adjust it by selecting relationships that are more likely to map to visual
features. For instances, triple (A cat, HasA, a nose) is more related to
visual information than triple (a cat, Desires, meow). We have explored the
results using “IsA” and “HasA’, but more experiments using different
combinations of relationships can be done in the future work.

###  3.2. Building a graph from the CNN

The visualization work of Zeiler and Fergus (2014) and other researchers have
already shown that in a CNN, the features in the earlier layers in the network
(close to the input) are detecting lower-level features. In a deep CNN such as
VGG-16, the features in CNN outline the contour of some part of the input
image (Zeiler and Fergus, 2014). The work of Grad-CAM(Selvaraju et al., 2017)
reveals that the area representing the shape of prediction is the weighted
combination of these CNN features. These works show that the prediction is
made according to the features that are detected. The purpose of this step is
to generate a graph from CNN that contains the outputs and the features that
will bring most contribution to the outputs (akin to components of the
output).The features in the higher network layers (close to output) are
combinations of lower level features with different weights. The visual
feature for building our CNN graph is from the flatten layer. The flatten
layer is the highest layer containing convolutional kernels, and has the
highest level features. After the flatten layer, the fully connected layer
combines the features in the flatten layer to detect the target classes.
However, information is also lost during this combination. Therefore, we use
the flatten layer features to build the CNN graph as it is a balance point of
selecting features that are between too low and too high. We let each feature
and each output be a node in the graph (Fig.2). We want to connect those most
relevant feature nodes with output nodes, to build a graph that the outputs
and the important features are connected with each other. When an image is
pushed into CNN, some features are activated. We want to choose the important
features among these activated features. There are several methods that we can
choose from to rank the importance of a feature for an output including
Pearson correlation (Benesty et al., 2009), DeConvolution, Guided Backprop
(Springenberg et al., 2014), LRP and DeepSHAP. We select Pearson correlation
because it is both intuitive and widely used for ranking associations. Other
methods can be tested in the future.

Step 1: Connect output nodes In the CNN side, the only thing we know are the
labels of outputs. In the knowledge graph, we can locate these output nodes (a
”dog” output in CNN will map to a ”dog” node in the knowledge graph).
Therefore, the CNN output nodes are the anchors to find meanings of visual
features. Before attempting to connect any CNN features to CNN output nodes,
we connect those CNN outputs to each other, where those equivalent output
nodes in the knowledge graph are directly connected to each other. This helps
the entity alignment process to locate at least these output nodes, and thus
enable the process to converge.

Step 2: Connect output nodes with visual features We then want to connect the
most important CNN feature nodes correctly to the CNN output nodes, by
identifying the subset of features that are most activated for each class. We
iterate through all the images in the dataset that trains the CNN(e.g.
ImageNet), and derive an average of the importance (correlation) of each
feature per output class. Now we know for each output node which features are
most important. For example, for all images containing dog, features detecting
tails and legs may have a higher activation. For each class, We average the
correlation for each feature through the whole ImageNet dataset for that
class. So, in the case of output node ’dog’, visual feature nodes related to
tails and legs likely be connected to that output node after averaging the
feature correlations. Fig.2 shows a diagram of this process, simplified to
show a subset of feature and output nodes. In Fig.2(a), the width of each line
represents the strength of correlation of a feature to an output node. On the
right side, the red nodes are the output nodes, and the blue ones are the CNN
features.

Visual
---
feature
output(a) CNN (b) CNN
---
graph
Figure 2. Diagram for building a graph from CNNs

###  3.3. Entity Alignment: aligning the CNN and knowledge graphs

Now we have our CNN graph and the knowledge graph. The process in the first
two steps of Fig.1 is finished. The CNN graph contains the output nodes (one
per label) and the most important features nodes associated with each output
node. The knowledge graph contains the equivalent output nodes as the CNN,
having been specifically selected from the knowledge base. The entity
alignment algorithm will take the two graphs as input and generate embeddings
for each nodes, so that all the nodes in the two graphs will be represented in
the same embedding space. Similar to word embeddings (Le and Mikolov, 2014),
the distance in the embedding space between each pair of nodes reflects the
semantic distance between them. For a node in a graph, and that node is
represented in the embedding space, the process of finding the corresponding
node from the second graph with similar meaning is to look for the closest
neighbour in the embedding space.

To align two graphs successfully, their structure must be similar. However,
the number of edges in the knowledge graph is significantly smaller than those
in the CNN graph. Therefore, we have to select a subset of the edges in the
CNN graph to match the number in the knowledge graph. There are two ways we
can approach this: (1) For a knowledge graph that has $k$ edges, we sort all
the correlations between visual features and output nodes in CNN, and take the
top $k$ correlations with the feature nodes into account. (2) For a CNN output
node $node_{i}$, there are $k_{i}$ visual features. We select the same number
of $k_{i}$ edges of $node_{i}$ in the knowledge graph in order to build two
graphs with similar structure.

Correlations may be positive or negative. Negative gradients (Selvaraju et
al., 2017) and correlations indicate that a particular feature is not
predictive of a current output node. Thus, we only consider positive
correlations, by matching the edge number($k_{i}$) of knowledge graph.

In order to create the embedding space during the alignment process, we need
to train GCN-Align with matching pairs of CNN/ knowledge graph output nodes.
The outputs classes (from the ImageNet dataset) in CNN form our CNN nodes, and
the equivalent concepts in Concept net form our knowledge graph nodes, to
provide us with 1000 node pairs. We select 90% of the pairs randomly as the
training set. The remaining 10% of the pairs will be the test set. The
training process is to minimize the distance between these 90% pairs of nodes.
The testing process is to check among those 10% pairs of nodes to test whether
in the generated embedding space, the output node is the closest to the
equivalent node in the knowledge base.

###  3.4. Evaluation approach

To explore our hypotheses in the introduction session, we examine the
embedding space and evaluate our feature labelling system in two ways.

Alignment performance Firstly, we evaluate the entity alignment process by
measuring whether we can align the equivalent output nodes across the
knowledge graph and the CNN graph. We use Hit@1 score to show the performance
of entity alignment, as used in other papers (Wang et al., 2018a; Xu et al.,
2019). In the embedding space, we select the nearest knowledge graph node to
each CNN output class node in the test set, based on Euclidean distance. The
Hit@1 measures the proportion of the CNN output nodes in the test set whose
closest neighbour is the corresponding knowledge graph node with the same
meaning.

Hit@k-nns chart Secondly, once we have established via the Hit@1 score that
equivalent output nodes (from the CNN and knowledge graphs) are located
together, the neighbourhood of a Knowledge Graph node should be fully
explored. We will use the closest knowledge base node to a CNN feature node to
label the CNN feature node. The feature labelling system is the embedding
space generated by the previous aligning step. A smaller distance between them
means the visual feature is more likely to detect that meaning. We introduce
the Hit@k-nns Chart. Here an independent image test dataset is used, which is
independent of the training set used to train the CNN. The target classes i.e.
output nodes are different from the dataset we used for training in Entity
Alignment. For each test image, Hit@k-nns Chart will explore $k$ nearest
visual feature neighbour nodes of the label of that image. The level of
activated visual nodes in these neighbours will reflect how well the embedding
space has captured the semantic meaning of visual features. Here is the steps
of plotting Hit@k-nns Chart:

  * •

A test image is passed through the pre-trained CNN.

  * •

Feature nodes that are activated for that image will be identified.

  * •

The set of k feature nodes that are the nearest neighbours of the target
output knowledge graph node for the test image will be identified.

  * •

The percentage of nearest neighbour feature nodes that are activated will be
measured. (i.e. Hit@k-NN score)

  * •

Both Hit@k-NN for a single image as k increases, or aggregated over a set of
test images, is plotted.

The Hit@k-NN score of a test image will be plotted in a graph where the x-axis
is the value $k$, which is the number of explored CNN feature nodes that are
nearest neighbours. The y-axis is the percentage of activated nodes. In a
perfect situation (as shown in Fig.4), all these nearest CNN feature nodes
should be activated. The shape of the curve should be high for low values of k
and dropping down as k increases. The perfect shape is drawn for the situation
that assuming that all the closest CNN feature nodes are activated. All other
not activated is not near the target output knowledge graph node.

Embedding space visualisation We also visualise the generated embedding space
to clearly show the placement of the nodes and the distances between CNN
feature nodes and knowledge graph nodes by mapping it to a two-dimensional
space using t-SNE (Maaten and Hinton, 2008). We integrated Deconvolution
(Simonyan et al., 2013) into the visualisation so that we display the contour
that the visual feature node is detecting. An example of this can be seen in
Figure 5. In the middle it is the t-SNE mapped embedding space. For each
visual feature, there is the visualisation in the grey box to the right of the
test images.

###  3.5. Experiment details

CNN backbone | VGG-16(ImageNet)
---|---
Importance measurement | Pearson Correlation
Hit@k chart test set | COCO dataset
Knowledge Base | ConceptNet 5.5
Entity Alignment method | GCN-Align
Table 1. Experiment details

This section outlines the implementation details of our experimental
evaluation.

We use the VGG-16 (Simonyan and Zisserman, 2014) model as our CNN. The version
of VGG-16 that we used is trained on the ImageNet dataset. In VGG-16 the
flatten layer is a 25088 dimensional vector and output is a 1000 dimensional
vector, as the VGG-16 used here is trained to distinguish between 1000
classes. We use the training set of ImageNet (Deng et al., 2009) to calculate
the Pearson Correlation between image features and output nodes to select
important features from the flatten layer for our CNN graph. In the entity
alignment process, typically a large number of nodes - up to 100,00 - are used
for training the alignment algorithm. (Sun et al., 2017; Wang et al., 2018a;
Zhu et al., 2020). CNNs, however, do not typically detect enough categories to
support this scale of entity alignment training. In our case, the entity
alignment process is driven by the number of CNN outputs nodes. The 1000
categories in the ImageNet datasets form the combined testing/ training set in
GCN-Align training process. To further test the embedding space generated by
GCN-Align, we use a separate image dataset, the COCO dataset (Lin et al.,
2014). We identified test images from COCO whose labels do not occur in the
training images in ImageNet and these images were used for testing.

The knowledge base we used for building the knowledge graph is ConceptNet 5.5
(Speer et al., 2017). We identified entities in ConceptNet that were exactly
the same as the 1000 categories of the ImageNet dataset. These output nodes
became the nodes of the knowledge graph built from ConceptNet. We also
identified an additional 62 categories from COCO which were the different from
the ImageNet categories and these entities were included as nodes in the
knowledge graph to allow us to use them at a later stage for testing. We
selected the “IsA” and “HasA” relationships, as we expect that these
relationships are more likely to be between visual features. We suggest that
the “IsA” relationship has the potential of an inference relationship between
nodes and may be useful for zero-shot Learning. The “HasA” relationship could
help us find the components of an object which may be useful in explanation.
We built two knowledge graphs for our evaluation, the first from the “IsA”
relationships with the output nodes, the second from the “HasA” relationships.

For entity alignment, we used GCN-Align (Wang et al., 2018a). This method is
purely based on the graph structure, which is suitable for our purposes. The
input to GCN-align does not need any word embeddings or attribute embeddings.
Table.1 shows details of experiments.

##  4\. Results and discussion

###  4.1. Alignment performance

(a) Hit@1 score of using the “IsA” relationship.

(b) Hit@1 score of using the “HasA” relationship

Figure 3. Hit@1 score comparison for three different graph building
approaches. (1) KG-to-CNN: Letting the knowledge graph match the edge number
of each node in the CNN graph. (2) CNN-to-KG: Letting the CNN graph match the
edge number of each node in the knowledge graph. (3) TOTAL: Do not match the
edge number of each node, and just match the total number of edges in two
graphs

Hit@1 score measures whether the closest neighbour of a CNN output node is the
corresponding knowledge graph node with the same meaning. Fig.3 shows the
align scores (Hit@1) based on three different approaches to building our two
graphs, depending on how the edges are added between nodes. The x-axis is the
percentage of total edges selected from the knowledge base and the y-axis is
the value of Hit@1 score. Edges are added 10% at a time in all cases. This
score measures the proportion of test set nodes from one graph that are the
closest node to the matching node from the other graph in the embedding space.

Here are the three ways of adding edges:

  * •

KGtoCNN starts with the nodes in the knowledge graph and at each iteration
adds 10% of the total number of KG edges to each CNN node. At the same time,
two corresponding nodes with same meaning in two graphs have the same edge
number.

  * •

CNNtoKG does the opposite starting with the CNN and at each iteration adding
10% of the total number of CNN edges to each KG node. Also, two corresponding
nodes with same meaning in two graphs have the same edge number.

  * •

TOTAL starts with the nodes in the knowledge graph and at each iteration adds
10% of the total number of KG edges to each CNN node, too. But TOTAL does not
keep the edge number of each pair of nodes same.

In Fig.3(a) (3) TOTAL shows how the Hit@1 score changes when we add the same
number of edges over the 1000 categories each time when building our CNN graph
and knowledge graphs (and where the knowledge graph is derived from extracting
the is-a relationships from ConceptNet). The Hit@1 score for adding 10% of
total number of edges is 43% which is the average performance achieved by GCN-
Align generally(Wang et al., 2018a). We observe that the performance is
dropping as we add more edges. Hit@1 score becomes lower than 10% after we add
40% of the edges. (1)KG-to-CNN has a different strategy of adding edges.We
still add 10% of the total edges each time. However, across all those 1000
output nodes, we let the number of edges from the knowledge graph node match
the number of edges in the corresponding node in CNN. For example, at the step
of 10%, on the CNN side, we select the top 28994 edges among all the 289944
edges. For example, the output node “bench” may have $k$ edges in this 28994
edges. On the knowledge graph side, we sort all the edges of “bench” in
descending order and select the top $k$ edges. If in the knowledge graph
“bench” does not have enough edges, we will select all of them. This process
ensures that every output node will have almost the same number of edges. The
Hit@1 score for KGtoCNN is more stable compared to (3)TOTAL. Unlike (1)KG-to-
CNN, which is making knowledge graph match CNN graph, (2)CNN-to-KG is letting
the CNN match the knowledge graph. The alignment score performance is similar.

Fig.3(b) shows the Hit@1 score of aligning the CNN graph with the knowledge
graph, but here the knowledge graph is created from the “HasA” relationships
in ConceptNet. Compared to the graph generated from “IsA” relationships, the
graph created from “HasA” relationships is more sparse. In total there are
13774 edges and 1263 nodes. The figure shows that entity alignment performance
as the number of edges increase is very similar to that of the “IsA”
relationships across the three different approaches to building the graphs -
although the performance using a “HasA” graph is slightly lower than using an
“IsA” graph. This suggests that the CNN graph can be aligned with knowledge
base graphs that are generated from different kinds of relationships.

###  4.2. Hit@k-nns chart

(a) Person(“IsA”)

(b) Truck(“IsA”)

(c) Person(“HasA”)

(d) Truck(“HasA”)

(e) The example image

Figure 4. Hit@k-nns charts of the example image. It has the label of “person”
and “truck”.

As we have a reasonable alignment performance for entity alignment based on
the Hit@1 score, we can confirm that the graph alignment process places output
nodes in our two graphs with the same semantic meaning very close to each
other in an embedding space. Now we explore this embedding space to see
whether it places CNN feature nodes close to the associated knowledge base
nodes. To assess this, we used the embedding spaces generated by (1)KGtoCNN in
Fig.3 with 50% of edges.

The COCO dataset is used to evaluate the embedding space generated from the
graph alignment process. The VGG-16 CNN has never been exposed to the COCO
dataset so it provides us with an independent set of test images.

Fig.4 is an example of Hit@k-nns chart for an image from COCO validation set.
Fig.4(e) shows the image in question. It has the labels of “truck” and
“person”, both of which can be seen in the image. Neither of these two labels
are in the ImageNet dataset. We identified the 200 nearest CNN feature nodes
around the “truck” and “person” node from the knowledge graph. The graph
represents the percentage of CNN feature nodes within that 200 nodes with an
activation larger than a threshold of 0 to exclude negative correlations.

In Fig.4 the green dot line is the Hit@k-NN chart is the theoretically perfect
situation that all the activated CNN feature nodes are the nearest neighbours
of the label node, using a threshold of 0. Fig.4(a) shows the embedding space
generated only using “IsA” relationship, it shows that a high proportion of
the nearest neighbours of the node “person” are activated CNN feature nodes.

Fig.4(b) shows that at $k=1$ all CNN features that are closest to the truck
node are all activated. Although the proportion of activated nodes drops as
$k$ increases, this does not necessarily suggest poor performance as all the
visual feature nodes will not necessarily need to be activated to detect the
content in an image.

For the other embedding space generated by “HasA”, the result is not as good.
Fig.4(c) and Fig.4(d) show that the Hit@k-nns line is far from the perfect
shape. This shows when using the “HasA” relationship to build the knowledge
graph that the nearest CNN feature nodes surrounding “person” and “truck” do
not directly relate to these visual concepts.

###  4.3. Embedding space visualization

Figure 5. An example of t-SNE mapped embedding space generated only using
“IsA” Figure 6. An example of t-SNE mapped embedding space generated only
using “HasA”

Since the feature labelling system we proposed uses GCN-Align to generate the
embedding space, it produces a 200-dimensional vector for describing the
location of each node i.e. the node embedding. To visualize the embedding
space, we applied t-SNE(Maaten and Hinton, 2008) it and mapped the node
embedding to a 2-dimensional space. Fig.5 is a section of the visualisation of
the nearest nodes in the embedding space generated from the “IsA” for the test
image shown in Fig.4. The nodes shown in this visualisation are all those that
are found in the section of the embedding space represented in the
visualisation. No nodes have been removed. In this figure the red circles are
the ConceptNet output nodes. The yellow “+” are the VGG-16 network output
nodes. The blue “$\times$” symbols represent the visual CNN feature nodes in
the flatten layer. The red square symbols are the ConceptNet nodes that are
connected to the output nodes. For each CNN feature node in flatten layer, we
select the test image from the COCO dataset that has the highest activation of
that feature. In addition to visualising the embedding space we use
Deconvolution to visualise the CNN feature which is displayed beside the test
image (the grey box) and shows the focus of the CNN feature for that image.

In Fig.5 we can see that nodes with similar semantic meanings are located
close to each other, for example, the “screen” node network output node from
VGG-16 and the “screen” node from ConceptNet. Nodes that have similar
meanings, such as “tv” and television from ConceptNet, are positioned almost
at the same point. The closest network output features all relate to objects
that are similar semantically, “monitor”, “television”, “desktop computer”,
“screen”. For each of these visual features, the Deconvolution visualization
shows that the content they are detecting matches the meanings of these nodes.
This visualisation shows that for the embedding space generated for the graph
alignment of ConceptNet using the “IsA” relationship captures this “IsA”
relationship with all nodes in this version of the embedding space related to
similar objects.

Fig.6 is an example of the embedding space generated from a ConceptNet graph
using the “HasA” relationship. In this figure, features detecting books are
also near to the nodes from both VGG-16 and ConceptNet. Moreover, in
comparison to Fig.5, there are significantly more ConceptNet nodes in this
neighbourhood, the green squares which are not the output nodes that the
graphs were aligned on. These nodes are potentially new categories and the
embedding space has located them close to relevant visual features. The
visualization shows that our approach can place visual features in a semantic
embedding space and the meanings of nearest neighbour nodes could be a
reference for naming the visual feature.

##  5\. Conclusion

Current research works can explain visual features with several external
meanings, but the feature labelling system is limited because the concepts for
explaining visual features are selected manually. In this paper, we have
presented an approach, based on entity alignment, to finding the meanings of
visual features in CNNs using all the related concepts selected automatically
from a large knowledge base, ConceptNet. We built a graph from VGG-16 by
selecting the important features, and applied entity alignment between the
sub-graph of ConceptNet and the VGG-16 graph to match the visual features with
semantic meanings. The entity alignment reached an average level Hit@1 score
of GCN-Align. The Hit@k-nns charts shows that the true label of an image is
surrounded by activated visual features. The visualization of visual features
with t-SNE mapping of embedding space shows that the visual features are close
to the concepts that they are detecting, and those concepts will explain the
meaning of that visual feature.

The main limitation of our work at this point is the knowledge base is too
noisy. It is full of non-visual concepts, such as (a cat, Desires, meow). We
know it may be irrelevant because it is describing a sound. In other cases,
however, it is hard to tell whether the triple is visual or not. These non-
visual triples will interfere with the alignment process. Another potential
limitation is that the visual feature may not have the same meaning for all
the time. A same visual feature may be detecting the screen of a monitor in
one image but it may be still detecting the frame of a window in another
image.

Having shown the promise of enriching CNNs with external knowledge bases via
entity alignment, the next phase of our work will involve the following (1)
Acquiring a more suitable knowledge source. The current external knowledge
source, ConceptNet, is built from a large language corpus. It was not created
for the purpose of labelling or analysing visual features, so the noise in the
knowledge base will perturb the feature labelling system. (2) Evaluating
additional methods for finding the most important features/nodes. (3) Using
the embedding space to reason for unseen classes for Zero-shot learning based
on feature matching. (4) Combining with object detection: After doing
reasoning using the embedding space, we could follow the existing methods
(Selvaraju et al., 2017) to locate the area of an object.

###### Acknowledgements.

This work was funded by Science Foundation Ireland through the SFI Centre for
Research Training in Machine Learning (18/CRT/6183)

## References

  * (1)
  * Akbari et al. (2009) Ismail Akbari, Mohammad Fathian, and Kambiz Badie. 2009\.  An improved mlma+ and its application in ontology matching. In _2009 Innovative Technologies in Intelligent Systems and Industrial Applications_. IEEE, 56–60.
  * Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020\.  Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  _Information Fusion_ 58 (2020), 82–115.
  * Bach et al. (2015) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ 10, 7 (2015), e0130140.
  * Bau et al. (2017) David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017.  Network dissection: Quantifying interpretability of deep visual representations. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6541–6549.
  * Benesty et al. (2009) Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009\.  Pearson correlation coefficient.  In _Noise reduction in speech processing_. Springer, 1–4.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating embeddings for modeling multi-relational data. In _Advances in neural information processing systems_. 2787–2795.
  * Carlson et al. (2010) Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka, and Tom M Mitchell. 2010.  Toward an architecture for never-ending language learning. In _Twenty-Fourth AAAI conference on artificial intelligence_.
  * Castelvecchi (2016) Davide Castelvecchi. 2016\.  Can we open the black box of AI?  _Nature News_ 538, 7623 (2016), 20.
  * Chen et al. (2013) Xinlei Chen, Abhinav Shrivastava, and Abhinav Gupta. 2013\.  Neil: Extracting visual knowledge from web data. In _Proceedings of the IEEE international conference on computer vision_. 1409–1416.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Gao et al. (2019) Junyu Gao, Tianzhu Zhang, and Changsheng Xu. 2019.  I Know the Relationships: Zero-Shot Action Recognition via Two-Stream Graph Convolutional Networks and Knowledge Graphs.  (2019).
  * Gatys et al. (2016) Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016\.  Image style transfer using convolutional neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2414–2423.
  * Gilpin et al. (2018) Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018\.  Explaining explanations: An approach to evaluating interpretability of machine learning.  _arXiv preprint arXiv:1806.00069_ (2018).
  * Hao et al. (2016) Yanchao Hao, Yuanzhe Zhang, Shizhu He, Kang Liu, and Jun Zhao. 2016.  A joint embedding method for entity alignment of knowledge bases. In _China Conference on Knowledge Graph and Semantic Computing_. Springer, 3–14.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Kim et al. (2016) Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016\.  Examples are not enough, learn to criticize! criticism for interpretability. In _Advances in neural information processing systems_. 2280–2288.
  * Koh and Liang (2017) Pang Wei Koh and Percy Liang. 2017.  Understanding black-box predictions via influence functions.  _arXiv preprint arXiv:1703.04730_ (2017).
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _International conference on machine learning_. 1188–1196.
  * Lecue (2019) Freddy Lecue. 2019\.  On the role of knowledge graphs in explainable AI.  _Semantic Web_ Preprint (2019), 1–11.
  * Lee et al. (2018) Chung-Wei Lee, Wei Fang, Chih-Kuan Yeh, and Yu-Chiang Frank Wang. 2018.  Multi-label zero-shot learning with structured knowledge graphs. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1576–1585.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _European conference on computer vision_. Springer, 740–755.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions. In _Advances in neural information processing systems_. 4765–4774.
  * Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, Nov (2008), 2579–2605.
  * Marino et al. (2016) Kenneth Marino, Ruslan Salakhutdinov, and Abhinav Gupta. 2016\.  The more you know: Using knowledge graphs for image classification.  _arXiv preprint arXiv:1612.04844_ (2016).
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_. 3111–3119.
  * Miller (1995) George A Miller. 1995\.  WordNet: a lexical database for English.  _Commun. ACM_ 38, 11 (1995), 39–41.
  * Mitsuhara et al. (2019) Masahiro Mitsuhara, Hiroshi Fukui, Yusuke Sakashita, Takanori Ogata, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. 2019.  Embedding human knowledge in deep neural network via attention map.  _arXiv preprint arXiv:1905.03540_ 5 (2019).
  * Mordvintsev et al. (2015) Alexander Mordvintsev, Christopher Olah, and Mike Tyka. 2015\.  Inceptionism: Going deeper into neural networks.  (2015).
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization. In _Proceedings of the IEEE international conference on computer vision_. 618–626.
  * Simonyan et al. (2013) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013\.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ (2013).
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge. In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Springenberg et al. (2014) Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014.  Striving for simplicity: The all convolutional net.  _arXiv preprint arXiv:1412.6806_ (2014).
  * Sun et al. (2015) Yufei Sun, Liangli Ma, and Shuang Wang. 2015.  A comparative evaluation of string similarity metrics for ontology alignment.  _Journal of Information &Computational Science_ 12, 3 (2015), 957–964.
  * Sun et al. (2017) Zequn Sun, Wei Hu, and Chengkai Li. 2017.  Cross-lingual entity alignment via joint attribute-preserving embedding. In _International Semantic Web Conference_. Springer, 628–644.
  * Sun et al. (2018) Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. 2018\.  Bootstrapping Entity Alignment with Knowledge Graph Embedding.. In _IJCAI_. 4396–4402.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1–9.
  * Trisedya et al. (2019) Bayu Distiawan Trisedya, Jianzhong Qi, and Rui Zhang. 2019\.  Entity alignment between knowledge graphs using attribute embeddings. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 297–304.
  * Wang et al. (2018b) Xiaolong Wang, Yufei Ye, and Abhinav Gupta. 2018b.  Zero-shot recognition via semantic embeddings and knowledge graphs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 6857–6866.
  * Wang et al. (2018a) Zhichun Wang, Qingsong Lv, Xiaohan Lan, and Yu Zhang. 2018a.  Cross-lingual knowledge graph alignment via graph convolutional networks. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 349–357.
  * Xu et al. (2019) Kun Xu, Liwei Wang, Mo Yu, Yansong Feng, Yan Song, Zhiguo Wang, and Dong Yu. 2019\.  Cross-lingual knowledge graph alignment via graph matching neural network.  _arXiv preprint arXiv:1905.11605_ (2019).
  * Zeiler and Fergus (2014) Matthew D Zeiler and Rob Fergus. 2014.  Visualizing and understanding convolutional networks. In _European conference on computer vision_. Springer, 818–833.
  * Zhu et al. (2020) Qi Zhu, Hao Wei, Bunyamin Sisman, Da Zheng, Christos Faloutsos, Xin Luna Dong, and Jiawei Han. 2020.  Collective Multi-type Entity Alignment Between Knowledge Graphs. In _Proceedings of The Web Conference 2020_. 2241–2252.