tiny[table]font=small

# KGPool: Dynamic Knowledge Graph Context Selection for Relation Extraction

Abhishek Nadgeri1,2 Anson Bastos1,3 Kuldeep Singh1,5 Isaiah Onando Mulang’1,7
Johannes Hoffart6, Saeedeh Shekarpour4, and Vijay Saraswat6
1Zerotha Research  2RWTH Aachen  3IIT Hyderabad  4University of Dayton
5Cerence GmbH  6Goldman Sachs  7IBM Research  Africa
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
sshekarpour1<EMAIL_ADDRESS>
###### Abstract

We present a novel method for relation extraction (RE) from a single sentence,
mapping the sentence and two given entities to a canonical fact in a knowledge
graph (KG). Especially in this presumed sentential RE setting, the context of
a single sentence is often sparse. This paper introduces the _KGPool_ method
to address this sparsity, dynamically expanding the context with additional
facts from the KG. It learns the representation of these facts (entity alias,
entity descriptions, etc.) using neural methods, supplementing the sentential
context. Unlike existing methods that statically use all expanded facts,
_KGPool_ conditions this expansion on the sentence. We study the efficacy of
_KGPool_ by evaluating it with different neural models and KGs (Wikidata and
Freebase). Our experimental evaluation on standard datasets shows that by
feeding the _KGPool_ representation into a Graph Neural Network, the overall
method is significantly more accurate than state-of-the-art methods.

##  1 Introduction

Knowledge graphs (KGs) are the foundation for many downstream applications and
are growing ever larger. However, due to the sheer volume of knowledge and the
world’s dynamic nature where new entities emerge and unknown facts about them
are learned, KGs need to be continuously updated. Distantly supervised
Relation Extraction (RE) is an important KG completion task aiming at finding
a semantic relationship between two entities annotated on the unstructured
text with respect to an underlying knowledge graph Ye and Ling (2019). In the
literature, researchers mainly studied two variants in the RE: 1) multi-
instance RE and 2) sentential RE. The multi-instance RE assumes that in a
given bag of sentences, if two entities participate in a relation, there
exists at least one sentence with these two entities, which may contain the
target relation Riedel et al. (2010); Vashishth et al. (2018). In this
setting, researchers aim to incorporate contextual signals from the previous
occurrences of an entity pair into the neural models to support relation
extraction Ye and Ling (2019); Xu and Barbosa (2019); Wu et al. (2019). In
contrast, sentential RE restricts the scope of document context only to the
input sentence (disregards other occurrences of entity pairs) while predicting
the KG relation Sorokin and Gurevych (2017). Hence, sentential RE makes the
multi-instance setting more difficult by limiting the available context.

Figure 1: Illustration of Knowledge Graph Context associated with the
annotated entities in a sentence. Here, entity aliases do not play any role in
the understanding of the sentence for finding the KG relation. 222sentence
taken from Sorokin and Gurevych (2017)

Recent approaches for RE not only base KGs for relation inventory but also
consider it for extending contextual knowledge for further improvement of RE
task Vashishth et al. (2018); Bastos et al. (2021). A few multi-instance RE
methods rely on entity attributes (properties) such as descriptions, aliases,
and types (as additional context) along with entity pair occurrences from
previous sentences to improve the overall extraction quality Ji et al. (2017);
Vashishth et al. (2018). For the sentential RE, the RECON approach Bastos et
al. (2021) aims to effectively encode KG context derived from the entity
attributes and entity neighborhood triples. RECON employs a Graph Neural
Network (GNN) as a context aggregator for combining sentential context
(annotated entities and sentence) and structured KG representation. Although
the additional KG context has a positive effect on the overall relation
extraction in multi-instance and sentential RE settings, not all KG context
forms are necessary for every input sentence. Consider Figure 2, where the
task is to infer a semantic relation ‘occupation’ between two entities
wd:Q568631 (Marc Davis)333wd: binds to https://www.wikidata.org/wiki/ and
wd:Q266569 (animator). Wikidata Vrandecic (2012) KG provides semantic
information such as description, instance-of, and aliases about entities.
Here, the entity alias (Marc Fraser Davis, Fraser Davis for wd:Q568631; and
cartoonist for wd:Q266569) has no impact on understanding the sentence because
the entities are explicitly mentioned in the sentence. Furthermore, there is
empirical evidence in the literature that for several sentences, statically
adding all KG context offered minimal or negative impact Bastos et al. (2021).
Hence, there are open research questions as to how an RE approach can
dynamically utilize the sufficient context from KG and whether or not the
selected KG context positively impacts the overall performance?

This paper studies these concerning questions proposing the KGPool approach.
KGPool utilizes a self-attention mechanism in a Graph Convolution Network
(GCN) Kipf and Welling (2017) for selecting a sub-graph from the KG to extend
the sentential context. The concept of dynamically mapping the structural
representation of a KG to a latent representation of a sentence has not been
widely studied in prior literature. In RE, KGPool is the initial attempt. The
existing approaches Bastos et al. (2021); Xu and Barbosa (2019); Wu et al.
(2019); Vashishth et al. (2018) feed all the available context (either derived
from a bag of sentences or a KG or both) into a neural model and relied on the
model to figure out the consequences, resulting in limited performance in many
cases Bastos et al. (2021). Conversely, we study the efficacy of KGPool in
dynamically choosing KG context for the sentential RE task using two standard
community datasets (NYT Freebase Riedel et al. (2010), and Wikidata Sorokin
and Gurevych (2017)). Our work makes the following key contributions:

  * •

The KGPool approach dynamically selects structural knowledge and transform it
into a representation suitable to supplement the latent representation of
sentential context learned using a neural model. We deduce that KGPool is the
first approach that works independently of the underlying context aggregators
used in the literature (Graph Neural Network Zhu et al. (2019) or LSTM-based
Sorokin and Gurevych (2017)).

  * •

We are the first to map the task of KG Context Selection to a Graph Pooling
Problem. Therefore, our proposed approach legitimizes the application of graph
pooling algorithms for choosing the relevant context.

  * •

KGPool, paired with a GNN as context aggregator, significantly outperforms the
existing baselines on both datasets, in one experiment increasing the
precision by 12 points over to baseline (P@30 on NYT Freebase). Furthermore,
our empirical results (cf., Table 3) conclude that an LSTM model paired with
KGPool is able to notably outperform a GNN-based approach Zhu et al. (2019)
and nearly all multi-instance baselines Ye and Ling (2019); Wu et al. (2019);
Vashishth et al. (2018) published in the recent years.

This paper is structured as follows: Section 2 reviews the related work.
Section 3 formalizes the problem and the proposed approach is described in
Section 4. Section 5 describes the experimental setup. The results are in
Section 6. We conclude in Section 7.

##  2 Related Work

Multi-instance RE: a few multi-instance RE approaches use convolution neural
network dos Santos et al. (2015), attention CNN Wang et al. (2016) and
attention-based recurrent neural models for relation extraction Zhou et al.
(2016). Other approaches such as Ji et al. (2017); Vashishth et al. (2018)
incorporate entity descriptions, entity and relation aliases from KG to
supplement context from the previous sentences. Work in Vashishth et al.
(2018) employs a graph convolution network to encode entity and relation
aliases derived from Wikidata. HRERE Xu and Barbosa (2019) proposes an
approach for jointly learning sentence and KG representation using cross-
entropy loss function. To effectively capture the available entity context in
the documents, Ye and Ling (2019) suggest an approach incorporating intra-bag
and inter-bag attentions. For a detailed survey, we point readers to Smirnova
and Cudré-Mauroux (2018).
Sentential RE: researchers Sorokin and Gurevych (2017) utilized additional
relations present in the sentence to assist the process of extracting the
target relation using an LSTM-based model. GP-GNN Zhu et al. (2019) generates
parameters of GNN based on the input sentence, which enables GNNs to process-
relational reasoning on unstructured text inputs. RECON Bastos et al. (2021)
is an approach that uses the entity attributes (aliases, labels, descriptions,
instance-of) and KG triples to signal an underlying GNN model for sentential
RE. Authors conclude that the multi-instance requirement can be relaxed
provided a good representation of KG context to enrich the sentential RE
model. However, RECON and multi-instance approaches Xu and Barbosa (2019);
Vashishth et al. (2018) utilize statically derived context from the KG, i.e.,
KG context does not vary depending the sentence.

Graph Pooling and Dynamic Context Selection: researchers proposed several
models for the graph classification aka. graph pooling task Cangea et al.
(2018); Ying et al. (2018); Gao and Ji (2019). These models employ various
approaches such as graph topology-based Rhee et al. (2018), and by learning
the hierarchical graph-structure Ying et al. (2018). Another graph pooling
model relies on node features and topological information using self-attention
Lee et al. (2019) in which a specific number of nodes are always eliminated.
In KGPool, the elimination of nodes depends on a context coefficient and node
importance (Section 4). For context selection, a recent work focuses on
dynamically selecting the KG context to optimize a Pre-Trained Language Model
(PLM) for entity typing and relation classification Su et al. (2020). KGPool
has the following fundamental differences compared to Su et al. (2020): KGPool
inspires its self-attention mechanism from Lee et al. (2019); Vaswani et al.
(2017) to learn a representation of the KG context. Hence, KGPool works
agnostic of the underlying model used for the context aggregation (unlike Su
et al. (2020), which is tightly coupled with PLM). Approaches such as Liu et
al. (2017); Zhang et al. (2018); Kang et al. (2020) also perform dynamic
context selection for respective tasks. However, these approaches are not
focused on knowledge graph context selection.

##  3 Problem Statement

We define the KG as a tuple given by
$KG=(\mathcal{E},\mathcal{R},\mathcal{T}^{+})$ where $\mathcal{E}$ denotes the
set of all vertices in the graph representing entities, $\mathcal{R}$ is the
set of edges representing relations, and
$\mathcal{T}^{+}\subseteq\mathcal{E}\times\mathcal{R}\times\mathcal{E}$ is a
set of all KG triples. The RE Task predicts the target relation
$r^{c}\in\mathcal{R}$ between a given pair of entities $\langle
e_{i},e_{j}\rangle$ from the sentence $\mathcal{W}=(w_{1},w_{2},...,w_{l})$.
If no relation is inferred, it returns ’NA’ label. We aim for the sentential
RE task which put a constraint that the sentence within which a given pair of
entities occurs is the only visible sentence from the bag of sentences. We
view RE as a classification task similar to Sorokin and Gurevych (2017). In a
KG triple $\uptau=(e_{h},r,e_{t})\in\mathcal{T}^{+}$, the relation
$r\in\mathcal{R}$, $e_{h}$ is the head entity (relation origin) while $e_{t}$
is the tail entity. For each entity, associated semantic properties such as
entity label, description, instance-of, and aliases are known as entity
attribute ($At^{e}$) (cf., graph construction step of Figure 2). We aim to
model KG contextual information to improve the classification. This is
achieved by learning the effective representations of the sets $At^{e}$,
$e_{h},e_{t}$, and $\mathcal{W}$ (cf. section 4).

Figure 2: KGPool approach has three components to supplement sentential
context with necessary KG context.

##  4 KGPool Approach

KGPool consists of three components illustrated in Figure 2: 1) Graph
Construction aggregates the sentence, entities and its attributes as a
Heterogeneous Information Graph ($HIG$) for input representation. 2) Context
Pooling step utilizes a self-attention mechanism in a graph convolution to
calculate attention scores for entity attributes using node features and graph
topology. The pooling process allows KGPool to construct a Context Graph
($CG$), which is a contextualized representation of $HIG$ with lesser number
of nodes. 3) Context Aggregator takes as input the sentence, entities,
contextual representations of $HIG$, and classifies the target relation
between the entities. We detail the approach in the following.

###  4.1 Graph Construction

As first step, we extract entity attributes ($At^{e}$) from public dumps of
Freebase Bollacker et al. (2007) and Wikidata Vrandecic (2012) KGs depending
on the dataset. For the KG context, we rely on the most commonly available
properties of the entities as suggested by Bastos et al. (2021): aliases,
description, instance-of, and label. An example of various entity attributes
is given in Figure 2 at the Graph Construction step. Then, the sentence
$\mathcal{W}$ is transformed to another representation using Bi-LSTM Schuster
and Paliwal (1997) by concatenating its word and character embeddings:

$\vec{\mathcal{W}}=\operatorname*{\textit{BiLSTM}}(\mathcal{W})$ (1)

Similar representation is created for each entity $e_{i}$ where $i=(h,t)$:

$\vec{e_{i}}=\operatorname*{\textit{BiLSTM}}(e_{i})$ (2)

For entity $e_{i}$, its KG contexts (entity attributes) $At_{j}^{e^{i}}$
(where $j=[0...N]$) are independently converted into associated embedding
representations:

$\vec{At_{j}}^{e^{i}}=\operatorname*{\textit{BiLSTM}}(At_{j}^{e^{i}})$ (3)

For a knowledge representation of the KG context concerning the sentential
context (sentence and annotated entities), we introduce the special graph
$HIG=(A,F)$, a Heterogeneous Information Graph, represented in the adjacency
matrix $A\in\\{0,1\\}^{n\times n}$, where $n$ is the maximum number of
neighboring nodes for an entity $e_{i}$. Here, $F\in\mathbb{R}^{n\times f}$ is
the node feature matrix assuming each node has $f$ features learned from the
Bi-LSTM in the equations 1, 2, and 3. In these equations, BERT Devlin et al.
(2019), or any other recent Transformer-based model can be used. Due to
hardware limitations, we are bound to Bi-LSTM using Glove embeddings
Pennington et al. (2014).

###  4.2 Context Pooling

Context pooling is built upon three layers of Graph Convolutional Networks
(GCN) and a readout layer associated with each of them. Moreover, the last
layer of GCN is coupled with a pooling layer (cf., ablation studies for
architectural design choice experiments).

####  4.2.1 Graph Convolution

Since KGPool is expected to select the sufficient context, the Context Graph
$CG$ is a reduction of $HIG$ using the mapping $\Psi:HIG\longrightarrow CG$.
The challenge here is the no natural notion of spatial locality, i.e., it is
not viable to pool together all context nodes in an “m × m” patch on $HIG$
because the complex topological structure of graphs prevents any
straightforward, deterministic definition of a “patch”. Furthermore, entities
nodes have a varying number of neighboring nodes, making the graph pooling
challenging (similar to other graph classification problems Ying et al.
(2018)). In $HIG$, entity nodes do not contain information of their neighbors.
Hence, we aim to enrich each entity node with the adjacent node’s contextual
information. Therefore, we employ a GNN variant to utilize its message-passing
architecture to learn node embeddings from a message propagation function. The
message propagation function depends on the adjacency matrix $A$, trainable
parameters $\theta$, and the node embeddings $F$ Ying et al. (2018). We rely
on a GCN model by Kipf and Welling (2017). The GCN layer is defined as:

$F^{(k)}=\operatorname{ReLU}\left(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}F^{(k-1)}\theta^{(k-1)}\right)$
(4)

where $\tilde{A}=A+I$, $\tilde{D}=\sum_{j}\tilde{A_{i,j}}$ and $\theta^{(k)}$
is the trainable matrix. The GCN module might run $k$ iteration and normally
is in the range of two to six Ying et al. (2018). A few graph representation
learning approaches propose to use readout layer that aggregates node features
to learn a fixed size representation Xu et al. (2018); Cangea et al. (2018).
We perform this summarizing after each block of the network (Equation 4), and
aggregate all of the intermediate representation together by taking their sum.
We define readout layer $R$ as:

$R^{(k)}=\frac{1}{N}\sum_{i=1}^{N}F^{(k)}_{i}\|\max_{i=1}^{N}F^{(k)}_{i}$ (5)

where $N$ is the number of nodes in the graph and $F$ is the node feature
embedding.

####  4.2.2 KG Self-Attention Mask

Until Equation 5, KGPool focuses on learning node features. Next, KGPool
learns the importance of each entity attribute node using self-attention.
Please note, in $HIG$, pooling happens only for entity attribute nodes
($\vec{At_{j}}^{e^{i}}$ from Equation 3). The sentence $\vec{\mathcal{W}}$ and
entities $\vec{e_{h}},\vec{e_{t}}$ remain intact. Hence, each entity
representation $\vec{e_{h}}$ and $\vec{e_{t}}$ is enriched by the useful
attribute context (KG context). The entity attribute nodes which do not
provide relevant context are excluded from the graph. To choose the relevant
entity attribute nodes, we use a self-attention score $Z$ Lee et al. (2019)
calculated as follows:

$Z=\tanh\left(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}F^{(k)}\Theta_{att}\right)$
(6)

where $\Theta_{att}\in\mathbb{R}^{F\times 1}$ is the only parameter of the
pooling layer. For ranking, we take the attention score and pass it through a
softmax layer where $Z_{score}$ is the normalized self attention score.

$Z_{score}=\exp{(Z_{i})}/\sum_{i}\exp{(Z_{i})}$ (7)

After Equation 7, we compute the scores for each entity attribute node. Next,
we propose a node selection method in which nodes are selected on the basis of
Context Coefficient $\alpha$ which is a hyper parameter. The top nodes are
selected as:

$idx=\max{(Z_{score})}-\alpha*\sigma{(Z_{score})}$ (8)

where $\sigma{(Z_{score})}$ is the standard deviation of $Z_{score}$, $idx$
represents the node selection result, and $Z_{mask}$ is the corresponding
attention mask. Equation 8 acts as a soft constraint in selecting the context
nodes for each $HIG$ which depends on the value of $\alpha$. Learning $\alpha$
during training may cause over-fitting. Hence, we decided to consider $\alpha$
as a trade-off parameter similar to $\lambda$ in regularization Bühlmann and
Van De Geer (2011). Next, the Context Graph ($CG$) is formed by pooling out
the less essential entity attribute nodes as:

$F^{\prime}=F^{(k)}_{\mathrm{idx},:},F_{out}=F^{\prime}\odot
Z_{mask},A_{out}=A_{\mathrm{idx},\mathrm{idx}}$ (9)

In addition to the dynamically selected nodes, we also inherit the
intermediate node and graph representations of $k-1$ layers similar to ResNET
He et al. (2016). The intermediate representations ($k-1$ ) and the $CG$
($k^{th}$ layer) is given as follows:

$\displaystyle\vec{e_{h}}^{\prime}=F^{(1)}_{e_{h}}\oplus
F^{(2)}_{e_{h}}\oplus....F^{(k)}_{e_{h}}$ (10)
$\displaystyle\vec{e_{t}}^{\prime}=F^{(1)}_{e_{t}}\oplus
F^{(2)}_{e_{t}}\oplus....F^{(k)}_{e_{t}}$
$\displaystyle\vec{\mathcal{W}}^{\prime}=F^{(1)}_{\mathcal{W}}\oplus
F^{(2)}_{\mathcal{W}}\oplus....F^{(k)}_{\mathcal{W}}$
$\displaystyle\vec{R}^{\prime}=R^{(1)}\oplus R^{(2)}\oplus....R^{(k)}$

where in the $i^{th}$ layer: $F^{(i)}_{e_{l}}$ is the node embedding of
$e_{l}$, $l=(h,t)$, $F^{(i)}_{\mathcal{W}}$ is the node embedding of sentence
$\mathcal{W}$, and $R^{(i)}$ is the readout. In the $k^{th}$ layer, $F^{(k)}$
is the $F_{out}$ from Equation 9. The $\oplus$ is the concatenation among the
vectors.

###  4.3 Context Aggregator

Finally, KGPool combines the latent representation (sentential context) with
the structured representation learned in Equation 10. As such, we employ a
model $M$ which learns latent relation vector $\vec{r^{\prime}}$. In the
state-of-the-art approaches that use KG context, the representation of
$\vec{r^{\prime}}$ is learned using sentential and all static KG context
Vashishth et al. (2018); Bastos et al. (2021). However, in KGPool, relation
$r$ is realized based on the sentential context and dynamically chosen KG
context. Hence, we employ context aggregators similar to the baselines
(section 5.3) for jointly learning the enriched KG context in the form of $CG$
and sentential context. The final relation is:

$P(r\mid e_{h},e_{t},W)=softmax(\\\
\operatorname*{\textit{MLP}}(\vec{r^{\prime}}\oplus\vec{e_{h}}^{\prime}\oplus\vec{e_{t}}^{\prime}\oplus\vec{\mathcal{W}}^{\prime}\oplus\vec{R}^{\prime}))$
(11)

##  5 Experimental Setup

###  5.1 Datasets

We consider two standard datasets (English version): Wikidata dataset Sorokin
and Gurevych (2017) and NYT Freebase Riedel et al. (2010). Both datasets were
annotated using distant supervision (associated stats are in Table 1).
Datasets include ’NA’ as one of the target relations.

Dataset |  #Train Sentences |  #Test sentences |  #Relations
---|---|---|---
Wikidata |  372,059 |  360,334 |  353
NYT |  455,771 |  172,448 |  53
Table 1: Statistics of the Datasets

###  5.2 KGPool Configurations

KGPool is configured with two context aggregator modules. We inherit context
aggregators from existing sentential RE baselines. Our experimental aim is to
assess as how KGPool performs along with the state-of-the-art context
aggregators (comparative study). Our two settings are:
1. $KGPool_{+lstm}$: KGPool is coupled with a context aware LSTM model from Sorokin and Gurevych (2017) as context aggregator.
2. $KGPool_{+gnn}$: this implementation has KGPool plugged-in with a variant of GNN module used by Zhu et al. (2019); Bastos et al. (2021).

###  5.3 Baseline Models

We consider the recent sentential RE approaches for our empirical study:
RECON Bastos et al. (2021): induces KG context (entity attributes and 1&2 hop
entity triples) along with the sentence in a GNN.
RECON-EAC Bastos et al. (2021): a variant of RECON contains entity attributes
as only KG context (same context as KGPool).
GP-GNN Zhu et al. (2019): performs multi-hop reasoning using a GNN.
Context-LSTM Sorokin and Gurevych (2017): uses context from other sentential
relations.
Sorokin-LSTM Sorokin and Gurevych (2017): the NYT dataset contains one
relation per sentence, but Context-LSTM requires at least two relations in a
sentence. Thus, the other setting is an LSTM model without a sentential
relation context, is used as a baseline on the NYT dataset.
Multi-instance RE Approaches: Please note, the Wikidata dataset does not have
multiple instances for an entity pair. Hence multi-instance baselines do not
have values on it. We inherit the recent multi-instance baselines and all
empirical values from Bastos et al. (2021): (i) HRERE Xu and Barbosa (2019)
(ii) Wu-2019 Wu et al. (2019), (iii) Yi-Ling-2019 Ye and Ling (2019), (iii)
RESIDE Vashishth et al. (2018).

(a) Micro P-R Curve on Wikidata Dataset

(b) Micro P-R Curve on NYT-Freebase Dataset

Figure 3: KGPool’s best configuration (Tables: 2, 3) perform better than
baselines over the entire recall range.

###  5.4 Metrics and Hyper-parameters

Graph Construction step (section 4.1) use a Bi-LSTM with one hidden layer of
size 50 Bastos et al. (2021). The word embedding dimension is 50 initialized
by Glove embeddings Pennington et al. (2014). The context pooling parameters
are from Lee et al. (2019). For model $M$, we used the default parameters
provided by the authors Zhu et al. (2019); Sorokin and Gurevych (2017). For
brevity, details are in the appendix.
Metric and Optimization: Our experiment settings are borrowed from Bastos et
al. (2021). Hence, on Wikidata dataset, we use (micro) precision (P), recall
(R), and F-score (F1). On the NYT Freebase dataset, (micro) P@10 and P@30 is
reported. P@K here represents precision at K percent recall. We also study the
effect of Context Coefficient ($\alpha$) for both KGPool configurations
(trained end-to-end). We ignore the probability predicted for the NA relation
during testing. We employ the Adam optimizer Kingma and Ba (2015) with
categorical cross entropy loss where each model is run three times on the
whole training set. For the P/R curves (with best $\alpha$ values of KGPool
variants), the result from the first run of each model is selected. For
ablation, we use the McNemar’s test for statistical significance to find if
the reduction in error in the KGPool configurations are significant. The
differences in the models is statistically significant if the $p-value<0.05$
Dietterich (1998). We release all experiment code and data on a public
GitHub444https://github.com/nadgeri14/KGPool.

##  6 Results

We conduct our experiments and analysis in response to the question RQ: ”What
is the efficacy of KGPool in dynamically selecting the KG context for the
sentential RE task?” As such, we also compare KGPool against approaches that
do not dynamically treat the context.

Model |  P |  R |  F1
---|---|---|---
Context-LSTM |  72.09 |  72.06 |  72.07
GP-GNN |  82.30 |  82.28 |  82.29
RECON-EAC |  85.44 |  85.41 |  85.42
RECON |  87.24 |  87.23 |  87.23
$KGPool_{+lstm}$ ($\alpha$=1) |  84.20 |  82.19 |  84.20
$KGPool_{+lstm}$ ($\alpha$=2) |  84.12 |  84.13 |  84.12
$KGPool_{+lstm}$ ($\alpha$=3) |  84.00 |  83.97 |  83.98
$KGPool_{+lstm}$ ($\alpha$=4) |  83.81 |  83.79 |  83.80
$KGPool_{+gnn}$ ($\alpha$=1) |  88.60 |  88.59 |  88.60
$KGPool_{+gnn}$ ($\alpha$=2) |  88.57 |  88.56 |  88.57
$KGPool_{+gnn}$ ($\alpha$=3) |  88.54 |  88.55 |  88.54
$KGPool_{+gnn}$ ($\alpha$=4) |  88.52 |  88.50 |  88.51
Table 2: Comparison of KGPool configurations with sentential RE models on Wikidata dataset. Best score in bold. Model |  P@10 |  P@30
---|---|---
HRERE |  86.1 |  76.6
Wu-2019 |  81.7 |  61.8
RESIDE |  73.6 |  59.5
Ye-Ling-2019 |  78.9 |  62.4
Sorokin-LSTM |  75.4 |  58.7
GP-GNN |  81.3 |  63.1
RECON-EAC |  83.5 |  73.4
RECON |  87.5 |  74.1
$KGPool_{+lstm}$ ($\alpha$=1) |  83.7 |  72.7
$KGPool_{+lstm}$ ($\alpha$=2) |  83.5 |  71.6
$KGPool_{+lstm}$ ($\alpha$=3) |  84.1 |  70.6
$KGPool_{+lstm}$ ($\alpha$=4) |  83.1 |  72.1
$KGPool_{+gnn}$ ($\alpha$=1) |  90.1 |  86.7
$KGPool_{+gnn}$ ($\alpha$=2) |  91.0 |  85.0
$KGPool_{+gnn}$ ($\alpha$=3) |  92.3 |  85.4
$KGPool_{+gnn}$ ($\alpha$=4) |  90.6 |  84.4
Table 3: Comparison of KGPool with sentential and multi-instance RE models on NYT Freebase dataset. Best score in bold. Compared Models | Contingency table |  Statistic |  p-value |  Significance |  Dataset
---|---|---|---|---|---
$KGPool_{+gnn}$ ($\alpha$=3) vs |  160916 |  4702 |  298.18 |  $8.18*10^{-67}$ |  Statistically |  NYT-
RECON |  3169 |  3613 |  |  |  Significant |  Freebase
$KGPool_{+gnn}$ ($\alpha$=1) vs |  617266 |  38652 |  1300.08 |  $1.08*10^{-284}$ |  Statistically |  Wikidata
RECON |  29255 |  55593 |  |  |  Significant |
Table 4: The McNemar’s test for statistical significance for KGPool’s best
configuration Vs previous baseline.

Performance on Wikidata Dataset: Table 2 summarizes the performance of KGPool
variants against the sentential RE models. Agnostic of the underlying
aggregator (LSTM or GNN), KGPool effectively captures the KG context
complimenting the sentential context. The $KGPool_{+gnn}$ ($\alpha$=1)
configuration outperforms other KGPool variants along with all sentential RE
baselines. We can observe that even when the available context is limited to
entity attributes, the $KGPool_{+gnn}$ variant surpasses RECON that also
contains context from 1&2 hop triples besides the entity attributes. RECON-EAC
and $KGPool_{+gnn}$ rely on entity attributes as KG context with the same
context aggregator. When $KGPool_{+gnn}$ variants choose KG context
dynamically, they perform better than RECON-EAC. It is interesting to notice
that when an LSTM model is fed with the dynamically chosen context, the
performance gain is more than ten absolute points ($KGPool_{+lstm}$ Vs
Context-LSTM), even outperforming GP-GNN.
Performance on NYT Freebase Dataset: Similar to the Wikidata dataset, the
$KGPool_{+gnn}$ variants significantly outperform all baselines (cf. Table 3).
The P@30 is comparatively high for $KGPool_{+gnn}$ against baselines. The
behavior could be interpreted as follows: dynamically adding context from the
KG for the entity pairs keeps the precision higher over a more extended recall
range. For both datasets, KGPool configurations ($KGPool_{+gnn}$ and
$KGPool_{+lstm}$) have the best-reported performance varying as per the
$\alpha$. This validates our choice to introduce a soft constraint in
selecting the context nodes (cf., Equation 8). The P/R curves in Figure 3 show
that KGPool performs better than baselines over the entire recall range. We
conclude that the effective dynamic context selection by KGPool has a positive
impact on the sentential RE task (which successfully answers our research
question).

Models |  DEG (HIG) |  DEG (CG) |  Dataset
---|---|---|---
$KGPool_{+gnn}$ ($\alpha$=1) |  5.33 |  1.15 |  Wikidata
$KGPool_{+gnn}$ ($\alpha$=2) |  5.33 |  1.52 |
$KGPool_{+gnn}$ ($\alpha$=3) |  5.33 |  2.87 |
$KGPool_{+gnn}$ ($\alpha$=4) |  5.33 |  4.71 |
$KGPool_{+gnn}$ ($\alpha$=1) |  6.34 |  1.67 |  NYT
$KGPool_{+gnn}$ ($\alpha$=2) |  6.34 |  1.91 |  Freebase
$KGPool_{+gnn}$ ($\alpha$=3) |  6.34 |  2.73 |
$KGPool_{+gnn}$ ($\alpha$=4) |  6.34 |  5.16 |
Table 5: Effect of Context Pooling. ‘DEG’ denotes average degree of an entity
node ($e_{i}$). ‘DEG’ of entity nodes in $CG$ is drastically reduced wrt the
$HIG$.

###  6.1 Ablation Studies

We conducted two ablation studies to understand the behavior of KGPool
configurations:
Significance of Dynamic Context Selection: we perform McNemar’s test for the
best KGPool configuration against the previous sentential state-of-the-art
(i.e. RECON). The results in Table 4 are statistically significant on both
datasets, illustrating KGPool’s robustness. Although $KGPool_{+gnn}$ variants
achieve statistically significant results against RECON, there exist several
sentences for which our approach is unable to select supplementary KG context
(($RW$) values in the contingency table). It requires further investigation,
and we plan it for our future work.
Effect on the Degree of Nodes for Entities: for studying the effect of context
pooling (Section 4.2), we also conducted a study to understand the impact of
KGPool on the reduction of the average degree of entity nodes ($e_{i}$) in the
$HIG$. Table 5 summarizes the effect of Context Coefficient on the average
degree of entity nodes. Irrespective of $\alpha$, KGPool notably reduces the
degree of $e_{i}$ by removing less relevant nodes.
Architectural Choice Experiment: In KGPool, we chose to introduce pooling in
the last layer of a three-layered architecture (three blocks). To support our
choice, we performed several additional experiments by introducing pooling in
various layers. We employ the Wikidata dataset for our experiments. We use
best configuration of our model ( $KGPool_{+gnn}$ ($\alpha$=1)) and created
several variants of it. For instance, $KGPool_{+gnn}$ ($P$=all) comprises the
configuration where we introduce pooling in all three GCN blocks. The
configuration $KGPool_{+gnn}$ ($\P$=2&3) has no pooling in the first layer but
has a pooling layer in the remaining two GCN blocks. $KGPool_{+gnn}$ is the
best configuration of KGPool where pooling is just in the final layer. In
Table 6, we observe that $KGPool_{+gnn}$ with pooling only in the last GCN
block has the superior performance compared to other two variants. Here, the
first two layers are used to learn the node features, which are then employed
with self-attention for node selection. Our experiments justify the
architectural choice decision. However, with a newer graph pooling technique,
such decisions will solely depend on the performance of the approach, and we
can not generalize the results of these experiments.

Model |  F1
---|---
$KGPool_{+gnn}$ ($\P$=all) |  84.19
$KGPool_{+gnn}$ ($\P$=2&3) |  86.87
$KGPool_{+gnn}$ ($\P$=3) (best) |  88.60
Table 6: When we introduce pooling in all three layers or in two layers, the
performance of KGPool’s variants drop. Hence, it justify our choice to add
pooling only in the third layer that gives the best performance (values in
bold). We use best configuration of our model ($KGPool_{+gnn}$ ($\alpha$=1)).

Case-Studies: To understand the KGPool’s performance gain, we report a few top
relations in Table 7. It can be observed from this table that in a few cases,
with lesser context, KGPool can perform significantly better. In the next case
study, to understand the KGPool’s performance while adding additional context
(more noise), we induce extra context in the form of 1&2-hop triples along
with entity attributes. For the same, we considered KGPool’s best
configurations on the Wikidata dataset. The configurations $KGPool_{+gnn}$
($+T$) and $KGPool_{+lstm}$ ($+T$) represent KGPool fed with additional triple
context. For both configurations agnostic of underlying aggregator, we observe
a slight increase in performance (Table 8). There are several triples which
are the irrelevant source of information not needed for a given sentence.
KGPool can remove that information and does not suffer the performance drop
due to added noise in the context. Details on error analysis, performance for
worst performing individual relations, and on a human-annotated dataset are
provided in the appendix.

Relation |  KGPool |  RECON |  GP-GNN
---|---|---|---
vocal specialization |  1.00 |  0.00 |  0.00
list of works |  1.00 |  0.00 |  0.00
track gauge |  1.00 |  0.92 |  0.00
position played |  0.99 |  0.99 |  0.92
sport |  0.99 |  0.99 |  0.97
record label |  0.95 |  0.90 |  0.64
list of episodes |  0.95 |  0.00 |  0.49
wing configuration |  0.94 |  0.57 |  0.00
numeric value |  0.93 |  0.27 |  0.46
vessel class |  0.87 |  0.00 |  0.00
Table 7: Micro F-score of Top performing Relations for $KGPool_{gnn}$ ($\alpha$=1) (on Wikidata dataset). Dynamically chosen context significantly improves performance for many relations. Model |  F1
---|---
$KGPool_{+gnn}$ ($+T$) |  88.85
$KGPool_{+lstm}$ ($+T$) |  84.42
$KGPool_{+gnn}$ |  88.60
$KGPool_{+lstm}$ |  84.12
Table 8: To scale the sources of the contexts, we induce additional triple
context in the KGPool shown as ($+T$) configurations. We use best
configurations of our model ($KGPool_{+gnn}$ ($\alpha$=1) and $KGPool_{+lstm}$
($\alpha$=1)). We observe a slight jump in the performance, however, KGPool is
still able to pool irrelevant context.

##  7 Discussion and Conclusion

Although KGs are often employed for providing background context in the RE
tasks (cf. Section 2), yet there is limited research about defining relevant
context. In this work, we proposed KGPool and provide a set of experiments
proving: 1) Given the limited context that is in individual sentences,
dynamically bringing context from KG significantly improves the RE
performance. 2) We introduced Context Coefficient ($\alpha$), which acts as a
soft constraint in determining the relevant entity context nodes. 3) Our
approach KGPool is invariant of the context aggregator and enables us to learn
effective knowledge representation of the required KG context for a given
sentential context. Our evaluation concerns several key questions:

  * •

Data quality impact on an effective knowledge representation: in spite
KGPool’s significant performance, there exist several sentences for which our
model finds a limitation compared to the baseline (cf. Table 4). One potential
interpretation could be about the noise injected due to the data quality of
the KG context Weichselbraun et al. (2018). Hence, how does the quality of
contextual data impact the performance of context selection approaches is an
open direction.

  * •

Impact of additional sources of KG context: In ablation, we provide a study by
adding 1 & 2-hop triples in addition to entity attributes. There is no
significant increase in the performance, although KGPool is able to remove
irrelevant context for a given sentence. Furthermore, we did not consider edge
features in $HIG$ although KGPool can be extended to support edge features
using techniques such as Simonovsky and Komodakis (2017). Additional
experiments are needed to verify that our empirical observations hold in this
setting, and we leave it for future work.

Overall, KGPool provides an effective knowledge representation for set-ups
where sentence context is sparse. It is interesting to observe that effective
knowledge representation learned using KGPool paired with an LSTM model
outperforms GP-GNN Zhu et al. (2019), and nearly all multi-instance baselines.
Our conclusive results open a new research direction: is it possible to apply
effective context selection techniques coupled with deep learning models to
other downstream NLP tasks? For example, our results can encourage researchers
to extend KGPool or develop novel context selection methods for the tasks
where KGs have been extensively used as additional background knowledge, such
as in entity linking Mulang’ et al. (2020); Mulang et al. (2020), KG
completion Wang et al. (2020); Shi et al. (2017), and recommendation system
Yang et al. (2020).

##  8 Ethics/ Impact Statement:

In this work, we present significant progress in solving sentential RE task.
Harvesting knowledge is an essential goal that human beings seek along with
the advancement of technology. This research and many RE approaches rely on
additional signals from the public KGs to design systems that extract
structured knowledge from unstructured contents. When it comes to who may be
disadvantaged from this research, we do not think it is applicable since our
study of addressing the KG context capabilities is still at an early stage.
Having said so, we are fully supporting the development of ethical and
responsible AI. The potential bias in the standard public datasets that may
lead to wrong knowledge needs to be cleaned or corrected with validation
mechanisms.

## References

  * Bastos et al. (2021) Anson Bastos, Abhishek Nadgeri, Kuldeep Singh, Isaiah Onando Mulang, Saeedeh Shekarpour, Johannes Hoffart, and Manohar Kaul. 2021.  Recon: Relation extraction using knowledge graph context in a graph neural network.  In _Proceedings of The Web Conference (WWW) (long papers)_ , page :N/A.
  * Bollacker et al. (2007) Kurt D. Bollacker, Robert P. Cook, and Patrick Tufts. 2007.  Freebase: A Shared Database of Structured General Human Knowledge.  In _AAAI_.
  * Bühlmann and Van De Geer (2011) Peter Bühlmann and Sara Van De Geer. 2011.  _Statistics for high-dimensional data: methods, theory and applications_.  Springer Science & Business Media.
  * Cangea et al. (2018) Catalina Cangea, Petar Velickovic, Nikola Jovanovic, Thomas Kipf, and Pietro Liò. 2018.  Towards sparse hierarchical graph classifiers.  _CoRR_ , abs/1811.01287.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Dietterich (1998) Thomas G Dietterich. 1998.  Approximate statistical tests for comparing supervised classification learning algorithms.  _Neural computation_ , 10(7):1895–1923.
  * Gao and Ji (2019) Hongyang Gao and Shuiwang Ji. 2019.  Graph u-nets.  In _International Conference on Machine Learning_ , pages 2083–2092.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Ji et al. (2017) Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2017.  Distant supervision for relation extraction with sentence-level attention and entity descriptions.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_ , pages 3060–3066. AAAI Press.
  * Kang et al. (2020) Xiaomian Kang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2020.  Dynamic context selection for document-level neural machine translation via reinforcement learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 2242–2254. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Lee et al. (2019) Junhyun Lee, Inyeop Lee, and Jaewoo Kang. 2019.  Self-attention graph pooling.  In _International Conference on Machine Learning_ , pages 3734–3743.
  * Liu et al. (2017) Liping Liu, Francisco Ruiz, Susan Athey, and David Blei. 2017.  Context selection for embedding models.  _Advances in Neural Information Processing Systems_ , 30:4816–4825.
  * Mulang’ et al. (2020) Isaiah Onando Mulang’, Kuldeep Singh, Chaitali Prabhu, Abhishek Nadgeri, Johannes Hoffart, and Jens Lehmann. 2020.  Evaluating the impact of knowledge graph context on entity disambiguation models.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 2157–2160.
  * Mulang et al. (2020) Isaiah Onando Mulang, Kuldeep Singh, Akhilesh Vyas, Saeedeh Shekarpour, Maria-Esther Vidal, and Soren Auer. 2020.  Encoding knowledge graph entity aliases in attentive neural network for wikidata entity linking.  In _International Conference on Web Information Systems Engineering_ , pages 328–342. Springer.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1532–1543. ACL.
  * Rhee et al. (2018) Sungmin Rhee, Seokjun Seo, and Sun Kim. 2018.  Hybrid approach of relation network and localized graph convolutional filtering for breast cancer subtype classification.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , pages 3527–3534.
  * Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010.  Modeling relations and their mentions without labeled text.  In _Machine Learning and Knowledge Discovery in Databases, European Conference, ECML PKDD_ , volume 6323 of _Lecture Notes in Computer Science_ , pages 148–163. Springer.
  * dos Santos et al. (2015) Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015.  Classifying relations by ranking with convolutional neural networks.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 626–634.
  * Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997.  Bidirectional recurrent neural networks.  _IEEE transactions on Signal Processing_ , 45(11):2673–2681.
  * Shi et al. (2017) Jun Shi, Huan Gao, Guilin Qi, and Zhangquan Zhou. 2017.  Knowledge graph embedding with triple context.  In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , pages 2299–2302.
  * Simonovsky and Komodakis (2017) Martin Simonovsky and Nikos Komodakis. 2017.  Dynamic edge-conditioned filters in convolutional neural networks on graphs.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 29–38. IEEE Computer Society.
  * Smirnova and Cudré-Mauroux (2018) Alisa Smirnova and Philippe Cudré-Mauroux. 2018.  Relation extraction using distant supervision: A survey.  _ACM Computing Surveys (CSUR)_ , 51(5):1–35.
  * Sorokin and Gurevych (2017) Daniil Sorokin and Iryna Gurevych. 2017.  Context-aware representations for knowledge base relation extraction.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017_ , pages 1784–1789.
  * Su et al. (2020) YuSheng Su, Xu Han, Zhengyan Zhang, Peng Li, Zhiyuan Liu, Yankai Lin, Jie Zhou, and Maosong Sun. 2020.  Contextual knowledge selection and embedding towards enhanced pre-trained language models.  _arXiv preprint arXiv:2009.13964_.
  * Vashishth et al. (2018) Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha P. Talukdar. 2018.  RESIDE: improving distantly-supervised neural relation extraction using side information.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1257–1266.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Vrandecic (2012) Denny Vrandecic. 2012.  Wikidata: a new platform for collaborative data collection.  In _Proceedings of the 21st World Wide Web Conference, WWW 2012 (Companion Volume)_ , pages 1063–1064.
  * Wang et al. (2020) Hongwei Wang, Hongyu Ren, and Jure Leskovec. 2020.  Entity context and relational paths for knowledge graph completion.  _arXiv preprint arXiv:2002.06757_.
  * Wang et al. (2016) Linlin Wang, Zhu Cao, Gerard De Melo, and Zhiyuan Liu. 2016.  Relation classification via multi-level attention cnns.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1298–1307.
  * Weichselbraun et al. (2018) Albert Weichselbraun, Philipp Kuntschik, and Adrian MP Braşoveanu. 2018.  Mining and leveraging background knowledge for improving named entity linking.  In _Proceedings of the 8th International Conference on Web Intelligence, Mining and Semantics_ , pages 1–11.
  * Wu et al. (2019) Shanchan Wu, Kai Fan, and Qiong Zhang. 2019.  Improving distantly supervised relation extraction with neural noise converter and conditional optimal selector.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019_ , pages 7273–7280. AAAI Press.
  * Xu et al. (2018) Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018.  Representation learning on graphs with jumping knowledge networks.  In _ICML_.
  * Xu and Barbosa (2019) Peng Xu and Denilson Barbosa. 2019.  Connecting language and knowledge with heterogeneous representations for neural relation extraction.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1_ , pages 3201–3206.
  * Yang et al. (2020) Susen Yang, Yong Liu, Yonghui Xu, Chunyan Miao, Min Wu, and Juyong Zhang. 2020.  Contextualized graph attention network for recommendation with item knowledge graph.  _arXiv preprint arXiv:2004.11529_.
  * Ye and Ling (2019) Zhi-Xiu Ye and Zhen-Hua Ling. 2019.  Distant supervision relation extraction with intra-bag and inter-bag attentions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1 (Long and Short Papers)_ , pages 2810–2819.
  * Ying et al. (2018) Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. 2018.  Hierarchical graph representation learning with differentiable pooling.  In _Advances in neural information processing systems_ , pages 4800–4810.
  * Zhang et al. (2018) Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018.  Graph convolution over pruned dependency trees improves relation extraction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 2205–2215. Association for Computational Linguistics.
  * Zhou et al. (2016) Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016\.  Attention-based bidirectional long short-term memory networks for relation classification.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 207–212.
  * Zhu et al. (2019) Hao Zhu, Yankai Lin, Zhiyuan Liu, Jie Fu, Tat-Seng Chua, and Maosong Sun. 2019\.  Graph neural networks with generated parameters for relation extraction.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL_ , pages 1331–1339.

##  Appendix A Appendix

Due to page limit, we could not put several empirical results in the main
paper. This section describes the remaining empirical studies.

###  A.1 Error Analysis

To understand the failure cases of KGPool, we conducted exhaustive error
analysis. We calculated (micro) F1-Score of each relation in Sorokin dataset
Sorokin and Gurevych (2017). Table 10 illustrates performance of ten relations
on which KGPool performs the worst (ascending order of Micro-F1 score). To put
the study in the right perspective, we also report all sentential RE
baselines’ performance on these relations. While analyzing the errors, we
observe three patterns. First, all models fail in the relations for which
number of instances are sparse. For example, the relation mother has only 190
instances (occurances) and the relation killed by has 48 instances. The
scarcity in training data has made all models to fail on certain relations.
Secondly, our model fails in very closed relations. For example, instead of
predicting the relation drafted
by555https://www.wikidata.org/wiki/Property:P647, our model predicts member of
sport team666 https://www.wikidata.org/wiki/Property:P54. Similarly, in case
of unmarried partner, our model predicts spouse. We believe that introducing
logical reasoning in the model can help these borderline cases. The third
observed pattern for errors is the quality of context. It is worthwhile to
mention that in GP-GNN and Context-LSTM, there is only a sentential context.
RECON and KGPool use KG context. Still, performance is limited for many
relations such as use and different from as reported in the table 10. The lack
of quality context in the KG possibly a reason for limited performance for KG-
context-induced models in erroneous cases. Detailed exploration is needed to
understand the impact of data quality on KGPool performance, and we leave it
for the future work.

###  A.2 Effect of Context Pooling

Models |  DEG (HIG) |  DEG (CG) |  Dataset
---|---|---|---
$KGPool_{+lstm}$ ($\alpha$=1) |  5.33 |  1.06 |  Wikidata
$KGPool_{+lstm}$ ($\alpha$=2) |  5.33 |  2.12 |
$KGPool_{+lstm}$ ($\alpha$=3) |  5.33 |  4.32 |
$KGPool_{+lstm}$ ($\alpha$=4) |  5.33 |  4.81 |
$KGPool_{+lstm}$ ($\alpha$=1) |  6.34 |  1.23 |  NYT
$KGPool_{+lstm}$ ($\alpha$=2) |  6.34 |  1.74 |  Freebase
$KGPool_{+lstm}$ ($\alpha$=3) |  6.34 |  3.05 |
$KGPool_{+lstm}$ ($\alpha$=4) |  6.34 |  6.30 |
Table 9: Effect of Context Pooling. ‘DEG’ denotes average degree of an entity node ($e_{i}$). We observe a reduction in the degree of entity nodes in $CG$ compared to the $HIG$. Relation |  KGPool |  RECON |  GP-GNN |  Context-LSTM
---|---|---|---|---
has quality |  0.00 |  0.00 |  0.00 |  0.00
enclave within |  0.00 |  0.00 |  0.00 |  0.00
drafted by |  0.01 |  0.08 |  0.00 |  0.02
different from |  0.01 |  0.00 |  0.0 |  0.00
mother |  0.03 |  0.05 |  0.02 |  0.00
unmarried partner |  0.04 |  0.01 |  0.00 |  0.00
killed by |  0.04 |  0.01 |  0.00 |  0.04
use |  0.09 |  0.00 |  0.00 |  0.00
lyrics by |  0.10 |  0.13 |  0.00 |  0.00
relative |  0.12 |  0.10 |  0.00 |  0.00
Table 10: Micro F-score of 10-worst performing Relations for $KGPool_{gnn}$
($\alpha$=1) on Wikidata dataset. We also provide corresponding values of
other sentential RE baselines. The main reason for limited performance across
all models is the scarcity of training data for these relation types.

In the main paper, we presented the effect of context pooling on KGPool’s best
configuration ($KGPool_{+gnn}$). Table 9 describes the reduction in the
average degree of nodes for $KGPool_{+lstm}$ configuration for various context
coefficient ($\alpha$). On both datasets, there is a significant reduction in
the degree of nodes. On Wikidata dataset Sorokin and Gurevych (2017),
$KGPool_{+lstm}$ with ($\alpha$=1) reports the highest value among its other
configurations. For the same, the average degree of nodes is reduced from 5.33
to 1.06. Please note, the degree of nodes in $HIG$ remains the same. However,
for $CG$, the degree of nodes differs based on the context aggregator. We
train the model end to end, and due to back-propagation, context weights
adjust as per the context aggregator.

###  A.3 Results on a Human Annotated Dataset

The employed datasets Wikidata Sorokin and Gurevych (2017) and NYT Freebase
Riedel et al. (2010) are created using distant supervision techniques.
Considering distant supervision techniques inherit a noise, to provide a
comprehensive ablation study, Zhu et al. (2019) provided a human evaluation
setting. Following the same setting, RECON provided human-annotated data from
Wikidata dataset Sorokin and Gurevych (2017). This is to verify that the
distantly supervised dataset is correct for every pair of entities. Sentences
accepted by all annotators are part of the human-annotated dataset. There are
500 sentences and 1846 triples in the test set. Table 11 reports $KGPool$’s
performance against the sentential baselines. $KGPool_{+gnn}$ continues to
outperform the baselines, maintaining similar behavior as seen on test sets of
original datasets. The results further re-assure the robustness of our
proposed approach.

Model |  P |  R |  F1
---|---|---|---
Context Aware LSTM |  77.77 |  78.69 |  78.23
GP-GNN |  81.99 |  82.31 |  82.15
RECON-EAC |  86.10 |  86.58 |  86.33
RECON |  87.34 |  87.55 |  87.44
$KGPool_{+lstm}$ ($\alpha$=1) |  86.34 |  86.07 |  86.20
$KGPool_{+gnn}$ ($\alpha$=1) |  89.36 |  89.31 |  89.33
Table 11: Sentential RE performance on Human Annotation Dataset. KGPool again outperforms the baselines. We report Micro P,R, and F1 values. (Best score in bold) Hyperparameters |  Value
---|---
learning rate |  0.001
batch size |  50
hidden state size |  128
context coefficient ($\alpha$) |  1,2,3,4
$\\#$ of propagation layers |  3
Table 12: Hyper-parameters for Context Pooling module Hyperparameters |  Value
---|---
learning rate |  0.001
batch size |  50
dropout ratio |  0.5
hidden state size |  256
non-linear activation |  relu
$\\#$ of propagation layers |  3
entity embedding size |  8
adjacent matrices |  untied
optimizer |  adam
$\beta_{1}$ |  0.9
$\beta_{2}$ |  0.999
$\varepsilon$ |  1e-08
pretrained embeddings |  glove
word embedding dim |  50
Table 13: Hyper-parameters for GNN-Aggregator module Hyperparameters |  Value
---|---
learning rate |  0.001
batch size |  50
dropout ratio |  0.5
hidden state size |  256
non-linear activation |  relu
$\\#$ of layers |  1
optimizer |  adam
pretrained embeddings |  glove
word embedding dim |  50
Table 14: Hyper-parameters for ContextAware-Aggregator module Hyperparameters |  Value
---|---
learning rate |  0.001
batch size |  50
initial embedding size |  50
final embedding size |  50
pretrained embeddings |  glove
$\\#$ of layers |  1
Table 15: Hyper-parameters for Graph Construction module

###  A.4 Datasets and Hyper-parameters

We augmented two datasets Wikidata dataset and Riedel Freebase dataset with
our proposed KG context. The Wikidata dataset has 353 unique relations,
372,059 sentences in training, 123824 sentences in validation and 360,334 for
testing. The number of sentences in the training and test set are 455,771 and
172,448 respectively in the Riedel dataset. No explicit validation set has
been provided for Riedel dataset. For augmenting entity attribute context, we
relied on public dumps of Wikidata and Freebase. From these dumps, we
automatically extracted entities and its properties: labels, aliases, instance
of, descriptions. For Wikidata, we used public
API777https://query.wikidata.org/ using a SPARQL query and for Freebase, we
took original depreciated dump888https://developers.google.com/freebase.

We use the nltk english tokenizer for splitting the sentence into its
corresponding tokens in the Riedel dataset. We do not do any further data
preprocessing. We used 1 GPU NVIDIA TITAN X Pascal with 12GB of GPU storage to
run our experiments. We train the models upto a maximum of 14 epochs and
select the best performing model based on the micro F1 scores of the
validation set. The tables 14, 15 and 12 detail the hyper-parameter settings
used in our experiments. We do not do any further hyper-parameter tuning.


# Finding Tori: Self-supervised Learning for Analyzing Korean Folk Song

###### Abstract

In this paper, we introduce a computational analysis of the field recording
dataset of approximately 700 hours of Korean folk songs, which were recorded
around 1980-90s. Because most of the songs were sung by non-expert musicians
without accompaniment, the dataset provides several challenges. To address
this challenge, we utilized self-supervised learning with convolutional neural
network based on pitch contour, then analyzed how the musical concept of tori,
a classification system defined by a specific scale, ornamental notes, and an
idiomatic melodic contour, is captured by the model. The experimental result
shows that our approach can better capture the characteristics of tori
compared to traditional pitch histograms. Using our approaches, we have
examined how musical discussions proposed in existing academia manifest in the
actual field recordings of Korean folk songs.

##  1 Introduction

Folk songs are considered as a musical language. Not only that, but they are
also regarded as the root of all traditional music. Folk songs are potent
embodiments of a region’s cultural and linguistic characteristics, serving as
a foundation for all artistic and musical developments since their inception.
It is believed that research for folk music can provide new inspiration for
existing art music research, and facilitate an interdisciplinary approach that
encompasses both music and other fields.

In Korean musicology, there have been ongoing discussions aimed at identifying
regional characteristics in folk music. Through numerous debates, tori is used
as the most representative theory in many studies. Even though the use of tori
as a term to describe the musical characteristics of Korean folk music is
widespread and its utility is well acknowledged, the existing tori
classification methods were unable to fully explain the features of the actual
music. Various scholars have defined the different tori according to musical
characteristics such as scales (intervals between notes), primary notes,
ornamentation, ending note and idiomatic melodic patterns in folk songs,
leading to a refinement process for the classification of tori. So far, there
are still ongoing opinions, controversies, and discussions concerning the
existing tori[1, 2].

In order to conduct valuable discussions on analyzing folk songs, the
relationships within the audio must be examined and systematically shared.
However, checking numerous audio files individually is extremely time-
consuming and difficult, so previous research has focused on analyzing small
amounts of audio. The most common method used by musicologists has been
transcription, which involves listening to the music and notating it in a
specific music notation system. However, the task of transcribing orally
transmitted music into a music notation system has inherent limitations[3],
such as quantizing the pitch and rhythmic features of folk songs.

For these reasons, there have been several research on analyzing folk music
empowered with methods derived from music information retrieval (MIR) on audio
recording. For example, Indian raga music has been analyzed based on first-
order pitch distrubutions [4], and for traditional three-part Georgian
singing, F0-based tonal analysis[5] or development of tuning systems[6] has
been introduced. [7] also presented analysis using audio-signal processing on
Flamenco and Arab-Andalusian vocal music.

In this paper, we take a computational approach to analyze a vast corpus of
Korean folk songs, utilizing deep-learning-based methodologies. Our primary
aim is to investigate the connection between conventional musicological
classifications of Korean folk songs, tori, and the actual field recordings.
Through a comparison of our analytical results with established musicological
frameworks, our objective is to offer insights on identifying meaningful
clustering or distinguishable features among Korean folk songs. We also share
our code and metadata that contains links to original audio with our manual
tori labels for 218 songs111https://github.com/danbinaerinHan/finding-tori.

##  2 Korean Folk Songs and Tori

###  2.1 Dataset

The “Anthology of Korean Traditional Folksongs” is an audio collection
consisting solely of traditional Korean folk songs [8]. As part of a cultural
project conducted by Munhwa Broadcasting Corporation (MBC) from 1989 to 1995,
the folk song collection was compiled under the direction of Sang-il Choi. The
available audio contains 15,861 songs, with approximately 700 hours in total
length.

The audio was field-recorded across 153 city/county and 1,010 villages in
South Korea. The metadata accompanying the recordings includes title, machine-
readable lyrics, regions, recording dates, and the singer’s name and age. The
region represents the administrative districts of South Korea at the time of
recording, which consists of nine categories in total. Due to its extensive
audio corpus and detailed accompanying metadata, this dataset has become a
crucial resource for research on Korean folk music within the domestic
academic community[9, 10, 11]. We obtained the audio material through crawling
the original website,222http://urisori.co.kr/urisori-en/doku.php/ which has
been hosted by MBC since April 2022, where every audio and metadata is openly
published. We received official approval for using these data for research
purposes from MBC.

###  2.2 Tori

Tori was proposed by a Korean musicologist named Bo-hyeong Lee in the early
1980s to explain the musical characteristics of regional folk songs[12, 13].
Prior to the development of tori theory, Korean music academia used the terms
jo and cheong to explain the musical features of folk songs, based on the
tonal center and intervals between notes[14]. However, Bo-hyeong Lee argued
that jo and cheong are insufficient in capturing the unique musical
characteristics of Korean folk music. Therefore, he proposed the tori
classification system as an alternative approach, which covers primary notes,
ornamentation, ending note and idiomatic melodic patterns.

The most widely-used tori classification method divides songs into four
categories. We encourage readers to refer Figure 3, which presents scale of
three tori with staff notation. The pitch name in this section is used as an
conventional representation, not absolute pitches. Gyung-tori utilizes five
notes and is mainly sung in the capital region. It often uses a gentle vibrato
overall and finessed ornamental melody. Menari-tori is widely distributed
throughout the eastern regions and the entire Korean Peninsula. When the
melody ascends, it leaps through notes, while during the descending melody, it
comes down through passing notes. Yukjabaegi-tori is commonly found in the
southern regions of the Korean Peninsula. It is characterized by a thick and
vibrant vibrato in G note and passing briefly through E in a descending melody
from E to D. Sushimga-tori commonly appears in the northwestern region of the
Korean Peninsula, characterized by its unique vibrato inflecting upward.

###  2.3 Tori annotation

As the dataset did not include any tori labels, one of the authors, who has
over 10 years of experience in Korean traditional music, selected a subset of
218 high-quality recordings and manually annotated them in terms of tori
classification.

To create this tori-subset, we focused on identifying clear musical
characteristics present in the tori classification method, such as pitch
scale, ornamentation, and idiomatic melodies, rather than considering the
audio’s recorded region. We found that there are not many instances of
sushimga-tori in the audio dataset, as the dataset predominantly consists of
folk songs from the central and southern regions of the Korean Peninsula.
Also, we observed that songs belonging to menari-tori appeared not only in the
eastern region but also nationwide. In conclusion, we mainly focused on the
remaining three types of tori, gyung-tori, yukjabaegi-tori, and menari-tori.
Finally, audio recordings that were closer to speech or chanting (non-musical)
and did not exclusively belong to any other subdivided tori types were labeled
as ‘others’. We have labeled totally 218 songs: 65 gyung, 73 menari, 49 yukja,
and 31 others.

##  3 Methodology

In this study, we take an approach of representing the musical characteristics
of a given folk song using a high-dimensional single vector, which can also be
called an embedding of the song. If this embedding accurately represents the
musical characteristics of tori, it can be utilized for various purposes
including tori classification, song similarity searches and clustering songs
in the corpus based on tori similarity.

Even though the dataset is provided with audio recordings, we focus on the
contour of fundamental frequency (F0) rather than using audio directly. By
doing so, our embedding could better capture the melodic features of the songs
rather than timbral characterstics such as the dialect of the singer. To
extract the F0 contour from each audio recording, we utilized CREPE [15], a
widely-used CNN-based model. It extracts F0 value for every 10 ms as well as
confidence score ranging from 0 to 1. Figure 1 illustrates an example of an
extracted pitch contour.

Figure 1: Example of pitch contour from the dataset extracted by CREPE. F0
value under confidence of 0.8 was masked out

###  3.1 Pitch Histogram

Since the characteristics of a musical system can be largely studied from the
pitch distribution of its melodies, there has been numerous research using
pitch histograms to analyze traditional musics from different culture, such as
for Indian Carnatic music [16], Turkish Makam [17], Arab-Andalusian music
[18], or Iranian dastgāhi music [19]. Therefore, we applied a similar approach
to Korean folk music.

One of the important features one has to know to exploit pitch histogram is
the tonic of the song so that one can normalize the pitch histograms of each
song into relative pitch, rather than directly using absolute pitch value.
During our preliminary experiment, we found that the tonic of each recording
in the dataset is usually the most frequently appearing pitch in the song.
Therefore, we determined the tonic by counting the number of appearances of
each pitch in terms of time frames, utilizing a 100 cents range to count pitch
(i.e. MIDI pitch 60.49 and 59.51 are counted as the same pitch). As the center
pitch may differ from the typical 440 Hz tuning, we identified the best
matching pitch by adjusting the pitch center by increments of 10 cents and
selecting the pitch that showed the maximum pitch count.

###  3.2 CNN contour encoder

While the pitch histogram can offer a comprehensible representation of each
song, it cannot fully capture the concept of tori as it does not consider the
relationship between each note in the scale. Additionally, because most of the
recordings were sung by non-professional singers without instrumental
accompaniment, the intervals of singing and tonic frequency has signifcant
amount of noise. Therefore, we employ a convolutional neural network (CNN) to
learn the representation of a given pitch contour. This approach allows us to
better capture the underlying musical characteristics of each song in relation
to tori classification.

The problem is setting the training objective of the CNN model. One option is
to train the model in a supervised manner, as typically done in other MIR
tasks, such as classification tasks with an annotated dataset. In our dataset,
region labels, which represents the administrative region where the recording
took place, can be used for training labels. It can be regarded as a
reasonable approach considering that tori has strong correlation with the
regional characteristics of each folk song. However, we discovered several
different types of tori in a single recorded region, which would make it
difficult to learn distinguishable musical characteristics only using the
region label.

To address this issue, we propose to adopt a self-supervised representation
learning, so that we can obtain a representation of given pitch contour that
is consistent within a song without extra annotated labels as shown in Fig. 2.
This approach of exploiting intra-song similarity has been widely used for
music audio representation learning [20, 21]. We use triplet loss with hinge
margin as Equation 1,

$L=\max(0,m-\text{Sim}(v_{a},v_{p})+\text{Sim}(v_{a},v_{n}))$ (1)

where $m$ denotes hinge margin, $\text{Sim}(v_{a},v_{p,n})$ represents cosine
similarity between anchor vector $v_{a}$ and positive vector $v_{p}$ or
negative vector $v_{n}$.

Figure 2: Self-supervised learning with triplet loss using cosine similarity,
with four layers of convolutional neural network

As presented in Fig. 2, the model consists of 4 layers of 1D convolution
layer, each with a kernel size of three and channel size of 64, 128, 256, and
256, respectively, each followed by batch normalization layer. We used a max
pooling layer between each convolutional layer with size 5 for the first layer
and 4 for the second and third layers. On top of the convolutional stack, we
used context attention to summarize the arbitrary length of vectors, which was
initially proposed in hierarchical document classification [22], with a slight
modification of multi-head attention weight [23] instead of single-head.
Context attention is a type of attention mechanism that uses an independent
learnable parameter named context vector as a query while using the input
vectors as key and value. The final embedding size of the model was 256.

The input is a pitch contour in the shape of $T\times 2$, where $T$ denotes
the number of time step of pitch contour. Throughout the experiment, we used
the frame rate of 20 Hz, which means that 30 seconds of pitch contour is
converted to 600 time steps. During the training, we randomly slice the
contour to have 30 seconds length, while using the entire contour in the test
or visualization. Two channels are tonic-normalized MIDI pitch and confidence
of the F0 estimation at that time frame. Since the automatic F0 estimation
includes noise, especially in the unsung silent part, we masked the estimated
pitch value to zero if the confidence is lower than 0.8.

##  4 Experiments

###  4.1 Evaluation Design

In the experiments, we explored the quantitative relation between tori and
different embedding schemes to see which type of embedding can explain the
concept of tori better. To evaluate the correlation between tori and the given
embedding scheme, we exploit our tori-labeled subset with two metrics.

The first metric is the ranking of cosine similarity. If the embedding shares
important characteristics with the concept of tori, we can expect that songs
with high cosine similarity in the given embedding space share the same tori
with the query song. Therefore, we calculated the cosine similarity between
each song in the tori-subset, and obtained normalized discounted cumulative
gain (nDCG), which is a frequently used metric to evaluate the quality of
search results. If all the other $n$ songs with the same tori are ranked
within $n$-th order in the similarity, nDCG becomes one.

The second metric is the tori classification accuracy using a random forest
classifier. If the embedding includes essential characteristics that define
tori, one can expect that it can be exploited to classify the tori for a given
song. Among many options, we used a simple random forest classifier. The
random forest classifier with 100 trees was trained with 75% of the tori-set
and tested with the remaining 25%. For each embedding scheme, we repeated the
procedure 30 times with different train/test split and reported mean and
deviation of accuracy to ensure that the results were not dependent on the
specific dataset split.

###  4.2 Training

While a pitch histogram can be obtained from a given song without additional
training data, CNN models require training procedures. We employ F0 contour
extracted from the “Anthology of Korean Traditional Folksongs” to train our
model.

Some of the songs in the dataset were recorded with multiple singers or
percussion accompaniment. Because these can degrade the performance of F0
estimation, we excluded them while training the CNN model. Instead of manually
filtering the dataset, we used a CNN-based sound event detection model [24] to
calculate the activation of ‘choir’ and ‘percussion,’ which are included among
the model’s event vocabulary. We filtered the song by the maximum activation
value exceeding a certain threshold, which was manually decided by observing
the activation distribution across the dataset.

The CNN model was trained for 25,000 updates. For the self-supervised learning
using triplet loss, we used eight negative samples with one positive sample,
and hinge loss using a margin of 0.4. For the region-supervised training, we
use cross-entropy loss with class weight to address the class imbalance issue.
The architecture of the CNN remained the same, but we added a fully connected
layer to project the embedding to logits for nine different region categories.
We used the Adam optimizer with an initial learning rate of 0.001 and batch
size of 128. The entire training process was conducted on a single RTX A5000,
which took approximately 10 minutes to complete 25,000 updates. We validated
the model’s training procedure and hyperparameters using a specific split of
the tori-subset and chose the simplest setting to avoid overfitting the
hyperparameters to the small subset.

###  4.3 Results

| Similarity Ranking | Random Forest Classifier
---|---|---
Embedding | nDCG | Accuracy
Hist. (25 bin) | 0.783 | 0.744 $\pm$ 0.058
Hist. (124 bin) | 0.777 | 0.722 $\pm$ 0.054
CNN (region ) | 0.792 | 0.634 $\pm$ 0.055
CNN (SSL) | 0.853 |  0.848 $\pm$ 0.039

Table 1: Experiment results on the tori subset. Hist. denotes normalized pitch
histogram, specified with the number of bins to cover two-octave range. region
and SSL denotes the region-supervised and self-supervised learning.

The evaluation result of each embedding scheme is presented in Table 1. As
tori is closely related to the use of pitch, the pitch histogram showed about
74 % accuracy when combined with a random forest classifier, and 0.783 or
0.777 nDCG in the similar song search. The result shows that increasing the
resolution of the histogram does not achieve better performance. The
performance of CNN embeddings trained with region classification was worse
than the pitch histogram in classification accuracy. However, the CNN
embedding trained with self-supervised learning showed the best nDCG and
classification accuracy, even though both CNN shares the same architecture
except the final projection layer.

The performance gain compared to the pitch histogram can be caused by the fact
the concept of tori includes not only the pitch scale the song uses but also
ornamentation and idiomatic phrases it, which can be easily captured by
contour CNN.

The result also shows that the region label did not help to learn tori-related
characteristics. Even though the concept of tori is strongly related to each
region, folk songs were widely spread nationwide at the time of the recording,
which made it difficult to learn coherent musical characteristics from the
region label.

It is also worth noting that self-supervised learning did not use any other
musical knowledge that is related to tori, except that it was trained to
extract constant embedding throughout a given song regardless of the segment
position. From a machine learning point of view, the model has to extract an
embedding that can explain the entire song or distinguish it from other songs
from a given fragment. The high correlation between the trained embedding and
tori labels shown in the evaluation implies that the concept of tori is
clearly related to the distinguishable characteristics of each Korean folk
song.

##  5 Analysis and Discussion

Figure 3: Average pitch histogram of each tori obtained from the tori-subset
and corresponding scale description.

In this section, we introduce interesting musicological findings that we have
found from the pitch histogram and embedding learned from self-supervised
training.

###  5.1 Pitch Histogram Analysis

To explain the scale and characteristics alongside with pitch histogram of
each tori, we present the staff notation of each tori in Figure 3 based on
various studies describing tori [25, 26, 2]. In the score, the cheong (a main
note that generally appears most frequently) was fixed as the C5. The unfilled
notes are primary tones, and the filled notes are passing tones. In cases
where there is a vibrato mark below the note, it is called yoseong, meaning
the pitch generally oscillates up and down. In addition, the arrow on the left
side of the note indicates that the pitch is slightly raised or lowered
compared to the equal-tempered pitch. The slur mark flowing down to the right
of the note represents twoeseong, which means the pitch slides down.

While menari-tori and yukjabaegi-tori remain within a perfect fifth range from
the center, in gyung-tori, there is a larger proportion of high notes (G5-A5)
or more from the center. This indicates that the distribution of notes from
low to high in gyung-tori is evenly spread. This could be due to the fact that
folk songs of gyung-tori are more commonly found in popular folk songs than in
those that appear nationwide. Perfect 4th below the center tone is common in
all Korean music. However for example in gyung-tori, there is vibrato in the
G4 note, and the A4 note has a low pitch, resulting in a distinct distribution
in that part. In menari-tori, it can be observed that the B4 note is used as a
passing tone between C5 and G4. Yukjabaegi-tori is characterized by a vibrant
vibrato in the G4, resulting in a gentle distribution in the nearby pitch than
other tori.

Whether a major third or a minor third is used in the scale is an important
factor in distinguishing the regional musical characteristics of the eastern
and western parts of the Korean Peninsula in the traditional method. Subtle
pitch differences can be observed in all three tori. In gyung-tori, the D5
note is clearly distributed lower and the E5 note is definitely higher than
the line appearing the equal-tempered tune. In yukjabaegi-tori, it is also
confirmed that the E note is slightly higher than the minor third note’s

Through the above analysis, we found that pitch histograms can be utilized to
roughly identify musical characteristics, such as the range, interval from
tones and subtle pitches, etc. However, we also acknowledged that this
approach solely exhibits the constituent pitches and their frequencies,
without accounting for other aspects like uses of melody and ornamentation and
more.

Figure 4: UMAP visualization of folk song embeddings obtained from our self-
supervised-trained CNN model. Note that no tori label was used to train the
model.

###  5.2 UMAP visualization

We visualized the embeddings learned through self-supervised learning in 2D
using UMAP, a dimension reduction technique that is frequently used for
visualization of high-dimensional vectors in Figure 4. In the first figure, we
demonstrated the distribution of four different tori labels. In addition, by
examining the metadata information of the embeddings, we figured out several
interesting topics from this visualization. We also implemented a web demo
using this visualization where one can directly access the corresponding audio
recording333https://danbinaerinhan.github.io/korean-folksong-visualization/.

Menari-tori has a much wider distribution area and encompasses a broader
musical range. In Figure 4 (A), the menari-tori distribution can be seen to be
wider compared to the other tori. We mentioned that several subdivided tori of
it have been identified by scholars. Among them, eosayong-tori is a
representative example of refining the musical characteristics of eastern folk
songs[11]. Eosayong-tori, a lamenting song sung by lumberjacks, was suggested
to have distinct musical characteristics compared to menari-tori. For
instance, in eosayong-tori, the lowest pitch is a semitone higher, and it
concludes on the lowest pitch of the scale instead of the final pitch and the
middle pitch found in menari-tori. We examined embeddings corresponding to
eosayong-tori in the metadata’s title information, and they were clearly
gathered in a different space than the area annotated as menari-tori, as
presented in Figure 4 (B).

Similar results can be observed in the following cases with arari. Arirang,
having repetitive refrain lyrics such as “arirang” or “arari”, is the most
representative Korean folk song, with countless versions appearing in various
regions. Among them, arari, another name for Jeongseon arirang, was sung
primarily in Gangwon province before spreading nationwide in the 1930s[27,
28]. Arari is also regarded as a separated tori from menari-tori for some
researchers[29], with its own slower tempo, monotonous skeleton tones, and the
use of decorative tones. In Figure 4 (C), we can see the clustering of
entities with the title arari extracted from the metadata.

Another interesting example is songs from Jeju. Folk songs sung from Jeju
island, which is the largest island in Korea, have fewer research results
compared to other regions. In Jeju folk songs, the interval between the
pitches is narrower compared to other provinces’. Due to these reasons, the
traditional method of transcribing into staff notation was unsuitable for
capturing the characteristics of Jeju music [30, 31, 32]. However, using our
embeddings, we can identify three clusters as in Figure 4 (D). It implies the
potential usefulness of our approach to uncover the unique characteristics of
folk songs in Jeju Island.

We also checked the UMAP of the pitch histograms but could not find clear
clustering as in the presented examples.

##  6 Conclusions

In this paper, we have presented our computational approach to obtaining a
high-dimensional embedding vector for a given pitch contour, which allows us
to analyze a vast amount of Korean folk songs. Using these embeddings and a
manually labeled subset, we have examined how musical discussions proposed in
existing academia are manifested in our dataset. As our results cover various
music characteristics, the learned embeddings can be utilized as a monitoring
aid when dealing with numerous undefined data to review and find the tori.
Also, we have discussed the methods and possibilities for utilizing and
interpreting experimental results in the future research. Through this
material, we can review and refine our understanding of the concept of the
tori, and provide easily accessible resources and utilize them as appropriate
evidence. Furthermore, there is potential to clarify the musical
characteristics of regions that are distinct from other regions, like Jeju,
about which existing research has not been as active. Ultimately, we expect
this research to be valuable in illuminating the relationships and
transformations of folk songs over time.

##  7 Acknowledgement

We wish to express our deepest gratitude to Sang-il Choi and his team for
their invaluable work on the “Anthology of Korean Traditional Folksongs”.
Their effort to collect more than ten thousand unique recordings from across
Korea has not only significantly contributed to the preservation of the rich
tapestry of Korean folk music but has also created a monumental resource for
academic research. Without their extraordinary efforts, our work would not
have been possible. We sincerely thank them for laying such a robust
foundation and for enabling future researchers and us to build upon it. We
also appreciate MBC for kindly allowing us to utilize the dataset for this
research.

The web demo was implemented with the kind help of Dongmin Kim.

## References

  * [1] E. J. Shin, “Reconsideration on the modes(tori) of korean folk song(한국 민요 선법 (토리) 의 연구 성과 검토 및 논점),” _The Society Of Korean Folk Song(한국민요학_ , vol. 46, pp. 153–195, 2016.
  * [2] G. R. Sung, _Controversial Topics in the Research of Korean Music Theory, 한국음악이론 연구의 쟁점_. Publishing department of the academy of Korean studies(한국학중앙연구원출판부), 2020.
  * [3] H. J. Kim, _Transcription and Interpretation of Korean Folk Songs(민요의 채보와 해석)_. Publish Company Minsokwon(민속원), 2013.
  * [4] G. K. Koduri, S. Gulati, P. Rao, and X. Serra, “Rāga recognition based on pitch distribution methods,” _Journal of New Music Research_ , vol. 41, no. 4, pp. 337–350, 2012.
  * [5] S. Rosenzweig, F. Scherbaum, and M. Müller, “Detecting stable regions in frequency trajectories for tonal analysis of traditional georgian vocal music.” in _Proc. of Int. Society of Music Information Retrieval Conf. (ISMIR)_ , 2019, pp. 352–359.
  * [6] F. Scherbaum, N. Mzhavanadze, S. Rosenzweig, and M. Müller, “Tuning systems of traditional georgian singing determined from a new corpus of field recordings,” _Musicologist_ , vol. 6, no. 2, pp. 142–168, 2022.
  * [7] N. Kroher, E. Gómez, A. Chaachoo, M. Sordo, J.-M. Díaz-Báñez, F. Gómez, and J. Mora, _Computational Ethnomusicology: A Study of Flamenco and Arab-Andalusian Vocal Music", bookTitle="Springer Handbook of Systematic Musicology_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2018, pp. 885–897.
  * [8] S. I. Choi, “Articles on recordings : ‘anthology of korean traditional folksongs’ about the project and the records published(창간 10 주년 기념호: 음반;’한국민요대전’사업과 음반 발간),” _Korean Recording Studies(한국음반학)_ , vol. 10, pp. 459–480, 2000\.
  * [9] I. G. Bae, “Musicological study of the tori of chungcheong province : focusing on the rice field weeding folksongs contained in 『mbc hankuk minyo daejeon (the compilation of korean folksongs)』(충청도 토리를 찾기 위한 시론:[mbc 한국민요대전] 수록 논매기소리를 중심으로),” _The Society Of Korean Folk Song(한국민요학_ , vol. 12, pp. 159–184, 2003.
  * [10] N. kyong Choi, “Distribution of folk songs and its musical characteristics(1) (민요의 종류와 음악적 특성의 분포 연구 (1)-모심는 소리, 노젓는 소리, 만선 풍장소리를 중심으로),” _Research Institute of Korean Studies(민족문화연구)_ , vol. 41, pp. 85–127, 2004.
  * [11] Y. W. Kim, “"a study of the yoengnam folk song, eosayong(嶺南民謠 어사용의 音組織 硏究),” _The Society Of Korean Folk Song(The Society Of Korean Folk Song(한국민요학_ , vol. 6, pp. 45–131, 1999.
  * [12] B. H. Lee, “Music and folk song of central and western region and tori(경서토리권의 무가, 민요),” _Collection of Korean Music Studies(나운영박사 회갑기념 한국음악논총)_ , 1982.
  * [13] ——, “Music culture of the menari-tori folk song(메나리토리 무가 민요권의 음악문화),” _Koanthro(한국문화인류학)_ , vol. 15, pp. 233–249, 1983.
  * [14] D. W. Baek, “The definition and interpretation of terminology of korean musicology, 한국음악학에서의 용어 개념 정의와 해석,” _National Folk Museum of Korea(민속학연구)_ , no. 14, pp. 193–230, 2004\.
  * [15] J. W. Kim, J. Salamon, P. Li, and J. P. Bello, “Crepe: A convolutional representation for pitch estimation,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 161–165.
  * [16] G. K. Koduri, J. Serrà Julià, and X. Serra, “Characterization of intonation in carnatic music by parametrizing pitch histograms,” in _Proc. of the 13th Int. Society for Music Information Retrieval Conf._ International Society for Music Information Retrieval (ISMIR), 2012.
  * [17] A. C. Gedik and B. Bozkurt, “Pitch-frequency histogram-based music information retrieval for turkish music,” _Signal Processing_ , vol. 90, no. 4, pp. 1049–1063, 2010.
  * [18] N. Pretto, B. Bozkurt, R. Caro Repetto, and X. Serra, “Nawba recognition for arab-andalusian music using templates from music scores,” in _Proc. of the 15th Sound and Music Computing Conference (SMC2018)_. Cyprus University of Technology, 2018, pp. 394–99.
  * [19] B. Nikzat and R. Caro Repetto, “KDC: An open corpus for computational research of dastgāhi music,” in _Proc. of the 23rd Int. Society for Music Information Retrieval Conf._ International Society for Music Information Retrieval (ISMIR), 2022, pp. 321–328.
  * [20] J. Lee, J. Park, and J. Nam, “Representation learning of music using artist, album, and track information,” in _Machine Learning for Music Discovery Workshop, International Conference on Machine Learning_ , 2019.
  * [21] J. Spijkervet and J. A. Burgoyne, “Contrastive learning of musical representations,” in _Proc. of the 22nd Int. Society for Music Information Retrieval (ISMIR)_ , 2021.
  * [22] Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, “Hierarchical attention networks for document classification,” in _Proc. of the 2016 conf. of the North American chapter of the association for computational linguistics: human language technologies_ , 2016, pp. 1480–1489.
  * [23] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [24] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [25] B. H. Lee, “Joga jisihaneun seonbeobgwa torieui gaenyeom(조(調)가 지시하는 선법과 토리의 개념),” _한국음악연구_ , vol. 51, pp. 245–271, 2012.
  * [26] Y. W. Kim, _Introduction to Korean Traditional Music(국악개론)_. Eumaksekye(음악세계), 2020.
  * [27] D. H. Kang, “A study on the folksong’s ecology and cultural meaning of chongson arari in korean folksong arirang(정선아라리의 민요생태와 문화적 의미),” _The Society Of Korean Folk Song(The Society Of Korean Folk Song(한국민요학_ , vol. 23, pp. 257–288, 2008.
  * [28] Y. W. Kim, “A study of the formation of arirang(<아리랑> 형성과정에 대한 음악적 연구),” _한국문학과 예술_ , vol. 7, pp. 5–55, 2011\.
  * [29] ——, “Reexamination of eastern folk song tori －necessity to set the lower category of menari-tori and proposal(동부민요 토리의 재검토-동부민요 하층위 설정의 필요성을 중심으로),” _Studies in Korean music(한국음악연구)_ , vol. 61, pp. 35–64, 2017\.
  * [30] Y. B. Cho, “A study of the musical style of jeju folk songs, 濟州道 民謠의 音樂樣式 硏究,” Ph.D. dissertation, the academy of Korean studies(한국정신문화연구원 한국학대학원), 1997.
  * [31] B. H. Lee, “Research methodology of jejudo tori based on traditional tori features(전통적 토리 특성에 의한 제주도 토리 연구 방법론),” _Korean Recording Studies(한국음반학)_ , vol. 28, pp. 105–130, 2018.
  * [32] E. J. Shin, “A review and proposal of studies of the mode of the folk songs in cheju island(제주 민요 음조직에 대한 연구 검토 및 제언),” _The Society for Korean History Musicology(한국음악사학회)_ , vol. 62, pp. 169–206, 2019.