#  NestE: Modeling Nested Relational Structures for Knowledge Graph Reasoning

Bo Xiong1, Mojtaba Nayyeri1, Linhao Luo2, Zihao Wang1,
Shirui Pan3, Steffen Staab1,4

###### Abstract

Reasoning with knowledge graphs (KGs) has primarily focused on triple-shaped
facts. Recent advancements have been explored to enhance the semantics of
these facts by incorporating more potent representations, such as hyper-
relational facts. However, these approaches are limited to _atomic facts_ ,
which describe a single piece of information. This paper extends beyond
_atomic facts_ and delves into _nested facts_ , represented by quoted triples
where subjects and objects are triples themselves (e.g., ((_BarackObama_ ,
_holds_position_ , _President_), _succeed_by_ , (_DonaldTrump_ ,
_holds_position_ , _President_))). These nested facts enable the expression of
complex semantics like _situations_ over time and _logical patterns_ over
entities and relations. In response, we introduce NestE, a novel KG embedding
approach that captures the semantics of both atomic and nested factual
knowledge. NestE represents each atomic fact as a $1\times 3$ matrix, and each
nested relation is modeled as a $3\times 3$ matrix that rotates the $1\times
3$ atomic fact matrix through matrix multiplication. Each element of the
matrix is represented as a complex number in the generalized 4D hypercomplex
space, including (spherical) quaternions, hyperbolic quaternions, and split-
quaternions. Through thorough analysis, we demonstrate the embedding’s
efficacy in capturing diverse logical patterns over nested facts, surpassing
the confines of first-order logic-like expressions. Our experimental results
showcase NestE’s significant performance gains over current baselines in
triple prediction and conditional link prediction. The code and pre-trained
models are open available at https://github.com/xiongbo010/NestE.

## Introduction

Knowledge graphs (KGs) depict relationships between entities, commonly through
triple-shaped facts such as (_JoeBiden_ , _holds_position_ , _VicePresident_).
KG embeddings map entities and relations into a lower-dimensional vector space
while retaining their relational semantics. This empowers the effective
inference of missing relationships between entities directly from their
embeddings. Prior research (Bordes et al. 2013; Trouillon et al. 2016) has
primarily centered on embedding triple-shaped facts and predicting the missing
elements of these triples. Yet, to augment the triple-shaped representations,
recent endeavors explore knowledge that extends beyond these triples. For
instance, $n$-ary facts (Liu, Yao, and Li 2020; Fatemi et al. 2020) describe
relationships between multiple entities, and hyper-relational facts (Galkin et
al. 2020; Xiong et al. 2023a) augment primal triples with key-value qualifiers
that provide contextual information. These approaches allow for expressing
complex semantics and enable answering more sophisticated queries with
additional knowledge (Alivanistos et al. 2022).

Figure 1: An example of a nested factual KG consisting of 1) a set of atomic
facts describing the relationship between entities and 2) a set of nested
facts describing the relationship between atomic facts. Nested factual
relations are colored and they either describe situations in/over time (e.g.,
_succeed_by_ and _works_for_when_) or logical patterns (e.g.,
_implies_profession_ and _implies_language_).

However, these beyond-triple representations typically focus only on
relationships between entities that jointly define an _atomic fact_ ,
overlooking the significance of relationships that describe multiple facts
together. Indeed, within a KG, each atomic fact may have a relationship with
another atomic fact. Consider the following two atomic facts:
$T_{1}$=(_JoeBiden_ , _holds_position_ , _VicePresident_) and
$T_{2}$=(_BarackObama_ , _holds_position_ , _President_). We can depict the
scenario where _JoeBiden_ held the position of _VicePresident_ under the
_President_ _BarackObama_ using a triple
$(T_{1},\emph{works\\_for\\_when},T_{2})$. Such a fact about facts is referred
to as a _nested fact_ 111This is also called a _quoted triple_ in RDF star
(Champin 2022). and the relation connecting these two facts is termed a
_nested relation_. Fig. 1 provides an illustration of a KG containing both
atomic and nested facts.

These nested relations play a crucial role in expressing complex semantics and
queries in two ways: 1) Expressing situations involving facts in or over time.
This facilitates answering complex queries that involve multiple facts. For
example, KG embeddings face challenges when addressing queries like ” _Who was
the president of the USA after _DonaldTrump_?_” because the query about the
primary fact (?, _holds_position_ , _President_) depends on another fact
(_DonaldTrump_ , _holds_position_ , _President_). As depicted in Fig. 1,
_succeed_by_ conveys such temporal situation between these two facts, allowing
the direct response to the query through conditional link prediction. 2)
Expressing logical patterns (implications) using a non-first-order logical
form $\psi\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\phi$. As
illustrated in Fig. 1, (_Location A_ , _uses_language_ , _Language B_)
$\stackrel{{\scriptstyle\emph{implies\\_language}}}{{\rightarrow}}$ (_Location
in A_ , _uses_language_ , _Language B_) represents a logical pattern, as it
holds true for all pairs of (_Location A_ , _Location in A_). Modeling such
logical patterns is crucial as it facilitates generalization. Once these
patterns are learned, new facts adhering to these patterns can be inferred. A
recent study (Chung and Whang 2023) explored link prediction over nested
facts.222In their work, the KG is referred to as a bi-level KG, and the term
”high-level facts” is synonymous with nested facts. However, their method
embeds facts using a multilayer perceptron (MLP), which fails to capture
essential logical patterns and thus has limited generalization capabilities.

In this paper, we introduce NestE, an innovative approach designed to embed
the semantics of both atomic facts and nested facts that enable representing
temporal situations and logical patterns over facts. NestE represents each
atomic fact as a $1\times 3$ hypercomplex matrix, with each element signifying
a component of the atomic fact. Furthermore, each nested relation is modeled
through a $3\times 3$ hypercomplex matrix that rotates the $1\times 3$ atomic
fact matrix via a matrix-multiplicative Hamilton product. Our matrix-like
modeling for facts and nested relations demonstrates the capacity to encode
diverse logical patterns over nested facts. The modeling of these logical
patterns further enables efficient modeling of logical rules that extend
beyond the first-order-logic-like expressions (e.g., Horn rules). Moreover, we
propose a more general hypercomplex embedding framework that extends the
quaternion embedding (Zhang et al. 2019) to include hyperbolic quaternions and
split-quaternions. This generalization of hypercomplex space allows for
expressing rotations over hyperboloid, providing more powerful and distinct
inductive biases for embedding complex structural patterns (e.g.,
hierarchies). Our experimental findings on triple prediction and conditional
link prediction showcase the remarkable performance gain of NestE.

## Related Work

#### Beyond-Triple KGs

To enrich the semantics of triple-base KGs, several lines of work have
explored more powerful representations (Xiong et al. 2023b). Temporal KGs
(Dasgupta, Ray, and Talukdar 2018; Leblay and Chekol 2018; Wang et al. 2023)
introduce an additional timestamp to each triple to specify the temporal
validity of the fact. Hyper-relational facts (Guan et al. 2020; Rosso, Yang,
and Cudré-Mauroux 2020; Galkin et al. 2020; Xiong et al. 2023a) attach a set
of key-value qualifiers to the primal triple, where each qualifier specify
certain semantics of the primal fact. $N$-ary facts (Liu, Yao, and Li 2020,
2021; Fatemi et al. 2020) represent a fact as an abstracted relationship
between $n$ entities. Bilinear models are generalized to n-ary facts by
replacing the bilinear product with multi-linear products (Liu, Yao, and Li
2020, 2021). These representations capture relationships between entities or
between entities and facts, but they do not capture the relationships between
multiple facts.

#### Describing relationships between facts

Rule-based approaches (Niu et al. 2020; Meilicke et al. 2019; Demeester,
Rocktäschel, and Riedel 2016; Guo et al. 2016; Yang, Yang, and Cohen 2017;
Sadeghian et al. 2019) consider relationships between facts, but they are
confined to first-order-logic-like expressions (i.e., Horn rules), i.e.,
$\forall
e_{1},e_{2},e_{3}:(e_{1},r_{1},e_{2})\wedge(e_{2},r_{2},e_{3})\Rightarrow(e_{1},r_{3},e_{3})$,
where there must exist a path connecting $e_{1}$, $e_{2}$, and $e_{3}$ in the
KG. Notably, (Chung and Whang 2023) marked an advancement by examining KG
embeddings with relationships between facts as nested facts, denoted as
$(x,r_{1},y)\xRightarrow{\hat{r}}(p,r_{2},q)$. The proposed embeddings (i.e.,
BiVE-Q and BiVE-B333Note that BiVE-B, despite being described as based on the
biquaternian–BiQUE, employs quaternion space with an additional translation
component based on our analysis of the code.) concatenate the embeddings of
the head, relation, and tail, subsequently embedding them via an MLP. However,
such modeling does do not explicitly capture crucial logical patterns over
nested facts, which bear significant importance in KG embeddings.

#### Algebraic and geometric embeddings

Algebraic embeddings like QuatE (Zhang et al. 2019) and BiQUE (Guo and Kok
2021) represent relations as algebraic operations and score triples using
inner products. They can be viewed as a unification of many earlier functional
(Bordes et al. 2013) and multiplication-based (Trouillon et al. 2016) models.
Geometric embeddings like hyperbolic embeddings (Chami et al. 2020; Balazevic,
Allen, and Hospedales 2019) further extend the functional models to non-
Euclidean hyperbolic space, enabling the representation of hierarchical
relations.

## Preliminaries

A KG is denoted as a graph $G=(\mathcal{V},\mathcal{R},\mathcal{T})$, where
$\mathcal{V}$ represents the set of entities, $\mathcal{R}$ stands for the set
of relation names, and
$\mathcal{T}=\\{(h,r,t):h,t\in\mathcal{V},r\in\mathcal{R}\\}$ represents the
set of triples. We refer to $G$ as an atomic factual KG, and each
$(h,r,t)\in\mathcal{T}$ is referred to as an atomic triple. The nested triple
and nested factual KG are defined as follows.

######  Definition 1 (Nested Triple).

Given an atomic factual KG $G=(\mathcal{V},\mathcal{R},\mathcal{T})$, a set of
nested triples is defined by $\widehat{\mathcal{T}}=\\{\langle
T_{i},\widehat{r},T_{j}\rangle:T_{i},T_{j}\in\mathcal{T},\widehat{r}\in\widehat{\mathcal{R}}\\}$,
where $\mathcal{T}$ is the set of atomic triples and $\widehat{\mathcal{R}}$
is the set of nested relation names.

######  Definition 2 (Nested Factual Knowledge Graph).

Given a KG $G=(\mathcal{V},\mathcal{R},\mathcal{T})$, a set of nested relation
names $\widehat{\mathcal{R}}$, and a set of nested triples
$\widehat{\mathcal{T}}$ defined on $G$ and $\widehat{\mathcal{R}}$, a nested
factual KG is defined as
$\widehat{G}=(\mathcal{V},\mathcal{R},\mathcal{T},\widehat{\mathcal{R}},\widehat{\mathcal{T}})$.

We can now define triple prediction and conditional link prediction (Chung and
Whang 2023) as follows.

######  Definition 3 (Triple Prediction).

Given a nested factual KG
$\widehat{G}=(\mathcal{V},\mathcal{R},\mathcal{T},\widehat{\mathcal{R}},\widehat{\mathcal{T}})$,
the triple prediction problem involves answering a query $\langle
T_{i},\widehat{r},?t\rangle$ or $\langle h?,\widehat{r},T_{j}\rangle$ with
$T_{i},T_{j}\in\mathcal{T}$ and $\widehat{r}\in\widehat{\mathcal{R}}$, where
the variable $?h$ or $?t$ needs to be bounded to an atomic triple within
$\widehat{G}$.

######  Definition 4 (Conditional Link Prediction).

Given a nested factual KG
$\widehat{G}=(\mathcal{V},\mathcal{R},\mathcal{T},\widehat{\mathcal{R}},\widehat{\mathcal{T}})$,
let $T_{i}=(h_{i},r_{i},t_{i})$ and $T_{j}=(h_{j},r_{j},t_{j})$. The
conditional link prediction problem involves queries $\langle
T_{i},\widehat{r},(h_{j},r_{j},?)\rangle$, $\langle
T_{i},\widehat{r},(?,r_{j},t_{j})\rangle$,
$\langle(h_{i},r_{i},?),\widehat{r},T_{j}\rangle$, or
$\langle(?,r_{i},t_{i}),\widehat{r},T_{j}\rangle$, where the variables need to
be bound to entities within $\widehat{G}$.

## NestE: Embedding Atomic and Nested Facts

### Unified Hypercomplex Embeddings

We first extend QuatE (Zhang et al. 2019), a KG embedding in 4D hypercomplex
quaternion space, into a more general 4D hypercomplex number system including
three variations: (spherical) quaternions, hyperbolic quaternions, and split
quaternions. Each of these 4D hypercomplex numbers is composed of one real
component and three imaginary components denoted by
$s+x\textit{i}+y\textit{j}+z\textit{k}$ with $s,x,y,z\in\mathbb{R}$ and
$i,j,k$ being the three imaginary parts. The distinctive feature among these
hyper-complex number systems lies in their multiplication rules of the
imaginary components.

(Spherical) quaternions $\mathcal{Q}$ follow the multiplication rules:

$\begin{array}[]{l}\textit{i}^{2}=\textit{j}^{2}=\textit{k}^{2}=1,\\\
\textit{ij}=\textit{k}=-ji,\textit{jk}=\textit{i}=-kj,\textit{ki}=\textit{j}=-ik.\end{array}$
(1)

Hyperbolic quaternions $\mathcal{H}$ follow the multiplication rules:

$\begin{array}[]{l}\textit{i}^{2}=-1,\textit{j}^{2}=\textit{k}^{2}=1,\\\
\textit{ij}=\textit{k},\textit{jk}=-\textit{i},\textit{ki}=\textit{j},\
\textit{ji}=-\textit{k},\textit{kj}=\textit{i},\textit{ik}=-\textit{j}.\end{array}$
(2)

Split quaternions $\mathcal{S}$ follow the multiplication rules:

$\begin{array}[]{l}\textit{i}^{2}=-1,\textit{j}^{2}=\textit{k}^{2}=1,\\\
\textit{ij}=\textit{k},\textit{jk}=-\textit{i},\textit{ki}=\textit{j},\
\textit{ji}=-\textit{k},\textit{kj}=\textit{i},\textit{ik}=-\textit{j}.\end{array}$
(3)

#### Geometric intuitions

The distinctions in the multiplication rules of various hypercomplex numbers
give rise to different geometric spaces that provide suitable inductive biases
for representing different types of relations. Specifically, spherical
quaternions, hyperbolic quaternions, and split quaternions with the same norm
$c$ correspond to 4D hypersphere, Lorentz model of hyperbolic space (i.e., the
upper part of the double-sheet hyperboloid), and pseudo-hyperboloid (i.e.,
one-sheet hyperboloid, with curvature $\sqrt{c}$, respectively. These are
denoted as follows:

$\begin{array}[]{l}|\mathcal{Q}|=s^{2}+x^{2}+y^{2}+z^{2}=c>0\
(\text{hypersphere})\\\ |\mathcal{H}|=s^{2}-x^{2}-y^{2}-z^{2}=c>0\
(\text{Lorentz hyperbolic space})\\\
|\mathcal{S}|=s^{2}+x^{2}-y^{2}-z^{2}=c>0\ (\text{pseudo-hyperboloid}).\\\
\end{array}$ (4)

These spaces have well-known characteristics: spherical spaces are adept at
modeling cyclic relations (Wang et al. 2021), hyperbolic spaces provide
geometric inductive biases for hierarchical relations (Chami et al. 2020), and
the pseudo-hyperboloid (Xiong et al. 2022) offers a balance between spherical
and hyperbolic spaces, making it suitable for embedding both cyclic and
hierarchical relations. Moreover, by representing relations as geometric
rotations over these spaces (i.e., Hamilton product), fundamental logical
patterns such as symmetry, inversion, and compositions can be effectively
inferred (Zhang et al. 2019; Chami et al. 2020; Xiong et al. 2022). Our
proposed embeddings can be viewed as a unification of previous approaches that
leverages these geometric inductive biases in these geometric spaces within a
single geometric algebraic framework.

For convenience, we parameterize each entity and relation as a Cartesian
product of $d$ 4D hypercomplex numbers
$\mathbf{s}+\mathbf{x}\textit{i}+\mathbf{y}\textit{j}+\mathbf{z}\textit{k}$,
where $\mathbf{s},\mathbf{x},\mathbf{y},\mathbf{z}\in\mathbb{R}^{d}$. This
enables us to define all algebraic operations involving these hypercomplex
vectors in an element-wise manner.

### Atomic Fact Embeddings

Each atomic relation is represented by a rotation hypercomplex vector
$\mathbf{r}_{\theta}$ and a translation hypercomplex vector $\mathbf{r}_{b}$.
For a given triple $(h,r,t)$, we apply the following operation:

$\mathbf{h}^{\prime}=(\mathbf{h}\oplus\mathbf{r}_{b})\otimes\mathbf{r}_{\theta},$
(5)

where $\oplus$ and $\otimes$ stand for addition and Hamilton product between
hypercomplex numbers, respectively. The addition involves an element-wise sum
of each hypercomplex component. The Hamilton product rotates the head entity.
To ensure proper rotation on the unit sphere, we normalize the rotation
hypercomplex number
$\mathbf{r}_{\theta}=\mathbf{s}_{r}^{\theta}+\mathbf{x}_{r}^{\theta}\textit{i}+\mathbf{y}_{r}^{\theta}\textit{j}+\mathbf{z}_{r}^{\theta}\textit{k}$
by
$\mathbf{r}_{\theta}=\frac{\mathbf{s}_{r}^{\theta}+\mathbf{x}_{r}^{\theta}\boldsymbol{i}+\mathbf{y}_{r}^{\theta}\boldsymbol{j}+\mathbf{z}_{r}^{\theta}\boldsymbol{k}}{\sqrt{{\mathbf{s}_{r}^{\theta}}^{2}+{\mathbf{x}_{r}^{\theta}}^{2}+{\mathbf{y}_{r}^{\theta}}^{2}+{\mathbf{z}_{r}^{\theta}}^{2}}}$.
Hamilton product is defined by combining the components of the hypercomplex
numbers.

$\begin{split}&\mathbf{h}^{\prime}=\mathbf{h}\otimes\mathbf{r}_{\theta}\\\
&=\left(\mathbf{s}_{h}\circ\mathbf{s}_{r}^{\theta}\circ
1+\mathbf{x}_{h}\circ\mathbf{x}_{r}^{\theta}\circ
i^{2}+\mathbf{y}_{h}\circ\mathbf{y}_{r}^{\theta}\circ
j^{2}+\mathbf{z}_{h}\circ\mathbf{z}_{r}^{\theta}\circ k^{2}\right)\\\
&+\left(\mathbf{s}_{h}\circ\mathbf{x}_{r}^{\theta}\circ
i+\mathbf{x}_{h}\circ\mathbf{s}_{r}^{\theta}\circ
i+\mathbf{y}_{h}\circ\mathbf{z}_{r}^{\theta}\circ
jk+\mathbf{z}_{h}\circ\mathbf{y}_{r}^{\theta}\circ kj\right)\\\
&+\left(\mathbf{s}_{h}\circ\mathbf{y}_{r}^{\theta}\circ
j+\mathbf{x}_{h}\circ\mathbf{z}_{r}^{\theta}\circ
ik+\mathbf{y}_{h}\circ\mathbf{s}_{r}^{\theta}\circ
j+\mathbf{z}_{h}\circ\mathbf{x}_{r}^{\theta}\circ ik\right)\\\
&+\left(\mathbf{s}_{h}\circ\mathbf{z}_{r}^{\theta}\circ
k+\mathbf{x}_{h}\circ\mathbf{y}_{r}^{\theta}\circ
ij+\mathbf{y}_{h}\circ\mathbf{x}_{r}^{\theta}\circ
ij+\mathbf{z}_{h}\circ\mathbf{s}_{r}^{\theta}\circ k\right)\\\
&=\mathbf{s}_{h^{\prime}}+\mathbf{x}_{h^{\prime}}\textit{i}+\mathbf{y}_{h^{\prime}}\textit{j}+\mathbf{z}_{h^{\prime}}\textit{k},\end{split}$
(6)

where the multiplication of imaginary components follows the rules (Eq.1-3) of
the chosen hypercomplex systems.

The scoring function $\phi(h,r,t)$ is defined as:

$\phi(h,r,t)=\langle\mathbf{h}^{\prime},\mathbf{t}\rangle=\left\langle\mathbf{s}_{h^{\prime}},\mathbf{s}_{t}\right\rangle+\left\langle\mathbf{x}_{h^{\prime}},\mathbf{x}_{t}\right\rangle+\left\langle\mathbf{y}_{h^{\prime}},\mathbf{y}_{t}\right\rangle+\left\langle\mathbf{z}_{h^{\prime}},\mathbf{z}_{t}\right\rangle,$
(7)

where $\langle\cdot,\cdot\rangle$ represents the inner product.

### Nested Fact Embeddings

To represent an atomic fact $(h,r,t)$ without losing information, we embed
each atomic triple as a $1\times 3$ matrix, where each column corresponds to
the embedding of the respective element. Consequently, we have
$\mathbf{T}_{i}=\left[\mathbf{h}_{i},\mathbf{r}_{i},\mathbf{t}_{i}\right]$.

To embed various shapes of nested relations between $T_{i}$ and $T_{j}$ with
relation $\widehat{r}$, we model each nested relation using a $1\times 3$
translation matrix and a $3\times 3$ rotation matrix, where each element is a
4D hypercomplex number. Specifically, we first translate the head triple
$\mathbf{T}_{i}$ with $\mathbf{\widehat{r}}_{b}$, followed by applying a
matrix-like rotation of $\mathbf{\widehat{r}}_{\theta}$, as defined by

$\mathbf{T}i^{\prime}=(\mathbf{T}i\oplus_{1\times
3}\mathbf{\widehat{r}}_{b})\otimes_{3\times 3}\mathbf{\widehat{r}}_{\theta},$
(8)

where the matrix addition $\oplus_{1\times 3}$ is performed through an
element-wise summation of the hypercomplex components within the matrices. The
matrix-like Hamilton product $\otimes_{3\times 3}$ is defined as a product
akin to matrix multiplication:

$\begin{split}\mathbf{T}_{i}^{\prime}=\mathbf{T}_{i}\otimes_{3\times
3}\mathbf{\widehat{r}}_{\theta}=\left[\begin{array}[]{c}\mathbf{h}_{i}\\\
\mathbf{r}_{i}\\\ \mathbf{t}_{i}\\\
\end{array}\right]^{\top}\times\left[\begin{array}[]{ccc}\mathbf{\widehat{r}}_{11}^{\theta}&\mathbf{\widehat{r}}_{12}^{\theta}&\mathbf{\widehat{r}}_{13}^{\theta}\\\
\mathbf{\widehat{r}}_{21}^{\theta}&\mathbf{\widehat{r}}_{22}^{\theta}&\mathbf{\widehat{r}}_{23}^{\theta}\\\
\mathbf{\widehat{r}}_{31}^{\theta}&\mathbf{\widehat{r}}_{32}^{\theta}&\mathbf{\widehat{r}}_{33}^{\theta}\\\
\end{array}\right]=\\\
\left[\begin{array}[]{c}\mathbf{h}_{i}\otimes\mathbf{\widehat{r}}_{11}^{\theta}+\mathbf{r}_{i}\otimes\mathbf{\widehat{r}}_{21}^{\theta}+\mathbf{t}_{i}\otimes\mathbf{\widehat{r}}_{31}^{\theta}\\\
\mathbf{h}_{i}\otimes\mathbf{\widehat{r}}_{12}^{\theta}+\mathbf{r}_{i}\otimes\mathbf{\widehat{r}}_{22}^{\theta}+\mathbf{t}_{i}\otimes\mathbf{\widehat{r}}_{32}^{\theta}\\\
\mathbf{h}_{i}\otimes\mathbf{\widehat{r}}_{13}^{\theta}+\mathbf{r}_{i}\otimes\mathbf{\widehat{r}}_{23}^{\theta}+\mathbf{t}_{i}\otimes\mathbf{\widehat{r}}_{33}^{\theta}\\\
\end{array}\right]^{\top}=\left[\begin{array}[]{c}\mathbf{h}_{i}^{\prime}\\\
\mathbf{r}_{i}^{\prime}\\\ \mathbf{t}_{i}^{\prime}\\\
\end{array}\right]^{\top},\end{split}$ (9)

where $\otimes$ is the Hamilton product.

#### Remarks

This matrix-like modeling of nested facts provides flexibility to capture
diverse shapes of logical patterns inherent in nested relations. In essence,
different shapes of situations or patterns can be effectively modeled by
manipulating the $3\times 3$ rotation matrix. For instance, relational
implications can be represented using a diagonal matrix, while inversion can
be captured using an anti-diagonal matrix. See theoretical justification for
further analysis.

To assess the plausibility of the nested fact $(T_{i},\widehat{r},T_{j})$, we
calculate the inner product between the transformed head
$\mathbf{T}_{i}^{\prime}$ fact and the tail fact $\mathbf{T}_{j}$ as:

$\rho(T_{i},\widehat{r},T_{j})=\langle\mathbf{T}_{i}^{\prime},\mathbf{T}_{j}\rangle,$
(10)

where $\langle\cdot,\cdot\rangle$ denotes the matrix inner product.

#### Learning objective

We sum up the loss of atomic fact embedding $\mathcal{L}_{\text{atomic}}$, the
loss of nested fact embedding $\mathcal{L}_{\text{meta}}$, and additionally
the loss term $\mathcal{L}_{\text{aug}}$ for augmented triples generated by
random walking as used in (Chung and Whang 2023). The overall loss is defined
as

$\mathcal{L}=\mathcal{L}_{\text{atomic}}+\lambda_{1}\mathcal{L}_{\text{nested}}+\lambda_{2}\mathcal{L}_{\text{aug}}$
(11)

where $\lambda_{1}$ and $\lambda_{2}$ are the weight hyperparameters
indicating the importance of each loss. Negative sampling is applied by
randomly replacing one of the head or tail entity/triple. These losses are
defined as follows:

$\begin{array}[]{@{} *{6}{>{\displaystyle}c}
@{}}\mathcal{L}_{\text{atomic}}=\sum_{(h,r,t)\in\mathcal{T}}g\left(-\phi\left(h,r,t\right)\right)+\sum_{g\left(\left(h^{\prime},r^{\prime},t^{\prime}\right)\right)\notin\mathcal{T}}g\left(\phi\left(h^{\prime},r^{\prime},t^{\prime}\right)\right)\\\
\mathcal{L}_{\text{aug}}=\sum_{(h,r,t)\in\mathcal{T}^{\prime}}g\left(-\phi\left(h,r,t\right)\right)+\sum_{(h^{\prime},r^{\prime},t^{\prime})\notin\mathcal{T}^{\prime}}g\left(\phi(h^{\prime},r^{\prime},t^{\prime})\right)\\\
\mathcal{L}_{\text{nested}}=\sum_{(T_{i},\widehat{r},T_{j})\notin\widehat{\mathcal{T}}}g\left(-\rho(T_{i},\widehat{r},T_{j})\right)+\sum_{(T_{i}^{\prime},\widehat{r},T_{j}^{\prime})\notin\widehat{\mathcal{T}}}g\left(\rho(T_{i}^{\prime},\widehat{r},T_{j}^{\prime})\right),\\\
\end{array}$ (12)

where $g=\log(1+\exp(x))$ and $\mathcal{T}^{\prime}$ is the set of augmented
triples.

### Theoretical Justification

Modeling logical patterns is of great importance for KG embeddings because it
enables generalization, i.e., once the patterns are learned, new facts that
respect the patterns can be inferred. A logical pattern is a logical form
$\psi\rightarrow\phi$ with $\psi$ and $\phi$ being the body and head, implying
that if the body is satisfied then the head must also be satisfied.

#### First-order-logic-like logical patterns

Existing KG embeddings studied logical patterns expressed in the first-order-
logic-like form. Prominent examples include symmetry $\forall
h,t\colon(h,r,t)\rightarrow(t,r,h)$, anti-symmetry $\forall
h,t\colon(h,r,t)\rightarrow\neg(h,r,t)$, inversion $\forall
h,t\colon(h,r_{1},t)\rightarrow(t,r_{2},h)$ and composition $\forall
e_{1},e_{2},e_{3}\colon(e_{1},r_{1},e_{2})\wedge(e_{2},r_{2},e_{3})\rightarrow(e_{1},r_{3},e_{3})$.

######  Proposition 1.

NestE can infer symmetry, anti-symmetry, inversion, and composition,
regardless of the specific choices of hypercomplex number systems.

This proposition holds because NestE subsumes ComplEx (Trouillon et al. 2016)
(i.e., 4D complex numbers generalize 2D complex numbers).

Figure 2: A structural illustration of different shapes of logical patterns,
where the colored circles are free variables.

#### Logical patterns over nested facts

We extend the vanilla logical patterns in KGs to include nested facts. This
can be expressed in a non-first-order-logic-like form
$\psi\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\phi$.

  * •

Relational symmetry (R-symmetry): an atomic relation $r$ is symmetric w.r.t a
nested relation $\widehat{r}$ if $\forall x,y\in\mathcal{E},\langle
x,r,y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\leftrightarrow}}\langle
y,r,x\rangle$.

  * •

Relational inverse (R-inverse): two atomic relations $r_{1}$ and $r_{2}$ are
inverse w.r.t a nested relation $\widehat{r}$ if $\forall x,y\in\mathcal{E},$
$(\langle
x,r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\leftrightarrow}}\langle
y,r_{2},x\rangle)$.

  * •

Relational implication (R-implication): an atomic relation $r_{1}$ implies a
atomic relation $r_{2}$ w.r.t a nested relation $\widehat{r}$ if $\forall
x,y\in\mathcal{E},(\langle
x,r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x,r_{2},y\rangle)$.

  * •

Relational inverse implication (R-Inv-implication): an atomic relation $r_{1}$
inversely implies an atomic relation $r_{2}$ w.r.t a nested relation
$\widehat{r}$ if $\forall x,y\in\mathcal{E},(\langle
x,r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
y,r_{2},x\rangle)$.

  * •

Entity implication (E-implication): an entity $x_{1}$ (resp. $y_{1}$) implies
entity $x_{2}$ (resp. $y_{2}$) w.r.t an atomic relation $r$ and a nested
relation $\widehat{r}$ if $\forall y\in\mathcal{E},(\langle
x_{1},r,y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x_{2},r,y\rangle)$ (resp. $\forall x\in\mathcal{E},(\langle
x,r,y_{1}\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x,r,y_{2}\rangle)$ ).

  * •

Entity relational implication (E-R-implication): an entity $x_{1}$ and
relation $r_{1}$ (resp. $y_{1}$ and relation $r_{1}$) implies entity $x_{2}$
and relation $r_{2}$ (resp. $y_{2}$ and relation $r_{2}$) w.r.t a nested
relation $\widehat{r}$ if $\forall y\in\mathcal{E},(\langle
x_{1},r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x_{2},r_{2},y\rangle)$ (resp. $\forall x\in\mathcal{E},(\langle
x,r_{1},y_{1}\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x,r_{2},y_{2}\rangle)$).

  * •

Entity relational inverse implication (E-R-Inv-implication): an entity $x_{1}$
and relation $r_{1}$ (resp. $y_{1}$ and relation $r_{1}$) inversely implies
entity $x_{2}$ and relation $r_{2}$ (resp. $y_{2}$ and relation $r_{2}$) w.r.t
a nested relation $\widehat{r}$ if $\forall y\in\mathcal{E},(\langle
x_{1},r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
y,r_{2},x_{2}\rangle)$ (resp. $\forall x\in\mathcal{E},(\langle
x,r_{1},y_{1}\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
y_{2},r_{2},x\rangle)$).

  * •

Dual Entity implication (Dual E-implication): an entity pair ($x_{1}$,
$x_{2}$) implies another entity pair ($y_{1}$, $y_{2}$) iff both ($x_{1}$,
$y_{1}$) and ($x_{2}$, $y_{2}$) satisfy E-implication.

Fig. 2 illustrates the structure of the introduced patterns and Table 1
presents exemplary patterns of nested facts.

Table 1: Exemplary logical patterns of nested facts.

Pattern | $\widehat{r}$ | triple template |
---|---|---|---
R-Symmetry | EquivalentTo | (Person A, IsMarriedTo, Person B) |
(Person B, IsMarriedTo, Person A) |
R-Inverse | EquivalentTo | (Location A, UsesLanguage, Language B) |
(Language B, IsSpokenIn, Location A) |
R-Implication | ImpliesLocation | (Country A, CapitalIsLocatedIn, City B) |
(Country A, Contains, City B) |
R-Inv-Implication | ImpliesLocation | (Organization A, Headquarter, Location B) |
(Location B, Contains, Organization A) |
E-Implication | ImpliesTimeZone | (Location A, TimeZone, Time Zone B) |
(Location in A, TimeZone, Time Zone B) |
E-R-Implication | ImpliesProfession | (Person A, HoldsPosition, Government Position B) |
(Person A, IsA, Politician) |
E-R-Inv-Implication | ImpliesProfession | (Work A, CinematographyBy, Person B) |
(Person B, IsA, Cinematographer) |
Dual E-Implication | ImpliesLocation | (Location A, Contains, Location B) |
(Location containing A, Contains, Location in B) |

######  Proposition 2.

NestE can infer R-symmetry, R-inverse, R-implication, R-Inv-implication,
E-implication, E-R-implication, E-R-Inv-implication, and Dual E-implication.

###### Proof Sketch.

To infer different logical patterns via different free variables, we can set
some elements of the relation matrix to be zero-valued or one-valued complex
numbers. For example, the implication and inverse implication relations can be
inferred by setting the matrix to be diagonal or anti-diagonal. See Appendix
for details. ∎

## Experimental Results

### Experiment Setup

#### Datasets

We utilize three benchmark KGs: FBH, FBHE, and DBHE, that contain nested facts
and are constructed by (Chung and Whang 2023). FBH and FBHE are based on
FB15K237 from Freebase (Bollacker et al. 2008) while DBHE is based on DB15K
from DBpedia (Auer et al. 2007). FBH contains only nested facts that can be
inferred from the triple facts, e.g., _prerequisite_for_ and
_implies_position_ , while FBHE and DBHE further contain externally-sourced
knowledge crawled from Wikipedia articles, e.g., _next_almaMater_ and
_transfers_to_. The authors of (Chung and Whang 2023) spent six weeks manually
defining these nested facts and adding them to the KGs. The dataset details
are presented in Table 2. We split $\mathcal{T}$ and $\widehat{\mathcal{T}}$
into training, validation, and test sets in an 8:1:1 ratio.

Table 2: Statistics of $\widehat{G}=(V,R,\mathcal{T},\widehat{\mathcal{R}},\widehat{\mathcal{T}})$. $|\mathcal{T}|^{\prime}$ denotes the number of atomic triples involved in the nested triples. | $|V|$ | $|R|$ | $|\mathcal{T}|$ | $|\widehat{\mathcal{R}}|$ | $|\widehat{\mathcal{T}}|$ | $|\mathcal{T}|^{\prime}$
---|---|---|---|---|---|---
FBH | 14,541 | 237 | 310,117 | 6 | 27,062 | 33,157
FBHE | 14,541 | 237 | 310,117 | 10 | 34,941 | 33,719
DBHE | 12,440 | 87 | 68,296 | 8 | 6,717 | 8,206

#### Baselines

We consider BiVE-Q and BiVE-B (Chung and Whang 2023) as our major baselines as
they are specifically designed for KGs with nested facts and have demonstrated
significant improvements over triple-based methods. We also compare some rule-
based approaches as they indirectly consider relations between facts in first-
order-logic-like expression, including Neural-LP (Yang, Yang, and Cohen 2017),
DRUM (Sadeghian et al. 2019), and AnyBURL (Meilicke et al. 2019). We further
include QuatE (Zhang et al. 2019) and BiQUE (Guo and Kok 2021) as they are the
SoTA triple-based methods and they are also based on 4D hypercomplex numbers.
However, these triple-based methods do not directly apply to the nested facts.
Following (Chung and Whang 2023), we create a new triple-based KG $G_{T}$
where the atomic facts are converted into entities and nested facts are
converted into triples (see Appendix for details). For our approach, we
implement three variants of NestE: NestE-Q (using quaternions), NestE-H (using
hyperbolic quaternions), NestE-S (split quaternions), as well as their
counterparts with translations: NestE-QB, NestE-HB, and NestE-SB. In the
Appendix, we also extend BiVE-Q and BiVE-B to other hypercomplex numbers:
BiVE-H, BiVE-HB BiVE-S, and BiVE-SB for further comparison. We employ three
standard metrics: Filtered MR (Mean Rank), MRR (Mean Reciprocal Rank), and
Hit@$10$. We report the mean performance over $10$ random seeds for each
method, and the relatively small standard deviations are omitted.

#### Implementation details

We implement the framework based on OpenKE 444https://github.com/thunlp/OpenKE
and the code 555https://github.com/bdi-lab/BiVE/. We train our methods on
triple prediction and evaluate them on other tasks. The dimensionality is set
to be $d=200$. We reuse the hyperparameters as used in (Chung and Whang 2023)
and do not perform further hyperparameter searches. The detailed
hyperparameter settings can be found in the Appendix.

Table 3: Results of triple prediction. Shaded numbers are better results than the best baseline. The best scores are boldfaced and the second best scores are underlined. * denotes results taking from (Chung and Whang 2023).  | FBH | FBHE | DBHE
---|---|---|---
| MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$) | MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$) | MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$)
QuatE* | 145603.8 | 0.103 | 0.114 | 94684.4 | 0.101 | 0.209 | 26485.0 | 0.157 | 0.179
BiQUE* | 81687.5 | 0.104 | 0.115 | 61015.2 | 0.135 | 0.205 | 19079.4 | 0.163 | 0.185
Neural-LP* | 115016.6 | 0.070 | 0.073 | 90000.4 | 0.238 | 0.274 | 21130.5 | 0.170 | 0.209
DRUM* | 115016.6 | 0.069 | 0.073 | 90000.3 | 0.261 | 0.274 | 21130.5 | 0.166 | 0.209
AnyBURL* | 108079.6 | 0.096 | 0.108 | 83136.8 | 0.191 | 0.252 | 20530.8 | 0.177 | 0.214
BiVE-Q | 6.20 | 0.855 | 0.941 | 8.35 | 0.711 | 0.866 | 3.63 | 0.687 | 0.958
BiVE-B | 8.63 | 0.833 | 0.924 | 9.53 | 0.705 | 0.860 | 4.66 | 0.718 | 0.945
NestE-Q (Ours) | 6.56 | 0.863 | 0.953 | 5.77 | 0.811 | 0.943 | 3.51 | 0.809 | 0.960
NestE-H (Ours) | 4.69 | 0.858 | 0.964 | 3.99 | 0.781 | 0.943 | 2.65 | 0.806 | 0.969
NestE-S (Ours) | 3.87 | 0.867 | 0.977 | 3.60 | 0.795 | 0.947 | 2.55 | 0.809 | 0.966
NestE-QB (Ours) | 6.04 | 0.898 | 0.958 | 5.55 | 0.845 | 0.947 | 2.54 | 0.847 | 0.973
NestE-HB (Ours) | 3.82 | 0.899 | 0.971 | 3.53 | 0.828 | 0.955 | 2.62 | 0.842 | 0.972
NestE-SB (Ours) | 3.34 | 0.922 | 0.982 | 3.05 | 0.851 | 0.962 | 2.07 | 0.862 | 0.984
Table 4: Results of conditional link prediction. Shaded numbers are better results than the best baseline. The best scores are boldfaced and the second best scores are underlined. * denotes results taking from (Chung and Whang 2023).  | FBH | FBHE | DBHE
---|---|---|---
| MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$) | MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$) | MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$)
QuatE* | 163.7 | 0.346 | 0.494 | 1546.4 | 0.124 | 0.189 | 551.6 | 0.208 | 0.309
BiQUE* | 111.0 | 0.423 | 0.641 | 90.1 | 0.387 | 0.617 | 29.5 | 0.378 | 0.677
Neural-LP* | 185.9 | 0.433 | 0.648 | 146.2 | 0.466 | 0.716 | 32.2 | 0.517 | 0.756
DRUM* | 262.7 | 0.394 | 0.555 | 207.6 | 0.413 | 0.620 | 49.0 | 0.470 | 0.732
AnyBURL* | 228.5 | 0.380 | 0.563 | 166.0 | 0.418 | 0.607 | 81.7 | 0.403 | 0.594
BiVE-Q | 4.33 | 0.826 | 0.948 | 6.56 | 0.761 | 0.886 | 2.69 | 0.852 | 0.971
BiVE-B | 5.34 | 0.836 | 0.940 | 7.49 | 0.761 | 0.872 | 2.91 | 0.858 | 0.967
NestE-Q (Ours) | 1.70 | 0.930 | 0.986 | 2.89 | 0.863 | 0.948 | 1.68 | 0.930 | 0.987
NestE-H (Ours) | 1.68 | 0.909 | 0.987 | 2.87 | 0.843 | 0.945 | 1.82 | 0.912 | 0.986
NestE-S (Ours) | 1.54 | 0.925 | 0.991 | 3.04 | 0.850 | 0.941 | 1.76 | 0.910 | 0.988
NestE-QB (Ours) | 1.71 | 0.935 | 0.987 | 3.00 | 0.865 | 0.949 | 1.70 | 0.931 | 0.986
Fact-HB (Ours) | 1.60 | 0.924 | 0.989 | 2.76 | 0.855 | 0.950 | 1.92 | 0.918 | 0.981
NestE-SB (Ours) | 1.52 | 0.934 | 0.991 | 2.61 | 0.867 | 0.951 | 1.72 | 0.919 | 0.990

### Main Results

#### Triple prediction

Table 3 presents the results of triple prediction. First, it shows that all
triple-based approaches yield relatively modest results compared to BiVE-Q and
BiVE-B, designed specifically for KGs with nested facts. Our approach,
NestE-Q, the quaternionic version, already outperforms the baselines across
most metrics. Particularly notable are the pronounced enhancements in FBHE and
DBHE, with MRR improvements of 14.1% and 17.7% respectively, underscoring the
efficacy of the proposed NestE model. Furthermore, NestE-H and NestE-S
demonstrate heightened performance over NestE-Q across various evaluation
metrics, particularly in terms of MR. This highlights the advantages that
hyperbolic quaternions and split quaternions offer over standard quaternions.
Impressively, the split quaternionic version attains the highest performance,
followed closely by the hyperbolic quaternionic variant. Moreover, through the
incorporation of a hypercomplex translation component, NestE-QB, Fact-HB, and
NestE-SB consistently outperform their non-translation counterparts,
illustrating the advantages of combining multiple transformations (rotation
and translation) within the hypercomplex space.

#### Conditional link prediction

Table 4 shows the outcomes of conditional link prediction. It is evident that
all three NestE variants substantially outperform the two SoTA baselines,
BiVE-Q and BiVE-B, across all datasets. Notably, the best NestE variant
surpasses the baselines by 11.8%, 13.9%, and 8.5% in terms of MRR for FBH,
FBHE, and DBHE, respectively. This remarkable performance gain underscores the
effectiveness of the proposed method. Similar to the trends observed in triple
prediction, the incorporation of translation components in NestE-QB, Fact-HB,
and Fact-SB leads to further improvements over their counterparts without
translation components. This reaffirms the advantages gained from the
integration of multiple hypercomplex transformations. Intriguingly, we noticed
that varying hypercomplex number systems yield the best performance on
different datasets, contrasting the observations from triple prediction. We
conjecture that this stems from the inherent variance in inductive biases
offered by different hypercomplex number systems, making them more suitable
for certain datasets over others. We believe the choices of spaces can be
linked to a hyperparameter that offers flexibility in adapting to diverse
dataset characteristics.

Table 5: Results of base link prediction. The best scores are boldfaced and
the second best scores are underlined. * denotes results taking from (Chung
and Whang 2023).

| FBHE | DBHE
---|---|---
| MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$) | MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$)
QuatE* | 139.0 | 0.354 | 0.581 | 409.6 | 0.264 | 0.440
BiQUE* | 134.9 | 0.356 | 0.583 | 376.6 | 0.274 | 0.446
Neural-LP* | 1942.5 | 0.315 | 0.486 | 2904.8 | 0.233 | 0.357
DRUM* | 1945.6 | 0.317 | 0.490 | 2904.7 | 0.237 | 0.359
AnyBURL* | 342.0 | 0.310 | 0.526 | 879.1 | 0.220 | 0.364
BiVE-Q | 136.13 | 0.369 | 0.603 | 827.18 | 0.271 | 0.428
BiVE-B | 136.54 | 0.370 | 0.607 | 795.59 | 0.274 | 0.422
NestE-Q (Ours) | 131.72 | 0.365 | 0.605 | 749.75 | 0.284 | 0.446
Fact-H (Ours) | 153.00 | 0.349 | 0.593 | 868.82 | 0.266 | 0.423
NestE-S (Ours) | 149.64 | 0.350 | 0.592 | 895.85 | 0.272 | 0.432
NestE-QB (Ours) | 130.13 | 0.371 | 0.608 | 751.18 | 0.289 | 0.443
Fact-HB (Ours) | 155.74 | 0.353 | 0.594 | 801.76 | 0.271 | 0.423
NestE-SB (Ours) | 149.73 | 0.355 | 0.594 | 827.89 | 0.273 | 0.431

#### Base link prediction

Table 5 illustrates the results of base link prediction. Among our approaches,
namely NestE-Q, NestE-H, and NestE-S, we observe competitive or improved
results in comparison to SoTA embedding-based and rule-based methods on the
FBHE and DBHE datasets. The best performance is achieved by NestE-QB, which
outperforms the baselines across a majority of metrics. This outcome
substantiates the fact that the incorporation of nested facts into triple-
based KGs indeed enhances the inference capabilities for base link prediction.

### Ablation Analysis

#### Embedding analysis of logical patterns

To verify whether the learned embeddings capture the inference of logical
patterns over nested facts, we visualized the real part of the embeddings of
the $8$ relations in DBHE. The analysis of the embeddings yields insightful
observations. As shown in Fig. 3, the lower left element and upper right
element of the embedding _EquivalentTo_ are $1$, showcasing that
_EquivalentTo_ predominantly adheres to R-symmetry or R-inverse. On the other
hand, the upper left element and lower right element of the _ImpliesLang._ are
$1$, affirming its alignment with the R-implication rule. Similarly, the
embeddings of _NextAlmaM._ , _TransfersTo_ , and _ImpliesGenre_ indicate high
adherence to E-implicationsas as only one of the corners is $1$. We find that
the embedding of relation _ImpliesProf._ does not have a significant pattern.
We conjecture that this is because _ImpliesProf._ follows many rule patterns
and there exists no global solution that satisfies all rules. See the Appendix
for the statistics of the logical patterns in the datasets.

Figure 3: The visualization of the average of the real component embeddings of
the $8$ nested relations in DBHE.

#### Influence of nested fact embeddings

To evaluate the influence of nested fact embeddings, we perform a comparison
by excluding the loss associated with nested fact embeddings (i.e., setting
$\lambda_{1}=0$). The outcomes presented in Table 6 underscore the significant
enhancements achieved by incorporating nested fact embeddings, particularly
evident in the improvements in MR and H@10 for DBHE.

Table 6: Ablation study on the nested fact embeddings for base link
prediction. Best results are boldfaced.

| FBHE | DBHE
---|---|---
| MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$) | MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$)
NestE-Q ($\lambda_{1}=0$) | 135.38 | 0.368 | 0.604 | 799.97 | 0.281 | 0.431
NestE-Q ($\lambda_{1}=0.5$) | 131.72 | 0.365 | 0.605 | 749.75 | 0.284 | 0.446
NestE-H ($\lambda_{1}=0$) | 154.75 | 0.347 | 0.589 | 922.34 | 0.267 | 0.420
NestE-H ($\lambda_{1}=0.5$) | 153.00 | 0.349 | 0.593 | 868.82 | 0.266 | 0.423
NestE-S ($\lambda_{1}=0$) | 151.84 | 0.347 | 0.589 | 910.16 | 0.272 | 0.427
NestE-S ($\lambda_{1}=0.5$) | 149.64 | 0.350 | 0.592 | 895.85 | 0.272 | 0.432

Table 7: Performance per relation on triple prediction. Freq. indicates the
number of nested facts in the test set.

$\widehat{r}$ | Freq. | NestE-Q | NestE-QB | NestE-H | NestE-HB | NestE-S | NestE-SB
---|---|---|---|---|---|---|---
Equiv.To | 98 | 0.994 | 0.997 | 0.992 | 0.997 | 0.997 | 0.995
ImpliesLang. | 29 | 0.671 | 0.602 | 0.680 | 0.662 | 0.614 | 0.622
ImpliesProf. | 210 | 0.807 | 0.916 | 0.830 | 0.935 | 0.832 | 0.936
ImpliesLocat. | 163 | 0.929 | 0.869 | 0.893 | 0.810 | 0.929 | 0.958
ImpliesTime. | 44 | 0.305 | 0.297 | 0.307 | 0.329 | 0.290 | 0.293
ImpliesGenre | 84 | 0.726 | 0.762 | 0.719 | 0.741 | 0.742 | 0.796
NextAlmaM. | 14 | 0.770 | 0.812 | 0.689 | 0.688 | 0.751 | 0.795
Transf.To | 29 | 0.977 | 0.952 | 0.964 | 0.949 | 0.921 | 0.953

#### Relation-specific performance

In Table 7, we present the performance results for each relation within the
DBHE dataset. Notably, the diverse hypercomplex number systems lead to optimal
performance for different relations. This reiterates our conjecture that
distinct benefits are offered by varying hypercomplex number systems, catering
to the specific characteristics of different relation types. Remarkably, our
findings reveal that the incorporation of a hypercomplex translation component
(as seen in NestE-QB, NestE-HB, and NestE-SB) notably enhances the embeddings
of relations such as _ImpliesProf._ and _ImpliesGenre_ across all variants of
hypercomplex number systems. However, this does not extend to relations like
_ImpliesLocat._ and _ImpliesLang._ , suggesting a more complex relationship
between these specific relations and the hypercomplex translation.

## Conclusion

This paper considers a novel perspective by extending traditional atomic
factual knowledge representation to include nested factual knowledge. This
enables the representation of both temporal situations and logical patterns
that go beyond conventional first-order logic expressions (Horn rules). Our
proposed approach, NestE, presents a family of hypercomplex embeddings capable
of embedding both atomic and nested factual knowledge. This framework
effectively captures essential logical patterns that emerge from nested facts.
Empirical evaluation demonstrates the substantial performance enhancements
achieved by NestE compared to existing baseline methods. Additionally, our
generalized hypercomplex embedding framework unifies previous algebraic (e.g.,
quaternionic) and geometric (e.g., hyperbolic) embedding methods, offering
versatility in embedding diverse relation types. One under-explored version of
our method is to consider scoring function based on distance instead of inner
product. Another interesting avenue for future exploration involves devising
efficient strategies for selecting appropriate algebraic and geometric
operations tailored to specific relations in KGs.

## Acknowledgement

The authors thank the International Max Planck Research School for Intelligent
Systems (IMPRS-IS) for supporting Bo Xiong and Zihao Wang. This research has
been partially funded by Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy - EXC 2075 - 390740016. We
acknowledge the support by the Stuttgart Center for Simulation Science
(SimTech). Bo Xiong is funded by the European Union’s Horizon 2020 research
and innovation programme under the Marie Skłodowska-Curie grant agreement No:
860801. Zihao Wang was funded by the the Bundesministerium für Wirtschaft und
Energie (BMWi), grant aggrement No. 01MK20008F.

## References

  * Alivanistos et al. (2022) Alivanistos, D.; Berrendorf, M.; Cochez, M.; and Galkin, M. 2022.  Query Embedding on Hyper-Relational Knowledge Graphs.  In _ICLR_. OpenReview.net.
  * Auer et al. (2007) Auer, S.; Bizer, C.; Kobilarov, G.; Lehmann, J.; Cyganiak, R.; and Ives, Z. G. 2007\.  DBpedia: A Nucleus for a Web of Open Data.  In _ISWC/ASWC_ , volume 4825 of _Lecture Notes in Computer Science_ , 722–735. Springer.
  * Balazevic, Allen, and Hospedales (2019) Balazevic, I.; Allen, C.; and Hospedales, T. M. 2019.  Multi-relational Poincaré Graph Embeddings.  In _NeurIPS_ , 4465–4475.
  * Bollacker et al. (2008) Bollacker, K. D.; Evans, C.; Paritosh, P. K.; Sturge, T.; and Taylor, J. 2008.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _SIGMOD Conference_ , 1247–1250. ACM.
  * Bordes et al. (2013) Bordes, A.; Usunier, N.; García-Durán, A.; Weston, J.; and Yakhnenko, O. 2013.  Translating Embeddings for Modeling Multi-relational Data.  In _NIPS_ , 2787–2795.
  * Chami et al. (2020) Chami, I.; Wolf, A.; Juan, D.; Sala, F.; Ravi, S.; and Ré, C. 2020.  Low-Dimensional Hyperbolic Knowledge Graph Embeddings.  In _ACL_ , 6901–6914. Association for Computational Linguistics.
  * Champin (2022) Champin, P. 2022.  RDF-star: Paving the Way to the Next generation of Linked Data.  _ERCIM News_ , 2022(128).
  * Chung and Whang (2023) Chung, C.; and Whang, J. J. 2023.  Learning Representations of Bi-Level Knowledge Graphs for Reasoning beyond Link Prediction.  In _AAAI_.
  * Dasgupta, Ray, and Talukdar (2018) Dasgupta, S. S.; Ray, S. N.; and Talukdar, P. P. 2018.  HyTE: Hyperplane-based Temporally aware Knowledge Graph Embedding.  In _EMNLP_ , 2001–2011. Association for Computational Linguistics.
  * Demeester, Rocktäschel, and Riedel (2016) Demeester, T.; Rocktäschel, T.; and Riedel, S. 2016.  Lifted Rule Injection for Relation Embeddings.  In _EMNLP_ , 1389–1399. The Association for Computational Linguistics.
  * Fatemi et al. (2020) Fatemi, B.; Taslakian, P.; Vázquez, D.; and Poole, D. 2020.  Knowledge Hypergraphs: Prediction Beyond Binary Relations.  In _IJCAI_ , 2191–2197. ijcai.org.
  * Galkin et al. (2020) Galkin, M.; Trivedi, P.; Maheshwari, G.; Usbeck, R.; and Lehmann, J. 2020.  Message Passing for Hyper-Relational Knowledge Graphs.  In _EMNLP (1)_ , 7346–7359. Association for Computational Linguistics.
  * Guan et al. (2020) Guan, S.; Jin, X.; Guo, J.; Wang, Y.; and Cheng, X. 2020.  NeuInfer: Knowledge Inference on N-ary Facts.  In _ACL_ , 6141–6151. Association for Computational Linguistics.
  * Guo and Kok (2021) Guo, J.; and Kok, S. 2021.  BiQUE: Biquaternionic Embeddings of Knowledge Graphs.  In _EMNLP (1)_ , 8338–8351. Association for Computational Linguistics.
  * Guo et al. (2016) Guo, S.; Wang, Q.; Wang, L.; Wang, B.; and Guo, L. 2016.  Jointly Embedding Knowledge Graphs and Logical Rules.  In _EMNLP_ , 192–202. The Association for Computational Linguistics.
  * Leblay and Chekol (2018) Leblay, J.; and Chekol, M. W. 2018.  Deriving Validity Time in Knowledge Graph.  In _WWW (Companion Volume)_ , 1771–1776. ACM.
  * Liu, Yao, and Li (2020) Liu, Y.; Yao, Q.; and Li, Y. 2020.  Generalizing Tensor Decomposition for N-ary Relational Knowledge Bases.  In _WWW_ , 1104–1114. ACM / IW3C2.
  * Liu, Yao, and Li (2021) Liu, Y.; Yao, Q.; and Li, Y. 2021.  Role-Aware Modeling for N-ary Relational Knowledge Bases.  In _WWW_ , 2660–2671. ACM / IW3C2.
  * Meilicke et al. (2019) Meilicke, C.; Chekol, M. W.; Ruffinelli, D.; and Stuckenschmidt, H. 2019.  Anytime Bottom-Up Rule Learning for Knowledge Graph Completion.  In _IJCAI_ , 3137–3143. ijcai.org.
  * Nayyeri et al. (2023) Nayyeri, M.; Xiong, B.; Mohammadi, M.; Akter, M. M.; Alam, M. M.; Lehmann, J.; and Staab, S. 2023.  Knowledge Graph Embeddings using Neural Ito Process: From Multiple Walks to Stochastic Trajectories.  In _ACL (Findings)_ , 7165–7179. Association for Computational Linguistics.
  * Nickel, Tresp, and Kriegel (2011) Nickel, M.; Tresp, V.; and Kriegel, H. 2011.  A Three-Way Model for Collective Learning on Multi-Relational Data.  In _ICML_ , 809–816. Omnipress.
  * Niu et al. (2020) Niu, G.; Zhang, Y.; Li, B.; Cui, P.; Liu, S.; Li, J.; and Zhang, X. 2020.  Rule-Guided Compositional Representation Learning on Knowledge Graphs.  In _AAAI_ , 2950–2958. AAAI Press.
  * Rosso, Yang, and Cudré-Mauroux (2020) Rosso, P.; Yang, D.; and Cudré-Mauroux, P. 2020.  Beyond Triplets: Hyper-Relational Knowledge Graph Embedding for Link Prediction.  In _WWW_ , 1885–1896. ACM / IW3C2.
  * Sadeghian et al. (2019) Sadeghian, A.; Armandpour, M.; Ding, P.; and Wang, D. Z. 2019.  DRUM: End-To-End Differentiable Rule Mining On Knowledge Graphs.  In _NeurIPS_ , 15321–15331.
  * Sun et al. (2019) Sun, Z.; Deng, Z.; Nie, J.; and Tang, J. 2019.  RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space.  In _ICLR (Poster)_. OpenReview.net.
  * Trouillon et al. (2016) Trouillon, T.; Welbl, J.; Riedel, S.; Gaussier, É.; and Bouchard, G. 2016\.  Complex Embeddings for Simple Link Prediction.  In _ICML_ , volume 48 of _JMLR Workshop and Conference Proceedings_ , 2071–2080. JMLR.org.
  * Wang et al. (2023) Wang, J.; Wang, B.; Qiu, M.; Pan, S.; Xiong, B.; Liu, H.; Luo, L.; Liu, T.; Hu, Y.; Yin, B.; and Gao, W. 2023.  A Survey on Temporal Knowledge Graph Completion: Taxonomy, Progress, and Prospects.  _CoRR_ , abs/2308.02457.
  * Wang et al. (2021) Wang, S.; Wei, X.; dos Santos, C. N.; Wang, Z.; Nallapati, R.; Arnold, A. O.; Xiang, B.; Yu, P. S.; and Cruz, I. F. 2021.  Mixed-Curvature Multi-Relational Graph Neural Network for Knowledge Graph Completion.  In _WWW_ , 1761–1771. ACM / IW3C2.
  * Wang et al. (2014) Wang, Z.; Zhang, J.; Feng, J.; and Chen, Z. 2014.  Knowledge Graph Embedding by Translating on Hyperplanes.  In _AAAI_ , 1112–1119. AAAI Press.
  * Xiong et al. (2023a) Xiong, B.; Nayyer, M.; Pan, S.; and Staab, S. 2023a.  Shrinking Embeddings for Hyper-Relational Knowledge Graphs.  In _ACL_. International Committee on Computational Linguistics.
  * Xiong et al. (2023b) Xiong, B.; Nayyeri, M.; Daza, D.; and Cochez, M. 2023b.  Reasoning beyond Triples: Recent Advances in Knowledge Graph Embeddings.  In _CIKM_ , 5228–5231. ACM.
  * Xiong et al. (2022) Xiong, B.; Zhu, S.; Nayyeri, M.; Xu, C.; Pan, S.; Zhou, C.; and Staab, S. 2022.  Ultrahyperbolic Knowledge Graph Embeddings.  In _KDD_ , 2130–2139. ACM.
  * Yang et al. (2015) Yang, B.; Yih, W.; He, X.; Gao, J.; and Deng, L. 2015.  Embedding Entities and Relations for Learning and Inference in Knowledge Bases.  In _ICLR (Poster)_.
  * Yang, Yang, and Cohen (2017) Yang, F.; Yang, Z.; and Cohen, W. W. 2017.  Differentiable Learning of Logical Rules for Knowledge Base Reasoning.  In _NIPS_ , 2319–2328.
  * Zhang et al. (2019) Zhang, S.; Tay, Y.; Yao, L.; and Liu, Q. 2019.  Quaternion Knowledge Graph Embeddings.  In _NeurIPS_ , 2731–2741.

## Appendix

##  Appendix A Supplementary Related Work

#### Triple-shaped Representations

The majority of KG embedding models focus on triple-shaped KGs, where each
fact is represented as a triple. Notable instances include the _additive_ (or
_translational_) family (Bordes et al. 2013; Wang et al. 2014; Sun et al.
2019). This family embodies facts as translations, with representations like
$\mathbf{h}+\mathbf{r}\approx\mathbf{t}$ or rotations as
$\mathbf{h}\circ\mathbf{r}\approx\mathbf{t}$. On the other hand, the
_multiplicative_ (or _semantic matching_) family (Nickel, Tresp, and Kriegel
2011; Yang et al. 2015), models the relationship between two entities through
bilinear interactions of $\mathbf{h}$, $\mathbf{r}$, and $\mathbf{t}$. The
translational family excels in inferring various inference patterns (e.g.,
symmetry, inversion, and composition), while the multiplicative family
facilitates expressive relational mapping (e.g., one-to-many and many-to-many)
between head and tail entities. Notably, certain approaches (Trouillon et al.
2016; Zhang et al. 2019) amalgamate the strengths of both families. A prime
example, ComplEx (Trouillon et al. 2016), embeds entities and relations in a
complex space and employs the Hermitian product, possessing geometric rotation
significance, to model diverse patterns. QuatE (Zhang et al. 2019) and BiQUE
(Guo and Kok 2021) further extends ComplEx to a 4-dimensional hypercomplex
space with quaternion representations. This extension empowers rich and
expressive mapping between head and tail entities, facilitated by relational
rotation endowed with additional degrees of freedom. Other methods (Chami et
al. 2020; Balazevic, Allen, and Hospedales 2019; Nayyeri et al. 2023) embed
entities in hyperbolic space, which excels in modeling hierarchical
structures. Our hypercomplex embedding method unifies these algebraic and
geometric embeddings in a single framework.

##  Appendix B Theoretical Justifications

We extend the vanilla logical patterns in KGs to include nested facts. This
can be expressed in a non-first-order-logic-like form
$\psi\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\phi$.

  * •

Relational symmetry (R-symmetry): an atomic relation $r$ is symmetric w.r.t a
nested relation $\widehat{r}$ if $\forall x,y\in\mathcal{E},\langle
x,r,y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\leftrightarrow}}\langle
y,r,x\rangle$.

  * •

Relational inverse (R-inverse): two atomic relations $r_{1}$ and $r_{2}$ are
inverse w.r.t a nested relation $\widehat{r}$ if $\forall x,y\in\mathcal{E},$
$(\langle
x,r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\leftrightarrow}}\langle
y,r_{2},x\rangle)$.

  * •

Relational implication (R-implication): an atomic relation $r_{1}$ implies a
atomic relation $r_{2}$ w.r.t a nested relation $\widehat{r}$ if $\forall
x,y\in\mathcal{E},(\langle
x,r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x,r_{2},y\rangle)$.

  * •

Relational inverse implication (R-Inv-implication): an atomic relation $r_{1}$
inversely implies an atomic relation $r_{2}$ w.r.t a nested relation
$\widehat{r}$ if $\forall x,y\in\mathcal{E},(\langle
x,r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
y,r_{2},x\rangle)$.

  * •

Entity implication (E-implication): an entity $x_{1}$ (resp. $y_{1}$) implies
entity $x_{2}$ (resp. $y_{2}$) w.r.t an atomic relation $r$ and a nested
relation $\widehat{r}$ if $\forall y\in\mathcal{E},(\langle
x_{1},r,y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x_{2},r,y\rangle)$ (resp. $\forall x\in\mathcal{E},(\langle
x,r,y_{1}\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x,r,y_{2}\rangle)$ ).

  * •

Entity relational implication (E-R-implication): an entity $x_{1}$ and
relation $r_{1}$ (resp. $y_{1}$ and relation $r_{1}$) implies entity $x_{2}$
and relation $r_{2}$ (resp. $y_{2}$ and relation $r_{2}$) w.r.t a nested
relation $\widehat{r}$ if $\forall y\in\mathcal{E},(\langle
x_{1},r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x_{2},r_{2},y\rangle)$ (resp. $\forall x\in\mathcal{E},(\langle
x,r_{1},y_{1}\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
x,r_{2},y_{2}\rangle)$).

  * •

Entity relational inverse implication (E-R-Inv-implication): an entity $x_{1}$
and relation $r_{1}$ (resp. $y_{1}$ and relation $r_{1}$) inversely implies
entity $x_{2}$ and relation $r_{2}$ (resp. $y_{2}$ and relation $r_{2}$) w.r.t
a nested relation $\widehat{r}$ if $\forall y\in\mathcal{E},(\langle
x_{1},r_{1},y\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
y,r_{2},x_{2}\rangle)$ (resp. $\forall x\in\mathcal{E},(\langle
x,r_{1},y_{1}\rangle\stackrel{{\scriptstyle\widehat{r}}}{{\rightarrow}}\langle
y_{2},r_{2},x\rangle)$).

  * •

Dual Entity implication (Dual E-implication): an entity pair ($x_{1}$,
$x_{2}$) implies another entity pair ($y_{1}$, $y_{2}$) iff both ($x_{1}$,
$y_{1}$) and ($x_{2}$, $y_{2}$) satisfy E-implication.

######  Proposition 3.

NestE can infer R-symmetry, R-inverse, R-implication, R-Inv-implication,
E-implication, E-R-implication, E-R-Inv-implication, and Dual E-implication.

To infer different logical patterns via different free variables, we can set
some elements of the relation matrix to be zero-valued or one-valued complex
numbers. We prove proposition 3 by showing the following special solutions for
each case of pattern.

######  Proposition 4.

Relational symmetry can be inferred by setting

$\mathbf{R}=\left[{\begin{array}[]{ccc}\mathbf{0}&\mathbf{0}&\mathbf{1}\\\
\mathbf{0}&\mathbf{R}_{22}&\mathbf{0}\\\ \mathbf{1}&\mathbf{0}&\mathbf{0}\\\
\end{array}}\right]$ (13)

where $\mathbf{r}\otimes\mathbf{R}_{22}=\mathbf{r}$ implying
$\mathbf{R}_{22}=\mathbf{1}$.

######  Proposition 5.

Relational inversion can be inferred by setting

$\mathbf{R}=\left[{\begin{array}[]{ccc}\mathbf{0}&\mathbf{0}&\mathbf{1}\\\
\mathbf{0}&\mathbf{R}_{22}&\mathbf{0}\\\ \mathbf{1}&\mathbf{0}&\mathbf{0}\\\
\end{array}}\right]$ (14)

where $\mathbf{r}_{1}\otimes\mathbf{R}_{22}=\mathbf{r}_{2}$ and
$\mathbf{r}_{2}\otimes\mathbf{R}_{22}=\mathbf{r}_{1}$ implying
$\mathbf{R}_{22}=\mp 1$.

######  Proposition 6.

Relational implication can be inferred by setting

$\mathbf{R}=\left[{\begin{array}[]{ccc}\mathbf{1}&\mathbf{0}&\mathbf{0}\\\
\mathbf{0}&\mathbf{R}_{22}&\mathbf{0}\\\ \mathbf{0}&\mathbf{0}&\mathbf{1}\\\
\end{array}}\right]$ (15)

where $\mathbf{r}_{1}\otimes\mathbf{R}_{22}=\mathbf{r}_{2}$.

######  Proposition 7.

Relational inverse implication can be inferred by setting

$\mathbf{R}=\left[{\begin{array}[]{ccc}\mathbf{0}&\mathbf{0}&\mathbf{1}\\\
\mathbf{0}&\mathbf{R}_{22}&\mathbf{0}\\\ \mathbf{1}&\mathbf{0}&\mathbf{0}\\\
\end{array}}\right]$ (16)

where $\mathbf{r}_{1}\otimes\mathbf{R}_{22}=\mathbf{r}_{2}$.

######  Proposition 8.

Entity implication can be inferred by setting

$\mathbf{R}=\left[{\begin{array}[]{ccc}\mathbf{R}_{11}&\mathbf{R}_{12}&\mathbf{0}\\\
\mathbf{R}_{21}&\mathbf{R}_{21}&\mathbf{0}\\\
\mathbf{0}&\mathbf{0}&\mathbf{1}\\\ \end{array}}\right]$ (17)

where

$\displaystyle\mathbf{x}_{1}\otimes\mathbf{R}_{11}+\mathbf{r}\otimes\mathbf{R}_{21}=\mathbf{x}_{2}$
(18)
$\displaystyle\mathbf{x}_{1}\otimes\mathbf{R}_{12}+\mathbf{r}\otimes\mathbf{R}_{22}=\mathbf{r}$
(19)

or

$\mathbf{R}=\left[{\begin{array}[]{ccc}\mathbf{1}&\mathbf{0}&\mathbf{0}\\\
\mathbf{0}&\mathbf{R}_{22}&\mathbf{R}_{23}\\\
\mathbf{0}&\mathbf{R}_{32}&\mathbf{R}_{33}\\\ \end{array}}\right]$ (20)

where

$\displaystyle\mathbf{r}\otimes\mathbf{R}_{32}+\mathbf{y}_{1}\otimes\mathbf{R}_{33}=\mathbf{y}_{2}$
(21)
$\displaystyle\mathbf{r}\otimes\mathbf{R}_{22}+\mathbf{y}_{1}\otimes\mathbf{R}_{32}=\mathbf{r}$
(22)

######  Proposition 9.

Entity inverse implication can be inferred by setting

$\mathbf{R}=\left[{\begin{array}[]{ccc}\mathbf{0}&\mathbf{0}&\mathbf{R}_{13}\\\
\mathbf{0}&\mathbf{1}&\mathbf{0}\\\ \mathbf{R}_{31}&\mathbf{0}&\mathbf{1}\\\
\end{array}}\right]$ (23)

where $\mathbf{x}_{1}\otimes\mathbf{R}_{31}=\mathbf{x}_{2}$. Or,

$\mathbf{R}=\left[{\begin{array}[]{ccc}\mathbf{0}&\mathbf{0}&\mathbf{R}_{13}\\\
\mathbf{0}&\mathbf{1}&\mathbf{0}\\\ \mathbf{R}_{31}&\mathbf{0}&\mathbf{0}\\\
\end{array}}\right]$ (24)

where $\mathbf{y}_{1}\otimes\mathbf{R}_{33}=\mathbf{y}_{2}$.

##  Appendix C Experimental Details

### Dataset Details

The datasets FBH, FBHE, and DBHE are constructed as detailed in (Chung and
Whang 2023). Both FBH and FBHE are derived from FB15K237, sourced from
Freebase (Bollacker et al. 2008), while DBHE is based on DB15K from DBpedia
(Auer et al. 2007). FBH contains only nested facts that can be inferred from
triple facts, including relations such as _prerequisite_for_ and
_implies_position_. In contrast, FBHE and DBHE incorporate additional
externally-sourced information gathered from Wikipedia articles, giving rise
to relations like _next_almaMater_ and _transfers_to_. For instance, the
(vice)presidents of the United States and politicians’ alma mater information
were extracted by crawling relevant Wikipedia articles. The datasets were
meticulously curated, involving six weeks of manual effort to define and
incorporate the nested facts (Chung and Whang 2023). The detailed data
augmentation process can be found in Section 4 of the same paper.

Table 10 provides an overview of all nested relations and their corresponding
examples that constitute FBH, FBHE, and DBHE. Specifically, FBHE contains ten
higher-level relations, while FBH contains the initial six higher-level
relations. Certain relations within FBHE, namely WorksFor, SucceededBy,
TransfersTo, and HigherThan, as well as NextAlmaMater and TransfersTo within
DBHE, necessitated externally-sourced information. In the case of WorksFor,
for instance, the authors crawled Wikipedia articles to obtain insights into
the affiliations of individuals. Similarly, for HigherThan in FBHE, recent
rankings from sources such as Fortune 1000 and Times University Ranking were
leveraged to construct the respective triples. Further details regarding the
types of nested triples utilized in the creation of FBH, FBHE, and DBHE can be
found in Table 11 and Table 12. Additionally, these tables provide statistics
pertaining to the logical patterns discussed in the paper.

Hyperparameter | value
---|---
learning rate $\alpha$ | 0.1
regularization rate $\beta$ | 0.1
$\lambda_{1}$ | 0.5
$\lambda_{2}$ | 0.2
dimensionality $d$ | 200
Table 8: The hyperparameter settings used in all experiments across all
datasets.

### Implementation Details

All experiments were conducted on machines equipped with 4 Nvidia A100 GPUs,
each with 40GB of VRAM. Our framework was implemented based on
OpenKE666https://github.com/thunlp/OpenKE and the provided
code777https://github.com/bdi-lab/BiVE/. The training was performed using
triple prediction, and evaluations were carried out for the other two tasks.
The embedding dimensionality was set to $d=200$.

Hyperparameters utilized in this work were reused from (Chung and Whang 2023),
and no additional hyperparameter tuning was performed. Specifically, all
hyperparameters are listed in Table 8. The validation process occurred every
50 epochs, up to a maximum of 500 epochs. The best epoch for evaluation was
selected based on the validation results.

### Supplemental Experiments

#### Comparison of different hypercomplex spaces for BiVEs

We also implemented variants of BiVE in different hypercomplex spaces and
present the results in Table 9. It was observed that the hyperbolic and split
quaternionic versions of BiVE, denoted as BiVE-H and BiVE-S respectively, did
not demonstrate significant improvements over the performance of BiVE-Q, even
when leveraging their respective translation-enhanced versions. This
phenomenon could potentially be attributed to the concatenation operation
employed by BiVE and the use of a multilayer perceptron (MLP) for
approximation. These factors might disrupt the geometric properties intrinsic
to hypercomplex numbers, thus diminishing the potential advantages these
geometries offer.

Table 9: The performance comparison of BiVEs implemented via different hypercomplex spaces on triple prediction.  | FBH | FBHE | DBHE
---|---|---|---
| MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$) | MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$) | MR ($\downarrow$) | MRR ($\uparrow$) | Hit@10 ($\uparrow$)
BiVE-Q | 6.20 | 0.855 | 0.941 | 8.35 | 0.711 | 0.866 | 3.63 | 0.687 | 0.958
BiVE-H (Ours) | 6.70 | 0.838 | 0.934 | 8.16 | 0.687 | 0.850 | 4.27 | 0.648 | 0.935
BiVE-S (Ours) | 6.13 | 0.824 | 0.938 | 8.28 | 0.677 | 0.850 | 4.85 | 0.645 | 0.926
BiVE-B | 8.63 | 0.833 | 0.924 | 9.53 | 0.705 | 0.860 | 4.66 | 0.718 | 0.945
BiVE-HB (Ours) | 7.65 | 0.803 | 0.919 | 10.54 | 0.675 | 0.834 | 6.40 | 0.703 | 0.922
BiVE-SB (Ours) | 9.77 | 0.799 | 0.914 | 9.83 | 0.678 | 0.853 | 7.58 | 0.712 | 0.914
| $\widehat{r}$ | $\langle T_{i},\widehat{r},T_{j}\rangle$ | Description
---|---|---|---
FBHE | PrerequisiteFor |  $T_{i}$: (BAFTA_Award, Nominates, The_King’s_Speech) | For The King’s Speech to win BAFTA Award, BAFTA Award should nominate The
$T_{j}$: (The_King’s_Speech, Wins, BAFTA_Award) | King’s Speech.
2-4 | EquivalentTo |  $T_{i}$: (Hillary_Clinton, IsMarriedTo, Bill_Clinton) | The two triples indicate the same information.
|  $T_{j}$: (Bill_Clinton, IsMarriedTo, Hillary_Clinton)
2-4 | ImpliesLocation |  $T_{i}$: (Sweden, CapitalIsLocatedIn, Stockholm) | ‘The capital of Sweden is Stockholm’ implies ‘Sweden contains Stockholm’.
|  $T_{j}$: (Sweden, Contains, Stockholm)
2-4 | ImpliesProfession |  $T_{i}$: (Liam_Neeson, ActsIn, Love_Actually) | ‘Liam Neeson acts in Love Actually’ implies ‘Liam Neeson is an actor’.
|  $T_{j}$: (Liam_Neeson, IsA, Actor)
2-4 | ImpliesSports |  $T_{i}$: (Boston_Red_Socks, HasPosition, Infield) | ‘Boston Red Socks has an infield position’ implies ‘Boston Red Socks plays baseball’.
|  $T_{j}$: (Boston_Red_Socks, Plays, Baseball)
2-4 | NextEventPlace |  $T_{i}$: (1932_Summer_Olympics, IsHeldIn, Los_Angeles) | Summer Olympics in 1932 and 1936 were held in Los Angeles and Berlin, respectively.
|  $T_{j}$: (1936_Summer_Olympics, IsHeldIn, Berlin) | 1936 Summer Olympics is the next event of 1932 Summer Olympics.
2-4 | WorksFor |  $T_{i}$: (Joe_Biden, HoldsPosition, Vice_President) | Joe Biden was a vice president when Barack Obama was a president of the United States.
|  $T_{j}$: (Barack_Obama, HoldsPosition, President)
2-4 | SucceededBy |  $T_{i}$: (George_W._Bush, HoldsPosition, President) | President Barack Obama succeeded President George W. Bush.
|  $T_{j}$: (Barack_Obama, HoldsPosition, President)
2-4 | TransfersTo |  $T_{i}$: (David_Beckham, PlaysFor, Real_Madrid_CF) | David Beckham transferred from Real Madrid CF to LA Galaxy.
|  $T_{j}$: (David_Beckham, PlaysFor, LA_Galaxy)
2-4 | HigherThan |  $T_{i}$: (Walmart, IsRankedIn, Fortune_500) | Walmart is ranked higher than Bank of America in Fortune 500.
|  $T_{j}$: (Bank_of_America, IsRankedIn, Fortune_500)
1-4 DBHE | EquivalentTo |  $T_{i}$: (David_Beckham, IsMarriedTo, Victoria_Beckham) | The two triples indicate the same information.
$T_{j}$: (Victoria_Beckham, IsMarriedTo, David_Beckham)
2-4 | ImpliesLanguage |  $T_{i}$: (Italy, HasOfficialLanguage, Italian_Language) | ‘The official language of Italy is the Italian language’ implies ‘The Italian language is
|  $T_{j}$: (Italy, UsesLanguage, Italian_Language) | used in Italy’.
2-4 | ImpliesProfession |  $T_{i}$: (Psycho, IsDirectedBy, Alfred_Hitchcock) | ‘Psycho is directed by Alfred Hitchcock’ implies ‘Alfred Hitchcock is a film producer’.
|  $T_{j}$: (Alfred_Hitchcock, IsA, Film_Producer)
2-4 | ImpliesLocation |  $T_{i}$: (Mariah_Carey, LivesIn, New_York_City) | ‘Mariah Carey lives in New York City’ implies ‘Mariah Carey lives in New York’
|  $T_{j}$: (Mariah_Carey, LivesIn, New_York)
2-4 | ImpliesTimeZone |  $T_{i}$: (Czech_Republic, TimeZone, Central_European) | ‘Czech Republic is included in Central European Time Zone’ implies ‘Prague is included
|  $T_{j}$: (Prague, TimeZone, Central_European) | in Central European Time Zone’.
2-4 | ImpliesGenre |  $T_{i}$: (Pharrell_Williams, Genre, Progressive_Rock) | ‘Pharrell Williams is a progressive rock musician’ implies ‘Pharrell Williams is a rock
|  $T_{j}$: (Pharrell_Williams, Genre, Rock_Music) | musician’.
2-4 | NextAlmaMater |  $T_{i}$: (Gerald_Ford, StudiesIn, University_of_Michigan) | Gerald Ford studied in University of Michigan. Then, he studied in Yale University.
|  $T_{j}$: (Gerald_Ford, StudiesIn, Yale_University)
2-4 | TransfersTo |  $T_{i}$: (Ronaldo, PlaysFor, FC_Barcelona) | Ronaldo transferred from FC Barcelona to Inter Millan.
|  $T_{j}$: (Ronaldo, PlaysFor, Inter_Millan)
Table 10: The nested factual relations and the corresponding examples of the nested triples used to create FBHE, and DBHE. Table taken from (Chung and Whang 2023). |  | Example | Frequency
---|---|---|---
$\langle T_{i}$, PrerequisiteFor, $T_{j}\rangle$ |  $T_{i}$: (Person A, DatesWith, Person B) | (Bruce_Willis, DatesWith, Demi_Moore) | 222
$T_{j}$: (Person A, BreaksUpWith, Person B) | (Bruce_Willis, BreaksUpWith, Demi_Moore)
2-4 |  $T_{i}$: (Award A, Nominates, Work B) | (BAFTA_Award, Nominates, The_King’s_Speech) | 2,335
|  $T_{j}$: (Work B, Wins, Award A) | (The_King’s_Speech, Wins, BAFTA_Award)
2-4 |  $T_{i}$: (Person A, HasNationality, Country B) | (Neymar, HasNationality, Brazil) | 109
|  $T_{j}$: (Person A, PlaysFor, National Team of B) | (Neymar, PlaysFor, Brazil_National_Football_Team)
1-4 $\langle T_{i}$, EquivalentTo, $T_{j}\rangle$ |  $T_{i}$: (Person A, IsASiblingTo, Person B) | (Serena_Williams, IsASiblingTo, Venus_Williams) | 120
$T_{j}$: (Person B, IsASiblingTo, Person A) | (Venus_Williams, IsASiblingTo, Serena_Williams)
2-4 |  $T_{i}$: (Person A, IsMarriedTo, Person B) | (Hillary_Clinton, IsMarriedTo, Bill_Clinton) | 352
|  $T_{j}$: (Person B, IsMarriedTo, Person A) | (Bill_Clinton, IsMarriedTo, Hillary_Clinton)
2-4 |  $T_{i}$: (Person A, HasAFriendshipWith, Person B) | (Bob_Dylan, HasAFriendshipWith, The_Beatles) | 1,832
|  $T_{j}$: (Person B, HasAFriendshipWith, Person A) | (The_Beatles, HasAFriendshipWith, Bob_Dylan)
2-4 |  $T_{i}$: (Person A, IsAPeerOf, Person B) | (Jimi_Hendrix, IsAPeerOf, Eric_Clapton) | 132
|  $T_{j}$: (Person B, IsAPeerOf, Person A) | (Eric_Clapton, IsAPeerOf, Jimi_Hendrix)
1-4 $\langle T_{i}$, ImpliesLocation, $T_{j}\rangle$ |  $T_{i}$: (Location A, Contains, Location B) | (England, Contains, Warwickshire) | 2,415
$T_{j}$: (Location containing A, Contains, Location in B) | (United_Kingdom, Contains, Birmingham)
2-4 |  $T_{i}$: (Organization A, Headquarter, Location B) | (Kyoto_University, Headquarter, Kyoto) | 820
|  $T_{j}$: (Location B, Contains, Organization A) | (Kyoto, Contains, Kyoto_University)
2-4 |  $T_{i}$: (Country A, CapitalIsLocatedIn, City B) | (Sweden, CapitalIsLocatedIn, Stockholm) | 83
|  $T_{j}$: (Country A, Contains, City B) | (Sweden, Contains, Stockholm)
1-4 $\langle T_{i}$, ImpliesProfession, $T_{j}\rangle$ |  $T_{i}$: (Person A, IsA, Specialized Profession of B) | (Mariah_Carey, IsA, Singer-songwriter) | 2,364
$T_{j}$: (Person A, IsA, Profession B) | (Mariah_Carey, IsA, Musician)
2-4 |  $T_{i}$: (Rock&Roll Hall of Fame, Inducts, Person A) | (Rock&Roll_Hall_of_Fame, Inducts, Bob_Dylan) | 66
|  $T_{j}$: (Person A, IsA, Musician/Artist) | (Bob_Dylan, IsA, Musician)
2-4 |  $T_{i}$: (Film A, IsWrittenBy, Person B) | (127_Hours, IsWrittenBy, Danny_Boyle) | 893
|  $T_{j}$: (Person B, IsA, Writer/Film Producer) | (Danny_Boyle, IsA, Film_producer)
2-4 |  $T_{i}$: (Person A, ActsIn, Film B) | (Liam_Neeson, ActsIn, Love_Actually) | 10,864
|  $T_{j}$: (Person A, IsA, Actor) | (Liam_Neeson, IsA, Actor)
2-4 |  $T_{i}$: (Person A, HoldsPosition, Government Position B) | (Barack_Obama, HoldsPosition, President) | 120
|  $T_{j}$: (Person A, IsA, Politician) | (Barack_Obama, IsA, Politician)
1-4 $\langle T_{i}$, ImpliesSports, $T_{j}\rangle$ |  $T_{i}$: (Team A, HasPosition, Position of B) | (Boston_Red_Socks, HasPosition, Infield) | 2,936
$T_{j}$: (Team A, Plays, Sports B) | (Boston_Red_Socks, Plays, Baseball)
2-4 |  $T_{i}$: (League of A, Includes, Team B) | (National_League, Includes, New_York_Mets) | 824
|  $T_{j}$: (Team B, Plays, Sports A) | (New_York_Mets, Plays, Baseball)
2-4 |  $T_{i}$: (Team A, ParticipatesIn, Draft of B) | (Atlanta_Braves, ParticipatesIn, MLB_Draft) | 528
|  $T_{j}$: (Team A, Plays, Sports B) | (Atlanta_Braves, Plays, Baseball)
1-4 $\langle T_{i}$, NextEventPlace, $T_{j}\rangle$ |  $T_{i}$: (Event A, IsHeldIn, Location A) | (1932_Summer_Olympics, IsHeldIn, Los_Angeles) | 47
$T_{j}$: (Next Event of A, IsHeldIn, Location B) | (1936_Summer_Olympics, IsHeldIn, Berlin)
1-4 $\langle T_{i}$, WorksFor, $T_{j}\rangle$ |  $T_{i}$: (Person A, HoldsPosition, Vice President) | (Joe_Biden, HoldsPosition, Vice_President) | 13
$T_{j}$: (Person B, HoldsPosition, President) | (Barack_Obama, HoldsPosition, President)
1-4 $\langle T_{i}$, SucceededBy, $T_{j}\rangle$ |  $T_{i}$: (Person A, HoldsPosition, President/Vice President) | (George_W._Bush, HoldsPosition, President) | 30
$T_{j}$: (Person B, HoldsPosition, President/Vice President) | (Barack_Obama, HoldsPosition, President)
1-4 $\langle T_{i}$, TransfersTo, $T_{j}\rangle$ |  $T_{i}$: (Person A, PlaysFor, Team B) | (David_Beckham, PlaysFor, Real_Madrid_CF) | 377
$T_{j}$: (Person A, PlaysFor, Team C) | (David_Beckham, PlaysFor, LA_Galaxy)
1-4 $\langle T_{i}$, HigherThan, $T_{j}\rangle$ |  $T_{i}$: (Item A, IsRankedIn, Ranking List C) | (Walmart, IsRankedIn, Fortune_500) | 7,459
$T_{j}$: (Item B, IsRankedIn, Ranking List C) | (Bank_of_America, IsRankedIn, Fortune_500)
Table 11: All types of nested triples to create FBH and FBHE. Table taken from (Chung and Whang 2023). |  | Example | Frequency
---|---|---|---
$\langle T_{i}$, EquivalentTo, $T_{j}\rangle$ |  $T_{i}$: (Person A, IsMarriedTo, Person B) | (Hillary_Clinton, IsMarriedTo, Bill_Clinton) | 314
$T_{j}$: (Person B, IsMarriedTo, Person A) | (Bill_Clinton, IsMarriedTo, Hillary_Clinton)
2-4 |  $T_{i}$: (Location A, UsesLanguage, Language B) | (Brazil, UsesLanguage, Portuguese_Language) | 120
|  $T_{j}$: (Language B, IsSpokenIn, Location A) | (Portuguese_Language, IsSpokenIn, Brazil)
2-4 |  $T_{i}$: (Person A, Influences, Person B) | (Baruch_Spinoza, Influences, Immanuel_Kant) | 394
|  $T_{j}$: (Person B, IsInfluencedBy, Person A) | (Immanuel_Kant, IsInfluencedBy, Baruch_Spinoza)
1-4 $\langle T_{i}$, ImpliesLanguage, $T_{j}\rangle$ |  $T_{i}$: (Location A, HasOfficialLanguage, Language B) | (Italy, HasOfficialLanguage, Italian_Language) | 196
$T_{j}$: (Location A, UsesLanguage, Language B) | (Italy, UsesLanguage, Italian_Language)
2-4 |  $T_{i}$: (Location A, UsesLanguage, Language B) | (United_States, UsesLanguage, English_Language) | 75
|  $T_{j}$: (Location in A, UsesLanguage, Language B) | (California, UsesLanguage, English_Language)
1-4 $\langle T_{i}$, ImpliesProfession, $T_{j}\rangle$ |  $T_{i}$: (Work A, MusicComposedBy, Person B) | (Forrest_Gump, MusicComposedBy, Alan_Silvestri) | 553
$T_{j}$: (Person B, IsA, Musician/Composer) | (Alan_Silvestri, IsA, Composer)
2-4 |  $T_{i}$: (Work A, Starring, Person B) | (Love_Actually, Starring, Liam_Neeson) | 737
|  $T_{j}$: (Person B, IsA, Actor) | (Liam_Neeson, IsA, Actor)
2-4 |  $T_{i}$: (Work A, CinematographyBy, Person B) | (Jurassic_Park, CinematographyBy, Dean_Cundey) | 299
|  $T_{j}$: (Person B, IsA, Cinematographer) | (Dean_Cundey, IsA, Cinematographer)
2-4 |  $T_{i}$: (Work A, IsDirectedBy, Person B) | (Psycho, IsDirectedBy, Alfred_Hitchcock) | 295
|  $T_{j}$: (Person B, IsA, Film_Director/Television_Director) | (Alfred_Hitchcock, IsA, Film_Director)
2-4 |  $T_{i}$: (Work A, IsProducedBy, Person B) | (King_Kong, IsProducedBy, Merian_C._Cooper) | 354
|  $T_{j}$: (Person B, IsA, Film_Producer/Television_Producer) | (Merian_C._Cooper, IsA, Film_Producer)
2-4 |  $T_{i}$: (Person A, AssociatesWithRecordLabel, Record B) | (Bo_Diddley, AssociatesWithRecordLabel, Atlantic_Records) | 155
|  $T_{j}$: (Person A, IsA, Record_Producer) | (Bo_Diddley, IsA, Record_Producer)
1-4 $\langle T_{i}$, ImpliesLocation, $T_{j}\rangle$ |  $T_{i}$: (Location A, IsPartOf, Location B) | (Ann_Arbor, IsPartOf, Washtenaw_County_Michigan) | 1,174
$T_{j}$: (Location A, IsPartOf, Location Containing B) | (Ann_Arbor, IsPartOf, Michigan)
2-4 |  $T_{i}$: (Organization A, IsLocatedIn, Location B) | (Adobe_Systems, IsLocatedIn, San_Jose_California) | 250
|  $T_{j}$: (Organization A, IsLocatedIn, Location Containing B) | (Adobe_Systems, IsLocatedIn, California)
2-4 |  $T_{i}$: (Person A, LivesIn, Location B) | (Mariah_Carey, LivesIn, New_York_City) | 213
|  $T_{j}$: (Person A, LivesIn, Location Containing B) | (Mariah_Carey, LivesIn, New_York)
1-4 $\langle T_{i}$, ImpliesTimeZone, $T_{j}\rangle$ |  $T_{i}$: (Location A, TimeZone, Time Zone B) | (Czech_Republic, TimeZone, Central_European_Time) | 409
$T_{j}$: (Location in A, TimeZone, Time Zone B) | (Prague, TimeZone, Central_European_Time)
1-4 $\langle T_{i}$, ImpliesGenre, $T_{j}\rangle$ |  $T_{i}$: (Musician A, Genre, Genre B) | (Pharrell_Williams, Genre, Progressive_Rock) | 767
$T_{j}$: (Musician A, Genre, Parent Genre of B) | (Pharrell_Williams, Genre, Rock_Music)
1-4 $\langle T_{i}$, NextAlmaMater, $T_{j}\rangle$ |  $T_{i}$: (Person A, StudiesIn, Institution B) | (Gerald_Ford, StudiesIn, University_of_Michigan) | 112
$T_{j}$: (Person A, StudiesIn, Institution C) | (Gerald_Ford, StudiesIn, Yale_University)
1-4 $\langle T_{i}$, TransfersTo, $T_{j}\rangle$ |  $T_{i}$: (Person A, PlaysFor, Team B) | (Ronaldo, PlaysFor, FC_Barcelona) | 300
$T_{j}$: (Person A, PlaysFor, Team C) | (Ronaldo, PlaysFor, Inter_Millan)
Table 12: All types of nested triples to create DBHE. Table taken from (Chung
and Whang 2023).