THE ${J}^{2+}$-INVARIANT FOR PAIRS OF GENERIC IMMERSIONS
Hanna Häußler, 09.04.2021

<EMAIL_ADDRESS>
ABSTRACT: This paper introduces the $J^{2+}$-invariant for pairs of generic
immersions. The new invariant behaves like Arnold’s $J^{+}$-invariant for
generic immersions as it is invariant under homotopy including inverse
tangencies and traversing triple points, but changes by traversing direct
tangencies. It has several other properties, for example the invariant for
pairs is completely independent of the $J^{+}$-invariants of the single
immersions, which we will prove in Theorem B.
KEYWORDS: Arnold’s $J^{+}$-invariant, Viro’s formula

###### Contents

  1. 1 ${J}^{+}$-invariant and Viro’s formula
  2. 2 The ${J}^{2+}$-invariant for pairs of generic immersions
  3. 3 Further properties and perspectives

Introduction

Arnold introduces the $J^{+}$-invariant for generic immersions. This paper
extends the observation to homotopies of pairs of generic immersions and
therefore studies Viro’s formula [3]:

$J^{+}(S)=1+n-\sum\omega_{C}(S)^{2}+\sum ind_{p}(S)^{2},$ (1)

here $n$ is the amount of double points of the generic immersion $S$ and
$\omega_{C}(S)$ is the winding number of the component $C$ of
$\mathbb{C}\setminus S$. The index $ind_{p}(S)$ for all double points $p$ is
defined as the arithmetic mean of the winding numbers of the four adjacent
components.
The paper gives an alternative proof for Viro’s formula and introduces the
$J^{2+}$-invariant for pairs of generic immersions $K$ by

$J^{2+}(K)=2+n-\sum\omega_{c}(K)^{2}+\sum ind_{p}(K)^{2}+u(K)^{2},$ (2)

here $n$, $\omega_{c}(K)$ and $ind_{p}(K)$ are defined analogously to Viro’s
formula for a pair of immersions $K$. The additional variable $u(K)$, called
the encircling index, describes the relation of the single immersions forming
$K$ towards each other and is necessary in order to satisfy the following
conditions.

Theorem A: The $J^{2+}$-invariant is constant under homotopy including
traversing inverse (self-)tangency and triple points and changes through
traversing direct (self-)tangency. The invariant increases (decreases) by the
value of $2$ through positively (negatively) traversing direct tangencies.

The Theorem is proven similarly to the alternative proof that will be given
for Viro’s formula. We then discuss the independency of the $J^{+}$\- and
$J^{2+}$-invariants and prove the following theorem.

Theorem B: For arbitrary even numbers $x$, $y$ and $z$ always exists a pair of
generic immersions $K$ out of immersions $S_{1}$ and $S_{2}$ such that
${J}^{+}(S_{1})=x$, ${J}^{+}(S_{2})=y$ and ${J}^{2+}(K)=z$.

The proof is given by an algorithm which constructs a suitable pair of
immersions satisfying the given conditions.

This paper has the following structure: In section [1] we recall the
definition of Arnold’s $J^{+}$-invariant and Viro’s formula. The alternative
proof of the formula is given in [1.2]. Section [2] introduces the
$J^{2+}$-invariant for pairs of immersions and Theorem A is proven. In section
[3] we look at further properties of the $J^{2+}$-invariant, prove Theorem B
and give different perspectives on how the new found invariant can be used in
further work.

##  1 ${J}^{+}$-invariant and Viro’s formula

The first chapter introduces Arnold’s $J^{+}$-invariant and Viro’s formula,
then chapter 1.2 provides an alternative proof of the formula, which is
essential for the generalization, construction and proof of the properties of
the $J^{2+}$-invariant in chapter 2.

1.1 The $J^{+}$-invariant

We consider immersions in the plane and define these as smooth functions
${S:S^{1}\rightarrow\mathbb{C}}$, whose derivative never vanishes. As
${S}^{1}$ and $\mathbb{R}/\mathbb{Z}$ are diffeomorphic, we can also consider
smooth functions $S:\mathbb{R}/\mathbb{Z}\rightarrow\mathbb{R}^{2}$, whose
derivative never vanishes. In the following we examine generic immersions with
at most double points. Immersions are either oriented clockwise (negatively)
or anti-clockwise (positively). We use $S$ as notation for the image of a
smooth function ${S:S^{1}\rightarrow\mathbb{C}}$.

Figure 1: Critical homotopy scenarios

###### Definition 1.1

Two immersions $S$ and $S^{\prime}$ are homotopic, if there exists a smooth
function ${g:{S}^{1}\times[0,1]\rightarrow\mathbb{C}}$, with $g(\cdot,0)=S$
and $g(\cdot,1)=S^{\prime}$ such that
$g(\cdot,t):{S}^{1}\rightarrow\mathbb{C}$ is an immersion for all $t\in[0,1]$.

During a homotopy of generic immersions three critical homotopy scenarios can
occur: double points can vanish, new ones can occur by passing direct or
inverse self-tangencies or the immersion can pass triple points. In "Plane
curves their invariants, perestroikas and classifications"[1] Arnold
introduces three invariants, which are constant for two of these three
scenarios, as visualized in figure 1. Positively passing direct or inverse
self-tangency means that new double points occur, whereas negatively passing
means that double points vanish. In figure 1 positively passing is visualized
left to right, negatively passing right to left. Before and after passing a
triple point the immersion has the same amount of double points.
In his paper Arnold introduces the $J^{+}$-invariant, which is constant under
inverse self-tangency and triple points and changes with direct self-tangency.

Figure 2: standard immersions $K_{i}$ for $i\in\mathbb{N}_{0}$

The ${J}^{+}$-invariant of an immersion $S$ has the following properties:

  1. 1.

${J}^{+}(S)\in 2\mathbb{Z}$ is unique and independent of orientation .

  2. 2.

${J}^{+}(S)$ is constant under homotopy including triple points and inverse
self-tangency.

  3. 3.

${J}^{+}(S)$ increases or decreases by the value of $2$ under traversing
direct self-tangency positively or negatively.

  4. 4.

For pre-defined standard immersions $K_{i}$ with $i\in\mathbb{N}_{0}$, as
visualized in figure 2, one can calculate

${J}^{+}(K_{i})=\begin{cases}2-2i&\text{f\"{u}r }i\neq 0\\\ 0&\text{f\"{u}r
}i=0.\end{cases}$

Due to the Theorem of Whitney-Graustein, discussed in "On regular closed
curves in the plane"[2], every arbitrary immersion $S$ is transformable in a
suitable standard immersion $K_{i}$ by homotopy including crossings of triple
points, inverse and direct self-tangencies. Hence the $J^{+}$-invariant of
every immersion can be calculated by finding such a homotopy.
As notation for the $J^{+}$-invariant of an immersion under homotopy, we write

$\triangle J^{+}(S)=J^{+}(S_{\text{new}})-J^{+}(S_{\text{old}}),$

whereas the immersions $S_{\mathrm{new}}$ and $S_{\mathrm{old}}$ will be
denoted as $S$.
A proof of the propositions mentioned above is given in [1]. The independence
of orientation will be discussed in chapter 1.2.

1.2 Viro’s formula

In the first chapter the $J^{+}$-invariant is defined using standard
immersions. In "$J^{+}$-like invariants of periodic orbits of the second kind
in the restricted three-body problem"[3] Viro’s formula is introduced as a new
method to calculate the $J^{+}$-invariant of an arbitrary immersion. With that
it is not necessary anymore to construct a homotopy to a standard immersion.
Instead, we can directly calculate the $J^{+}$-invariant by determining
topological properties.

###### Definition 1.2

The winding number $\omega_{C}(S)\in\mathbb{Z}$ characterizes how often every
point in a connected component $C$ in $\mathbb{C}\setminus S$ is circled by
running through the immersion.
The index $ind_{p}(S)\in\mathbb{Z}$ of a double point $p$ of an immersion $S$
is the arithmetic mean of the winding numbers of the four adjacent components
at $p$.

###### Theorem 1.3

(Viro’s formula) The ${J}^{+}$-invariant of an immersion $S$ with $n$ double
points can be calculated using the following formula:

${J}^{+}(S)=1+n-\sum\omega_{C}(S)^{2}+\sum ind_{p}(S)^{2}$

The first sum runs over all components $C$ in $\mathbb{C}\setminus S$, the
second sum runs over all double points $p$ of $S$.

Figure 3: How to determine the winding number of a component [6]

The two following examples illustrate the defined variables and the
calculation of the $J^{+}$-invariant using Viro’s formula. For the
determination of the winding numbers we use the rule, that by traversing a
line into a new component the winding number increases (decreases) by one if
the line is oriented from left to right (right to left). Figure 3 visualizes
that rule.

Figure 4: Homotopic immersions $S$ and $K_{2}$

Example 1 (Figure 4): An immersion $S$ intersects $\mathbb{C}$ in five
components, whose winding numbers $\omega_{C}(S)$ are registered in the
belonging component. For all $n=3$ double points $p$ we calculate
$ind_{p}(S)=0$. Hence by using Viro’s formula we get ${J}^{+}(S)=0$. This is
what we also get by constructing a homotopy transforming $S$ into $K_{2}$,
which includes traversing a triple point and a negative crossing of a direct
self-tangency. Therefore we obtain
${J}^{+}(S)=J^{+}(K_{2})-1\cdot(-2)=-2-(-2)=0$.

Figure 5: Homotopic immersions $S$ and $K_{1}$

Example 2 (Figure 5): The immersion $S$ divides $\mathbb{C}$ into six
components, whose $\omega_{C}(S)$ are registered in the belonging component.
The central component having $\omega_{C}(S)=0$ is once circled clockwise and
anti-clockwise. For all $n=4$ double points $p$ we get $ind_{p}(S)=1$. Using
Viro’s formula we calculate ${J}^{+}(S)=2$. A construced homotopy forming $S$
into a standard immersion includes a negative crossing of a direct self-
tangency, therefore we obtain ${J}^{+}(S)={J}^{+}(K_{1})-(-2)=0-(-2)=2$, which
agrees with the calculation by formula.

Figure 6: Winding numbers of components of $\mathbb{C}\setminus K_{i}$

Using Viro’s formula we prove the following proposition.

###### Proposition 1.4

The $J^{+}$-invariant is independent of orientation.

Proof: By changing the orientation of an immersion $S$ the amount of double
points does not change. The winding numbers of the components change their
sign, therefore also the sign of the indices of the double points change. As
for the calculation of the $J^{+}$-invariant all variables are summed squared,
and therefore the invariant needs to be independent of orientation.

$\square$

Elements of the following proof are used to show Theorem A.
$\underline{\textit{Proof of Viro's formula (Theorem 1.3): }}$ We start by
showing that Viro’s formula is correct for all standard immersions $K_{i}$.
Therefore we use positively oriented immersions $K_{i}$, as visualized in
Figure 6. This suffices as the $J^{+}$-invariant is independent of
orientation.
The immersion $K_{0}$ intersects $\mathbb{C}$ into three components with the
winding numbers $1$, $-1$ and $0$ and has one double point $p$ $(n=1)$ with
$ind_{p}(K_{0})=0$. Hence, we get ${J}^{+}(K_{0})=0$.
Next to that $\mathbb{C}\setminus K_{1}$ has two components with the winding
numbers $0$ and $1$ and $K_{1}$ has no double points $(n=0)$. Hence, we
calculate ${J}^{+}(K_{1})=0$.
All $K_{i}$ with $i>1$ have $n=i-1$ double points $p$ with index
$ind_{p}(S)=1$ and they intersect $\mathbb{C}$ into $i+1$ components. Hence,
we have
${{J}^{+}(K_{i})=1+(i-1)-(1^{2}+(i-1)\cdot 2^{2}+0^{2})+(i-1)\cdot
1^{2}=-2*(i-1)}$.
Therefore the formula calculates the $J^{+}$-invariants of the standard
immersions $K_{i}$ defined by Arnold correctly.
Now let $S$ be an arbitrary immersion, which is homotopic to a standard
immersion $K_{i}$. During a homotopy three different critical situations can
occur, which lead to a change of the topological variables that are used in
Viro’s formula: crossing direct and inverse self-tangency and traversing
triple points. If the immersion $S$ has just passed one of these three
situations during a homotopy, we can write

$\displaystyle\triangle{J}^{+}(S)=J^{+}(S_{\text{new}})-J^{+}(S_{\text{old}})$
$\displaystyle=\triangle\left(1+n-\sum\omega_{c}(S)^{2}+\sum
ind_{p}(S)^{2}\right)$ $\displaystyle=\triangle
n-\triangle\sum\omega_{c}(S)^{2}+\triangle\sum ind_{p}(S)^{2}.$ (3)

Figure 7: Components by traversing a self-tangency

Now we study $\triangle J^{+}(S)$ for all three critical situations. Figure 7
shows the local neighbourhood before and after traversing self-tangency. In
the figure the right and the left path can be connected in an arbitrary way.
By passing a self-tangency the immersion $S$ gets two new double points $A$
and $B$, so is $\triangle n=2$. Furthermore, two new components occur in
$\mathbb{C}\setminus S$. The component $C_{s}$ is the new arised component
with winding number $s$. Besides that the component with winding number $z$,
which is first in between the tangencies, is divided into two components. One
of them is called $C_{z}$ with winding number $z$. Figure 7 illustrates the
situation whereas $x$, $y$, $z$ and $s$ are the winding numbers of the
relevant components. Therefore we have
${\triangle\sum\omega_{C}(S)^{2}=\omega_{C_{s}}(S)^{2}+\omega_{C_{z}}(S)^{2}}$,
as these two new components occur and the winding numbers of the other
components do not change. Furthermore, ${\triangle\sum
ind_{p}(S)^{2}=ind_{A}(S)^{2}+ind_{B}(S)^{2}}$ with
$ind_{A}(S)=ind_{B}(S)=\frac{1}{4}(x+y+z+s)$, follows as $ind_{p}(S)$ is
constant for all other double points $p$ of $S$.
Using the rule for determining the winding numbers of the single components,
we see that for traversing direct self-tangency for an arbitrary
$z\in\mathbb{Z}$ we have $x=z+1$, $y=z-1$ and $s=z$. Therefore we conclude

$\displaystyle\triangle{J}^{+}(S)$
$\displaystyle=2-(z^{2}+s^{2})+2\left(\frac{1}{4}(x+y+z+s)\right)^{2}$
$\displaystyle=2-2(z)^{2}+2(z)^{2}=2$

For traversing an inverse self-tangency for an arbitrary $z\in\mathbb{Z}$ we
obtain $x=z+1=y$ and $s=z+2$. Therefore we get

$\displaystyle\triangle{J}^{+}(S)$
$\displaystyle=2-(z^{2}+s^{2})+2\left(\frac{1}{4}(x+y+z+s)\right)^{2}$
$\displaystyle=2-(z)^{2}-(z+2)^{2}+2(z+1)^{2}=0$

Figure 8: Winding numbers by traversing a triple point

By traversing a triple point the amount of double points $n$ as well as the
amount of components $C$ in $\mathbb{C}\setminus S$ do not change ($\triangle
n=0$). Only the winding number of the new emerging triangle $D_{\mathrm{new}}$
compared to the old triangle $D_{\mathrm{old}}$ changes. Therefore also the
indices of the double points $A$, $B$ and $C$, which are the corners of the
triangle, change. In Figure 8 two axes are fixed orthogonally, the third axis
moves through the triple point, the figure shows all four possibilities of
orientation of the axes. The winding numbers are recorded on the basis of the
winding number of the vanishing triangle $\omega_{D_{\mathrm{old}}}(S)=x$.
By traversing a triple point Viro’s formula changes in the following way

$\displaystyle\triangle{J}^{+}(S)$
$\displaystyle=-\triangle\sum\omega_{C}(S)^{2}+\triangle\sum ind_{p}(S)^{2}$
$\displaystyle=-\left(\omega_{D_{\mathrm{new}}}(S)^{2}-\omega_{D_{\mathrm{old}}}(S)^{2}\right)+\triangle
ind_{A}(S)^{2}+\triangle ind_{B}(S)^{2}+\triangle ind_{C}(S)^{2}.$
1\. Version: $\triangle{J}^{+}(S)$ | $=-((x+1)^{2}-x^{2})+(x^{2}+2(x+1)^{2}-2x^{2}-(x+1)^{2})$
---|---
| $=x^{2}-(x+1)^{2}-x^{2}+(x+1)^{2}=0$
2\. Version: $\triangle{J}^{+}(S)$ | $=-((x-3)^{2}-x^{2})+(3(x-2)^{2}-3(x-1)^{2})$
| $=-x^{2}+6x-9+x^{2}+(3x^{2}-12x+12-3x^{2}+6x-3)=0$
3\. Version: $\triangle{J}^{+}(S)$ | $=-((x+1)^{2}-x^{2})+(x^{2}+2(x+1)^{2}-2x^{2}-(x+1)^{2})$
| $=-(x+1)^{2}+x^{2}-x^{2}+(x+1)^{2}=0$
4\. Version: $\triangle{J}^{+}(S)$ | $=-((x+1)^{2}-x^{2})+(x^{2}+2(x+1)^{2}-2x^{2}-(x+1)^{2})$
| $=-(x+1)^{2}+x^{2}-x^{2}+(x+1)^{2}=0$

In all four versions of traversing a triple point, we get $\triangle
J^{+}(S)=0$.
Hence, it is shown, that the number calculated by Viro’s formula behaves like
the $J^{+}$-invariant is defined. It changes by the value of $2$ through
traversing direct self-tangency and is constant under traversing inverse self-
tangency and triple points. Therefore the theorem is proven.

$\square$

##  2 The ${J}^{2+}$-invariant for pairs of generic immersions

This chapter introduces the $J^{2+}$-invariant for pairs of generic immersions
$K$. The invariant is constant when traversing inverse (self-)tangency and
triple points, it changes through direct (self-)tangency. As we need to
consider the relation of the single immersions towards each other, the
encircling index is introduced which we need in order to calculate the
$J^{2+}$-invariant. The second part of the chapter discusses the proof of
Theorem A stating that the $J^{2+}$-invariant behaves as described above.

2.1 Introduction of the $J^{2+}$-invariant

Let $K$ be a pair of two generic immersions $S_{1}:S^{1}\rightarrow\mathbb{C}$
and $S_{2}:S^{1}\rightarrow\mathbb{C}$ defined as function

$K:S^{1}\sqcup S^{1}\rightarrow\mathbb{C}$

with at most double points. In the following $K$ is also used as notation for
the image of this function.
Such a pair of immersions $K$ has different topological properties. The amount
of double points of $K$ is called $n$. The double points can be double points
of a single immersion $S_{1}$ or $S_{2}$, or be formed by an intersection of
$S_{1}$ and $S_{2}$. The winding number $\omega_{C}(K)$ of the components $C$
of $\mathbb{C}\setminus K$ describes, how often every point in $C$ is circled
while going through $S_{1}$ and $S_{2}$. The index $ind_{p}(K)$ is defined for
every double point $p$ and is calculated as the arithmetic mean of winding
numbers of the four adjacent components at $p$.
The new found invariant can not easily be calculated as the sum of the
$J^{+}$-invariants of the single immersions, because the relation of the
immersions towards each other needs to be considered. Therefore we introduce a
new variable.

###### Definition 2.1

Let $K$ be a pair of generic immersions of $S_{1}$ and $S_{2}$, then
$u_{S_{i}}(K)\in\mathbb{Z}$ describes how often the immersion $S_{i}$ for
$i\in\\{1,2\\}$ is entirely encircled by going through $S_{j}$ for
$j\in\\{1,2\\}\setminus i$ without an intersection of the immersions.
Furthermore, the encircling index is defined as

$u(K)=\begin{cases}0&\textnormal{if }S_{1}\cap S_{2}\neq\emptyset\\\
\max\\{|u_{S_{i}}(K)|:i=1,2\\}&\textnormal{elsewise}\end{cases}$

We call a pair of immersions out of $S_{1}$ and $S_{2}$ disjoint, if the
immersions do not intersect each other and none encircles the other one
entirely.

Figure 9: Determination of $u(K)$ with $K$ as pair of two standard immersions
$K_{1}$

Figure 9 shows three examples in which two immersions $K_{1}$ are related
towards each other differently. If $K$ is a system of two disjoint immersions
or if $K$ is a system of two immersions intersecting each other, $u(K)=0$
follows immediately. In the last example one immersion is entirely encircled
by the other one without intersection, we get $u(K)=1$.

###### Definition 2.2

Two pairs of immersions $K$ and $K^{\prime}$ are homotopic, if there exists a
smooth function ${g:(S^{1}\sqcup S^{1})\times[0,1]\rightarrow\mathbb{C}}$ with
$g(\cdot,0)=K$ and $g(\cdot,1)=K^{\prime}$ such that for all $t\in[0,1]$
$g(\cdot,t):S^{1}\sqcup S^{1}\rightarrow\mathbb{C}$ is a pair of immersions.

Remark:  Every arbitrary pair of immersions $K$ out of $S_{1}$ and $S_{2}$ is
homotopic to a pair of immersions $K^{*}$ out of disjoint generic immersions
$S_{1}$ and $S_{2}$.

###### Theorem 2.3

(Theorem A) For a pair of immersions $K$, we define the $J^{2+}$-invariant as

${J}^{2+}(K)=2+n-\sum\omega_{c}(K)^{2}+\sum ind_{p}(K)^{2}+u(K)^{2}$ (4)

whereas the first sum runs over all components $C$ of $\mathbb{C}\setminus K$
and the second sum runs over all double points $p$ of $K$.
The $J^{2+}$-invariant is constant under homotopy including traversing inverse
(self-)tangency and triple points but changes through traversing direct
(self-)tangency.

The following two examples illustrate the $J^{2+}$-invariant.

Figure 10: Homotopy of a pair of immersions $K$ out of $K_{1}$ and $K_{2}$

In Example 3 (Figure 10) we can see two disjoint standard immersions $K_{1}$
and $K_{2}$. We calculate
${J}^{2+}(K)=2+1-(1^{2}+1^{2}+2^{2})+(1^{2})+0^{2}=3-6+1=-2$. By positively
traversing an inverse tangency one new component and two new double points
occur. We get
${J}^{2+}(K)=2+3-(1^{2}+2^{2}+1^{2}+2^{2})+(1^{2}+1^{2}+1^{2})+0^{2}=-2$. It
follows $J^{2+}(K)=J^{+}(K_{1})+J^{+}(K_{2})=0+(-2)=-2$. The value of the
formula is constant. In the last phase of the homotopy by traversing a direct
tangent negatively $K_{1}$ is encircled by $K_{2}$ entirely without
intersection. We calculate
${{J}^{2+}(K)=2+1-(2^{2}+2^{2}+1^{2})+(1^{2})+1^{2}=-4}$. By negatively
traversing the direct tangency the $J^{2+}$-invariant decreases by the value
of $2$.

Figure 11: Homotopy of a pair of immersions $K$ out of two standard immersions
$K_{0}$

Example 4 (Figure 11) shows two disjoint immersions $K_{0}$ having the same
orientation. We get
${J}^{2+}(K)=2+2-(1^{2}+1^{2}+(-1)^{2}+(-1)^{2})+(0^{2}+0^{2})+0^{2}=4-4=0$.
By positively traversing a direct tangency we obtain
${J}^{2+}(K)=2+4-(1^{2}+1^{2}+(-1)^{2}+(-1)^{2})+(4\cdot 0^{2})+0^{2}=2$. The
${J}^{2+}$-invariant increases by the value of $2$. By traversing a triple
point, the amount of double points and the amount of components do not change
in contrary to the winding number of the triangle. Therefore,we receive
${J}^{2+}(K)=2+4-(1^{2}+1^{2}+(-1)^{2}+(-2)^{2})+(0^{2}+3\cdot
1^{2})+0^{2}=6-7+3=2$. The $J^{2+}$-invariant remains constant. During the
last phase of the homotopy a direct tangency is traversed negatively such that
$S_{1}$ is entirely encircled by $S_{2}$. We conclude
${J}^{2+}(K)=2+2-(1^{2}+(-1)^{2}+(-2)^{2})+(1^{2}+0^{2})+1^{2}=4-6+1+1=0$. The
$J^{2+}$-invariant decreases by the value of $2$.

2.2 Proof of Theorem A

The proof of Theorem A is divided into four lemmas. At first we discuss that
the $J^{2+}$-invariant is unique and well defined. After that we treat the
special case of a pair of disjoint immersions and finally we examine critical
homotopy scenarios of a single immersion and between the two immersions
$S_{1}$ and $S_{2}$.

###### Lemma 2.4

The ${J}^{2+}$-invariant of a pair of immersions $K$ is well defined.

Proof: This is true as for an arbitrary pair of immersions all summands of the
formula are well defined as $K$ is generic and defined as smooth function with
non negative derivative. The amount of double points $n$, the winding numbers
of the components $\omega_{C}(K)$ of $\mathbb{C}\setminus K$, the indices
$ind_{p}(K)$ for all double points $p$ as well as $u(K)$ exists and is
uniquely determined for all pairs of immersions $K$.

$\square$

###### Lemma 2.5

Let $K$ be a pair of two disjoint generic immersions $S_{1}$ and $S_{2}$, then

${J}^{2+}(K)={J}^{+}(S_{1})+{J}^{+}(S_{2}).$ (5)

Proof: In this case the $J^{2+}$-invariant can be split into the Viro formulas
for $S_{1}$ and $S_{2}$. As the immersions do not intersect each other, we
know $u(K)=0$ and $n=n_{1}+n_{2}$, whereas $n_{i}$ with $i=1,2$ is the amount
of double points of the immersion $S_{i}$ and $n$ is the amount of double
points of the pair of immersions $K$. The sum $\sum ind_{p}(K)^{2}$ runs over
the double points of the single immersions and can therefore be split. This is
also possible for the sum $\sum\omega_{c}(K)^{2}$ running over the components,
as every component is ether encircled by $S_{1}$ or $S_{2}$ or none of them.

$\square$

###### Lemma 2.6

The ${J}^{2+}$-invariant of a pair of immersions of $S_{1}$ and $S_{2}$
changes by traversing direct self-tangency and is constant by traversing
inverse self-tangency or triple points of one single immersion $S_{i}$ for
$i=1,2$.

Proof: The Proof of Viro’s formula can be applied analogously. We know
$\triangle u(K)=0$, as a homotopy as described in the Lemma does not change
the relation of $S_{1}$ and $S_{2}$ towards each other. This means either the
two generic immersions do intersect each other before and after, one encircles
the other one entirely without intersection before and after, or they are
disjoint before and after the homotopy. Therefore the local critical
neighbourhood is either encircled entirely by the other immersion or it isn’t
encircled at all. Therefore we conclude $\triangle{J}^{2+}(K)=2$ for
positively traversing a direct self-tangency and $\triangle{J}^{2+}(K)=0$ for
traversing an inverse self-tangency or a triple point.

$\square$

###### Lemma 2.7

The ${J}^{2+}$-invariant changes by the value of $2$ by traversing direct
tangency of $S_{1}$ and $S_{2}$ and is constant by traversing inverse tangency
and triple points of $S_{1}$ and $S_{2}$.

Figure 12: Local neighbourhood during traversing a tangency of $S_{1}$ and
$S_{2}$

Proof: Let $K$ be a pair of immersions, which traverses a tangency during a
homotopy. Figure 12 shows the local neighbourhood by traversing a direct or
inverse tangency. The left branch is part of the immersion $S_{1}$, the right
one is part of the immersion $S_{2}$. The values $x$, $y$, $z$ and $s$ are the
winding numbers of the single components. We consider

$\triangle J^{2+}(K)=\triangle
n-\triangle\sum{\omega_{C}(K)}^{2}+\triangle\sum{ind_{p}(K)}^{2}+\triangle
u(K)^{2}.$

By positively traversing a tangency we get two new double points $p$ with
$ind_{p}(K)=\frac{1}{4}(x+y+z+s)$, hence we obtain $\triangle n=2$ and
$\triangle\sum ind_{p}(K)^{2}=2(\frac{1}{4}(x+y+z+s))^{2}$, as the indices of
all other double points of $K$ do not change.
Considering $\triangle\sum\omega_{C}(K)^{2}$ and $\triangle u(K)$ we need to
distinguish two cases.
1\. Either the encircling index is $u(K)=0$ and $S_{1}$ and $S_{2}$ intersect
each other, then we get two new components with winding numbers $z$ and $s$ by
traversing the tangency. Furthermore, we have
$\triangle\sum\omega_{C}(K)^{2}=z^{2}+s^{2}$ and $\triangle u(K)^{2}=0$.
2\. But if we have $u(K)\neq 0$ or $u(K)=0$ with the condition, that $S_{1}$
and $S_{2}$ do not intersect each other, there is only one new component $C$
with $\omega_{C}(K)=s$ and hence we have
$\triangle\sum\omega_{c}(K)^{2}=s^{2}$. Therefore we conclude $\triangle
u(K)^{2}=-z^{2}$, as for the starting pair of immersions we have $u(K)=z$ and
after positively traversing a tangency, we have $u(K)=0$.
For a fixed $z\in\mathbb{Z}$ we determine for the winding numbers $x=z+1$,
$y=z-1$ and $s=z$ for traversing a direct tangency, whereas for traversing an
inverse tangency we know that $x=z+1=y$ and $s=z+2$. Therefore we receive the
following values:
direct inverse tangency tangency $\triangle n$ $2$ $2$ $\triangle\sum
ind_{p}(K)^{2}$ $2z^{2}$ $2(z+1)^{2}$ $1.u(K)=0$ and intersection
$\triangle\sum\omega_{C}(K)^{2}$ $2z^{2}$ $z^{2}+(z+2)^{2}$ $\triangle
u(K)^{2}$ $0$ $0$ $2.u(K)\neq 0$ or $\triangle\sum\omega_{c}(K)^{2}$ $z^{2}$
$(z+2)^{2}$ $u(K)=0$ and no intersection $\triangle u(K)^{2}$ $-z^{2}$
$-z^{2}$

For positively traversing a direct tangency starting with a pair of immersions
with $u(K)=0$ with intersection, we calculate
$\triangle{J}^{2+}(K)=2-2z^{2}+2z^{2}-0=2$. Starting with a pair of immersions
with $u(K)\neq 0$ or $u(K)=0$ without intersection of $S_{1}$ and $S_{2}$ we
calculate $\triangle{J}^{2+}(K)=2-z^{2}+2z^{2}-z^{2}=2$.
For positively traversing an inverse tangency starting with a pair of
immersions with $u(K)=0$ and intersection of the single immersions we
calculate $\triangle{J}^{2+}(K)=2-z^{2}-(z+2)^{2}+2(z+1)^{2}-0=0$. Starting
with a pair of immersions with $u(K)\neq 0$ or $u(K)=0$ without intersection
of $S_{1}$ and $S_{2}$ we calculate
$\triangle{J}^{2+}(K)=2-(z+2)^{2}+2(z+1)^{2}-z^{2}=0$.
Considering an inverse tangency with opposed orientation, we conclude for the
winding numbers $x=z-1=y$ and $s=z-2$ for a fixed but arbitrary
$z\in\mathbb{Z}$. In this case we obtain $\triangle J^{2+}(K)=0$ too. For
negatively traversing a direct or an inverse tangency we conclude directly
$\triangle J^{2+}(K)=-2$ and $\triangle J^{2+}=0$.

Figure 13: Winding numbers of components by traversing a triple point

Now let $K$ be a pair of immersions traversing a triple point of $S_{1}$ and
$S_{2}$ through homotopy. This general case is related to one of the four
illustrated versions visualized in Figure 13 analogously to the proof of
Viro’s formula. Hereby the winding numbers of the adjacent components are the
same as for the consideration of a single immersion. Let the fixed orthogonal
axes be part of $S_{1}$, the moveable axe is part of $S_{2}$.
By traversing the triple point a triangle with winding number $x$ vanishes and
a new triangle with winding number not equal to $x$ emerges. The amount of
components of $\mathbb{C}\setminus K$ and the amount of double points $n$
remain constant ($\triangle n=0$). However the indices of the double points
$A$, $B$ and $C$ surrounding the triangle change. The indices of all other
double points remain constant. Furthermore we know $\triangle u(K)=0$, as
$S_{1}$ and $S_{2}$ intersect before and after traversing the triple point.
For the change of the ${J}^{2+}$-invariant we get

$\displaystyle\triangle{J}^{2+}(K)$
$\displaystyle=-\triangle\sum\omega_{C}(K)^{2}+\triangle\sum ind_{p}(K)^{2}$
$\displaystyle=-\left(\omega_{D_{\mathrm{new}}}(K)^{2}-\omega_{D_{\mathrm{old}}}(K)^{2}\right)+\triangle
ind_{A}(K)^{2}+\triangle ind_{B}(K)^{2}+\triangle ind_{C}(K)^{2}$

Inserting what we know we get for the four versions

1\. Version: $\triangle{J}^{2+}(K)$
$=-((x+1)^{2}-x^{2})+(x^{2}+2(x+1)^{2}-2x^{2}-(x+1)^{2})=0$ 2\. Version:
$\triangle{J}^{2+}(K)$ $=-((x-3)^{2}-x^{2})+(3(x-2)^{2}-3(x-1)^{2})=0$ 3\.
Version: $\triangle{J}^{2+}(K)$
$=-((x+1)^{2}-x^{2})+(x^{2}+2(x+1)^{2}-2x^{2}-(x+1)^{2})=0$ 4\. Version:
$\triangle{J}^{2+}(K)$
$=-((x+1)^{2}-x^{2})+(x^{2}+2(x+1)^{2}-2x^{2}-(x+1)^{2})=0$.
Therefore we obtain $\triangle J^{2+}(K)=0$ for traversing a triple point .
Hence, the $J^{2+}$-invariant behaves in all three critical homotopy scenarios
as assumed and the lemma is proven.

$\square$

##  3 Further properties and perspectives

This chapter highlights several other properties of the $J^{2+}$-invariant.
Especially interesting is Theorem 3.5 (Theorem B) which states that the
$J^{2+}$-invariant of a pair of immersions $K$ is completely independent of
the $J^{+}$-invariants of the single immersions. The proof of the theorem is
an algorithm constructing a suitable pair of immersions for a given
$J^{2+}$-invariant and given $J^{+}$-invariants. Chapter 3.2 discusses
interesting questions resulting from the introduction of the
$J^{2+}$-invariant.

3.1 Properties of the ${J}^{2+}$-invariant

We begin by showing that the $J^{2+}$-invariant is always even.

###### Proposition 3.1

For all arbitrary pairs of immersions $K$, we know ${J}^{2+}(K)\in
2\mathbb{Z}$.

Proof:  ${J}^{+}(S)$ $\in 2\mathbb{Z}$ holds for all immersions $S$.
Considering an arbitrary pair of immersions $K$ consisting of $S_{1}$ and
$S_{2}$ there always exists a homotopic pair of immersions $K^{*}$ with
disjoint immersions $S_{1}$ and $S_{2}$. In this case we know
${J}^{2+}(K^{*})={J}^{+}(S_{1})+{J}^{+}(S_{2})$ $\in 2\mathbb{Z}$. During the
homotopy of $K^{*}$ into $K$ the $J^{2+}$-invariant does not change or changes
in steps by the value of $2$.

$\square$

Figure 14: Construction of an immersion $S$ with ${J}^{+}(S)>0$

We define $P_{k}$ as a $k$-times connected sum of the immersion $S$ with
$J^{+}(S)=2$, as illustrated in Figure 14. We know $J^{+}(P_{k})=k\cdot 2$,
since the ${J}^{+}$-invariant is additive under taking connected sums. This is
explained in [1] in more detail.

###### Proposition 3.2

For every arbitrary number $n$ $\in 2\mathbb{Z}$ exists an immersion $S$ such
that ${J}^{+}(S)=n$.

Proof:  For $n\leq 0$, we choose $i=1-\frac{n}{2}$ and the corresponding
standard immersion $K_{i}$ with ${J}^{+}(K_{i})=n$. If $n>0$, we choose
$k=\frac{n}{2}$ and the immersion $P_{k}$ with $J^{+}(P_{k})=n$.

$\square$

###### Proposition 3.3

For arbitrary $x$, $y\in 2\mathbb{Z}$ exists a pair of immersions $K$ out of
$S_{1}$ and $S_{2}$ such that $J^{+}(S_{1})=x$, $J^{+}(S_{2})=y$ and
${J}^{2+}(K)=x+y$.

Proof: This follows immediately by Proposition 3.2 and Lemma 2.5.

$\square$

Figure 15: Definition of the immersions $L_{i}$

We define $L_{i}$ for $i$ $\in\mathbb{N}_{0}$ to be the immersions visualized
in Figure 15. By construction of $L_{i}$ and Viro’s formula we conclude:

$\displaystyle{J}^{+}(L_{i})$
$\displaystyle=1+i-\sum\nolimits_{k=1}^{i+1}k^{2}+\sum\nolimits_{k=1}^{i}k^{2}$
$\displaystyle=1+i-(i+1)^{2}=-i-i^{2}=-i\cdot(1+i)$

Before stating a proposition discussing the dependence of orientation of the
$J^{2+}$-invariant we look at the following example showing a homotopy of
$L_{0}$ and $L_{2}$ with varying orientations. First the immersions are
disjoint, then they are merged into one another.

Figure 16: Homotopy of a system of the two immersions $L_{0}$ and $L_{2}$,
oriented differently

Example 5, Figure 16, discusses a homotopy of two immersions $L_{0}$ and
$L_{2}$ differently oriented. Here the $J^{2+}$-invariant increases by merging
the immersions into one another. While traversing three direct tangencies the
${J}^{2+}$-invariant increases each time by the value of $2$. We conclude
${J}^{2+}(K)=-6+3\cdot(+2)=0$.

Figure 17: Homotopy of a system of two immersions $L_{0}$ and $L_{2}$,
oriented similarly

Figure 17 visualizing Example 6 shows two similarly oriented immersions
$L_{0}$ and $L_{2}$, whose $J^{2+}$-invariant decreases by merging them into
one another. During the homotopy three inverse tangencies are traversed, but
this does not change the ${J}^{2+}$-invariant. Similarly, traversing triple
points does not change the $J^{2+}$-invariant. By negatively traversing three
direct tangencies the ${J}^{2+}$-invariant decreases each time by the value of
$2$. Therefore we get ${J}^{2+}(K)=-6+3\cdot(-2)=-12$.
These examples show, that the $J^{2+}$-invariant can change by varying the
orientation of one of the immersions of the pair $K$. For a simultaneous
change of orientation of both immersions, we conclude:

###### Proposition 3.4

The $J^{2+}$-invariant remains constant under a simultaneous change of
orientation of both immersions of a pair $K$.

Proof: Let $K$ be an arbitrary pair of immersions, and we change the
orientation of both immersions simultaneously. The amount of double points and
$u(K)$ do not change. The winding numbers of all components change their sign,
as well as the indices of the double points. As in the formula to calculate
$J^{2+}$ all these values are summed squared, hence $J^{2+}(K)$ does not
change.

$\square$

###### Theorem 3.5

(Theorem B)For three arbitrary even numbers $x$, $y$ and $z$ exists a pair of
immersions $K$ out of $S_{1}$ and $S_{2}$ such that $J^{+}(S_{1})=x$,
$J^{+}(S_{2})=y$ and $J^{2+}(K)=z$.

Proof: Proposition 5.3 already discusses the special case $z=x+y$. For $z\neq
x+y$, the immersions $S_{1}$ and $S_{2}$ must not be disjoint in the pair $K$.
We set $d:=z-x-y$.

Figure 18: Construction of $K^{*}$ out of immersions $L_{\frac{|d|}{2}}$,
$P_{k}$ and $K_{i}$

  * 1.

We define the immersion $S_{1}$ as a connected sum of the immersions $S_{1,1}$
and $S_{1,2}$, whereas $S_{1,2}$ is $L_{\frac{|d|}{2}}$ and $S_{1,1}$ is
defined as immersions such that
${J}^{+}(S_{1,1})=x-{J}^{+}(L_{\frac{|d|}{2}})=x-(-(\frac{|d|}{2})-(\frac{|d|}{2})^{2})$.
For $x+(\frac{|d|}{2})+(\frac{|d|}{2})^{2}>0$, $S_{1,1}$ is the suitable
$P_{k}$, otherwise we choose the appropriate standard immersions $K_{i}$.
Hence, we get

$J^{+}(S_{1})={J}^{+}(S_{1,1})+{J}^{+}(S_{1,2})=x.$

For $y>0$ we choose $S_{2}$ as immersion $P_{\frac{y}{2}}$ otherwise as
$K_{1-\frac{y}{2}}$.
In $K^{*}$ (figure 18) $S_{1}$ and $S_{2}$ are disjoint, hence
${J}^{2+}(K^{*})=x+y$.

  * 2.

For $d>0$, we orientate $S_{1}$ positively and $S_{2}$ negatively. The outer
branch of $S_{2}$ is pushed into the pre-innermost component of $S_{1,2}$ with
winding number $\frac{|d|}{2}$. Hereby $K$ traverses $\frac{|d|}{2}$ direct
tangencies positively, therefore the ${J}^{2+}$-invariant increases by the
value of $2\cdot\frac{|d|}{2}=|d|=d$. For the constructed system $K$ we
calculate

${J}^{2+}(K)={J}^{2+}(K^{*})+\triangle J^{2+}(K)=x+y+d=z.$

  * For $d<0$, we orientate $S_{1}$ and $S_{2}$ positively and the outer branch of $S_{2}$ pushes into $S_{1}$ until $\frac{|d|}{2}+1$ inverse tangencies are traversed positively, $\frac{|d|}{2}$ triple points points are traversed and $\frac{|d|}{2}$ direct tangencies are traversed negatively. Hence, the ${J}^{2+}$-invariant decreases by the value of $2\cdot\frac{|d|}{2}=|d|=-d$, therefore we conclude

${J}^{2+}(K)={J}^{2+}(K^{*})+\triangle J^{2+}(K)=x+y-(-d)=z.$

$\square$

The quantities ${J}^{+}(S_{1})$, ${J}^{+}(S_{2})$ and ${J}^{2+}(K)$ for a
system $K$ out of two immersions $S_{1}$ and $S_{2}$ are therefore completely
independent of each other. The following example illustrates the algorithm.

Figure 19: Construction of $S_{1}$ and $S_{2}$ with given $J^{+}$

Let $x=-6$, $y=-4$ and $z=-16$ be given. The aim is to construct a pair of
immersions $K$ out of $S_{1}$ and $S_{2}$ such that $J^{+}(S_{1})=-6$,
$J^{+}(S_{2})=-4$ and $J^{2+}(K)=-16$. Initially we note that $x+y\neq z$,
therefore $K$ is not a pair of disjoint immersions $S_{1}$ and $S_{2}$.

Figure 20: Pair of immersions $K^{*}$ out of $P_{3}$,$L_{3}$ and $K_{3}$

We calculate $d=-6$.

  * 1\.

For the construction of $S_{1}$ we choose the immersions
$S_{1,2}=L_{\frac{|d|}{2}}=L_{3}$ with $J^{+}(L_{3})=-3-3^{2}=-12$ and
$S_{1,1}$ with $J^{+}(S_{1,1})=x-{J}^{+}(L_{3})=6$, such that $S_{1,1}=P_{3}$
as illustrated in figure 19.
We choose $S_{2}$ as $K_{3}$ with $J^{+}(K_{3})=-4$.
Figure 20 shows the pair of immersions $K^{*}$ with disjoint $S_{1}$ and
$S_{2}$. We calculate $J^{2+}(K^{*})=-10$.

Figure 21: Constructed pair of immersions $K$ with $J^{2+}(K)=-16$

  * 2\.

As $d<0$ we orientate $S_{1}$ and $S_{2}$ positively and push $S_{2}$ into
$S_{1}$, until four inverse tangencies are traversed positively and three
triple points and three direct tangencies are traversed negatively. The
emerging pair of immersions $K$ is illustrated in Figure 21. The
$J^{2+}$-invariant does not change under inverse tangencies and triple points
and decreases under the negative traversing of direct tangencies by the value
of $2$, therefore we get
$J^{2+}(K)=J^{2+}(K^{*})+\triangle J^{2+}(K)=-10+3\cdot(-2)=-16$.

3.2 Further discussion and perspective

The newly introduced $J^{2+}$-invariant poses several questions. For example
whether there are similar invariants for three or more generic immersions in
analogous systems. If so, a more widely applicable definition of the
encircling index is inevitable.
As Arnold has not only introduced the $J^{+}$-invariant, but also $St$, the
Strangeness and the $J^{-}$-invariant, it would be interesting to know whether
there exist analogous invariants for two or more immersions similar to $St$
and $J^{-}$.
Furthermore, in physical systems there occur other crucial events apart from
triple points and inverse and direct (self-)tangencies during homotopies of
orbits. For example the orbit might have velocity zero for some isolated
moment in time or might have collisions. An investigation of the connection of
the $J^{2+}$-invariant and the invariants $\mathcal{J}^{-}$ and
$\mathcal{J}^{+}$ as defined in "Periodic orbits in the restricted three-body
problem and Arnolds $J^{+}$-invariant"[7] could be very interesting.

## Bibliography

$[1]$ Arnold, V.I.. (1994).  Plane curves, their invariants, perestroikas and
classifications. Adv. Soviet Math.. 21.

$[2]$ Whitney, Hassler. (1973). On regular closed curves in the plane.
International Journal of Computer Mathematics - IJCM. 4.
10.1007/978-1-4612-2974-2 3.

$[3]$ Kim, Joontae und Kim, Seongchan. (2017). $J^{+}$-like invariants of
periodic orbits of the second kind in the restricted three-body problem.
Journal of Topology and Analysis. 10.1142/S1793525319500614. S.3-7.

$[4]$ Viro, Oleg. (1996). Generic immersions of circle to surfaces and complex
topology of real algebraic curves. Amer. Math. Soc. Transl. Ser. 2. 173.
10.1090/trans2/173/19.

$[5]$ Haeussler, Hanna. (2020). Die $J^{+}$-Invarianz nach Arnold.
Seminararbeit an der Universität Augsburg.

$[6]$ Jänich, Klaus. (2011). Funktionentheorie - Eine Einführung. 6. Auflage,
S.67-68.

$[7]$ K. Cieliebak, U. Frauenfelder and O. van Koert. (2017). Periodic orbits
in the restricted three-body problem and Arnold’s $J^{+}$-invariant. Regul.
Chaotic Dyn. 22. 408–434.


# Repurposing Knowledge Graph Embeddings for Triple Representation via Weak
Supervision

1st Alexander Kalinowski  Department of Information Science
Drexel University
Philadelphia, USA
<EMAIL_ADDRESS>2nd Yuan An  Department of Information Science
Drexel University
Philadelphia, USA
<EMAIL_ADDRESS>
###### Abstract

The majority of knowledge graph embedding techniques treat entities and
predicates as separate embedding matrices, using aggregation functions to
build a representation of the input triple. However, these aggregations are
lossy, i.e. they do not capture the semantics of the original triples, such as
information contained in the predicates. To combat these shortcomings, current
methods learn triple embeddings from scratch without utilizing entity and
predicate embeddings from pre-trained models. In this paper, we design a novel
fine-tuning approach for learning triple embeddings by creating weak
supervision signals from pre-trained knowledge graph embeddings. We develop a
method for automatically sampling triples from a knowledge graph and
estimating their pairwise similarities from pre-trained embedding models.
These pairwise similarity scores are then fed to a Siamese-like neural
architecture to fine-tune triple representations. We evaluate the proposed
method on two widely studied knowledge graphs and show consistent improvement
over other state-of-the-art triple embedding methods on triple classification
and triple clustering tasks.

###### Index Terms:

triple embedding, knowledge graph embedding, weakly supervised learning

##  I Introduction

Knowledge graphs (KG) are a paradigm for representing head entities
(subjects), tail entities (objects) and the predicates between them ($\langle
h,p,t\rangle$ triples) and form a computational-basis for the Semantic Web.
Due to their size and incompleteness, many approaches have been developed for
compressed representations of KGs, with the majority of these approaches
relying on low-dimensional vector representations, henceforth referred to as
_embeddings_. These embeddings have been shown to be highly effective in link
prediction and knowledge graph completion tasks. However, almost all methods
encode entities and predicates in their own independent embedding vectors,
requiring additional functions to combine the resulting vectors into a single
‘triple’ representation. Such ‘triple’ embeddings have applications in many
downstream tasks, including relation extraction [1] and e-commerce predictions
[2].

Functions explored in the literature typically combine the head and tail
entity representations for a composite ‘triple’ representation, with good
performance on simple, homogeneous graphs. Knowledge graphs, on the other
hand, are heterogeneous multi-graphs where each predicate edge is associated
with a particular semantic label, thus causing issues with the traditional
methods of aggregation for triple representation.

In a heterogeneous multi-graph, there may exist multiple different predicates
between two entities. For example, a person, David Lynch, can play the role of
both an actor and a director in a particular piece of work, in this case the
television series Twin Peaks: ”David Lynch” – actor_in – ”Twin Peaks”, and
”David Lynch” – director_of – ”Twin Peaks”. Triple representations that rely
only on an aggregation of the embeddings of the head and tail entities miss
the contextual information contained in the multitude of predicates. An
alternate approach for encoding a triple is to include the embeddings of all
three components of the triple. However, aggregations of all three components
may offset the information encoded in the entity and predicate embeddings. For
example, a main category of KG embedding approaches is translation-based
models such as TransE [3]. These models assume that predicates can be modelled
as translations from the head entity to the tail entity,
$\mathbf{h}+\mathbf{p}\approx\mathbf{t}$. When combining these representations
into a triple, a simple mechanism would be to sum or average all component
vectors that comprise the triple, i.e.
$\mathbf{e}_{triple}=\mathbf{h}+\mathbf{p}+\mathbf{t}$ where
$\mathbf{e}_{triple}$ represents the embedding vector of the entire triple
$\langle h,p,t\rangle$. Clearly, this is problematic given the assumption that
$\mathbf{h}$ $+$ $\mathbf{p}$ $\approx$ $\mathbf{t}$, substitution into the
equation (I) would lead to a representation of the triple that is twice the
embedding of the tail entity, i.e.

$\mathbf{e}_{triple}=\mathbf{h}+\mathbf{p}+\mathbf{t}\approx\mathbf{t}+\mathbf{t}=2\mathbf{t}$

where the relational semantics of the triple, including any information about
the predicate involved, are entirely lost.

To treat a knowledge graph triple as a first-class object and represent it as
a single vector, Triple2vec [2] learns triple embeddings completely from
scratch, based on the idea of random walks on the line graph of the original
graph. Building new triple representations from scratch ignores the
information already captured in embedding algorithms designed for link
prediction. In contrast, we believe that knowledge graph embedding methods
still maintain their merits, yet require re-working to craft a triple
representation. Rather than building these triple embeddings from scratch, we
aim at re-usability and flexibility. Following suit with the works detailed in
[4], we build a post-processing and fine-tuning technique to re-use knowledge
graph embeddings, saving on computational cost, especially in the case where
pre-trained models are readily available [5].

The key idea of our method is to learn triple embeddings by post-processing
pre-trained entity and predicate embeddings in a _weakly supervised manner_.
Our main inspiration comes from the ideas of semantic textual similarity [6]
and Siamese neural networks [7]. Figure 1 illustrates the architecture of our
system. The method consists of two main components (labeled as (B) and (C) in
Figure 1). In the first component (B), we propose a novel method for
generating weak supervision signals by combining pre-trained entity and
predicate embeddings (from (A)) in a semantic way. In the second component
(C), we apply these signals to fine-tune triple embeddings composed by a
Siamese-like deep neural network.

Figure 1:  The architecture of the weakly supervised learning system for
triple embeddings. (A) The input of the system consists of a knowledge graph
and its pre-trained entity and predicate embeddings. (B) A sampling method
automatically draws pairs of triples for generating pairwise triple similarity
scores (PTSS). (C) A Siamiese-like neural network takes a pair of triples and
composes an initial triple embedding using the constituent embeddings. Initial
embeddings are forward propagated through the network with the resulting
outputs scored by a similarity function. Error between this score and the PTSS
is used to refine the parameters of the entire network. (D) After training,
the system is able to generate embeddings for all triples.

Following the pioneering work in Triple2vec [2] on evaluating triple
embeddings, we also used the tasks of triple classification and triple
clustering and compared their approach to our new proposed methodology. When
evaluated on two widely studied knowledge graphs, our method consistently
surpasses all baselines.

The rest of the paper is organized as follows. Section II presents
preliminaries and formalizes the problem. Section III describes the process of
sampling triples and building pairwise triple similarity scores. Section IV
presents the weakly supervised method for learning triple embeddings. Section
V describes the evaluation settings and process. Section VI analyzes the
evaluation results in comparison to baselines. Section VII discusses related
work. Finally, Section VIII concludes the paper with ideas on future
improvements.

##  II Problem Formulation

A knowledge graph is a 3-tuple $G=(E,P,T)$, where $E$ is a set of entities,
$P$ is a set of predicates, and $T$ is a set of triples $\langle h,p,t\rangle$
connecting a head entity $h\in E$ to a tail entity $t\in E$ by a predicate
$p\in P$ as an edge. A knowledge graph embedding (KGE) method is function
$f_{e}:E\rightarrow\mathbb{R}^{d}$ which encodes entities $v\in E$ as
$d$-dimensional vectors $\mathbf{e}_{v}\in\mathbb{R}^{d}$ and predicates $r\in
P$ as a scoring functions
$f_{r}:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}$.

The _PTSS triple embedding problem_ is to find a function
$g:T\rightarrow\mathbb{R}^{d}$ such that $g$ reuses the output of KGE
functions to encode each triple as a $d$-dimensional vector. In this paper, we
implement $g$ via weak supervision signals from KGE similarity scores.

##  III Building Triple Similarity Scores

We begin by building weak supervision signals for learning triple embeddings.
We seek signals corresponding to similarity measures between the triples that
share a single element (somewhat similar), those that share multiple elements
(more similar), and those that do not share any elements (not similar).
Unfortunately, knowledge graph benchmark datasets (i.e. WordNet or FreeBase)
do not have explicit measures of the similarity between two triples. Building
these similarity scores could require subjective human labelling, usually
accomplished through crowdsourcing, as is done with semantic textual
similarity (STS) tasks [6]. Rather than trust human judgment of these
similarities, we build a scoring mechanism based on the low-dimensional, pre-
trained embeddings of the entities and predicates that compose each triple.

Let $G$ be a KG and $f$ a corresponding KGE function. For triples $\langle
h,p,t\rangle$ with $h,t\in E$ and $p\in P$, the embeddings learned through $f$
may already encode vector space notions of similarities between entities and
predicates. For example, two entities sharing many properties and
relationships will participate in many of the same triples, and we assume that
their respective embeddings should exhibit these similarities. This assumption
is highly dependent on the selection of the pre-trained embedding function
$f$, and there exists a line of research into how much semantic information
these embeddings actually encode [8]. We can use these embeddings, henceforth
referred to as seed embeddings, as a means to proxy pairwise triple similarity
scores. Specifically, given two triples $T_{a}=\langle
h_{a},p_{a},t_{a}\rangle$ and $T_{b}=\langle h_{b},p_{b},t_{b}\rangle$ in the
knowledge graph, we define the pairwise triple similarity score (PTSS) between
$T_{a}$ and $T_{b}$ as an average of the similarities between the embeddings
of their head entities, tail entities, and predicates, respectively. Formally,

$PTSS(T_{a},T_{b})=avg(sim(\mathbf{e}_{h_{a}},\mathbf{e}_{h_{b}}),sim(\mathbf{e}_{p_{a}},\mathbf{e}_{p_{b}}),sim(\mathbf{e}_{t_{a}},\mathbf{e}_{t_{b}}))$,

where $\mathbf{e}_{h_{a}}$ is the embedding vector of the head entity $h_{a}$
and so on, $avg(x,y,z)$ computes an average of its arguments, and the function
$sim(\mathbf{e}_{1},\mathbf{e}_{2})$ is a function for computing the
similarities of two embeddings. The similarity function
$sim(\mathbf{e}_{1},\mathbf{e}_{2})$ could be an arbitrary function capturing
the geometric structures in KG embedding spaces. In our work, we choose cosine
similarity and arithmetic mean, leaving the evaluation of more general
functions to future work. Our evaluation demonstrates that these simple
functions generate useful weak supervision signals.

For the proposed weakly supervised training, we extract from the knowledge
graph a set of training examples containing both _”positive”_ and _”negative”_
examples. For a triple $T_{a}=\langle h_{a},p_{a},t_{a}\rangle$, we define
positive training examples as a set of potential candidate matches. In
particular, triple $T_{a}$ is composed of three elements, or three potential
‘slots’ to match on: the head, the tail and the predicate. For each ‘slot’ we
identify candidate matches as follows. Select $N$ other triples with the same
head entity, $N$ other triples with the same tail entity, $N$ other triples
with the same predicate. For negative examples, we select $N$ other triples
with no commonalities in any available slot. In instances where the set of
candidate matches has cardinality less than $N$, we add the entire set to be
scored. Thus, for each triple in the graph, we build a set of at most $4N$
triples to compare and contrast with.

These scores and training examples can then be utilized as an input to a fine-
tuning process to convert the low-dimensional representations of entities and
predicates into low-dimensional representations of the composite triple. For
any arbitrary knowledge graph, we can build these similarity estimates in an
automated way given that we specify an initial embedding function $f$, a
similarity function $sim(\mathbf{e}_{1},\mathbf{e}_{2})$, an averaging
function $avg(x,y,z)$, and a sampling parameter $N$. We note that these scores
are not meant to be gold standard labels; instead, they are utilized as weak
supervision signals to bootstrap the triple similarity training process.

##  IV Learning Triple Embeddings

For learning triple embeddings, we define a Siamese-like neural network that
takes two triples as input and initializes values at an embedding layer. The
network subsequently forward propagates the values at the embedding layer
through a series of encoding (ENC) layers. The final layer is a scoring (SCO)
layer where the network computes the cosine similarity of the representations.
This score is then compared to the estimated pairwise similarity scores (PTSS)
defined in the previous section, with the errors back-propagated through the
network. To update the triple embeddings, we make the embedding layer tunable
and use the final values as the triple embeddings. In many neural network
approaches, embedding layers are typically initialized randomly or following
the Xavier normalization scheme [9]. In our approach, we are leveraging pre-
trained knowledge graph embeddings, and thus use this information as a
starting point for the initialization of this layer. We experiment with five
aggregation methods for combining the embeddings of the head and tail entities
into a triple representation. Four of these aggregation methods are utilized
in [10]; here, we introduce a fifth: the concatenation of the head and tail
vectors. These choices are outlined in Table I. We investigate the impacts of
these aggregation functions to the learned triple embeddings in conjunction
with the selected underlying KGE methods. The results provide insights for
adopting or generalizing our methods in future applications.

TABLE I: A summary of aggregation operators for constructing triple representations Operator | Definition
---|---
Average (AVG) | $[f(\mathbf{u})\star f(\mathbf{v})]_{i}=\frac{f_{i}(\mathbf{u})+f_{i}(\mathbf{v})}{2}$
Hadamard (HAD) | $[f(\mathbf{u})\star f(\mathbf{v})]_{i}=f_{i}(\mathbf{u})*f_{i}(\mathbf{v})$
Weighted-L1 (L1) | $|f_{i}(\mathbf{u})-f_{i}(\mathbf{v})|$
Weighted-L2 (L2) | $|f_{i}(\mathbf{u})-f_{i}(\mathbf{v})|^{2}$
Concatenation (HT) | $[f(\mathbf{u})\star f(\mathbf{v})]_{i}=f_{i}(\mathbf{u})\|f_{i}(\mathbf{v})$
Notes:
‘$*$’ denotes element-wise product operation.
‘$\|$’ denotes vector concatenation operation.

Formally, an encoding (ENC) and scoring (SCO) layer of the network is defined
as

$\mathbf{o}_{a}=\sigma(\mathbf{W}_{1}(\mathbf{e}_{T_{a}})+\mathbf{b}_{1})$

$\mathbf{o}_{b}=\sigma(\mathbf{W}_{1}(\mathbf{e}_{T_{b}})+\mathbf{b}_{1})$

$\hat{s}=\cos(\mathbf{o}_{a},\mathbf{o}_{b})$

$\mathcal{L}=MSE(s_{a,b},\hat{s})$

where $\mathbf{e}_{T_{a}}$ and $\mathbf{e}_{T_{b}}$ are embeddings of two
triples $T_{a}$ and $T_{b}$, $\mathbf{W}_{1}$ is a shared dense linear layer,
$\mathbf{b}_{1}$ the bias of that layer, and $\sigma$ a non-linearity, in this
case hyperbolic tangent. The outputs $\mathbf{o}_{1}$ and $\mathbf{o}_{2}$ are
then compared using cosine similarity to produce a final estimate of the
similarity between the two inputs, $\hat{s}$. The loss is defined as the mean
squared error between the computed similarity $\hat{s}$ and the estimated
pairwise triple similarity score $s_{a,b}=PTSS(T_{a},T_{b})$ defined in
Section III. For our experiments, we use a single encoding and scoring layer
(ENC-SCO layer) to measure the efficacy of the weakly supervised approach. We
note that many of these layers could be stacked to arbitrary depth. Our
evaluation shows that a single layer can lead to significant improvements over
the baselines and aligns with our goal of re-usability in compute-restricted
environments. We leave the evaluation of more complex network structures for
future applications.

##  V Evaluation

###  V-A Data Sets Selection

In line with Triple2vec [2], we construct two tasks, triple classification and
triple clustering, to evaluate our learned embeddings. However, we deviate
from their work in selection of our benchmark datasets and utilize two well-
established knowledge graphs, WordNet (WN18RR) and Freebase (FB15K-237).
Summary statistics for both datasets are presented in Table II. These
benchmark datasets were selected for two reasons. First, FB15K-237 and WN18RR
are utilized as a benchmark for all of the seed knowledge graph embedding
methods we tested. The frequency of usage in the research community also
allows us to take advantage of publicly available, pre-trained models, such as
those found in LibKGE [5]. Secondly, these graphs are complex in nature,
making our evaluation more reflective of knowledge graphs that occur in
practice.

TABLE II:  Summary of Benchmarks Dataset | WN18RR | FB15K-237
---|---|---
Entities | 40,943 | 14,541
Predicates | 11 | 237
Triples | 93,003 | 310,116
Multi-edge Triples | 218 | 49,214
Num. Strongly CC | 23,105 | 2,678
Num. Weakly CC | 46 | 6

Important to our discussion is the underlying topology of these graphs. We
note that neither graph is connected, meaning that a random walker on each
graph may fail to find a path between two arbitrarily selected nodes.
Furthermore, the distributions of strongly and weakly connected components are
very different. WN18RR presents itself as a denser graph: more strongly and
weakly connected components as compared to FB15K-237, which is sparser and has
only six weakly connected components. The differences in the graph topologies
are an important factor for us to consider in the analysis of our results,
especially in comparison to the Triple2vec approach which is dependent on
random walks over the graph. Of particular interest to our approach is the
number of triples that contain multiple predicates between the head and tail
entities. In WN18RR, this occurs very infrequently; the overwhelming majority
of triples contain a single predicate. On the other hand, 16% of triples found
in FB15K-237 contain more than one predicate. From this difference, we would
expect our approach to have a greater impact on triple classification metrics
when applied to FB15K-237 over WN18RR.

###  V-B Triple Classification Task

To test the efficacy of our proposed model, we probe the resulting triple
embeddings for their ability to predict their respective predicate labels.
Following the work of [11] and [2], we perform this classification using both
a low-capacity and high-capacity model. For our low-capacity model, we employ
one-vs-rest logistic regression, providing the model with triple
representations as features and the predicate indices as labels. For the high-
capacity model, we replace the logistic regression with a multi-layer
perceptron with 512 hidden nodes, trained with the Adam optimizer and batch
size of 256 for a total of 10 epochs. In each case, we split the triples into
training and test sets, with the training set comprising of 80% of the total
triples with the remaining 20% for testing, repeated using five non-
overlapping folds. Both models are then evaluated on their respective Micro-F1
scores.

###  V-C Triple Clustering Task

Motivated by work on word embedding regularities [12], we wish to probe the
triple embedding spaces generated in our experiments to measure the degree to
which they exhibit an underlying structure. Our hypothesis is that embedding
techniques that exhibit higher degrees of clusterability are able to capture
more of the semantics expressed in the input triples. To formalize this
notion, we introduce a definition of clusterability [13]. For some dataset
$X\subseteq\mathbb{R}^{n}$, a description of the clusterability of $X$ is a
function $c:X\rightarrow v$ where $v\in\mathbb{R}$ is a real value. Here, $v$
is a measure of how strong a clustering presence is in the underlying set $X$.
As t-SNE plots, such as those in the clustering evaluation of [2], can be
difficult to tune and interpret [14], we choose to evaluate clusterability via
a metric, although code for generating these plots is included in our open
source implementation. Our process for measuring embedding space
clusterability is as follows. For each triple embedding space, we apply
k-means clustering, selecting k to be the number of known predicates in the
graph. To measure the quality of these clusters, we select the Calinski-
Harabasz (CH) index [15], which is the ratio of between cluster dispersion to
within cluster dispersion. Mathematically, this score is computed as

$v=\frac{\operatorname{Tr}(B_{k})}{\operatorname{Tr}(W_{k})}\times\frac{n_{T}-k}{k-1},$

where $n_{T}$ is the number of triples, $k$ the number of clusters,
$\operatorname{Tr}(B_{k})$ the trace of the between cluster dispersion matrix
and $\operatorname{Tr}(W_{k})$ the trace of the within cluster dispersion
matrix. Higher values of the CH metric indicate a better fit of the k-means
partitioning. This metric has an added benefit in that it tends to be inflated
for convex clusters; we hypothesize convex structures will allow for well-
defined areas corresponding to the predicates we seek to predict.

###  V-D Experimental Design

We choose several popular knowledge graph embedding models for seed
embeddings, including TransE, DistMult, RESCAL, ConvE, RotatE and ComplEx (See
Section VII for more details). We leverage the open source LibKGE package and
associated pre-trained models [5] for their implementations. These pre-trained
models have undergone extensive hyperparameter tuning to achieve the highest
link prediction accuracies. For comparison with Triple2vec, we use an
implementation provided by the authors [2], extended to our benchmark
knowledge graphs and trained for 30 epochs. To minimize the impact of the
choice of embedding dimensionality on the triple classification task, we train
Triple2vec with dimension matching the dimension of pre-trained models from
our approach and set the hyperparameter for the number of walks to be 10.

For all of our experiments, we set the PTSS hyperparameter $N$ to be 5,
selected randomly from the set of all candidate matches, although we note that
this hyperparameter deserves further investigation and tuning to measure its
full effect. We depend on the embedding dimension of each pre-trained
knowledge graph model to be the input dimension of each triple embedding
space. For the remainder of model training, we set the batch size to 128,
optimize using Adam with an initial learning rate of 2e-3 and warm up each
model using 10 percent of the training data.

###  V-E Implementation and Baselines

We dub our proposed method as PTSS in the evaluation. The source code
implementation is available here111https://github.com/yur7nd/ptss. The
baselines to which we compare our method are (1) the state-of-the-art triple
embeddings method, Triple2vec [2] and (2) aggregated pre-trained KG embeddings
by applying the operators in Table I without our PTSS fine-tuning.

Figure 2: Comparison of Micro-F1 scores for Triple2vec and PTSS for the triple
classification task. For display purposes, we selected the best performing
model/aggregation combination.

##  VI Analysis of Results

###  VI-A Comparison of Triple2vec and PTSS

Overall, we find that the vectors generated by PTSS serve as better features
for triple classification than those generated from Triple2vec. Figure 2 show
the comparison of Micro-F1 scores between Triple2vec and the best performing
model and aggregation combinations from our PTSS approach. On WN18RR, our low-
capacity models perform better than, but on par with, the results of
Triple2vec, while our triple vectors serve as better features for the high-
capacity classifier. The best model, a combination of TransE seed vectors and
weighted-L2 aggregation, has F1 score of 0.6723, while Triple2vec
representations with equal dimension only achieves F1 score of 0.4008. The
performance gains are even more dramatic on FB15K-237. Our best model (RotatE
seed vectors with aggregation by concatenation) achieves F1 score of 0.7949
while the best Triple2vec representation of equal dimension only reaches
0.059. The rationale for such a gap in performance on FB15K-237 comes from the
topology of the graph; fewer connected components and lower degree means
random walks are more likely to reach dead ends. Triple2vec was inherently
designed to take advantage of these random walkers; hence, when the graph is
less densely connected, fewer pieces of information are available for their
approach to learn from. Our approach is not dependent on random walks and is
invariant to the underlying graphs topology. In comparison to Triple2vec, we
also note that our approach is more computationally efficient. Building the
line graph $G_{L}$ from the input knowledge graph $G$ can be burdensome, with
a cost of $\mathcal{O}(|T|^{2}+costWeight)$, where $costWeight$ is a factor
for computing edge weights (see VII-C). In comparison, building the PTSS
dataset can be accomplished in $\mathcal{O}(4N\times|T|)$ where $N\ll|T|$.

###  VI-B Assessing the Aggregation Operator

We analyze the impact of the aggregation operations on the triple
classification performance. We find that there is no clear aggregation
function that consistently outperforms all others. For WN18RR, two models
showed best performance using the Hadamard product (ComplEx and ConvE), two
models with top performance using weighted-L1 (DistMult and TransE), while
RESCAL and RotatE performed best with the concatenation and average operators,
respectively. On FB15K-237, ComplEx and DistMult performed best with the
Hadamard product, ConvE and RESCAL with weighted-L2 and RotatE and TransE with
the concatenation operator. These results are counter to our intuition; our
expectation, aligned with results of other lines of research [10], was that
Hadamard product would consistently outperform all other operators. It was
interesting to note that the performance of these operators with respect to
the selected embedding model was also inconsistent across datasets. Only the
ComplEx model showed best performance when combined with the Hadamard product
for both datasets. We also note that while simple, the concatenation method
does yield positive results for three of our tests: in conjunction with RESCAL
on WN18RR and with RotatE and TransE on FB15K-237. We expected this simple
concatenation to benefit from a performance boost due to the extra triple
feature dimensionality (twice that of all other approaches due to the nature
of vector concatenation), yet we do not see a consistent boost across the
board. This is further explained by the correlation of triple feature
dimensionality with model performance, which has a neutral correlation of
0.143. Given the lack of concrete evidence suggesting that any aggregation
operator is superior, we recommend treating this choice as a hyperparameter to
be tuned when applying our methodology to novel knowledge graphs.

###  VI-C Quality of Input Embeddings

We compare the quality of the input seed embeddings to the results of the
triple classification task. As the input embeddings are all trained for the
link prediction task, we present their reported MRR, Hits@1, Hits@3 and
Hits@10 from [5] and assess the correlation of these metrics with our reported
Micro-F1 scores on triple classification. Actual performance metrics from [5]
are reproduced with the best triple classification Micro-F1 scores in Tables
III and IV.

TABLE III: Link Prediction Performance on FB15K-237 Approach | MRR | Hits@1 | Hits@3 | Hits@5 | Best F1*
---|---|---|---|---|---
ComplEx | 0.348 | 0.253 | 0.384 | 0.536 | 0.7346
ConvE | 0.339 | 0.248 | 0.369 | 0.521 | 0.7896
DistMult | 0.343 | 0.250 | 0.378 | 0.531 | 0.7268
RESCAL | 0.356 | 0.263 | 0.393 | 0.541 | 0.0657
RotatE | 0.333 | 0.240 | 0.368 | 0.522 | 0.7949
TransE | 0.313 | 0.221 | 0.347 | 0.497 | 0.7008
*Best F1: the best Micro-F1 scores of our method on triple classification.
TABLE IV: Link Prediction Performance on WN18RR Approach | MRR | Hits@1 | Hits@3 | Hits@5 | Best F1*
---|---|---|---|---|---
ComplEx | 0.475 | 0.438 | 0.490 | 0.547 | 0.582
ConvE | 0.442 | 0.411 | 0.451 | 0.504 | 0.600
DistMult | 0.452 | 0.413 | 0.466 | 0.530 | 0.424
RESCAL | 0.467 | 0.439 | 0.480 | 0.517 | 0.484
RotatE | 0.478 | 0.439 | 0.494 | 0.553 | 0.431
TransE | 0.228 | 0.053 | 0.368 | 0.520 | 0.672
*Best F1: the best Micro-F1 scores of our method on triple classification.

We find little evidence of consistent, strong positive correlations between
performance on the link prediction task and performance on the triple
classification task ($\rho=-0.6377$ and $\rho=0.4285$ for FB15K-237 and
WN18RR, respectively). It is important to note that the best performance on
the link prediction task does not imply best performance on triple
classification. Our architecture is not merely a pass-through mechanism–we are
able to add semantics back to triple representations that were previously lost
in the pre-trained embedding models. It is also interesting to note that poor
performing models on the link prediction task can still be re-purposed to
yield strong triple embeddings. On WN18RR, the TransE approach has the worst
performance for link prediction, but produces entity and predicate embeddings
that can be best used for the pairwise triple similarity scoring required by
our model. For the more complex data in FB15K-237, increasing the complexity
of the pre-trained embeddings does help performance on the triple
classification task–more complicated models such as RotatE and ConvE are able
to achieve superior performance. As with the impact of the aggregation
operator, the choice of which model used to seed the entity and predicate
embeddings entirely depends on the underlying dataset; there is no silver
bullet that always achieves superior performance. The inability to find a
clear, consistent winner across all our experiments further highlights an
important feature of our approach: flexibility. Our methodology is general
enough to be adapted to different aggregation operators and sets of pre-
trained embeddings, all dependent on the complexity of the knowledge graph
being investigated.

###  VI-D Ablation of PTSS

In addition to measuring the impact of the seed embeddings, we also analyze
the lift in Micro-F1 our method provides over simply using the initialized
vectors as features in the triple classification task. We restrict the triple
classification task to include only triples with multiple predicates between
the head and the tail. This allows us to asses the impact our architecture has
on the problem area identified in Section I, namely, the missing predicate
information when aggregating the head and tail representations. These metrics
are shown in Figure 3.

Figure 3: Comparison of Micro-F1 scores for pre-trained KGE models with and
without the fine-tuning of PTSS for the triple classification task, restricted
to the multi-predicate triples from each graph.

In every case, we see a lift in triple classification Micro-F1 on triples with
multiple predicates when using our approach. As suspected, this increase is
greater on FB15K-237, where 20% of the graph features such triples. Even on
WN18RR, where the multi-predicate issue is less prevalent, we still see
higher-quality triple embedding vectors generated from the PTSS approach. For
FB15K-237, the two best performing models are generated from using ConvE and
RotatE as seed vectors, indicating that these seed approaches may have
increased capacity for encoding multi-predicate information into the head and
tail embeddings due to their more complex architectures and scoring functions.
This concurs with our findings in the ablation of PTSS, where ConvE and RotatE
as seed embeddings to our approach outperformed their respective pre-trained
models.

###  VI-E Analysis of Clusterability

We report the Calinski-Harabasz index for all triple embedding spaces in Table
V. For both datasets, we find higher CH scores for the PTSS approach over
Triple2vec. Additionally, scores are across the board higher on FB15K-237 than
WN18RR, indicating we are able to learn higher-quality vectors from this
graph. To test our hypothesis that triple vectors demonstrating greater
clusterability will perform well on the triple classification task, we measure
the correlations between these two scores. There is a strong, positive
correlation ($\rho=0.7820$) between the CH metric and Micro-F1 scores on the
triple classification task for Freebase. The correlation is weaker for WN18RR,
where the embeddings generated from PTSS alone have correlation of
$\rho=0.5421$ with the CH index. When analyzing the CH scores on FB15K-237, we
find that the triple embedding spaces generated using ConvE and RotatE lead to
the highest scores and thus are most clusterable. This agrees with our
findings in overall triple classification performance and our ablation of the
PTSS mechanism. These two approaches lead to the highest-quality triple
embedding spaces; these spaces cluster well with respect to the predicate
labels, and this clusterability is a factor in performance on the triple
classification task.

TABLE V: Calinski-Harabasz scores for evaluating the quality of k-means
clustering on each respective triple embedding space. WN18RR
---
| d=32 | d=64 | d=128 | d=256 | d=512 | d=1024
Triple2vec | 1.0809 | 1.1909 | 1.1211 | 1.0642 | 1.0904 | 1.1031
Agg. | ComplEx | ConvE | DistMult | RESCAL | RotatE | TransE
HT | 10.3760 | 13.1284 | 1.4678 | 11.1171 | 4.3188 | 19.5217
Had. | 19.1498 | 23.7984 | 1.9967 | 19.2833 | 5.0234 | 93.6061
Avg. | 9.8583 | 16.0167 | 2.4876 | 12.4854 | 12.5828 | 28.7761
L1 | 12.3999 | 14.1780 | 4.2113 | 10.6682 | 9.0190 | 74.1507
L2 | 15.8628 | 17.9349 | 1.7894 | 11.9538 | 2.3756 | 88.6926
FB15K-237
| d=32 | d=64 | d=128 | d=256 | d=512 | d=1024
Triple2vec | 0.9838 | 1.000 | 0.9976 | 0.9875 | 0.9924 | 0.9989
Agg. | ComplEx | ConvE | DistMult | RESCAL | RotatE | TransE
HT | 260.9607 | 472.0966 | 254.6109 | 3.0954 | 489.9187 | 282.7902
Had. | 266.2164 | 454.5830 | 258.2619 | 157.2214 | 447.6694 | 185.8585
Avg. | 255.6484 | 454.1992 | 247.7288 | 4.2279 | 454.4069 | 295.6048
L1 | 261.3924 | 450.9421 | 253.9956 | 2.9730 | 454.5488 | 298.8541
L2 | 253.7461 | 459.0331 | 252.6248 | 4.6564 | 450.9278 | 181.9362

##  VII Related Work

We cover the requisite background on the knowledge graph embedding models
tested in our experiments. For a full coverage of these techniques, please see
[16]. These methods fall into three main groups: translation-based models,
semantic-matching models, and graph-based models.

###  VII-A Translational Models

The intuition behind translation-based models is to build vector
representations of $\langle h,p,t\rangle$ such that
$\mathbf{h}+\mathbf{p}\approx\mathbf{t}$. Model choices then depend on which
space or spaces the entities and relations are embedded in as well as the
scoring function used to help the model learn to differentiate between true
triples from the graph and noise triples that do not reflect real-world facts.
TransE [3] is the simplest of these models. It embeds entities and predicates
by using a distance function defined by

$f_{p}(\mathbf{h},\mathbf{t})=-\left\lVert\mathbf{h}+\mathbf{p}-\mathbf{t}\right\rVert_{1/2}$
(1)

While this model is simple, it struggles to properly encode one to many
triples, where a single relation may hold between a head entity and several
tail entities. An entire family of translation-based models exists, adding
various complexities and constraints to the embedding spaces and operations
used to best recover the relations described in the original graph.

###  VII-B Semantic-matching Models

While the translational assumption $\mathbf{h}+\mathbf{p}\approx\mathbf{t}$
gives good geometric intuition as to the types of relations learned during
model training, it is prohibitive for a wide class of relations, including
those with anti-symmetric or complex properties. Semantic-matching models
deviate away from the distance-based assumption and focus on using similarity-
based scoring functions to recover facts from the low-dimensional
representations of entities and relations. Rather than relying on norms and
translations, these methods leverage dot-product-like scoring functions to
measure angles between low-dimensional representations, sometimes referred to
as ‘semantic energy’ functions. The simplest of such models is RESCAL [17],
which relies on a tensor representation of the underlying knowledge graph $X$,
where each entry of the tensor $\mathbf{X}_{ijk}=1$ if the fact is represented
in the knowledge graph, otherwise zero. This tensor can then be factorized
into latent components,

$\mathbf{X}_{k}\approx\mathbf{A}\mathbf{P}_{k}\mathbf{A}^{\top}$ for
$k=1,\ldots,r$

where $P_{k}$ is a matrix of dimension $r\times r$ representing interactions
between each corresponding component and $A$ contains the $r$ dimensional
representations of the entities. Thus, for each $(h,t)$ pair, we can compute
the likelihood they participate in the $k$-th relation as

$f_{k}(\mathbf{h},\mathbf{t})=\mathbf{h}^{\top}\mathbf{P}_{k}\mathbf{t}$.

Contrasted with translation-based models, RESCAL takes advantage of vector
products while capturing interactions between elements of each entity and all
relations. In a simplification, DistMult [18] requires each $P_{k}$ to be
diagonal, reducing the parameters of the model while sacrificing some of its
representational capacity. This reduction in capacity is especially felt when
modeling anti-symmetric relations as interactions in these diagonal matrices
have no notion of directionality. To circumvent this issue, the ComplEx [19]
model allows for the low-dimensional representations to live in the complex
space $\mathbb{C}$. The scoring function used by the ComplEx model is defined
as

$f_{k}(\mathbf{h},\mathbf{t})=\Re(\langle\mathbf{h},\mathbf{p}_{k},\mathbf{t}\rangle)$
where $\mathbf{p}_{k}\in\mathbb{C}^{r}$.

By allowing the representations to be complex-valued, the model can handle the
asymmetries of many relations present in knowledge graphs, yet score the
likelihoods of facts existing using only the real-valued vectors. The RotatE
approach also utilizes complex-valued representations, but it also models each
individual relation as a rotation between the head and tail entities. This
additional parameterization allows for increased learning capacity with the
end goal of capturing anti-symmetry, inversion, and composition. The work of
[20] takes this one step further, defining the ConvE model where entities
interact through the convolution operator. This introduces additional non-
linearities through which the model can increase the capacity for learning
complicated relational structures.

###  VII-C Graph Models

Besides exploiting the information in knowledge graph triples through
translational or factorization models, graphical approaches seek to exploit
information contained in the graph’s structure itself. This represents a shift
from thinking of knowledge graphs embeddings of having separate embedding
spaces for entities and relations. In the work of [10], the authors capture
this shift by suggesting aggregation methods for combining nodes (entities in
the KG) to represent edges (triples in the KG). We reproduce the summary of
these operators along with new aggregation methods we use in our study in
Table I.

In order to generate node embeddings to use with these aggregation schemes,
the node2vec approach utilizes random walks over the graph to generate
sequences of nodes, where each sequence is treated as a sentence and fed into
a word2vec-like model [10]. In our experiments, we consider the Triple2vec [2]
approach, which directly learns triple representations without using
aggregations of nodes by converting the original graph $G$ to a line graph
$G_{L}$. To reflect edge weights in this new line graph, the authors introduce
the triple frequency and inverse triple frequency, defined as

$TF(p_{i},p_{j})=\log(1+\mathbf{C}_{i,j})$
$ITF(p_{j},E)=\log\frac{|E|}{|{p_{i}:\mathbf{C}_{i,j}>0}|}$

where $C_{i,j}$ counts the number of times the predicates $p_{i}$ and $p_{j}$
link the same heads and tails, and $E$ is the set of edges. They use these
metrics to build a symmetric matrix

$\mathbf{C}_{M}(i,j)=TF(p_{i},p_{j})\times ITF(p_{j},E)$

to represent a vector for each predicate in the graph; each vector in
$\mathbf{C}_{M}$ can then be used to build a matrix of pairwise similarities
between predicates, denoted $\mathbf{M}_{R}$. This weighting scheme is
expected to push the embeddings of triples with similar predicates to similar
neighborhoods in the embedding space; we hypothesize that if this metric is
effective, the resulting triple embedding vectors should serve as good
features for triple classification. Given this weighted line graph, a random
walk strategy to create sequences of triples is then applied. The walk-based
nature of this approach suits small, connected graphs and hence may not extend
to larger, heterogeneous and complex-typed knowledge graphs.

###  VII-D Post-Processing of Embeddings

Aside from leveraging previous knowledge graph embedding models, our work
introduces a novel way of repurposing these embeddings for alternate tasks.
This work was inspired by post-processing procedures for word and sentence
embeddings, many of which are detailed in [4]. A key idea in this area is to
regularize the embedding space, leaning on the notion of isotropy of the space
to better reflect pairwise similarities, which we directly estimate in an
unsupervised way in our approach. The first of these was published by [21],
who utilize pre-trained word embeddings, average the corresponding vectors for
each word in a sentence for a singular representation, and then modify that
representation using principal component analysis and singular value
decomposition. The authors find that many of the dimensions used in the
sentence representation are unnecessary and subsequently introduce noise; the
application of PCA/SVD reduces this noise and builds more compact features
that retain the majority of the learned information. The removal of these
dominating components, some of which merely reflect the frequency of a word in
the underlying corpus, leads to more robust representation. In the work of
[22], the authors leverage a pre-trained word2vec model (GloVe [23] in their
case). By analyzing the geometry of the subspace generated by building a
matrix $\mathbf{A}\in\mathbb{R}^{d\times n}$ for $d$-dimensional embeddings of
the $n$ words in a given sentence, the authors build a new orthogonal basis
vector that captures the general semantics of the words and their contexts.
Applying a sliding window QR factorization to the matrix $\mathbf{A}$ and re-
weighting by three metrics: a word’s novelty, significance, and uniqueness–a
new semantically motivated sentence representation is generated. Not only is
the approach computationally efficient, but it allows for the re-purposing of
pre-trained word vectors, such as those from GloVe, allowing for new
representations to be generated using no new training data. The work of [1]
found that these sentence representations best align with low-dimensional
knowledge graph representations; we hope to extend their work utilizing our
novel triple representations.

##  VIII Conclusion and Future Work

In this work, we introduce an approach for re-using pre-trained knowledge
graph embeddings for the purpose of building explicit triple representations.
To this end, we introduce the pairwise triple similarity score as a means of
bootstrapping similarity scores between triples in a weakly supervised way.
Using these scores, we develop a novel model to build semantically motivated
triple embeddings. We show that this model consistently beats the state-of-
the-art method, Triple2vec, on two well-known knowledge graph benchmarks.
Future work on our methodology involves additional hyperparameter tuning, an
ablation study on the impact of the number of pairwise triple similarity score
samples and expansion to other knowledge graph benchmark datasets. We are also
interested in investigating alternatives to our PTSS scoring scheme that may
better leverage information contained in the pre-trained seed embeddings, such
as including the notion of triple frequency and inverse triple frequency used
to calculate predicate weights in [2].

## Acknowledgment

We would like to acknowledge the authors of Triple2vec [2], who shared their
implementation to serve as a baseline for comparison. We would also like to
thank the maintainers of LibKGE [5] for providing the pre-trained knowledge
graph embedding models.

## References

  * [1] A. Kalinowski and Y. An, “Exploring sentence embedding structures for semantic relation extraction,” in 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–7, 2021.
  * [2] V. Fionda and G. Pirrò, “Learning triple embeddings from knowledge graphs,” vol. 34, pp. 3874–3881, Apr. 2020.
  * [3] A. Bordes, N. Usunier, A. García-Durán, J. Weston, and O. Yakhnenko, “Translating embeddings for modeling multi-relational data,” in Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems (NeurIPS) 2013. (C. J. C. Burges, L. Bottou, Z. Ghahramani, and K. Q. Weinberger, eds.), pp. 2787–2795, 2013.
  * [4] S. Tang, M. Mousavi, and V. R. de Sa, “An empirical study on post-processing methods for word embeddings,” 2019.
  * [5] S. Broscheit, D. Ruffinelli, A. Kochsiek, P. Betz, and R. Gemulla, “LibKGE \- A knowledge graph embedding library for reproducible research,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, pp. 165–174, 2020.
  * [6] D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia, “Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation,” Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), 2017.
  * [7] N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 11 2019.
  * [8] N. Jain, J.-C. Kalo, W.-T. Balke, and R. Krestel, “Do embeddings actually capture knowledge graph semantics?,” in Eighteenth Extended Semantic Web Conference (ESWC) - Research Track, 2021.
  * [9] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” Journal of Machine Learning Research - Proceedings Track, vol. 9, pp. 249–256, 01 2010.
  * [10] A. Grover and J. Leskovec, “Node2vec: Scalable feature learning for networks,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, (New York, NY, USA), p. 855–864, Association for Computing Machinery, 2016.
  * [11] A. Conneau and D. Kiela, “SentEval: An evaluation toolkit for universal sentence representations,” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), (Miyazaki, Japan), European Language Resources Association (ELRA), May 2018.
  * [12] T. Mikolov, Q. V. Le, and I. Sutskever, “Exploiting similarities among languages for machine translation,” CoRR, vol. abs/1309.4168, 2013.
  * [13] A. Adolfsson, M. Ackerman, and N. C. Brownstein, “To cluster, or not to cluster: An analysis of clusterability methods,” Pattern Recognition, vol. 88, p. 13–26, Apr 2019.
  * [14] M. Wattenberg, F. Viegas, and I. Johnson, “How to use t-sne effectively,” Distill, 2016.
  * [15] T. Caliński and H. JA, “A dendrite method for cluster analysis,” Communications in Statistics - Theory and Methods, vol. 3, pp. 1–27, 01 1974\.
  * [16] Q. Wang, Z. Mao, B. Wang, and L. Guo, “Knowledge graph embedding: A survey of approaches and applications,” IEEE Transactions on Knowledge and Data Engineering, vol. PP, pp. 1–1, 09 2017.
  * [17] M. Nickel, V. Tresp, and H.-P. Kriegel, “A three-way model for collective learning on multi-relational data,” in Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, (Madison, WI, USA), p. 809–816, Omnipress, 2011.
  * [18] B. Yang, W. Yih, X. He, J. Gao, and L. Deng, “Embedding entities and relations for learning and inference in knowledge bases,” in 3rd International Conference on Learning Representations, ICLR 2015, 2015.
  * [19] T. Trouillon, C. R. Dance, É. Gaussier, J. Welbl, S. Riedel, and G. Bouchard, “Knowledge graph completion via complex tensor factorization,” Journal of Machine Learning Research (JMLR), vol. 18, no. 130, pp. 1–38, 2017.
  * [20] T. Dettmers, M. Pasquale, S. Pontus, and S. Riedel, “Convolutional 2d knowledge graph embeddings,” in Proceedings of the 32th AAAI Conference on Artificial Intelligence, pp. 1811–1818, February 2018.
  * [21] S. Arora, Y. Liang, and T. Ma, “A simple but tough-to-beat baseline for sentence embeddings,” in 5th International Conference on Learning Representations (ICLR) 2017 - Toulon, France, Jan. 2017.  5th International Conference on Learning Representations.
  * [22] Z. Yang, C. Zhu, and W. Chen, “Parameter-free sentence embedding via orthogonal basis,” in Proceedings of the 2019 Conference EMNLP-IJCNLP, 2018\.
  * [23] J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543, Oct. 2014.