# Hydrodynamics of thermally-driven chiral propulsion and separation

E. Kirkinis<EMAIL_ADDRESS>Department of Materials Science &
Engineering, Robert R. McCormick School of Engineering and Applied Science,
Northwestern University, Evanston IL 60208 USA  Center for Computation and
Theory of Soft Materials, Northwestern University, Evanston IL 60208 USA  A.
V. Andreev  Department of Physics, University of Washington, Seattle WA 98195
USA  M. Olvera de la Cruz  Department of Materials Science & Engineering,
Robert R. McCormick School of Engineering and Applied Science, Northwestern
University, Evanston IL 60208 USA  Center for Computation and Theory of Soft
Materials, Northwestern University, Evanston IL 60208 USA

###### Abstract

Considerable effort has been directed towards the characterization of chiral
mesoscale structures, as shown in chiral protein assemblies and carbon
nanotubes. Here, we establish a thermally-driven hydrodynamic description for
the actuation and separation of mesoscale chiral structures in a fluid medium.
Cross flow of a Newtonian liquid with a thermal gradient gives rise to chiral
structure propulsion and separation according to their handedness. In turn,
the chiral suspension alters the liquid flow which thus acquires a transverse
(chiral) velocity component. Since observation of the predicted effects
requires a low degree of sophistication, our work provides an efficient and
inexpensive approach to test and calibrate chiral particle propulsion and
separation strategies.

††preprint: APS/123-QED

Chirality, denoting the lack of superposition ability of structures on mirror
images, is a characteristic of various assemblies including carbon nanotubes,
viruses and actin filaments, and is essential for their function. Since left-
and right-handed amino acids lead to different protein structures, their
homochirality is required for biological function such as gene encoding [1].
Chiral proteins can sometimes lead to chiral mesoscale structures; some
organisms with chiral body structures have chiral cells [2]. Therefore, the
chirality of proteins may be responsible for the chiral mesoscale structures
found in cell media. Chiral mesoscale assemblies are formed in peptide
amphiphiles with chiral aminoacids [3] and in many carbon based systems, which
have required great efforts to understand and characterize [4]. However, the
mechanism by which chirality manifests at the mesoscale is not well
understood. Here, we propose ways of actuating and separating mesoscale chiral
structures such as helices [5, *McCourt2022], helicoidal scrolls [7] and
twisted ribbons [8].

In chemical and biological systems various mesoscale structures move and
function in an aqueous environment in the presence of thermal gradients
induced by chemical reactions [9]. Temperature gradients may alter the liquid
material parameters and in particular, viscosity, as this was demonstrated in
laser-induced thermophoresis experiments [10] and the associated theory of a
single hot particle in a viscous liquid [11]. This motivates us to study the
_hydrodynamic_ motion of chiral suspensions in Stokes flow in the presence of
temperature gradients. The corresponding chiral current
$\mathbf{j}^{\textrm{ch}}$ is perpendicular to the plane formed by the base
flow direction and the temperature gradient (cf. Fig. 1). The motion of the
chiral suspension also perturbs the base flow and endows it with a transverse
(chiral) velocity component. It is noteworthy that the chiral suspension also
exerts a screw torque on the confining walls, in the direction of the base
flow. The hydrodynamic description developed in this article implies averaging
over the tumbling motion of the chiral particles and applies at time scales
longer than the tumbling time [12]. It can be understood as a “continuum”
formulation for the motion of a chiral suspension and thus _differs_ from the
majority of propulsion descriptions which are based on a resistance matrix at
the level of a single suspended particle [13]. The equivalence of the two
approaches was discussed in the recent review article by Witten and Diamant
[14].

Figure 1:  A chiral current ${j}^{\textrm{ch}}\hat{\mathbf{z}}$ of $n$ chiral
particles suspended in a classical liquid is induced by a vertical temperature
gradient $\nabla T=\partial_{y}T\hat{\mathbf{y}}$ and shear flow
$\mathbf{v}=u(y)\hat{\mathbf{x}}$. In a nonracemic suspension of chiral
particles the liquid is also perturbed by chirality and acquires a transverse
chiral velocity component $\delta v(y)\hat{\mathbf{z}}$ perpendicular to the
direction of the base flow $u(y)\hat{\mathbf{x}}$.

Motion of chiral particles suspended in a classical liquid is associated with
a chiral current and in particular
$\mathbf{j}^{\pm}=n^{\pm}\mathbf{v}_{p}^{\pm}$ where $n^{\pm}$ is the number
density of right and left-handed particles, respectively and
$\mathbf{v}_{p}^{\pm}$ is their respective velocity. For simplicity we
consider an incompressible liquid where the right and left-handed particles
are mirror-images of each other. We can thus define a chiral current
$\mathbf{j}^{\textrm{ch}}$ (cf. Fig. 1) of the form

$\mathbf{j}^{\textrm{ch}}=\mathbf{j}^{+}-\mathbf{j}^{-}.$ (1)

In the presence of temperature gradients a phenomenological expression for the
chiral current $\mathbf{j}^{\textrm{ch}}$ based on symmetry considerations and
which has a low power of derivatives of vorticity is

$\mathbf{j}^{\textrm{ch}}=\frac{n}{T}\left[\beta_{1}(\nabla
T\cdot\nabla)\textrm{curl}\mathbf{v}+\beta_{2}\nabla
T\times\nabla^{2}\mathbf{v}\right],$ (2)

where $n=n^{+}+n^{-}$ and $T,\mathbf{v}$ are the liquid’s, undisturbed by
chirality, temperature and velocity respectively. $\beta_{1,2}$ are described
below. As mentioned above, the magnitude of the chiral current is determined
by the complicated tumbling motion of the particles caused by thermal
fluctuations and the inhomogeneous flow. The phenomenological expression (2)
is written to lowest order in the driving flow. At strong drives thermal
fluctuations are subdominant, and the magnitude of the chiral current is
determined by averaging over corresponding Jeffery orbits in the nonuniform
flow. This problem, but in a different context, was discussed in [15].

In Ref. [12] the chiral current in an isothermal system was shown to be
described by the expression

$\mathbf{j}^{\textrm{ch}}=n\beta\nabla^{2}\textrm{curl}\mathbf{v}.$ (3)

Insight into the physical origin of Eq.(2) may be obtained by applying Eq.(3)
to a shear flow in the presence of temperature gradients. In particular,
allowing temperature dependence of the liquid viscosity $\eta$ and
implementing the resulting vorticity equation, Eq.(3) leads to

$\mathbf{j}^{\textrm{ch}}\sim n\beta\frac{\eta^{\prime}}{\eta}\left[\nabla
T\times\nabla^{2}\mathbf{v}+(\nabla
T\cdot\nabla)\textrm{curl}\mathbf{v}\right],$ (4)

where a prime denotes differentiation with respect to temperature $T$ and we
retained only leading order terms in temperature gradients. Thus, the
coefficients $\beta_{1}$ and $\beta_{2}$ in (2) can be expressed in terms of
the logarithmic derivative of viscosity with respect to temperature. We note
that in the absence of temperature gradients chiral separation is possible
only in nonstationary or nonlinear flows [12] as can be seen by the vorticity
equation and (3). In the presence of temperature gradients chiral separation
is possible even in the creeping flow regime. This is important for biological
systems, which operate at low Reynolds numbers.

Recent literature examines the way microorganisms and active particles move in
liquids with spatially-varying viscosity [10, 11, 16, *Shoele2018]. In what
follows we are primarily concerned with suspensions where the viscosity $\eta$
of the base liquid varies with temperature. A temperature gradient will then
give rise to a chiral current of the forms (3) and (4).

The chiral density $n^{\textrm{ch}}=n^{+}-n^{-}$ satisfies the conservation
law [12]

$\partial_{t}n^{\textrm{ch}}+\textrm{div}(\mathbf{v}n^{\textrm{ch}})+\textrm{div}\left[\mathbf{j}(n^{\textrm{ch}})+\mathbf{j}^{\textrm{ch}}(n)\right]=0,$
(5)

where $\mathbf{j}(n^{\textrm{ch}})=-D\nabla
n^{\textrm{ch}}-n^{\textrm{ch}}\lambda_{T}\nabla
T-n^{\textrm{ch}}\lambda_{p}\nabla p,$ is the diffusive current relative to
the liquid [18, 12] and $\mathbf{j}^{\textrm{ch}}(n)$ is Eq. (3). The density
$n$ of chiral particles satisfies a similar conservation law, which is
affected by chirality in a non-racemic mixture

$\partial_{t}n+\textrm{div}(\mathbf{v}n)+\textrm{div}\left[\mathbf{j}(n)+\mathbf{j}^{\textrm{ch}}(n^{\textrm{ch}})\right]=0,$
(6)

so that Eq. (5) and (6) satisfy the Onsager principle of the symmetry of the
kinetic coefficients [18].

A chiral suspension imparts stresses on the suspending liquid. To leading
order in velocity gradients these stresses, allowed by symmetry, read

$\displaystyle\sigma_{ij}^{{\textrm{ch}}}$ $\displaystyle=$
$\displaystyle\eta(T)n^{\textrm{ch}}\left\\{\alpha\left[\partial_{i}(\textrm{curl}\mathbf{v})_{j}+\partial_{j}(\textrm{curl}\mathbf{v})_{i}\right]\right.$
(7)
$\displaystyle+\frac{\alpha_{1}}{T}\left.\left[\epsilon_{kli}V_{kj}+\epsilon_{klj}V_{ki}\right]\partial_{l}T\right\\},$

where $V_{ij}$ is the rate-of-strain tensor. The first term of Eq. (7)
introduced in [12] was discussed in the recent review [14]. The second term
exists only when temperature gradients are present in the liquid.

The coefficients $\beta$, $\alpha$ and $\alpha_{1}$ in (3) and (7) are
determined in the low Reynolds number regime by studying the particle motion
in the surrounding liquid [13]. They may be estimated as

$\alpha\sim\alpha_{1}\sim\chi R^{4}\quad\textrm{and}\quad\beta\sim\chi R^{3},$
(8)

where $R$ is the chiral particle radius and $\chi$ is the degree of chirality
in the shape of the particles. Eq. (8) provides the order of magnitude
estimates of these coefficients. Their precise determination for a specific
particle shape however requires solving hydrodynamic equations for a tumbling
particle in the presence of temperature and velocity gradients, and is beyond
the scope of our work.

In the _absence_ of chirality the liquid satisfies the Navier-Stokes equations
and is considered incompressible

$\rho{Du_{i}}/{Dt}=\partial_{k}\sigma_{ik}\quad\textrm{and}\quad\partial_{i}u_{i}=0,$
(9)

where the Cauchy stress tensor $\sigma_{ik}$ is given by
$\sigma_{ik}=-p\delta_{ik}+\eta\left(\frac{\partial u_{i}}{\partial
x_{k}}+\frac{\partial u_{k}}{\partial x_{i}}\right)i,k=1,2,3,$ $\rho$ the
density of the liquid and $p$ is the pressure. Conservation of energy in an
incompressible liquid is expressed in the form [18]

$\rho c_{p}(\partial_{t}T+\mathbf{v}\cdot\textrm{grad}T)=k_{th}\nabla^{2}T$
(10)

where $c_{p}$ is the specific heat at constant pressure and $k_{th}$ the
thermal conductivity of the liquid.

Consider pressure-driven flow, in the absence of chiral particles, in a
channel with unevenly heated walls (cf. Fig. 1). With
$\mathbf{v}=u(y)\hat{\mathbf{x}}$, $\nabla T=\partial_{y}T\hat{\mathbf{y}}$,
the Navier-Stokes equations (9) and energy balance (10) in the creeping flow
approximation reduce to

$\frac{d}{dy}\left[\eta(T)\frac{du}{dy}\right]=\frac{dp}{dx},\qquad\frac{d^{2}T}{dy^{2}}=0,$
(11)

respectively, with boundary conditions

$u(0)=u(d)=0,\quad T(0)=T_{0},\quad T(d)=T_{0}+\Delta T.$ (12)

The temperature profile thus obtained is $T(y)=T_{0}+\frac{y}{d}\Delta T$. The
solution of the first of Eq. (11) with boundary conditions (12) becomes

$u(y)=\frac{T_{e}^{2}d^{2}\partial_{x}p}{2\eta(T)(\Delta
T)^{2}}\frac{\sum\limits_{\left\\{i,j,k\right\\}}e^{X_{i}}\left\\{\textrm{Ei}_{1}(X_{i})\left[\frac{e^{X_{j}}}{X_{k}^{2}}-\frac{e^{X_{k}}}{X_{j}^{2}}\right]+\frac{1}{X_{j}X_{k}}\left(\frac{1}{X_{k}}-\frac{1}{X_{j}}\right)\right\\}}{\left[\textrm{Ei}_{1}(X_{0})-\textrm{Ei}_{1}(X_{1})\right]e^{X_{0}+X_{1}}+\frac{e^{X_{0}}}{X_{1}}-\frac{e^{X_{1}}}{X_{0}}},$
(13)

where the symbol $\left\\{i,j,k\right\\}$ denotes cyclic permutation of $i,j$
and $k$, and $\textrm{Ei}_{1}(X)=\int_{1}^{\infty}\frac{e^{-kX}}{k}dk$ is the
exponential integral. Here we employed the well-documented Arrhenius-type law
[19]

$\eta(T)=\eta_{0}e^{\frac{E}{R_{g}(T+T_{A})}},$ (14)

valid in the $243$ to $373$ K temperature range, since it encompasses linear
and other exponential laws [20, *Davis1983, *Wall1996], as special cases. $E$
is the activation energy, $R_{g}$ is the gas constant and $T_{A}$ is a
temperature correction, unique to each viscous liquid, cf. [19] and Table 1.
In Eq. (13) $X_{i}=\frac{Te}{T_{A}+T_{i}},\quad i=0,1,2,\quad
T_{e}=\frac{E}{R_{g}},$ where $T_{0}$ and $T_{1}=T_{0}+\Delta T$ are the lower
and upper channel wall fixed temperatures, respectively (cf. Fig. 1) and
$T_{2}\equiv T=T_{0}+\frac{y}{d}\Delta T$.

Now consider the presence of chiral particles and define the chiral separation
velocity

$\mathbf{v}^{\textrm{ch}}\equiv\mathbf{j}^{\textrm{ch}}/n$ (15)

relative to the liquid by employing (3). With respect to the geometry
displayed in Fig. 1 it has the form
$\mathbf{v}^{\textrm{ch}}=v^{\textrm{ch}}(y)\hat{\mathbf{z}}$ and its
magnitude is

$v^{\textrm{ch}}(y)=\chi\frac{R^{3}}{d}\frac{X_{2}^{4}\Delta
T\partial_{x}p}{2\eta(T)T_{e}}\frac{\left[\textrm{Ei}_{1}(X_{1})-\textrm{Ei}_{1}(X_{0})\right]e^{X_{0}+X_{1}}+\sum\limits_{i\neq
j=0,1}\frac{(-1)^{i}e^{X_{j}}}{X_{i}^{2}}\left[\frac{1}{2}+\frac{1}{X_{i}}\left(\frac{1}{X_{2}}-\frac{1}{2}\right)\right]}{\left[\textrm{Ei}_{1}(X_{0})-\textrm{Ei}_{1}(X_{1})\right]e^{X_{0}+X_{1}}+\frac{e^{X_{0}}}{X_{1}}-\frac{e^{X_{1}}}{X_{0}}}.$
(16)

In Fig. 2 we plot the closed form expression (16) for the chiral separation
velocity $v^{\textrm{ch}}$ in cm/sec vs. channel elevation $y$ in cm for two
temperature variations $\Delta T$ between the lower (at $y=0$) and upper (at
$y=0.1$ cm) channel walls. The chiral separation velocity is non-zero close to
the solid walls located at $y=0,d$, even though no-slip boundary conditions
are satisfied by the base liquid. This is the case because, according to Eq.
(3), chiral particle velocities become prominent in the vicinity of large
vorticity gradients, and these are present close to the walls.

Table 1: Definitions and material parameters [19] Quantity | Value | Definition
---|---|---
$\eta$ ($\textrm{g}\>\textrm{cm}^{-1}\textrm{sec}^{-1}$) | 2.03 | viscosity of BM-4 oil at $25\text{\,}\mathrm{\SIUnitSymbolCelsius}$ [19]
$R$ (cm) | $5\times 10^{-3}$ | chiral particle radius
$d$ (cm) | $0.1$ | channel width
$U_{0}$ (cm/sec) | $0.1$ | Poiseuille velocity
$T_{0}$ (K) | $298.15$ | lower channel wall temperature
$\gamma$ ($K^{-1}$) | $0.07$ | BM-4 oil [19]
$E$ (kJ/mol) | $7.5$ | activation energy of BM-4 oil [19]
$R_{g}$ (J/ (mol K) | $8.31441$ | gas constant
$T_{\textrm{A}}$ (K) | $-186$ | BM-4 oil temp. correction [19]
$n^{\textrm{ch}}(\textrm{cm}^{-3})$ | $R^{-3}$ | chiral density $n^{+}-n^{-}$
$n\;(\textrm{cm}^{-3})$ | $R^{-3}$ | particle number density $n^{+}+n^{-}$
$u$ (cm/sec) |  | basic shear flow velocity
$v^{\textrm{ch}}$ (cm/sec) |  | chiral separation velocity
$\delta v$ (cm/sec) |  | chiral correction to flow velocity

To obtain a better understanding of the effect, we average (16) over the
channel width $d$ and expand with respect to $\Delta T$ to obtain $\langle
v^{\textrm{ch}}\rangle\sim 2\chi R^{3}\gamma\frac{\Delta
T}{d}\frac{\partial_{x}p}{\eta}$, to leading order in $\Delta T$. It is more
illuminating however to replace the pressure gradient with a characteristic
velocity $U_{0}$ of Poiseuille flow by averaging the (undisturbed by
chirality) base Poiseuille profile
$u\sim\frac{\partial_{x}p}{2\eta(T_{0})}(y^{2}-yd)$ over the channel width
$d$. This gives $U_{0}=-\frac{\partial_{x}p}{12\eta(T_{0})}d^{2}$, and
substituting into the expression for $\langle v^{\textrm{ch}}\rangle$ we
obtain the chiral separation velocity

$\langle
v^{\textrm{ch}}\rangle=24\chi\left(\frac{R}{d}\right)^{3}U_{0}\gamma\Delta T,$
(17)

where we defined the average of a function $f(y)$ with respect to the channel
width to be $\langle f\rangle=\frac{1}{d}\int_{0}^{d}f(y)dy$, $R$ is chiral
particle size and $d$ channel width.
$\gamma={E}/\left[{R_{g}(T_{0}+T_{A})^{2}}\right]$ is the logarithmic
derivative of the viscosity which arises from the linearization of (14)
$\eta(T)\sim\eta(T_{0})\left[1-\gamma(T-T_{0})\right].$

Considering BM-4 oil [19], $\Delta T=$10\text{\,}\mathrm{K}$$ and the values
displayed in Table 1, Eq. (17) leads to the estimate

$v^{\textrm{ch}}\sim 2\chi\;\mu\textrm{m/sec}.$ (18)

The Reynolds number is $\textrm{Re}\sim 3.4\times 10^{-3}$. Analogous results
can be derived for silicon oils employed in the experiments of Ehrhard [23]
and other liquids reported in the literature [19]. Water can also be used,
although it leads to Reynolds numbers higher than those reported here.

_Pertubation of liquid velocity by the chiral suspension._ \- A chiral
suspension imparts stresses on the suspending liquid. To leading order in
gradients of vorticity these stresses, allowed by symmetry are given by (7).
The liquid velocity $\mathbf{v}$ acquires a chirality-induced component
$\delta v$ perpendicular to the plane of the base flow

$\mathbf{v}=u(y)\hat{\mathbf{x}}+\delta v(y)\hat{\mathbf{z}}.$ (19)

With the correction (7) the Cauchy stress tensor reads

$\sigma_{ij}=-p\delta_{ij}+\eta(\partial_{i}u_{j}+\partial_{j}u_{i})+\sigma_{ij}^{{\textrm{ch}}}.$
(20)

Conservation of linear momentum $\partial_{j}\sigma_{ij}=0$ along the flow
direction $\hat{\mathbf{x}}:-\partial_{x}p+\partial_{y}(\eta\partial_{y}u)=0,$
is now accompanied by its chirality-induced counterpart that is perpendicular
to the base flow direction

$\hat{\mathbf{z}}:\partial_{y}(\eta\partial_{y}\delta v)-n^{\textrm{ch}}\chi
R^{4}\partial_{y}(\eta\partial_{y}^{2}u)=0,$ (21)

and satisfies no-slip boundary conditions $\delta v(0)=\delta v(d)=0.$

Figure 2:  Chiral separation velocity $v^{\textrm{ch}}$ in cm/sec from Eq.
(16), perpendicular to the $x$-$y$ plane formed by a base Poiseuille flow and
the vertical temperature gradient directions (cf. Fig.1). $y$ is vertical
channel coordinate in cm and we employed the Arrhenius-type temperature
dependent viscosity law (14) for a BM-4 oil [19]. $\chi$ has been set equal to
$1$.

The solution $\delta v$ of Eq. (21) is displayed in Fig. 3 in cm/sec vs.
channel elevation $y$ in cm for two temperature variations $\Delta T$ between
the lower (at $y=0$) and upper (at $y=0.1$ cm) channel walls, employing the
Arrhenius-type temperature dependent viscosity law (14). Its profile is skewed
due to the reduction of viscosity close to the upper heated channel wall which
is also the location of high chiral separation velocity $v^{\textrm{ch}}$. To
leading order in $\Delta T$, and averaging over the channel width $d$, we
obtain $\langle\delta v\rangle\sim\chi
R\frac{d\partial_{x}p}{12\eta}\gamma\Delta T$. Replacing the pressure gradient
with its Poiseuille flow counterpart, leads to

$\langle\delta v\rangle\sim\chi\frac{R}{d}U_{0}\gamma\Delta T.$ (22)

Employing the material parameters for BM-4 oil displayed in Table 1 and
setting $\Delta T=$10\text{\,}\mathrm{K}$$ we obtain $\delta v\sim
35\chi\;\mu\textrm{m/sec},$ which agrees well, in order of magnitude, with the
exact solution displayed in Fig. 3. The momentum equation displayed in (21)
was formulated by considering only the first term of the constitutive law (7)
since the second term, and for the material parameters employed in this
article, gives velocities that are one order of magnitude smaller than the
ones derived here.

Figure 3:  Transverse chiral component of liquid velocity $\delta v$ in
cm/sec, perpendicular to the $x$-$y$ plane formed by a Poiseuille flow and the
temperature gradient directions (cf. Fig.1) and obtained by solution of Eq.
(21) with boundary conditions $\delta v(0)=\delta v(d=0.1)=0$. $y$ is vertical
channel coordinate in cm and we employed the Arrhenius-type temperature
dependent viscosity law (14) for a BM-4 oil [19] giving rise to the skewness
of chiral velocity profiles. $\chi$ has been set equal to $1$.

_Screw torque in a non-racemic suspension._ \- A non-racemic mixture will
apply shear stresses on the channel walls that are perpendicular to the plane
of the paper. These forces arise from the chiral momentum flux density (7),
cf. [12]. Employing the geometry of the channel Poiseuille flow displayed in
Fig. 1 this stress is

$\hat{\mathbf{z}}:\quad\sigma_{zy}^{{\textrm{ch}}}=\chi
R\eta\partial^{2}_{y}u.$ (23)

In Fig. 4 we display the chiral stress $\sigma_{zy}^{{\textrm{ch}}}$ as a
function of channel width employing the exact form for the liquid velocity
profile (13). Since the normal vectors to the two channel walls have opposite
sign, the chiral suspension exerts on the walls two forces of opposite sign
directed into and out of the page. Hence, there is a screw torque exerted by
the chiral flow on the confining walls and is directed along the
$\hat{\mathbf{x}}$ direction of the flow. The average of the chiral stress
$\langle\sigma_{zy}^{{\textrm{ch}}}\rangle$ over the channel width to leading
order in $\Delta T$ becomes

$\langle\sigma_{zy}^{{\textrm{ch}}}\rangle=\chi
R\partial_{x}p(1+\frac{1}{2}\gamma\Delta T+O((\Delta T)^{2}).$ (24)

Expression (24) implies that a chiral stress exists even in the absence of
temperature gradients. This was also noted in [12]. Replacing the pressure
gradient by the base Poiseuille profile, as carried out in the foregoing
sections and employing the material values appearing in Table 1 for $\Delta
T=10K$, Eq.(24) gives
$\langle\sigma_{zy}^{{\textrm{ch}}}\rangle=1.73\chi\textrm{ dynes/cm}^{-2}.$
This estimate agrees, in order of magnitude, with those of same-size systems
known in the literature [24, *Ando2009].

Figure 4:  Distribution of the chiral stress $\sigma_{zy}^{{\textrm{ch}}}$ in
(23) in dynes/$\textrm{cm}^{2}$ imparted by the chiral suspension on the
liquid in a direction perpendicular to the $x$-$y$ plane formed by the
Poiseuille flow and the vertical temperature gradient (cf. Fig.1), vs. the
vertical coordinate $y$ of the channel in cm and we employed the Arrhenius-
type temperature dependent viscosity law (14) for a BM-4 oil [19]. Since the
normal vectors to the two channel walls have opposite signs, the chiral
suspension exerts on the walls two forces of opposite sign directed into and
out of the page. Hence, there is a screw torque exerted by the chiral flow on
the confining walls and is directed along the base flow direction. $\chi$ has
been set equal to $1$.

We note the existence of a related thermal effect for the propulsion of chiral
particles when the viscosity is temperature-dependent and thermal gradients
are generated at the interior by viscous heating [26]. The energy equation is
now replaced by
$\kappa\partial_{y}^{2}T+\eta(T)\left(\partial_{y}u\right)^{2}=0$, where
$\kappa$ is the thermal conductivity of the liquid. The external stimuli may
be supplied, for instance, by sliding one channel wall at constant speed $V$.
In this case the vorticity equation and the chiral current depend on the
Brinkman number $Br$, that is, the ratio of viscous heating to conduction
$Br=\frac{\gamma\eta V^{2}}{\kappa}$. The chiral separation velocity becomes
$v^{\textrm{ch}}\sim\frac{\chi}{8}\left(\frac{R}{d}\right)^{3}Br.$

Another related thermally-induced chiral particle propulsion effect can take
place in a Rayleigh-Bènard cell [27], driven by variations of the liquid
density with temperature. Here, the coefficient of thermal expansion
$\alpha_{T}$ is the logarithmic derivative of density with respect to
temperature, in the same sense that $\gamma$ in Eq. (17) is the logarithmic
derivative of viscosity. Diffusion of vorticity, perpendicular to the plane of
the cell, is now induced by $\rho\alpha_{T}\nabla T\times\mathbf{g}$, where
$\rho$ is the unperturbed mass density of the liquid and both the temperature
gradient and gravitational acceleration $\mathbf{g}$ lie on the plane of the
cell. The chiral separation velocity is
$v^{\textrm{ch}}\sim\chi\frac{R^{3}\rho g\alpha_{T}\Delta T}{\eta d}.$ This
effect is present when the Rayleigh number exceeds its critical value.

We thank the Department of Energy, Office of Basic Energy Sciences for support
under contract DE-FG02-08ER46539 (M.O.C and E.K.). The work of A.V.A. was
supported, in part, by the US National Science Foundation through the MRSEC
Grant No. DMR-1719797.

## References

  * Inaki  _et al._ [2016] M. Inaki, J. Liu, and K. Matsuno, Philosophical Transactions of the Royal Society B: Biological Sciences 371, 20150403 (2016).
  * Fan  _et al._ [2019] J. Fan, H. Zhang, T. Rahman, D. Stanton, and L. Wan, Communicative & Integrative Biology 12, 78 (2019).
  * Gao  _et al._ [2019] C. Gao, S. Kewalramani, D. Valencia, H. Li, J. McCourt, M. Olvera de la Cruz, and M. Bedzyk, Proceedings of the National Academy of Sciences 116, 22030 (2019).
  * Arnold  _et al._ [2006] M. Arnold, A. Green, J. Hulvat, S. Stupp, and M. Hersam, Nature nanotechnology 1, 60 (2006).
  * Shimizu  _et al._ [2005] T. Shimizu, M. Masuda, and H. Minamikawa, Chemical Reviews 105, 1401 (2005).
  * McCourt  _et al._ [2022] J. McCourt, S. Kewalramani, C. Gao, E. Roth, S. Weigand, M. Olvera de la Cruz, and M. Bedzyk, ACS Central Science 8, 1169 (2022).
  * Nagarsekar  _et al._ [2016] K. Nagarsekar, M. Ashtikar, F. Steiniger, J. Thamm, F. Schacher, and A. Fahr, Soft Matter 12, 3797 (2016).
  * Oda  _et al._ [1999] R. Oda, I. Huc, M. Schmutz, S. Candau, and F. MacKintosh, Nature 399, 566 (1999).
  * Zhang  _et al._ [2014] H. Zhang, W. Duan, M. Lu, X. Zhao, S. Shklyaev, L. Liu, T. Huang, and A. Sen, ACS Nano 8, 8537 (2014).
  * Schermer  _et al._ [2011] R. Schermer, C. Olson, J. Coleman, and F. Bucholtz, Optics Express 19, 10571 (2011).
  * Oppenheimer  _et al._ [2016] N. Oppenheimer, S. Navardi, and H. Stone, Physical Review Fluids 1, 014001 (2016).
  * Andreev  _et al._ [2010] A. Andreev, D. Son, and B. Spivak, Physical Review Letters 104, 198301 (2010).
  * Happel and Brenner [1965] J. Happel and H. Brenner,  _Low Reynolds number hydrodynamics with special applications to particulate media_ (Prentice-Hall Inc., Englewood Cliffs, N.J., 1965) pp. xiii+553.
  * Witten and Diamant [2020] T. Witten and H. Diamant, Reports on Progress in Physics 83, 116601 (2020).
  * Kirkinis  _et al._ [2012] E. Kirkinis, A. Andreev, and B. Spivak, Physical Review E 85, 016321 (2012).
  * Datt and Elfring [2019] C. Datt and G. Elfring, Physical Review Letters 123, 158006 (2019).
  * Shoele and Eastham [2018] K. Shoele and P. Eastham, Physical Review Fluids 3, 043101 (2018).
  * Landau and Lifshitz [1987] L. D. Landau and E. M. Lifshitz,  _Fluid Mechanics. Course of Theoretical Physics, Vol. 6_ (Pergamon Press Ltd., London-Paris, 1987) pp. xii+515.
  * Fogel’son and Likhachev [2001] R. Fogel’son and E. Likhachev, Technical Physics 46, 1056 (2001).
  * Potter and Graber [1972] M. Potter and E. Graber, The Physics of Fluids 15, 387 (1972).
  * Davis  _et al._ [1983] S. Davis, G. Kriegsmann, R. Laurence, and S. Rosenblat, The Physics of Fluids 26, 1177 (1983).
  * Wall and Wilson [1996] D. Wall and S. Wilson, Journal of Fluid Mechanics 323, 107 (1996).
  * Ehrhard [1993] P. Ehrhard, Journal of Fluid Mechanics 257, 463 (1993).
  * Kataoka and Troian [1999] D. Kataoka and S. Troian, Nature 402, 794 (1999).
  * Ando and Yamamoto [2009] J. Ando and K. Yamamoto, Circulation Journal 73, 1983 (2009).
  * Kirkinis and Andreev [2019] E. Kirkinis and A. Andreev, Journal of Fluid Mechanics 872, 308 (2019).
  * Chandrasekhar [1961] S. Chandrasekhar,  _Hydrodynamic and hydromagnetic stability_ (Oxford University Press, 1961).


1 Hamann et al. W6: Text Mining and Generation (TMG-2022) 2017

# IRT2: Inductive Linking and Ranking in Knowledge Graphs of Varying Scale

Felix Hamann111RheinMain University of Applied Sciences, Wiesbaden, Germany
felix.hamann, adrian.ulges<EMAIL_ADDRESS>Adrian Ulges11footnotemark:
1 Maurice Falk11footnotemark: 1

###### Abstract

We address the challenge of building domain-specific knowledge models for
industrial use cases, where labelled data and taxonomic information is
initially scarce. Our focus is on inductive link prediction models as a basis
for practical tools that support knowledge engineers with exploring text
collections and discovering and linking new (so-called open-world) entities to
the knowledge graph. We argue that – though neural approaches to text mining
have yielded impressive results in the past years – current benchmarks do not
reflect the typical challenges encountered in the industrial wild properly.
Therefore, our first contribution is an open benchmark coined IRT2 (inductive
reasoning with text) that (1) covers knowledge graphs of varying sizes
(including very small ones), (2) comes with incidental, low-quality text
mentions, and (3) includes not only triple completion but also ranking, which
is relevant for supporting experts with discovery tasks.

We investigate two neural models for inductive link prediction, one based on
end-to-end learning and one that learns from the knowledge graph and text data
in separate steps. These models compete with a strong bag-of-words baseline.
The results show a significant advance in performance for the neural
approaches as soon as the available graph data decreases for linking. For
ranking, the results are promising, and the neural approaches outperform the
sparse retriever by a wide margin.

###### keywords:

Text Mining Knowledge Graphs NLP.

##  1 Introduction

Knowledge graphs (KG) are known to be powerful tools in various applications
such as web search or personal assistants. They are key to addressing complex
information needs that require reasoning and cannot be answered with simple
text matching. Usually, knowledge graphs are assumed to be available at large
scale [hogan2021knowledge], examples include the Linked Open Data
Cloud222https://lod-cloud.net or KGs in biomedicine [hoehndorf15biomedical].
However, knowledge graphs may also be interesting in other industries where no
large-scale knowledge exists yet, be it in the technical service, where KGs
can help reason about the root causes of machine failures, in insurance, where
they can model the details of a contract, or in finance rating, where they
capture businesses’ supply chains. Explicit knowledge is scarce in these
domains. However, what is often commonly available in abundance is text data,
such as service tickets, insurance claims, or business reports.

The focus of this paper is to support experts with building knowledge graphs
by inspecting text. To do so, we assume that the expert adds triples to the
graph in an iterative process, in which he/she locates and identifies new
entities in the text collection and links them to the graph. We address two
key steps of this process, as illustrated in Figure 1:

Figure 1: Our two-step approach towards interactive knowledge acquisition:
First, the user discovers sentences containing relevant entities
(Discovery/Ranking, left). Second, given a mention of interest, triples for
this mention are suggested as if it was an entity (Linking, right).

  1. 1.

Discovery/Ranking: The expert explores the text collection based on a partial
fact of the KG (i.e. a relation-vertex tuple). He/she specifies an information
need (such as “find me actors”, or more specifically “find sentences likely to
contain mentions of entities which have the relation  profession with the
known entity  actor”). The system returns a ranked list of potentially
interesting sentences, giving rise to a ranking task.

  2. 2.

Linking: The expert studies the sentences. Once he/she has discovered an
interesting entity $x$ (such as  Frances McDormand), we would like to link $x$
to the graph by adding triples, not only via the  profession relation but also
via other relations (such as the birthplace, languages she speaks, or movies
she appeared in). To do so, the system collects textual evidence on the entity
in form of other mentions in the text collection. From those, it infers
triples that link $x$ to the knowledge graph.

We address both tasks using embedding-based semi-inductive [ali2021improving]
open-world [shi2017open] link prediction, which is targeted at predicting the
likelihood of triples $(h,r,t)$. In contrast to standard link prediction,
semi-inductive link prediction addresses new entities to be linked to the
known graph and the open-world scenario connotes that the new entities are
solely described via free text.

Since domain-specific data is often confidential, link prediction research
employs open data from graphs such as Freebase
[Bollacker:2008:FCC:1376616.1376746] or Wikidata [vrandevcic2014wikidata]. We
argue that the insight these benchmarks offer for industrial knowledge
acquisition is limited, and contribute a new benchmark called Inductive
Reasoning with Text V2 (IRT2) with the following benefits333The data is
publicly available under https://github.com/lavis-nlp/irt2: (1) To assess
models at different stages of graph construction, we benchmark on graphs of
varying size. (2) While text contexts in other datasets consist of concise
descriptions of entities, text in practice contains rather incidental mentions
of entities. We sample our text data accordingly. (3) To our knowledge, our
study is the first to not only address the linking task but also the ranking
task.

We evaluate three inductive link models on IRT2: One baseline based on keyword
matching, and two neural models that combine a transformer text encoder with a
link predictor. We show that the end-to-end approach and separate training
both work well, while the latter generally performs a little better (even for
small graphs). Also, only through evaluation on smaller graphs, the baseline
model is outperformed when linking, which favours the neural models in
scenarios where structured data becomes sparse.

##  2 Related Work

Link Prediction Models: Closed-world link prediction (or also knowledge graph
completion, KGC) has attracted interest in the research community over the
past years. Earlier approaches such as RESCAL [nickel2011three], TransE
[bordes2013translating], and DistMult [yang2014embedding] laid the foundations
for many following completion models such as ComplEx [trouillon2016complex],
ConvE [dettmers2018convolutional], RotatE [sun2019rotate], or KBGAT
[nathani2019learning]. We are, however, interested in open-world scenarios,
where, given a textual description of an entity, geometric reasoning is
applied through the alignment of dense graph- and text-representations. This
combines KGC—which aims at the prediction of missing links between existing
entities—with language modelling [mikolov2013distributed,
bojanowski2016enriching, devlin2018bert] which produces dense vector
representations for natural language text. Specifically, in this work, a semi-
inductive setting [ali2021improving] is studied, where out-of-kg entities are
to be linked to an existing graph using their textual description. One of the
first approaches to combining language- and graph-reasoning models is NTN
[socher2013reasoning], where entity description embeddings are trained to be
similar if their entities are connected in the KG. Later approaches build on
this idea: DKRL [xie2016representation], ConMask [shi2017open], MIA
[fu2020multiple], and KEPLER [wang2021kepler] introduce models that jointly
learn the connection between entity and graph representations. A different
approach decouples the KGC scorer and text encoder such that first the scorer
is trained independently and in a separate step a projection from text-based
entity descriptions to their associated graph-based embeddings is learned
[shah2019open, shah-etal-2020-relation, zhou2020weighted]. We study and
compare both approaches with the models described in Section 4. Highly related
to our work is the recently published approach by Daza et al
[daza2021inductive]. Here, a BERT model (BLP) is trained to directly score KGC
triples for plausibility using the textual representations instead of training
a separate entity embedding. However, we cannot rely on high quality text and
thus a separate entity embedding tuned with many different observations for
input text is more suitable.

KGC Benchmarks: Closed-world KGC models are usually evaluated on subsets of
publicly available knowledge graphs such as Freebase
[Bollacker:2008:FCC:1376616.1376746], DBPedia [auer2007dbpedia] or Wikidata
[vrandevcic2014wikidata]. Famous benchmarks include FB15K
[bordes2013translating], succeeded by FB15k237 [toutanova-chen-2015-observed]
due to test-leakage, WN18, succeeded by WN18RR [dettmers2018convolutional], or
CoDEx [safavi2020codex], among others. Open-world KGC combines KGs with
textual descriptions of its entities. Approaches include the FB20K benchmark
as part of DKRL [xie2016representation], DBPedia50k and DBPedia500k as part of
ConMask [shi2017open], and FB15k237-OWE as part of OWE [shah2019open]. More
recent work introduces Wikidata5M [wang2021kepler] (a large Wikidata subset)
and the FB15k237/WN18RR adaptions of [daza2021inductive]. Since data in
industrial use cases is not publicly available, we also sample our benchmark
data from open knowledge graphs (CoDEx) and text sources (Wikipedia). However,
all the above benchmarks tackle the open-world scenario with a single concise
description of graph entities. We are, however, interested in linking entities
associated with many, noisy text contexts. To our knowledge, the only
benchmark to attempt this is IRT [hamann2021open]. Here, a a set of $30$ text
contexts with incidental mentions is associated with each entity in the
knowledge graph. We extend this benchmark by offering more text, multiple
graph sizes, a greater variety of mentions, and evaluation protocols for both
ranking and linking.

##  3 Problem Description

We define a knowledge graph (KG) as a directed graph
$\mathcal{G}=(\mathcal{V},\mathcal{R},\mathcal{T},\mathcal{M},\mathcal{C})$
where $\mathcal{V}$ is the set of vertices $v$ (e.g.
$v={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\textsc{Fargo}}$)
and $\mathcal{R}$ a set of relation types (e.g.
$r={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\text{genre}}$).
Triples
$(h,r,t)\in\mathcal{T}\subseteq\mathcal{V}\times\mathcal{R}\times\mathcal{V}$
connect the vertices, e.g. ( Fargo,  genre,  Thriller Film). Additionally, the
set $\mathcal{M}$ contains textual mentions, each associated with a vertex via
a function $M:\mathcal{V}\mapsto\mathcal{P}(\mathcal{M})$. For example, the
entity v= Thriller Film has the mentions M(v) = {  “crime thriller”, …,
“gangster film” }. Note that mentions can be ambiguous (e.g.  “the film” as
mention of  Fargo). Furthermore, each of the mentions $m$ is assumed to occur
in textual context sentences $c$ from a corpus $\mathcal{C}$. In general,
these contexts do not express triples but solely contain incidental mentions
of entities. They are accessible through a function
$C:\mathcal{V}\times\mathcal{M}\mapsto\mathcal{P}(\mathcal{C})$. For example,
with
$v={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\textsc{Crime
Film}}$ and
$m={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\textit{``\text@underline{gangster
film}''}}$: $C(v,m)=\\{$ “Corman called it the most accurate, authentic
gangster film” $,\dots\\}$. Note that – via M and C – we can access an
entity’s mentions and contexts, and vice versa.

Open/Closed-World: While the text corpus $\mathcal{C}$ contains known (or
"closed-world") entities, a second corpus $\mathcal{Q}$ contains undiscovered
open-world mentions $\mathcal{M}^{o}$. Our goal is to discover these and link
them to the graph. We define a closed-world graph $\mathcal{G}^{c}$
(containing subsets of the respective sets of $\mathcal{G}$ and the text
corpus $\mathcal{C}$) and an open-world graph
$\mathcal{G}^{o}=(\mathcal{V}^{o},\mathcal{R},\mathcal{T}^{o},\mathcal{M}^{o},\mathcal{Q})$
with $\mathcal{V}^{o}\subseteq\mathcal{V}$,
$\mathcal{T}^{o}\subseteq\mathcal{T}$,
$\mathcal{M}^{o}=\mathcal{M}\setminus\mathcal{M}^{c}$: Namely a graph with a
set of undiscovered mentions not known in the closed-world graph
$\mathcal{G}^{c}$. This gives rise to the two tasks we address:

Ranking Task: The first task is to retrieve text contexts from $\mathcal{Q}$
which contain mentions of interest, i.e. unknown mentions of (possibly unseen)
entities which should complete a given entity-relation pair. For example, for
relation
$r={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\text{genre}}$
and tail
$t={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\textsc{crime
film}}$, we search for text contexts where any crime films are mentioned.

Linking Task: Now, given an open-world mention was just discovered and marked,
all text contexts of $\mathcal{Q}$ are bundled by mention. For each such
bundle of text, links $r\in\mathcal{R}$ to the graph vertices $V^{c}$ must be
found. For example:  “What genre is associated with all text contexts that
contain the mention fargo?”.

##  4 Models

We tackle the above tasks with three models: (1) JOINT, where a text encoder
and a KGC scorer are trained jointly, (2) OWE, where an encoder learns to
project text representations into a pre-trained graph embedding space, and (3)
BOW, a model that employs bag-of-words representations for text similarity.
All three models are evaluated on both the ranking and linking tasks. Note
that, although our notation is (for brevity) generally focused on finding
tails, we do also always include head-prediction for given relation-tail
pairs.

###  4.1 Neural Models

Both neural models combine a text encoder $\phi$ and a KGC module $\psi$ (see
Figure 2). As a text encoder, we use a pre-trained BERT [devlin2018bert], a
popular state of the art transformer [vaswani2017attention] encoder. Given a
text context $c$, we run it through the encoder and select the [CLS]-token
embedding as the context representation
$\phi(c)_{\text{CLS}}\in\mathbb{R}^{d^{\prime}}$ (following the observations
of [hamann2021open]). This representation is mapped using a learned affine
projection ($W,\mathbf{b}$) into a complex-valued space, obtaining a
representation $\mathbf{c}$. The representation is then used to estimate a
tripel’s plausibility. We use the ComplEx scorer [trouillon2016complex],
which, given complex-valued context, relation and tail embeddings
$\mathbf{c},\mathbf{r},\mathbf{v}\in\mathbb{C}^{d}$, calculates the
plausibility score as
$\psi(c,r,v)=\textbf{Re}(\mathbf{c}\odot\mathbf{r}\odot\mathbf{\overline{v}})$.
Note that—since the projection maps from real-valued to complex space—we
choose the matrix and bias vector dimensions accordingly
($W\in\mathbb{R}^{d^{\prime}\times 2d},\mathbf{b}\in\mathbb{R}^{2d}$).
Overall, our neural models can be written as:

$s(c,r,v)=\psi(\;\underbrace{W\cdot\phi(c)_{\text{CLS}}+\mathbf{b}}_{\mathbf{c}},\mathbf{r},\mathbf{v}\;)$
(1)

Multi-Context Extension As discussed above, instead of a single context $c$, a
set $\Sigma$ of multiple contexts may be available describing the same entity.
We would like our entity representations to take the whole set $\Sigma$ into
account. To do so, we use a simple early fusion that averages the text
representations before projecting them:

$\displaystyle\mathbf{c}$
$\displaystyle=W\cdot\Bigg{(}\frac{1}{|\Sigma|}\sum_{c\in\Sigma}{\phi}(c)_{\text{CLS}}\Bigg{)}+\mathbf{b}$
(2)

We then use the resulting representation $\mathbf{c}$ as a drop-in replacement
in Equation (1). Concerning training, we use two variants of this setup: In
the first variant (referred to as single), training happens on individual
contexts as in Equation (1), but we apply Equation (2) in inference. In the
second variant (multi), we apply Equation (2) both in training and inference.

Figure 2: Data flow in our model (top) and the two training strategies
(bottom): Strategy 1 (JOINT) applies an end-to-end training of both link
predictor and text encoder. Strategy 2 (OWE) first trains the link predictor
(2a), and then aligns the text-based entity representations $\mathbf{c}$ with
the graph-based ones $\mathbf{v}$ (2b). Learnable parameters are highlighted
in orange.

Training

As illustrated in Figure 2, we study two different training strategies to fit
the model parameters (text encoder $\phi$, projection $W,\mathbf{b}$, and
knowledge graph embeddings $\mathbf{r},\mathbf{v}$). Our first approach uses a
joint (end-to-end) training (JOINT): We iteratively sample a random entity $v$
from the knowledge graph. For $v$, we draw a random triple $(v,r,v^{\prime})$
and context $c$. The model’s scores $s(c,r,v^{\prime})$ are mapped to a
probability distribution using a softmax, and we apply a cross-entropy loss:

$\mathcal{L}_{\text{JOINT}}=-\sum_{v^{\prime}\in\mathcal{V}^{c}}\Bigg{(}\log\frac{\exp(s(c,r,v^{\prime}))}{\sum_{u^{\prime}\in\mathcal{V}^{c}}\exp(s(c,r,u^{\prime}))}\Bigg{)}\cdot\mathbf{1}_{(v,r,v^{\prime})\in\mathcal{T}}$
(3)

Note that – though the above notation samples triples $(v,r,v^{\prime})$ to
predict tails $v^{\prime}$ – we also sample triples $(v^{\prime},r,v)$ and
predict the heads $v^{\prime}$ accordingly.

Our second approach follows Shah et al’s OWE model [shah2019open] and applies
a separate training. Here, training happens in two steps: First, the link
prediction model $\psi$ is trained on the closed-world graph, obtaining a
graph-based embedding space. Second, the text encoder $\phi$ and projection
$(W,\mathbf{b})$ are trained to align entities’ text-based representations
with the (now fixed) graph-based ones. To do so, we reduce the geometric
distance between context representations
$\mathbf{c}=W\cdot\phi(c)_{CLS}+\mathbf{b}$ and their associated graph-based
representations, using a mean squared error loss:

$\mathcal{L}_{\text{OWE}}=\frac{1}{2d}\cdot\sum_{0<i\leq
d}\big{(}\textbf{Re}(\mathbf{c}_{i})-\textbf{Re}(\mathbf{v}_{i})\big{)}^{2}+\big{(}\textbf{Im}(\mathbf{c}_{i})-\textbf{Im}(\mathbf{v}_{i})\big{)}^{2}$
(4)

Inference: This section outlines how the neural models are used in our two
tasks. First, in ranking, a closed-world entity $v$ and a relation of interest
$r$ are given, and the task is to retrieve a ranked list of text contexts from
the query corpus $\mathcal{Q}$ that potentially contain mentions of relevant
open-world entities to be identified by the expert. To do so, we compute the
single-context scores $s(c,r,v)$ for all contexts $c\in Q$, $r\in\mathcal{R}$
and $v\in\mathcal{V}^{c}$. The scores are normalized per relation using the
softmax function as to better compare the independently scored contexts $c$
(the raw ComplEx scores are unbounded). The contexts in the query corpus are
then ranked by this normalized score.

Second, in linking, the expert is interested in a certain mention of interest
$m\in\mathcal{M}^{o}$ representing an open-world entity. His/her goal is to
link said entity to the graph. To do so, he/she selects a relation
$r\in\mathcal{R}$ and collects a set of contexts $\Sigma$ containing $m$ to
describe the open-world entity. Using the contexts, all entity candidates
$v\in\mathcal{V}^{c}$ are ranked by their scores $s(\Sigma,r,v)$. Note that in
principle we can repeat this procedure for each relation and also for
predicting heads instead of tails.

###  4.2 BOW: Bag-of-Words Retrieval

To evaluate the tasks at hand with a common industrial approach, we compare
the above neural models with a bag-of-words approach. Given one or several
mentions, this approach concatenates text contexts containing the mentions
into documents and then conducts a similarity matching between documents using
the well-known BM25[robertson1994some] keyword matching implemented by the
Elasticsearch search engine444https://www.elastic.co.

Ranking: Given an entity relation pair such as
$v={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\textsc{Thriller
Film}}$,
$r={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\text{genre}}$
(“Find text contexts which mention thriller films”), the baseline’s strategy
is to find open-world entities that are similar to known closed-world entities
$v^{\prime}$ for which $(v^{\prime},r,v)$ holds (i.e., known thriller films).
We sample a set of random representatives $v^{\prime}$ and some of their
associated text contexts. These contexts are concatenated into a document
which is then used as a query against an index containing all open-world
contexts $\mathcal{Q}$.

Linking: Given a mention $m\in\mathcal{M}^{o}$ and a relation $r$, we sample
contexts containing $m$ into a document $D(m)$, which is used to query against
an index containing documents $D(v)$ for all closed-world vertices
$v\in\mathcal{V}^{c}$. The top-n predicted results allow us to compile a set
of reference vertices which we use as blueprints for prediction. For example:
When searching for
$r={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\text{profession}}$
of text contexts containing  “s. a. corey” ( “Predict the profession of the
text contexts containing s. a. corey”), the result list may contain other
texts describing authors. The final prediction of vertices is compiled from
the retrieved vertices targets of relation $r$ and scored by the inverse of
the position of the document in the result list.

##  5 Benchmark Construction

Our benchmark picks up the work of [hamann2021open] and extends it: (1) We
study the behaviour of models with varying knowledge graph size, and
particularly for small graphs, (2) we test models for both the ranking and
linking task. We offer four variants of the benchmark: tiny, small, medium,
and large. The variants offer different ratios of structural information (i.e.
graph triples) and associated text. The datasets are based on CoDEx-M
[safavi2020codex] which is sampled from Wikidata [vrandevcic2014wikidata]. The
following steps are executed:

1\. Identifying Concept Vertices: We assume that in a real-world scenario
world-knowledge is more likely to be already modelled in the given KG.
Information to be discovered is usually much more volatile. Consider for
example a graph for a machine manufacturer. We know that things can catch fire
(the world knowledge) but we aim to find the specific parts that actually do.
A heuristic to identify such world knowledge is based on the assumption that
such entities are characterised by relations which exhibit a strong
disproportion between their heads and tails. The ratio between a relation
$r$’s domain size (its number of heads)
$\text{dom}(r):=|\\{h\>|\>\exists\>t:\>(h,r,t)\in\mathcal{T}\\}|$ and the
range size (its number of tails)
$\text{rg}(r):=|\\{t\>|\>\exists\>h:\>(h,r,t)\in\mathcal{T}\\}|$ is
$\text{min}\big{(}\text{dom}(r),\text{rg}(r)\big{)}/\text{max}\big{(}\text{dom}(r),\text{rg}(r)\big{)}$.
Per size variant of the dataset, we select a subset of the relations offered
by CoDEx, order by ratio and select a subset of those to determine concept
entities that will remain in the closed-world split. The appendix in Section 8
enumerates all relations and corresponding selections in greater detail.

2\. Sampling Mentions and Text Contexts: As our graph is based on Wikidata, we
can exploit its links to Wikipedia. For each vertex $v$, we identify the
associated Wikipedia page $p_{v}$ and all pages which link back to it
$\mathcal{N}(p_{v})$. Using the link text of the hyperlinks, we build the set
of mentions which are associated with the vertex $M(v)$. Lastly, to build the
contexts $C(v,m)$, all occurrences of the mentions in $\mathcal{N}(p_{v})$ are
identified and the surrounding sentence kept as context. This is a heuristic
to obtain incidental text contexts: The entity is mentioned but usually not
the subject of the sentence. As example for
$v={\color[rgb]{0.3984375,0.3984375,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.3984375,0.3984375,0.3984375}\textsc{Scotland}}$:
“Aberdeen is a city in northeast Scotland”.

3\. Open/Closed-World Split: The open/closed-world split, in this case, is not
done on vertex but mention level. We separate the mention set $\mathcal{M}$
into a partition $\mathcal{M}^{c}$ for closed-world- and $\mathcal{M}^{o}$ for
open-world mentions respectively. First, all mentions associated with concept
entities are set aside for the closed-world part. Then, for each remaining
vertex $v$, its associated mentions are distributed randomly between open- and
closed-world. We set an additional pruning parameter which limits the maximum
amount of mentions allowed for a closed-world vertex. With the information
about closed-world mentions, we identify all triples whose vertices are
associated with them and set them aside as the closed-world triple set
$\mathcal{T}^{c}\subseteq\mathcal{T}$. The so-called test triples
$(m\in\mathcal{M}^{o},r\in\mathcal{R},v\in\mathcal{V}^{c})$ (explained below
the table) are derived directly from the triple set $\mathcal{T}$ by
identifying in which triples the mention’s vertex occurs. We set aside a
subset of the open-world split for validation. The above steps are executed
for the four different dataset variants. These have different parameters set
for concept relations, total relations to keep and closed-world mention
pruning (see the appendix for more details).

|  |  | Tiny | Small | Medium | Large
---|---|---|---|---|---|---
Relations $|\mathcal{R}|$ |  |  | 5 | 12 | 45 | 45
Entities $|\mathcal{E}|$ |  |  | 1,174 | 2,887 | 3,592 | 9,952
Training Triples $|\mathcal{T}^{c}|$ |  |  | 2,928 | 7,527 | 26,335 | 102,289
Training Text $|\mathcal{C}|$ |  |  | 9,100,422 |  15,117,184 |  17,398,943 |  18,654,485
Test Text $|\mathcal{Q}|$ |  |  | 6,058,557 | 2,390,017 | 1,905,151 | 864,598
Ranking Queries |  |  | 695 | 1,225 | 4,972 | 7,336
Linking Queries |  |  | 47,857 | 52,654 | 97,921 | 48,801
Test Triples |  |  | 102,866 | 102,616 | 174,312 | 90,780

The last four rows of the table describe the scale of the challenges. A
ranking query is a tuple $(v\in\mathcal{V}^{c},r\in\mathcal{R})$ (e.g. (
writer,  profession) – “Find texts with writers in them”) and an associated
ground truth set of open-world mentions $\\{m_{1},\dots\\}\in M^{o}$ to be
discovered (e.g. {  “Tolkiens”,  “the author”,  “Corey”, …}). A linking query
is a tuple $(m\in M^{o},r\in\mathcal{R})$ (e.g. ( “Tolkiens”,  profession) –
“What professions does the entity have, given texts in which  “Tolkiens”
occurs?”) and a set of associated closed-world vertices $v\in\mathcal{V}^{c}$
(e.g. {  author,  linguist, …}). One can see that both ranking and linking are
different views of the same data: unique $(m\in
M^{o},r\in\mathcal{R},v\in\mathcal{V}^{c})$ triples. These triples are
presented as test triples in the table above and allow to gauge the ratio of
the queries and their associated ground truth. For example, in the most
extreme case, ranking on the tiny variant, has $102866/695\approx 148$
mentions to discover per query on average.

##  6 Experiments

We run a series of experiments to build a solid understanding of the
challenge’s difficulty. We employ all three formerly described models (JOINT,
OWE, BOW) for both tasks (ranking, linking) on all four splits (tiny, small,
medium, and large). We measure hits@k and the mean reciprocal rank (MRR) for
each test triple (i.e. micro-averaged). Following common practice, we apply
target filtering [shah2019open] to the scored predictions (mentions for
ranking and vertices for ranking). Target filtering is a method where, for a
given ranking, only a single true positive of interest is inspected, while all
other true positives are removed from the list. This helps to not artificially
worsen the result for rankings with many true positives. We offer the
datasets, dataset creation code, and evaluation protocol
online555https://github.com/lavis-nlp/irt2. The model implementations and all
presented trained models are also supplied
separately666https://github.com/lavis-nlp/irt2m. We evaluate our models on a
subset of the test text contexts $\mathcal{Q}$ (400k sentences drawn randomly
for ranking; 100 sentences per mention for linking).

###  6.1 Hyperparameters

We determine the final hyperparameters using a combination of random-, and
grid searches over the parameter space. A single closed-world ComplEx model is
trained per dataset split and commonly used to train the OWE model using
Adagrad [duchi2011adaptive]. The hyperparameters studied comprise learning
rate, L2 regularization weight, and embedding space dimensions. We employ
PyKEEN[ali2020pykeen] for training and build our own open-world extension. We
implement and train the JOINT and OWE models using PyTorch [paszke2019pytorch]
and PyTorch Lightning[falcon2019lightning]. Models are optimized using
Adam[kingma2014adam]. For each model architecture and dataset split, we run a
random parameter sweep to fix a subset of parameters and subject the remaining
to a grid search. We study the effect of regularization of the entity and
relation embeddings, weight-decay and learning rate of the optimizer, how many
layers of the encoder are frozen during training, how many text contexts are
sampled per vertex per training, and whether the mention is masked during
training. Additionally, we study how many contexts to provide per optimizer
step for multi-context models. Generally, all models share a similar set of
values for learning-rate, weight-decay and regularization weight. The other
parameters are dependent on the split. We found to freeze parts of the encoder
works well against overfitting. Contrary to the observations of
[hamann2021open], masking did not significantly increase model performance.
The final hyperparameters are provided in the appendix (Section 8) and with
the models online.

###  6.2 Linking

First, we evaluate the linking task as the designated challenge for the
presented models. Table 1 details our findings. Generally, a few trends can be
observed: (1) The neural models significantly outperform the baseline approach
on the three smaller variants but yield inferior results on the large dataset.
This supports our claim that a benchmark should offer insights into varying
degrees of structural data scarcity. Even with an abundance of data (usually
favouring neural approaches), the BOW baseline can beat the neural model’s
performance on large but falls short for the smaller variants. (2) Overall,
the OWE approach outperforms JOINT albeit not by a great margin for tiny,
small, and large. The biggest performance gap can be observed on medium, where
the OWE model profits the most from the decoupled training. We argue that this
hints towards the need for better sampling/overfitting strategies during
training for JOINT as it seems to struggle with the lower variety of mentions
seen during training. (3) Although the multi-context models generally perform
better than the single-context counterparts, the difference in performance is
not as pronounced as reported by [hamann2021open]. Contrary to their findings
we also did not observe much performance improvement through masking the
mention. These observations indicate that both models struggle to incorporate
much of the textual clues and instead rely heavily on the structural
information present in the vertex- and relation embeddings. Overall, the
metrics show that the models can link an entity described by text contexts to
a given graph with relatively high precision.

|  | HITS@10 | MRR
---|---|---|---
|  | Tiny | Small | Med. | Large | Tiny | Small | Med. | Large
BOW |  | 53.82 | 55.18 | 46.43 | 71.38 | 33.63 | 34.62 | 29.81 | 50.61
JOINT | single | 72.06 | 70.20 | 47.14 | 65.75 | 50.61 | 45.95 | 33.72 | 48.29
JOINT | multi | 73.56 | 74.27 | 53.77 | 65.12 | 51.28 | 52.39 | 37.50 | 45.26
OWE | single | 74.09 | 74.33 | 61.98 | 64.27 | 50.25 | 50.57 | 40.60 | 42.69
OWE | multi | 75.39 | 71.49 | 64.41 | 66.36 | 53.06 | 47.17 | 43.25 | 45.51
Table 1: Model performance on the linking task. The bag-of-words model
outperforms the neural approaches on the large dataset. The neural models both
outperform the baseline on the smaller variants by a large margin. Among the
neural models, OWE generally yields better performance on all splits.

###  6.3 Ranking

Secondly, for ranking, we study whether the models which were originally
trained to do link-prediction can be employed for discovery, too. We report
the hits@100 performance in Table 2. (1) The results show that the scoring
mechanism of the neural models outperforms the BM25 search results. At first
glance, this is a surprising insight, but can be explained by the nature of
text samples. As the mention of interest is usually not the subject of the
text context, the query does not directly describe its properties. This leads
the ranker astray. For example: searching for other comedians (i.e. ?,
profession,  comedian) the model uses text samples sampled from the Wikipedia
page about New York among others. This leads to other text samples to be
retrieved which also talk about things in the context of New York but seldom
other comedians. This contextualization of queries (by queried relation) is
better considered with the neural models. (2) Overall ranking quality is not
high enough for practical application.

|  | HITS@100
---|---|---
|  | Tiny | Small | Med. | Large
BOW |  | 2.86 | 4.29 | 6.42 | 14.83
JOINT | single | 7.91 | 6.78 | 6.37 | 19.47
JOINT | multi | 13.28 | 16.17 | 14.38 | 30.68
OWE | single | 6.30 | 8.19 | 6.88 | 10.81
OWE | multi | 9.98 | 13.00 | 6.36 | 31.40
Table 2: Results for the ranking task. The neural models all outperform the
baseline and, contrary to the linking task, the multi-context JOINT model is
generally the best performing approach.

We suspect two main factors which hinder effective ranking: First, the scores
are calculated independently from each other. This makes an intra-sample
comparison of scores difficult. Secondly, the ranking task requires scores
first and foremost produced by head prediction. These scores usually are of
lower quality because the models are much better at predicting a few biased
tails (e.g. professions) than hundreds or thousands of volatile heads (e.g.
authors). This is worsened by the formerly described observation that text
contexts have a comparatively small influence on the score. To obtain better
performance, new approaches must be devised which incorporate the graph
context on the one hand and neural semantic retrieval [mitra2017neural,
lin2021pretrained] on the other.

##  7 Conclusion

We present IRT2, a benchmark to study a knowledge acquisition pipeline to
extend a knowledge graph (KG) from noisy text. We offer knowledge graphs of
different sizes and text associated with its vertices. The benchmark comprises
two tasks, ranking and linking, where first, entities need to be discovered
from unlabelled text, and in the second stage, linked to the KG. Alongside, we
study three approaches for the tasks at hand: two neural open-world triple
scorers and a bag-of-words model. We found the models to perform well for the
linking task but less so for ranking. For linking, the BOW baseline
outperforms the neural models on the large variant. For the remaining, smaller
variants, the OWE approach outperforms all other approaches even if graph data
is scarce. For ranking, the JOINT model produces the best results. Although
the models show a promising direction to be used for ranking, they do not
perform at the level for practical application yet. We recommend studying how
semantic ranking techniques can profit from the given links between graph- and
text data. We invite the community to use the benchmark to (1) devise ways to
better incorporate the noisy but abundant text data to make better
predictions, (2) find ways to contextualize semantic retrievers with the
specific graph information, and (3) derive insights for application in an
industrial context with similar constraints and conditions. The dataset and
evaluation scripts can be found here: https://github.com/lavis-nlp/irt2.

Acknowledgement: This work was supported by the BMBF program FH-Kooperativ,
project SCENT (13FH003KX0).

##  8 Appendix

This appendix details configuration options used for benchmark construction
and model training in detail to allow for the reproduction of the reported
results.

###  8.1 Benchmark

Split | Tiny | Small | Medium | Large
---|---|---|---|---
Concept Relations | 4 | 8 | 27 | 27
Total Relations | 5 | 12 | 45 | 45
Closed World Threshold | 400 | 800 | 800 | -
Target Mention Split | 70% | 70% | 70% | 70%
Target Validation Split | 10% | 20% | 20% | 80%
Mention Threshold | 5 | 5 | 5 | 5
Concept Vertices | 471 | 962 | 2,200 | 2,200
Training Vertices | 1,174 | 2,887 | 3,592 | 9,952
Training Mentions | 4,945 | 9,231 | 14,417 | 22,866
Training Triples | 2,928 | 7,527 | 26,335 | 102,289
Training Text Contexts | 9,100,422 | 15,117,184 | 17,398,943 | 18,654,485
Validation Vertices | 1,741 | 3,375 | 3,279 | 2,644
Validation Mentions | 1,894 | 3,870 | 3,649 | 2,940
Validation Vertex Triples | 13,639 | 24,764 | 43,532 | 35,209
Validation Task Triples | 11,588 | 25,722 | 43,716 | 38,582
Validation Text Contexts | 850,351 | 597,050 | 505,979 | 383,351
Test Vertices | 10,909 | 10,527 | 10,221 | 5,607
Test Mentions | 17,055 | 15,481 | 14,600 | 6,860
Test Vertex Triples | 70,094 | 71,360 | 124,050 | 71,951
Test Task Triples | 102,866 | 102,616 | 174,312 | 90,780
Test Text Contexts | 6,058,577 | 2,390,017 | 11,705,151 | 864,598

The final benchmark proportions rely heavily on the selected relations and
pruning options for construction. The following table enumerates both the
configuration which was used for sampling and the resulting key figures for
the training, validation and test splits. Relations are picked manually per
split, are ordered by their ratio and a subset is selected as concept
relations. The Closed World Threshold determines how many mentions per
relation are retained at most in the closed world split. The Mention Threshold
says to keep only mentions with at least that many associated text contexts.
Validation- and Test Vertex Triples are the triple sets (vertex, relation,
vertex) from which the final Task Triples are derived (mention, relation,
vertex). They are pruned by filtering out true open-world vertices (the
necessary zero-shot entity linking is out of scope for the presented
tasks)—which explains the decrease that can be observed for the tiny variant.

The above figures detail the shift in proportions between the dataset
variants. The test mention counts decrease proportionally to the increase in
training. Concept mention count increases and is equal for M and L. The
proportion of concept mentions decreases with increasing dataset size.
Correspondingly, validation text context counts stay constant while an
increase in training data decreases available test data.

The bar plots above report the absolute counts of triples (called Vertex
Triples in the former table) and associated text. The resulting triple split
behaves correspondingly to the mention split with a proportional behaviour
between training/test data. The overall triple count more than doubles between
T and L. Associated text data increases with respect to the increase in
closed-world mentions.

The following table shows all relations present in CoDEx ordered by ratio as
described in 5. On the right-hand side, it is detailed which relations are
present in which dataset variant. The respective right side shows whether it
is selected to remain in the dataset. The left side marks whether the relation
was selected to pick concept entities.

ID Relation Ratio Heads Tails Triples T S M L P1412 languages spoken (…)
6.32e-3 9,816 62 12,584 X X X X X X X X P1303 instrument 1.05e-2 3,622 38
6,076 X X X X P140 religion 1.59e-2 2,520 40 2,651 X X X X X X P27 country of
citizenship 1.69e-2 13,036 220 16,828 X X X X X X X P30 continent 1.98e-2 353
7 391 X X X X X X X X P509 cause of death 2.02e-2 3,071 62 3,210 X X X X P172
ethnic group 2.48e-2 2,013 50 2,293 X X X X P2348 time period 2.63e-2 152 4
152 X X X X P102 member of political party 2.76e-2 2,175 60 2,668 X X X X P106
occupation 2.85e-2 13,145 375 71,596 X X X X X X X X P495 country of origin
3.85e-2 1,299 50 2,049 X X X X X X P136 genre 4.32e-2 5,303 229 11,761 X X X X
P641 sport 4.43e-2 384 17 418 X X X X P19 place of birth 6.11e-2 7,185 439
7,214 X X X X X X X P69 educated at 6.47e-2 6,502 421 9,752 X X X X P463
member of 6.77e-2 3,705 251 11,490 X X X X P264 record label 6.84e-2 2,002 137
3,456 X X X X P20 place of death 6.96e-2 5,417 377 5,442 X X X X P1050 medical
condition 7.59e-2 395 30 408 X X X X P101 field of work 8.13e-2 1,967 160
2,421 X X X X P2283 uses 8.33e-2 12 1 12 X X X X P135 movement 9.20e-2 413 38
446 X X X X P119 place of burial 9.67e-2 1,944 188 1,972 X X X X X X P108
employer 1.27e-1 3,016 382 4,795 X X X X X P37 official language 1.83e-1 306
56 403 X X X X P840 narrative location 1.97e-1 986 194 1,506 X X X X P17
country 2.23e-1 641 143 1,323 X X X X P50 author 2.35e-1 4 17 17 P452 industry
2.50e-1 16 4 17 X X X P551 residence 2.52e-1 1,426 359 1,934 X X P749 parent
organization 3.15e-1 73 23 82 X X P407 language of work or name 3.24e-1 34 11
43 X X P361 part of 3.48e-1 138 48 171 X X P57 director 3.71e-1 542 201 561 X
X P159 headquarters location 4.20e-1 157 66 169 X X X P161 cast member 4.80e-1
1,227 2,557 9,249 X X P1056 product (…) produced 5.00e-1 2 1 2 P740 location
of formation 5.19e-1 133 69 133 X X P131 located in (…) 7.61e-1 163 124 212 X
X P737 influenced by 8.71e-1 514 590 1,508 X X P138 named after 8.91e-1 49 55
57 X X P112 founded by 9.48e-1 55 58 67 X X P40 child 9.54e-1 309 324 391 X X
X P451 unmarried partner 9.62e-1 328 341 393 X X P530 diplomatic relation
9.68e-1 214 221 6,225 X X P3373 sibling 9.95e-1 394 396 785 X X P26 spouse
1.00e+0 804 804 866 X X P3095 practiced by 1.00e+0 2 2 2 P54 member of sports
team 1.00e+0 2 2 2 P113 airline hub 1.00e+0 1 1 1 P780 symptoms 1.00e+0 1 1 1

###  8.2 Hyperparameters

All models are trained on Nvidia GTX 2080TI or RTX A6000 graphics cards using
PyTorch version 1.11. KGC models use the 1.8 implementation of PyKEEN and the
text encoder uses Huggingface’s transformer implementation 4.19. The amount of
text samples per epoch varies between different model configurations and is a
combination of “max contexts” (the total amount of text contexts associated
with an entity during training) “max contexts per sample” (the number of text
contexts used by a model per training step). Model training is stopped if the
performance did not increase on the target metric (i.e. hits@10) on the
validation split by more than 0.001 for a few epochs. In the following, the
hyperparameters for the models presented in 6 are enumerated.

| Ranking | Linking
---|---|---
| Tiny | Small | Medium | Large | Tiny | Small | Medium | Large
Embedding Dims. | 800 | 500 | 800 | 200 | 200 | 800 | 800 | 200
Unfrozen Layer | 5 | 0 | 11 | 0 | 5 | 11 | 11 | 0
Regularizer Weight | 7.16e-1 | 7.73e-1 | 7.12e-2 | 5.86e-3 | 6.56e-1 | 9.02e-3 | 7.12e-2 | 5.86e-3
Contexts per Sample | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
Maximum Contexts | 10 | 100 | 10 | 10 | 10 | 10 | 10 | 10
Masked | false | false | false | false | true | true | false | false
Batch Size | 4 | 8 | 40 | 20 | 4 | 8 | 40 | 20
Learning Rate | 3.94e-6 | 4.93e-6 | 4.39e-6 | 8.99e-6 | 8.47e-6 | 6.67e-6 | 4.39e-6 | 8.99e-6
Weight Decay | 3.48e-2 | 1.19e-2 | 6.64e-4 | 1.49e-4 | 1.22e-4 | 2.16e-4 | 6.64e-4 | 1.49e-4
Seed | 5629275 | 4828059 | 2773483 | 4076657 | 8697782 | 9387603 | 2773483 | 4076657
Table 3: JOINT (single context) - best model selected after random search. | Ranking | Linking
---|---|---
| Tiny | Small | Medium | Large | Tiny | Small | Medium | Large
Embedding Dims. | 800 | 800 | 800 | 800 | 800 | 800 | 800 | 800
Unfrozen Layer | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
Regularizer Weight | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01
Contexts per Sample | 40 | 40 | 40 | 10 | 40 | 40 | 40 | 10
Maximum Contexts | 100 | 100 | 1000 | 1000 | 100 | 100 | 1000 | 1000
Masked | false | false | false | false | false | false | false | false
Batch Size | 1 | 1 | 1 | 4 | 1 | 1 | 1 | 4
Subbatch Size | 5 | 5 | 10 | 10 | 5 | 5 | 10 | 10
Learning Rate | 5.00e-6 | 5.00e-6 | 5.00e-6 | 5.00e-6 | 5.00e-6 | 5.00e-6 | 5.00e-6 | 5.00e-6
Weight Decay | 0.0001 | 0.0001 | 0.0001 | 0.0001 | 0.0001 | 0.0001 | 0.0001 | 0.0001
Seed | 4733486 | 5969007 | 7460979 | 6929361 | 4733486 | 5969007 | 7460979 | 6929361
Table 4: JOINT (multi-context) - best model selected after grid search. | Ranking | Linking
---|---|---
| Tiny | Small | Medium | Large | Tiny | Small | Medium | Large
Contexts per Sample | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
Maximum Contexts | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
Masked | false | false | false | false | false | false | false | false
Batch Size | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10
Subbatch Size | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10
Learning Rate | 5.00e-5 | 5.00e-5 | 5.00e-5 | 5.00e-5 | 5.00e-5 | 5.00e-5 | 5.00e-5 | 5.00e-5
Seed | 1705119 | 2298300 | 3899079 | 8847385 | 1705119 | 2298300 | 3899079 | 8847385
Table 5: OWE (single context) - best model selected after grid search. | Ranking | Linking
---|---|---
| Tiny | Small | Medium | Large | Tiny | Small | Medium | Large
Contexts per Sample | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10
Maximum Contexts | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100
Masked | false | false | false | false | true | false | true | false
Batch Size | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
Subbatch Size | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10
Learning Rate | 1.00e-6 | 1.00e-6 | 1.00e-6 | 1.00e-6 | 1.00e-6 | 1.00e-6 | 1.00e-6 | 1.00e-6
Seed | 3443300 | 3250085 | 4051229 | 6248903 | 9294686 | 9773709 | 9790717 | 6248903
Table 6: OWE (multi-context) - best model selected after grid search. | ComplEx
---|---
| Tiny | Small | Medium | Large
Embedding Dims. | 300 | 300 | 300 | 500
Learning Rate | 1 | 0.6 | 0.4 | 0.6
Regularizer Weight | 0.3 | 0.1 | 0.1 | 0.1
Batch Size | 64 | 64 | 64 | 64
Seed | 1174584270 | 2593575093 | 2203942071 | 2649116927
Table 7: OWE reference embeddings - best model selected after grid search.