# Mutually-paced Knowledge Distillation for Cross-lingual Temporal Knowledge
Graph Reasoning

Ruijie Wang, Zheng Li2†, Jingfeng Yang2, Tianyu Cao2, Chao Zhang3,
Bing Yin2 and Tarek Abdelzaher1† 1Department of Computer Science, University
of Illinois Urbana-Champaign, 2Amazon.com Inc3School of Computational Science
and Engineering, Georgia Institute of Technology ruijiew2<EMAIL_ADDRESS><EMAIL_ADDRESS>amzzhe, jingfe, caoty<EMAIL_ADDRESS>
(2023)

###### Abstract.

This paper investigates cross-lingual temporal knowledge graph reasoning
problem, which aims to facilitate reasoning on Temporal Knowledge Graphs
(TKGs) in low-resource languages by transfering knowledge from TKGs in high-
resource ones. The cross-lingual distillation ability across TKGs becomes
increasingly crucial, in light of the unsatisfying performance of existing
reasoning methods on those severely incomplete TKGs, especially in low-
resource languages. However, it poses tremendous challenges in two aspects.
First, the cross-lingual alignments, which serve as bridges for knowledge
transfer, are usually too scarce to transfer sufficient knowledge between two
TKGs. Second, temporal knowledge discrepancy of the aligned entities,
especially when alignments are unreliable, can mislead the knowledge
distillation process. We correspondingly propose a mutually-paced knowledge
distillation model MP-KD, where a teacher network trained on a source TKG can
guide the training of a student network on target TKGs with an alignment
module. Concretely, to deal with the scarcity issue, MP-KD generates pseudo
alignments between TKGs based on the temporal information extracted by our
representation module. To maximize the efficacy of knowledge transfer and
control the noise caused by the temporal knowledge discrepancy, we enhance MP-
KD with a temporal cross-lingual attention mechanism to dynamically estimate
the alignment strength. The two procedures are mutually paced along with model
training. Extensive experiments on twelve cross-lingual TKG transfer tasks in
the EventKG benchmark demonstrate the effectiveness of the proposed MP-KD
method.

Temporal Knowledge Graph, Cross-lingual Transfer, Knowledge Distillation,
Self-training

†Corresponding authors in UIUC and Amazon.com Inc

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
ACM Web Conference 2023; April 30-May 4, 2023; Austin, TX, USA††booktitle:
Proceedings of the ACM Web Conference 2023 (WWW ’23), April 30-May 4, 2023,
Austin, TX, USA††price: 15.00††doi: 10.1145/3543507.3583407††isbn:
978-1-4503-9416-1/23/04††ccs: Computing methodologies Temporal reasoning

##  1\. introduction

Temporal Knowledge Graphs (TKGs) (YAGO, ; ICEWS18, ; WIKI, ; acekg, )
characterize temporally evolving events, where each event, represented as
(subject, relation, object), is associated with temporal information (time),
e.g., (Macron, reelected, French president, 2022). TKGs has facilitated
various knowledge-intensive Web applications with timeliness, such as question
answering (KBQA, ), product recommendation (RippleNet, ; TKG4Rec, ; TKG4Rec2,
; RETE, ), and social event forecasting (KG4Social, ; DyDiff-VAE, ; andgan, ;
belief, ; misinfo, ; polarization, ).

As new events are continually emerging, modern TKGs are still far from being
complete. Conventionally, the TKG construction process relies primarily on
information extraction from unstructured corpus (WIKI, ; YAGO, ; EventKG, ),
which necessitates extensive manual annotations to keep up with changing
events. For instance, the recent transition from Trump to Biden as the
President of the United States has not been reflected in many TKGs,
highlighting the need for timely updates. This spurs research on temporal
knowledge graph reasoning to automate evolving events prediction over time
(TA-DistMult, ; Know-Evolve, ; Renet, ; RE-GCN, ). Unfortunately, the problem
of TKG incompleteness is particularly pronounced in low-resource languages,
where it is unable to collect enough corpus and annotations to support robust
TKG construction. This results in suboptimal reasoning performance and
distinctly unsatisfying accuracy in predicting recent and future events.

Figure 1. An illustrative example of cross-lingual reasoning on TKGs. 1) We
aim to transfer knowledge from English TKG to Japanese TKG, where the English
version provides more complete information; 2) Cross-lingual alignments only
cover a small ratio of entities, e.g., Apple Inc; 3) Cross-lingual alignments
can be noisy and misleading, e.g., A city called Ventura is linked to new
macOS Ventura at $t_{2}$, introducing noise for reasoning in Japanese.

Inspired by the incompleteness issue facing low-resource languages in
constructing TKGs, we introduce a novel task named Cross-Lingual Temporal
Knowledge Graph Reasoning (as shown in Figure 1). This task aims to alleviate
the reliance on supervision for TKGs in low-resource languages (referred to as
the target language) by transferring temporal knowledge from high-resource
languages (referred to as the source language) 111In this paper, for the sake
of brevity, we interchangeably use the terms high-resource/low-resource and
source/target.. In contrast, all the existing efforts are either limited to
reasoning in monolingual TKGs (usually high-resource languages, e.g., English)
(TA-DistMult, ; Know-Evolve, ; Renet, ; RE-GCN, ), or multilingual static KGs
(KEnS, ; AlignKGC, ; SS-AGA, ). To the best of our knowledge, cross-lingual
TKG reasoning that transfers temporal knowledge between TKGs has not been
investigated.

The fulfillment of this task poses tremendous challenges in two aspects: 1)
Scarcity of cross-lingual alignment: as the informative bridge of two separate
TKGs, cross-lingual alignment is imperative for cross-lingual knowledge
transfer (AlignKGC, ; KEnS, ; SS-AGA, ). However, obtaining alignments between
languages is a time-consuming and resource-intensive process that heavily
relies on human annotations. The transfer of knowledge through a limited
number of alignments is often insufficient to fully enhance the TKG in the
target language. 2) Temporal knowledge discrepancy: the information associated
with two aligned entities is not necessarily identical, especially with
regards to temporal patterns. Utilizing a rough approach to equate the aligned
entities at all times can result in the transfer of misleading knowledge and
negatively impact performance. This becomes more pronounced when the
alignments are noisy and unreliable. For example, at the time step $t_{2}$, a
new event about operating system “Ventura” from Apple company occurs in the
source English TKG, and meanwhile there is a noisy aligned entity “Ventura
city” in the target Japanese TKG. Directly pulling those two entities at this
point, can inevitably introduce noise and fail to predict a set of related
events in the target TKG. Therefore, it is crucial to dynamically regulate the
alignment strength of each local graph structure over time in order to
maximize the effectiveness of cross-lingual knowledge distillation.

In this paper, we propose a novel Mutually-paced Knowledge Distillation (MP-
KD) framework, where a teacher network learns more enriched temporal knowledge
and reasoning skills from the source TKG to facilitate the learning of a
student network in the low-data target one. The knowledge transfer is enabled
via an alignment module, which estimates entity correspondence across
languages based on temporal patterns. Firstly, to alleviate the limited
language alignments (Challenge #1), such a knowledge distillation process is
mutually paced over time. This means, on one hand, we encourage the mutually
interactive learning between the teacher and student. Concretely, the
alignment module between the teacher and the student learns to generate pseudo
alignment between TKGs to maximally expand the upper bound of knowledge
transfer. And subsequently, it empowers the student to encode more informative
knowledge in target TKG, which can in turn boost the alignment module to
explore more reasonable alignments as the bridge across TKGs. One the other
hand, inspired by self-paced learning (spl-1, ; spl-2, ), we make the
generations as a progressively easy-to-hard process over time. We start from
generating reliable pseudo data with high confidence. As time goes by, we then
gradually increase the generation amount by relieving the restriction over
time. Secondly, to inhibit the temporal knowledge mismatch (Challenge #2), the
attention module can estimate the graph alignment strength distribution over
time. This is achieved by a temporal cross-lingual attention in terms of the
local graph structure and temporal-evolving patterns of aligned entities. As
such, it can dynamically control the negative effect and suppress noise
propagation from the source TKG. Moreover, we provide a theoretical
convergence guarantee for the training objective on both initial ground-truth
data and pseudo data. To evaluate MP-KD, we conduct extensive experiments of
12 cross-lingual TKG transfer tasks in multilingual EventKG dataset (EventKG,
). Our empirical results show that the MP-KD method outperforms state-of-the-
art baselines in both with and without alignment noise settings, where only
$20\%$ of temporal events in the target KG and $10\%$ of cross-lingual
alignments are preserved.

To sum up, our contributions are three-fold:

  * •

Problem formulation: We propose the cross-lingual temporal knowledge graph
reasoning task, to boost the temporal reasoning performance in target TKG by
transferring knowledge from source TKG;

  * •

Novel framework: We propose a novel MP-KD framework, which enables the
mutually-paced learning between the teacher and student networks, to promote
both pseudo alignments and knowledge transfer reliability. Besides, MP-KD
involves a dynamic alignment estimation across TKGs that inhibits the
influence of temporal knowledge discrepancy.

  * •

Extensive evaluations: Empirically, extensive experiments on 12 cross-lingual
TKG transfer tasks in multilingual EventKG benchmark dataset demonstrate the
effectiveness of MP-KD.

##  2\. Preliminaries and Notations

In this section, we formally define the cross-lingual temporal knowledge graph
reasoning task, and summarize the notations in Table 1. A temporal knowledge
graph can be defined as follows:

Table 1. Symbols and Notations.

Symbol | Definition
---|---
$(e,r,e^{\prime},t)$ | A quadruple in TKG.
$\mathcal{G}_{s}$, $\mathcal{G}_{t}$ | Source TKG and Target TKG.
$e_{s}$, $e_{t}$ | Entities in the source and target TKGs.
$\Gamma_{s\leftrightarrow t}$ | Alignments between the source and target TKGs.
$\tilde{\mathcal{G}}_{t}$, $\tilde{\Gamma}_{s\leftrightarrow t}$ | Incomplete target TKG and alignments.
$\tilde{\mathcal{G}}_{t}^{ST}$, $\tilde{\Gamma}_{s\leftrightarrow t}^{ST}$ | Pseudo target TKG and pseudo alignment.
$f(\cdot;\Theta_{s})$ | Teacher network on the source TKG.
$f(\cdot;\Theta_{t})$ | Student network on the target TKG.
$g(e_{s},e_{t},t;\Phi)$ |  Alignment module measuring correspondence of $(e_{s},e_{t})$ at $t$.
$\mathcal{L}_{\tilde{\mathcal{G}}_{t}}$, $\mathcal{L}_{\tilde{\mathcal{G}}_{t}^{ST}}$ | Reasoning loss on groundtruth/pseudo target TKG.
$\mathcal{L}_{\tilde{\Gamma}_{s\leftrightarrow t}}$, $\mathcal{L}_{\tilde{\Gamma}_{s\leftrightarrow t}^{ST}}$ | Alignment loss on groundtruth/pseudo alignment pairs.
$\mathcal{L}_{s\rightarrow t}$ | Cross-lingual reasoning loss from source TKG to target TKG.
$\mathcal{L}_{s\rightarrow t}^{ST}$ | Cross-lingual reasoning loss on both groundtruth and pseudo data.

###### Definition 2.1 (Temporal Knowledge Graph).

A temporal knowledge graph (TKG) is denoted as
$\mathcal{G}=\\{(e,r,e^{\prime},t)|t\leq
T\\}\subseteq\mathcal{E}\times\mathcal{R}\times\mathcal{E}\times\mathcal{T}$,
where $\mathcal{E}$ denotes the entities set, $\mathcal{R}$ denotes the
relation set, $\mathcal{T}$ denotes the timestamp set, and $T$ denotes the
latest update time. Each quadruple $(e,r,e^{\prime},t)$ refers to an event
that a subject entity $e\in\mathcal{E}$ has a relation $r\in\mathcal{R}$ with
an object entity $e^{\prime}\in\mathcal{E}$ at timestamp $t\in\mathcal{T}$.

###### Definition 2.2 (Multilingual TKGs and Alignments).

To denote multilingual TKGs, we further utilize subscript to represent
specific languages, i.e., $\mathcal{G}_{s}$ denotes TKG in the source language
and $\mathcal{G}_{t}$ denotes TKG in the target language. The corresponding
entities can be denoted as $e_{s}$ and $e_{t}$ respectively. Given two
different languages $s$,$t$, we have the cross-lingual alignment set
$\Gamma_{s\leftrightarrow t}$. To be more practical, we further assume the TKG
in target language $\mathcal{G}_{t}$ and alignment set
$\Gamma_{s\leftrightarrow t}$ are incomplete: $\tilde{\mathcal{G}}_{t}$,
$\tilde{\Gamma}_{s\leftrightarrow t}$.

Figure 2. An overview of MP-KD. (a) The source TKG is more complete than the
target TKG, and the cross-lingual alignments are also scarce; (b) A
teacher/student representation module to represent source/target TKG, and an
alignment module for knowledge transfer; (c) Mutually-paced knowledge
distillation between knowledge transfer and pseudo alignment generation.

Based on the definition above, we formalize our cross-lingual reasoning task
on TKGs as follows:

###### Definition 2.3 (Cross-lingual reasoning on TKGs).

Given the TKG $\mathcal{G}_{s}$ in the source language and the incomplete TKG
$\tilde{\mathcal{G}}_{t}$ in the target language before the latest update time
$T$, and the incomplete cross-lingual alignment
$\tilde{\Gamma}_{s\leftrightarrow t}$, we aim to predict future events in the
target TKG after time $T$. Concretely, we aim to predict missing entity in
each future quadruple:
$\\{(e_{t},r,?,t)~{}\text{or}~{}(?,r,e_{t}^{\prime},t)|t>T\\}$ in the target
TKG.

##  3\. Methodology

In this section, we present the proposed MP-KD framework for the cross-lingual
temporal knowledge graph reasoning task.

###  3.1. Overview

Figure 2 shows an overview of MP-KD. Given the TKGs in source language and
target language, the teacher network and the student network first represent
the source and target TKGs in a temporally evolving uni-space respectively. To
facilitate training of the student, the knowledge distillation is enabled by a
cross-lingual alignment module and an explicit temporal event transfer
process. To deal with the scarcity issue of cross-lingual alignments, we
propose a pseudo alignment generation technique to facilitate the knowledge
distillation process, which is mutually-paced along with model training. To
address the temporal knowledge discrepancy issue, the alignment module pulls
the aligned entities close to each other based on the alignment strength which
is dynamically adjusted.

Section 3.2 and Section 3.3 introduce our teacher/student network and the
knowledge distillation respectively, followed by Section 3.4 which details how
we generate pseudo alignments. Finally, Section 3.5 specifies our learning
objective on both groundtruth data and pseudo data, and summarizes the
training of MP-KD.

###  3.2. The Teacher/Student Network

We train two identical temporal representation modules on source and target
TKGs with different parameters. The representation module $f(\cdot;\Theta)$
parameterized by $\Theta$ is designed to measure the plausibility of each
quadruple, which represents each entity $e$ into a low-dimensional latent
space at each time: $\mathbf{h}_{e}(t)\in\mathbb{R}^{d}$. On a TKG
$\mathcal{G}$, entities $e\in\mathcal{E}$ are evolving, as they interact with
different entities over time. Such temporally interacted entities are defined
as temporal neighbors. Therefore, we aim to model the temporal pattern of each
entity $e$ by encoding the changes of temporal neighbors.

Towards this goal, $f(\cdot;\Theta)$ first samples temporal neighbors
$\mathcal{N}_{e}(t)$ from the TKG for each entity $e\in\mathcal{E}$.
$\mathcal{N}_{e}(t)$ consists of a set of the most recently interacted
entities at time $t$. Then $f(\cdot;\Theta)$ attentively aggregates
information from the temporal neighbors. Specifically, given the temporal
neighbor $\mathcal{N}_{e}(t)$, we represent the entity $e$ as
$\mathbf{h}_{e}(t)$ at time $t$:

(1)
$\small\mathbf{h}^{l}_{e}(t)=\sigma\left(\sum_{(e_{i},r_{i},t_{i})\in\mathcal{N}_{e}(t)}\alpha^{l}_{e,e_{i}}\left(\mathbf{h}_{e_{i}}^{l-1}(t_{i})\mathbf{W}\right)\right),$

where $l$ denotes the layer number, $\sigma(\cdot)$ denotes the activation
function ReLU, $\alpha^{l}_{e,e_{i}}$ denotes the attention weight of entity
$e_{i}$ to the represented entity $e$, and $\mathbf{W}$ is the trainable
transformation matrix. To aggregate from history, $\alpha^{l}_{e,e_{i}}$ is
supposed to be aware of entity feature, time delay and topology feature
induced by relations. Thus, we design the attention weight
$\alpha^{l}_{e,e_{i}}$ as follows:

(2)
$\small\alpha^{l}_{e,e_{i}}=\frac{\exp(q^{l}_{e,e_{i}})}{\underset{(e_{k},r_{k},t_{k})\in\mathcal{N}_{e}(t)}{\sum}\exp(q^{l}_{e,e_{k}})},~{}~{}q^{l}_{e,e_{k}}=\mathbf{a}\left(\mathbf{h}^{l-1}_{e}\|\mathbf{h}^{l-1}_{e_{k}}\|\mathbf{h}_{r_{k}}\|\kappa(t-t_{k})\right),$

where $q^{l}_{e,e_{k}}$ measures the pairwise importance by considering the
entity embedding, relation embedding and time embedding,
$\mathbf{a}\in\mathbb{R}^{4d}$ is the shared parameter in the attention
mechanism. Following (TGAT, ) we adopt random Fourier features as time
encoding $\kappa(\Delta t)$ to reflect the time difference.

To measure plausibility of each possible quadruple, we utilize TransE (TransE,
) as the score function
$f(e,r,e^{\prime},t;\Theta)=-\|\mathbf{h}^{l}_{e}(t)+\mathbf{h}_{r}-\mathbf{h}^{l}_{e^{\prime}}(t)\|^{2}$,
where true quadruples should have higher scores. To optimize the parameter
$\Theta$ on a TKG $\mathcal{G}$, we set the objective to rank the scores of
true quadruples higher than all other false quadruples produced by negative
sampling:

(3)
$\small\mathcal{L}_{\mathcal{G}}=\underset{(e,r,e^{\prime},t)\in\mathcal{G}}{\mathbb{E}}\left[\max(0,\lambda_{1}-f(e,r,e^{\prime},t;\Theta)+f(e,r,e^{-},t;\Theta))\right],$

where $(e,r,e^{-},t)$ is negative samples with object $e^{\prime}$ replaced by
$e^{-}$, $\lambda_{1}$ is the margin to distinguish positive and negative
quadruples.

###  3.3. Knowledge Distillation

The incomplete target TKG, $\tilde{\mathcal{G}}t$, can be used to train the
corresponding parameter $\Theta_{t}$ through minimization of
$\mathcal{L}{\tilde{\mathcal{G}}_{t}}$. However, the low-resource nature of
the target language often results in an incomplete target TKG, leading to
suboptimal $\Theta_{t}$. In light of this, we propose a knowledge distillation
approach to transfer temporal knowledge from the source TKG to the target TKG.
The proposed approach consists of two components: an alignment module that
enhances $\Theta_{t}$ using the more informative $\Theta_{s}$ learned from the
source TKG, and an explicit temporal event transfer based on the improved
parameters. This integrated approach aims to improve the completeness and
quality of the target TKG by leveraging the knowledge contained in the source
TKG.

The Alignment Module. In general, the source parameters $\Theta_{s}$ provide a
more informative representation of each entity $e\in\mathcal{E}$ compared to
the target parameters $\Theta_{t}$. To take advantage of this, we utilize
$\Theta_{s}$ to guide the optimization of $\Theta_{t}$ through the alignment
module $g(\cdot;\Phi)$, which measures the correspondence between each pair of
entities and is parameterized by $\Phi$.

Directly pulling embeddings of aligned entities at all time steps can transfer
misleading knowledge due to the temporal knowledge discrepancy. Therefore, the
alignment module first utilizes a temporal attention layer to integrate
information of each entity from history in both source and target TKGs, i.e.,
$\mathbf{H}^{s}_{e}(t),\mathbf{H}^{t}_{e}(t)\in\mathbb{R}^{d}$, then it pulls
such integration $\mathbf{H}^{s}_{e}(t)$ close to $\mathbf{H}^{t}_{e}(t)$
instead of the initial $\mathbf{h}^{s}_{e}(t)$ and $\mathbf{h}^{t}_{e}(t)$.
Moreover, the temporal integration $\mathbf{H}^{s}_{e}(t)$ and
$\mathbf{H}^{t}_{e}(t)$ also encode the temporal evolution information for
each entity, which can be utilized to estimate the adaptive alignment strength
at different time to improve the alignment module. Concretely, the temporal
integration is learned by:

(4) $\displaystyle\small\mathbf{H}^{s}_{e}(t)$ $\displaystyle=\text{Temporal-
Attn}(\mathbf{h}^{s}_{e}(1),\mathbf{h}^{s}_{e}(2),\cdots,\mathbf{h}^{s}_{e}(t)),$
$\displaystyle\mathbf{H}^{t}_{e}(t)$ $\displaystyle=\text{Temporal-
Attn}(\mathbf{h}^{t}_{e}(1),\mathbf{h}^{t}_{e}(2),\cdots,\mathbf{h}^{t}_{e}(t)),$
(5) $\small
g(e_{s},e_{t},t;\Phi)=\frac{\mathbf{H}^{s}_{e}(t)\cdot\mathbf{H}^{t}_{e}(t)}{\|\mathbf{H}^{s}_{e}(t)\|_{2}\cdot\|\mathbf{H}^{t}_{e}(t)\|_{2}},$

where Temporal-Attn is the temporal attention network designed to integrate
information on the temporal domain. The correspondence between each pair of
entities $(e_{s},e_{t})$ across source and target languages at time $t$ is
measured by $g(e_{s},e_{t},t;\Phi)$. As the temporal knowledge for aligned
entities is not identical, the alignment strength between them should vary
across time $t$. The alignment strength is strong when the two entities share
similar information, and weak when the information is dissimilar or the
alignment is unreliable. This variability is achieved through the design of a
trainable weight $\beta_{e,t}$ to adjust the alignment strength for different
entities at different times, which is generated by a cross-lingual attention
layer:

(6) $\small\beta_{e,t}=\text{Cross-
Attn}(key=\mathbf{H}^{t}_{e}(1:T),query=\mathbf{H}^{s}_{e}(1:T))_{tt}.$

Due the page limitation, we refer readers to Appendix A.1 for the detailed
implementation of $\text{Temporal-Attn}(\cdot)$ and $\text{Cross-
Attn}(\cdot)$.

To optimize the parameter $\Phi$ on the incomplete alignments
$\tilde{\Gamma}_{s\leftrightarrow t}$, we set the objective in order to rank
the correspondence of true alignments higher than false alignments:

(7) $\small\mathcal{L}_{\tilde{\Gamma}_{s\leftrightarrow
t}}=\underset{\tilde{\Gamma}_{s\leftrightarrow
t}}{\mathbb{E}}\left[\underset{t\in\mathcal{T}}{\mathbb{E}}\left[\beta_{e,t}\cdot\max(0,\lambda_{2}-g(e_{s},e_{t},t;\Phi)+g(e_{s},e_{t}^{-},t;\Phi))\right]\right],$

where the entity pair $(e_{s},e_{t})\in\tilde{\Gamma}_{s\leftrightarrow t}$ is
the aligned entities across languages, $(e_{s},e_{t}^{-})$ is the negative
samples, $\lambda_{2}$ is the margin value.

Temporal Event Transfer. Cross-lingual alignments offer the potential to
directly transfer temporal events towards the progressive completion of the
target TKG. This is based on the premise that entities that are reliably
aligned are likely to experience similar temporal events across languages,
with the same relations.

Given an aligned pair $(e_{s},e_{t})$, the temporal event
$(e_{t},r,e_{t}^{?},t)$ or $(e_{t}^{?},r,e_{t},t)$ is added to the target TKG
if the corresponding event $(e_{s},r,e_{s}^{?},t)$ or $(e_{s}^{?},r,e_{s},t)$
exists in the source TKG $\mathcal{G}_{s}$. To determine the missing entity
$e_{t}^{?}$, we first verify if $(e_{s}^{?},e_{t}^{?})$ is present in the
alignment set. If so, the temporal event is directly added to the target TKG.
Otherwise, the updated student network $f(\cdot;\Theta_{t})$ is utilized to
predict the missing entity and the top-1 entity is utilized to complete the
temporal event. We define the set of transferred temporal events in the target
TKG as $\tilde{\mathcal{G}}_{t}^{ST}$ for ease of discussion.

###  3.4. Generating Pseudo Alignments

The limited amount of cross-lingual alignments negatively constrain the effect
of the knowledge distillation process. In this section, we introduce how to
generate pseudo alignments $\tilde{\Gamma}_{s\leftrightarrow t}^{ST}$ with
high confidence to boost cross-lingual transfer effectiveness.

To expand the range of alignments used in the knowledge transfer process, we
generate pseudo-alignments with high confidence scores and incorporate them
into the training data. The confidence score for each pair of entities
$(e_{s},e_{t})$ is calculated as the average cosine similarity:
$sim(e_{s},e_{t})=\underset{t}{\mathbb{E}}\left[g(e_{s},e_{t},t;\Phi)\right]$.
While pair-wise similarity comparison is computationally intensive, we improve
efficiency by first adding alignments for entities that are neighbors of
already aligned entities
$\tilde{\mathcal{E}}t=\\{e_{t}|(e_{s},e_{t})\in\tilde{\Gamma}_{s\leftrightarrow
t}\\}$ in the target TKG, as they are likely to be represented well to produce
reliable alignment. Following (bootEA, ), we formulate the generation process
as solving the following optimization problem:

(8) $\displaystyle\max$
$\displaystyle\sum_{e_{t}\in\mathcal{N}(\tilde{\mathcal{E}}_{t})}\sum_{e_{s}\in\mathcal{E}_{s}}sim(e_{s},e_{t})\cdot\phi(e_{s},e_{t}),$
$\displaystyle s.t.$
$\displaystyle\sum_{e_{s}\in\mathcal{E}_{s}}\phi(e_{s},e_{t})=1,~{}~{}~{}\sum_{e_{t}\in\mathcal{N}(\tilde{\mathcal{E}}_{t})}\phi(e_{s},e_{t})=1,$

where $\phi(e_{s},e_{t})$ is a binary indicator of whether to add
$(e_{s},e_{t})$ as pseudo alignment, $\phi(e_{s},e_{t})=1$ if we choose to add
this pair, otherwise $\phi(e_{s},e_{t})=0$. The two constrains can guarantee
each entity $e_{t}\in\mathcal{N}(\tilde{\mathcal{E}}_{t})$ is aligned to at
most one entity in source language $e_{s}\in\mathcal{E}_{s}$. Finally, all
pairs that satisfying $\phi(e_{s},e_{t})=1$ can be viewed as candidates to be
added into alignment data. We further select the top ones in terms of
$sim(e_{s},e_{t})$ to control the total size of pseudo alignments. Notably, in
each generation, the target entities to be aligned can already have the
alignment, i.e.,
$\mathcal{N}(\tilde{\mathcal{E}}_{t})\bigcup\tilde{\mathcal{E}}_{t}\neq\emptyset$.
In this case, we can update the existing alignments with the pseudo ones to
eliminate the possible alignment noise.

Input: Source TKG $\mathcal{G}_{s}$, incomplete target TKG
$\tilde{\mathcal{G}}_{t}$, incomplete alignment
$\tilde{\Gamma}_{s\leftrightarrow t}$.

Output: Student Model Parameter $\Theta_{t}$ for target TKG.

1 Optimize $\Theta_{s}$ by minimizing $\mathcal{L}_{\mathcal{G}_{s}}$ on
source TKG;

2 Initialize $\Theta_{t}$ $\leftarrow$ $\Theta_{s}$ for target TKG;

3 while  _model not converged_ do

4 Optimize alignment module $g(\cdot;\Phi)$:

5 Minimize $\mathcal{L}_{s\rightarrow t}^{ST}$ in Eq. (10) w.r.t. alignment
parameter $\Phi$;

6 Transfer temporal events $\tilde{\mathcal{G}}_{t}^{ST}$ based on updated
$\Theta_{t}$;

7 Optimize student representation module $f_{t}(\cdot;\Theta_{t})$:

8 for  _Each time step $T_{i}$ during training_ do

9 Prepare training data
$\\{(e_{t},r,e_{t}^{\prime},t)|(e_{t},r,e_{t}^{\prime},t)\in\tilde{\mathcal{G}}_{t}\bigcup\tilde{\mathcal{G}}_{t}^{ST}~{}\text{and}~{}T_{i}<t<T_{i+1}\\}$

10 Update $\Theta_{t}$ by minimizing $\mathcal{L}_{s\rightarrow t}^{ST}$ in
Eq. (10);

11

12 end for

13 Generate pseudo alignments $\tilde{\Gamma}_{s\leftrightarrow t}^{ST}$ based
on updated $\Phi$;

14 end while

Algorithm 1 The optimization process for MP-KD.

###  3.5. Mutually-paced Optimization

Learning Objective. Given a source TKG $\mathcal{G}_{s}$, the incomplete
target TKG $\tilde{\mathcal{G}}_{t}$, and the incomplete cross-lingual
alignment $\tilde{\Gamma}_{s\leftrightarrow t}$, the objective of cross-
lingual temporal knowledge graph reasoning $\mathcal{L}_{s\rightarrow t}$ can
be summarized as follows:

(9) $\mathcal{L}_{s\rightarrow
t}=\mathcal{L}_{\tilde{G}_{t}}+\mathcal{L}_{\tilde{\Gamma}_{s\leftrightarrow
t}},$

where $\mathcal{L}_{\tilde{G}_{t}}$ denotes knowledge graph reasoning loss
which measures the correctness of each quadruple,
$\mathcal{L}_{\tilde{\Gamma}_{s\leftrightarrow t}}$ denotes the alignment loss
which measures the distance of aligned entities in the uni-space. To enlarge
the knowledge distillation effect, we progressively transfer temporal events
$\tilde{\mathcal{G}}_{t}$ and generate high-quality pseudo alignment
$\tilde{\Gamma}_{s\leftrightarrow t}^{ST}$. Therefore, the training objective
on both ground-truth data and pseudo data $\mathcal{L}_{s\rightarrow t}^{ST}$
becomes:

(10) $\displaystyle\mathcal{L}_{s\rightarrow t}^{ST}$
$\displaystyle=\frac{|\tilde{\mathcal{G}}_{t}|}{|\tilde{\mathcal{G}}_{t}|+|\tilde{\mathcal{G}}_{t}^{ST}|}\cdot\mathcal{L}_{\tilde{\mathcal{G}}_{t}}+\frac{|\tilde{\mathcal{G}}_{t}^{ST}|}{|\tilde{\mathcal{G}}_{t}|+|\tilde{\mathcal{G}}_{t}^{ST}|}\cdot\mathcal{L}_{\tilde{\mathcal{G}}_{t}^{ST}}$
$\displaystyle+\frac{|\tilde{\Gamma}_{s\leftrightarrow
t}|}{|\tilde{\Gamma}_{s\leftrightarrow t}|+|\tilde{\Gamma}_{s\leftrightarrow
t}^{ST}|}\cdot\mathcal{L}_{\tilde{\Gamma}_{s\leftrightarrow
t}}+\frac{|\tilde{\Gamma}_{s\leftrightarrow
t}^{ST}|}{|\tilde{\Gamma}_{s\leftrightarrow
t}|+|\tilde{\Gamma}_{s\leftrightarrow
t}^{ST}|}\cdot\mathcal{L}_{\tilde{\Gamma}_{s\leftrightarrow t}^{ST}},$

where $|\cdot|$ denotes the set size. Eq. (10) formulates the learning
objective on both scarce data and pseudo data for cross-lingual temporal
knowledge graph reasoning in target languages. We give the convergence
analysis in the following theorem:

###### Theorem 3.1.

Let $N$ denote the number of negative samples for optimization, $\epsilon$
denotes the portion of correct pseudo data, $\beta$ denotes the proportion of
pseudo data to the initial ground-truth data. As the number of negative
samples $N\rightarrow\infty$, the $\mathcal{L}_{s\rightarrow t}^{ST}$
converges to its limit with an absolute deviation decaying in
$O(\frac{1+\epsilon}{1+\beta}\cdot N^{-2/3})$.

###### Proof.

Refer to Appendix A.2. ∎

Mutually-paced Optimization and Generation. The MP-KD framework can be
optimized by minimizing Eq. (10) w.r.t. $\Theta$ and $\Phi$ alternatively. The
generated pseudo alignments can help the training of the representation
modules by the knowledge distillation, and in turn transferring temporal
events in the target TKG can improves alignment module by providing high-
quality representations. In light of this, we propose a mutually-paced
optimization and generation procedure. Generally speaking, we iteratively
generate pseudo alignments and update the representation module and alignment
module respectively. To be concrete, as shown in Algorithm 1, we first update
the alignment module $g(\cdot;\Phi)$ and transfer temporal events to transfer
knowledge from source to target. Then we divide the time span into several
time steps to update the student representation module from recent time step
to far away ones. Finally, we generate the pseudo alignments, as the
optimization $\Theta_{t}$ on all temporal events can improve the entity
feature quality, which is beneficial for alignment prediction. Algorithm 1
summarizes the training procedure.

##  4\. Experiment

Table 2. Statistics of the datasets.

Languages | Entity | Relation | Quadruple | Train/Val/Test | Time
---|---|---|---|---|---
English (EN) | 34,416 | 105 | 602K | 602K/0K/0K | 28
French (FR) | 32,546 | 105 | 580K | 580K/0K/0K | 28
Spanish (ES) | 31,808 | 105 | 316K | 114K/136K/66K | 40
German (DE) | 27,657 | 105 | 268K | 97K/114K/56K | 40
Italian (IT) | 23,734 | 94 | 236K | 84K/100K/51K | 40
Danish (DA) | 15,710 | 94 | 125K | 48K/50K/26K | 40
Slovene (SL) | 13,250 | 94 | 55K | 24K/21K/10K | 40
Bulgarian (BG) | 3,508 | 105 | 23K | 8K/9K/6K | 40

Table 3. Overall Performance without alignment noise. Average results on $5$
independent runs are reported. $*$ indicates the statistically significant
improvements over the best baseline, with $p$-value smaller than $0.01$.

Models | Target | ES | DE | IT | DA | BG | SL | Avg.
---|---|---|---|---|---|---|---|---
| Source | MRR | H@10 | MRR | H@10 | MRR | H@10 | MRR | H@10 | MRR | H@10 | MRR | H@10 | MRR | H@10
RE-GCN w/o source | NA | 14.31 | 31.85 | 16.32 | 34.19 | 14.59 | 31.64 | 14.19 | 31.24 | 10.27 | 23.44 | 9.33 | 21.63 | 13.17 | 29.00
Static KG embedding methods
TransE (TransE, ) | EN | 11.67 | 26.73 | 15.19 | 31.37 | 9.15 | 21.44 | 12.71 | 23.31 | 10.17 | 23.72 | 9.73 | 21.83 | 11.44 | 24.73
FR | 12.37 | 27.79 | 14.01 | 28.30 | 11.38 | 23.19 | 10.05 | 22.10 | 11.88 | 23.01 | 10.63 | 22.44 | 11.72 | 24.47
T.R. | 0.84 | 0.86 | 0.89 | 0.87 | 0.70 | 0.71 | 0.80 | 0.73 | 1.07 | 1.00 | 1.09 | 1.02 | 0.88 | 0.85
TransR (TransR, ) | EN | 11.88 | 28.66 | 16.01 | 32.01 | 8.14 | 22.07 | 13.34 | 24.73 | 10.33 | 23.51 | 8.89 | 22.12 | 11.43 | 25.52
FR | 12.01 | 28.32 | 14.58 | 29.51 | 9.93 | 24.66 | 11.90 | 22.64 | 11.98 | 23.44 | 9.27 | 23.88 | 11.61 | 25.41
T.R. | 0.83 | 0.89 | 0.94 | 0.90 | 0.62 | 0.74 | 0.89 | 0.76 | 1.09 | 1.00 | 0.97 | 1.06 | 0.87 | 0.88
DistMult (DistMult, ) | EN | 13.66 | 29.77 | 17.46 | 33.19 | 11.63 | 26.63 | 14.63 | 25.91 | 9.97 | 22.92 | 9.08 | 20.44 | 12.74 | 26.48
FR | 12.58 | 28.73 | 16.03 | 31.81 | 12.12 | 27.76 | 11.64 | 22.97 | 9.01 | 23.77 | 10.13 | 21.07 | 11.92 | 26.02
T.R. | 0.92 | 0.92 | 1.03 | 0.95 | 0.81 | 0.86 | 0.93 | 0.78 | 0.92 | 1.00 | 1.03 | 0.96 | 0.94 | 0.91
RotatE (RotatE, ) | EN | 12.99 | 28.89 | 19.87 | 35.46 | 15.62 | 30.14 | 13.44 | 25.79 | 11.10 | 22.98 | 11.37 | 23.99 | 14.07 | 27.88
FR | 13.01 | 29.33 | 17.63 | 34.81 | 14.99 | 31.04 | 11.62 | 23.17 | 10.73 | 23.14 | 11.10 | 24.66 | 13.18 | 27.69
T.R. | 0.91 | 0.91 | 1.15 | 1.03 | 1.05 | 0.97 | 0.88 | 0.78 | 1.06 | 0.98 | 1.20 | 1.12 | 1.03 | 0.96
Temporal KG embedding methods
TA-DistMult (TA-DistMult, ) | EN | 15.83 | 34.77 | 18.99 | 37.46 | 14.98 | 29.99 | 14.97 | 30.01 | 9.02 | 21.10 | 8.74 | 17.76 | 13.75 | 28.51
FR | 16.61 | 35.83 | 17.81 | 37.96 | 15.58 | 31.21 | 13.21 | 28.58 | 9.63 | 22.91 | 9.03 | 18.83 | 13.65 | 29.22
T.R. | 1.13 | 1.11 | 1.13 | 1.10 | 1.05 | 0.97 | 0.99 | 0.94 | 0.91 | 0.94 | 0.95 | 0.85 | 1.04 | 1.00
RE-Net (Renet, ) | EN | 17.58 | 37.97 | 19.03 | 39.46 | 15.88 | 33.69 | 15.03 | 34.77 | 12.01 | 25.72 | 11.07 | 25.64 | 15.10 | 32.88
FR | 17.01 | 36.79 | 18.32 | 38.07 | 15.47 | 34.83 | 15.63 | 33.86 | 12.31 | 25.03 | 11.79 | 24.97 | 15.09 | 32.26
T.R. | 1.21 | 1.17 | 1.14 | 1.13 | 1.07 | 1.08 | 1.08 | 1.10 | 1.18 | 1.08 | 1.23 | 1.17 | 1.15 | 1.12
RE-GCN (RE-GCN, ) | EN | 16.88 | 36.54 | 19.84 | 40.17 | 16.17 | 34.84 | 15.99 | 35.62 | 12.22 | 26.02 | 10.63 | 23.38 | 15.29 | 32.76
FR | 17.14 | 37.01 | 19.63 | 41.01 | 16.44 | 35.61 | 15.03 | 33.19 | 11.91 | 25.13 | 11.09 | 22.77 | 15.21 | 32.45
T.R. | 1.19 | 1.15 | 1.21 | 1.19 | 1.12 | 1.11 | 1.09 | 1.10 | 1.17 | 1.09 | 1.16 | 1.07 | 1.16 | 1.12
Multilingual KG embedding methods
KEnS (KEnS, ) | EN | 15.98 | 33.91 | 17.33 | 37.62 | 14.41 | 31.44 | 14.47 | 29.61 | 12.88 | 26.77 | 11.03 | 24.99 | 14.35 | 30.72
FR | 17.02 | 34.07 | 16.61 | 37.99 | 15.57 | 33.82 | 13.62 | 30.24 | 12.03 | 24.32 | 10.51 | 23.86 | 14.23 | 30.72
T.R. | 1.15 | 1.07 | 1.04 | 1.11 | 1.03 | 1.03 | 0.99 | 0.96 | 1.21 | 1.09 | 1.15 | 1.13 | 1.09 | 1.06
AlignKGC (AlignKGC, ) | EN | 13.59 | 33.19 | 16.44 | 33.14 | 13.71 | 34.07 | 12.13 | 31.07 | 11.33 | 26.63 | 8.32 | 20.77 | 12.59 | 29.81
FR | 13.90 | 34.71 | 17.14 | 34.81 | 14.97 | 33.65 | 12.07 | 30.44 | 10.92 | 25.31 | 9.64 | 21.28 | 13.11 | 30.03
T.R. | 0.96 | 1.07 | 1.03 | 0.99 | 0.98 | 1.07 | 0.85 | 0.98 | 1.08 | 1.11 | 0.96 | 0.97 | 0.98 | 1.03
SS-AGA (SS-AGA, ) | EN | 15.11 | 32.19 | 16.49 | 36.14 | 14.83 | 33.31 | 12.27 | 30.68 | 12.99 | 27.03 | 11.55 | 25.07 | 13.87 | 30.74
FR | 16.54 | 33.99 | 18.32 | 37.19 | 15.02 | 32.99 | 11.73 | 29.98 | 11.13 | 25.62 | 11.01 | 23.64 | 13.96 | 30.57
T.R. | 1.11 | 1.04 | 1.07 | 1.07 | 1.02 | 1.05 | 0.85 | 0.97 | 1.17 | 1.12 | 1.21 | 1.13 | 1.06 | 1.06
MP-KD * | EN | 19.51 | 41.55 | 22.84 | 49.30 | 17.18 | 37.62 | 18.79 | 40.01 | 14.33 | 30.13 | 13.87 | 30.30 | 17.75 | 38.15
FR | 19.05 | 42.86 | 21.67 | 46.57 | 17.92 | 39.18 | 17.95 | 37.95 | 13.85 | 29.27 | 12.54 | 27.36 | 17.16 | 37.20
T.R. | 1.35 | 1.33 | 1.36 | 1.40 | 1.20 | 1.21 | 1.29 | 1.25 | 1.37 | 1.27 | 1.42 | 1.33 | 1.33 | 1.3
Gains |  | 11% | 13% | 15% | 20% | 9% | 10% | 18% | 12% | 10% | 11% | 18% | 18% | 16% | 16%

We evaluate MP-KD on EventKG data (EventKG, ) including 2 source languages and
6 target languages, and we aim to answer the following research questions:

  * •

RQ1: How does MP-KD perform compared with state-of-the-art models on the low-
resource target languages?

  * •

RQ2: How do reliability of alignment information (with various noise ratio)
affect model performances?

  * •

RQ3: How do each component and important parameters affect MP-KD performance?

###  4.1. Datasets

We evaluate MP-KD by 12 cross-lingual TKG transfer tasks on EventKG data
(EventKG, ), which is a multilingual TKG including 2 source languages and 6
target languages. For each language, we collect events during 1980 to 2022 and
split the time span into 40 time steps for training, validation and testing
(28/4/8). Table 2 shows the dataset statistics. We describe the dataset
details in Appendix A.3.

###  4.2. Experimental Setup

Baselines. We compare ten state-of-the-art baselines from three related areas.
We describe the baseline details in Appendix A.4.

  * •

Static KG embedding methods: TransE (TransE, ), TransR (TransR, ), DistMult
(DistMult, ), RotatE (RotatE, );

  * •

Temporal KG embedding methods: TA-DistMult (TA-DistMult, ), RE-NET (Renet, ),
RE-GCN (RE-GCN, );

  * •

Multilingual KG embedding methods: KEnS (KEnS, ); AlignKGC (AlignKGC, ); SS-
AGA (SS-AGA, ).

Evaluation Protocol and Metrics. For each prediction $(e,r,?,t)$ or
$(?,r,e,t)$, we rank missing entities to evaluate the performance. Following
(RE-GCN, ), we adopt raw mean reciprocal rank (MRR) and raw Hits at 10 (H@10)
as evaluation metrics. To quantitatively compare how well the transferred
knowledge from the source languages can improve predictions on the low-
resource languages, we adopt Transfer Ratio (T.R.) to evaluate the average
improvement of each method over the best baseline without knowledge
transferring, i.e.:

(11) $\small
T.R.(t_{i})=\frac{1}{|S|}\sum_{s_{i}\in\mathcal{S}}\frac{\text{Model}(s_{i}\rightarrow
t_{i})}{\text{BestBaseline}(t_{i})}$

where $t_{i}$ denotes each target language, $s_{i}\in\mathcal{S}$ denotes each
source language, and $\text{BestBaseline}(t_{i})$ denotes the best baseline
performance on the target language $t_{i}$ without any knowledge transferring,
i.e., RE-GCN w/o source.

Implementation. To simulate scarce setting, by default, we utilize $10\%$
alignments and $20\%$ events of target TKG by random selection. For
static/temporal KG embedding methods, we merge source graph and target graph
by adding one new type of relation (alignment). For multilingual baselines, we
train them on 1-to-1 knowledge transferring (instead of the original setting)
for fair comparison. We introduce implementation details of baseline models
and MP-KD in Appendix A.5. Code and data are open-source and available at
https://github.com/amzn/mpkd-thewebconf-2023.

Figure 3. Experimental results under various alignment noise ratios. Average
H@10 on 6 target languages are reported. MP-KD achieves relatively robust
results, with only $3.7\%$ relative drop, others have over $10\%$ drop.

###  4.3. Experiments on Cross-lingual Reasoning (RQ1)

We first evaluate the model performance with incomplete cross-lingual
alignments, where we randomly preserve $10\%$ alignments of the target
entities for distilling knowledge. Table 3 reports the overall results for the
cross-lingual experiments. By utilizing only $10\%$ cross-lingual alignments,
MP-KD achieves $33\%$ (MRR) and $30\%$ (H@10) relative improvement over best
baseline without the knowledge transferring (RE-GCN w/o source) on average,
demonstrating the effectiveness of MP-KD in modeling alignments for knowledge
transferring. Compared with ten baselines using alignments, MP-KD still
achieves relative $14\%$ relative improvements over the second best results.
Specifically, we have the following observations:

  * •

Static baseline (TransE, TransR, DistMult, RotatE) fail to beat RE-GCN w/o
source, although using alignments, due the insufficient modeling of temporal
information. Similarly, multilingual methods (KEnS, AlignKGC, SS-AGA) also
produce unsatisfying results;

  * •

All temporal baselines (TA-DistMult, RE-Net, RE-GCN) manage to beat RE-GCN w/o
source, as the modeling of both temporal evolution and cross-lingual alignment
can facilitate the representation learning of target entities. But the
improvements are marginal compared with our model, as the effect of knowledge
distillation is constrained by the limited amount of cross-lingual alignments;

  * •

Our model consistently achieves the best performance. Through $10\%$
alignments, MP-KD can progressively transfer temporal knowledge and generate
pseudo alignments with high confidence to boost the effect and range of the
knowledge distillation;

  * •

We also notice the uneven improvements across languages, (e.g., $40\%$
improvements for German, $20\%$ for Italian). We hypothesize it is because of
various language dependencies with source languages.

###  4.4. Experiments under Alignment Noises (RQ2)

In reality, cross-lingual alignments can be obtained by human labeling or
rule-based inference modules, which may introduce indispensable noises. We
evaluate how the reliability of alignment information affects baseline models
and MP-KD. In this experiment, we still utilize $10\%$ alignments. To simulate
unreliable alignments, we select a subset of alignments (measured by Noise
Ratio) and randomly change the aligned target entity to another entity without
alignment information.

Table 4. Ablation Studies.

Ablations | Target | ES | SL | Avg.
---|---|---|---|---
| Source | MRR | H@10 | MRR | H@10 | MRR | H@10
MP-KD w/o Align. Strength Control | EN | 17.61 | 38.59 | 13.07 | 28.51 | 16.24 | 37.04
FR | 17.07 | 39.46 | 13.03 | 28.14 | 16.33 | 36.61
T.R. | 1.21 | 1.23 | 1.27 | 1.21 | 1.21 | 1.22
MP-KD w Pure Training | EN | 17.09 | 37.03 | 11.89 | 26.25 | 14.81 | 31.90
FR | 16.99 | 37.10 | 11.78 | 25.91 | 14.97 | 32.15
T.R. | 1.19 | 1.16 | 1.15 | 1.11 | 1.13 | 1.09
MP-KD w/o Pseudo Align. | EN | 17.97 | 38.04 | 12.31 | 27.83 | 15.98 | 34.83
FR | 17.55 | 38.45 | 12.19 | 26.32 | 15.79 | 35.27
T.R. | 1.24 | 1.20 | 1.19 | 1.16 | 1.22 | 1.19
MP-KD w/o Event Transfer | EN | 18.79 | 39.03 | 13.07 | 28.07 | 16.03 | 36.74
FR | 18.83 | 39.88 | 12.95 | 28.79 | 15.99 | 36.95
T.R. | 1.31 | 1.24 | 1.27 | 1.21 | 1.27 | 1.24
MP-KD | EN | 19.51 | 41.55 | 14.33 | 30.13 | 17.75 | 38.15
FR | 19.05 | 42.86 | 13.85 | 29.27 | 17.16 | 37.20
T.R. | 1.35 | 1.33 | 1.37 | 1.27 | 1.33 | 1.3

We vary the noise ratio from $0.0$ to $0.2$ to evaluate the models
performance, as shown in Figure 3. We report the average H@10 on 6 target
languages by utilizing English TKG and French TKG respectively. As expected,
with the increase of noise ratio, the performances of all compared models
degrade, as the wrong alignment links mislead the knowledge transfer process.
Most baselines fail to beat RE-GCN w/o Source even with $10\%$ noise, and all
lose with $20\%noise$, which indicates that the quality of alignments
significantly influences the model effectiveness in the cross-lingual TKG
reasoning task. Notably, MP-KD achieves relatively robust results, with only
$3.7\%$ performance drop, while other strong baselines have over $10\%$ drop.
This is because during the generation of pseudo alignments, MP-KD can
automatically replace those unreliable ones based on the confidence score.
Also, in the alignment module, MP-KD can assign small alignment strength to
unreliable alignments.

###  4.5. Model Analysis (RQ3)

Ablation Study. We evaluate performance improvements brought by the MP-KD
framework by following ablations:

  * •

MP-KD w/o Align. Strength Control uniformly set the alignment strength for all
entities across all time steps;

  * •

MP-KD w Pure Training optimizes the teacher-student framework without pseudo
alignment generation and temporal event transfer;

  * •

MP-KD w/o Pseudo Align eliminates the pseudo alignments generation process;

  * •

MP-KD w/o Event Transfer eliminates the explicit transfer of temporal events.

Table 4 reports the results measured by $H@10$. Each component leads to
performance boost. MP-KD with uniform alignment strength largely degrades
performance, due to temporal knowledge discrepancy. MP-KD without pseudo data
generation achieves similar performance with temporal baselines RE-Net, RE-
GCN, because of the limited amount of cross-lingual alignments. Bother
generating pseudo alignments and explicitly transferring temporal events
increase the performance, and combining them together in a mutually-paced
procedure (in MP-KD) can achieve the best results.

The Effect of Pseudo Alignments Ratio. To investigate the effects of the
pseudo alignments on the reasoning performance, we vary the amount of pseudo
alignments during training period and compare the corresponding performance
measured by H@10, as shown in Figure 4(a). The blue line and red line show the
performances of single model on complete target TKG and single model on $20\%$
target TKG (our setting) respectively. From $0.1$, MP-KD starts to generate
and expand the initially available alignments. We observe a significant
performance improvement, demonstrating the positive effects of the pseudo
alignments. As expected, we find a performance decrease at $50\%$, as the
added pseudo data with relatively low confidence start to introduce noise that
hurt the performance.

(a) Pseudo align. analysis.

(b) Efficiency analysis.

Figure 4. Efficiency analysis and pseudo alignment analysis.

###  4.6. Efficiency Comparison

To demonstrate the efficiency of MP-KD framework, we train MP-KD and baseline
models from scratch on both target language and source language, and compare
the training time. Figure 4(b) shows that MP-KD significantly outperforms
baseline models with reasonable training time. More details are provided in
Appendix A.5.3.

##  5\. Related Work

Knowledge Graph Reasoning. Knowledge graph reasoning aims to predict missing
facts to automatically complete KGs (DBPedia, ; WIKI, ; YAGO, ; acekg, ). It
is mostly formulated as measuring the correctness of factual samples and
negative samples by specially designed score functions (TransE, ; RotatE, ;
factKG, ; multiKG, ). Recently, reasoning on temporal KGs attracts a lot of
interests from the community (Know-Evolve, ; TA-DistMult, ; Renet, ; RE-GCN,
). Compared with static KG reasoning task, the main challenge lies in how to
incorporate time information. Several embedding-based methods have been
proposed. They encode time-dependent information of entities and relations by
decoupling embeddings into static component and time-varying component
(TKGC_TE, ; GoelAAAI2020, ; metatkgr, ), utilizing recurrent neural networks
(RNNs) to adaptively learn the dynamic evolution from historical fact sequence
(Renet, ; RE-GCN, ), or learning a sequence of evolving representations from
discrete knowledge graph snapshots (TKGC-ODE, ; DBKGE, ; Renet, ; RE-GCN, ).
However, all of the existing temporal KG reasoning models aim to extrapolate
future facts based on relatively complete TKGs in high-resource languages, and
how to boost reasoning performance for TKGs in low-resource languages through
cross-lingual alignments is largely under-explored.

Multilingual KG Reasoning. Entity alignment methods on KGs (EA1, ; DKGA, ;
bright, ; selfKG, ; multi-network, ) can automatically enlarge the alignments
by predicting the correspondence between the two KGs. But most of them, if not
all, require the relatively even completeness of two KGs to capture the
structural similarities, which can not be satisfied in our case, as target
TKGs are far from complete. Inspired by recent transfer learning advances
(pan2009survey, ; li2018hierarchical, ; li2019exploiting, ; li2019sal, ;
li2020learn, ; juan2022disentangle, ), some recent works start to study the
multilingual KG reasoning on static graphs (AlignKGC, ; KEnS, ; SS-AGA, ),
which aim to extract knowledge from several source KGs to boost the reasoning
performance in the target KG, while they still require a sufficient amount of
seed alignments and totally ignore the temporal information in our task. We
extend this line of works on TKGs, where transferring temporal knowledge is
more complex.

Self-training. Self-training is one of the learning strategies that addresses
data scarcity issue by fully utilizing abundant unlabeled data (ST, ; ST1, ;
li2021queaco, ). Recent works start to study self-training strategy for graph
data, as GNNs typically require large amount of data labeling (ST-GCN, ;
CaGCN-st, ; Shift, ). In summary, most efforts are put on node classification
problem, where node labels are largely unavailable. We focus on utilizing
self-training technique to deal with link scarcity (events + alignments),
which is also a bottleneck for improving the performance on graphs.

##  6\. Conclusion

In this paper, we studied a realistic but underexplored cross-lingual temporal
knowledge graph reasoning problem, which aims at facilitating TKG reasoning in
low-resource languages by distilling knowledge from a corresponding TKG in
high-resource language through a small set of entity alignments as bridges. To
this end, we proposed a novel mutually-paced teacher student framework, namely
MP-KD. During training, MP-KD iteratively generates pseudo alignments to
expand the cross-lingual connection, as well as transfers temporal facts to
facilitate student model training in low-resource languages. Our alignment
module is learned to adjust the alignment strength for different entities at
different time, thereby maximizing the benefits of knowledge transferring. We
empirically validated the effectiveness of MP-KD on 12 language pairs of
EventKG data, on which the proposed framework significantly outperforms an
extensive set of state-of-the-art baselines.

###### Acknowledgements.

The authors would like to thank the anonymous reviewers for their valuable
comments and suggestions. Research reported in this paper was sponsored in
part by DARPA award HR001121C0165, DARPA award HR00112290105, Basic Research
Office award HQ00342110002, the Army Research Laboratory under Cooperative
Agreement W911NF-17-20196, and Amazon.com Inc.

## References

  * [1] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.  Translating embeddings for modeling multi-relational data.  In Advances in Neural Information Processing Systems, 2013.
  * [2] Elizabeth Boschee, Jennifer Lautenschlager, Sean O’Brien, Steve Shellman, James Starz, and Michael Ward.  Icews coded event data.  2015\.
  * [3] Muhao Chen, Yingtao Tian, Mohan Yang, and Carlo Zaniolo.  Multilingual knowledge graph embeddings for cross-lingual knowledge alignment.  In IJCAI’17.
  * [4] Xuelu Chen, Muhao Chen, Changjun Fan, Ankith Uppunda, Yizhou Sun, and Carlo Zaniolo.  Multilingual knowledge graph completion via ensemble knowledge transfer.  In EMNLP, pages 3227–3238, 2020.
  * [5] da Xu, chuanwei ruan, evren korpeoglu, sushant kumar, and kannan achan.  Inductive representation learning on temporal graphs.  In ICLR’20, 2020.
  * [6] Alberto García-Durán, Sebastijan Dumančić, and Mathias Niepert.  Learning sequence encoders for temporal knowledge graph completion.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
  * [7] Rishab Goel, Seyed Mehran Kazemi, Marcus A. Brubaker, and Pascal Poupart.  Diachronic Embedding for Temporal Knowledge Graph Completion.  In AAAI’20, 2020.
  * [8] Simon Gottschalk and Elena Demidova.  Eventkg: the hub of event knowledge on the web and biographical timeline generation.  Semantic Web, 2019.
  * [9] M. Gutmann and A. Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Y.W. Teh and M. Titterington, editors, Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS), volume 9 of JMLR W & CP, pages 297–304, 2010.
  * [10] Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp.  Learning neural ordinary equations for forecasting future links on temporal knowledge graphs.  In NeurIPS’21, 2021.
  * [11] Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato.  Revisiting self-training for neural sequence generation.  CoRR, abs/1909.13788, 2019.
  * [12] Zijie Huang, Zheng Li, Haoming Jiang, Tianyu Cao, Hanqing Lu, Bing Yin, Karthik Subbian, Yizhou Sun, and Wei Wang.  Multilingual knowledge graph completion with self-supervised adaptive graph alignment.  In ACL’22, 2022.
  * [13] Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren.  Renet: Autoregressive structure inference over temporal knowledge graphs.  In EMNLP, 2020.
  * [14] M. Kumar, Benjamin Packer, and Daphne Koller.  Self-paced learning for latent variable models.  In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, NIPS’10, volume 23. Curran Associates, Inc., 2010.
  * [15] Julien Leblay and Melisachew Wudage Chekol.  Deriving validity time in knowledge graph.  In WWW’18, 2018.
  * [16] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer.  DBpedia - a large-scale, multilingual knowledge base extracted from wikipedia.  Semantic Web Journal, 6(2):167–195, 2015.
  * [17] Jinning Li, Huajie Shao, Dachun Sun, Ruijie Wang, Yuchen Yan, Jinyang Li, Shengzhong Liu, Hanghang Tong, and Tarek Abdelzaher.  Unsupervised belief representation learning with information-theoretic variational graph auto-encoders.  In SIGIR’22, 2022.
  * [18] Qimai Li, Zhichao Han, and Xiao-Ming Wu.  Deeper insights into graph convolutional networks for semi-supervised learning.  In AAAI’18, 2018.
  * [19] Zheng Li, Mukul Kumar, William Headden, Bing Yin, Ying Wei, Yu Zhang, and Qiang Yang.  Learn to cross-lingual transfer with meta graph learning across heterogeneous languages.  In EMNLP’20, pages 2290–2301, 2020.
  * [20] Zheng Li, Xin Li, Ying Wei, Lidong Bing, Yu Zhang, and Qiang Yang.  Transferable end-to-end aspect-based sentiment analysis with selective adversarial learning.  In EMNLP’19, pages 4590–4600, Hong Kong, China, 2019.
  * [21] Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang.  Hierarchical attention transfer network for cross-domain sentiment classification.  In AAAI’18, 2018.
  * [22] Zheng Li, Ying Wei, Yu Zhang, Xiang Zhang, and Xin Li.  Exploiting coarse-to-fine task transfer for aspect-level sentiment classification.  In AAAI’19, 2019.
  * [23] Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng.  Temporal knowledge graph reasoning based on evolutional representation learning.  In SIGIR, 2021.
  * [24] Siyuan Liao, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang.  Learning dynamic embeddings for temporal knowledge graphs.  In WSDM ’21, 2021.
  * [25] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu.  Learning entity and relation embeddings for knowledge graph completion.  In AAAI’15, 2015.
  * [26] Hongrui Liu, Binbin Hu, Xiao Wang, Chuan Shi, Zhiqiang Zhang, and Jun Zhou.  Confidence may cheat: Self-training on graph neural networks under distribution shift.  CoRR, abs/2201.11349, 2022.
  * [27] Lihui Liu, Boxin Du, Yi Ren Fung, Heng Ji, Jiejun Xu, and Hanghang Tong.  Kompare: A knowledge graph comparative reasoning system.  In KDD’21, page 3308–3318, New York, NY, USA, 2021. Association for Computing Machinery.
  * [28] Lihui Liu, Boxin Du, Jiejun Xu, Yinglong Xia, and Hanghang Tong.  Joint knowledge graph completion and question answering.  In KDD ’22, page 1098–1108, New York, NY, USA, 2022. Association for Computing Machinery.
  * [29] Xiao Liu, Haoyun Hong, Xinghao Wang, Zeyi Chen, Evgeny Kharlamov, Yuxiao Dong, and Jie Tang.  Selfkg: Self-supervised entity alignment in knowledge graphs.  Proceedings of the ACM Web Conference 2022, 2022.
  * [30] Farzaneh Mahdisoltani, Joanna Biega, and Fabian M. Suchanek.  Yago3: A knowledge base from multilingual wikipedias.  In CIDR, 2015.
  * [31] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009.
  * [32] Jianwei Qian, Xiang-Yang Li, Chunhong Zhang, Linlin Chen, Taeho Jung, and Junze Han.  Social network de-anonymization and privacy inference with knowledge graph model.  IEEE Transactions on Dependable and Secure Computing, pages 679–692, 2019.
  * [33] Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar.  Question answering over temporal knowledge graphs.  In ACL’21, August 2021.
  * [34] H. J. Scudder.  Probability of error of some adaptive pattern-recognition machines.  IEEE Trans. Inf. Theory, 11:363–371, 1965.
  * [35] Huajie Shao, Shuochao Yao, Andong Jing, Shengzhong Liu, Dongxin Liu, Tianshi Wang, Jinyang Li, Chaoqi Yang, Ruijie Wang, and Tarek Abdelzaher.  Misinformation detection and adversarial attack cost analysis in directional social networks.  In ICCCN, pages 1–11, 2020.
  * [36] Harkanwar Singh, Soumen Chakrabarti, PRACHI JAIN, Sharod Roy Choudhury, and Mausam.  Multilingual knowledge graph completion with joint relation and entity alignment.  In Conference on Automated Knowledge Base Construction, 2021.
  * [37] Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu.  Bootstrapping entity alignment with knowledge graph embedding.  In IJCAI’18, page 4396–4402, 2018.
  * [38] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang.  Rotate: Knowledge graph embedding by relational rotation in complex space.  In International Conference on Learning Representations, 2019.
  * [39] Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song.  Know-evolve: Deep temporal reasoning for dynamic knowledge graphs.  In ICML’17.
  * [40] Haiwen Wang, Ruijie Wang, Chuan Wen, Shuhao Li, Yuting Jia, Weinan Zhang, and Xinbing Wang.  Author name disambiguation on heterogeneous information network with adversarial representation learning.  In AAAI ’20, 2020.
  * [41] Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo.  Ripplenet: Propagating user preferences on the knowledge graph for recommender systems.  In CIKM ’18, 2018.
  * [42] Ruijie Wang, Zijie Huang, Shengzhong Liu, Huajie Shao, Dongxin Liu, Jinyang Li, Tianshi Wang, Dachun Sun, Shuochao Yao, and Tarek Abdelzaher.  Dydiff-vae: A dynamic variational framework for information diffusion prediction.  In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, 2021.
  * [43] Ruijie Wang, Zheng Li, Danqing Zhang, Qingyu Yin, Tong Zhao, Bing Yin, and Tarek Abdelzaher.  Rete: Retrieval-enhanced temporal event forecasting on unified query product evolutionary graph.  In Proceedings of the ACM Web Conference 2022, WWW ’22, page 462–472, 2022.
  * [44] Ruijie Wang, Yuchen Yan, Jialu Wang, Yuting Jia, Ye Zhang, Weinan Zhang, and Xinbing Wang.  Acekg: A large-scale knowledge graph for academic data mining.  In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM ’18, 2018.
  * [45] Ruijie Wang, zheng li, Dachun Sun, Shengzhong Liu, Jinning Li, Bing Yin, and Tarek Abdelzaher.  Learning to sample and aggregate: Few-shot reasoning over temporal knowledge graphs.  In Advances in Neural Information Processing Systems, 2022.
  * [46] Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  CoRR, abs/2005.10242, 2020.
  * [47] Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang.  Be confident! towards trustworthy graph neural networks via confidence calibration.  In NeurIPS’21.
  * [48] Chenjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Hamed Yazdi, and Jens Lehmann.  Temporal knowledge graph completion based on time series gaussian embedding.  In The Semantic Web – ISWC 2020: 19th International Semantic Web Conference, Athens, Greece, November 2–6, 2020, Proceedings, Part I, page 654–671, 2020.
  * [49] Miao Xu, Bingcong Li, Gang Niu, Bo Han, and Masashi Sugiyama.  Revisiting sample selection approach to positive-unlabeled learning: Turning unlabeled data into positive rather than negative.  CoRR, abs/1901.10155, 2019.
  * [50] Yuchen Yan, Lihui Liu, Yikun Ban, Baoyu Jing, and Hanghang Tong.  Dynamic knowledge graph alignment.  In AAAI, volume 35, 2021.
  * [51] Yuchen Yan, Si Zhang, and Hanghang Tong.  Bright: A bridging algorithm for network alignment.  In Proceedings of the Web Conference 2021, 2021.
  * [52] Yuchen Yan, Qinghai Zhou, Jinning Li, Tarek Abdelzaher, and Hanghang Tong.  Dissecting cross-layer dependency inference on multi-layered inter-dependent networks.  In CIKM’22, 2022.
  * [53] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and li Deng.  Embedding entities and relations for learning and inference in knowledge bases.  In ICLR’14.
  * [54] Chaoqi Yang, Jinyang Li, Ruijie Wang, Shuochao Yao, Huajie Shao, Dongxin Liu, Shengzhong Liu, Tianshi Wang, and Tarek Abdelzaher.  Hierarchical overlapping belief estimation by structured matrix factorization.  In ASONAM’20, 2020.
  * [55] Juan Zha, Zheng Li, Ying Wei, and Yu Zhang.  Disentangling task relations for few-shot text classification via self-supervised hierarchical task clustering.  In EMNLP’22, 2022.
  * [56] Danqing Zhang, Zheng Li, Tianyu Cao, Chen Luo, Tony Wu, Hanqing Lu, Yiwei Song, Bing Yin, Tuo Zhao, and Qiang Yang.  Queaco: Borrowing treasures from weakly-labeled behavior data for query attribute value extraction.  In CIKM’21.
  * [57] Yuyue Zhao, Xiang Wang, Jiawei Chen, Yashen Wang, Wei Tang, Xiangnan He, and Haiyong Xie.  Time-aware path reasoning on knowledge graph for recommendation.  ACM Trans. Inf. Syst., 2022.
  * [58] Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu.  Improving conversational recommender systems via knowledge graph based semantic fusion.  In KDD’20, page 1006–1014, 2020.

##  Appendix A Appendix

###  A.1. Model Description

In this section, we introduce the temporal attention layer and cross-lingual
attention layer for entity alignments utilized in Section 3.3. We first
introduce the general attention mechanism we utilized, then specify the two
layers respectively.

Given two representation sequence from temporal domain: key sequence
$\boldsymbol{H}_{K}=\\{\boldsymbol{h}_{K}^{1},\boldsymbol{h}_{K}^{2},\cdots,\boldsymbol{h}_{K}^{T}\\}$
and query sequence
$\boldsymbol{H}_{Q}=\\{\boldsymbol{h}_{Q}^{1},\boldsymbol{h}_{Q}^{2},\cdots,\boldsymbol{h}_{Q}^{T}\\}$
from all time steps, we propose the following attention to calculate the
pairwise importance:

(12)
$\small\mathbf{\beta}=\text{Attn}(key=\boldsymbol{H}_{K},query=\boldsymbol{H}_{Q})=\operatorname{softmax}\left(\frac{\boldsymbol{H}_{Q}\boldsymbol{W}^{Q}(\boldsymbol{H}_{K}\boldsymbol{W}^{K})^{T}}{\sqrt{d}}+\boldsymbol{M}\right),$

where $\boldsymbol{W}^{Q}$, $\boldsymbol{W}^{L}$ are trainable temporal
parameters, $\mathbf{\beta}$ is learned temporal weight indicating pairwise
importance, $d$ denotes dimension of input representations, and
$\boldsymbol{M}$ is added to ensure auto-regressive setting, i.e., preventing
future information affecting current state. We define $\boldsymbol{M}_{ij}=0$
if $i\leq j$, otherwise $-\infty$.

For temporal attention layer, we use
$\boldsymbol{h}_{e}=\\{\boldsymbol{h}_{e}(1),\boldsymbol{h}_{e}(2),\cdots,\boldsymbol{h}_{e}(T)\\}$
for both query and key sequence to obtain the temporal attention weights
$\beta$:

(13)
$\small\mathbf{\beta}=\text{Attn}(key=\boldsymbol{h}_{e},query=\boldsymbol{h}_{e}),$

then the desired $\boldsymbol{H}_{e}(t)$ is leaned as the combination of input
sequence, where $\boldsymbol{W}^{V}$ is a trainable matrix.:

(14) $\displaystyle\small\boldsymbol{H}_{e}(t)$ $\displaystyle=\text{Temporal-
Attn}(\boldsymbol{h}_{e}(1),\cdots,\boldsymbol{h}_{e}(t))$
$\displaystyle=\sum_{i=1}^{t}\beta_{it}\boldsymbol{h}_{e}(i)\boldsymbol{W}^{V}$

For cross-lingual attention layer, we use
$\boldsymbol{H}^{s}_{e}=\\{\boldsymbol{H}^{s}_{e}(1),\cdots,\boldsymbol{H}^{s}_{e}(T)\\}$
in source language as query sequence and
$\boldsymbol{H}^{t}_{e}=\\{\boldsymbol{H}^{t}_{e}(1),\cdots,\boldsymbol{H}^{t}_{e}(T)\\}$
in target language as key sequence to obtain the attention weights $\beta$:

(15)
$\small\mathbf{\beta}_{e,t}=\text{Attn}(key=\boldsymbol{H}^{t}_{e},query=\boldsymbol{H}^{s}_{e})_{tt},$

where $\mathbf{\beta}_{e,t}$ is trainable weight to adjust the alignment
strength of different entities at different time.

###  A.2. Theorem Proof

###### Theorem A.1.

Let $N$ denote the number of negative samples for optimization, $\epsilon$
denotes the ratio of correct pseudo data, $\beta$ denotes the ratio of pseudo
data amount to the initial groundtruth data amount. As the number of negative
samples $N\rightarrow\infty$, the $\mathcal{L}_{s\rightarrow t}^{ST}$
converges to its limit with an absolute deviation decaying in
$O(\frac{1+\epsilon}{1+\beta}\cdot N^{-2/3})$.

###### Proof.

In representation learning, the margin loss has been widely adopted as the
similarity metric. Without loss of generality, they can be expressed in the
form of Noise Contrastive Estimation (NCE) [9]. Therefore, we express
$\mathcal{L}_{\mathcal{G}}$ and $\mathcal{L}_{\Gamma_{s\leftrightarrow t}}$ in
the form of Noise Contrastive Estimation (NCE) by introducing the negative
sampling:

(16)
$\scriptsize\mathcal{L}_{\mathcal{G}}\triangleq\mathbb{E}\left[-\log\frac{e^{f(\cdot;\Theta)/\tau}}{e^{f(\cdot;\Theta)/\tau}+\sum_{-}e^{\left(f^{-}(\cdot;\Theta)\right)/\tau}}\right],$

(17) $\scriptsize\mathcal{L}_{\Gamma_{s\leftrightarrow
t}}\triangleq\mathbb{E}\left[-\log\frac{e^{g(\cdot;\Phi))/\tau}}{e^{g(\cdot;\Phi)/\tau}+\sum_{-}e^{g^{-}(\cdot;\Phi)/\tau}}\right],$

for simplicity, $f^{-}(\cdot;\Theta)$ denotes the score for negative
quadruple, and $g^{-}(\cdot;\Phi)$ denotes score for negative alignment pair.

For our training objective $\mathcal{L}_{s\rightarrow t}^{ST}$, we show the
convergence analysis of four terms one by one, then prove the overall
convergence results. First of all, following [46, 29], let $N$ denote the
number of negative samples per each quadruple, and we have:

(18)
$\displaystyle\lim_{N\rightarrow\infty}\left[\mathcal{L}_{\tilde{\mathcal{G}}_{t}}-\log
M\right]$
$\displaystyle=-\frac{1}{\tau}\underset{(e_{t},r,e_{t}^{\prime},t)\in\tilde{\mathcal{G}}_{t}}{\mathbb{E}}\left[f(\cdot;\Theta)\right]$
$\displaystyle+\lim_{N\rightarrow\infty}\underset{e^{-}_{t}\in\mathcal{E}_{t}}{\underset{(e_{t},r,e_{t}^{\prime},t)\in\tilde{\mathcal{G}}_{t}}{\mathbb{E}}}\left[\log\left(\frac{\lambda}{N}e^{f(\cdot;\Theta)/\tau}+\frac{1}{N}\sum_{-}e^{f^{-}(\cdot;\Theta)/\tau}\right)\right]$
$\displaystyle=-\frac{1}{\tau}\underset{(e_{t},r,e_{t}^{\prime},t)\in\tilde{\mathcal{G}}_{t}}{\mathbb{E}}\left[f(\cdot;\Theta)\right]+\underset{(e_{t},r,e_{t}^{\prime},t)\in\tilde{\mathcal{G}}_{t}}{\mathbb{E}}\left[\log\underset{e^{-}_{t}\in\mathcal{E}_{t}}{\mathbb{E}}\left[e^{f^{-}(\cdot;\Theta)}\right]\right],$

where $\lambda$ denotes the duplicate quadruples co-existing in both
incomplete $\tilde{\mathcal{G}}_{t}$ and negative samples. The convergence
speed is derived as follows:

For one side:

(19) $\scriptsize\mathcal{L}_{\tilde{\mathcal{G}}_{t}}-\log
N-\lim_{N\rightarrow\infty}\left[\mathcal{L}_{\tilde{\mathcal{G}}_{t}}-\log
N\right]\leq\frac{\lambda}{N}e^{\frac{2}{\tau}}.$

For another side:

(20)
$\scriptsize\lim_{N\rightarrow\infty}\left[\mathcal{L}_{\tilde{\mathcal{G}}_{t}}-\log
N\right]-\left[\mathcal{L}_{\tilde{\mathcal{G}}_{t}}-\log
N\right]\leq\frac{\lambda}{N}e^{2/\tau}+\frac{5}{4}N^{-\frac{2}{3}}e^{\frac{1}{\tau}}(e^{\frac{1}{\tau}}-e^{-\frac{1}{\tau}}).$

We then generalize the above results to the loss term on pseudo data. Suppose
$\epsilon$ of the pseudo data are correct. Then we can have the following two
inequality. For one side:

(21) $\displaystyle\mathcal{L}_{\tilde{\mathcal{G}}_{t}^{ST}}-\log
N-\lim_{N\rightarrow\infty}\left[\mathcal{L}_{\tilde{\mathcal{G}}_{t}^{ST}}-\log
N\right]$
$\displaystyle\leq\epsilon\underset{e\in\mathcal{E}_{t}}{\mathbb{E}}\left[\log\frac{\underset{e^{-}\in\mathcal{E}_{t}}{\mathbb{E}}\left[\frac{\lambda}{N}e^{1/\tau}+e^{f^{-}(\cdot;\Theta_{t})/\tau}\right]}{\underset{e^{-}\in\mathcal{E}_{t}}{\mathbb{E}}e^{f^{-}(\cdot;\Theta_{t})/\tau}}\right]\leq\epsilon\frac{\lambda}{N}e^{\frac{2}{\tau}}$

Therefore, for $\mathcal{L}_{s\rightarrow t}^{ST}$ in this side, we have:

(22) $\scriptsize\mathcal{L}_{s\rightarrow t}^{ST}-\log
N-\lim_{N\rightarrow\infty}\left[\mathcal{L}_{s\rightarrow t}^{ST}-\log
N\right]\leq\frac{1+\epsilon}{1+\beta}\frac{\lambda}{N}e^{\frac{2}{\tau}},$

where $\beta$ is the ratio of pseudo data amount to groundtruth data amount
during training.

Similarly, for another side, we have:

(23) $\displaystyle\lim_{N\rightarrow\infty}\left[\mathcal{L}_{s\rightarrow
t}^{ST}-\log N\right]$ $\displaystyle-\left[\mathcal{L}_{s\rightarrow
t}^{ST}-\log
N\right]\leq\frac{1+\epsilon}{1+\beta}\frac{\lambda}{N}e^{2/\tau}$
$\displaystyle+\frac{1+\epsilon}{1+\beta}\frac{5}{4}N^{-\frac{2}{3}}e^{\frac{1}{\tau}}(e^{\frac{1}{\tau}}-e^{-\frac{1}{\tau}}).$

Therefore, we conclude that the $\mathcal{L}_{s\rightarrow t}^{ST}$ converges
to its limit with an absolute deviation decaying in
$O(\frac{1+\epsilon}{1+\beta}\cdot N^{-2/3})$

∎

###  A.3. Datasets

Dataset Information. The commonly utilized benchmark TKGs are divided into two
categories: temporal event graphs [2] and knowledge graphs where temporally
associated facts have valid periods [15, 30, 16]. In this paper, we mainly
evaluate MP-KD on the EventKG [8], which is a multilingual resource
incorporating event-centric information extracted from several large-scale
knowledge graphs such as Wikidata [15], DBpedia [16] and YAGO [30]. Each
temporal event is organized as $(e,r,e^{\prime},t_{s},t_{e})$, where each
piece of data is attached with a valid time period from start time $t_{s}$ to
end time $t_{e}$. Following [13], we preprocess the format such that each fact
is converted to a sequence
$\\{(e,r,e^{\prime},t_{s}),(e,r,e^{\prime},t_{s}+1),\cdots,(e,r,e^{\prime},t_{e})\\}$
from $t_{s}$ to $t_{e}$, with the minimum time unit as one step.

Splitting Scheme. We collect events during 1980 to 2022, and noisy events of
early years are removed. To construct multilingual TKGs, we first preserve
important entities and relations by excluding infrequent ones that have less
than $20$ events in each language. Then we collect the events and cross-
lingual alignments.To guarantee the relation match, we only preserve relations
appearing in English TKG. We split the time span into 40 equal time steps for
training, validation and testing (28/4/8), where each time step roughly lasts
for one year. To focus on the prediction on existing entities during training
period, and eliminate the negative effects possibly caused by the randomly
appearning new entities in val/test period, we only preserve entities having
events during training period, following [23]. Table 2 shows the dataset
statistics, including 2 source languages and 6 target languages. We
purposefully choose 6 different target languages with diverse characteristics
in term of the TKG size, which can evaluate MP-KD from different data
granularity. It is worth noting that to simulate the scarcity issue in target
TKGs, the training quadruples presented in Table 2 are randomly selected from
original TKGs, with random ratio $20\%$.

###  A.4. Baselines

We describe the baselines utilized in the experiments in detail:

  * •

TransE [1] is a translation-based embedding model, where both entities and
relations are represented as vectors in the latent space. The relation is
utilized as a translation operation between the subject and the object entity;

  * •

TransR [25] advances TransE by optimizing modeling of n-n relations, where
each entity embedding can be projected to hyperplanes defined by relations;

  * •

DistMult [53] is a general framework with bilinear objective for multi-
relational learning that unifies most multi-relational embedding models;

  * •

RotatE [38] represents entities as complex vectors and relations as rotation
operations in a complex vector space;

  * •

TA-DistMult [6] is a temporal knowledge graph reansoing method aiming at
predicting missing events in history. We utilize it for predicting future
events;

  * •

RE-NET [13] is a generative model to predict future facts on temporal
knowledge graphs, which employs a recurrent neural network to model the entity
evolution, and utilizes a neighborhood aggregator to consider the connection
of facts at the same time intervals;

  * •

RE-GCN [23] learns the temporal representations of both entities and relations
by modeling the KG sequence recurrently;

  * •

KEnS [4] starts to directly improve KGR performance on static KGs given a set
of seed alignment, and proposes an ensemble-based approach for the task;

  * •

AlignKGC [36] jointly optimizes entity alignment loss and knowledge graph
reasoning loss to improve the performance;

  * •

SS-AGA [12] views alignments as new edge type and employ a relation-aware GNN
with learnable attention weight to model the influence of the aligned
entities.

###  A.5. Reproducibility

####  A.5.1. Baseline Setup

For static knowledge graph reasoning methods, i.e., TransE, TransR, DistMult,
and RotatE, we ignore all time information in quadruples, and view temporal
knowledge graphs as static, cumulative ones. For static/temporal KG embedding
methods, we merge source graph and target graph by adding one new type of
relation (alignment), as they do not explicitly model cross-lingual entity
alignment. For multilingual baselines, we train them on 1-to-1 knowledge
transferring (instead of the original setting) for fair comparison. For static
baselines, we utilize the static embeddings for predictions in all time steps.
For fair comparisons, we keep the dimension of all embeddings as $128$, we
feed pre-trained TransE embeddings on the merge graph including both source
and target TKGs to those that require initial entity/relation embeddings. We
tune learning rate of baselines based on MRR on validation set, and we train
all baseline models and MP-KD on same GPUs (Nvidia A100) and CPUs (Intel(R)
Xeon(R) Platinum 8275CL).

####  A.5.2. MP-KD Setup

We first utilize the source TKG to train the teacher representation module.
Then we initialize the student module with the parameters of the teacher.
During the training procedure, we first optimize the objective without
generating pseudo data in the first 10 epoch. After that, we start to generate
high-quality pseudo data. For the generation in each epoch, we gradually
increase the amount of pseudo alignments from $10\%$ to $40\%$, and transfer
all temporal events that meet the requirement. During evaluation, we tune
hyperparameters based on MRR on validation set, and report the performance on
the test set. Next, we report the choices of hyperparameters. For model
training, we utilize Adam optimizer, and set maximum number of epochs as $50$.
We set batch size as $256$, the dimension of all embeddings as $128$, and
dropout rate as $0.5$. For the sake of efficiency, we set number of temporal
neighbors $b$ as $8$, and employ $1$ neighborhood aggregation layer in
temporal encoder. For TKG reasoning, we set negative sampling factor as 10.
For entity alignment, we set negative sampling factor as 50. For temporal
generation process, We divide time span into $4$ time intervals. For model
training, we mainly tune margin value $\lambda_{1}$, $\lambda_{2}$ in score
functions in range $\\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9\\}$, learning rate
in range $\\{0.02,0.01,0.005,0.001,0.0005\\}$.

####  A.5.3. Efficiency Comparison

To demonstrate the efficiency of MP-KD framework, we train MP-KD and baseline
models from scratch on both target language and source language, and compare
the training time. We train all baseline models and MP-KD on same GPUs (Nvidia
A100) and CPUs (Intel(R) Xeon(R) Platinum 8275CL). Figure 4(b) shows that MP-
KD significantly outperforms baseline models with reasonable training time.
Notably, we include the pseudo data generation time. Compared with slow
temporal models RE-NET, RE-GCN for knowledge graph reasoning, MP-KD is more
efficient because our temporal encoder can learn temporal entity embeddings
via sampled temporal neighbors at each time without using RNNs.