# Chain of History: Learning and Forecasting with LLMs for Temporal Knowledge
Graph Completion

Ruilin Luo1, Tianle Gu1∗, Haoling Li1∗, Junzhe Li2, Zicheng Lin1, Jiayi Li3,
Yujiu Yang1†

1Tsinghua Shenzhen International Graduate School, Tsinghua University
2School of Computer Science, Peking University
3Baidu Inc.
<EMAIL_ADDRESS>Equal contribution. $\dagger$
Corresponding author<EMAIL_ADDRESS>
###### Abstract

Temporal Knowledge Graph Completion (TKGC) is a challenging task of predicting
missing event links at future timestamps by leveraging established temporal
structural knowledge. Given the formidable generative capabilities inherent in
LLMs (LLMs), this paper proposes a novel approach to conceptualize temporal
link prediction as an event generation task within the context of a historical
event chain. We employ efficient fine-tuning methods to make LLMs adapt to
specific graph textual information and patterns discovered in temporal
timelines. Furthermore, we introduce structure-based historical data
augmentation and the integration of reverse knowledge to emphasize LLMs’
awareness of structural information, thereby enhancing their reasoning
capabilities. We conduct thorough experiments on multiple widely used datasets
and find that our fine-tuned model outperforms existing embedding-based models
on multiple metrics, achieving SOTA results. We also carry out sufficient
ablation experiments to explore the key influencing factors when LLMs perform
structured temporal knowledge inference tasks.

##  1 Introduction

Knowledge Graphs (KGs), as meticulously structured repositories of
deterministic knowledge, find application across various domains, including
recommender systems Qin et al. (2024), question-answering Liu et al. (2023),
and more recently, the eagerly anticipated field of Retrieval-augmented
Generation (RAG) Sun et al. (2023); Feng et al. (2023). In contrast, Temporal
Knowledge Graphs (TKGs), enriched with timestamped information, offer a more
precise archival of knowledge Cai et al. (2023). However, this precision
simultaneously results in a more generalized and complex landscape for
reasoning. The recent surge in the application of powerful generative
capabilities inherent in LLMs has demonstrated promising performance in tasks
beyond natural language processing Tang et al. (2023). Nevertheless, unlike
tasks in static scenarios, temporal reasoning mandates the model to capture
potential knowledge evolution patterns from historical data. Thus, can LLMs
exhibit comparable proficiency in generating events for the next time? We
posit that delving into TKGs, structures inherently characterized by textual
and temporal sequences, holds the potential to furnish an initial response to
this inquiry.

Figure 1: LLM undergoes fine-tuning on known data and subsequently utilizes
the chain of known factual information to generate the next event.

A Temporal Knowledge Graph (TKG) stores several facts in the form of quadruple
$(e_{h},r,e_{t},t_{T})$, which represents that $e_{h}$ has a directional edge
$r$ into $e_{t}$ at timestamp $t_{T}$. Given a series of seen facts denoted as
$\mathcal{F}=\\{(s,p,o,t_{s})|s,o\in\mathcal{S},p\in\mathcal{P},t_{s}<T\\}$,
TKGC under extrapolative setting requires a forecast on invisible links in the
form of $(s_{i},p_{j},?,t_{k})$, in which $t_{k}\geq T$ Zhu et al. (2021).
Furthermore, a reverse inference $(?,p_{j},o_{l},t_{k})$ should also be taken
into consideration. In static knowledge graph completion tasks, due to the
determinism of textual knowledge and the relatively simple semantic structure
of a single graph, many methods resort to language models to improve
performance and generalization Wang et al. (2022a, b). However, the innate
evolutionary characteristics of TKGs make the absolute representation of text
embeddings appear limited. Therefore, previous embedding-based methods more
concentrate on learning the evolvement of entities using relational graph
encoder and recurrent networks such as RNN and GRU Chung et al. (2014); Jin et
al. (2020); Li et al. (2021a).

Since the LLMs have demonstrated their powerful generative capabilities, we
emphasize that TKGC can be seen as a generative sequel to LLMs within the
chain of historical events. The swift evolution of instruction-tuning
techniques for LLMs allows for the adaptation of LLMs to this sequence
prediction task at a minimal cost. We construct instructions on known temporal
facts by selecting queries in the form of $(s_{i},p_{j},?,t_{k})$ and the
history associated with them. In the fine-tuning process, we inject
proprietary knowledge of specific TKGs into the model, along with the
exploration of various contextualized reasoning rules within the historical
chain.

In this study, we utilize LLMs to complete event generation on the history
chain to aid the TKGC task as shown in Fig. 1. During fine-tuning, we
partition the known data into an input section and a supervised labeling
segment, guiding LLMs in adapting the mapping relationship between the textual
information of the specific TKG and the intricate logic inherent in temporal
events. Furthermore, we propose to use structural information on TKGs for
historical data augmentation to explore the ability of LLMs to perceive graph
information. In addition, we explore different means of reverse data
incorporation to mitigate the curse of reversal Lv et al. (2023) problem in
structured knowledge reasoning.

We conduct comprehensive experiments on widely used TKGC datasets, including
the ICEWS Li et al. (2021a) series from news and the commonsense dataset YAGO
Mahdisoltani et al. (2015). Significantly, we report the Hits@n metric under
the most stringent raw setting and the more popular time-aware filtered
setting, achieving highly competitive results. We also provide the 8-shot in-
context learning (ICL) performance of several open-source models as a
comparative reference. Furthermore, we conduct thorough ablation experiments
to validate the effectiveness of structure-based historical data augmentation
methods and the introduction of reverse logic. Additionally, we investigate
the impact of historical chain length, model size, and the performance of LLMs
like GPT-4, aiming to uncover key factors influencing temporal structural
information reasoning using LLMs.

##  2 Related Work

Temporal Knowledge Graph Completion involves two essential reasoning settings:
interpolation and extrapolation. Interpolation-based TKG reasoning addresses
the challenge of filling in missing links within observed timestamps. Many
static knowledge graph embedding models have incorporated timestamp
information to tackle this issue García-Durán et al. (2018); Goel et al.
(2020). TTransE Leblay and Chekol (2018) introduces time-based encoding
through translation operations. Additionally, TNTComplEx Lacroix et al. (2020)
and TuckERTNT Shao et al. (2022) propose complex decomposition and TuckER
decomposition of four-order tensors, respectively, to augment model
expressiveness under temporal conditions. However, the interpolation setting
has limitations, as it cannot infer missing information in future timestamps,
thereby restricting its applicability.

Extrapolative reasoning in TKGC, involving the prediction of facts for future
timestamps, represents a more challenging yet valuable task. Recent works have
concentrated on leveraging multi-relational graph convolutional networks Li et
al. (2021a); Jin et al. (2020). GHT Sun et al. (2022) introduces a transformer
that incorporates graph Hawkes transformation, aiming to model the evolution
of entity representations over time. xERTE Han et al. (2021a) captures query-
related subgraph information through dynamic pruning operations. TANGO Han et
al. (2021b) adopts neural ordinary differential equations to model the
temporal representation of entities. TITer Sun et al. (2021) stands out as the
first model to utilize temporal-path-based reinforcement learning for TKG
reasoning. Furthermore, TLogic Liu et al. (2022) enhances interpretability by
extracting temporal logic rules through random exploration of time.

LLMs-as-Predictors LLM has demonstrated promising potential in the field of
graph machine learning, manifesting itself in two primary forms: LLM as an
enhancer and LLM as a predictor. Chen et al. (2023) The former harnesses LLMs
to enrich nodes’ textual attributes by leveraging their extensive knowledge
base. Enriched information is then utilized in conjunction with Graph Neural
Networks (GNNs) to generate predictions. However, such an approach only
utilizes the pre-trained knowledge of LLMs and ignores the reasoning ability
of them.

Hence, some studies transform graph structure information into sequential
representations and utilize LLMs as standalone predictors. Graph4GPT Guo et
al. (2023) uses InstructGPT-3 Ouyang et al. (2022) to conduct an empirical
study to assess LLMs’ capabilities in graph understanding, and GraphLLM Chai
et al. (2023) uses LLaMA2 for the graph reasoning task, but these work ignore
LLM’s ability to TKGC. Most relevant to our work, Lee et al. (2023a) uses ICL
with LLMs for TKGC, which may not fully exploit the extensive learning
capabilities of LLMs.

Parameter-Efficient Fine-tuning Transformer-based pre-trained models has
emerged as the cornerstone of NLP tasks, and fine-tune them with full
parameter updates has become a prevalent approach. However, recent studies
have introduced several more efficient fine-tuning techniques, which can be
seen as Parameter-Efficient Fine-tuning (PEFT) He et al. (2022) methods. These
methods include the addition of adapters Rebuffi et al. (2017); Houlsby et al.
(2019); Bapna et al. (2019), which entail the insertion of small trainable
feed-forward networks between fixed pre-trained models. Additionally, low-rank
updates Hu et al. (2021) have been proposed as an alternative, wherein the
fine-tuning process leverages low-dimensional representations. Moreover,
prompt tuning Lester et al. (2021) and prefix tuning Li and Liang (2021) have
been developed, which involve augmenting the model’s input or activations with
learnable parameters.

##  3 Preliminary

Definition 3.1. (TKGC) A TKG is defined as a sequence
$\mathcal{G}=\\{\mathcal{G}_{1},\cdots,\mathcal{G}_{t},\cdots,\mathcal{G}_{n}\\}$
comprising static KGs. Here, each static KG denoted as $\mathcal{G}_{t}$
contains factual triplets at timestamp $t$. A single static KG is formulated
as $\\{\mathcal{E},\mathcal{R},\mathcal{T}\\}$, in which $\mathcal{E}$,
$\mathcal{R}$ and $\mathcal{T}=\\{s_{i},p_{j},o_{k}\\}$ respectively represent
entities, relations and triplets within it. TKGC involves bidirectional
prediction of query quadruples, specifically, $(s_{i},p_{j},?,t_{s})$ and
$(o_{k},p_{j}^{-1},?,t_{s})$.

Definition 3.2. (Fine-tuning) Given a pre-trained LLM denoted as $\mathcal{M}$
with parameters $\mathcal{\theta}$, and a dataset comprising $n$ instances
$\\{\text{Query}_{i},\text{Response}_{i}\\}$, the fine-tune processing aims to
minimize the following loss function:

$\boldsymbol{\theta}^{\star}=\arg\min_{\boldsymbol{\theta}^{\prime}}\sum_{i=0}^{n-1}\mathcal{L}\left(\mathcal{M}\left(\mathcal{Q}|\boldsymbol{\theta}^{\prime}\right);\mathcal{R}\right)$
(1)

where $\mathcal{M}(|\theta^{\prime})$ denotes the output of the fine-tuned LLM
$\mathcal{M}$ with parameters $\theta^{\prime}$, $\mathcal{Q}$ represents
Query and $\mathcal{R}$ represents response.

##  4 Methodology

###  4.1 Structure-augmented History Modeling

The LLM’s predictions of undiscovered links in the TKG rely on knowledge
derived from historical facts. In particular, when dealing with a query
quadruple represented as $q=(s_{i},p_{j},?,t_{q})$ in a forward reasoning
mode, we aim to model the historical chain $\mathcal{H}_{q}$ associated with
this query.

Schema-matching History. The initial set of historical facts we leverage
originates from schema-matching records, denoted as
$H_{s}=\\{(s_{i},p_{j},o,t)|o\in\mathcal{E},t<t_{q}\\}$. Specifically, given a
query (Japan, Make_a_visit, ?, 305), $\mathcal{H}_{s}=\\{(\textit{Japan,
Make\\_a\\_visit, North\\_Korea, 296}),\cdots,\\\ (\textit{Japan,
Make\\_a\\_visit, North\\_Korea, 304})\\}$ encompasses relevant schema-
matching facts that align with the subject and predicate of the query $q$,
providing inference basis for LLMs.

Entity-augmented History. Similar to many prior works that leverage structural
information from KGs to enhance the reasoning capabilities of LLMs Luo et al.
(2023); Tian et al. (2023), we focus on semantically enriching the
representation of central entities by utilizing links with neighbors in TKGs.
The entity-augmented history $\mathcal{H}_{e}$ is defined as
$\\{(s_{i},p,o,t)|(s_{i},p,o,t)\in\mathcal{G}_{t},p\in\mathcal{R},o\in\mathcal{E},t<t_{q}\\}$
formally.

Relation-augmented History. In addition to completing the historical chain
based on entity-based neighbor information, we introduce a supplementary
strategy based on relations. We believe that it’s beneficial for enhancing the
model’s intrinsic understanding of relation inference Xiong et al. (2018).
Formally, relation-augmented history set
$\mathcal{H}_{r}=\\{(s,p_{j},o,t)|(s,p_{j},o,t)\in\mathcal{G}_{t},p,o\in\mathcal{E},t<t_{q}\\}$.

When modeling $\mathcal{H}_{q}$, we adhere to two criteria for selecting data
from $\mathcal{H}_{s},\mathcal{H}_{e}$, and $\mathcal{H}_{r}$. i) We
prioritize the ground-truth history directly related to $q$, which is
$\mathcal{H}_{s}$. If the history length does not meet the specified value, we
then sequentially incorporate facts from $\mathcal{H}_{e}$ and
$\mathcal{H}_{r}$. ii) Data close to the current timestamp is introduced with
priority. By following these two criteria, we aim to select the most relevant
knowledge to inspire forecasting capabilities in LLMs.

###  4.2 Introduction of Reverse Logic

Similar to reasoning on static KGs, we require the model to also possess the
capability of reverse inference on TKG Li et al. (2021a). However, recent
research indicates that LLM’s reasoning has encountered the issue of reversal
curse Qi et al. (2023); Berglund et al. (2023); Lv et al. (2023). In this
problem, models often succeed in correctly deducing questions like ’Who is Tom
Cruise’s mother?’ but struggle to answer ’Who is the son of Mary Lee
Pfeiffer?’. We believe that this phenomenon also exists in structured
knowledge reasoning. We propose using three prompt strategies to incorporate
reverse quadruples during the fine-tuning phase to alleviate this issue, and
explore the performance patterns in the context of structured knowledge
reasoning scenarios.

As demonstrated in Tbl. 1, the most ordinary construction is to treat the
structure of backward inferences as forward inferences. The text-aware prompt
leverages reverse to indicate reverse reasoning, and the position-aware prompt
follows the order of backward inference, providing different head entities in
the historical records.

Strategy | Prompt
---|---
Ordinary | 280: [Japan, Make_a_visit, China]
281: [Japan, Make_a_visit, Vietnam]
$\cdots$
304: [Japan, Make_a_visit, Kiichi_Miyazawa]
Query: 305: [Japan, Make_a_visit, ]
Text-aware | 280: [Japan, reverse Make_a_visit, China]
281: [Japan, reverse Make_a_visit, Vietnam]
$\cdots$
304: [Japan, reverse Make_a_visit, Kiichi_Miyazawa]
Query: 305: [Japan, reverse Make_a_visit, ]
Position-aware | 280: [China, Make_a_visit, Japan]
281: [Vietnam, Make_a_visit, Japan]
$\cdots$
304: [Kiichi_Miyazawa, Make_a_visit, Japan]
Query: 305: [ , Make_a_visit, Japan]

Table 1: Prompts for query (Japan, Make_a_visit-1, ?, 305).

###  4.3 Instruction-tuning in TKGC

Instruction Tuning Wei et al. (2021) achieves remarkable zero-shot
generalization results by training LLMs on different tasks with instructions.
While prior work has demonstrated the effectiveness of fine-tuning LLMs via
full-parameter updates, this approach presents considerable challenges at
large scale. Hence, we apply the Low-Rank Adaptation (LoRA) Hu et al. (2021)
method due to its demonstrated effectiveness for LLaMA-style models. This
method, founded on the plugin encapsulation strategy of PEFT, furnishes us
with lightweight task-specific plugins.

The LLM $\mathcal{M}$ generates a sequence of tokens
$\hat{\mathcal{R}}=\\{\hat{r_{1}},\hat{r_{2}},...\hat{r_{n}}\\}$, where
response $\mathcal{R}$ we need must be extracted and consists of a set of
consecutive tokens. Similarly to most fine-tuning LLMs process using LoRA, the
parameter update for a pre-trained weight matrix
${W_{0}}\in\mathbb{R}^{d\times k}$ is specified by product of two low-rank
matrices $W_{A}$ and $W_{B}$:

$\delta W=W_{A}W_{B}$ (2)

where ${W_{A}}\in\mathbb{R}^{d\times r}$ and ${W_{B}}\in\mathbb{R}^{r\times
k}$ are matrices of trainable parameters and
$\operatorname{rank}r\ll\min(d,k)$. Therefore, the forward pass for $h=W_{0}x$
is altered as :

$h=W_{0}{x}+\delta W_{x}=W_{0}x+W_{A}W_{B}x$ (3)

We employ cross-entropy loss which constrains the similarity between estimated
and ground-truth tokens, to fine-tune LLMs by LoRA, which can be presented as

$\mathcal{L}=CE(\mathcal{\hat{R}},\tilde{R})$ (4)

where $\hat{\mathcal{R}}$ is the temporal knowledge graph completion predicted
by LLM $\mathcal{M}$ and $\tilde{R}$ is the given label.

###  4.4 Predict with LLMs

The instructions constructed are fed into the trained LLMs for prediction. The
response is obtained by beam search, which is a decoding strategy that
maintains $k$ beams of possible generated responses at each time step $t$. The
generation of response is updated as follows: for each generated response, the
$k$ tokens with the highest probabilities are selected based on Eq. 5. This
results in $k\times k$ new response candidates. The next $k$ beams of response
are obtained by selecting the top $k$ responses with the highest probabilities
from the generated response candidates. The highest probability is determined
by the product of probabilities of $|\mathcal{\hat{R}}|$ tokens that
constitute the response, where $|\mathcal{\hat{R}}|$ represents the length of
the current response.

$r_{t}=argmax_{r}P(r|r_{1:t-1})$ (5)

In this context, the single step setting is employed, wherein for each test
query in the test dataset, the model can access the ground truth from past
timestamps. Consequently, after the prediction for this step is completed, the
ground truth from the current timestamp is added to the history of the next
timestamp before its execution.

##  5 Experiments

###  5.1 Datasets

In our experimental setup, we utilize the ICEWS14 dataset García-Durán et al.
(2018), ICEWS18 dataset Li et al. (2021a), ICEWS05-15 dataset Li et al.
(2021b), and YAGO dataset Mahdisoltani et al. (2015) as benchmarks for
evaluation. The specific statistics are listed in Tbl. 2. We employ partition
criteria widely accepted in prior studies Han et al. (2021a) and establish
instruction-tuning data on the validation set. Specifically, for the ordered
timestamp set
$T=\\{t_{train}^{1},t_{train}^{2},\cdots,t_{train}^{n},t_{val}^{1},\cdots,t_{val}^{m}\\}$,
comprising training and validation sets, when gathering historical data for
timestamp $t_{val}^{i}$, we observe only facts within the range
$t<t_{val}^{i}$. In the context of testing under a single-step setup Trivedi
et al. (2017), for a query at timestamp $t_{q}$, we construct a ground-truth
chain of history based on facts preceding timestamp $t_{q}$, serving as the
input to the model.

Datasets | Entity | Relation | Train | Valid | Test | Interval
---|---|---|---|---|---|---
ICEWS14 | 6869 | 230 | 74845 | 8514 | 7371 | 1 day
ICEWS05-15 | 10094 | 251 | 368868 | 46302 | 46159 | 1 day
ICEWS18 | 23033 | 256 | 373018 | 45995 | 49545 | 1 day
YAGO | 10623 | 10 | 161540 | 19523 | 20026 | 1 year

Table 2: Statistics of leveraged datasets.

Datasets | YAGO | ICEWS14 | ICEWS05-15 | ICEWS18
---|---|---|---|---
Model | Hits@1 | Hits@3 | Hits@10 | Hits@1 | Hits@3 | Hits@10 | Hits@1 | Hits@3 | Hits@10 | Hits@1 | Hits@3 | Hits@10
RE-NET Jin et al. (2020) | 0.404 | 0.530 | 0.629 | 0.293 | 0.431 | 0.575 | 0.334 | 0.478 | 0.611 | 0.192 | 0.323 | 0.483
RE-GCN Li et al. (2021a) | 0.499 | 0.663 | 0.779 | 0.297 | 0.441 | 0.586 | 0.336 | 0.487 | 0.658 | 0.193 | 0.331 | 0.494
xERTE Han et al. (2021a) | 0.506 | 0.719 | 0.828 | 0.312 | 0.453 | 0.570 | 0.347 | 0.497 | 0.633 | 0.206 | 0.330 | 0.458
TANGO$\dagger$ Han et al. (2021b) | 0.409 | 0.554 | 0.637 | 0.151 | 0.272 | 0.431 | 0.311 | 0.476 | 0.622 | 0.178 | 0.314 | 0.460
Timetraveler Sun et al. (2021) | 0.494 | 0.675 | 0.790 | 0.313 | 0.451 | 0.571 | 0.341 | 0.494 | 0.667 | 0.210 | 0.325 | 0.437
TiRGN Li et al. (2022) | 0.509 | 0.710 | 0.864 | 0.313 | 0.468 | 0.612 | 0.358 | 0.535 | 0.690 | 0.202 | 0.350 | 0.514
TLogic Han et al. (2021b) | 0.454 | 0.703 | 0.782 | 0.322 | 0.470 | 0.603 | 0.345 | 0.525 | 0.673 | 0.205 | 0.339 | 0.484
GPT-NeoX-20B-ICL Black et al. (2022) | 0.520 | 0.722 | 0.870 | 0.295 | 0.406 | 0.475 | 0.348 | 0.497 | 0.586 | 0.177 | 0.290 | 0.385
Llama-2-7b-ICL Touvron et al. (2023) | 0.517 | 0.725 | 0.868 | 0.275 | 0.391 | 0.453 | 0.353 | 0.490 | 0.563 | 0.177 | 0.295 | 0.364
Vicuna-7b-ICL Vicuna (2023) | 0.514 | 0.714 | 0.868 | 0.270 | 0.386 | 0.453 | 0.347 | 0.483 | 0.563 | 0.172 | 0.288 | 0.364
Llama-2-7b-CoH | 0.527 | 0.747 | 0.874 | 0.338 | 0.462 | 0.587 | 0.370 | 0.531 | 0.699 | 0.219 | 0.361 | 0.520

Table 3: Temporal forecasting with raw metrics Hits@1, Hits@3 and Hits@10. The
best results are highlighted in bold and the second-rank results are
underlined. The results of the model with $\dagger$ are derived from Han et
al. (2021b), while other models have been reproduced by us.

Datasets | YAGO | ICEWS14 | ICEWS05-15 | ICEWS18
---|---|---|---|---
Model | Hits@1 | Hits@3 | Hits@10 | Hits@1 | Hits@3 | Hits@10 | Hits@1 | Hits@3 | Hits@10 | Hits@1 | Hits@3 | Hits@10
RE-NET$\dagger$ Jin et al. (2020) | 0.586 | 0.715 | 0.868 | 0.301 | 0.440 | 0.582 | 0.336 | 0.488 | 0.627 | 0.197 | 0.326 | 0.485
RE-GCN$\dagger$ Li et al. (2021a) | 0.788 | 0.843 | 0.886 | 0.313 | 0.470 | 0.613 | 0.366 | 0.527 | 0.671 | 0.215 | 0.354 | 0.515
xERTE$\dagger$ Han et al. (2021a) | 0.801 | 0.880 | 0.898 | 0.327 | 0.457 | 0.573 | 0.378 | 0.523 | 0.639 | 0.210 | 0.335 | 0.465
TANGO$\ddagger$ Han et al. (2021b) | 0.590 | 0.646 | 0.677 | 0.272 | 0.408 | 0.550 | 0.344 | 0.499 | 0.640 | 0.191 | 0.318 | 0.462
Timetraveler$\dagger$ Sun et al. (2021) | 0.801 | 0.900 | 0.903 | 0.327 | 0.465 | 0.584 | 0.383 | 0.527 | 0.649 | 0.221 | 0.335 | 0.448
TiRGN Li et al. (2022) | 0.839 | 0.907 | 0.923 | 0.328 | 0.481 | 0.622 | 0.379 | 0.544 | 0.698 | 0.220 | 0.366 | 0.522
TLogic$\ddagger$ Han et al. (2021b) | 0.740 | 0.789 | 0.791 | 0.336 | 0.483 | 0.612 | 0.362 | 0.531 | 0.674 | 0.205 | 0.340 | 0.485
GPT-NeoX-20B-ICL Black et al. (2022) | 0.792 | 0.890 | 0.909 | 0.295 | 0.406 | 0.475 | 0.367 | 0.503 | 0.587 | 0.192 | 0.300 | 0.389
Llama-2-7b-ICL Touvron et al. (2023) | 0.767 | 0.852 | 0.868 | 0.286 | 0.397 | 0.453 | 0.353 | 0.490 | 0.563 | 0.177 | 0.294 | 0.364
Vicuna-7b-ICL Vicuna (2023) | 0.747 | 0.840 | 0.868 | 0.281 | 0.391 | 0.453 | 0.347 | 0.483 | 0.563 | 0.172 | 0.288 | 0.364
Llama-2-7b-CoH | 0.880 | 0.929 | 0.931 | 0.349 | 0.470 | 0.591 | 0.386 | 0.541 | 0.699 | 0.223 | 0.360 | 0.518

Table 4: Temporal forecasting with time-aware filtered metrics Hits@1, Hits@3
and Hits@10. The best results are highlighted in bold and the second-rank
results are underlined. The results of the model with $\dagger$ are derived
from Li et al. (2022), and results with $\ddagger$ are taken from Lee et al.
(2023b).

###  5.2 Baseline Models

The models selected for comparative analysis primarily fall into two
categories: embedding-based methods and LLM-based approaches. Within the realm
of embedding-based methods, we present the performance evaluations of RE-NET
Jin et al. (2020), RE-GCN Li et al. (2021a), TiRGN Li et al. (2022), xERTE Han
et al. (2021a), TANGO Han et al. (2021b), Timetraveler Sun et al. (2021) and
TiRGN Li et al. (2022). Regarding LLM-based approaches, to align with our
model settings and consider the choices made in prior studies Lee et al.
(2023b), we focus on the effects of 8-shot in-context learning for Llama-2-7b
Touvron et al. (2023), Vicuna-7b Vicuna (2023), and GPT-NeoX-20B Black et al.
(2022). In addition to these, we also include the rule-based method TLogic Liu
et al. (2022) in our comparison.

###  5.3 Evaluation Protocol

We acknowledge that, at the metric level, notable distinctions exist between
LLM-based methods and embedding-based approaches. The latter proves
advantageous as it can furnish a precise ranking of all entities in the graph
for a query presented in the form of $(s,q,?)$, facilitating the calculation
of metrics like Mean Reciprocal Rank Chao et al. (2021); Yu et al. (2022).
However, for LLM-based methods, we can only furnish the ranking of a
predetermined number of candidates, relying on the probabilities of output
paths from the open-source model Lee et al. (2023b). This is in contrast to
obtaining the ranking of all entities in the graph. This constraint stems from
the inability to compel the model to remember all entities directly, and it
introduces impractical search costs. Consequently, we choose to report
relatively accurate Hits@1, Hits@3, and Hits@10 Sun et al. (2019).
Furthermore, we align with the perspective outlined in Ding et al. (2021);
Jain et al. (2020) that directly excluding all other valid candidates to a
specific query in a filtering setting is not entirely reasonable.
Additionally, given that the proprietary LLMs we employ for comparison lack
the opportunities to output ranking lists, we report raw metrics without loss
of generality.

###  5.4 Main Results

As shown in Tbl. 3, Llama-2-7b-CoH achieves state-of-the-art performance in
multiple metrics under raw setting. Significantly, on the ICEWS14 and ICEWS18
datasets, Llama-2-7b-CoH shows an improvement of 5.0% and 6.3% in the Hits@1
metric compared to the current best models. We observe that on the YAGO
dataset, the 8-shot ICL performance of GPT-NeoX-20B, Llama-2-7b, and vicuna-7b
is not significantly worse than Llama-2-7b-CoH. However, there is a noticeable
gap on the ICEWS14 series datasets, even falling behind embedding-based
models. We also report the metrics under the time-aware filtered setting in
Tbl. 4, where Llama-2-7b-CoH outperforms the previous best-performing TiRGN
model by 4.1 percentage points in the Hits@1 and also exhibits a substantial
advantage on other datasets. The relative performance of the model remains
generally consistent under both settings.

##  6 Analysis

###  6.1 Effective Stucture-based Augmentation

To assess the efficacy of the structure-augmented history modeling strategy,
we conduct comprehensive ablation experiments on all used datasets, employing
Hits@1 as the evaluation criterion. For comparison, we exclude entity-
augmented and relation-augmented histories during both the fine-tuning and
inference phases, relying solely on schema-matching history for predictive
determination. The results of the ablation studies are depicted in Tbl. 5,
enabling a clear analysis that structure-augmented history is beneficial for
both forward and backward inference.

Illustrating with a practical case, when reasoning about the quadruple
(Economist (United Kingdom), Criticize or denounce, ?, 6960), due to schema-
matching history capturing only a historical fact (Economist (United Kingdom),
Criticize or denounce, Silvio Berlusconi, 120), this leads to an incorrect
inference of Afghanistan. However, the entity-augmented history contains
multiple instances of Economist (United Kingdom) linked through the Make
statement relation to United Kingdom. This similar behavior guides the model
to output the correct answer United Kingdom. Thus, supplementation enhances to
some extent the expression of structured information related to the central
node, thereby aiding LLM in making more accurate predictions beyond simply
relying on the ground truth history.

###  6.2 Effect of Introducing Reverse Logic

Datasets | ICEWS14 | ICEWS05-15 | ICEWS18 | YAGO
---|---|---|---|---
| Forward | Backward | Overall | Forward | Backward | Overall | Forward | Backward | Overall | Forward | Backward | Overall
Llama-2-7b-CoH w/o aug | 0.353 | 0.297 | 0.325 | 0.400 | 0.357 | 0.379 | 0.226 | 0.196 | 0.211 | 0.555 | 0.491 | 0.523
Llama-2-7b-CoH | 0.370 | 0.308 | 0.339 | 0.408 | 0.359 | 0.383 | 0.236 | 0.204 | 0.220 | 0.560 | 0.491 | 0.526
$\Delta$ | 4.8% | 3.7% | 4.3% | 2.0% | 0.6% | 1.1% | 4.4% | 4.1% | 4.3% | 0.9% | 0.0% | 0.6%

Table 5: Ablations on the structure-based history augmentation. We also report
Hits@1 metric. The strategy has achieved comprehensive improvement in bi-
directional forecasting.

Datasets | ICEWS14 | ICEWS05-15 | ICEWS18 | YAGO
---|---|---|---|---
| Forward | Backward | Overall | Forward | Backward | Overall | Forward | Backward | Overall | Forward | Backward | Overall
Llama-2-7b-CoH w/o rq | 0.367 | 0.298 | 0.333 | 0.396 | 0.343 | 0.369 | 0.238 | 0.188 | 0.213 | 0.560 | 0.489 | 0.524
Llama-2-7b-CoH | 0.370 | 0.308 | 0.339 | 0.408 | 0.359 | 0.383 | 0.236 | 0.204 | 0.220 | 0.560 | 0.491 | 0.526
$\Delta$ | 0.8% | 3.4% | 1.8% | 3.0% | 4.7% | 3.8% | 0.8% | 8.5% | 3.3% | 0.0% | 0.4% | 0.4%

Table 6: Ablations on the incorporation of reciprocal quadruples when fine-
tuning. We report Hits@1 on four datasets. Rising and falling trends are
indicated by green and red respectively. In order to more clearly observe
differences, we use historical chain with a length of 30 on the ICEWS18
dataset, while for other datasets, this value is set to 10.

We conduct a comprehensive ablation experiment for the introduction of reverse
quadruples in the fine-tuning phase. Considering the difficulty of ICEWS18
dataset, we set the length of the history chain to 30, and we set this value
to 10 on the other datasets. We use the ordinary prompt as a comparison to
verify the effect of the reverse data introduction. The results are
demonstrated in Tbl. 6, where Llama-2-7b-CoH (w/o rq) indicates that no
reverse quadruples are added during the fine-tuning phase. We can see that all
the results show an upward trend except for a slight dip in the forward
inference on the ICEWS18 dataset. Therefore, we can argue that the inclusion
of reverse logic in the fine-tuning stage is not only beneficial to alleviate
the curse of reversal in structured knowledge reasoning, but also largely
harmless to forward reasoning.

We still give a comparison of three proposed prompt styles in Tbl. 7. We
observe that ordinary and text-aware strategies always lead to better results,
so we believe that consistency in preserving the inflectional position of
different structured quadruples during fine-tuning is more critical.

Strategy | YAGO | ICEWS14 | ICEWS05-15 | ICEWS18
---|---|---|---|---
Ordinary | 0.526 | 0.339 | 0.383 | 0.209
Text-aware | 0.525 | 0.333 | 0.382 | 0.214
Position-aware | 0.525 | 0.330 | 0.381 | 0.213

Table 7: Overall Hits@1 metrics for three utilized prompt strategies under the
raw setting.

###  6.3 Exploration on History Length

The length of the historical chain $L$ significantly influences prediction
outcomes, reflecting the amount of information provided to the LLMs. We
conduct experiments with varying history lengths ($L=10,20,30,50$), while
maintaining other settings constant. We choose the ordinary prompt for
incorporating reverse quadruples and harness entity-augmented and relation-
augmented quadruples to enrich historical facts.

As illustrated in Fig. 2, except the ICEWS14 dataset, on other datasets, the
Hits@1 metric exhibits an upward trend followed by stabilization as $L$
increases. We calculate the average length of schema-matching history for each
query in the test sets of four datasets. For the ICEWS14 dataset, this value
is 30.05, significantly lower than the other datasets. On the ICEWS05-15
dataset, this value is 56.95. Consequently, an excessively long required
history length may negatively impact the reasoning of LLM due to interference
from numerous historical quadruples used for padding. However, even with a
smaller input cost (i.e., smaller $L$) on the ICEWS14 dataset, significant
effectiveness is already achievable.

Figure 2: The evolution pattern of the Hits@1 metric across four utilized
datasets concerning the history length $L$.

###  6.4 How Model Size Affects Results

Model | YAGO | ICEWS14 | ICEWS05-15 | ICEWS18
---|---|---|---|---
Llama-2-7b-CoH | 0.527 | 0.343 | 0.390 | 0.218
Llama-2-13b-CoH | 0.526 | 0.343 | 0.392 | 0.210

Table 8: Overall Hits@1 metrics on different model sizes.

Datasets | ICEWS14 | ICEWS05-15 |  ICEWS18 | YAGO
---|---|---|---|---
| Forward | Backward | Overall | Forward | Backward | Overall | Forward | Backward | Overall | Forward | Backward | Overall
GPT-3.5-turbo | 0.260 | 0.158 | 0.209 | 0.157 | 0.177 | 0.167 | 0.079 | 0.070 | 0.075 | 0.496 | 0.441 | 0.481
GPT-4 OpenAI (2023) | 0.298 | 0.233 | 0.266 | 0.293 | 0.260 | 0.277 | 0.096 | 0.092 | 0.094 | 0.510 | 0.484 | 0.497
Qwen-72B-Chat Bai et al. (2023) | 0.279 | 0.216 | 0.248 | 0.357 | 0.343 | 0.350 | 0.159 | 0.148 | 0.154 | 0.499 | 0.463 | 0.481

Table 9: The performance of some powerful commercial models on 1000 randomly
selected test samples in each dataset.

In this section, we explore how model size of LLMs affects performance in
TKGC. We choose Llama-2-13b as a comparison and consider leveraging history
chain with $L=20$, and both add inverse quadruples and structure-based
augmentation data for fine-tuning. The results, as shown in Tbl. 8, depict
that Llama-2-13b-CoH does not achieve a very significant improvement in
effectiveness. Specifically, Hits@1 on ICEWS18 dataset decreases by 3.7%
compared to Llama-2-7b-CoH. We speculate that in temporal reasoning tasks, the
model size of LLMs is not the dominant factor, but rather depends more on the
model input.

###  6.5 Performance of Commercial LLMs

In this section, we test the effectiveness of three powerful commercial LLMs
on the TKGC task, aiming to explore the performance differences after multi-
task instruction fine-tuning and Reinforcement Learning from Human Feedback
(RLHF). We provide the same 8-shot ICL prompt samples for each of the three
models on different datasets, as detailed in the appendix. For the test data,
we randomly select 1000 queries for forward reasoning and 1000 queries for
backward reasoning on each dataset. Since these models do not provide output
probabilities, we only present the most accurate exact match metric,
equivalent to the Hits@1 metric under the raw setting. After confirming that
there are no fine-tuning on TKGC task and related datasets in the available
technical reports OpenAI (2023); Bai et al. (2023), we consider this
comparison to be relatively fair.

The evaluation results are shown in Tbl. 9. Firstly, we can observe that
Qwen-72B-Chat is able to achieve performance comparable to or surpass GPT-4.
In contrast, the performance of GPT-3.5-turbo is not satisfactory. We are
currently observing that the few-shot capabilities of Qwen-72B-Chat on the
MMLU evaluation set are approaching those of GPT-4 and surpassing the
performance of GPT-3.5-turbo. This eliminates a significant bias in terms of
language tendency. On the other hand, we demonstrate that chat models,
carefully fine-tuned and applying RLHF, exhibit superior performance in TKGC
tasks. However, when we compare the results of Tbl. 9 and Tbl. 3, we can
observe that the 8-shot ICL capability of commercial LLMs is still
significantly lower on the ICEWS series dataset compared to the capabilities
of Llama-2-7b-CoH, while the difference is not substantial on the YAGO. This
is because YAGO is a dataset biased towards common knowledge, and therefore,
commercial LLMs may already be familiar with a considerable number of rules.
However, the reasoning in the ICEWS series news dataset emphasizes the
interaction and evolutionary information of nodes in the graph rather than
relying on textual features. This results in commercial LLMs underperforming
in ICL, as they struggle to effectively capture the evolutionary patterns
along historical chains and utilize augmented structure-based knowledge.

##  7 Conclusion

In this article, we model the TKGC task as a process of event generation on
the chain of historical facts, leveraging fine-tuning of LLMs to enhance their
ability to generate temporal events. We explore the capability of LLMs to
perceive graph structures through structure-based historical data augmentation
and introduce reverse quadruples into the fine-tuning process to alleviate the
curse of reversal problem. The majority of our results under both raw and
filtered settings surpass those of the current best models. Furthermore, we
conduct extensive ablation experiments to validate the effectiveness of data
augmentation strategies and the introduction of reverse logic. We also
investigate the impact of factors such as historical data length and model
size on the TKGC task. We find that longer historical awareness often assists
LLMs in making accurate inferences, but it gradually reaches saturation.
However, larger models do not necessarily provide greater benefits for
temporal structural knowledge reasoning. We also test the 8-shot ICL
performance of some powerful commercial models on this task, and the results
indicate that they still fall short of the ideal performance. We believe that
our analysis can offer more powerful tools and effective insights for LLMs in
performing tasks related to temporal knowledge graphs.

## References

  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.  _CoRR_ , abs/2309.16609.
  * Bapna et al. (2019) Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. 2019.  Simple, scalable adaptation for neural machine translation.
  * Berglund et al. (2023) Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023.  The reversal curse: Llms trained on "a is b" fail to learn "b is a".  _CoRR_ , abs/2309.12288.
  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  Gpt-neox-20b: An open-source autoregressive language model.  _CoRR_ , abs/2204.06745.
  * Cai et al. (2023) Borui Cai, Yong Xiang, Longxiang Gao, He Zhang, Yunfeng Li, and Jianxin Li. 2023.  Temporal knowledge graph completion: A survey.  In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China_ , pages 6545–6553. ijcai.org.
  * Chai et al. (2023) Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. 2023.  Graphllm: Boosting graph reasoning ability of large language model.
  * Chao et al. (2021) Linlin Chao, Jianshan He, Taifeng Wang, and Wei Chu. 2021.  Pairre: Knowledge graph embeddings via paired relation vectors.  In _ACL/IJCNLP(1)_ , pages 4360–4369.
  * Chen et al. (2023) Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, and Jiliang Tang. 2023.  Exploring the potential of large language models (llms) in learning on graphs.
  * Chung et al. (2014) Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _CoRR_ , abs/1412.3555.
  * Ding et al. (2021) Zifeng Ding, Zhen Han, Yunpu Ma, and Volker Tresp. 2021.  Temporal knowledge graph forecasting with neural ODE.  _CoRR_ , abs/2101.05151.
  * Feng et al. (2023) Chao Feng, Xinyu Zhang, and Zichu Fei. 2023.  Knowledge solver: Teaching llms to search for domain knowledge from knowledge graphs.  _CoRR_ , abs/2309.03118.
  * García-Durán et al. (2018) Alberto García-Durán, Sebastijan Dumancic, and Mathias Niepert. 2018.  Learning sequence encoders for temporal knowledge graph completion.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 4816–4821. Association for Computational Linguistics.
  * Goel et al. (2020) Rishab Goel, Seyed Mehran Kazemi, Marcus A. Brubaker, and Pascal Poupart. 2020.  Diachronic embedding for temporal knowledge graph completion.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 3988–3995. AAAI Press.
  * Guo et al. (2023) Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. 2023.  Gpt4graph: Can large language models understand graph structured data ? an empirical evaluation and benchmarking.
  * Han et al. (2021a) Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. 2021a.  Explainable subgraph reasoning for forecasting on temporal knowledge graphs.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Han et al. (2021b) Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp. 2021b.  Learning neural ordinary equations for forecasting future links on temporal knowledge graphs.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 8352–8364. Association for Computational Linguistics.
  * He et al. (2022) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022.  Towards a unified view of parameter-efficient transfer learning.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Jain et al. (2020) Prachi Jain, Sushant Rathi, Mausam, and Soumen Chakrabarti. 2020.  Temporal knowledge base completion: New algorithms and evaluation protocols.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 3733–3747. Association for Computational Linguistics.
  * Jin et al. (2020) Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. 2020.  Recurrent event network: Autoregressive structure inferenceover temporal knowledge graphs.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 6669–6683. Association for Computational Linguistics.
  * Lacroix et al. (2020) Timothée Lacroix, Guillaume Obozinski, and Nicolas Usunier. 2020.  Tensor decompositions for temporal knowledge base completion.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Leblay and Chekol (2018) Julien Leblay and Melisachew Wudage Chekol. 2018.  Deriving validity time in knowledge graph.  In _Companion Proceedings of the The Web Conference 2018_ , pages 1771–1776.
  * Lee et al. (2023a) Dong-Ho Lee, Kian Ahrabian, Woojeong Jin, Fred Morstatter, and Jay Pujara. 2023a.  Temporal knowledge graph forecasting without knowledge using in-context learning.  _arXiv preprint arXiv:2305.10613_.
  * Lee et al. (2023b) Dong-Ho Lee, Kian Ahrabian, Woojeong Jin, Fred Morstatter, and Jay Pujara. 2023b.  Temporal knowledge graph forecasting without knowledge using in-context learning.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pages 544–557. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Li et al. (2022) Yujia Li, Shiliang Sun, and Jing Zhao. 2022.  Tirgn: Time-guided recurrent graph network with local-global historical patterns for temporal knowledge graph reasoning.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022_ , pages 2152–2158. ijcai.org.
  * Li et al. (2021a) Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. 2021a.  Temporal knowledge graph reasoning based on evolutional representation learning.  In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_ , pages 408–417. ACM.
  * Li et al. (2021b) Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. 2021b.  Temporal knowledge graph reasoning based on evolutional representation learning.  In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_ , pages 408–417. ACM.
  * Liu et al. (2023) Pei Liu, Bing Qian, Qi Sun, and Longgang Zhao. 2023.  Prompt-wnqa: A prompt-based complex question answering for wireless network over knowledge graph.  _Comput. Networks_ , 236:110014.
  * Liu et al. (2022) Yushan Liu, Yunpu Ma, Marcel Hildebrandt, Mitchell Joblin, and Volker Tresp. 2022.  Tlogic: Temporal logical rules for explainable link forecasting on temporal knowledge graphs.  In _Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022_ , pages 4120–4127. AAAI Press.
  * Luo et al. (2023) Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2023.  Reasoning on graphs: Faithful and interpretable large language model reasoning.  _CoRR_ , abs/2310.01061.
  * Lv et al. (2023) Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, and Rui Yan. 2023.  Are we falling in a middle-intelligence trap? an analysis and mitigation of the reversal curse.  _CoRR_ , abs/2311.07468.
  * Mahdisoltani et al. (2015) Farzaneh Mahdisoltani, Joanna Biega, and Fabian M. Suchanek. 2015.  YAGO3: A knowledge base from multilingual wikipedias.  In _Seventh Biennial Conference on Innovative Data Systems Research, CIDR 2015, Asilomar, CA, USA, January 4-7, 2015, Online Proceedings_. www.cidrdb.org.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Qi et al. (2023) Chengwen Qi, Bowen Li, Binyuan Hui, Bailin Wang, Jinyang Li, Jinwang Wu, and Yuanjun Laili. 2023.  An investigation of llms’ inefficacy in understanding converse relations.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pages 6932–6953. Association for Computational Linguistics.
  * Qin et al. (2024) Yingrong Qin, Chen Gao, Shuangqing Wei, Yue Wang, Depeng Jin, Jian Yuan, Lin Zhang, Dong Li, Jianye Hao, and Yong Li. 2024.  Learning from hierarchical structure of knowledge graph for recommendation.  _ACM Trans. Inf. Syst._ , 42(1):18:1–18:24.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017.  Learning multiple visual domains with residual adapters.
  * Shao et al. (2022) Pengpeng Shao, Dawei Zhang, Guohua Yang, Jianhua Tao, Feihu Che, and Tong Liu. 2022.  Tucker decomposition-based temporal knowledge graph completion.  _Knowl. Based Syst._ , 238:107841.
  * Sun et al. (2022) Haohai Sun, Shangyi Geng, Jialun Zhong, Han Hu, and Kun He. 2022.  Graph hawkes transformer for extrapolated reasoning on temporal knowledge graphs.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 7481–7493. Association for Computational Linguistics.
  * Sun et al. (2021) Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. 2021.  Timetraveler: Reinforcement learning for temporal knowledge graph forecasting.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 8306–8319. Association for Computational Linguistics.
  * Sun et al. (2023) Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Heung-Yeung Shum, and Jian Guo. 2023.  Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph.  _CoRR_ , abs/2307.07697.
  * Sun et al. (2019) Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019.  Rotate: Knowledge graph embedding by relational rotation in complex space.  In _ICLR_.
  * Tang et al. (2023) Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. 2023.  Graphgpt: Graph instruction tuning for large language models.  _CoRR_ , abs/2310.13023.
  * Tian et al. (2023) Yijun Tian, Huan Song, Zichen Wang, Haozhu Wang, Ziqing Hu, Fang Wang, Nitesh V. Chawla, and Panpan Xu. 2023.  Graph neural prompting with large language models.  _CoRR_ , abs/2309.15427.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _CoRR_ , abs/2302.13971.
  * Trivedi et al. (2017) Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. 2017.  Know-evolve: Deep temporal reasoning for dynamic knowledge graphs.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3462–3471. PMLR.
  * Vicuna (2023) Vicuna. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  https://vicuna.lmsys.org/.
  * Wang et al. (2022a) Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022a.  Simkgc: Simple contrastive knowledge graph completion with pre-trained language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 4281–4294. Association for Computational Linguistics.
  * Wang et al. (2022b) Xintao Wang, Qianyu He, Jiaqing Liang, and Yanghua Xiao. 2022b.  Language models as knowledge embeddings.  _CoRR_ , abs/2206.12617.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Xiong et al. (2018) Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2018.  One-shot relational learning for knowledge graphs.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 1980–1990. Association for Computational Linguistics.
  * Yu et al. (2022) Long Yu, Zhicong Luo, Huanyong Liu, Deng Lin, Hongzhu Li, and Yafeng Deng. 2022.  Triplere: Knowledge graph embeddings via tripled relation vectors.  _CoRR_ , abs/2209.08271.
  * Zhu et al. (2021) Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan Cheng, and Yan Zhang. 2021.  Learning from history: Modeling temporal knowledge graphs with sequential copy-generation networks.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 4732–4740. AAAI Press.

##  Appendix A Prompt for TKGC

In this section, we provide a comprehensive design for the prompt, including
versions that utilize only entity text (Tbl. 10) and versions identified by
number id (Tbl. 11).

Section |  Prompt
---|---
Instruction |  Given contexts consisting of multiple quadruplets in the form of {time}: [{subject}, {relation}, {object}], please predict the missing entity in the query quadruplet {time}: [{subject}, {relation}, ] in the end.
Input |  295: [[Victor_Ponta, Make_statement, Romania]
|  296: [Victor_Ponta, Make_statement, North_Atlantic_Treaty_Organization]
|  296: [Victor_Ponta, Make_statement, Romania]
|  300: [Victor_Ponta, Make_statement, Viorel_Hrebenciuc]
|  301: [Victor_Ponta, Make_statement, Romania]
|  302: [Victor_Ponta, Make_statement, Romania]
|  303: [Victor_Ponta, Make_statement, National_Liberal_Party_(Romania)]
|  303: [Victor_Ponta, Make_statement, Romania]
|  304: [Victor_Ponta, Make_statement, Romania]
|  307: [Victor_Ponta, Make_statement, Representatives_(Romania)]
|  Query:
|  308: [Victor_Ponta, Make_statement, ]
Output |  The missing entity of query quadruplet is Romania.

Table 10: Prompt design using text only.

Section |  Prompt
---|---
Instruction |  Given contexts consisting of multiple quadruplets in the form of {time}: [{subject}, {relation}, {object}], please predict the missing entity in the query quadruplet {time}: [{subject}, {relation}, ] in the end.
Input |  295: [[Victor_Ponta, Make_statement, 0.Romania]
|  296: [Victor_Ponta, Make_statement, 1.North_Atlantic_Treaty_Organization]
|  296: [Victor_Ponta, Make_statement, 0.Romania]
|  300: [Victor_Ponta, Make_statement, 2.Viorel_Hrebenciuc]
|  301: [Victor_Ponta, Make_statement, 0.Romania]
|  302: [Victor_Ponta, Make_statement, 0.Romania]
|  303: [Victor_Ponta, Make_statement, 3.National_Liberal_Party_(Romania)]
|  303: [Victor_Ponta, Make_statement, 0.Romania]
|  304: [Victor_Ponta, Make_statement, 0.Romania]
|  307: [Victor_Ponta, Make_statement, 4.Representatives_(Romania)]
|  Query:
|  308: [Victor_Ponta, Make_statement, ]
Output |  The missing entity of query quadruplet is 0.Romania.

Table 11: Prompt design using text and id.