# Lifelong Embedding Learning and Transfer for Growing Knowledge Graphs

Yuanning Cui1, Yuxin Wang1, Zequn Sun1, Wenqiang Liu3, Yiqiao Jiang3, Kexin
Han3, Wei Hu1,2 Corresponding author

###### Abstract

Existing knowledge graph (KG) embedding models have primarily focused on
static KGs. However, real-world KGs do not remain static, but rather evolve
and grow in tandem with the development of KG applications. Consequently, new
facts and previously unseen entities and relations continually emerge,
necessitating an embedding model that can quickly learn and transfer new
knowledge through growth. Motivated by this, we delve into an expanding field
of KG embedding in this paper, i.e., lifelong KG embedding. We consider
knowledge transfer and retention of the learning on growing snapshots of a KG
without having to learn embeddings from scratch. The proposed model includes a
masked KG autoencoder for embedding learning and update, with an embedding
transfer strategy to inject the learned knowledge into the new entity and
relation embeddings, and an embedding regularization method to avoid
catastrophic forgetting. To investigate the impacts of different aspects of KG
growth, we construct four datasets to evaluate the performance of lifelong KG
embedding. Experimental results show that the proposed model outperforms the
state-of-the-art inductive and lifelong embedding baselines.

##  1 Introduction

Many knowledge-driven applications are built on knowledge graphs (KGs), which
store massive amounts of structured facts about the real world (Ji et al.
2022). Throughout the life-cycle of KG construction and application, new
facts, unseen entities, and unseen relations continually emerge into the KG on
a regular basis. As a result, the real-world KG is rarely a static graph, but
rather evolves and grows alongside the development. Figure 1 illustrates an
example excerpted from Wikidata (Vrandecic and Krötzsch 2014), which shows the
growth of the KG along with the continuous knowledge extraction. However, KG
embedding, a critical task for downstream applications, has primarily focused
on static KGs over the years (Wang et al. 2017). Learning from scratch every
time is inefficient and wastes previously acquired knowledge, and simply fine-
tuning new facts would quickly disrupt previously acquired knowledge. Hence,
this paper proposes to investigate lifelong embedding learning and transfer
for growing KGs, with the goal of learning new facts while retaining old
knowledge without re-training from scratch.

The key idea of this paper comes from the human learning process. Humans are
typical lifelong learners, with knowledge transfer and retention being the
most important aspects of lifelong learning. Humans, in particular, can
continually learn new knowledge given new facts and use previously learned
knowledge to help new knowledge learning (knowledge learning and transfer), as
well as update old knowledge while retaining useful knowledge (knowledge
update and retention). Motivated by this, we seek to build a lifelong KG
embedding model, namely LKGE, which is capable of learning, transferring and
retaining knowledge for growing KGs efficiently. Existing related work, such
as inductive KG embedding (Hamaguchi et al. 2017; Wang et al. 2019b), mainly
focuses on knowledge transfer, ignoring new knowledge learning and old
knowledge update.

Figure 1: An example of a growing KG. In each snapshot $i$, new facts are
added into the KG, and previously unseen entities and relations emerge with
the new facts.

The proposed lifelong KG embedding task faces two major challenges. First, how
to strike a balance between new knowledge learning and old knowledge transfer?
Learning embeddings for new entities and relations from scratch cannot
leverage previously learned knowledge, and inductively generating embeddings
for them ignores new knowledge in the new snapshot. Second, how to update old
knowledge while retaining useful knowledge? Learning new facts about an old
entity usually requires updating the previously learned embeddings, which can
be harmful to the old model. This is because updating an old entity embedding
would affect many other old embeddings of related entities. This would cause
the catastrophic forgetting issue, and therefore affect the applications built
on the old KG snapshot.

To resolve the above challenges, we propose three solutions in our LKGE model.
First, as the base embedding model for new knowledge learning and old
knowledge update, we design a masked KG autoencoder that masks and
reconstructs the entities or relations in new facts. It builds connections
between locally-related old and new entities, acting as a bridge for knowledge
transfer. Second, to aid in the learning of new knowledge, we propose a
knowledge embedding transfer strategy that uses previously learned knowledge
to initialize the embeddings of new entities and relations. These embeddings
are used by the KG autoencoder to learn new facts. Third, to avoid
catastrophic forgetting in the old knowledge update, we propose embedding
regularization to balance the learning of new facts and the update of old
embeddings.

We build four datasets to assess the lifelong KG embedding performance,
including the entity-centric, relation-centric, fact-centric, and hybrid
growth. Each dataset examines a different aspect of KG growth. By contrast,
existing datasets (Hamaguchi et al. 2017; Kou et al. 2020; Daruna et al. 2021)
all assume that a KG grows in an ideal way, with balanced new entities or
facts in each new snapshot. In our experiments, we compare the link prediction
accuracy, knowledge transfer ability, and learning efficiency of the proposed
model against baselines on the four datasets. The results show that the
proposed LKGE not only achieves the best performance on the four datasets, but
also has the best forward knowledge transfer ability and learning efficiency.
The main contributions of this paper are summarized as follows:

  * •

We study lifelong KG embedding learning and transfer. It is a practical task
since the real-world KGs continually evolve and grow, which requires the
embedding model to be capable of handling the knowledge growth.

  * •

We propose a novel lifelong learning model, LKGE. It includes a masked KG
autoencoder as the basis of embedding learning and update, an embedding
transfer strategy for knowledge transfer, and an embedding regularization to
prevent catastrophic forgetting in knowledge update.

  * •

We conduct extensive experiments on four new datasets. The results demonstrate
the effectiveness and efficiency of LKGE against a variety of state-of-the-art
models.

##  2 Related Work

###  2.1 Knowledge Graph Embedding

KG embedding seeks to encode the symbolic representations of KGs into vector
space to foster the application of KGs in downstream tasks. Most existing KG
embedding models (Bordes et al. 2013; Wang et al. 2014; Dettmers et al. 2018;
Schlichtkrull et al. 2018; Guo, Sun, and Hu 2019; Vashishth et al. 2020a, b)
focus on static graphs and cannot continually learn new knowledge on the
growing KGs.

To embed unseen entities, inductive KG embedding models learn to represent an
entity by aggregating its existing neighbors in the previous KG snapshot. MEAN
(Hamaguchi et al. 2017) uses a graph convolutional network (GCN) (Scarselli et
al. 2009) for neighborhood aggregation. When an unseen entity emerges, the GCN
would aggregate its previously seen neighboring entities to generate an
embedding. LAN (Wang et al. 2019b) adopts an attention mechanism to
attentively aggregate different neighbors. As MEAN and LAN rely on the entity
neighborhood for embedding learning, they cannot handle the new entities that
have no neighbors in the previous snapshot. Furthermore, inductive KG
embedding disregards learning the facts about new entities.

Our work is also relevant to dynamic KG embedding. puTransE (Tay, Luu, and Hui
2017) trains several new models when facts are added. DKGE (Wu et al. 2022)
learns contextual embeddings for entities and relations, which can be
automatically updated as the KG grows. They both need partial re-training on
old facts, but our model does not. In addition, some subgraph-based models,
such as GraIL (Teru, Denis, and Hamilton 2020), INDIGO (Liu et al. 2021), and
TACT (Chen et al. 2021), can also represent unseen entities using the entity-
independent features and subgraph aggregation. Their subgraph-building process
is time-consuming, making them only applicable to small KGs. In order to run
on large-scale KGs, NBFNet (Zhu et al. 2021) proposes a fast node pair
embedding model based on the Bellman-Ford algorithm, and NodePiece (Galkin et
al. 2022) uses tokenized anchor nodes and relational paths to represent new
entities. However, they do not consider learning new knowledge and cannot
support new relations.

###  2.2 Lifelong Learning

Lifelong learning seeks to solve new problems quickly without catastrophically
forgetting previously acquired knowledge. Lifelong learning models are broadly
classified into three categories. (i) Dynamic architecture models (Rusu et al.
2016; Lomonaco and Maltoni 2017) extend the network to learn new tasks and
avoid forgetting acquired knowledge. (ii) Regularization-based models
(Kirkpatrick et al. 2017; Zenke, Poole, and Ganguli 2017) capture the
importance of model parameters for old tasks and limit the update of important
parameters. (iii) Rehearsal-based models (Lopez-Paz and Ranzato 2017; Wang et
al. 2019a) memorize some data from old tasks and replay them when learning new
knowledge.

Few lifelong learning models focus on KG embedding. DiCGRL (Kou et al. 2020)
is a disentangle-based lifelong graph embedding model. It splits node
embeddings into different components and replays related historical facts to
avoid catastrophic forgetting. The work (Daruna et al. 2021) combines class-
incremental learning models with TransE (Bordes et al. 2013) for continual KG
embedding. However, it does not propose a specific lifelong KG embedding
model.

##  3 Lifelong Knowledge Graph Embedding

Figure 2: Overview of the proposed model for lifelong KG embedding.

###  3.1 Preliminaries

Growing KG. The growth process of a KG yields a snapshot sequence, i.e.,
$\mathcal{G}=\\{\mathcal{S}_{1},\mathcal{S}_{2},\ldots,\mathcal{S}_{t}\\}$.
Each snapshot $\mathcal{S}_{i}$ is defined as a triplet
$(\mathcal{T}_{i},\mathcal{E}_{i},\mathcal{R}_{i})$, where
$\mathcal{T}_{i},\mathcal{E}_{i}$ and $\mathcal{R}_{i}$ denote the fact,
entity and relation sets, respectively. We have
$\mathcal{T}_{i}\subseteq\mathcal{T}_{i+1}$,
$\mathcal{E}_{i}\subseteq\mathcal{E}_{i+1}$ and
$\mathcal{R}_{i}\subseteq\mathcal{R}_{i+1}$. We use $\mathcal{T}_{\Delta
i}=\mathcal{T}_{i}-\mathcal{T}_{i-1}$, $\mathcal{E}_{\Delta
i}=\mathcal{E}_{i}-\mathcal{E}_{i-1}$, and $\mathcal{R}_{\Delta
i}=\mathcal{R}_{i}-\mathcal{R}_{i-1}$ to denote the new facts, entities and
relations, respectively. Each fact is in the form of
$(s,r,o)\in\mathcal{T}_{i}$, where $s,o\in\mathcal{E}_{i}$ are the subject and
object entities, respectively, and $r\in\mathcal{R}_{i}$ is their relation.

Lifelong KG embedding. KG embedding seeks to encode the symbolic
representations of entities and relations into vector space and capture KG
semantics using vector operations. For a growing KG, a lifelong KG embedding
model learns to represent the snapshot sequence
$\mathcal{G}=\\{\mathcal{S}_{1},\mathcal{S}_{2},\ldots,\mathcal{S}_{t}\\}$
continually. When a new fact set $\mathcal{T}_{\Delta i}$ emerges, the current
KG embedding model $\mathcal{M}_{i-1}$ needs update to fit the new facts and
learn embeddings for the new entities $\mathcal{E}_{\Delta i}$ and new
relations $\mathcal{R}_{\Delta i}$. The resulting model is denoted by
$\mathcal{M}_{i}$.

Lifelong link prediction. The link prediction task asks the KG embedding model
to predict the missing subject or object entity in an incomplete fact like
$(s,r,?)$ or $(?,r,o)$. For each snapshot $\mathcal{S}_{i}$, the new fact set
$\mathcal{T}_{\Delta i}$ is divided into a training set $\mathcal{D}_{i}$, a
validation set $\mathcal{V}_{i}$ and a test set $\mathcal{Q}_{i}$. In the
lifelong setting, the model is required to learn the training data sets,
$\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{t}$, in turn. After
finishing the learning on $\mathcal{D}_{i}$, the model is evaluated on the
accumulated test data, which is $\cup_{j=1}^{i}\mathcal{Q}_{j}$, to assess the
overall learning performance. After learning $\mathcal{D}_{i}$, then
$\mathcal{D}_{i},\mathcal{V}_{i}$ would no longer be available for the
following learning. Note that the goal of lifelong KG embedding is to improve
the overall performance on all snapshots, which requires the KG embedding
model to continually learn new knowledge and retain the learned knowledge.

###  3.2 Model Overview

The overview of our model, LKGE, is shown in Figure 2. It continually learns
knowledge over a sequence of KG snapshots without re-training on previously
seen data. The foundation is a masked KG autoencoder that can reconstruct the
entity and relation embeddings from the masked related subgraph. To enable
knowledge transfer from old snapshots to the new one, we propose embedding
transfer to inject learned knowledge into unseen entities and relations and
then iteratively optimize the model to fit the new facts. We also propose a
lightweight regularization method to retain old knowledge.

###  3.3 Masked Knowledge Graph Autoencoder

In lifelong KG embedding learning, the new facts, e.g., $\mathcal{T}_{\Delta
i}$, bring new entities $\mathcal{E}_{\Delta i}$, and new relations
$\mathcal{R}_{\Delta i}$. It would also involve some old entities from
$\mathcal{E}_{i-1}$, and old relations from $\mathcal{R}_{i-1}$. Thus, a new
entity may be connected with both new and old entities (referred to as new
knowledge). The involved old entities also receive more facts, and therefore
their previously learned embeddings (referred to as old knowledge) need to be
updated to fit the new facts. Hence, the base KG embedding model for lifelong
learning should be capable of capturing new knowledge as well as updating old
knowledge. To this end, we propose a masked KG autoencoder motivated by the
recent success of self-supervised learning (Hou et al. 2022). The key idea is
to reconstruct the embedding for an entity or relation based on its masked
subgraph, which may include both other new entities and some old entities.
Specifically, we use the first-order subgraph of an entity or relation to
reconstruct its embedding $\bar{\mathbf{x}}_{i}$:

$\bar{\mathbf{x}}_{i}=\text{MAE}\big{(}\cup_{j=1}^{i}\mathcal{N}_{j}(x)\big{)},$
(1)

where $x$ denotes either an entity or a relation, and
$\mathcal{N}_{j}\subseteq\mathcal{D}_{j}$ denotes the involved facts of $x$ in
the $j$-th snapshot. $\text{MAE}()$ is an encoder to represent the input
subgraph. The objective of our KG encoder is to align the entity or relation
embedding with the reconstructed representation as follows:

$\mathcal{L}_{\text{MAE}}=\sum_{e\in\mathcal{E}_{i}}\|\mathbf{e}_{i}-\bar{\mathbf{e}}_{i}\|_{2}^{2}+\sum_{r\in\mathcal{R}_{i}}\|\mathbf{r}_{i}-\bar{\mathbf{r}}_{i}\|_{2}^{2}.$
(2)

The key then becomes how to design an effective and efficient encoder for
lifelong learning. GCN (Kipf and Welling 2017) and Transformer (Vaswani et al.
2017) are two common choices for the encoder. The two encoders both introduce
additional model parameters (e.g., the weight matrices). In our lifelong
learning setting, the encoder needs to be updated to fit new facts or
subgraphs. In this case, once the GCN or Transformer is updated, the changed
model parameters would affect the embedding generation of _all_ old entities
(not just the involved old entities in new facts), increasing the risk of
catastrophically forgetting previous snapshots. To avoid this issue, we use
the entity and relation embedding transition functions as encoders, which do
not introduce additional parameters. We borrow the idea of TransE (Bordes et
al. 2013) and interpret a relation embedding as the translation vector between
the subject and object entity embeddings, i.e.,
$\mathbf{s}+\mathbf{r}\approx\mathbf{o}$, where
$\mathbf{s},\mathbf{r},\mathbf{o}$ denote the embeddings of subject entity,
relation and object entity, respectively. Based on this, we can deduce two
transition functions for entity and relation embeddings. The subject entity of
$(s,r,o)$ can be represented by
$f_{\rm{sub}}(\mathbf{r},\mathbf{o})=\mathbf{o}-\mathbf{r}$, and the relation
embedding is $f_{\rm{rel}}(\mathbf{s},\mathbf{o})=\mathbf{o}-\mathbf{s}$. We
can define the encoders as

$\displaystyle\bar{\mathbf{e}}_{i}$
$\displaystyle=\frac{\sum_{j=1}^{i}\sum_{(s,r,o)\in\mathcal{N}_{j}(e)}f_{\rm{sub}}(\mathbf{r}_{i},\mathbf{o}_{i})}{\sum_{j=1}^{i}|\mathcal{N}_{j}(e)|},$
(3) $\displaystyle\bar{\mathbf{r}}_{i}$
$\displaystyle=\frac{\sum_{j=1}^{i}\sum_{(s,r,o)\in\mathcal{N}_{j}(r)}f_{\rm{rel}}(\mathbf{s}_{i},\mathbf{o}_{i})}{\sum_{j=1}^{i}|\mathcal{N}_{j}(r)|},$
(4)

where $\mathbf{s}_{i},\mathbf{r}_{i},\mathbf{o}_{i}$ are the embeddings of
$s,r,o$ during the training on $\mathcal{D}_{i}$.
$\mathcal{N}_{j}(x)\subseteq\mathcal{D}_{j}$ is the set of facts containing
$x$.

In lifelong learning, the model learns from a snapshot sequence. Eqs. (3) and
(4) require training samples from the first $i$ snapshots, which are not in
line with lifelong learning. To reduce the reliance on learned data, we use
$\mathbf{e}_{i-1}$ and $\mathbf{r}_{i-1}$ as the approximate average
embeddings of $e$ and $r$ in the first $i-1$ snapshots, respectively, and
rewrite the encoders as

$\bar{\mathbf{e}}_{i}\approx\frac{\sum_{j=1}^{i-1}|\mathcal{N}_{j}(e)|\mathbf{e}_{i-1}+\sum_{(s,r,o)\in\mathcal{N}_{i}(e)}f_{\rm{sub}}(\mathbf{r}_{i},\mathbf{o}_{i})}{\sum_{j=1}^{i-1}|\mathcal{N}_{j}(e)|+|\mathcal{N}_{i}(e)|},$

(5)

$\bar{\mathbf{r}}_{i}\approx\frac{\sum_{j=1}^{i-1}|\mathcal{N}_{j}(r)|\mathbf{r}_{i-1}+\sum_{(s,r,o)\in\mathcal{N}_{i}(r)}f_{\rm{rel}}(\mathbf{s}_{i},\mathbf{o}_{i})}{\sum_{j=1}^{i-1}|\mathcal{N}_{j}(r)|+|\mathcal{N}_{i}(r)|}.$

(6)

The encoders use the facts involving both old and new entities and relations
for embedding reconstruction and they build a bridge for knowledge transfer.

For each snapshot, to learn the knowledge from the new data and update the
learned parameters, we leverage TransE (Bordes et al. 2013) to train the
embedding model:

$\mathcal{L}_{\text{new}}=\sum\limits_{(s,r,o)\in\mathcal{D}_{i}}\max\big{(}0,\gamma+f(\mathbf{s},\mathbf{r},\mathbf{o})-f(\mathbf{s}^{\prime},\mathbf{r},\mathbf{o}^{\prime})\big{)},$

(7)

where $\gamma$ is the margin.
$(\mathbf{s}^{\prime},\mathbf{r},\mathbf{o}^{\prime})$ is the embedding of a
negative fact. For each positive fact, we randomly replace the subject or
object entity with a random entity $e^{\prime}\in\mathcal{E}_{i}$.

###  3.4 Embedding Transfer

During the lifecycle of a growing KG, there are abundant unseen entities and
some unseen relations emerge with the new facts. Learning effective embeddings
for them is an essential aspect of lifelong KG embedding learning. However,
these unseen ones are not included in any learned snapshots, so only
inheriting the learned parameters cannot transfer the acquired knowledge to
their embeddings. To avoid learning from scratch, we propose embedding
transfer that seeks to leverage the learned embeddings to help represent
unseen entities and relations. Specifically, we initialize the embeddings of
each unseen entity by aggregating its facts:

$\mathbf{e}_{i}=\frac{1}{|\mathcal{N}_{i}(e)|}\sum_{(e,r,o)\in\mathcal{N}_{i}(e)}f_{\rm{sub}}(\mathbf{r}_{i-1},\mathbf{o}_{i-1}),$
(8)

where $\mathcal{N}_{i}(e)\subseteq\mathcal{D}_{i}$ is the set of facts
containing $e$. For the new entities that do not have common facts involving
existing entities, we randomly initialize their embeddings. We also use this
strategy to initialize the embeddings of unseen relations:

$\mathbf{r}_{i}=\frac{1}{|\mathcal{N}_{i}(r)|}\sum_{(s,r,o)\in\mathcal{N}_{i}(r)}f_{\rm{rel}}(\mathbf{s}_{i-1},\mathbf{o}_{i-1}),$
(9)

where $\mathcal{N}_{i}(r)\subseteq\mathcal{D}_{i}$ is the set of facts
containing $r$.

###  3.5 Embedding Regularization

Learning new snapshots is likely to overwrite the learned knowledge from old
snapshots. To avoid catastrophic forgetting, some regularization methods
(Kirkpatrick et al. 2017; Lopez-Paz and Ranzato 2017) constrain the updates of
parameters that are important to old tasks. The loss function of
regularization methods is

$\mathcal{L}_{\text{old}}=\sum\limits_{e\in\mathcal{E}_{i-1}}\omega(e)\|\mathbf{e}_{i}-\mathbf{e}_{i-1}\|_{2}^{2}+\sum\limits_{r\in\mathcal{R}_{i-1}}\omega(r)\|\mathbf{r}_{i}-\mathbf{r}_{i-1}\|_{2}^{2},$

(10)

where $\omega(x)$ is the regularization weight for $x$.

Conventional regularization-based methods for classification tasks such as
(Kirkpatrick et al. 2017) model the importance of each parameter at a high
cost based on the gradient or parameter change during training. This problem
is even more severe for KG embedding models that have a large number of
embedding parameters (i.e. entity and relation embeddings). To resolve this
problem, we propose a lightweight embedding regularization method, which
calculates the regularization weight of each entity or relation by the number
of new and old facts containing it:

$\omega(x)=1-\frac{|\mathcal{N}_{i}(x)|}{\sum_{j=1}^{i}|\mathcal{N}_{j}(x)|}.$
(11)

As a lightweight technique, it only keeps the total number of involved trained
facts for each entity or relation and only updates regularization weights once
per snapshot.

###  3.6 Overall Learning Objective

To learn new knowledge while retaining acquired knowledge, the overall
lifelong learning objective $\mathcal{L}$ is defined as follows:

$\mathcal{L}=\mathcal{L}_{\text{new}}+\alpha\,\mathcal{L}_{\text{old}}+\beta\,\mathcal{L}_{\text{MAE}},$
(12)

where $\alpha,\beta$ are hyperparameters for balancing the objectives.

###  3.7 Complexity Analysis

Compared with fine-tuning, the proposed model requires few extra resources. It
does not increase the size of training samples like the rehearsal models (Wang
et al. 2019a; Lopez-Paz and Ranzato 2017; Kou et al. 2020). In addition to
fine-tuning, the proposed model calculates the loss of masked autoencoder and
embedding regularization. The additional time complexity in each iteration is
$O(|\mathcal{E}|+|\mathcal{R}|+|\mathcal{D}|)$. In practice, we find that the
loss of autoencoder can accelerate learning, and its time consumption is close
to that of fine-tuning. The space complexity of fine-tuning is
$O((|\mathcal{E}|+|\mathcal{R}|)\times d)$, and the space complexity of the
proposed model is $O((|\mathcal{E}|+|\mathcal{R}|)\times(d+1))$, where $d$ is
the dimension of embeddings.

Datasets | Snapshot 1 | Snapshot 2 | Snapshot 3 | Snapshot 4 | Snapshot 5
---|---|---|---|---|---
$|\mathcal{T}_{\Delta 1}|$ | $|\mathcal{E}_{1}|$ | $|\mathcal{R}_{1}|$ | $|\mathcal{T}_{\Delta 2}|$ | $|\mathcal{E}_{2}|$ | $|\mathcal{R}_{2}|$ | $|\mathcal{T}_{\Delta 3}|$ | $|\mathcal{E}_{3}|$ | $|\mathcal{R}_{3}|$ | $|\mathcal{T}_{\Delta 4}|$ | $|\mathcal{E}_{4}|$ | $|\mathcal{R}_{4}|$ | $|\mathcal{T}_{\Delta 5}|$ | $|\mathcal{E}_{5}|$ | $|\mathcal{R}_{5}|$
Entity | 46,388 |  2,909 | 233 | 72,111 |  5,817 | 236 | 73,785 |  8,275 | 236 |  70,506 | 11,633 | 237 | 47,326 | 14,541 | 237
Relation | 98,819 | 11,560 |  48 | 93,535 | 13,343 |  96 | 66,136 | 13,754 | 143 |  30,032 | 14,387 | 190 | 21,594 | 14,541 | 237
Fact | 62,024 | 10,513 | 237 | 62,023 | 12,779 | 237 | 62,023 | 13,586 | 237 |  62,023 | 13,894 | 237 | 62,023 | 14,541 | 237
Hybrid | 57,561 |  8,628 |  86 | 20,873 | 10,040 | 102 | 88,017 | 12,779 | 151 | 103,339 | 14,393 | 209 | 40,326 | 14,541 | 237

Table 1: Statistical data of the four constructed growing KG datasets. For the
$i$-th snapshot, $\mathcal{T}_{\Delta i}$ denotes the set of new facts in this
snapshot, and $\mathcal{E}_{i},\mathcal{R}_{i}$ denote the sets of cumulative
entities and relations in the first $i$ snapshots, respectively.

##  4 Dataset Construction

To simulate a variety of aspects of KG growth, we create four datasets based
on FB15K-237 (Toutanova and Chen 2015), which are entity-centric, relation-
centric, fact-centric, and hybrid. We denote them by Entity, Relation, Fact
and Hybrid, respectively. Given a KG
$\mathcal{G}=\\{\mathcal{E},\mathcal{R},\mathcal{T}\\}$, we construct five
snapshots with the following steps:

  1. 1.

Seeding. We randomly sample 10 facts from $\mathcal{T}$ and add them into
$\mathcal{T}_{1}$ for initialization. The entities and relations in the 10
facts form the initial $\mathcal{E}_{1}$ and $\mathcal{R}_{1}$, respectively.

  2. 2.

Expanding. To build Entity, Relation and Fact, we iteratively sample a fact
containing at least one seen entity in $\mathcal{E}_{i}$, add it into
$\mathcal{T}_{i}$, and extract the unseen entity and relation from it to
expand $\mathcal{E}_{i}$ and $\mathcal{R}_{i}$. For Entity, once
$|\mathcal{E}_{i}|\geq\frac{i+1}{5}|\mathcal{E}|$, we add all new facts
$\big{\\{}(s,r,o)\,|\,s\in\mathcal{E}_{i}\wedge o\in\mathcal{E}_{i}\big{\\}}$
into $\mathcal{T}_{i}$ and start building the next snapshot. In the same way,
we construct Relation and Fact. As for Hybrid, we uniformly sample an entity,
relation or fact without replacement from
$\mathcal{U}=\mathcal{E}\cup\mathcal{R}\cup\mathcal{T}$ to join
$\mathcal{E}_{i}$, $\mathcal{R}_{i}$ and $\mathcal{T}_{i}$. Note that when the
sampled fact contains an unseen entity or relation, we re-sample a fact that
only contains seen entities and relations to replace it. After each iteration,
we terminate the expansion of this snapshot with a probability
$\frac{5}{|\mathcal{U}|}$. Consequently, the expansion of Hybrid is uneven,
making it more realistic and challenging. For all datasets, we take the whole
KG as the last snapshot, i.e., $\mathcal{T}_{5}=\mathcal{T}$, and
$\mathcal{E}_{5}=\mathcal{E},\mathcal{R}_{5}=\mathcal{R}$.

  3. 3.

Dividing. For each snapshot, we randomly divide the new fact set
$\mathcal{T}_{\Delta i}$ into a training set $\mathcal{D}_{i}$, a validation
set $\mathcal{V}_{i}$ and a test set $\mathcal{Q}_{i}$ by a split ratio of
3:1:1.

The statistics of the four datasets are presented in Table 1.

##  5 Experimental Results

We conduct experiments regarding link prediction accuracy, knowledge transfer
capability, and learning efficiency to validate the proposed model, LKGE. The
datasets and source code are available at https://github.com/nju-websoft/LKGE.

###  5.1 Experiment Settings

Competitors. We compare our model with 12 competitors, including (i) three
baseline models: snapshot only, re-training, and fine-tuning; (ii) two
inductive models: MEAN (Hamaguchi et al. 2017), and LAN (Wang et al. 2019b);
(iii) two dynamic architecture models: PNN (Rusu et al. 2016), and CWR
(Lomonaco and Maltoni 2017); (iv) two regularization-based models: SI (Zenke,
Poole, and Ganguli 2017), and EWC (Kirkpatrick et al. 2017); and (v) three
rehearsal-based models: GEM (Lopez-Paz and Ranzato 2017), EMR (Wang et al.
2019a), and DiCGRL (Kou et al. 2020).

Evaluation metrics. Following the convention, we conduct the experiments on
link prediction. Given a snapshot $\mathcal{S}_{i}$, for each test fact
$(s,r,o)\in\mathcal{Q}_{i}$, we construct two queries $(s,r,?)$ and $(?,r,t)$.
When evaluating on $\mathcal{Q}_{i}$, we set all seen entities in
$\mathcal{E}_{i}$ as candidate entities. We select seven metrics to evaluate
all models, including (i) Four metrics on link prediction accuracy: mean
reciprocal rank (MRR) and Hits@$k$ ($k=1,3,10$, and H@$k$ for shot). We
conduct the model $\mathcal{M}_{5}$ trained on the last snapshot to evaluate
on the union of the test sets in all snapshots. (ii) Two metrics on knowledge
transfer capability: forward transfer (FWT) and backward transfer (BWT)
(Lopez-Paz and Ranzato 2017). FWT is the influence of learning a task to the
performance on the future tasks, while BWT is the influence of learning to the
previous tasks:

(13)

where $n$ is the number of snapshots, $h_{i,j}$ is the MRR scores on
$\mathcal{Q}_{j}$ after training the model $\mathcal{M}_{i}$ on the $i$-th
snapshot. Higher scores indicate better performance. (iii) Time cost: The
cumulative time cost of the learning on each snapshot.

Models | Entity | Relation | Fact | Hybrid
---|---|---|---|---
MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10
Snapshot | 0.084 | 0.028 | 0.107 | 0.193 | 0.021 | 0.010 | 0.023 | 0.043 | 0.082 | 0.030 | 0.095 | 0.191 | 0.036 | 0.015 | 0.043 | 0.077
Re-train | 0.236 | 0.137 | 0.274 | 0.433 | 0.219 | 0.128 | 0.250 | 0.403 | 0.206 | 0.118 | 0.232 | 0.385 | 0.227 | 0.134 | 0.260 | 0.413
Fine-tune | 0.165 | 0.085 | 0.188 | 0.321 | 0.093 | 0.039 | 0.106 | 0.195 | 0.172 | 0.090 | 0.193 | 0.339 | 0.135 | 0.069 | 0.151 | 0.262
MEAN | 0.117 | 0.068 | 0.123 | 0.212 | 0.039 | 0.024 | 0.040 | 0.067 | 0.084 | 0.051 | 0.088 | 0.146 | 0.046 | 0.029 | 0.049 | 0.080
LAN | 0.141 | 0.082 | 0.149 | 0.256 | 0.052 | 0.033 | 0.052 | 0.092 | 0.106 | 0.056 | 0.113 | 0.200 | 0.059 | 0.032 | 0.062 | 0.113
PNN | 0.229 | 0.130 | 0.265 | 0.425 | 0.167 | 0.096 | 0.191 | 0.305 | 0.157 | 0.084 | 0.188 | 0.290 | 0.185 | 0.101 | 0.216 | 0.349
CWR | 0.088 | 0.028 | 0.114 | 0.202 | 0.021 | 0.010 | 0.024 | 0.043 | 0.083 | 0.030 | 0.095 | 0.192 | 0.037 | 0.015 | 0.044 | 0.077
SI | 0.154 | 0.072 | 0.179 | 0.311 | 0.113 | 0.055 | 0.131 | 0.224 | 0.172 | 0.088 | 0.194 | 0.343 | 0.111 | 0.049 | 0.126 | 0.229
EWC | 0.229 | 0.130 | 0.264 | 0.423 | 0.165 | 0.093 | 0.190 | 0.306 | 0.201 | 0.113 | 0.229 | 0.382 | 0.186 | 0.102 | 0.214 | 0.350
GEM | 0.165 | 0.085 | 0.188 | 0.321 | 0.093 | 0.040 | 0.106 | 0.196 | 0.175 | 0.092 | 0.196 | 0.345 | 0.136 | 0.070 | 0.152 | 0.263
EMR | 0.171 | 0.090 | 0.195 | 0.330 | 0.111 | 0.052 | 0.126 | 0.225 | 0.171 | 0.090 | 0.191 | 0.337 | 0.141 | 0.073 | 0.157 | 0.267
DiCGRL | 0.107 | 0.057 | 0.110 | 0.211 | 0.133 | 0.079 | 0.147 | 0.241 | 0.162 | 0.084 | 0.189 | 0.320 | 0.149 | 0.083 | 0.168 | 0.277
LKGE | 0.234 | 0.136 | 0.269 | 0.425 | 0.192 | 0.106 | 0.219 | 0.366 | 0.210 | 0.122 | 0.238 | 0.387 | 0.207 | 0.121 | 0.235 | 0.379

Table 2: Result comparison of link prediction on the union of the test sets in
all snapshots.

Implementation details. We use TransE (Bordes et al. 2013) as the base model
and modify the competitors to do our task:

  * •

Snapshot only. For the $i$-th snapshot, we reinitialize and train a model only
on the training set $\mathcal{D}_{i}$.

  * •

Re-training. For the $i$-th snapshot, we reinitialize and train a model on the
accumulated training data $\cup_{j=1}^{i}\mathcal{D}_{j}$.

  * •

Fine-tuning. For the $i$-th snapshot, the model inherits the learned
parameters of the model trained on the previous snapshots, and we
incrementally train it on $\mathcal{D}_{i}$.

  * •

Inductive models. We train each model on the first snapshot and obtain the
embeddings of unseen entities in the following snapshots by neighborhood
aggregation.

  * •

Dynamic architecture models. For PNN, following the implementation of (Daruna
et al. 2021), we freeze the parameters learned on previous snapshots and
update new parameters. For CWR, after training on $\mathcal{D}_{1}$, we
replicate a model as the consolidated model. For the following $i$-th
snapshot, we reinitialize and train a temporal model on $\mathcal{D}_{i}$, and
merge the temporal model into the consolidated model by copying new parameters
or averaging old ones.

  * •

Regularization models. Since the base model parameters increase with the
emergence of unseen entities and relations, we only use the parameters learned
from the previous snapshot to calculate the regularization loss.

  * •

Rehearsal models. We store 5,000 training facts from previous snapshots and
add them to the current training set of the $i$-th snapshot. After the
learning, we randomly replace half of these facts with those in
$\mathcal{D}_{i}$.

For a fair comparison, we first tune the hyperparameters of the base model
using grid-search: learning rate in {0.0005, 0.0001, 0.001}, batch size in
{1024, 2048}, embedding dimension in {100, 200}. Then, we use the same base
model for LKGE and all competitors, and tune other hyperparameters. For the
regularization models, the $\alpha$ of regularization loss is in {0.01, 0.1,
1.0}. For our model, the $\beta$ of MAE loss is in {0.01, 0.1, 1.0}. For all
competitors, we use Adam optimizer and set the patience of early stopping to
3.

###  5.2 Link Prediction Accuracy

In Table 2, we run 5-seeds experiments for all models on our datasets and
report the means. The results show that: (i) our model consistently achieves
the best performance across all datasets. Some results of our model in Fact
even outperforms re-training. This is because our masked KG autoencoder
effectively improves information propagation based on both old and new
embeddings, and the embedding regularization avoids catastrophic forgetting.
Most competitors only work well on Entity, while our model shows stable and
promising results on all these datasets. (ii) Re-training is far superior to
most baseline models on Relation and Hybrid, while the gaps on Entity and Fact
are small. This is because the KG embedding model learns two aspects of
knowledge: relational patterns and entity embeddings. In Entity and Fact, the
relational patterns are stable, while in Relation and Hybrid, their relational
patterns are constantly changing due to unseen relations. These phenomena
illustrate that the variation of relational patterns is more challenging for
lifelong KG embedding. (iii) The inductive models are only trained on the
first snapshot, and cannot transfer knowledge to unseen relations. So, their
results are lower than other models, especially on Relation and Hybrid with
many unseen relations. (iv) Since the learned parameters are not updated, PNN
preserves the learned knowledge well. But on Fact, due to a few unseen
entities, it lacks new learnable parameters, and the performance is not well.
CWR averages the old and new model parameters, which does not work well on the
embedding learning task. (v) EWC performs well because it can model the
importance of each parameter using Fisher information matrices to the learned
snapshots. (vi) Unlike the classification tasks, most parameters of KG
embedding models correspond to specific entities, so the training data cannot
be divided into a few types, and we cannot use 5,000 old samples to replay the
learned facts for all entities. The performance of GEM, EMR and DiCGRL is
limited.

Figure 3: MRR changes. $\mathcal{M}_{i}$ is trained for the $i$-th snapshot
and evaluated using the test data of previous snapshots 1 to $i$.

Variants | Entity | Relation | Fact | Hybrid
---|---|---|---|---
MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10
LKGE (full) | 0.234 | 0.136 | 0.269 | 0.425 | 0.192 | 0.106 | 0.219 | 0.366 | 0.210 | 0.122 | 0.238 | 0.387 | 0.207 | 0.121 | 0.235 | 0.379
$-$ fine-tuning | 0.123 | 0.068 | 0.136 | 0.225 | 0.126 | 0.073 | 0.146 | 0.231 | 0.154 | 0.091 | 0.176 | 0.269 | 0.109 | 0.060 | 0.126 | 0.201
$-$ autoencoder | 0.222 | 0.124 | 0.255 | 0.415 | 0.185 | 0.100 | 0.212 | 0.355 | 0.191 | 0.105 | 0.215 | 0.369 | 0.198 | 0.111 | 0.227 | 0.367
$-$ embedding transfer | 0.240 | 0.141 | 0.275 | 0.433 | 0.174 | 0.091 | 0.201 | 0.339 | 0.210 | 0.123 | 0.237 | 0.390 | 0.200 | 0.112 | 0.229 | 0.372
$-$ regularization | 0.166 | 0.089 | 0.184 | 0.316 | 0.040 | 0.014 | 0.049 | 0.089 | 0.175 | 0.095 | 0.195 | 0.338 | 0.154 | 0.079 | 0.171 | 0.300

Table 3: Ablation results of link prediction on the union of the test sets in
all snapshots.

To show the performance evolution of LKGE during the learning process, we
evaluate the model $\mathcal{M}_{i}$ trained for the $i$-th snapshot using the
test data from previous snapshots. The MRR results are in Figure 3. LKGE can
maintain the learned knowledge during lifelong learning. On some snapshots
like Entity Snapshot 3, the knowledge update improves the performance on old
test data, which shows that old knowledge update has the potential for
backward knowledge transfer.

###  5.3 Knowledge Transfer Capability

To evaluate the knowledge transfer and retention capability of all models, we
report the FWT and BWT of MRR results in Figure 4. Because of the embedding
transfer, the FWT of LKGE is higher than all lifelong learning competitors.
Even on Relation and Hybrid where the KG schema changes, LKGE still keeps the
FWT capability well. MEAN and LAN are designed to transfer knowledge forward
to embed new entities. So, they work well on Entity. However, their FWT
capability is limited on other datasets since they cannot update the old
embeddings to adapt to new snapshots.

BWT is usually negative due to the overwriting of learned knowledge. PNN,
MEAN, and LAN do not update old parameters. Their BWT scores are “NA”. The
poor scores of CWR show the harmful effects of the average operation. The
scores of rehearsal models are also not good as they cannot store enough
facts. LKGE gets good BWT scores as the embedding regularization can well
maintain the learned knowledge.

###  5.4 Learning Efficiency

We show the training time on Fact as all snapshots of Fact have the same
training set size. Figure 5 shows the results. Unsurprisingly, re-training is
most time-consuming. Snapshot is also costly because it cannot inherit
knowledge from previous snapshots. By contrast, our model is most efficient,
and its advantage is more significant in the final snapshot. This is because
the embedding transfer can use the learned knowledge to accelerate the
learning of new facts.

###  5.5 Ablation Study

We conduct an ablation study by designing four variants of LKGE: “w/o fine-
tuning”, “w/o autoencoder”, “w/o embedding transfer” and “w/o regularization”.
The “w/o fine-tuning” variant is trained on $\mathcal{D}_{1}$ and performs the
embedding transfer on other $\mathcal{D}_{i}$. The latter three variants
disable the specific components in LKGE. The results are shown in Table 3. We
see that (i) although fine-tuning is disabled, “w/o fine-tuning” can still
perform well with only the knowledge from the first snapshot, showing that
embedding transfer can effectively transfer learned knowledge to unseen
entities and relations. (ii) Both “w/o autoencoder” and “w/o regularization”
significantly drop, showing the effects of masked KG autoencoder and knowledge
retention. (iii) Embedding transfer enables the model to be trained at a
starting point closer to the optimal parameters and stabilizes the embedding
space. There are declines when using the embedding transfer on Entity. This is
because Entity contains massive new entities and needs more plasticity rather
than stability. But on Relation and Hybrid, the results of “w/o embedding
transfer” are lower than the full model, showing that embedding transfer can
reduce the interference caused by the KG schema changes. On Fact, the results
of “w/o embedding transfer” are similar to the full model. This shows that,
even without embedding transfer, the model can still capture the knowledge.

Figure 4: Forward transfer and backward transfer of MRR. Figure 5: Cumulative
time cost on Fact.

##  6 Conclusion and Future Work

This paper studies lifelong embedding learning for growing KGs. For better
knowledge transfer and retention, we propose a lifelong KG embedding model
consisting of masked KG autoencoder, embedding transfer, and embedding
regularization. Experiments on new datasets show better link prediction
accuracy, knowledge transfer capability, and learning efficiency of our model.
In future, we plan to study lifelong embedding learning in long-tail and low-
resource settings.

## Acknowledgments

This work was supported by National Natural Science Foundation of China (No.
62272219).

## References

  * Bordes et al. (2013) Bordes, A.; Usunier, N.; García-Durán, A.; Weston, J.; and Yakhnenko, O. 2013.  Translating Embeddings for Modeling Multi-relational Data.  In _NeurIPS_ , 2787–2795.
  * Chen et al. (2021) Chen, J.; He, H.; Wu, F.; and Wang, J. 2021.  Topology-Aware Correlations Between Relations for Inductive Link Prediction in Knowledge Graphs.  In _AAAI_ , 6271–6278.
  * Daruna et al. (2021) Daruna, A. A.; Gupta, M.; Sridharan, M.; and Chernova, S. 2021.  Continual Learning of Knowledge Graph Embeddings.  _IEEE Robotics Autom. Lett._ , 6(2): 1128–1135.
  * Dettmers et al. (2018) Dettmers, T.; Minervini, P.; Stenetorp, P.; and Riedel, S. 2018.  Convolutional 2D Knowledge Graph Embeddings.  In _AAAI_ , 1811–1818.
  * Galkin et al. (2022) Galkin, M.; Denis, E. G.; Wu, J.; and Hamilton, W. L. 2022.  NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs.  In _ICLR_.
  * Guo, Sun, and Hu (2019) Guo, L.; Sun, Z.; and Hu, W. 2019.  Learning to Exploit Long-term Relational Dependencies in Knowledge Graphs.  In _ICML_ , volume 97, 2505–2514.
  * Hamaguchi et al. (2017) Hamaguchi, T.; Oiwa, H.; Shimbo, M.; and Matsumoto, Y. 2017.  Knowledge Transfer for Out-of-Knowledge-Base Entities: A Graph Neural Network Approach.  In _IJCAI_ , 1802–1808.
  * Hou et al. (2022) Hou, Z.; Liu, X.; Cen, Y.; Dong, Y.; Yang, H.; Wang, C.; and Tang, J. 2022.  GraphMAE: Self-Supervised Masked Graph Autoencoders.  _CoRR_ , abs/2205.10803: 1–11.
  * Ji et al. (2022) Ji, S.; Pan, S.; Cambria, E.; Marttinen, P.; and Yu, P. S. 2022.  A Survey on Knowledge Graphs: Representation, Acquisition, and Applications.  _IEEE Trans. Neural Netw. Learn. Syst._ , 33(2): 494–514.
  * Kipf and Welling (2017) Kipf, T. N.; and Welling, M. 2017.  Semi-Supervised Classification with Graph Convolutional Networks.  In _ICLR_.
  * Kirkpatrick et al. (2017) Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N. C.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; Hassabis, D.; Clopath, C.; Kumaran, D.; and Hadsell, R. 2017.  Overcoming Catastrophic Forgetting in Neural Networks.  _PNAS_ , 114(13): 3521–3526.
  * Kou et al. (2020) Kou, X.; Lin, Y.; Liu, S.; Li, P.; Zhou, J.; and Zhang, Y. 2020.  Disentangle-based Continual Graph Representation Learning.  In _EMNLP_ , 2961–2972.
  * Liu et al. (2021) Liu, S.; Grau, B. C.; Horrocks, I.; and Kostylev, E. V. 2021.  INDIGO: GNN-Based Inductive Knowledge Graph Completion Using Pair-Wise Encoding.  In _NeurIPS_ , 2034–2045.
  * Lomonaco and Maltoni (2017) Lomonaco, V.; and Maltoni, D. 2017.  CORe50: A New Dataset and Benchmark for Continuous Object Recognition.  In _CoRL_ , volume 78, 17–26.
  * Lopez-Paz and Ranzato (2017) Lopez-Paz, D.; and Ranzato, M. 2017.  Gradient Episodic Memory for Continual Learning.  In _NeurIPS_ , 6467–6476.
  * Rusu et al. (2016) Rusu, A. A.; Rabinowitz, N. C.; Desjardins, G.; Soyer, H.; Kirkpatrick, J.; Kavukcuoglu, K.; Pascanu, R.; and Hadsell, R. 2016.  Progressive Neural Networks.  _CoRR_ , abs/1606.04671: 1–14.
  * Scarselli et al. (2009) Scarselli, F.; Gori, M.; Tsoi, A. C.; Hagenbuchner, M.; and Monfardini, G. 2009\.  The Graph Neural Network Model.  _IEEE Trans. Neural Netw._ , 20(1): 61–80.
  * Schlichtkrull et al. (2018) Schlichtkrull, M.; Kipf, T. N.; Bloem, P.; van den Berg, R.; Titov, I.; and Welling, M. 2018.  Modeling Relational Data with Graph Convolutional Networks.  In _ESWC_ , 593–607.
  * Tay, Luu, and Hui (2017) Tay, Y.; Luu, A. T.; and Hui, S. C. 2017.  Non-Parametric Estimation of Multiple Embeddings for Link Prediction on Dynamic Knowledge Graphs.  In _AAAI_ , 1243–1249.
  * Teru, Denis, and Hamilton (2020) Teru, K.; Denis, E.; and Hamilton, W. 2020.  Inductive Relation Prediction by Subgraph Reasoning.  In _ICML_ , 9448–9457.
  * Toutanova and Chen (2015) Toutanova, K.; and Chen, D. 2015.  Observed Versus Latent Features for Knowledge Base and Text Inference.  In _CVSC_ , 57–66.
  * Vashishth et al. (2020a) Vashishth, S.; Sanyal, S.; Nitin, V.; Agrawal, N.; and Talukdar, P. P. 2020a.  InteractE: Improving Convolution-based Knowledge Graph Embeddings by Increasing Feature Interactions.  In _AAAI_ , 3009–3016.
  * Vashishth et al. (2020b) Vashishth, S.; Sanyal, S.; Nitin, V.; and Talukdar, P. P. 2020b.  Composition-based Multi-Relational Graph Convolutional Networks.  In _ICLR_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In _NeurIPS_ , 5998–6008.
  * Vrandecic and Krötzsch (2014) Vrandecic, D.; and Krötzsch, M. 2014.  Wikidata: A Free Collaborative Knowledgebase.  _Commun. ACM_ , 57(10): 78–85.
  * Wang et al. (2019a) Wang, H.; Xiong, W.; Yu, M.; Guo, X.; Chang, S.; and Wang, W. Y. 2019a.  Sentence Embedding Alignment for Lifelong Relation Extraction.  In _NAACL_ , 796–806.
  * Wang et al. (2019b) Wang, P.; Han, J.; Li, C.; and Pan, R. 2019b.  Logic Attention Based Neighborhood Aggregation for Inductive Knowledge Graph Embedding.  In _AAAI_ , 7152–7159.
  * Wang et al. (2017) Wang, Q.; Mao, Z.; Wang, B.; and Guo, L. 2017.  Knowledge Graph Embedding: A Survey of Approaches and Applications.  _IEEE Trans. Knowl. Data Eng._ , 29(12): 2724–2743.
  * Wang et al. (2014) Wang, Z.; Zhang, J.; Feng, J.; and Chen, Z. 2014.  Knowledge Graph Embedding by Translating on Hyperplanes.  In _AAAI_ , 1112–1119.
  * Wu et al. (2022) Wu, T.; Khan, A.; Yong, M.; Qi, G.; and Wang, M. 2022.  Efficiently Embedding Dynamic Knowledge Graphs.  _Knowl. Based Syst._ , 250: 109124.
  * Zenke, Poole, and Ganguli (2017) Zenke, F.; Poole, B.; and Ganguli, S. 2017.  Continual Learning through Synaptic Intelligence.  In _ICML_ , 3987–3995.
  * Zhu et al. (2021) Zhu, Z.; Zhang, Z.; Xhonneux, L. A. C.; and Tang, J. 2021.  Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction.  In _NeurIPS_ , 29476–29490.


homotopy category, even for sharp cubical sets like
$\amalg_{i=1}^{\infty}\square[0]$.

We will show that cubical cohomology can be expressed in terms of
representable functors on the Lipschitz homotopy category as follows. For each
Abelian group $\pi$, define $C(\pi,n)_{m}$ to be the following Abelian group
natural in $\square$-objects $[1]^{m}$:

$C(\pi,n)_{m}=\mathcal{Ch}_{\raisebox{0.4pt}{\scalebox{0.6}{+}}}\left(\mathfrak{ch}_{*}\mathfrak{ner}_{\square}[1]^{m},\Sigma^{n}\pi\right).$

######  Example 4.7.

For each Abelian group $\pi$, there exists a cubical homotopy equivalence

$\mathfrak{ner}_{\square}\,\pi\simeq C(\pi,1).$

Fix a seminormed Abelian group $\pi$. We represent
$H^{*}_{\ell_{\infty}}(-;\pi)$ in certain cases as follows. As a cubical set,
$C(\pi,n)$ admits a filtration whose $i$th stage is the subpresheaf of
$C(\pi,n)$ whose $m$-cubes are all chain maps
$\mathfrak{ch}_{*}\mathfrak{ner}_{\square}[1]^{m}\rightarrow\Sigma^{n}\pi$
which, in degree $n$, sends each $n$-cell of
$|\mathfrak{ner}_{\square}[1]^{m}|$ to an element in $(\Sigma^{n}\pi)_{n}=\pi$
with norm no greater than $i$.

######  Lemma 4.8.

There exists an isomorphism

(18)
$H^{*}_{\ell_{\infty}}(B^{\sharp};\pi)\cong\mathrm{colim}_{i\rightarrow\infty}\left[B,C(\pi,n)^{(i)}\right]$

natural in seminormed Abelian groups $\pi$.

###### Proof.

Let $D(\pi,n)$ be the cubical Abelian group

$D(\pi,n)=\mathcal{Ch}_{\raisebox{0.4pt}{\scalebox{0.6}{+}}}(\mathfrak{ch}_{*}\square[-],\Sigma^{n}\pi).$

Call a chain map from a cellular chain complex to $\Sigma^{n}\pi$ $i$-bounded
if it sends $n$-cells to elements in $(\Sigma^{n}\pi)_{n}=\pi$ with norm no
greater than $i$. Regard $D(\pi,n)$ as filtered so that the $m$-cubes in the
$i$th stage of its filtration consists of all $i$-bounded chain maps
$\mathfrak{ch}_{*}\square[1]^{m}\rightarrow\Sigma^{n}\pi$.

The inclusion $|\square[1]\hookrightarrow\mathfrak{ner}_{\square}[1]|$ of CW
complexes admits a cellular retraction mapping vertices to vertices and
$m$-cells onto the unique $1$-cell for each $m>0$. It therefore follows that
the inclusion
$\mathfrak{ch}_{*}(\square[1]\hookrightarrow\mathfrak{ner}_{\square}[1])$ of
chain complexes admits a retraction which degree-wise sends each cell in
$|\mathfrak{ner}_{\square}[1]|$ to either a cell in $|\square[1]|$ or $0$.
Every $i$-bounded chain map from
$\mathfrak{ch}_{*}(A\otimes\square[1])=\mathfrak{ch}_{*}A\otimes\mathfrak{ch}_{*}\square[1]$
to $\Sigma^{n}\pi$ thus extends to an $i$-bounded chain map on
$\mathfrak{ch}_{*}(A\otimes\mathfrak{ner}_{\square}[1])=\mathfrak{ch}_{*}A\otimes\mathfrak{ch}_{*}\mathfrak{ner}_{\square}[1]$.
It therefore follows that there exist natural bijections

$\displaystyle
c\mathscr{S}((-)\otimes\square[m],(D(\pi,n)^{(i)})_{\sharp})_{0}$
$\displaystyle\cong
c\mathscr{S}((-\otimes\square[m])^{\sharp},(D(\pi,n)^{(i)}))_{0}$
$\displaystyle\cong
c\mathscr{S}((-)^{\sharp}\otimes\mathfrak{ner}_{\square}[m],(D(\pi,n)^{(i)}))_{0}$
$\displaystyle\cong
c\mathscr{S}((-)^{\sharp}\otimes\square[m],(D(\pi,n)^{(i)}))_{0}$

natural in $\square$-objects of the form $[m]$ for $m=0,1$. Thus there exists
a natural bijection

(19)
$\pi_{0}c\mathscr{S}(-,(D(\pi,n)^{(i)})_{\sharp})\cong\pi_{0}c\mathscr{S}((-)^{\sharp},D(\pi,n)^{(i)}).$

Let $Z^{(n,i)}(-;\pi)$ denote the endofunctor on $c\mathscr{S}$ defined by

$Z^{(n,i)}(-;\pi)=c\mathscr{S}(-,D(\pi,n)^{(i)}).$

Under the Dold-Kan correspondence, $\pi_{0}Z^{(n,i)}(A;\pi)$ corresponds to
the the set of all $n$-cocycles $c$ of
$\mathfrak{ch}^{*}_{\ell_{\infty}}(A;\pi)$ with $\sup_{\sigma\in
A_{n}}\|c(\sigma)\|\leqslant i$ up to the smallest equivalence relating two
such cocycles if their difference is the coboundary of an $(n-1)$-cochain $h$
of $\mathfrak{ch}^{*}_{\ell_{\infty}}(A;\pi)$ with $\sup_{\sigma\in
A_{n-1}}\|h(\sigma)\|\leqslant i$.

There exist natural isomorphisms

(20) $\displaystyle\mathrm{colim}_{i}[B,C(\pi,n)^{(i)}]$
$\displaystyle\cong\mathrm{colim}_{i}[B,(D(\pi,n)^{(i)})_{\sharp}]$ (21)
$\displaystyle\cong\mathrm{colim}_{i,k}\pi_{0}c\mathscr{S}(\mathfrak{sd}^{k}B,(D(\pi,n)^{(i)}_{\sharp})^{\sharp})$
(22)
$\displaystyle\cong\mathrm{colim}_{i,k}\pi_{0}c\mathscr{S}(\mathfrak{sd}^{k}B^{\sharp},D(\pi,n)^{(i)}_{\sharp})$
(23)
$\displaystyle\cong\mathrm{colim}_{i,k}\pi_{0}Z^{(n,i)}(\mathfrak{sd}^{k}B^{\sharp};\pi)$
(24)
$\displaystyle\cong\mathrm{colim}_{k}H^{n}_{\ell_{\infty}}(\mathfrak{sd}^{k}B^{\sharp};\pi)$
(25) $\displaystyle\cong H^{n}_{\ell_{\infty}}(B^{\sharp};\pi),$

where the colimits over $k\rightarrow\infty$ are taken over all maps induced
by $\gamma^{\raisebox{0.4pt}{\scalebox{0.8}{-}}k}$, because: (20) by
$C(\pi,n)^{(i)}=(D(\pi,n)^{(i)})_{\sharp}$; (21) by a formula for the hom-sets
$[-,-]$ [Proposition 3.11]; (22) by the cubical homotopy equivalence
$(D(\pi,n)^{(i)})_{\sharp}\simeq((D(\pi,n)^{(i)})_{\sharp})^{\sharp}$ [Lemma
3.3]; (23) by (19); (24) by the Dold-Kan Correspondence; and (25) by
subdivision invariance of $\ell_{p}$ cohomology [Lemma 4.3]. ∎

Let $C_{\infty}(\pi,n)$ denote the coequalizer of the diagram

$\textstyle{\coprod_{i}C(\pi,n)^{(i)}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{(\amalg_{i}C(\pi,n)^{(i)})\otimes\mathfrak{ner}_{\square}\delta_{-}}$$\scriptstyle{\amalg_{i}(C(\pi,n)^{(i)}\hookrightarrow
C(\pi,n)^{(i+1)}\hookrightarrow\amalg_{j}C(\pi,n)^{(j)})\otimes\mathfrak{ner}_{\square}\delta_{+}}$$\textstyle{\coprod_{i}C(\pi,n)^{(i)}\otimes\mathfrak{ner}_{\square}[1]}$

######  Example 4.9.

Consider a finitely generated group $G$ with word-length function

$\ell:G\rightarrow\mathbb{N}.$

Fix a monotone function $\beta:\mathbb{N}\rightarrow\mathbb{N}$. Define
$\mathfrak{ner}_{\square}^{\beta}G$ as a mapping telescope like
$C_{\infty}(\pi,n)$ based on a filtration of $\mathfrak{ner}_{\square}G$ whose
$i$th stage is the maximal subpresheaf of $\mathfrak{ner}_{\square}G$ having
as its $1$-cubes all $g\in G$ with $\ell(g)\leqslant\beta(i)$. If
$\beta(n)\rightarrow\infty$, then the natural cubical function
$\mathfrak{ner}_{\square}^{\beta}G\rightarrow\mathfrak{ner}_{\square}G$ is a
classical weak equivalence but generally not a Lipschitz weak equivalence. In
the case $\mathbb{Z}$ is equipped with the standard absolute value word-length
function and $\beta=1_{\mathbb{N}}$, then
$\mathfrak{ner}_{\square}^{\beta}\mathbb{Z}=C_{\infty}(\mathbb{Z},1)$.

The cubical set $C_{\infty}(\pi,n)$ admits the structure of a cubical
semigroup whose multiplication is induced from dotted cubical functions making
commutative diagrams of the form

${(C(\pi,n)^{(i)}\otimes\square[1])\times(C(\pi,n)^{(j)}\otimes\mathfrak{ner}_{\square}[1])}$${C(\pi,n)^{(i)}\times
C(\pi,n)^{(j)}\otimes\mathfrak{ner}_{\square}[1]^{2}}$${C(\pi,n)^{(i+j)}\otimes\square[1]}$${C(\pi,n)^{(i)}\times
C(\pi,n)^{(j)}\otimes\mathfrak{ner}_{\square}[1]^{2}}$$\scriptstyle{\sigma}$$\scriptstyle{\mu\otimes\mathfrak{ner}_{\square}(\vee_{[1]^{2}})}$

where $\sigma$ permutes factors and $\mu$ is induced by addition on $\pi$. The
map

$C_{\infty}(\pi,n)\rightarrow C(\pi,n)$

induced by inclusions $C(\pi,n)^{(i)}\hookrightarrow C(\pi,n)$ is a cubical
semigroup homomorphism by the associativity of $\vee_{[1]^{2}}$ and addition
on $\pi$.

######  Theorem 4.10.

There exist dotted isomorphisms in the diagram

$\textstyle{H^{n}_{\ell_{\infty}}(B;\pi)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{H^{n}(B;\pi)}$$\textstyle{[B,C_{\infty}(\pi,n)]\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{[B,C_{\infty}(\pi,n)\rightarrow
C(\pi,n)]}$$\textstyle{[B,C(\pi,n)]\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\cong}$

natural in cubical sets $B$ that are either finite or both connected and Kan,
and normed Abelian groups $\pi$, making the entire square commute. Here
$[-,-]$ denotes a hom-set in $h_{\infty}(c\mathscr{S})$, the top horizontal
arrow is the comparison map, and the bottom horizontal arrow is induced by the
quotient cubical semigroup homomorphism $C_{\infty}(\pi,n)\rightarrow
C(\pi,n)$.

###### Proof.

Write $C_{\infty}(\pi,n)^{(j)}$ for the image of the canonical cubical
function

$\coprod_{i=1}^{j}C(\pi,n)^{(i)}\otimes\mathfrak{ner}_{\square}[1]\rightarrow
C_{\infty}(\pi,n).$

Consider the following solid arrow diagram of Abelian groups:

${H^{n}_{\ell_{\infty}}(B;\pi)}$${{[B,C_{\infty}(\pi,n)]}}$${H^{n}_{\ell_{\infty}}(B^{\sharp};\pi)}$${\mathrm{colim}_{i}[B,C(\pi,n)^{(i)}]}$${\mathrm{colim}_{i}[B,C_{\infty}(\pi,n)^{(i)}]}$$\scriptstyle{H^{n}_{\ell_{\infty}}(B\hookrightarrow
B^{\sharp};\pi)}$$\scriptstyle{\cong}$$\scriptstyle{\cong}$$\scriptstyle{\cong}$

Kan cubical sets are sharp by $C\hookrightarrow C^{\sharp}$ a weak equivalence
from a fibrant object in the test model structure [Lemma 3.25]. Therefore the
left vertical arrow is a natural isomorphism [Lemma 4.8]. There exists a
bottom horizontal natural dotted isomorphism [Lemma 4.8]. The bottom solid
horizontal arrow is an isomorphism because inclusion
$C(\pi,n)^{(i)}\hookrightarrow C_{\infty}(\pi,n)^{(i)}$ is a cubical homotopy
equivalence and hence a Lipschitz weak equivalence. The right vertical arrow
is an isomorphism because $|B|_{\sphericalangle}$ and hence
$|\mathfrak{sd}^{k}B|_{\sphericalangle}\times|\square[1]^{n}|_{\sphericalangle}$
has finite diameter [Lemma 2.27], hence every Lipschitz map
$|\mathfrak{sd}^{k}B|_{\sphericalangle}\times|\square[1]^{n}|_{\sphericalangle}\rightarrow|C_{\infty}(\pi,n)|_{\sphericalangle}$
factors through $|C_{\infty}(\pi,n)^{(i)}|_{\sphericalangle}$ for $i\gg 0$,
and hence every cubical function of the form
$\mathfrak{sd}^{k}(B\otimes\square[1]^{n})\rightarrow C_{\infty}(\pi,n)$
factors through $C_{\infty}(\pi,n)^{(i)}$ for $i\gg 0$. ∎

######  Example 4.11.

The first bounded cohomology of a group always vanishes. However, we can
define an alternative cohomology theory $H^{n}_{\beta}(-;\pi)$, with respect
to a bounding function $\beta$, on groups $G$ equipped with word-length
functions, by the rule

$H^{n}_{\beta}(G;\pi)=[\mathfrak{ner}_{\square}^{\beta}G,C_{\infty}(\pi,n)],$

where $\mathfrak{ner}_{\square}^{\beta}G$ is a certain surgical construction
on the cubical nerve $\mathfrak{ner}_{\square}G$ [Example 4.9]. The group
$H^{n}_{\beta}(G;\pi)$ is intepretable as ordinary group cohomology, but where
the growth of the cochains as word lengths increase is constrained by the
bounding function $\beta$. In the case $G=\mathbb{Z}$ is equipped its usual
absolute value norm and $\beta=1_{\mathbb{N}}$,
$H^{1}_{\beta}(\mathbb{Z};\mathbb{Z})\neq 0$.

###  4.2. Uniform

Let $K_{\infty}(\pi,n),K(\pi,n)$ denote the uniform realizations of the
respective cubical sets $C_{\infty}(\pi,n),C(\pi,n)$. The semigroup
multiplications induce a multiplication turning the induced uniform map
$K_{\infty}(\pi,n)\rightarrow K(\pi,n)$ into a homomorphism of commutative
semigroup objects in $\mathscr{U}$.

######  Example 4.12.

There exists a natural identification

$H^{*}_{\ell_{\infty}}(\mathfrak{sing}_{\Delta_{\raisebox{0.4pt}{\scalebox{0.6}{+}}}}X;\mathbb{R})=H^{*}_{\ell_{\infty}}(\mathfrak{sing}_{\square_{1}^{*}}X;\mathbb{R}),$

where $\mathfrak{sing}_{\Delta_{\raisebox{0.4pt}{\scalebox{0.6}{+}}}}$ and
$\mathfrak{sing}_{\square_{1}^{*}}$ are the right adjoint to topological
realization functors from, respectively, semi-simplicial sets and classically
defined cubical sets, and the cochain complexes used to define the
cohomologies are defined as the bounded subcomplexes of the usual cochain
complexes for semi-simplicial and classically defined cubical sets [27,
Example 3.2, Proposition 4.3]. The same kind of reasoning used to give this
kind of reasoning, the method of acyclic models [10], shows that we can make
the identification
$H^{*}_{\ell_{\infty}}(\mathfrak{sing}_{\Delta_{+}}X;\mathbb{R})=H^{*}_{\ell_{\infty}}(\mathfrak{sing}_{\square}X;\mathbb{R})$.

######  Corollary 4.13.

There exist dotted isomorphisms in the diagram

$\textstyle{H^{n}_{\ell_{\infty}}(\mathfrak{sing}_{\square}X;\pi)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{H^{n}(\mathfrak{sing}_{\square}X;\pi)}$$\textstyle{[X,K_{\infty}(\pi,n)]\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{[X,K_{\infty}(\pi,n)\rightarrow
K(\pi,n)]}$$\textstyle{[X,K(\pi,n)]\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\cong}$

natural in path-connected spaces $X$ and normed Abelian groups $\pi$, making
the entire square commute. Here $[-,-]$ denotes a hom-set in
$h_{\infty}\mathscr{U}$, $X$ is regarded in the bottom row as a uniform space
with its fine uniformity, the top horizontal arrow is the comparison map, and
the bottom horizontal arrow is induced by the quotient uniform map
$K_{\infty}(\pi,n)\rightarrow K(\pi,n)$.

###  4.3. Comparisons

For each $n$, there exists an isomorphism

$H^{*}(B;\pi)\cong H^{*}(|B|;\pi)$

natural in cubical sets $B$ and Abelian groups $\pi$, because
$\mathfrak{ch}^{*}(B,\pi)$ is the cellular cochain complex of $|B|$ when $\pi$
is trivially seminormed. A special case of the embedding of the Lipschitz
homotopy category into the uniform homotopy category [Corollary 3.29] is the
following refinement of that isomorphism.

######  Proposition 4.14.

Uniform realization induces an isomorphism

$\left[B,C_{\infty}(\pi,n)\right]\cong\left[|B|_{\infty},K_{\infty}(\pi,n)\right]$

of Abelian groups natural in cubical sets $B$ and seminormed Abelian groups
$\pi$.

##  5\. Conclusion

Lipschitz homotopy theory naturally extends to the model categorical setting.
Consider a closed Cartesian monoidal presheaf model category $\mathscr{M}$ in
which all objects are cofibrant. Further suppose there exists a distinguished
generating diagram $J$ for the acyclic cofibrations, a diagram in the arrow
category of $\mathscr{M}$ for which the fibrations in $\mathscr{M}$ are
characterized as having suitably natural right lifting properties against the
objects in $J$ [3]. The choice of $J$ determines a kind of metric structure on
$\mathscr{M}$ as follows. An algebraic small object argument builds a fibrant
replacement monad $R$ for $\mathscr{M}$ from $J$ as a transfinite colimit of
pointed endofunctors. A weak equivalence in $\mathscr{M}$ is exactly a
homotopy equivalence after applying $R$. We can call a weak equivalence
$\zeta$ in $\mathscr{M}$ a Lipschitz weak equivalence if for each
$\mathscr{M}$-object $o$, $\zeta^{o}$ is a homotopy equivalence after applying
one of the finite intermediate stages in the construction of $R$. In the case
$\mathscr{M}$ is a category of presheaves over a test category [6, 18], a
natural question is whether there exists a choice of generating diagram $J$ so
that the localization by the resulting class of Lipschitz weak equivalences is
equivalent to the Lipschitz homotopy category of cubical sets. The special
case where $\mathscr{M}$ is the category of simplicial sets, each of which is
plausibly sharp in a Lipschitz-theoretic sense (eg. [35]), is of particular
interest in simplicial $\ell_{p}$ cohomology.

##  6\. Acknowledgements

The first author was partially supported by AFOSR grant FA9550-16-1-0212.

##  Appendix A Simplicial sets

We factor both geometric realization and sharp replacement [Definition 2.7]
through simplicial sets. These factorizations make it easy to formally
construct certain maps needed in proofs throughout the paper. Let $\Delta$
denote the category of non-empty finite ordinals $[0],[1],\ldots$ and all
monotone functions between them. Let $s\mathscr{S}$ denote the category of
functors $\Delta^{\mathrm{op}}\rightarrow\mathscr{S}$ and natural
transformations between them. Simplicial sets and simplicial functions are the
objects and morphisms of $s\mathscr{S}$. Write
$\mathfrak{ner}_{\Delta}\mathscr{X}$ for the simplicial nerve natural in a
small category $\mathscr{X}$, defined as naturally sending each
$\Delta$-object $[n]$ to the set of functors $[n]\rightarrow\mathscr{X}$. Let
$\mathfrak{tri}\;\\!\\!\dashv\mathfrak{quad}\;\\!\\!$ denote the monoidal
adjunction

$\mathfrak{tri}\;\\!\\!:c\mathscr{S}\leftrightarrows
s\mathscr{S}:\mathfrak{quad}\;\\!\\!$

with $\mathfrak{tri}\;\\!\\!$ naturally sending a representable
$\square[1]^{n}$ to the simplicial nerve $\mathfrak{ner}_{\Delta}\,[1]^{n}$.
Formally, $\mathfrak{tri}\;\\!\\!$ is cocontinuous. Let
$\mathfrak{quad}\;\\!\\!$ denote the right adjoint to
$\mathfrak{tri}\;\\!\\!$. Less formally, the monad
$\mathfrak{quad}\;\\!\\!\,\mathfrak{tri}\;\\!\\!$ is cocontinuous.

######  Lemma A.1.

There exists an isomorphism

$C^{\sharp}\cong\mathfrak{quad}\;\\!\\!\,\mathfrak{tri}\;\\!\\!\,C$

natural in cubical sets $C$.

A proof for the corresponding statement under a different definition of the
$\square$ category [25, Lemma 4.2] only depends on the following facts: every
$\square$-morphism uniquely factors into a surjective $\square$-morphism
followed by an injective $\square$-morphism; and injective $\square$-morphisms
are uniquely determined by where they send extrema. Therefore a proof for the
lemma above is omitted.

######  Lemma A.2.

Geometric realization factors through triangulation.

###### Proof.

Let $F$ be the cocontinuous functor

$F:s\mathscr{S}\rightarrow\mathscr{M}$

sending each representable $\Delta[n]$ to the subset of $\mathbb{R}^{n}$
consisting of all tuples $(x_{1},\cdots,x_{n})$ with $1\geqslant
x_{1}\geqslant\cdots\geqslant x_{n}\geqslant 0$, equipped with the metric such
that two points $x,y$ are $\delta$-close if $x_{i},y_{i}$ are $\delta$-close
for all $i$, and sending each simplicial function
$\Delta[\phi]:\Delta[m]\rightarrow\Delta[n]$ to the linear extension of the
function sending the point $(1,1,\cdots,1,0,\cdots,0)$ with $m-i$ non-zero
coordinates to the point $(1,1,\cdots,1,0,\cdots,0)$ with $n-\phi(0)$ non-zero
entries. It suffices to show $F\mathfrak{tri}\;\\!\\!\,\square[-]$ is a
monoidal functor $\square\rightarrow\mathscr{M}$. For then there would exist
$\mathscr{M}$-isomorphisms

$\displaystyle F\mathfrak{tri}\;\\!\\!\,\square[1]^{n}$ $\displaystyle\cong
F(\mathfrak{tri}\;\\!\\!\,\square[1])^{n}$ $\displaystyle\cong
F(\Delta[1])^{n}$ $\displaystyle=|\square[1]|_{\sphericalangle}^{n}$
$\displaystyle=|\square[1]^{[1]^{n}}|_{\sphericalangle}$

natural in $\square$-object $[1]^{n}$. Then
$F\mathfrak{tri}\;\\!\\!=|-|_{\sphericalangle}$ would follow because all
functors in sight cocontinuous.

Let $\pi_{i}$ denote projection $[1]^{n}\rightarrow[1]$ onto the $i$th factor.
Let $\phi$ denote an injective monotone function $[n]\rightarrow[1]^{n}$. It
suffices to show that the $1$-Lipschitz function
$\prod_{i}F\mathfrak{ner}_{\Delta}\pi_{i}\phi$ for each $\phi$, and hence the
$1$-Lipschitz bijection
$\prod_{i}(F\,\mathfrak{ner}_{\Delta}\pi_{i}):F\mathfrak{tri}\;\\!\\!\,\square[1]^{n}\rightarrow(F\mathfrak{tri}\;\\!\\!\,\square[1])^{n}$,
is $1$-bi-Lipschitz.

Fix $\phi$. For each $z\in F\Delta[n]$, let
$\bar{z}_{i}=(F\mathfrak{ner}_{\Delta}\pi_{i}\phi)(z)$. Consider $x,y\in
F\Delta[n]$ for which $\prod_{i}\bar{x}_{i},\prod_{i}\bar{y}_{i}$ are
$\epsilon$-close. Then $\bar{x}_{i},\bar{y}_{i}$ are $\epsilon$-close for each
$i$. For each $1\leqslant i\leqslant n$, $\pi_{i}\phi:[n]\rightarrow[1]$ is
the monotone function sending $i-1$ to $0$ and $i$ to $1$. Therefore
$\bar{x}_{i}=x_{i}$ and $\bar{y}_{i}=y_{i}$ for each $1\leqslant i\leqslant
n$. Hence $x,y$ are $\epsilon$-close in $F\Delta[n]$. ∎

######  Remark A.3.

Geometric realization thus extends along $\mathfrak{tri}\;\\!\\!$ to a functor

(26) $s\mathscr{S}\rightarrow\mathscr{M}$

This geometric realization functor does not commute with abstract barycentric
subdivision up to natural homeomorphism. There does exist a homeomorphism,
linear in barycentric coordinates, between a topological simplicial complex
and its topological barycentric subdivision. There even exists a
homeomorphism, quadratic in barycentric coordinates, between more general
geometric realizations of a simplicial set and its abstract barycentric
subdivision [12]. However, neither of these homeomorphisms are bi-Lipschitz
for spaces of unbounded dimension. Geometric realization (26) does commute, up
to bi-Lipschitz equivalence linear in barycentric coordinates, with abstract
edgewise subdivision [37]. But iterated edgewise subdivision, unlike iterated
baryentric subdivision, does not factor even locally through a polyhedral
complex.

###### proof of Lemma 2.8.

The inclusion $C^{\sharp}\hookrightarrow C^{\sharp\sharp}$ can be identified
with

$\eta_{\mathfrak{quad}\;\\!\\!\,\mathfrak{tri}\;\\!\\!\,C}:\mathfrak{quad}\;\\!\\!\,\mathfrak{tri}\;\\!\\!\,C\hookrightarrow\mathfrak{quad}\;\\!\\!\,\mathfrak{tri}\;\\!\\!\,\mathfrak{quad}\;\\!\\!\,\mathfrak{tri}\;\\!\\!\,C,$

where $\eta$ is the unit of the adjunction
$\mathfrak{tri}\;\\!\\!\dashv\mathfrak{quad}\;\\!\\!$ [Lemma A.1], and
therefore admits a retraction
$\mathfrak{quad}\;\\!\\!\,\epsilon_{\mathfrak{tri}\;\\!\\!\,C}$ where
$\epsilon$ is the counit of the adjunction
$\mathfrak{tri}\;\\!\\!\dashv\mathfrak{quad}\;\\!\\!$. ∎

###### proof of Lemma 2.34.

The simplicial function

$\mathfrak{tri}\;\\!\\!\,(C\hookrightarrow
C^{\sharp})=\mathfrak{tri}\;\\!\\!\,\eta_{C}:C\hookrightarrow\mathfrak{tri}\;\\!\\!\,\mathfrak{quad}\;\\!\\!\,\mathfrak{tri}\;\\!\\!\,C$

[Lemma A.1] admits a retraction natural in cubical sets $C$ by the zig-zag
identities for the adjunction
$\mathfrak{tri}\;\\!\\!\dashv\mathfrak{quad}\;\\!\\!$. Therefore
$|C\hookrightarrow C^{\sharp}|_{\sphericalangle}$ admits a $1$-Lipschitz
retraction $r_{C}$ natural in cubical sets $C$ [Lemma A.2]. ∎

##  Appendix B Categories of (co)fibrant objects

We briefly recall definitions associated to categories of (co)fibrant objects
and the basic fact that such categories admit localizations by their weak
equivalences. The reader is referred elsewhere [4] for more details, including
concrete constructions of morphisms in the homotopy category and applications
to the construction of cocycles in cohomology. A category of fibrant objects
is a category $\mathscr{H}$ with weak equivalences together with a class of
morphisms, called the weak equivalences, and a class of morphisms, called the
fibrations, such that the following conditions hold:

  1. (1)

$\mathscr{H}$ has all finite limits

  2. (2)

The weak equivalences in $\mathscr{H}$ satisfy the 2-out-of-3 axiom.

  3. (3)

Each diagonal map in $\mathscr{H}$ factors as a weak equivalence followed by a
fibration.

  4. (4)

Every terminal $\mathscr{H}$-morphism is a fibration.

  5. (5)

Fibrations in $\mathscr{H}$ are closed under pullback.

  6. (6)

The $\mathscr{H}$-morphisms that are at once fibrations and weak equivalences
are closed under pullbacks.

Consider a category $\mathscr{H}$ of fibrant objects. Call a
$\mathscr{H}$-morphism $\zeta$ an acyclic fibration if $\zeta$ is at once a
fibration and weak equivalence. A $\mathscr{H}$-morphism is a cofibration if
it has the left lifting property against all acyclic fibrations. Suppose
further that $\mathscr{H}$ has an initial object. Then the cofibrant objects
are those $\mathscr{H}$-objects $o$ for which every $\mathscr{H}$-morphism of
the form $o\rightarrow x$ lifts against every acyclic fibration to $x$.

######  Example B.1.

A model category with all objects are fibrant is a category of fibrant
objects.

Every category $\mathscr{H}$ of fibrant objects admits a localization

$\mathscr{H}\rightarrow h\mathscr{H}$

by the weak equivalences. Under some mild assumptions, the h-equivalences and
h-fibrations form a model structure [3]. Under no assumptions, the
h-equivalences and h-fibrations form a category of fibrant objects.

######  Example B.2.

A bicomplete $\mathscr{T}$-category is a category of fibrant objects in which
…

  1. (1)

weak equivalences are the h-equivalences

  2. (2)

fibrations are the h-fibrations

The verification of all but the factorization axiom is formal. The
factorization axiom essentially follows from observations elsewhere [29, Lemma
4.3.1 (iii)] but for the inconsequential detail that $\mathscr{T}$, while
complete, cocomplete, and Cartesian closed, is not the usual category of weak
Hausdorff k-spaces.

A category of cofibrant objects is a category $\mathscr{H}$ together with a
class of morphisms, called the weak equivalences, and a class of morphisms,
called the cofibrations, such that $\mathscr{H}^{\mathrm{op}}$ is a category
of fibrant objects with the same weak equivalences but where the fibrations
are the cofibrations in $\mathscr{H}$. Consider a category $\mathscr{H}$ of
cofibrant objects. Call a $\mathscr{H}$-morphism $\zeta$ an acyclic
cofibration if $\zeta$ is at once a cofibration and weak equivalence. Suppose
further that $\mathscr{H}$ has a terminal object. Then the fibrant objects in
$\mathscr{H}$ are those $\mathscr{H}$-objects $o$ for which every
$\mathscr{H}$-morphism to $o$ extends along every acyclic cofibration. By
duality, every category $\mathscr{H}$ of cofibrant objects admits a
localization

$\mathscr{H}\rightarrow h\mathscr{H}$

by the weak equivalences.

## References

  * [1] Yves André.  Uniform Sheaves and Differential Equations.  Rendiconti del Seminario Matematico della Università di Padova, 128:345–372, 2012.
  * [2] Mohammed Bahauddin and John Thomas.  The Homology of Uniform spaces.  Canadian Journal of Mathematics, 25(3):449–455, 1973.
  * [3] Tobias Barthel and Emily Riehl.  On the Construction of Functorial Factorizations for Model Categories.  Algebraic & Geometric Topology, 13(2):1089–1124, 2013.
  * [4] Kenneth S Brown.  Abstract Homotopy Theory and Generalized Sheaf Cohomology.  Transactions of the American Mathematical Society, 186:419–458, 1973.
  * [5] Ronald Brown, Philip J. Higgins, and Rafael Sivera.  Nonabelian Algebraic Topology, volume 15 of EMS Tracts in Mathematics.  European Mathematical Society (EMS), Zürich, 2011.  Filtered spaces, crossed complexes, cubical homotopy groupoids, With contributions by Christopher D. Wensley and Sergei V. Soloviev.
  * [6] Denis-Charles Cisinski.  Les Préfaisceaux comme Modèles des Types d’Homotopie.  Astérisque, (308):xxiv+390, 2006.
  * [7] Michael Cole.  Mixing Model Structures.  Topology and its Applications, 153(7):1016–1032, 2006.
  * [8] Alain Connes and Henri Moscovici.  Conjecture de Novikov et Groupes Hyperboliques.  C. R. Acad. Sci. Paris Sér. I Math., 307(9):475–480, 1988.
  * [9] Edward B Curtis.  Simplicial Homotopy Theory.  Advances in Mathematics, 6(2):107–209, 1971.
  * [10] Samuel Eilenberg and Saunders MacLane.  Acyclic Models.  American journal of mathematics, 75(1):189–199, 1953.
  * [11] Gábor Elek.  Coarse Cohomology and lp-Cohomology.  K-theory, 13(1):1–22, 1998.
  * [12] Dieter Fritsch, Rudolf et Doll.  L’Homéomorphisme des Réalisations Gèométriques d’un Ensemble Semi-Simplicial et sa Subdivision Normale.  archive de mathématiques, 18(5):508–512, 1967.
  * [13] SM Gersten.  A Note on Cohomological Vanishing and the Linear Isoperimetric Inequality.  preprint available at http://www. math. utah. edu/ gersten, 1996\.
  * [14] VM Gol’dshtein, VI Kuz’minov, and IA Shvedov.  De Rham Isomorphism of the lp-Cohomology of Noncompact Riemannian Manifolds.  Siberian Mathematical Journal, 29(2):190–197, 1988.
  * [15] Eric Goubault and Samuel Mimram.  Directed Homotopy in Non-Positively Curved Spaces.  arXiv preprint arXiv:1908.06684, 2019.
  * [16] Marco Grandis and Luca Mauri.  Cubical sets and their site.  Theory Appl. Categ, 11(8):185–211, 2003.
  * [17] Michael Gromov.  Volume and Bounded Cohomology.  Publications Mathématiques de l’IHÉS, 56:5–99, 1982.
  * [18] Alexander Grothendieck.  Pursuing stacks.  unpublished manuscript, 1983.
  * [19] Allen Hatcher.  Algebraic Topology.  Cambridge University Press, Cambridge, 2002.
  * [20] John Rolfe Isbell.  Uniform Spaces.  Number 12. American Mathematical Soc., 1964.
  * [21] J. F. Jardine.  Simplicial Approximation.  Theory Appl. Categ., 12:No. 2, 34–72, 2004.
  * [22] John F Jardine.  Cubical Homotopy Theory: A Beginning.  preprint, 2002.
  * [23] Daniel M Kan.  On CSS Complexes.  American Journal of Mathematics, 79(3):449–476, 1957.
  * [24] Chris Kapulkin, Zachery Lindsey, and Liang Ze Wong.  A Co-reflection of Cubical Sets into Simplicial Sets with Applications to Model Structures.  arXiv preprint arXiv:1906.09203, 2019.
  * [25] Sanjeevi Krishnan.  Cubical Approximation for Directed Topology I.  Applied Categorical Structures, 23(2):177–214, 2015.
  * [26] F William Lawvere.  Metric Spaces, Generalized Logic, and Closed Categories.  Rendiconti del seminario matématico e fisico di Milano, 43(1):135–166, 1973.
  * [27] Clara Loeh, Cristina Pagliantini, and Sebastian Waeber.  Cubical Simplicial Volume of 3-manifolds.  arXiv preprint arXiv:1508.03017, 2015.
  * [28] Georges Maltsiniotis et al.  La Catégorie Cubique avec Connexions est une Catégorie Test Stricte.  Homology, Homotopy and Applications, 11(2):309–326, 2009.
  * [29] J Peter May and Johann Sigurdsson.  Parametrized Homotopy Theory.  Number 132. American Mathematical Soc., 2006.
  * [30] Igor Mineyev.  Bounded Cohomology Characterizes Hyperbolic Groups.  The Quarterly Journal of Mathematics, 53(1):59–73, 2002.
  * [31] Crichton Ogle.  Polynomially Bounded Cohomology and the Novikov Conjecture.  arXiv preprint arXiv:1004.4680, 2010.
  * [32] Pierre Pansu.  lp-Cohomology of Symmetric Spaces.  arXiv preprint math/0701151, 2007.
  * [33] Daniel G. Quillen.  Homotopical Algebra.  Lecture Notes in Mathematics, No. 43. Springer-Verlag, Berlin-New York, 1967.
  * [34] Bobby Ramsey.  A Spectral Sequence for Polynomially Bounded Cohomology.  J. Pure Appl. Algebra, 217(6):1153–1163, 2013.
  * [35] Charles Rezk.  Fibrations and Homotopy Colimits of Simplicial Sheaves.  arXiv preprint math/9811038, 1998.
  * [36] MD Rice and GJ Tashjian.  Cartesian-closed Coreflective Subcategories of Uniform Spaces Generated by Classes of Metric Spaces.  Topology and its Applications, 15(3):301–312, 1983.
  * [37] Graeme Segal.  Configuration-spaces and Iterated Loop-spaces.  Inventiones mathematicae, 21(3):213–221, 1973.
  * [38] Norman E Steenrod.  A Convenient Category of Topological Spaces.  Michigan Mathematical Journal, 14(2):133–152, 1967.
  * [39] Andrew P Tonks.  Cubical Groups which are Kan.  Journal of pure and applied algebra, 81(1):83–87, 1992.
  * [40] Oswald Wyler.  Convenient Categories for Topology.  General Topology and its Applications, 3(3):225–242, 1973.