Many methods have been developed to represent knowledge graph data, which implicitly exploit low-rank latent structure in the data to encode known information and enable unknown facts to be inferred. To predict whether a relationship holds between entities, their embeddings are typically compared in the latent space following a relation-specific mapping. Whilst link prediction has steadily improved, the latent structure, and hence why such models capture semantic information, remains unexplained. We build on recent theoretical interpretation of word embeddings as a basis to consider an explicit structure for representations of relations between entities. For identifiable relation types, we are able to predict properties and justify the relative performance of leading knowledge graph representation methods, including their often overlooked ability to make independent predictions. Knowledge graphs are large repositories of binary relations between words (or entities) in the form of fact triples (subject, relation, object). Many models have been developed for learning representations of entities and relations in knowledge graphs, such that known facts can be recalled and previously unknown facts can be inferred, a task known as link prediction. Recent link prediction models (e.g. Bordes et al., 2013; Trouillon et al., 2016; Balažević et al., 2019b ) learn entity representations, or embeddings, of far lower dimensionality than the number of entities, by capturing latent structure in the data. Relations are typically represented as a mapping from the embedding of a subject entity to its related object entity embedding(s). Although the performance of knowledge graphlink prediction models has steadily improved for nearly a decade, relatively little is understood of the low-rank latent structure that underpins these models, which we address in this work. We start by drawing a parallel between entity embeddings in knowledge graphs and unsupervised word embeddings, as learned by algorithms such as Word2Vec (W2V) (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) . We assume that words have latent features, e.g. meaning(s), tense, grammatical type, that are innate and fixed, irrespective of what an embedding may capture (which may be only a part, subject to the embedding method and/or the data source); and that this same latent structure gives rise to patterns observed in the data, e.g. in word co-occurrence statistics and in which words are related to which. As such, an understanding of the latent structure from one embedding task (e.g. word embedding) might be useful to another (e.g. knowledge graph entity embedding). Recent work theoretically explains how semantic properties are encoded in word embeddings that (approximately) factorise a matrix of word cooccurrence pointwise mutual information (PMI), e.g. as is known for W2V (Levy & Goldberg, 2014) . Semantic relationships between words (specifically similarity, relatedness, paraphrase and analogy) are proven to manifest as linear relationships between rows of the PMI matrix (subject to known error terms), of which word embeddings can be considered low-rank projections. This explains why similar words (e.g. synonyms) have similar embeddings; and embeddings of analogous word pairs share a common "vector offset". Importantly, this insight allows us to identify geometric relationships between such word embeddings necessary for other semantic relations to hold, such as those of knowledge graphs. These relation conditions describe relation-specific mappings between entity embeddings, i.e. relation representations, providing a "blue-print" against which to consider knowledge graph representation models. We find that various properties of knowledge graph representation models, including the relative DistMult (Yang et al., 2015) multiplicative (diagonal) e s Re o TuckER (Balažević et al., 2019b) multiplicative W × 1 e s × 2 r × 3 e o MuRE (Balažević et al., 2019a) performance of leading link prediction models, accord with predictions based on these relation conditions, suggesting a commonality to the latent structure learned in word embedding models and knowledge graph representation models, despite the significant differences between their training data and methodology. In summary, the key contributions of this work are: • to use recent understanding of PMI-based word embeddings to derive what a relation representation must achieve to map a subject word embedding to all related object word embeddings (relation conditions), based on which relations can be categorised into three types; • to show that properties of knowledge graph models fit predictions made from relation conditions, e.g. strength of a relation's relatedness aspect is reflected in the eigenvalues of its relation matrix; • to show that the performance per relation of leading link prediction models corresponds to the ability of the model's architecture to meet the relation conditions of the relation's type, i.e. the better the architecture of a knowledge graph representation model aligns with the form theoretically derived for PMI-based word embeddings, the better the model performs; and • noting how ranking metrics can be flawed, to provide novel insight into the prediction accuracy per relation of recent knowledge graph models, an evaluation metric we recommend in future. Many models learn low-rank representations for knowledge graph link prediction, yet little is known about the latent structure they learn. We build on recent understanding of PMI-based word embeddings to theoretically establish what a relation representation must achieve to map a word embedding to those it is related to for the relations of knowledge graphs (relation conditions). Such conditions partition relations into three types and also provide a framework to assess loss functions of knowledge graph models. Any model that satisfies a relation's conditions can represent it if its entity embeddings are set to PMI-based word embeddings, i.e. a solution is known to exist. Whilst knowledge graph models do not learn the parameters of word embeddings, we show that the better a model's architecture satisfies a relation's conditions, the better its link prediction performance, fitting the premise that similar latent structure is exploited. Overall, we extend previous understanding of how semantic relations are encoded in relationships between PMI-based word embeddings -generalising from a limited set, e.g. similarity and analogy; we demonstrate commonality between the latent structure learned by PMI-based word embeddings (e.g. W2V) and knowledge graph representation models; and we provide novel insight into knowledge graph models by evaluating their predictive performance. A CATEGORISING WORDNET RELATIONS Table 7 describes how each WN18RR relation was assigned to its respective category. Carlson et al., 2010) ), which span our identified relation types (see Table 8 ). Explanation for the relation category assignment is shown in Table 9 . (Balažević et al., 2019b ).
Understanding the structure of knowledge graph representation using insight from word embeddings. This paper attempts to understand the latent structure underlying knowledge graph embedding methods, and demonstrates that a model's ability to represent a relation type depends on the model architecture's limitations with respect to relation conditions. This paper proposes a detailed study on the explainability of link prediction (LP) models by utilizing a recent interpretation of word embeddings to provide a better understanding of LPs' model performance.