Entailment vectors are a principled way to encode in a vector what information is known and what is unknown.   They are designed to model relations where one vector should include all the information in another vector, called entailment.   This paper investigates the unsupervised learning of entailment vectors for the semantics of words.   Using simple entailment-based models of the semantics of words in text (distributional semantics), we induce entailment-vector word embeddings which outperform the best previous results for predicting entailment between words, in unsupervised and semi-supervised experiments on hyponymy.
 Modelling entailment, is a fundamental issue in the semantics of natural language, and there has been a lot of interest in modelling entailment using vector-space representations. But, until recently, unsupervised models such as word embeddings have performed surprisingly poorly at detecting entailment BID11 ; BID9 , not beating a frequency baseline BID11 . Entailment is the relation of information inclusion, meaning that y entails x if and only if everything that is known given x is also known given y. As such, representations which support entailment need to encode not just what information is known, but also what information is unknown. The results on lexical entailment seem to indicate that standard word embeddings, such as Word2Vec, do not reflect the relative abstractness of words, and in this sense do not reflect how much information is left unspecified by a word.In contrast with the majority of the work in this area, which simply uses existing vector-space embeddings of words in their models of entailment, recent work has addressed this issue by proposing new vector-space models which are specifically designed to capture entailment. In particular, BID10 use variances to represent the uncertainty in values in a continuous space, and BID4 use probabilities to represent uncertainty about a discrete space. We will refer to the latter as the "entailment-vectors" framework. In this work, we use this framework from BID4 to develop new entailment-based models for the unsupervised learning of word embeddings, and demonstrate that these embeddings achieve unprecedented results in predicting entailment between words.Our unsupervised models use the distribution of words in a large text corpus to induce vector-space representations of the meaning of words. This approach to word meaning is called distributional semantics. The distributional semantic hypothesis BID3 says that the meaning of a word is reflected in the distribution of text contexts which it appears in. Many methods (e.g. BID2 BID8 BID6 and this paper) have been proposed for inducing vector representations of the meaning of words (word embeddings) from the distribution of wordcontext pairs found in large corpora of text.In the framework of BID4 , each dimension of the vector-space represents something that might be known, and continuous vectors represent probabilities of these features being known or unknown. BID4 illustrate their framework by proposing a reinterpretation of existing Word2Vec BID6 word embeddings which maps them into entailment vectors, which in turn successfully predict entailment between words (hyponymy). To motivate this reinterpretation of existing word embeddings, they propose a model of distributional semantics and argue that the Word2Vec training objective approximates the training objective of this distributional semantic model given the mapping.In this paper, we implement this distributional semantic model and train new word embeddings using the exact objective. Based on our analysis of this model, we propose that this implementation can be done in several ways, including the one which motivates BID4 's reinterpretation of Word2Vec embeddings. In each case, training results in entailment vector embeddings, which directly encode what is known and unknown given a word, and thus do not require any reinterpretation to predict hyponymy.To model the semantic relationship between a word and its context, the distributional semantic model postulates a latent pseudo-phrase vector for the unified semantics of the word and its neighbouring context word. This latent vector must entail the features in both words' vectors and must be consistent with a prior over semantic vectors, thereby modelling the redundancy and consistency between the semantics of two neighbouring words.Based on our analysis of this entailment-based distributional semantic model, we hypothesise that the word embeddings suggested by BID4 are in fact not the best way to extract information about the semantics of a word from this model. They propose using a vector which represents the evidence about known features given the word (henceforth called the likelihood vectors). We propose to instead use a vector which represents the posterior distribution of known features for a phrase containing only the word. This posterior vector includes both the evidence from the word and its indirect consequences via the constraints imposed by the prior. Our efficient implementation of this model allows us to test this hypothesis by outputting either the likelihood vectors or the posterior vectors as word embeddings.To evaluate these word embeddings, we predict hyponymy between words, in both an unsupervised and semi-supervised setting. Given the word embeddings for two words, we measure whether they are a hypernym-hyponym pair using an entailment operator from BID4 applied to the two embeddings. We find that using the likelihood vectors performs as well as reinterpreting Word2Vec embeddings, confirming the claims of equivalence by BID4 . But we also find that using the posterior vectors performs significantly better, confirming our hypothesis that posterior vectors are better, and achieving the best published results on this benchmark dataset. In addition to these unsupervised experiments, we evaluate in a semi-supervised setting and find a similar pattern of results, again achieving state-of-the-art performance.In the rest of this paper, section 2 presents the formal framework we use for modelling entailment in a vector space, the distributional semantic models, and how these are used to predict hyponymy. Section 3 discusses additional related work, and then section 4 presents the empirical evaluation on hyponymy detection, in both unsupervised and semi-supervised experiments. Some additional analysis of the induced vectors is presented in section 4.4. The relative success of our distributional semantic models at unsupervised hyponymy detection indicates that they are capturing some aspects of lexical entailment. But the gap between the unsupervised and semi-supervised results indicates that other features are also being captured. This is not surprising, since many other factors influence the co-occurrence statistics of words. Table 3 : Ranking of the abstractness (0 > X) of frequent words from the hyponymy dataset, using Word2Hyp-Skipgram-posterior embeddings.To get a better understanding of these word embeddings, we ranked them by degree of abstractness. Table 3 shows the most abstract and least abstract frequent words that occur in the hyponymy data.To measure abstractness, we used our best unsupervised embeddings and measured how well they are entailed by the zero log-odds vector, which represents a uniform half probability of knowing each feature. For a vector to be entailed by the zero vector, it must be that its features are mostly probably unknown. The less you know given a word, the more abstract it is.An initial ranking found that six of the top ten abstract words had frequency less than 300 in the Wikipedia data, but none of the ten least abstract terms were infrequent. This indicates a problem with the current method, since infrequent words are generally very specific (as was the case for these low-frequency words, submissiveness, implementer, overdraft, ruminant, warplane, and londoner).Although this is an interesting characteristic of the method, the terms themselves seem to be noise, so we rank only terms with frequency greater than 300.The most abstract terms in table 3 include some clearly semantically abstract terms, in particular something and anything are ranked highest. Others may be affected by lexical ambiguity, since the model does not disambiguate words by part-of-speech (such as end, good, sense, back, and saw). The least abstract terms are mostly very semantically specific, but it is indicative that this list includes primate, which is an abstract term in Zoology but presumably occurs in very specific contexts in Wikipedia. In this paper, we propose unsupervised methods for efficiently training word embeddings which capture semantic entailment. This work builds on the work of BID4 , who propose the entailment-vectors framework for modelling entailment in a vector-space, and a distributional semantic model for reinterpreting Word2Vec word embeddings. Our contribution differs from theirs in that we provide a better understanding of their distributional semantic model, we choose different vectors in the model to use as word embeddings, and we train new word embeddings using our modification of the Word2Vec code. Empirical results on unsupervised and semi-supervised hyponymy detection confirm that the model's likelihood vectors, which BID4 suggest to use, do indeed perform equivalently to their reinterpretation of Word2Vec vectors. But these experiments also show that the model's posterior vectors, which we propose to use, perform significantly better, outperforming all previous results on this benchmark dataset.The success of these unsupervised models demonstrates that the proposed distributional semantic models are effective at extracting information about lexical entailment from the redundancy and consistency of words with their contexts in large text corpora. The use of the entailment-vectors framework to efficiently model entailment relations has been crucial to this success. This result suggests future work using the entailment-vectors framework in unsupervised models that leverage other distributional evidence about semantics, particularly in models of compositional semantics. The merger of word embeddings with compositional semantics to get representation learning for larger units of text is currently an important challenge in the semantics of natural language, and the work presented in this paper makes a significant contribution towards solving it.
We train word embeddings based on entailment instead of similarity, successfully predicting lexical entailment. The paper presents a word embedding algorithm for lexical entailment which follows the work of Henderson and Popa (ACL, 2016).