Learning high-quality word embeddings is of significant importance in achieving better performance in many down-stream learning tasks. On one hand, traditional word embeddings are trained on a large scale corpus for general-purpose tasks, which are often sub-optimal for many domain-specific tasks. On the other hand, many domain-specific tasks do not have a large enough domain corpus to obtain high-quality embeddings. We observe that domains are not isolated and a small domain corpus can leverage the learned knowledge from many past domains to augment that corpus in order to generate high-quality embeddings. In this paper, we formulate the learning of word embeddings as a lifelong learning process. Given knowledge learned from many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner, where the meta-learner is able to provide word context similarity information at the domain-level. Experimental results demonstrate that the proposed method can effectively learn new domain embeddings from a small corpus and past domain knowledges\footnote{We will release the code after final revisions.}. We also demonstrate that general-purpose embeddings trained from a large scale corpus are sub-optimal in domain-specific tasks. Learning word embeddings BID18 ; BID29 ; BID14 BID17 c) ; BID22 ) has received a significant amount of attention due to its high performance on many down-stream learning tasks. Word embeddings have been shown effective in NLP tasks such as named entity recognition BID26 ), sentiment analysis BID13 ) and syntactic parsing BID6 ). Such embeddings are shown to effectively capture syntactic and semantic level information associated with a given word BID14 ).The "secret sauce" of training word embedding is to turn a large scale in-domain corpus into billions of training examples. There are two common assumptions for training word embeddings: 1) the training corpus is largely available and bigger than the training data of the potential downstream learning tasks; and 2) the topic of the training corpus is closely related to the topic of the down-stream learning tasks. However , real-world learning tasks often do not meet one of these assumptions. For example , a domain-specific corpus that is closely related to a down-stream learning task may often be of limited size. If we lump different domain corpora together and train general-purpose embeddings over a large scale corpus (e.g., GloVe embeddings BID22 ) are trained from the corpus Common Crawl, which covers almost any topic on the web), the performance of such embeddings on many domain-specific tasks is sub-optimal (we show this in Section 6). A possible explanation is that although many domain words share similar meanings with the same out-of-domain words, with no in-domain awareness, dumping many out-of-domain co-occurrences as training examples may bias in-domain embeddings. (e.g., if the domain is about food, then an out-of-domain "python" as a programming language can bias "java", while the indomain word "chocolate" is more likely to help).To solve the problem of the limited domain corpus, one possible solution is to use transfer learning BID20 ) for training domain-specific embeddings BID2 ; BID31 ). However, these methods just manage to leverage out-of-domain embeddings trained from a large scale corpus to help limited in-domain corpus. The very in-domain corpus is never expanded. Also, one common assumption of these works is that a pair of similar source domain and target domain is manually identified in advance. In reality, given many domains , manually catching useful information in so many domains are very hard. In contrast, we humans learn the meaning of a word more smartly. We accumulate different domain contexts for the same word. When a new learning task comes , we may quickly identify the new domain contexts and borrow the word meanings from existing domain contexts. This is where lifelong learning comes to the rescue. Lifelong machine learning (LML) is a continual learning paradigm that retains the knowledge learned in past tasks 1, . . . , n, and uses it to help learning the new task n + 1 BID28 ; BID27 ; BID4 ). In the setting of word embedding : we assume that the learning system has seen n domain corpora: (D 1 , . . . , D n ), when a new domain corpus D n+1 comes by demands from that domain's potential down-stream learning tasks, the learning system can automatically generate word embeddings for the n + 1-th domain by effectively leveraging useful past domain knowledge.The main challenges of this task are 2 fold. 1) How to identify useful past domain knowledge to train the embeddings for the new domain. 2) How to automatically identify such kind of information, without help from human beings. To tackle these challenges, the system has to learn how to identify similar words in other domains for a given word in a new domain. This, in general, belongs to metalearning BID30 ; BID21 ). Here we do not focus on specific embedding learning but focus on learning how to characterize corpora of different domains for embedding purpose.The main contributions of this paper can be summarized as follows: 1) we propose the problem of lifelong word embedding, which may benefit many down-stream learning tasks. We are not aware of any existing work on word embedding using lifelong learning 2) we propose a lifelong embedding learning method , which leverages meta-learning to aggregate useful knowledge from past domain corpora to generate embeddings for the new domain. In this paper, we formulate a lifelong word embedding learning process. Given many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner. The meta-learner is able to provide word context similarity information on domain-level. Such information can help to accumulate new domain-specific training corpus in order to get better embedding. Experimental results show that the proposed method is effective in learning new domain embeddings from a small corpus and past domain knowledge.
learning better domain embeddings via lifelong learning and meta-learning Presents a lifelong learning method for learning word embeddings. This paper proposes an approach to learn embeddings in new domains and significantly beats the baseline on an aspect extraction task. 