Standard deep learning systems require thousands or millions of examples to learn a concept, and cannot integrate new concepts easily. By contrast, humans have an incredible ability to do one-shot or few-shot learning. For instance, from just hearing a word used in a sentence, humans can infer a great deal about it, by leveraging what the syntax and semantics of the surrounding words tells us. Here, we draw inspiration from this to highlight a simple technique by which deep recurrent networks can similarly exploit their prior knowledge to learn a useful representation for a new word from little data. This could make natural language processing systems much more flexible, by allowing them to learn continually from the new words they encounter. Humans are often able to infer approximate meanings of new words from context. For example, consider the following stanza from the poem "Jabberwocky" by Lewis Carroll:He took his vorpal sword in hand: Long time the manxome foe he sought So rested he by the Tumtum tree, And stood awhile in thought.Despite the fact that there are several nonsense words, we can follow the narrative of the poem and understand approximately what many of the words mean by how they relate other words. This a vital skill for interacting with the world -we constantly need to learn new words and ideas from context. Even beyond language, humans are often able adapt quickly to gracefully accomodate situations that differ radically from what they have seen before. Complementary learning systems theory BID5 suggests that it is the interaction between a slow-learning system that learns structural features of the world (i.e. a deep-learning like system) and a fast-learning system (i.e. a memory-like system) that allows humans to adapt rapidly from few experiences.By comparison, standard deep learning systems usually require much more data to learn a concept or task, and sometimes generalize poorly BID6 . They can be trained to learn a concept in one-shot if this is their sole task (Vinyals et al., 2016, e.g.) , but this limits the types of tasks that can be performed. Furthermore, these models typically discard this information after a single use. In order for deep learning systems to be adaptible, they will need to build on their prior knowledge to learn effectively from a few pieces of information. In other words, they will need to integrate learning experiences across different timescales, as complementary learning systems theory suggests that humans and other animals do. In this paper, we explore this broad issue in the specific context of creating a useful representation for a new word based on its context. Overall, using our technique of updating only the embedding vectors of a word while training on sentences containing it and negative sampled sentences from the networks past experience seems quite effective. It allows for substantial reductions in perplexity on text containing the new word, without greatly interfering with knowledge about other words. Furthermore, it seems to be capturing more useful structure about how the word is used in context than previous approaches, and performs close to as well as full training with the word. These results are exciting beyond their potential applications to natural language processing -this technique could easily be extended to adapting systems to other types of new experiences, for example a vision network for an RL agent could have a few new filters per layer added and trained to accomodate a new type of object.Under what circumstances will this strategy fail? Complementary learning systems theory BID5 , from which we drew inspiration, suggests that information which is schema-consistent (i.e. fits in with the network's previous knowledge) can be integrated easily, whereas schemainconsistent knowledge (i.e. knowledge that differs from the network's previous experience) will cause interference. Similar principles should apply here. Our approach should work for learning a new word on a topic which is already somewhat familiar, but would likely fail to learn from a new word in a context that is not well understood. For example, it would be difficult to learn a new German word from context if the model has only experienced English.On the other hand, this perspective also provides promises. We expect that our technique would perform even better in a system that had a more sophisticated understanding of language, because it would have more prior knowledge from which to bootstrap understanding of new words. Thus it would be very interesting to apply our technique on more complicated tasks like question answering, such as BID14 , or in a grounded context, such as BID4 . We have presented a technique for doing one-or few-shot learning of word embeddings from text data: freeze all the weights in the network except the embeddings for the new word, and then optimize these embeddings for the sentence, interleaving with negative examples from network's prior experience and stopping early. This results in substantial improvement of the ability to predict the word in context, with minimal impairment of prediction of other words. This technique could allow natural language processing systems to adapt more flexibly to a changing world, like humans do. More generally, it could serve as a model for how to integrate rapid adaptation into deep learning systems.A SUPPLEMENTARY FIGURES (a) Percent change in perplexity on 10 test sentences containing new word.(b ) Percent change in perplexity on full PTB test corpus. We used the "large" model described by BID17 , and use all their hyper-parameters for the pre-training. Specifically , the model consists of 2 layers of stacked LSTMs with a hidden size of 1500 units, 35 recurrent steps, and dropout (p keep = 0.35) applied to the non-recurrent connections. The gradients were clipped to a max global norm of 10. Weights were initialized uniformly from [−0.04, 0.04].
We highlight a technique by which natural language processing systems can learn a new word from context, allowing them to be much more flexible. A technique for exploiting prior knowledge to learn embedding representations for new words with minimal data.