Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference. Natural language has a sequential overt form as spoken and written, but the underlying structure of language is not strictly sequential. This structure is usually tree-like. Linguists agree on a set of rules, or syntax, that determine this structure BID10 BID11 BID46 and dictate how single words compose to form meaningful larger units, also called "constituents" BID30 . The human brain can also implicitly acquire the latent structure of language BID14 : during language acquisition, children are not given annotated parse trees. This observation brings more interest in latent structure induction with artificial neural network approaches, which are inspired by information processing and communication patterns in biological nervous systems. From a practical point of view, integrating a tree structure into a neural network language model may be important for multiple reasons:(i ) to obtain a hierarchical representation with increasing levels of abstraction, a key feature of deep neural networks BID1 BID34 BID48 ;(ii ) to model the compositional effects of language BID30 BID54 and help with the long-term dependency problem BID1 BID56 by providing shortcuts for gradient backpropagation BID12 ;(iii ) to improve generalization via a better inductive bias and at the same time potentially reducing the need of a large amount of training data.The study of deep neural network techniques that can infer and use tree structures to form better representations of natural language sentences has received a great deal of attention in recent years BID5 BID61 BID50 BID24 BID9 BID58 BID51 .Given a sentence, one straightforward way of predicting the corresponding latent tree structure is through a supervised syntactic parser. Trees produced by these parsers have been used to guide the composition of word semantics into sentence semantics BID54 BID4 , or even to help next word prediction given previous words BID59 . However , supervised parsers are limiting for several reasons: i) few languages have comprehensive annotated data for supervised parser training; ii) in some domains, syntax rules tend to be broken (e.g. in tweets); and iii) languages change over time with use, so syntax rules may evolve.On the other hand, grammar induction, defined as the task of learning the syntactic structure from raw corpora without access to expert-labeled data, remains an open problem. Many such recent attempts suffer from inducing a trivial structure (e.g., a left-branching or right-branching tree BID58 ), or encounter difficulties in training caused by learning branching policies with Reinforcement Learning (RL) BID61 . Furthermore, some methods are relatively complex to implement and train, like the PRPN model proposed in BID50 .Recurrent neural networks (RNNs) have proven highly effective at the task of language modeling BID41 BID39 . RNNs explicitly impose a chain structure on the data. This assumption may seem at odds with the latent non-sequential structure of language and may pose several difficulties for the processing of natural language data with deep learning methods, giving rise to problems such as capturing long-term dependencies BID1 , achieving good generalization BID4 , handling negation BID54 , etc. Meanwhile, some evidence exists that LSTMs with sufficient capacity potentially implement syntactic processing mechanisms by encoding the tree structure implicitly, as shown by BID21 ; and very recently by BID33 . We believe that the following question remains: Can better models of language be obtained by architectures equipped with an inductive bias towards learning such latent tree structures?In this work, we introduce ordered neurons, a new inductive bias for recurrent neural networks. This inductive bias promotes differentiation of the life cycle of information stored inside each neuron: high-ranking neurons will store long-term information which is kept for a large number of steps, while low-ranking neurons will store short-term information that can be rapidly forgotten. To avoid a strict division between high-ranking and low-ranking neurons, we propose a new activation function, the cumulative softmax, or cumax(), to actively allocate neurons to store long/short-term information. We use the cumax() function to produce a vector of master input and forget gates ensuring that when a given neuron is updated (erased), all of the neurons that follow it in the ordering are also updated (erased). Based on the cumax() and the LSTM architecture, we have designed a new model, ON-LSTM, that is biased towards performing tree-like composition operations. Our model achieves good performance on four tasks : language modeling, unsupervised constituency parsing, targeted syntactic evaluation BID38 and logical inference BID4 . The result on unsupervised constituency parsing suggests that the proposed inductive bias aligns with the syntax principles proposed by human experts better than previously proposed models. The experiments also show that ON-LSTM performs better than standard LSTM models in tasks requiring capturing long-term dependencies and achieves better generalization to longer sequences. In this paper, we propose ordered neurons, a novel inductive bias for recurrent neural networks. Based on this idea, we propose a novel recurrent unit, the ON-LSTM, which includes a new gating mechanism and a new activation function cumax(·). This brings recurrent neural networks closer to performing tree-like composition operations, by separately allocating hidden state neurons with long and short-term information. The model performance on unsupervised constituency parsing shows that the ON-LSTM induces the latent structure of natural language in a way that is coherent with human expert annotation. The inductive bias also enables ON-LSTM to achieve good performance on language modeling, long-term dependency, and logical inference tasks.
We introduce a new inductive bias that integrates tree structures in recurrent neural networks. This paper proposes ON-LSTM, a new RNN unit that integrates the latent tree structure into recurrent models and that has good results on language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.