The technology disclosed provides a so-called “joint many-task neural network model” to solve a variety of increasingly complex natural language processing (NLP) tasks using growing depth of layers in a single end-to-end model. The model is successively trained by considering linguistic hierarchies, directly connecting word representations to all model layers, explicitly using predictions in lower tasks, and applying a so-called “successive regularization” technique to prevent catastrophic forgetting. Three examples of lower level model layers are part-of-speech (POS) tagging layer, chunking layer, and dependency parsing layer. Two examples of higher level model layers are semantic relatedness layer and textual entailment layer. The model achieves the state-of-the-art results on chunking, dependency parsing, semantic relatedness and textual entailment.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method of training a stacked LSTM sequence processor, running on hardware, stacked in at least three layers according to an analytical hierarchy, the training method including: training first, second and third layers by backward propagation using training examples directed to each layer, with regularized pass down to underlying layers during training, including: training the first layer using first layer training examples, training the second layer using second layer training examples with regularized pass down training to the first layer, and training the third layer using third layer training examples with regularized pass down training to the first and second layers; and regularized pass down training is regularized by constraining a training objective function, having a fitness function with at least two regularization terms; and the two regularization terms regularize by penalizing growth in a magnitude of weights in coefficient matrices applied to the underlying layers and that successively regularize all changes in the weights in the coefficient matrices applied to the underlying layers. 2. The device component of claim 1: wherein the fitness function is negative log likelihood based cross-entropy. 3. The device component of claim 1, wherein the fitness function is Kullback-Leibler divergence (KL-divergence). 4. The device component of claim 1, wherein the fitness function is mean squared error. 5. The method of claim 1, wherein the fitness function is - ∑ s  ∑ t  log   p  ( y t ( n ) = α | h t ( n ) ) + regularization   terms where (n) denotes the n-th layer of the stacked LSTM, and p(yt(n)=α|ht(n) denotes the probability value that the correct label α is assigned to wt in the sentence s. 6. The method of claim 1, wherein the regularization term that penalizes growth in a magnitude of weights in coefficient matrices applied to the underlying layers is λ∥W(m)∥2 where (m), which is the same layer as n, denotes the layers 1 to m of the stacked LSTM, λ is a L2-norm regularization hyperparameter, and ∥w(m)∥ applies the squaring operation element wise to elements of weighting matrices for the layers 1 to m of the stacked LSTM. 7. The method of claim 1, wherein the successive regularization term δ∥θ(m−1)−θ(m−1)′∥2 where (m−1), which is the same layer as n−1, denotes the layers 1 to m−1 of the stacked LSTM, δ is a successive regularization hyperparameter, θ(m−1) denotes layer parameters of one or more underlying layers, θ(m−1)′ denotes layer parameters of one or more underlying layers persisted in a previous sub-epoch, and ∥θ(m−1)−θ(m−1)′∥ applies the squaring operation element wise to elements of a weighting matrix for the layers 1 to m−1 of the stacked LSTM. 8. The method of claim 1, further including at least five layers according to the analytical hierarchy in the stacked LSTM. 9. The method of claim 1, further including at least ten layers according to the analytical hierarchy in the stacked LSTM. 10. The method of claim 1, further basement layers below the stacked LSTM that are trained separately from the stacked LSTM and that produce input used by a lowest layer of the stacked LSTM. 11. The method of claim 1, further attic layers above the stacked LSTM that are trained separately from the stacked LSTM and that consume output from an upper most layer of the stacked LSTM. 12. A method of conveying intermediate results from an underlying layer to an overlying layer in a neural network stack of bidirectional LSTMs, in which the stack has layers corresponding to an analytical framework that process a sequence of tokens, and the underlying layer produces analytic framework label vectors for each of the tokens, the method including: for the sequence, analyzing the tokens using the underlying layer, including: applying the bidirectional LSTM to compute forward and backward state vectors for each of the tokens; applying a classifier to the forward and backward state vectors to embed each of the tokens in an analytic framework label space, as label space vectors that have dimensionality about the same as a number of available analytical framework labels; and projecting the label space vectors of each token into an extended dimensionality label space, which has dimensionality about the same as dimensionality of the forward and backward states, to produce extended token label vectors; and conveying, from the underlying layer to the overlying layer, vectors of the forward state, the backward state, and the extended token label, thereby supplying input needed by the overlying layer to perform its role in the analytical framework for processing the tokens. 13. The method of claim 12, further including conveying by a bypass to the overlaying layer, vectors received as input by the underlying layer, other than state vectors. 14. The method of claim 12, wherein the underlying layer is over two deeper layers, further including conveying by bypasses to the overlaying layer, vectors received as input by the two deeper layers and embedded label vectors produced as output by the two deeper layers. 15. The method of claim 12, wherein the number of available analytical framework labels is one-fifth or less the dimensionality of the forward and backward states, thereby forming a dimensionality bottleneck that reduces overfitting when training the neural network stack of bidirectional LSTMs. 16. The method of claim 12, wherein the number of available analytical framework labels is one-tenth or less the dimensionality of the forward and backward states, thereby forming a dimensionality bottleneck that reduces overfitting when training the neural network stack of bidirectional LSTMs. 17. A multi-layer neural network system running on hardware that processes a sequence of tokens in an input sequence, the system including: a stacked LSTM token sequence processor, running on hardware, stacked in layers according to an analytical hierarchy, with bypass connections that deliver input to underlying layers together with embedding outputs of the underlying layers to overlaying layers, the stacked layers including: a first embedding layer; and a second embedding layer overlying the first embedding layer; the first embedding layer, implemented as a bi-directional LSTM and a first label classifier, processes token embeddings representing the tokens in the input sequence, and produces analytic framework label vectors for each of the tokens, further including: the bi-directional LSTM computes forward and backward state vectors for each of the tokens; a classifier applied to the forward and backward state vectors embeds each of the tokens in an analytic framework label space, as label space vectors that have dimensionality about the same as a number of available analytical framework labels; and the first embedding layer projects the label space vectors of each token into an extended dimensionality label space, which has dimensionality about the same as dimensionality of the forward and backward states, to produce extended token label vectors; and the first embedding layer sending to the second embedding layer, vectors of the forward state, the backward state, and the extended token label, thereby supplying input needed by the second embedding layer to perform its role in the analytical framework for processing the tokens. 18. The system of claim 17, further including a bypass to the second embedding layer that conveys vectors received as input by the first embedding layer, other than state vectors. 19. The system of claim 17, wherein the first embedding layer is over two deeper layers, further including conveying by bypasses to the second embedding layer vectors received as input by the two deeper layers and embedded label vectors produced as output by the two deeper layers. 20. The system of claim 18, wherein a bypass conveys vectors without modification. 21. The system of claim 17, wherein the number of available analytical framework labels is one-fifth or less the dimensionality of the forward and backward states, thereby forming a dimensionality bottleneck that reduces overfitting when training the neural network stack of bidirectional LSTMs. 22. The system of claim 17, wherein the number of available analytical framework labels is one-tenth or less the dimensionality of the forward and backward states, thereby forming a dimensionality bottleneck that reduces overfitting when training the neural network stack of bidirectional LSTMs.