The technology disclosed provides a so-called “joint many-task neural network model” to solve a variety of increasingly complex natural language processing (NLP) tasks using growing depth of layers in a single end-to-end model. The model is successively trained by considering linguistic hierarchies, directly connecting word representations to all model layers, explicitly using predictions in lower tasks, and applying a so-called “successive regularization” technique to prevent catastrophic forgetting. Three examples of lower level model layers are part-of-speech (POS) tagging layer, chunking layer, and dependency parsing layer. Two examples of higher level model layers are semantic relatedness layer and textual entailment layer. The model achieves the state-of-the-art results on chunking, dependency parsing, semantic relatedness and textual entailment.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A multi-layer neural network system running on hardware that processes words in an input sentence, the system including: a stacked long-short-term-memory (abbreviated LSTM) sentence processor, running on hardware, stacked in layers according to a linguistic hierarchy, with bypass connections that deliver input to underlying layers together with embedding outputs of the underlying layers to overlaying layers, the stacked layers including: a part-of-speech (abbreviated POS) label embedding layer; a chunk label embedding layer overlying the POS label embedding layer; and a dependency parent identification and dependency relationship label embedding layer overlying the chunk label embedding layer; the POS label embedding layer, implemented as a bi-directional LSTM and a POS label classifier, processes word embedding vectors representing the words in the input sentence, and produces POS label embedding vectors and POS state vectors for each of the words; the chunk label embedding layer, implemented as a bi-directional LSTM and a chuck label classifier, processes at least the word embedding vectors, the POS label embedding vectors and the POS state vectors, to produce chunk label embeddings and chunk state vectors; the dependency parent identification and dependency relationship label embedding layer, implemented as a bi-directional LSTM and one or more classifiers, processes the word embeddings, the POS label embeddings, the chunk label embeddings and the chunk state vectors, to identify dependency parents of each of the words in the sentence to produce dependency relationship labels or label embeddings of relationships between the words and respective potential parents of the words; and an output processor that outputs at least results reflecting the identification of dependency parents and production of dependency relationship label embeddings for the words in the sentence. 2. The system of claim 1, further including: the POS label embedding layer further processes n-character-gram embedding vectors that represent the words in the input sentence, in addition to the word embedding vectors; and the bypass connections further deliver the n-character-gram embedding vectors to the chunk label embedding layer and the dependency parent and dependency relationship label embedding layer as input to respective bi-directional LSTMs in those overlaying layers. 3. The system of claim 1, further including: the POS label embedding layer further produces POS label probability mass vectors, by exponential normalization of a non-linear transformation of the POS state vectors, and produces the POS label embedding vectors, from the POS label probability mass vectors; the chunk label embedding layer further produces chunk label probability mass vectors, by exponential normalization of the chunk label state vectors, and produces the chunk label embedding vectors from the chunk label probability mass vectors; the dependency parent identification and dependency relationship label embedding layer further produces parent label probability mass vectors by classification and exponential normalization of parent label state vectors produced by the bi-directional LSTM; produces the parent label embedding vectors from the parent label probability mass vectors; produces dependency relationship label probability mass vectors by classification and exponential normalization of the parent label state vectors and the parent label embedding vectors; and produces the dependency relationship label embedding vectors from the dependency relationship label probability mass vectors; and dimensionalities of the POS label embedding vectors, the chunk label embedding vectors, and the dependency relationship label embedding vectors are similar, within +/− ten percent. 4. The system of claim 1, further including a word embedding processor, underlying the POS label embedding layer, that includes: a word embedder and an n-character-gram embedder; the word embedder maps the words in the sentence, when recognized, into a word embedding space represented by a word embedding vector; the n-character-gram embedder: processes character substrings of the word at multiple scales of substring length; maps each processed character substring into an intermediate vector representing a position in a character embedding space; and combines the intermediate vectors for each unique processed character substring to produce character embedding vectors for each of the words; and the sentence embedding processor combines results of the word embedder and the n-character-gram embedder, whereby a word not previously mapped into the word embedding space is nonetheless represented by the character embedding vector. 5. The system of claim 4, wherein the n-character-gram embedder combines the intermediate vectors to produce element wise average in the character embedding vector. 6. The system of claim 1, further including operating the system without a beam search in the POS label embedding layer, the chunk label embedding layer or the dependency parent identification and dependency relationship label embedding layer. 7. The system of claim 1, further including: a semantic relatedness layer, overlying the dependency parent identification and dependency relationship label embedding layer, that includes a relatedness vector calculator and a relatedness classifier; the semantic relatedness layer operates on pairs of first and second sentences processed through the system of claim 1; the relatedness vector calculator calculates a sentence-level representation of each of the first and second sentences, including: a bi-directional LSTM calculation of forward and backward state vectors for each of the words in the respective sentences; and an element-wise max pooling calculation over the forward and backward state vectors for the words in the respective sentences to produce sentence-level state vectors representing the respective sentences; and the relatedness vector calculator further calculates an element-wise sentence-level relatedness vector that is processed by the relatedness classifier to derive a categorical classification of relatedness between the first and second sentences. 8. The system of claim 7, further including: an entailment layer, overlying the semantic relatedness layer, that includes an entailment vector calculator and an entailment classifier; the entailment vector calculator calculates a sentence-level representation of each of the first and second sentences, including: a bi-directional LSTM calculation of forward and backward state vectors for each of the words in the respective sentences; and an element-wise max pooling calculation over the forward and backward state vectors for the words in the respective sentences to produce sentence-level state vectors representing the respective sentences; and the entailment vector calculator further calculates an element-wise sentence-level entailment vector that is processed by the entailment classifier to derive a categorical classification of entailment between the first and second sentences. 9. A multi-layer neural network system running on hardware that processes a sequence of tokens in an input sequence, the system including: a stacked LSTM token sequence processor, running on hardware, stacked in layers according to an analytical hierarchy, with bypass connections that deliver input to underlying layers together with embedding outputs of the underlying layers to overlaying layers, the stacked layers including: a first embedding layer; a second embedding layer overlying the first embedding layer; and a third embedding layer overlying the second embedding layer; the first embedding layer, implemented as a bi-directional LSTM and a first label classifier, processes token embeddings representing the tokens in the input sequence, and produces first embeddings and first state vectors of the tokens, the second embedding layer, implemented as a bi-directional LSTM and a second label classifier, processes at least the token embeddings, the first label embeddings and first state vectors, to produce second label embeddings and second state vectors, the third embedding layer, implemented as a bi-directional LSTM, processes at least the token embeddings, the first label embeddings, the second label embeddings and the second state vectors to produce third label embeddings and third state vectors, and an output processor that outputs at least results reflecting the third label embeddings for the tokens in the input sequence. 10. A method that processes words in an input sentence using a stacked layer long-short-term-memory (abbreviated LSTM) sentence processor running on hardware, stacked in layers according to a linguistic hierarchy, the stacked layers including: a part-of-speech (abbreviated POS) label embedding layer; a chunk label embedding layer overlying the POS label embedding layer; and a dependency parent identification and dependency relationship label embedding layer overlying the chunk label embedding layer; the method including: delivering, via bypass connections, input used by underlying layers together with embedding outputs from the underlying layers to overlaying layers; in the POS label embedding layer, applying a bi-directional LSTM and a POS label classifier to process word embedding vectors representing the words in the input sentence, and producing POS label embedding vectors and POS state vectors for each of the words; in the chunk label embedding layer, applying a bi-directional LSTM and a chuck label classifier to process at least the word embedding vectors, the POS label embedding vectors and the POS state vectors, and producing chunk label embeddings and chunk state vectors; in the dependency parent identification and dependency relationship label embedding layer, applying a bi-directional LSTM and one or more classifiers to process the word embeddings, the POS label embeddings, the chunk label embeddings and the chunk state vectors, to identify dependency parents of each of the words in the sentence and producing dependency relationship labels or label embeddings of relationships between the words and respective potential parents of the words; and outputting results reflecting the dependency relationship labels or label embeddings for the words in the sentence. 11. The method of claim 10, further including: in the POS label embedding layer, further processing n-character-gram embedding vectors that represent the words in the input sentence, in addition to the word embedding vectors; and the bypass connections further delivering the n-character-gram embedding vectors to the chunk label embedding layer and the dependency parent and dependency relationship label embedding layer as input to respective bi-directional LSTMs in those overlaying layers. 12. The method of claim 10, further including: in the POS label embedding layer, further producing POS label probability mass vectors, by exponential normalization of the POS state vectors, and producing the POS label embedding vectors, from the POS label probability mass vectors; in the chunk label embedding layer, further producing chunk label probability mass vectors, by exponential normalization of the chunk label state vectors, and producing the chunk label embedding vectors from the chunk label probability mass vectors; in the dependency parent identification and dependency relationship label embedding layer further: producing parent label probability mass vectors by classification and exponential normalization of parent label state vectors produced by the bi-directional LSTM; producing the parent label embedding vectors from the parent label probability mass vectors; producing dependency relationship label probability mass vectors by classification and exponential normalization of the parent label state vectors and the parent label embedding vectors; and producing the dependency relationship label embedding vectors from the dependency relationship label probability mass vectors; and dimensionalities of the POS label embedding vectors, the chunk label embedding vectors, and the dependency relationship label embedding vectors are similar, within +/− ten percent. 13. The method of claim 10, the stacked layers further including a sentence embedding layer, underlying the POS label embedding layer, that includes a word embedder and an n-character-gram embedder, the method further including: in the word embedder, mapping the words in the sentence, when recognized, into a word embedding space represented by a word embedding vector; in the n-character-gram embedder: processing character substrings of the word at multiple scales of substring length; mapping each processed character substring into an intermediate vector representing a position in a character embedding space; and combining the intermediate vectors for each unique processed character substring to produce character embedding vectors for each of the words; and the sentence embedding layer outputting vectors from the word embedder and the n-character-gram embedder, whereby a word not previously mapped into the word embedding space is nonetheless represented by the character embedding vector. 14. The method of claim 13, wherein the n-character-gram embedder combines the intermediate vectors to produce element wise average in the character embedding vector. 15. The method of claim 10, the stacked layers further including a semantic relatedness layer, overlying the dependency parent identification and dependency relationship label embedding layer, which includes a relatedness vector calculator and a relatedness classifier, the method further including: in the semantic relatedness layer, operating on pairs of first and second sentences processed through the method of claim 10; in the relatedness vector calculator, calculating a sentence-level representation of each of the first and second sentences, including: applying a bi-directional LSTM to calculate forward and backward state vectors for each of the words in the respective sentences; and calculating an element-wise maximum of the forward and backward state vectors for each of the respective sentences; in the relatedness vector calculator, further calculating an element-wise sentence-level relatedness vector; and processing the sentence-level relatedness vector to derive a categorical classification of relatedness between the first and second sentences. 16. The method of claim 15, the stacked layers further including an entailment layer, overlying the semantic relatedness layer, which includes an entailment vector calculator and an entailment classifier, the method further including: in the entailment vector calculator calculating a sentence-level representation of each of the first and second sentences, including: applying a bi-directional LSTM to calculate forward and backward state vectors for each of the words in the respective sentences; and calculating an element-wise maximum of the forward and backward state vectors for each of the respective sentences; in the entailment vector calculator, further calculating an element-wise sentence-level entailment vector; and processing the sentence-level entailment vector to derive a categorical classification of entailment between the first and second sentences. 17. A method that processes tokens in an input sequence using a stacked layer long-short-term-memory (abbreviated LSTM) sentence processor running on hardware, stacked in layers according to an analytical hierarchy, the stacked layers including: a first embedding layer; a second embedding layer overlying the first embedding layer; and a third embedding layer overlying the second embedding layer; the method including: delivering, via bypass connections, input used by underlying layers together with embedding outputs of the underlying layers to overlaying layers; the first embedding layer, applying a bi-directional LSTM and a first label classifier to process token embeddings representing the tokens in the input sequence, and producing first label embeddings and first state vectors of the tokens; the second embedding layer, applying a bi-directional LSTM and a second label classifier to process at least the token embeddings, the first label embeddings and first state vectors, to produce second label embeddings and second state vectors; the third embedding layer, applying a bi-directional LSTM, to process at least the token embeddings, the first label embeddings, the second label embeddings and the second state vectors to produce third label embeddings and third state vectors; and outputting results reflecting stacked LSTM analysis according to the analytical hierarchy, including the third label embeddings for the tokens in the input sequence.