Transformers have recently taken the center stage in language modeling after LSTM 's were considered the dominant model architecture for a long time . In this project , we investigate the performance of the Transformer architectures-BERT and Transformer-XL for the language modeling task . Automated design of neural network architectures tailored for a specific task is an extremely promising , albeit inherently difficult , avenue to explore . While most results in this domain have been achieved on image classification and language modelling problems , here we concentrate on dense per-pixel tasks , in particular , semantic image segmentation using fully convolutional networks . In contrast to the aforementioned areas , the design choices of a fully convolutional network require several changes , ranging from the sort of operations that need to be used - e.g. , dilated convolutions - to a solving of a more difficult optimisation problem . The second idea is of a technical nature : using patterns as a practical foundation for computing . The goal is to show that the systematic creation and layering of languages can be reduced to the elementary operations of pattern matching and instantiation and that this pattern-based approach provides a formal and practical foundation for language-driven modelling , programming and analysis . The underpinning of the work is a novel formalism for recognising , deconstructing , creating , searching , transforming and generally manipulating data structures . Typically , they are trained using word co-occurrence statistics from a large sample of text . In some language modelling applications , such as automatic speech recognition ( ASR ) , the availability of acoustic data provides an additional source of knowledge . This contains , amongst other things , the melodic and rhythmic aspects of speech referred to as prosody . The smoothing of n-gram models is a core technique in language modelling ( LM ) . Modified Kneser-Ney ( mKN ) ranges among one of the best smoothing techniques . We present a description of the development and evaluation of a first South African broadcast news transcription system . We describe a number of speech resources which have been collected in the resource-scarce South African environment for system development purposes : a 20 hour corpus of South African English ( SAE ) broadcast news ; a 109 M word corpus of South African newspaper text collected for language modelling purposes ; and a 60k word SAE pronunciation dictionary . The development of our system is based on similar state-of-the-art broadcast news transcription systems . Animating sign language requires both a model of the structure of the target language and a computer animation system capable of producing the resulting avatar motion . On the language modelling side , AZee proposes a methodology and formal description mechanism to build grammars of Sign languages . It has mostly assumed the existence of an avatar capable of rendering its low-level articulation specifications . The article presents an example of practical application of deep learning methods for language processing and modelling . Development of statistical language models helps to predict a sequence of recognized words and phonemes , and can be used for improving speech processing and speech recognition . Neural language models can also be used as contextualized word representations . Thus , language modeling is a good benchmark task for investigating the general frameworks of neural methods in NLP field . In language modeling , we compute joint probability using the product of conditional probabilities . section : Language Results Language modeling is a challenging problem due to the complex and long - range interactions between distant words [ reference ] . One hope is that large / deep models might be able to capture these complex interactions , but large models easily overfit on these tasks and exhibit large gaps between training set and testing set performance . CBS schedules effectively help us avoid overfitting , and in addition snapshot ensembling enables even greater performance . paragraph : Unsupervised Pretraining . The language modeling objective can be summarized as a maximum likelihood estimation problem for a sequence of tokens . We treat our model as a function with two parts : an encoder and decoder . Each layer is wrapped in a residual block that contains additional convolutional layers with filters of size . We adopt two variants of the residual blocks : one with ReLUs , which is used in the machine translation experiments , and one with Multiplicative Units , which is used in the language modelling experiments . Figure [ reference ] diagrams the two variants of the blocks . After training on Wikipedia corpora , our system was evaluated on the multilingual subtask of SemEval-2017 Task 2 and achieved a good level of performance , despite its great simplicity . Our results tend to demonstrate the power of the distributional approach in semantic similarity tasks , even without knowledge of the underlying language . We also show that dimensionality reduction has a considerable impact on the results . As a result , by considering the best run of each of the six teams , we have been able to achieve the best accuracy and F1 values in entailment recognition , in the Brazilian Portuguese set , and the best F1 score overall . In the semantic similarity task , our team was ranked second in the Brazilian Portuguese set , and third considering both sets . This citation linkage problem has been modeled as a semantic relatedness task where given a citing sentence the framework pairs this citing sentence with each sentence from the reference document and then tries to determine which sentence pair is semantically similar and which pair is not . Construction of the citation linkage framework involves corpus creation and utilizing deep-learning models for semantic similarity measurement . We use Kendall ’s τb correlation to compute vector similarities in the resulting non-Gaussian spaces . Most importantly , since E(R ) > 0 , weighting methods such as positive pointwise mutual information ( PPMI ) can be applied to PoP-constructed spaces after their construction for efficiently transferring PoP embeddings onto spaces that are discriminative for semantic similarity assessments . Our PoP-constructed models , combined with PPMI , achieve an average score of 0.75 in the MEN relatedness test , which is comparable to results obtained by state-of-the-art algorithms . We ground word embeddings in semantic spaces studied in the cognitive-psychometric literature , taking these spaces as the primary objects to recover . To this end , we relate log co-occurrences of words in large corpora to semantic similarity assessments and show that co-occurrences are indeed consistent with an Euclidean semantic space hypothesis . Framing word embedding as metric recovery of a semantic space unifies existing word embedding algorithms , ties them to manifold learning , and demonstrates that existing algorithms are consistent metric recovery methods given co-occurrence counts from random walks . Ontologies are key to information retrieval , semantic integration of datasets , and semantic similarity analyses . Evaluating ontologies ( especially defining what constitutes a " good " or " better " ontology ) is therefore of central importance for the Semantic Web community . ( soybean ) , and Solanum lycopersicum L. ( tomato ) . We applied the same ontologies , annotation standards , formats , and best practices across all six species , thereby ensuring that the shared dataset could be used for cross-species querying and semantic similarity analyses . Curated phenotypes were first converted into a common format using taxonomically broad ontologies such as the Plant Ontology , Gene Ontology , and Phenotype and Trait Ontology . Our proposed concept semantic similarity is based on the prototype view in cognitive psychology , and it can measure the semantic similarity between two concepts in a way which is approaching human cognition . It provides a novel way to measure semantic similarity based on prototypes in concepts and provides a new perspective in semantic similarity measure . We evaluate the effectiveness of the representation via the task of semantic similarity on standard benchmarks . The experimental results show the strong correlation between the proposed word representation to the perception of human in the task of semantic similarity measure . In this work we present a semantic framework suitable of being used as support tool for recommender systems . Our purpose is to use the semantic information provided by a set of integrated resources to enrich texts by conducting different NLP tasks : WSD , domain classification , semantic similarities and sentiment analysis . After obtaining the textual semantic enrichment we would be able to recommend similar content or even to rate texts according to different dimensions . Ontology offers a structured knowledge representation and provides formal interpretation of concepts which can be used in semantic similarity measuring . In this paper , we analyze these ontology-based approaches for semantic similarity computation and propose a new ontology-based measure relying on exploiting intent of the concept . When the parsing result is not a well - formed tree , we apply the first - order Eisner ’s algorithm eisner1996 to obtain a well - formed tree from it . subsection : Semantic Task : Semantic relatedness The next two tasks model the semantic relationships between two input sentences . subsection : Experiment - II : Matching Question and Answer Matching question answering ( MQA ) is a typical task for semantic matching . Given a question , we need select a correct answer from some candidate answers . Moreover , they may need additional NLP parsers or external knowledge sources that may not be available for some languages . Recently , researchers have been studying deep learning approaches to automatically learn semantic match between questions and answers . Such methods are built on the top of neural network models such as convolutional neural networks ( CNNs ) and Long Short - Term Memory Models ( LSTMs ) . However , CNNs are originally designed for computer vision ( CV ) , which uses position - shared weights with local perceptive filters , to learn spatial regularities in many CV tasks . However , such spatial regularities may not exist in semantic matching between questions and answers , since important similarity signals between question and answer terms could appear in any position due to the complex linguistic property of natural languages . Meanwhile , models based on LSTMs view the question / answer matching problem in a sequential way . document : Convolutional Neural Network Architectures for Matching Natural Language Sentences Semantic matching is of central importance to many natural language tasks . A successful matching algorithm needs to adequately model the internal structures of language objects and the interaction between them . More recently , neural approaches have come to dominate the scene , as shown by the recent RepEval 2017 task , where all submissions relied on bidirectional LSTM models , with or without pretrained embeddings . RTE also intersects with a number of related inference problems , including semantic text similarity and Question Answering , and some models have been proposed to address several such problems . In one popular approach , both P and H are encoded within the same embedding space , using a single RNN , with a decision made based on the encodings of the two sentences . Our model even reaches AdaSent performance on CR , suggesting that having a larger coverage for the training task helps learn even better general representations . On semantic textual similarity STS14 , we are also competitive with PPDB based paragram - phrase embeddings with a pearson score of 0.70 . Interestingly , on caption - related transfer tasks such as the COCO image caption retrieval task , training our sentence encoder on other genres from MultiNLI does not degrade the performance compared to the model trained only SNLI ( which contains mostly captions ) , which confirms the generalization power of our embeddings . In this paper , we propose an ensemble model which based on both word and character level neural networks such as the convolutional neural network ( CNN ) , and the long short-term memory network ( LSTM ) for modeling semantic similarity . And we adopt an enhanced deep semantic model ( R-ESIM ) which is proved to be more effective for sentence modeling . Our model takes 10-fold cross-validation into account to improve the generalization ability . Sentence pair modeling is a crucial problem in the field of natural language processing . In this paper , we propose a model to measure the similarity of a sentence pair focusing on the interaction information . In this paper , we study how effective subwordlevel ( character and character n-gram ) representations are in sentence pair modeling . Though it is well-known that subword models are effective in tasks with single sentence input , including language modeling and machine translation , they have not been systematically studied in sentence pair modeling tasks where the semantic and string similarities between texts matter . Our experiments show that subword models without any pretrained word embedding can achieve new state-of-the-art results on two social media datasets and competitive results on news data for paraphrase identification . Long Short-Term Memory ( LSTM ) , a significant network in deep learning has made tremendous achievements in capturing semantic information and predicting the semantic relatedness of two sentences . In this study , considering the similarity between predicting the relatedness of sentence pair task and semantic search , we provide a novel channel to process semantic search task : see semantic search as an atypical predicting the relatedness of sentence pair task . Furthermore , we propose an LSTM-Based Neural Network Model which is suitable for predicting the semantic relatedness between user query and potential documents . The recurrent neural networks ( RNNs ) have shown good performance for sentence similarity modeling in recent years . Most RNNs focus on modeling the hidden states based on the current sentence , while the context information from the other sentence is not well investigated during the hidden state generation . Abstract Sentence Pair Modeling is a critical and challenging problem in natural language processing ( NLP ) . It aims to seek the underlying semantic relationship between two sentences . We make three contributions . ( i ) The ABCNN can be applied to a wide variety of tasks that require modeling of sentence pairs . ( ii ) We propose three attention schemes that integrate mutual influence between sentences into CNNs ; thus , the representation of each sentence takes into consideration its counterpart . Modeling sentence pairs plays the vital role for judging the relationship between two sentences , such as paraphrase identification , natural language inference , and answer sentence selection . Previous work achieves very promising results using neural networks with attention mechanism . This paper extends such a general line of research and its main contributions are two-fold . First , we explore several effective formulations of proximity-based cues for use in the sentence modeling process involved in the LM-based summarization framework . Second , the utilities of the methods instantiated from the LM-based summarization framework and several well-practiced state-of-the-art methods are analyzed and compared extensively . Neural network based approaches for sentence relation modeling automatically generate hidden matching features from raw sentence pairs . However , the quality of matching feature representation may not be satisfied due to complex semantic relations such as entailment or contradiction . A lot of advanced methods have been developed to solve this task . In this paper , we propose an attention aware bidirectional GRU ( Bi-GRU ) framework to classify the sentiment polarity from the aspects of sentential-sequence modeling and word-feature seizing . It is composed of a pre-attention Bi-GRU to incorporate the complicated interaction between words by sentence modeling , and an attention layer to capture the keywords for sentiment apprehension . paragraph : Roadmap We start by introducing a convolution network in Section [ reference ] as the basic architecture for sentence modeling , and how it is related to existing sentence models . Based on that , in Section [ reference ] , we propose two architectures for sentence matching , with a detailed discussion of their relation . Table [ reference ] summarizes the results of model variants . We can again see that the use of cell states clearly improves sentence modeling performance ( baseline vs. ( i ) and ( iv ) vs. ( i ) ) . Also from the results of baseline and ( ii ) , we validate that the selection of does not significantly affect performance but introducing is beneficial ( baseline vs. ( iii ) ) possibly due to its effect on normalizing information from multiple sources , as mentioned in § [ reference ] . CNN has been very successful on image classification ; see e.g. , the winning solutions of ImageNet Large Scale Visual Recognition Challenge . On text , since the work on token - level applications ( e.g. , POS tagging ) by nnnlpJMLR11 , CNN has been used in systems for entity search , sentence modeling , word embedding learning , product feature mining , and so on . Notably , in many of these CNN studies on text , the first layer of the network converts words in sentences to word vectors by table lookup . The authors employ a word - by - word neural attention mechanism to reason about the entailment in two sentences . Also focused on modeling sentence pairs , the authors in propose an attention - based bigram CNN for jointly performing attention between two CNN hierarchies . In their work , three attention schemes are proposed and evaluated . NLI is in the core of natural language understanding and has wide applications in NLP , e.g. , question answering and automatic summarization . Moreover , NLI is also related to other tasks of sentence pair modeling , including paraphrase detection , relation recognition of discourse units , etc . Traditional approaches to NLI mainly fall into two groups : feature - rich models and formal reasoning methods . However , such approaches are limited in terms of scope and accuracy . The renewed prosperity of neural networks has made significant achievements in various NLP applications , including individual sentence modeling as well as sentence matching . A typical neural architecture to model sentence pairs is the “ Siamese ” structure , which involves an underlying sentence model and a matching layer to determine the relationship between two sentences . Finally , two sentences ’ information is combined by several heuristic matching layers , including concatenation , element - wise product and difference ; they are effective in capturing relationships between two sentences , but remain low complexity . To sum up , the main contributions of this paper are two - fold : ( 1 ) We are the first to introduce tree - based convolution to sentence pair modeling tasks like NLI ; ( 2 ) Leveraging additional heuristics further improves the accuracy while remaining low complexity , outperforming existing sentence encoding - based approaches to a large extent , including feature - rich methods and long short term memory ( LSTM )- based recurrent networks . section : Related Work tai2015 adjusted the standard LSTM to tree - structured topologies and obtained superior results over a sequential LSTM on related tasks . In this paper , we stack CNN and LSTM in a unified architecture for semantic sentence modeling . The combination of CNN and LSTM can be seen in some computer vision tasks like image caption and speech recognition . section : Related Work Our architecture for sentence pair encoding can be regarded as strong interaction models , which have been explored in previous models . An intuitive paradigm is to compute similarities between all the words or phrases of the two sentences . The attention weight for the ith word in the premise is the result of a weighted combination ( parameterized by w ) of values in m i . The final sentence - pair representation is obtained from a non - linear combination of the attentionweighted representation r of the premise and the last output vector h N using where W p , W x ∈ R k×k are trained projection matrices . section : Related Work Our method is motivated by the central role played by alignment in machine translation and previous approaches to sentence similarity modeling , natural language inference , and semantic parsing . The neural counterpart to alignment , attention , which is a key part of our approach , was originally proposed and has been predominantly used in conjunction with LSTMs and to a lesser extent with CNNs .
Transformers have recently taken the center stage in language modeling after LSTM 's were considered the dominant model architecture for a long time . In this project , we investigate the performance of the Transformer architectures-BERT and Transformer-XL for the language modeling task .
Automated design of neural network architectures tailored for a specific task is an extremely promising , albeit inherently difficult , avenue to explore . While most results in this domain have been achieved on image classification and language modelling problems , here we concentrate on dense per-pixel tasks , in particular , semantic image segmentation using fully convolutional networks . In contrast to the aforementioned areas , the design choices of a fully convolutional network require several changes , ranging from the sort of operations that need to be used - e.g. , dilated convolutions - to a solving of a more difficult optimisation problem .
The second idea is of a technical nature : using patterns as a practical foundation for computing . The goal is to show that the systematic creation and layering of languages can be reduced to the elementary operations of pattern matching and instantiation and that this pattern-based approach provides a formal and practical foundation for language-driven modelling , programming and analysis . The underpinning of the work is a novel formalism for recognising , deconstructing , creating , searching , transforming and generally manipulating data structures .
Typically , they are trained using word co-occurrence statistics from a large sample of text . In some language modelling applications , such as automatic speech recognition ( ASR ) , the availability of acoustic data provides an additional source of knowledge . This contains , amongst other things , the melodic and rhythmic aspects of speech referred to as prosody .
The smoothing of n-gram models is a core technique in language modelling ( LM ) . Modified Kneser-Ney ( mKN ) ranges among one of the best smoothing techniques .
We present a description of the development and evaluation of a first South African broadcast news transcription system . We describe a number of speech resources which have been collected in the resource-scarce South African environment for system development purposes : a 20 hour corpus of South African English ( SAE ) broadcast news ; a 109 M word corpus of South African newspaper text collected for language modelling purposes ; and a 60k word SAE pronunciation dictionary . The development of our system is based on similar state-of-the-art broadcast news transcription systems .
Animating sign language requires both a model of the structure of the target language and a computer animation system capable of producing the resulting avatar motion . On the language modelling side , AZee proposes a methodology and formal description mechanism to build grammars of Sign languages . It has mostly assumed the existence of an avatar capable of rendering its low-level articulation specifications .
The article presents an example of practical application of deep learning methods for language processing and modelling . Development of statistical language models helps to predict a sequence of recognized words and phonemes , and can be used for improving speech processing and speech recognition .
Neural language models can also be used as contextualized word representations . Thus , language modeling is a good benchmark task for investigating the general frameworks of neural methods in NLP field . In language modeling , we compute joint probability using the product of conditional probabilities .
section : Language Results Language modeling is a challenging problem due to the complex and long - range interactions between distant words [ reference ] . One hope is that large / deep models might be able to capture these complex interactions , but large models easily overfit on these tasks and exhibit large gaps between training set and testing set performance . CBS schedules effectively help us avoid overfitting , and in addition snapshot ensembling enables even greater performance .
paragraph : Unsupervised Pretraining . The language modeling objective can be summarized as a maximum likelihood estimation problem for a sequence of tokens . We treat our model as a function with two parts : an encoder and decoder .
Each layer is wrapped in a residual block that contains additional convolutional layers with filters of size . We adopt two variants of the residual blocks : one with ReLUs , which is used in the machine translation experiments , and one with Multiplicative Units , which is used in the language modelling experiments . Figure [ reference ] diagrams the two variants of the blocks .
After training on Wikipedia corpora , our system was evaluated on the multilingual subtask of SemEval-2017 Task 2 and achieved a good level of performance , despite its great simplicity . Our results tend to demonstrate the power of the distributional approach in semantic similarity tasks , even without knowledge of the underlying language . We also show that dimensionality reduction has a considerable impact on the results .
As a result , by considering the best run of each of the six teams , we have been able to achieve the best accuracy and F1 values in entailment recognition , in the Brazilian Portuguese set , and the best F1 score overall . In the semantic similarity task , our team was ranked second in the Brazilian Portuguese set , and third considering both sets .
This citation linkage problem has been modeled as a semantic relatedness task where given a citing sentence the framework pairs this citing sentence with each sentence from the reference document and then tries to determine which sentence pair is semantically similar and which pair is not . Construction of the citation linkage framework involves corpus creation and utilizing deep-learning models for semantic similarity measurement .
We use Kendall ’s τb correlation to compute vector similarities in the resulting non-Gaussian spaces . Most importantly , since E(R ) > 0 , weighting methods such as positive pointwise mutual information ( PPMI ) can be applied to PoP-constructed spaces after their construction for efficiently transferring PoP embeddings onto spaces that are discriminative for semantic similarity assessments . Our PoP-constructed models , combined with PPMI , achieve an average score of 0.75 in the MEN relatedness test , which is comparable to results obtained by state-of-the-art algorithms .
We ground word embeddings in semantic spaces studied in the cognitive-psychometric literature , taking these spaces as the primary objects to recover . To this end , we relate log co-occurrences of words in large corpora to semantic similarity assessments and show that co-occurrences are indeed consistent with an Euclidean semantic space hypothesis . Framing word embedding as metric recovery of a semantic space unifies existing word embedding algorithms , ties them to manifold learning , and demonstrates that existing algorithms are consistent metric recovery methods given co-occurrence counts from random walks .
Ontologies are key to information retrieval , semantic integration of datasets , and semantic similarity analyses . Evaluating ontologies ( especially defining what constitutes a " good " or " better " ontology ) is therefore of central importance for the Semantic Web community .
( soybean ) , and Solanum lycopersicum L. ( tomato ) . We applied the same ontologies , annotation standards , formats , and best practices across all six species , thereby ensuring that the shared dataset could be used for cross-species querying and semantic similarity analyses . Curated phenotypes were first converted into a common format using taxonomically broad ontologies such as the Plant Ontology , Gene Ontology , and Phenotype and Trait Ontology .
Our proposed concept semantic similarity is based on the prototype view in cognitive psychology , and it can measure the semantic similarity between two concepts in a way which is approaching human cognition . It provides a novel way to measure semantic similarity based on prototypes in concepts and provides a new perspective in semantic similarity measure .
We evaluate the effectiveness of the representation via the task of semantic similarity on standard benchmarks . The experimental results show the strong correlation between the proposed word representation to the perception of human in the task of semantic similarity measure .
In this work we present a semantic framework suitable of being used as support tool for recommender systems . Our purpose is to use the semantic information provided by a set of integrated resources to enrich texts by conducting different NLP tasks : WSD , domain classification , semantic similarities and sentiment analysis . After obtaining the textual semantic enrichment we would be able to recommend similar content or even to rate texts according to different dimensions .
Ontology offers a structured knowledge representation and provides formal interpretation of concepts which can be used in semantic similarity measuring . In this paper , we analyze these ontology-based approaches for semantic similarity computation and propose a new ontology-based measure relying on exploiting intent of the concept .
When the parsing result is not a well - formed tree , we apply the first - order Eisner ’s algorithm eisner1996 to obtain a well - formed tree from it . subsection : Semantic Task : Semantic relatedness The next two tasks model the semantic relationships between two input sentences .
subsection : Experiment - II : Matching Question and Answer Matching question answering ( MQA ) is a typical task for semantic matching . Given a question , we need select a correct answer from some candidate answers .
Moreover , they may need additional NLP parsers or external knowledge sources that may not be available for some languages . Recently , researchers have been studying deep learning approaches to automatically learn semantic match between questions and answers . Such methods are built on the top of neural network models such as convolutional neural networks ( CNNs ) and Long Short - Term Memory Models ( LSTMs ) .
However , CNNs are originally designed for computer vision ( CV ) , which uses position - shared weights with local perceptive filters , to learn spatial regularities in many CV tasks . However , such spatial regularities may not exist in semantic matching between questions and answers , since important similarity signals between question and answer terms could appear in any position due to the complex linguistic property of natural languages . Meanwhile , models based on LSTMs view the question / answer matching problem in a sequential way .
document : Convolutional Neural Network Architectures for Matching Natural Language Sentences Semantic matching is of central importance to many natural language tasks . A successful matching algorithm needs to adequately model the internal structures of language objects and the interaction between them .
More recently , neural approaches have come to dominate the scene , as shown by the recent RepEval 2017 task , where all submissions relied on bidirectional LSTM models , with or without pretrained embeddings . RTE also intersects with a number of related inference problems , including semantic text similarity and Question Answering , and some models have been proposed to address several such problems . In one popular approach , both P and H are encoded within the same embedding space , using a single RNN , with a decision made based on the encodings of the two sentences .
Our model even reaches AdaSent performance on CR , suggesting that having a larger coverage for the training task helps learn even better general representations . On semantic textual similarity STS14 , we are also competitive with PPDB based paragram - phrase embeddings with a pearson score of 0.70 . Interestingly , on caption - related transfer tasks such as the COCO image caption retrieval task , training our sentence encoder on other genres from MultiNLI does not degrade the performance compared to the model trained only SNLI ( which contains mostly captions ) , which confirms the generalization power of our embeddings .
In this paper , we propose an ensemble model which based on both word and character level neural networks such as the convolutional neural network ( CNN ) , and the long short-term memory network ( LSTM ) for modeling semantic similarity . And we adopt an enhanced deep semantic model ( R-ESIM ) which is proved to be more effective for sentence modeling . Our model takes 10-fold cross-validation into account to improve the generalization ability .
Sentence pair modeling is a crucial problem in the field of natural language processing . In this paper , we propose a model to measure the similarity of a sentence pair focusing on the interaction information .
In this paper , we study how effective subwordlevel ( character and character n-gram ) representations are in sentence pair modeling . Though it is well-known that subword models are effective in tasks with single sentence input , including language modeling and machine translation , they have not been systematically studied in sentence pair modeling tasks where the semantic and string similarities between texts matter . Our experiments show that subword models without any pretrained word embedding can achieve new state-of-the-art results on two social media datasets and competitive results on news data for paraphrase identification .
Long Short-Term Memory ( LSTM ) , a significant network in deep learning has made tremendous achievements in capturing semantic information and predicting the semantic relatedness of two sentences . In this study , considering the similarity between predicting the relatedness of sentence pair task and semantic search , we provide a novel channel to process semantic search task : see semantic search as an atypical predicting the relatedness of sentence pair task . Furthermore , we propose an LSTM-Based Neural Network Model which is suitable for predicting the semantic relatedness between user query and potential documents .
The recurrent neural networks ( RNNs ) have shown good performance for sentence similarity modeling in recent years . Most RNNs focus on modeling the hidden states based on the current sentence , while the context information from the other sentence is not well investigated during the hidden state generation .
Abstract Sentence Pair Modeling is a critical and challenging problem in natural language processing ( NLP ) . It aims to seek the underlying semantic relationship between two sentences .
We make three contributions . ( i ) The ABCNN can be applied to a wide variety of tasks that require modeling of sentence pairs . ( ii ) We propose three attention schemes that integrate mutual influence between sentences into CNNs ; thus , the representation of each sentence takes into consideration its counterpart .
Modeling sentence pairs plays the vital role for judging the relationship between two sentences , such as paraphrase identification , natural language inference , and answer sentence selection . Previous work achieves very promising results using neural networks with attention mechanism .
This paper extends such a general line of research and its main contributions are two-fold . First , we explore several effective formulations of proximity-based cues for use in the sentence modeling process involved in the LM-based summarization framework . Second , the utilities of the methods instantiated from the LM-based summarization framework and several well-practiced state-of-the-art methods are analyzed and compared extensively .
Neural network based approaches for sentence relation modeling automatically generate hidden matching features from raw sentence pairs . However , the quality of matching feature representation may not be satisfied due to complex semantic relations such as entailment or contradiction .
A lot of advanced methods have been developed to solve this task . In this paper , we propose an attention aware bidirectional GRU ( Bi-GRU ) framework to classify the sentiment polarity from the aspects of sentential-sequence modeling and word-feature seizing . It is composed of a pre-attention Bi-GRU to incorporate the complicated interaction between words by sentence modeling , and an attention layer to capture the keywords for sentiment apprehension .
paragraph : Roadmap We start by introducing a convolution network in Section [ reference ] as the basic architecture for sentence modeling , and how it is related to existing sentence models . Based on that , in Section [ reference ] , we propose two architectures for sentence matching , with a detailed discussion of their relation .
Table [ reference ] summarizes the results of model variants . We can again see that the use of cell states clearly improves sentence modeling performance ( baseline vs. ( i ) and ( iv ) vs. ( i ) ) . Also from the results of baseline and ( ii ) , we validate that the selection of does not significantly affect performance but introducing is beneficial ( baseline vs. ( iii ) ) possibly due to its effect on normalizing information from multiple sources , as mentioned in § [ reference ] .
CNN has been very successful on image classification ; see e.g. , the winning solutions of ImageNet Large Scale Visual Recognition Challenge . On text , since the work on token - level applications ( e.g. , POS tagging ) by nnnlpJMLR11 , CNN has been used in systems for entity search , sentence modeling , word embedding learning , product feature mining , and so on . Notably , in many of these CNN studies on text , the first layer of the network converts words in sentences to word vectors by table lookup .
The authors employ a word - by - word neural attention mechanism to reason about the entailment in two sentences . Also focused on modeling sentence pairs , the authors in propose an attention - based bigram CNN for jointly performing attention between two CNN hierarchies . In their work , three attention schemes are proposed and evaluated .
NLI is in the core of natural language understanding and has wide applications in NLP , e.g. , question answering and automatic summarization . Moreover , NLI is also related to other tasks of sentence pair modeling , including paraphrase detection , relation recognition of discourse units , etc . Traditional approaches to NLI mainly fall into two groups : feature - rich models and formal reasoning methods .
However , such approaches are limited in terms of scope and accuracy . The renewed prosperity of neural networks has made significant achievements in various NLP applications , including individual sentence modeling as well as sentence matching . A typical neural architecture to model sentence pairs is the “ Siamese ” structure , which involves an underlying sentence model and a matching layer to determine the relationship between two sentences .
Finally , two sentences ’ information is combined by several heuristic matching layers , including concatenation , element - wise product and difference ; they are effective in capturing relationships between two sentences , but remain low complexity . To sum up , the main contributions of this paper are two - fold : ( 1 ) We are the first to introduce tree - based convolution to sentence pair modeling tasks like NLI ; ( 2 ) Leveraging additional heuristics further improves the accuracy while remaining low complexity , outperforming existing sentence encoding - based approaches to a large extent , including feature - rich methods and long short term memory ( LSTM )- based recurrent networks . section : Related Work
tai2015 adjusted the standard LSTM to tree - structured topologies and obtained superior results over a sequential LSTM on related tasks . In this paper , we stack CNN and LSTM in a unified architecture for semantic sentence modeling . The combination of CNN and LSTM can be seen in some computer vision tasks like image caption and speech recognition .
section : Related Work Our architecture for sentence pair encoding can be regarded as strong interaction models , which have been explored in previous models . An intuitive paradigm is to compute similarities between all the words or phrases of the two sentences .
The attention weight for the ith word in the premise is the result of a weighted combination ( parameterized by w ) of values in m i . The final sentence - pair representation is obtained from a non - linear combination of the attentionweighted representation r of the premise and the last output vector h N using where W p , W x ∈ R k×k are trained projection matrices .
section : Related Work Our method is motivated by the central role played by alignment in machine translation and previous approaches to sentence similarity modeling , natural language inference , and semantic parsing . The neural counterpart to alignment , attention , which is a key part of our approach , was originally proposed and has been predominantly used in conjunction with LSTMs and to a lesser extent with CNNs .
Finnish Language Modeling with Deep Transformer Models Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells A pattern-based foundation for language-driven software engineering Prosodic features for a maximum entropy language model Conditional leaving-one-out and cross-validation for discount estimation in Kneser-Ney-like extensions Resource development and experiments in automatic south african broadcast news transcription Synthesizing Sign Language by Connecting Linguistically Structured Descriptions to a Multi-track Animation System Deep Learning for Natural Language Processing and Language Modelling Direct Output Connection for a High-Rank Language Model Parameter Re-Initialization through Cyclical Batch Size Schedules Practical Text Classification With Large Pre-Trained Language Models Neural Machine Translation in Linear Time Jmp8 at SemEval-2017 Task 2: A simple and general distributional approach to estimate word similarity Methodology and Results for the Competition on Semantic Similarity Evaluation and Entailment Recognition for PROPOR 2016 Investigating Citation Linkage as a Sentence Similarity Measurement Task Using Deep Learning Random Positive-Only Projections: PPMI-Enabled Incremental Semantic Space Construction Word Embeddings as Metric Recovery in Semantic Spaces A Snapshot of Ontology Evaluation Criteria and Strategies An ontology approach to comparative phenomics in plants Multi-prototype based semantic similarity of concepts in ontology Contextual Analysis for the Representation of Words A semantic framework for textual data enrichment Ontology-based semantic similarity: A new approach based on analysis of the concept intent A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks Modelling Interaction of Sentence Pair with Coupled-LSTMs aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model Convolutional Neural Network Architectures for Matching Natural Language Sentences Grounded Textual Entailment Supervised Learning of Universal Sentence Representations from Natural Language Inference Data An Ensemble Model Based on Siamese Neural Networks for the Question Pairs Matching Task Inter-Weighted Alignment Network for Sentence Pair Modeling THE IMPORTANCE OF SUBWORD EMBEDDINGS IN SENTENCE PAIR MODELING LSTM-Based Neural Network Model for Semantic Search CA-RNN: Using Context-Aligned Recurrent Neural Networks for Modeling Sentence Similarity Enhanced attentive convolutional neural networks for sentence pair modeling ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs Multiway Attention Networks for Modeling Sentence Pairs Incorporating proximity information in relevance language modeling for extractive speech summarization Enhancing Sentence Relation Modeling with Auxiliary Character-level Embedding Attention Aware Bidirectional Gated Recurrent Unit Based Framework for Sentiment Analysis Convolutional Neural Network Architectures for Matching Natural Language Sentences Cell-aware Stacked LSTMs for Modeling Sentences Effective Use of Word Order for Text Categorization with Convolutional Neural Networks Hierarchical Question-Image Co-Attention for Visual Question Answering Natural Language Inference by Tree-Based Convolution and Heuristic Matching A C-LSTM Neural Network for Text Classification Modelling Interaction of Sentence Pair with Coupled-LSTMs Reasoning about Entailment with Neural Attention A Decomposable Attention Model for Natural Language Inference
682c6d8fd421935c3622885c369f8695e6b742a9 47c71b62356adee63f63ff0f7d0a3579f981e973 991d325ba8c56ca03578c1def99d72d8e130ebad 029fa86dca22a619974b029d7fdbdd5618ec81f1 78dee03f871b503b6208c4baeabe7e3031c7748f 11abac9c5c7a7c7bc8c8a8326f4bc4ef8d19da96 10330bb5dc3f8a945d60f76798e50130f3ee8afc f73647c3290ed7d857cc103971e343f654b44685 6d3ac66659b0e13599bd85c9d46344248a549b32 11d056992b0019eab16e53c5f7e4c1e005ad52be 6f7c46e3b15e4807867c574a4a3d8f785a6ab860 fe6ca5d7bc922b86995524d233536a0a3b88ed97 637a2c0efa953bc5d8f00e797e6cb4606d640a11 8486c67fbedf9c02efa2ae96576477628cdf8169 d7c0cde65249aa141061b03c5584cfd32899dea0 ab649ceeac3d237b6338b9a5b9ece7826ac085cb 16664b1e5ea6a23ede203409f8c7b3be5268adf2 17cf817c88bc5a984d3321a4acba32d13e142d03 1f429edbc48f4dede4df84830a577feb07a3a0b2 c0610dbf48c8ea2b0e9e52141f9a645d7d5e7652 e291dc06d3feab30e2414226ae11718d991d1dbc c99fe106e7d1cc62f7cb73ea6fc745b8679e4d2f a85aadf2d93c9510a0976ab457e3acbf2c0b12d5 c58738fade559d02a71bedba51bc62d42587777c ffea84c37bc14ee03920636ae917186a2a10c7df 7f3ae283243e15e05f188a05779ccfae9a3567f4 2b32b4fa1e28c256745f1573b5444b1b2c8df30e 86a08fd3a52fedbd49d967a5b6d66c5226ef20cd d4c72bd0cb6728068e958e21a7fc494abc6eaf58 f1a9ac615f83379c9800b27a21a90187e8b28bcb
["Computer Science","Engineering","Mathematics"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Mathematics","Computer Science"] ["Geography","Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Biology","Engineering","Medicine","Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"]
2020 2018 2011 2006 2012 2012 2017 2018 2018 2018 2018 2016 2017 2017 2020 2016 2016 2017 2015 2012 2014 2016 2013 2017 2016 2016 2014 2018 2017 2018 2017 2018 2019 2018 2020 2016 2018 2015 2016 2018 2014 2019 2015 2016 2016 2015 2016 2016 2016
INTERSPEECH ICASSP SLTU HCI (8) SPA EMNLP ArXiv ArXiv ArXiv SemEval@ACL Canadian Conference on AI *SEM@ACL SEMANTICS ICMLC ICMLC EMNLP EMNLP CIKM NIPS COLING EMNLP CCKS Tasks EMNLP AAAI IJCAI APSIPA KSEM (1) NIPS ACML HLT-NAACL NIPS ACL ArXiv EMNLP ICLR EMNLP
https://api.semanticscholar.org/682c6d8fd421935c3622885c369f8695e6b742a9 https://api.semanticscholar.org/47c71b62356adee63f63ff0f7d0a3579f981e973 https://api.semanticscholar.org/991d325ba8c56ca03578c1def99d72d8e130ebad https://api.semanticscholar.org/029fa86dca22a619974b029d7fdbdd5618ec81f1 https://api.semanticscholar.org/78dee03f871b503b6208c4baeabe7e3031c7748f https://api.semanticscholar.org/11abac9c5c7a7c7bc8c8a8326f4bc4ef8d19da96 https://api.semanticscholar.org/10330bb5dc3f8a945d60f76798e50130f3ee8afc https://api.semanticscholar.org/f73647c3290ed7d857cc103971e343f654b44685 https://www.semanticscholar.org/paper/0be9ca65ad318ee3729928882ef2c403d4b6d24e https://www.semanticscholar.org/paper/4087ebc37a1650dbb5d8205af0850bee74f3784b https://www.semanticscholar.org/paper/14908a18ff831005b6b4fc953ce61e1b4e7b54ee https://www.semanticscholar.org/paper/98445f4172659ec5e891e031d8202c102135c644 https://api.semanticscholar.org/6d3ac66659b0e13599bd85c9d46344248a549b32 https://api.semanticscholar.org/11d056992b0019eab16e53c5f7e4c1e005ad52be https://api.semanticscholar.org/6f7c46e3b15e4807867c574a4a3d8f785a6ab860 https://api.semanticscholar.org/fe6ca5d7bc922b86995524d233536a0a3b88ed97 https://api.semanticscholar.org/637a2c0efa953bc5d8f00e797e6cb4606d640a11 https://api.semanticscholar.org/8486c67fbedf9c02efa2ae96576477628cdf8169 https://api.semanticscholar.org/d7c0cde65249aa141061b03c5584cfd32899dea0 https://api.semanticscholar.org/ab649ceeac3d237b6338b9a5b9ece7826ac085cb https://api.semanticscholar.org/16664b1e5ea6a23ede203409f8c7b3be5268adf2 https://api.semanticscholar.org/17cf817c88bc5a984d3321a4acba32d13e142d03 https://api.semanticscholar.org/1f429edbc48f4dede4df84830a577feb07a3a0b2 https://www.semanticscholar.org/paper/ade0c116120b54b57a91da51235108b75c28375a https://www.semanticscholar.org/paper/10d89efc96beb45676149c8a3237a86a72a2116e https://www.semanticscholar.org/paper/364c1a3df58d87cb40ab33fdf3831cf2862f3570 https://www.semanticscholar.org/paper/9f08b01251cb99f4ffae8c7b3e4468d3af9c98d3 https://www.semanticscholar.org/paper/1235dd37312cb20aced0e97d953f6379d8a0c7d4 https://www.semanticscholar.org/paper/ee7b883e35d754ae4f71c21bb71f9f03e4ffbb2c https://api.semanticscholar.org/c0610dbf48c8ea2b0e9e52141f9a645d7d5e7652 https://api.semanticscholar.org/e291dc06d3feab30e2414226ae11718d991d1dbc https://api.semanticscholar.org/c99fe106e7d1cc62f7cb73ea6fc745b8679e4d2f https://api.semanticscholar.org/a85aadf2d93c9510a0976ab457e3acbf2c0b12d5 https://api.semanticscholar.org/c58738fade559d02a71bedba51bc62d42587777c https://api.semanticscholar.org/ffea84c37bc14ee03920636ae917186a2a10c7df https://api.semanticscholar.org/7f3ae283243e15e05f188a05779ccfae9a3567f4 https://api.semanticscholar.org/2b32b4fa1e28c256745f1573b5444b1b2c8df30e https://api.semanticscholar.org/86a08fd3a52fedbd49d967a5b6d66c5226ef20cd https://api.semanticscholar.org/d4c72bd0cb6728068e958e21a7fc494abc6eaf58 https://api.semanticscholar.org/f1a9ac615f83379c9800b27a21a90187e8b28bcb https://www.semanticscholar.org/paper/9f08b01251cb99f4ffae8c7b3e4468d3af9c98d3 https://www.semanticscholar.org/paper/249b3b7421d3cdb932eecfe4b67203e0e46806b2 https://www.semanticscholar.org/paper/fbf417c83ae5b895fc645346e4efbf3a0aabeac9 https://www.semanticscholar.org/paper/fb9d253258d6b3beceb9d6cd7bba6e0a29ab875b https://www.semanticscholar.org/paper/ea407573bfcd39f9a478fe33cf6ce0ee1780a5f0 https://www.semanticscholar.org/paper/10f62af29c3fc5e2572baddca559ffbfd6be8787 https://www.semanticscholar.org/paper/10d89efc96beb45676149c8a3237a86a72a2116e https://www.semanticscholar.org/paper/2846e83d405cbe3bf2f0f3b5f635dd8b3c680c45 https://www.semanticscholar.org/paper/2cd8e8f510c89c7c18268e8ad51c061e459ad321