Having received these entities , we have an opportunity to form parameters space and to solve problems of text mining as concept normalization , speech recognition , etc . The recent advances in NER are related to the technology of contextualized word embeddings , which transforms text to the form being effective for Deep Learning . In the paper , we show how NER model detects pharmacological substances , compounds , and proteins in the dataset obtained from the Spanish Clinical Case Corpus ( SPACCC ) . Objective This study aims to improve the performance of NER in clinical text by combining multiple contextual embeddings and prior knowledge . Methods In this study , we investigate the effects of combining multiple contextualized word embeddings with classic word embedding in deep neural networks to predict named entities in clinical text . We also investigate whether using a semantic lexicon could further improve the performance of the clinical NER system . Contextual word embeddings ( e.g. GPT , BERT , ELMo , etc . ) have demonstrated state-of-the-art performance on various NLP tasks . Recent work with the multilingual version of BERT has shown that the model performs very well in cross-lingual settings , even when only labeled English data is used to finetune the model . For example , in a knowledge base , we may have an entity for Germany as a country but not for the more fuzzy concept of Germany that covers notions of German Population , German Drivers , and the German Government . Inspired by recent advances in contextual word embeddings , we introduce the concept of entity spaces specific representations of a set of associated entities with near-identity . Thus , these entity spaces provide a handle to an amorphous grouping of entities . Word embeddings that provide continuous low-dimensional vector representations of words have been extensively used for various natural language processing tasks . However , existing context-based word embeddings such as Word2vec and GloVe typically fail to capture sufficient sentiment information , which may result in words with similar vector representations having an opposite sentiment polarity ( e.g. , good and bad ) , thus degrading sentiment analysis performance . To tackle this problem , recent studies have suggested learning sentiment embeddings to incorporate the sentiment polarity ( positive and negative ) information from labeled corpora . However , these models have not been considered in the field of Arabic question classification . In this paper , we investigate the use of a contextual word representation named Embeddings from Language Models ( ELMo ) on Arabic question classification . We study the behaviour of this representation by building numerous neural network architectures trained to classify questions . section : Introduction Word embeddings play an important role in neural - based natural language processing ( NLP ) models . Neural word embeddings encapsulate the linguistic information of words in continuous vectors . Word embeddings : We use Skip - gram mikolov2013word2vec to train word embeddings . Character embeddings All weights except word embeddings are initialized randomly following . Word embeddings are initialized with Glove vectors of dimension 300 available publicly , trained for 6 billion words on Wikipedia and Gigaword . The word embeddings are fine - tuned with a learning rate of of the other weights . We choose to use Gated Recurrent Unit ( GRU ) in our experiments because it performs similarly but is computationally cheaper than LSTM . In this step , the goal is to compare the question embedding and all the contextual embeddings , and select the pieces of information that are relevant to the question . We compute a probability distribution depending on the degree of relevance between word ( in its context ) and the question and then produce an output vector which is a weighted combination of all contextual embeddings : is used in a bilinear term , which allows us to compute a similarity between and more flexibly than with just a dot product . Moreover , such word embedding methods as Skip - gram and vLBL originated from neural language models designed to handle much larger vocabulary and data sizes . Neural language models can also be used as contextualized word representations . Thus , language modeling is a good benchmark task for investigating the general frameworks of neural methods in NLP field . [ reference ] . On the target side , we continue to use only word - based embeddings as the representation . subsection : Modeling Rare / Unseen Words using Switching Generator - Pointer The idea of using a recurrent layer over characters to induce a complementary view of a word has occurred in numerous papers . Perhaps the earliest is Santos and Zadrozny ( 2014 ) who compare character - based LSTM encodings to traditional word - based embeddings . [ reference ] take this a step further and combine the word embeddings with a recurrent character encoding of the word - instead of just relying on one or the other . We reimplement the encoder of konstas2017neural , where the sequential linearization is the input to a bidirectional LSTM ( BiLSTM ; graves2013speech ) network . The hidden state of the BiLSTM at step is used as a context - aware word representation of the - th token in the sequence : where , is the size of the output embeddings . subsection : TreeLSTM Encoders On CoNLL - 2012 , our model gains more than 2.5 F1 absolute over the previous state - of - the - art . Our models also show improvements when using contextually - encoded word representations peters2018deep , obtaining nearly 1.0 F1 higher than the state - of - the - art on CoNLL - 2005 news and more than 2.0 F1 improvement on out - of - domain text . section : Model document : Deep contextualized word representations We introduce a new type of deep contextualized word representation that models both ( 1 ) complex characteristics of word use ( e.g. , syntax and semantics ) , and ( 2 ) how these uses vary across linguistic contexts ( i.e. , to model polysemy ) . Our word vectors are learned functions of the internal states of a deep bidirectional language model ( biLM ) , which is pre - trained on a large text corpus . We now provide a qualitative analysis of our model on the SQuAD dev set . First , we visualize the feature spaces after the word and contextual embedding layers . These two layers are responsible for aligning the embeddings between the query and context words which are the inputs to the subsequent attention layer . Since the introduction of context-aware token representation techniques such as Embeddings from Language Models ( ELMo ) and Bidirectional Encoder Representations from Transformers ( BERT ) , there have been numerous reports on improved performance on a variety of natural language tasks . Nevertheless , the degree to which the resulting context-aware representations can encode information about morpho-syntactic properties of the tokens in a sentence remains unclear . This paper presents the results for English and code-mixed Hindi language . Embeddings from Language Models ( ELMo ) , Glove and fastText embeddings , and TF-IDF features of character and word n-grams have been used to train the models . Our best models for Hindi and English language obtained F1 score of 81.05 and 74.62 respectively on the official run . Matching variations with their original questions was not trivial with a set of unsupervised baselines , especially for manually created variations . Besides high performances obtained with ELMo and BERT embeddings , an Information Retrieval system was surprisingly competitive when considering only the first hit . In the second experiment , text classifiers were trained with the original questions , and tested when assigning each variation to one of three possible sources , or assigning them as out-of-domain . To further investigate what contextualized word embeddings capture , this paper analyzes whether they can indicate the corresponding sense definitions and proposes a general framework that is capable of explaining word meanings given contextualized word embeddings for better interpretation . The experiments show that both ELMo and BERT embeddings can be well interpreted via a readable textual form , and the findings may benefit the research community for a better understanding of what the embeddings capture1 . In this study , we aim to investigate the extent to which the semblance of syntactic structure captured by language models adheres to a surface-syntactic or deep syntactic style of analysis , and whether the patterns are consistent across different languages . We apply a probe for extracting directed dependency trees to BERT and ELMo models trained on 13 different languages , probing for two different syntactic annotation styles : Universal Dependencies ( UD ) , prioritizing deep syntactic relations , and Surface-Syntactic Universal Dependencies ( SUD ) , focusing on surface structure . We find that both models exhibit a preference for UD over SUD - with interesting variations across languages and layers - and that the strength of this preference is correlated with differences in tree shape . Few other models with different settings and training designed to test and optimise the performance of both models . The results of the optimised BERT and ELMo models can achieve 68.4 % and 60.8 % , respectively . In the first part of this paper , we use transfer learning from English to English-paired code-switched languages for the language identification ( LID ) task by applying two simple yet effective techniques : 1 ) a hierarchical attention mechanism that enhances morphological clues from character n-grams , and 2 ) a secondary loss that forces the model to learn n-gram representations that are particular to the languages involved . We use the bottom layers of the ELMo architecture to learn these morphological clues by essentially recognizing what is and what is not English . Our approach outperforms the previous state of the art on Nepali-English , Spanish-English , and Hindi-English datasets . Abstract We investigate the application of deep learning to biocuration tasks that involve classification of text associated with biomedical evidence in primary research articles . We developed a large-scale corpus of molecular papers derived from PubMed and PubMed Central open access records and used it to train deep learning word embeddings under the GloVe , FastText and ELMo algorithms . We applied those models to a distant supervised method classification task based on text from figure captions or fragments surrounding references to figures in the main text using a variety or models and parameterizations . In this paper , we explore techniques to efficiently transfer the knowledge from these unlabeled utterances to improve model performance on Spoken Language Understanding ( SLU ) tasks . We use Embeddings from Language Model ( ELMo ) to take advantage of unlabeled data by learning contextualized word representations . Additionally , we propose ELMo-Light ( ELMoL ) , a faster and simpler unsupervised pre-training method for SLU . Three real-time Twitter datasets such as Nepal Earthquake , California Earthquake , and Typhoon hagupit are used and the performance is analyzed with parameters such as precision , recall , f1-score and accuracy . The proposed dense classifier with ELMo embeddings model gives better accuracy than the traditional classifier ( Support Vector Machine ) and deep learning classifiers ( Convolutional Neural Network and MultiLayer Perceptron-Convolutional Neural Network with Crisis word embedding ) . The deep methods include a vanilla bidirectional long short-term memory recurrent neural network ( biLSTM ) , an adapted biLSTM , and a bi-attentive classification network ( BCN ) with pre-trained contextualised ELMo embedding . For all the runs , we apply different word embeddings ( in-domain pre-trained , word-level pre-trained GloVe , character-level , or ELMo embeddings ) and data augmentation strategies ( SMOTE , loss weights , or GPT-2 ) to explore the influence they have on performance . Evaluation results show that our models perform better than the median for most situations . Neither approach leads to a significant gain on any of the four downstream tasks we considered relative to ELMo-only baselines . Further analysis using black-box probes confirms that our shallow-syntax-aware contextual embeddings do not transfer to linguistic tasks any more easily than ELMo 's embeddings . We take these findings as evidence that ELMo-style pretraining discovers representations which make additional awareness of shallow syntax redundant . The paper presents an evaluation of word embedding models in clustering of texts in the Polish language . Authors verified six different embedding models , starting from widely used word2vec , across fastText with character n-grams embedding , to deep learning-based ELMo and BERT . Moreover , four standardisation methods , three distance measures and four clustering methods were evaluated . We use vectors derived from a bidirectional LSTM that is trained with a coupled language model ( LM ) objective on a large text corpus . For this reason , we call them ELMo ( Embeddings from Language Models ) representations . Unlike previous approaches for learning contextualized word vectors Peters2017SemisupervisedST , McCann2017LearnedIT , ELMo representations are deep , in the sense that they are a function of all of the internal layers of the biLM . [ reference ] where a bi - attention layer follows the biLSTMs , or the coreference resolution experiments where a clustering model is layered on top of the biLSTMs . Finally , we found it beneficial to add a moderate amount of dropout to ELMo Srivastava2014DropoutAS and in some cases to regularize the ELMo weights by adding to the loss . This imposes an inductive bias on the ELMo weights to stay close to an average of all biLM layers . After adding ELMo , the model exceeds the baseline maximum at epoch 10 , a 98 % relative decrease in the number of updates needed to reach the same level of performance . In addition , ELMo - enhanced models use smaller training sets more efficiently than models without ELMo . Figure 1 compares the performance of baselines models with and without ELMo as the percentage of the full training set is varied from 0.1 % to 100 % . During finetuning , text is embedded with ELMo before being passed into a decoder . Error signals are backpropagated into the ELMo language model . Unlike our other models , we do not use an auxiliary language modeling loss during finetuning , as the ELMo language model is bidirectional . See Table [ reference ] . While the Transformer gets close but does not exceed the state of the art on the SST dataset , it exceeds both the mLSTM and ELMo baseline as well as both Watson and Google Sentiment APIs on the company tweets . This is despite optimally calibrating the API results on the test set . model toutanova2003feature . We use the pre - trained ELMo models and learn task - specific combinations of the ELMo representations which are provided as input instead of GloVe embeddings to the D & M parser with otherwise default settings . subsubsection : CoNLL - 2012 We present Cross-View Training ( CVT ) , a simple but effective method for deep semi-supervised learning . On labeled examples , the model is trained with standard cross-entropy loss . Inspired by human beings often using different views to find connection between domains , we argue that , there exist some fine-grained features which can be shared across different views of origin data . In this paper , we proposed a cross-view adaptation network , which use adversarial method to extract shared features and introduce cross-view training to fine-turn it . Besides , we construct some novel views of input data for cross-domain relation extraction . We apply contextualized ( ELMo and BERT ) word and sentence embeddings to this task , and propose supervised models that leverage these representations for prediction . Our models are further assisted by lexical substitute annotations automatically assigned to word instances by context2vec , a neural model that relies on a bidirectional LSTM . We perform an extensive comparison of existing word and sentence representations on benchmark datasets addressing both graded and binary similarity . We create a lexical substitution dataset for CDSM evaluation from an Englishlanguage corpus with manual “ all-words ” lexical substitution annotation . Our experiments indicate that the Practical Lexical Function CDSM outperforms simple component-wise CDSMs and performs on par with the context2vec lexical substitution model using the same context . Other recent work has also focused on learning context - dependent representations . context2vec Melamud2016context2vecLG uses a bidirectional Long Short Term Memory [ LSTM ; ] [ ] LSTM : Hochreiter1997 to encode the context around a pivot word . Other approaches for learning contextual embeddings include the pivot word itself in the representation and are computed with the encoder of either a supervised neural machine translation ( MT ) system [ CoVe ; ]
Having received these entities , we have an opportunity to form parameters space and to solve problems of text mining as concept normalization , speech recognition , etc . The recent advances in NER are related to the technology of contextualized word embeddings , which transforms text to the form being effective for Deep Learning . In the paper , we show how NER model detects pharmacological substances , compounds , and proteins in the dataset obtained from the Spanish Clinical Case Corpus ( SPACCC ) .
Objective This study aims to improve the performance of NER in clinical text by combining multiple contextual embeddings and prior knowledge . Methods In this study , we investigate the effects of combining multiple contextualized word embeddings with classic word embedding in deep neural networks to predict named entities in clinical text . We also investigate whether using a semantic lexicon could further improve the performance of the clinical NER system .
Contextual word embeddings ( e.g. GPT , BERT , ELMo , etc . ) have demonstrated state-of-the-art performance on various NLP tasks . Recent work with the multilingual version of BERT has shown that the model performs very well in cross-lingual settings , even when only labeled English data is used to finetune the model .
For example , in a knowledge base , we may have an entity for Germany as a country but not for the more fuzzy concept of Germany that covers notions of German Population , German Drivers , and the German Government . Inspired by recent advances in contextual word embeddings , we introduce the concept of entity spaces specific representations of a set of associated entities with near-identity . Thus , these entity spaces provide a handle to an amorphous grouping of entities .
Word embeddings that provide continuous low-dimensional vector representations of words have been extensively used for various natural language processing tasks . However , existing context-based word embeddings such as Word2vec and GloVe typically fail to capture sufficient sentiment information , which may result in words with similar vector representations having an opposite sentiment polarity ( e.g. , good and bad ) , thus degrading sentiment analysis performance . To tackle this problem , recent studies have suggested learning sentiment embeddings to incorporate the sentiment polarity ( positive and negative ) information from labeled corpora .
However , these models have not been considered in the field of Arabic question classification . In this paper , we investigate the use of a contextual word representation named Embeddings from Language Models ( ELMo ) on Arabic question classification . We study the behaviour of this representation by building numerous neural network architectures trained to classify questions .
section : Introduction Word embeddings play an important role in neural - based natural language processing ( NLP ) models . Neural word embeddings encapsulate the linguistic information of words in continuous vectors .
Word embeddings : We use Skip - gram mikolov2013word2vec to train word embeddings . Character embeddings
All weights except word embeddings are initialized randomly following . Word embeddings are initialized with Glove vectors of dimension 300 available publicly , trained for 6 billion words on Wikipedia and Gigaword . The word embeddings are fine - tuned with a learning rate of of the other weights .
We choose to use Gated Recurrent Unit ( GRU ) in our experiments because it performs similarly but is computationally cheaper than LSTM . In this step , the goal is to compare the question embedding and all the contextual embeddings , and select the pieces of information that are relevant to the question . We compute a probability distribution depending on the degree of relevance between word ( in its context ) and the question and then produce an output vector which is a weighted combination of all contextual embeddings : is used in a bilinear term , which allows us to compute a similarity between and more flexibly than with just a dot product .
Moreover , such word embedding methods as Skip - gram and vLBL originated from neural language models designed to handle much larger vocabulary and data sizes . Neural language models can also be used as contextualized word representations . Thus , language modeling is a good benchmark task for investigating the general frameworks of neural methods in NLP field .
[ reference ] . On the target side , we continue to use only word - based embeddings as the representation . subsection : Modeling Rare / Unseen Words using Switching Generator - Pointer
The idea of using a recurrent layer over characters to induce a complementary view of a word has occurred in numerous papers . Perhaps the earliest is Santos and Zadrozny ( 2014 ) who compare character - based LSTM encodings to traditional word - based embeddings . [ reference ] take this a step further and combine the word embeddings with a recurrent character encoding of the word - instead of just relying on one or the other .
We reimplement the encoder of konstas2017neural , where the sequential linearization is the input to a bidirectional LSTM ( BiLSTM ; graves2013speech ) network . The hidden state of the BiLSTM at step is used as a context - aware word representation of the - th token in the sequence : where , is the size of the output embeddings . subsection : TreeLSTM Encoders
On CoNLL - 2012 , our model gains more than 2.5 F1 absolute over the previous state - of - the - art . Our models also show improvements when using contextually - encoded word representations peters2018deep , obtaining nearly 1.0 F1 higher than the state - of - the - art on CoNLL - 2005 news and more than 2.0 F1 improvement on out - of - domain text . section : Model
document : Deep contextualized word representations We introduce a new type of deep contextualized word representation that models both ( 1 ) complex characteristics of word use ( e.g. , syntax and semantics ) , and ( 2 ) how these uses vary across linguistic contexts ( i.e. , to model polysemy ) . Our word vectors are learned functions of the internal states of a deep bidirectional language model ( biLM ) , which is pre - trained on a large text corpus .
We now provide a qualitative analysis of our model on the SQuAD dev set . First , we visualize the feature spaces after the word and contextual embedding layers . These two layers are responsible for aligning the embeddings between the query and context words which are the inputs to the subsequent attention layer .
Since the introduction of context-aware token representation techniques such as Embeddings from Language Models ( ELMo ) and Bidirectional Encoder Representations from Transformers ( BERT ) , there have been numerous reports on improved performance on a variety of natural language tasks . Nevertheless , the degree to which the resulting context-aware representations can encode information about morpho-syntactic properties of the tokens in a sentence remains unclear .
This paper presents the results for English and code-mixed Hindi language . Embeddings from Language Models ( ELMo ) , Glove and fastText embeddings , and TF-IDF features of character and word n-grams have been used to train the models . Our best models for Hindi and English language obtained F1 score of 81.05 and 74.62 respectively on the official run .
Matching variations with their original questions was not trivial with a set of unsupervised baselines , especially for manually created variations . Besides high performances obtained with ELMo and BERT embeddings , an Information Retrieval system was surprisingly competitive when considering only the first hit . In the second experiment , text classifiers were trained with the original questions , and tested when assigning each variation to one of three possible sources , or assigning them as out-of-domain .
To further investigate what contextualized word embeddings capture , this paper analyzes whether they can indicate the corresponding sense definitions and proposes a general framework that is capable of explaining word meanings given contextualized word embeddings for better interpretation . The experiments show that both ELMo and BERT embeddings can be well interpreted via a readable textual form , and the findings may benefit the research community for a better understanding of what the embeddings capture1 .
In this study , we aim to investigate the extent to which the semblance of syntactic structure captured by language models adheres to a surface-syntactic or deep syntactic style of analysis , and whether the patterns are consistent across different languages . We apply a probe for extracting directed dependency trees to BERT and ELMo models trained on 13 different languages , probing for two different syntactic annotation styles : Universal Dependencies ( UD ) , prioritizing deep syntactic relations , and Surface-Syntactic Universal Dependencies ( SUD ) , focusing on surface structure . We find that both models exhibit a preference for UD over SUD - with interesting variations across languages and layers - and that the strength of this preference is correlated with differences in tree shape .
Few other models with different settings and training designed to test and optimise the performance of both models . The results of the optimised BERT and ELMo models can achieve 68.4 % and 60.8 % , respectively .
In the first part of this paper , we use transfer learning from English to English-paired code-switched languages for the language identification ( LID ) task by applying two simple yet effective techniques : 1 ) a hierarchical attention mechanism that enhances morphological clues from character n-grams , and 2 ) a secondary loss that forces the model to learn n-gram representations that are particular to the languages involved . We use the bottom layers of the ELMo architecture to learn these morphological clues by essentially recognizing what is and what is not English . Our approach outperforms the previous state of the art on Nepali-English , Spanish-English , and Hindi-English datasets .
Abstract We investigate the application of deep learning to biocuration tasks that involve classification of text associated with biomedical evidence in primary research articles . We developed a large-scale corpus of molecular papers derived from PubMed and PubMed Central open access records and used it to train deep learning word embeddings under the GloVe , FastText and ELMo algorithms . We applied those models to a distant supervised method classification task based on text from figure captions or fragments surrounding references to figures in the main text using a variety or models and parameterizations .
In this paper , we explore techniques to efficiently transfer the knowledge from these unlabeled utterances to improve model performance on Spoken Language Understanding ( SLU ) tasks . We use Embeddings from Language Model ( ELMo ) to take advantage of unlabeled data by learning contextualized word representations . Additionally , we propose ELMo-Light ( ELMoL ) , a faster and simpler unsupervised pre-training method for SLU .
Three real-time Twitter datasets such as Nepal Earthquake , California Earthquake , and Typhoon hagupit are used and the performance is analyzed with parameters such as precision , recall , f1-score and accuracy . The proposed dense classifier with ELMo embeddings model gives better accuracy than the traditional classifier ( Support Vector Machine ) and deep learning classifiers ( Convolutional Neural Network and MultiLayer Perceptron-Convolutional Neural Network with Crisis word embedding ) .
The deep methods include a vanilla bidirectional long short-term memory recurrent neural network ( biLSTM ) , an adapted biLSTM , and a bi-attentive classification network ( BCN ) with pre-trained contextualised ELMo embedding . For all the runs , we apply different word embeddings ( in-domain pre-trained , word-level pre-trained GloVe , character-level , or ELMo embeddings ) and data augmentation strategies ( SMOTE , loss weights , or GPT-2 ) to explore the influence they have on performance . Evaluation results show that our models perform better than the median for most situations .
Neither approach leads to a significant gain on any of the four downstream tasks we considered relative to ELMo-only baselines . Further analysis using black-box probes confirms that our shallow-syntax-aware contextual embeddings do not transfer to linguistic tasks any more easily than ELMo 's embeddings . We take these findings as evidence that ELMo-style pretraining discovers representations which make additional awareness of shallow syntax redundant .
The paper presents an evaluation of word embedding models in clustering of texts in the Polish language . Authors verified six different embedding models , starting from widely used word2vec , across fastText with character n-grams embedding , to deep learning-based ELMo and BERT . Moreover , four standardisation methods , three distance measures and four clustering methods were evaluated .
We use vectors derived from a bidirectional LSTM that is trained with a coupled language model ( LM ) objective on a large text corpus . For this reason , we call them ELMo ( Embeddings from Language Models ) representations . Unlike previous approaches for learning contextualized word vectors Peters2017SemisupervisedST , McCann2017LearnedIT , ELMo representations are deep , in the sense that they are a function of all of the internal layers of the biLM .
[ reference ] where a bi - attention layer follows the biLSTMs , or the coreference resolution experiments where a clustering model is layered on top of the biLSTMs . Finally , we found it beneficial to add a moderate amount of dropout to ELMo Srivastava2014DropoutAS and in some cases to regularize the ELMo weights by adding to the loss . This imposes an inductive bias on the ELMo weights to stay close to an average of all biLM layers .
After adding ELMo , the model exceeds the baseline maximum at epoch 10 , a 98 % relative decrease in the number of updates needed to reach the same level of performance . In addition , ELMo - enhanced models use smaller training sets more efficiently than models without ELMo . Figure 1 compares the performance of baselines models with and without ELMo as the percentage of the full training set is varied from 0.1 % to 100 % .
During finetuning , text is embedded with ELMo before being passed into a decoder . Error signals are backpropagated into the ELMo language model . Unlike our other models , we do not use an auxiliary language modeling loss during finetuning , as the ELMo language model is bidirectional .
See Table [ reference ] . While the Transformer gets close but does not exceed the state of the art on the SST dataset , it exceeds both the mLSTM and ELMo baseline as well as both Watson and Google Sentiment APIs on the company tweets . This is despite optimally calibrating the API results on the test set .
model toutanova2003feature . We use the pre - trained ELMo models and learn task - specific combinations of the ELMo representations which are provided as input instead of GloVe embeddings to the D & M parser with otherwise default settings . subsubsection : CoNLL - 2012
We present Cross-View Training ( CVT ) , a simple but effective method for deep semi-supervised learning . On labeled examples , the model is trained with standard cross-entropy loss .
Inspired by human beings often using different views to find connection between domains , we argue that , there exist some fine-grained features which can be shared across different views of origin data . In this paper , we proposed a cross-view adaptation network , which use adversarial method to extract shared features and introduce cross-view training to fine-turn it . Besides , we construct some novel views of input data for cross-domain relation extraction .
We apply contextualized ( ELMo and BERT ) word and sentence embeddings to this task , and propose supervised models that leverage these representations for prediction . Our models are further assisted by lexical substitute annotations automatically assigned to word instances by context2vec , a neural model that relies on a bidirectional LSTM . We perform an extensive comparison of existing word and sentence representations on benchmark datasets addressing both graded and binary similarity .
We create a lexical substitution dataset for CDSM evaluation from an Englishlanguage corpus with manual “ all-words ” lexical substitution annotation . Our experiments indicate that the Practical Lexical Function CDSM outperforms simple component-wise CDSMs and performs on par with the context2vec lexical substitution model using the same context .
Other recent work has also focused on learning context - dependent representations . context2vec Melamud2016context2vecLG uses a bidirectional Long Short Term Memory [ LSTM ; ] [ ] LSTM : Hochreiter1997 to encode the context around a pivot word . Other approaches for learning contextual embeddings include the pivot word itself in the representation and are computed with the encoder of either a supervised neural machine translation ( MT ) system [ CoVe ; ]
Named Entity Recognition in Spanish Biomedical Literature: Short Review and Bert Model Combining Contextualized Embeddings and Prior Knowledge for Clinical Named Entity Recognition: Evaluation Study Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER Towards Entity Spaces Refining Word Embeddings Using Intensity Scores for Sentiment Analysis Exploring Contextual word representation for Arabic question classification Compressing Word Embeddings via Deep Compositional Code Learning A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks Graph-Structured Representations for Visual Question Answering A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task Direct Output Connection for a High-Rank Language Model Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings Structural Neural Encoders for AMR-to-text Generation Linguistically-Informed Self-Attention for Semantic Role Labeling Deep contextualized word representations Bidirectional Attention Flow for Machine Comprehension Neural Token Representations and Negation and Speculation Scope Detection in Biomedical and General Domain Text IIITG-ADBU at HASOC 2019: Automated Hate Speech and Offensive Content Detection in English and Code-Mixed Hindi Text AIA-BDE: A Corpus of FAQs in Portuguese and their Variations What Does This Word Mean? Explaining Contextualized Embeddings with Natural Language Definition Do Neural Language Models Show Preferences for Syntactic Formalisms? Hyperpartisan News and Articles Detection Using BERT and ELMo From English to Code-Switching: Transfer Learning with Strong Morphological Clues Building deep learning models for evidence classification from the open access biomedical literature Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents Improved Classification of Crisis-Related Data on Twitter using Contextual Representations Classification for Crisis-Related Tweets Leveraging Word Embeddings and Data Augmentation Shallow Syntax in Deep Water Evaluation of vector embedding models in clustering of text documents Deep contextualized word representations Practical Text Classification With Large Pre-Trained Language Models Linguistically-Informed Self-Attention for Semantic Role Labeling Cross-View Training for Semi-Supervised Learning Cross-View Adaptation Network for Cross-Domain Relation Extraction Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutes Lexical Substitution for Evaluating Compositional Distributional Models Deep contextualized word representations
e1b188b33a8ddc314679551a74bb561bef30af06 02e0ffe141c1e00f711121981e155d2ca2dd6a06 35480cc277b94dd653721ebc95d3b8d9cde6a46a 3660ea5bc9a72bbf357dbb3f49b5d82dea8d20cf 4588c518deebb639174025498927a2945734b753 bf30144192efa93dc65465bbbe4ea1dfcf417abb ab293e2efc083437df5f9fed58c026240a12ca4d c4ce31ec4e167b008557f85561942f21295c2719 4fa3daf3287b9e9725d70c8701fb384a72f2e84e 02e51b30387cacb8e4ebd4b8e1a5860cd00665a2 f2a9ef36b79c1b25d15bf10ff061db5eaad06798 207b3a97a0fb13d7c04d1107807f0e990f837a21 9abb8ff6bfb8ec175179f9605421b77bb1f09338 f23dae278cf98a7b46ad3ca93385f18c7903009d 0884a1b616d64e5b2c252f5980acf2049b2d8b80 d3fbef9ed6136e200c8ae7f296e46f9241102119 08cd047b370b37c171a00825e06f758235f7ec7f 4c636fa1843d15d0b0b7d3b8e3dca376b606abc7 cb292f86a3fea9cc964539e5c544aac5828389c6 6dc0eceea3f8bafcec62cedc2054b65f86dd7c73 c8b65d126477f00a848d94e8d99cbb03a5d965a6 330e998bd6c28ef407a0003e2b90b761e4cbb9b7 5f9455eea572d7b341ac6016f71de9e7c5384b85
["Computer Science"] ["Medicine","Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Medicine","Computer Science"] ["Computer Science","Mathematics"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"]
2020 2019 2019 2020 2018 2020 2018 2017 2017 2016 2018 2016 2018 2019 2018 2018 2017 2019 2019 2020 2019 2020 2019 2019 2019 2019 2020 2019 2019 2019 2018 2018 2018 2018 2019 2019 2018 2018
FRUCT EMNLP/IJCNLP (1) LREC ICLR EMNLP 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ACL EMNLP CoNLL ACL NAACL-HLT EMNLP NAACL-HLT ICLR LOUHI@EMNLP FIRE (Working Notes) LREC EMNLP/IJCNLP (1) AAAI TREC RANLP NAACL-HLT ArXiv EMNLP CCL *SEM@NAACL-HLT NAACL-HLT (2) NAACL-HLT
https://api.semanticscholar.org/e1b188b33a8ddc314679551a74bb561bef30af06 https://api.semanticscholar.org/02e0ffe141c1e00f711121981e155d2ca2dd6a06 https://api.semanticscholar.org/35480cc277b94dd653721ebc95d3b8d9cde6a46a https://api.semanticscholar.org/3660ea5bc9a72bbf357dbb3f49b5d82dea8d20cf https://api.semanticscholar.org/4588c518deebb639174025498927a2945734b753 https://api.semanticscholar.org/bf30144192efa93dc65465bbbe4ea1dfcf417abb https://www.semanticscholar.org/paper/4bf6ce4a9366cdba069a45651606538f2febd8e6 https://www.semanticscholar.org/paper/ade0c116120b54b57a91da51235108b75c28375a https://www.semanticscholar.org/paper/c7d007ba376faddf0046930ea7375ed59600cee9 https://www.semanticscholar.org/paper/b1e20420982a4f923c08652941666b189b11b7fe https://www.semanticscholar.org/paper/0be9ca65ad318ee3729928882ef2c403d4b6d24e https://www.semanticscholar.org/paper/f37076f426023241f19cdc2fb0a0fd733a6fa7fa https://www.semanticscholar.org/paper/cc5582606aef717855bb37c7aa35639da755cb13 https://www.semanticscholar.org/paper/0fc47b976051366cc1b1ec00e0f3d512deb3232a https://www.semanticscholar.org/paper/060ff1aad5619a7d6d6cdfaf8be5da29bff3808c https://www.semanticscholar.org/paper/3febb2bed8865945e7fddc99efd791887bb7e14f https://www.semanticscholar.org/paper/3a7b63b50c64f4ec3358477790e84cbd6be2a0b4 https://api.semanticscholar.org/ab293e2efc083437df5f9fed58c026240a12ca4d https://api.semanticscholar.org/c4ce31ec4e167b008557f85561942f21295c2719 https://api.semanticscholar.org/4fa3daf3287b9e9725d70c8701fb384a72f2e84e https://api.semanticscholar.org/02e51b30387cacb8e4ebd4b8e1a5860cd00665a2 https://api.semanticscholar.org/f2a9ef36b79c1b25d15bf10ff061db5eaad06798 https://api.semanticscholar.org/207b3a97a0fb13d7c04d1107807f0e990f837a21 https://api.semanticscholar.org/9abb8ff6bfb8ec175179f9605421b77bb1f09338 https://api.semanticscholar.org/f23dae278cf98a7b46ad3ca93385f18c7903009d https://api.semanticscholar.org/0884a1b616d64e5b2c252f5980acf2049b2d8b80 https://api.semanticscholar.org/d3fbef9ed6136e200c8ae7f296e46f9241102119 https://api.semanticscholar.org/08cd047b370b37c171a00825e06f758235f7ec7f https://api.semanticscholar.org/4c636fa1843d15d0b0b7d3b8e3dca376b606abc7 https://api.semanticscholar.org/cb292f86a3fea9cc964539e5c544aac5828389c6 https://www.semanticscholar.org/paper/3febb2bed8865945e7fddc99efd791887bb7e14f https://www.semanticscholar.org/paper/14908a18ff831005b6b4fc953ce61e1b4e7b54ee https://www.semanticscholar.org/paper/060ff1aad5619a7d6d6cdfaf8be5da29bff3808c https://api.semanticscholar.org/6dc0eceea3f8bafcec62cedc2054b65f86dd7c73 https://api.semanticscholar.org/c8b65d126477f00a848d94e8d99cbb03a5d965a6 https://api.semanticscholar.org/330e998bd6c28ef407a0003e2b90b761e4cbb9b7 https://api.semanticscholar.org/5f9455eea572d7b341ac6016f71de9e7c5384b85 https://www.semanticscholar.org/paper/3febb2bed8865945e7fddc99efd791887bb7e14f