We tested a broad range of textual representations for their capacity to address these issues . We found that , as expected , contextualized word representations perform better than static word embeddings , more so on detecting meaning shift than in recovering implicit information , in which their performance is still far from that of humans . Our evaluation suite , consisting of six tasks related to lexical composition effects , can serve future research aiming to improve representations . This paper describes our submission for the SemEval-2019 Suggestion Mining task . A simple Convolutional Neural Network ( CNN ) classifier with contextual word representations from a pre-trained language model was used for sentence classification . The model is trained using tri-training , a semi-supervised bootstrapping mechanism for labelling unseen data . In this paper , we focus on the identification of biomedical abstracts related to protein-protein interactions . We propose a novel feature representation , contextual-bag-of-words , to exploit named entity information . Our method outperforms well-known methods that use named entity information as additional features . Contextual representations of words derived by neural language models have proven to effectively encode the subtle distinctions that might occur between different meanings of the same word . However , these representations are not tied to a semantic network , hence they leave the word meanings implicit and thereby neglect the information that can be derived from the knowledge base itself . Our contributions are as follows : First , we enable multi-granular matches between question and answer n-grams . Second , by contextualizing word representation with surrounding n-grams , MICRON can naturally utilize word-based signals for query term weighting , known to be effective in information retrieval . We validate MICRON in two public non-factoid question answering datasets : WikiPassageQA and InsuranceQA , showing our model achieves the state of the art among baselines with reported performances on both datasets . Word Embedding Layer maps each word to a vector space using a pre - trained word embedding model . Contextual Embedding Layer utilizes contextual cues from surrounding words to refine the embedding of the words . These first three layers are applied to both the query and context . This reduces the information loss caused by early summarization . The inputs to the layer are contextual vector representations of the context and the query . The outputs of the layer are the query - aware vector representations of the context words , , along with the contextual embeddings from the previous layer . The output of the modeling layer captures the interaction among the context words conditioned on the query . This is different from the contextual embedding layer , which captures the interaction among context words independent of the query . We use two layers of bi - directional LSTM , with the output size of for each direction . What makes a universal sentence encoder universal ? The notion of a generic encoder of text appears to be at odds with the inherent contextualization and non-permanence of language use in a dynamic world . Apresentamos um modelo preliminar baseado em aprendizado de máquina para classificar a opinião de tweets em português sobre o tema da Reforma de Previdência no contexto político brasileiro . Nossa abordagem utiliza o Multilingual Universal Sentence Encoder for Semantic Retrieval , um modelo pré-treinado recém-lançado por pesquisadores do Google para gerar representações vetoriais semanticamente ricas . Treinamos e classificamos nosso dataset em uma rede neural profunda feedfoward . The main aim of building a new model is to provide better fine-grained results for use in the Natural Language Processing fields . In our proposed methodology we utilized transfer learning with deep neural networks to build a Pooled-GRU model combined with the Multilingual Universal Sentence Encoder . Our proposed model scored about 17 % enhancement when being compared to previous work . In this study , we introduce two new Polish datasets for evaluating sentence embeddings and provide a comprehensive evaluation of eight sentence representation methods including Polish and multilingual models . We consider classic word embedding models , recently developed contextual embeddings and multilingual sentence encoders , showing strengths and weaknesses of specific approaches . We also examine different methods of aggregating word vectors into a single sentence vector . We also provide an upper bound baseline for both tasks . We found that for common XCMRC task , translation-based method and multilingual sentence encoder-based method can obtain reasonable performance but still have much room for improvement . As for pseudo low-resource XCMRC task , due to strict restrictions on the use of language resources , our two approaches are far below the upper bound so there are many challenges ahead . The current study aims to leverage visual information in order to capture sentence level semantics without the need for word embeddings . We use a multimodal sentence encoder trained on a corpus of images with matching text captions to produce visually grounded sentence embeddings . Deep Neural Networks are trained to map the two modalities to a common embedding space such that for an image the corresponding caption can be retrieved and vice versa . The first model is based on Convolutional Neuronal Networks whose outputs feed a LSTM Neural Network . The second one uses the pre-trained Universal Sentence Encoder for encoding sentences into embedding vectors . Finally , the third one consists in a simple Multi-layer Perceptron . BiLSTM with max pooling is adopted as a well-performed supervised universal sentence encoder . Max pooling is a common mechanism to get a fixed-size sentence representation . In this paper we present the impact of top-performing sentence embedding methodologies on the accuracy of a neural model trained to assess the quality of English sentences . We focus our efforts in the methodologies called Language Agnostic SEntence Representation ( LASER ) , Sentence to Vector ( S2V ) , and Universal Sentence Encoder ( USE ) to observe their ability to capture information related to sentence quality . Our study suggests that these state-of-the-art sentence embeddings are unable to capture sufficient information regarding sentence correctness and quality in the English language . Several attempts at learning unsupervised representations of sentences have not reached satisfactory enough performance to be widely adopted . In this paper , we show how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks . Much like how computer vision uses ImageNet to obtain features , which can then be transferred to other tasks , our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks . This work combines two research directions , which we describe in what follows . First , we explain how the NLI task can be used to train universal sentence encoding models using the SNLI task . We subsequently describe the architectures that we investigated for the sentence encoder , which , in our opinion , covers a suitable range of sentence encoders currently in use . ( i ) sentence encoding - based models that explicitly separate the encoding of the individual sentences and ( ii ) joint methods that allow to use encoding of both sentences ( to use cross - features or attention from one sentence to the other ) . Since our goal is to train a generic sentence encoder , we adopt the first setting . As illustrated in Figure [ reference ] , a typical architecture of this kind uses a shared sentence encoder that outputs a representation for the premise and the hypothesis . Once the sentence vectors are generated , 3 matching methods are applied to extract relations between and : ( i ) concatenation of the two representations ; ( ii ) element - wise product ; and ( iii ) absolute element - wise difference . We showed that models learned on NLI can perform better than models trained in unsupervised conditions or on other supervised tasks . By exploring various architectures , we showed that a BiLSTM network with max pooling makes the best current universal sentence encoding methods , outperforming existing approaches like SkipThought vectors . We believe that this work only scratches the surface of possible combinations of models and tasks for learning generic sentence embeddings . Some tasks ( BUCC , MLDoc ) tend to perform better when the encoder is trained on long and formal sentences , whereas other tasks ( XNLI , Tatoeba ) benefit from training on shorter and more informal sentences . In an attempt to achieve a general purpose sentence encoder that performs well on all tasks , we aimed at balancing the size of training corpora with long and short sentences . For that purpose , we used at most two million sentences from OpenSubtitles , although more data is available for some languages . The most notable effort in this regard is probably the XNLI corpus , an NLI test set similar to MultiNLI for which the premises and hypotheses were translated into 14 languages by professional translators . We train an NLI classifier on top of our multilingual sentence embedding using English training data , and evaluate its zero - shot transfer performance in the remaining languages ( Section [ reference ] ) . So as to obtain a more complete picture of the behavior of our multilingual sentence representations , we also evaluate them in cross - lingual document classification ( MLDoc , Section [ reference ] ) , and bitext mining ( BUCC , Section [ reference ] ) . Note that no human translated training data is provided ; instead , different systems are to use English training data from MultiNLI , and their transfer performance is evaluated on the rest of languages . We train a classifier on top of our multilingual encoder using the usual combination of the two sentence embeddings : , where and are the premise and hypothesis . For that purpose , we use a feed - forward neural network with two hidden layers of size 512 and 384 , trained with Adam . In this paper , we propose a new method for this task based on multilingual sentence embeddings . Our approach uses an encoder - decoder trained over an initial parallel corpus to build multilingual sentence representations , which are then incorporated into a new margin - based method to score , mine and filter parallel sentences . In contrast to previous approaches , which rely on nearest neighbor retrieval with a hard threshold over cosine similarity , our proposed method accounts for the scale inconsistencies of this measure , considering the margin between a given sentence pair and its closest candidates instead . section : Scoring and filtering parallel sentences Having learned a multilingual sentence encoder as described in Section [ reference ] , parallel sentences can be mined by taking the nearest neighbor of each source sentence in the target side according to cosine similarity , and filtering those below a fixed threshold . While this approach has been reported to be competitive in previous work schwenk2018filtering , we argue that it suffers from the fact that the scale of cosine similarity is not globally consistent across different sentences . We present a comparison of different word and context representations on the task of proposing substitutes for a target word in context ( lexical substitution ) . We also experiment with tuning contextualized word embeddings on a dataset of sense-specific instances for each target word . We show that powerful contextualized word representations , which give high performance in several semantics-related tasks , deal less well with the subtle in-context similarity relationships needed for substitution . We compare classical machine learning techniques to the use of a more advanced model : the contextual word embeddings in the particular case of classification of abusive short-texts for the Spanish language . As contextual word embeddings , we use Bidirectional Encoder Representation from Transformers ( BERT ) , we show that it mostly outperforms classical techniques . Far beyond the experimental impact of our research , this project aims at planting the seeds for an innovative technological tool with a high potential social impact and aiming at being part of the initiatives in artificial intelligence for social good . Therefore , this thesis proposes ways of employing machine learning techniques to process the incident reports more efficiently and effectively . More specifically , we show how context-based word embeddings , vector representations of words , can be used to enhance searching capabilities within the report database and how they can be combined with classifiers to predict labels attributed to the incidents . For these purposes , we subjected several word embedding and classification techniques to a comparative analysis . Compared to our work these models were trained with Dropout regularization which might improve single model performance . However , ensemble of our models outperforms these models even though they use pre - trained word embeddings . On the CNN dataset our single model with best validation accuracy achieves a test accuracy of 69.5 % . [ reference ] use a sentencelevel character LSTM encoding for parsing . [ reference ] show that contextual embeddings using character convolutions improve accuracy for number of NLP tasks . [ reference ] is probably the jumping - off point for most current architectures for tagging models with recurrent neural networks . First , this layer projects each of the one - hot vectors ( of size ) into a - dimensional continuous vector space with a pre - trained weight matrix such as GloVe PenningtonSM14 . Next , it uses contextualized word representations , ELMo PetersNIGCLZ18 , which is a character - level two - layer bidirectional language model pre - trained on a large - scale corpus . ELMo representations allow our model to use morphological clues to form robust representations for out - of - vocabulary words unseen in training . Our adaptation of AT for use in software engineering has evolved over time as we have used it in a case study and developed limited tools that can support designers both during initial system design and during system evolution . Here we describe how the USE tool was applied to develop the DSL and how we have used this tool to create instances of AT models and analyze them for structural constraint inconsistencies . We identify some of the issues encountered in this process and some of the remaining open issues regarding a USE model as an implementation of our DSL . We describe the features available in the UML/OCL tool USE and extend its features by introducing a language for defining properties of desired snapshots and by showing how such snapshots are generated . We explain the functionality offered by the USE tool . In particular , we demonstrate how the diverse windows , e.g. , object diagram , class invariant , class extent , or OCL expression evaluation window , support the development process in early phases of software development . When designing nanomagnetic circuitry , several challenges arise , such as the design of a clocking system able to avoid signal disruption due to the thermal noise effect . In this paper , we compare four NML clocking schemes : BANCS , USE , RES , and 2DDWave by analyzing scalability and area overhead of combinational and sequential circuits . However , we believe that , in addition to recognising the expressive power of modelling languages , it is crucial to detect the capabilities and the weaknesses of analysis tools that parse and analyse models written in these languages . In order to explore this area , we have chosen four model analysis tools : USE , Alloy Analyzer , ZLive and ProZ and observed how modelling and validation stages of MBT are handled by these tools for the same system . Through this experiment , we not only concretise the tasks that form the modelling and validation stages of MBT process , but also reveal how efficiently these tasks are carried out in different tools . EIGHTY-THREE REFERENCES ARE PRESENTED ON LAND USE STUDIES . LAND USE METHODOLOGY IS IN A RAPID STATE OF EVOLUTION . THE THREE MAJOR COMPONENTS OF A LAND USE INVENTORY APPEAR TO BE : ( 1 ) PLACE , OR SOME FORM OF GEOGRAPHIC IDENTIFICATION , ( 2 ) A MEASURE OF QUANTITY OR INTENSITY , SUCH AS AREA OF THE PARCEL , AND ( 3 ) CLASSIFICATION OF THE LAND USE OR ACTIVITY THROUGH A CODING SYSTEM . Abstract This paper presents the User Software Engineering ( USE ) approach for developing specifications for an interactive information system ( IIS ) and shows how the method is applied to the specification of a Perinatal Data Registry system . Two linked views of the system are developed : a user view suitable for computer-naive users , and a design/verification view , suitable for computer-knowledgeable users . Sponsored by SIG USE , SIG DL , SIG ED . It is increasingly easy to imagine the time in the future when most of the information and some of the support needed by library users is available electronically . They also pose questions that the existing metrics may not yet answer , which may be critical in solving the toughest problems . System methodologies include the USE method , workload characterization , drill-down analysis , off-CPU analysis , and more . This talk will discuss various system performance issues , and the methodologies , tools , and processes used to solve them . Now , with the availability of 3-D ultrasound data , a lot of research efforts are being devoted to developing 3-D ultrasound strain elastography ( USE ) systems . Because 3-D motion tracking , a core component in any 3-D USE system , is computationally intensive , a lot of efforts are under way to accelerate 3-D motion tracking . In the literature , the concept of Sum-Table has been used in a serial computing environment to reduce the burden of computing signal correlation , which is the single most computationally intensive component in 3-D motion tracking . We enable users to control lighting , a TV set , a coffee machine and the shutters of the smart home . We evaluate the performance ( accuracy , interaction time ) , usability and feasibility ( USE questionnaire ) on 12 healthy subjects and 2 disabled subjects . We find that healthy subjects achieve 77 % task accuracy . Abstract . This paper describes the structure of the LAND USE SCANNER model , a GIS based model developed to generate spatial forecasts for various types of land use for a large number of grids . The model basically allocates land according to bid prices for various types of land use . Its network and value-added services enable customers to reduce costs , raise productivity and control risks . Security is one of the key benefits that S.W.I.F.T. offers to customers , and the User Security Enhancement ( USE ) programme is a major part of the company 's security strategy . In this paper , we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation ( MT ) to contextualize word vectors . We show that adding these context vectors ( CoVe ) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks : sentiment analysis ( SST , IMDb ) , question classification ( TREC ) , entailment ( SNLI ) , and question answering ( SQuAD ) . For fine-grained sentiment analysis and entailment , CoVe improves performance of our baseline models to the state of the art . The addition of ELMo representations alone significantly improves the state of the art in every case , including up to 20 % relative error reductions . For tasks where direct comparisons are possible , ELMo outperforms CoVe McCann2017LearnedIT , which computes contextualized representations using a neural machine translation encoder . Finally , an analysis of both ELMo and CoVe reveals that deep representations outperform those derived from just the top layer of an LSTM . Our trained models and code are publicly available , and we expect that ELMo will provide similar gains for many other NLP problems . This is competitive with a state - of - the - art WSD - specific supervised model using hand crafted features Iacobacci2016EmbeddingsFW and a task specific biLSTM that is also trained with auxiliary coarse - grained semantic labels and POS tags Raganato2017NeuralSL . The CoVe biLSTM layers follow a similar pattern to those from the biLM ( higher overall performance at the second layer compared to the first ) ; however , our biLM outperforms the CoVe biLSTM , which trails the WordNet first sense baseline . POS tagging To examine whether the biLM captures basic syntax , we used the context representations as input to a linear classifier that predicts POS tags with the Wall Street Journal portion of the Penn Treebank ( PTB ) Marcus1993BuildingAL . However , unlike WSD , accuracies using the first biLM layer are higher than the top layer , consistent with results from deep biLSTMs in multi - task training Sgaard2016DeepML , joint - many - iclr07 and MT Belinkov2017WhatDN . CoVe POS tagging accuracies follow the same pattern as those from the biLM , and just like for WSD , the biLM achieves higher accuracies than the CoVe encoder . Implications for supervised tasks Taken together , these experiments confirm different layers in the biLM represent different types of information and explain why including all biLM layers is important for the highest performance in downstream tasks .
We tested a broad range of textual representations for their capacity to address these issues . We found that , as expected , contextualized word representations perform better than static word embeddings , more so on detecting meaning shift than in recovering implicit information , in which their performance is still far from that of humans . Our evaluation suite , consisting of six tasks related to lexical composition effects , can serve future research aiming to improve representations .
This paper describes our submission for the SemEval-2019 Suggestion Mining task . A simple Convolutional Neural Network ( CNN ) classifier with contextual word representations from a pre-trained language model was used for sentence classification . The model is trained using tri-training , a semi-supervised bootstrapping mechanism for labelling unseen data .
In this paper , we focus on the identification of biomedical abstracts related to protein-protein interactions . We propose a novel feature representation , contextual-bag-of-words , to exploit named entity information . Our method outperforms well-known methods that use named entity information as additional features .
Contextual representations of words derived by neural language models have proven to effectively encode the subtle distinctions that might occur between different meanings of the same word . However , these representations are not tied to a semantic network , hence they leave the word meanings implicit and thereby neglect the information that can be derived from the knowledge base itself .
Our contributions are as follows : First , we enable multi-granular matches between question and answer n-grams . Second , by contextualizing word representation with surrounding n-grams , MICRON can naturally utilize word-based signals for query term weighting , known to be effective in information retrieval . We validate MICRON in two public non-factoid question answering datasets : WikiPassageQA and InsuranceQA , showing our model achieves the state of the art among baselines with reported performances on both datasets .
Word Embedding Layer maps each word to a vector space using a pre - trained word embedding model . Contextual Embedding Layer utilizes contextual cues from surrounding words to refine the embedding of the words . These first three layers are applied to both the query and context .
This reduces the information loss caused by early summarization . The inputs to the layer are contextual vector representations of the context and the query . The outputs of the layer are the query - aware vector representations of the context words , , along with the contextual embeddings from the previous layer .
The output of the modeling layer captures the interaction among the context words conditioned on the query . This is different from the contextual embedding layer , which captures the interaction among context words independent of the query . We use two layers of bi - directional LSTM , with the output size of for each direction .
What makes a universal sentence encoder universal ? The notion of a generic encoder of text appears to be at odds with the inherent contextualization and non-permanence of language use in a dynamic world .
Apresentamos um modelo preliminar baseado em aprendizado de máquina para classificar a opinião de tweets em português sobre o tema da Reforma de Previdência no contexto político brasileiro . Nossa abordagem utiliza o Multilingual Universal Sentence Encoder for Semantic Retrieval , um modelo pré-treinado recém-lançado por pesquisadores do Google para gerar representações vetoriais semanticamente ricas . Treinamos e classificamos nosso dataset em uma rede neural profunda feedfoward .
The main aim of building a new model is to provide better fine-grained results for use in the Natural Language Processing fields . In our proposed methodology we utilized transfer learning with deep neural networks to build a Pooled-GRU model combined with the Multilingual Universal Sentence Encoder . Our proposed model scored about 17 % enhancement when being compared to previous work .
In this study , we introduce two new Polish datasets for evaluating sentence embeddings and provide a comprehensive evaluation of eight sentence representation methods including Polish and multilingual models . We consider classic word embedding models , recently developed contextual embeddings and multilingual sentence encoders , showing strengths and weaknesses of specific approaches . We also examine different methods of aggregating word vectors into a single sentence vector .
We also provide an upper bound baseline for both tasks . We found that for common XCMRC task , translation-based method and multilingual sentence encoder-based method can obtain reasonable performance but still have much room for improvement . As for pseudo low-resource XCMRC task , due to strict restrictions on the use of language resources , our two approaches are far below the upper bound so there are many challenges ahead .
The current study aims to leverage visual information in order to capture sentence level semantics without the need for word embeddings . We use a multimodal sentence encoder trained on a corpus of images with matching text captions to produce visually grounded sentence embeddings . Deep Neural Networks are trained to map the two modalities to a common embedding space such that for an image the corresponding caption can be retrieved and vice versa .
The first model is based on Convolutional Neuronal Networks whose outputs feed a LSTM Neural Network . The second one uses the pre-trained Universal Sentence Encoder for encoding sentences into embedding vectors . Finally , the third one consists in a simple Multi-layer Perceptron .
BiLSTM with max pooling is adopted as a well-performed supervised universal sentence encoder . Max pooling is a common mechanism to get a fixed-size sentence representation .
In this paper we present the impact of top-performing sentence embedding methodologies on the accuracy of a neural model trained to assess the quality of English sentences . We focus our efforts in the methodologies called Language Agnostic SEntence Representation ( LASER ) , Sentence to Vector ( S2V ) , and Universal Sentence Encoder ( USE ) to observe their ability to capture information related to sentence quality . Our study suggests that these state-of-the-art sentence embeddings are unable to capture sufficient information regarding sentence correctness and quality in the English language .
Several attempts at learning unsupervised representations of sentences have not reached satisfactory enough performance to be widely adopted . In this paper , we show how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks . Much like how computer vision uses ImageNet to obtain features , which can then be transferred to other tasks , our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks .
This work combines two research directions , which we describe in what follows . First , we explain how the NLI task can be used to train universal sentence encoding models using the SNLI task . We subsequently describe the architectures that we investigated for the sentence encoder , which , in our opinion , covers a suitable range of sentence encoders currently in use .
( i ) sentence encoding - based models that explicitly separate the encoding of the individual sentences and ( ii ) joint methods that allow to use encoding of both sentences ( to use cross - features or attention from one sentence to the other ) . Since our goal is to train a generic sentence encoder , we adopt the first setting . As illustrated in Figure [ reference ] , a typical architecture of this kind uses a shared sentence encoder that outputs a representation for the premise and the hypothesis . Once the sentence vectors are generated , 3 matching methods are applied to extract relations between and : ( i ) concatenation of the two representations ; ( ii ) element - wise product ; and ( iii ) absolute element - wise difference .
We showed that models learned on NLI can perform better than models trained in unsupervised conditions or on other supervised tasks . By exploring various architectures , we showed that a BiLSTM network with max pooling makes the best current universal sentence encoding methods , outperforming existing approaches like SkipThought vectors . We believe that this work only scratches the surface of possible combinations of models and tasks for learning generic sentence embeddings .
Some tasks ( BUCC , MLDoc ) tend to perform better when the encoder is trained on long and formal sentences , whereas other tasks ( XNLI , Tatoeba ) benefit from training on shorter and more informal sentences . In an attempt to achieve a general purpose sentence encoder that performs well on all tasks , we aimed at balancing the size of training corpora with long and short sentences . For that purpose , we used at most two million sentences from OpenSubtitles , although more data is available for some languages .
The most notable effort in this regard is probably the XNLI corpus , an NLI test set similar to MultiNLI for which the premises and hypotheses were translated into 14 languages by professional translators . We train an NLI classifier on top of our multilingual sentence embedding using English training data , and evaluate its zero - shot transfer performance in the remaining languages ( Section [ reference ] ) . So as to obtain a more complete picture of the behavior of our multilingual sentence representations , we also evaluate them in cross - lingual document classification ( MLDoc , Section [ reference ] ) , and bitext mining ( BUCC , Section [ reference ] ) .
Note that no human translated training data is provided ; instead , different systems are to use English training data from MultiNLI , and their transfer performance is evaluated on the rest of languages . We train a classifier on top of our multilingual encoder using the usual combination of the two sentence embeddings : , where and are the premise and hypothesis . For that purpose , we use a feed - forward neural network with two hidden layers of size 512 and 384 , trained with Adam .
In this paper , we propose a new method for this task based on multilingual sentence embeddings . Our approach uses an encoder - decoder trained over an initial parallel corpus to build multilingual sentence representations , which are then incorporated into a new margin - based method to score , mine and filter parallel sentences . In contrast to previous approaches , which rely on nearest neighbor retrieval with a hard threshold over cosine similarity , our proposed method accounts for the scale inconsistencies of this measure , considering the margin between a given sentence pair and its closest candidates instead .
section : Scoring and filtering parallel sentences Having learned a multilingual sentence encoder as described in Section [ reference ] , parallel sentences can be mined by taking the nearest neighbor of each source sentence in the target side according to cosine similarity , and filtering those below a fixed threshold . While this approach has been reported to be competitive in previous work schwenk2018filtering , we argue that it suffers from the fact that the scale of cosine similarity is not globally consistent across different sentences .
We present a comparison of different word and context representations on the task of proposing substitutes for a target word in context ( lexical substitution ) . We also experiment with tuning contextualized word embeddings on a dataset of sense-specific instances for each target word . We show that powerful contextualized word representations , which give high performance in several semantics-related tasks , deal less well with the subtle in-context similarity relationships needed for substitution .
We compare classical machine learning techniques to the use of a more advanced model : the contextual word embeddings in the particular case of classification of abusive short-texts for the Spanish language . As contextual word embeddings , we use Bidirectional Encoder Representation from Transformers ( BERT ) , we show that it mostly outperforms classical techniques . Far beyond the experimental impact of our research , this project aims at planting the seeds for an innovative technological tool with a high potential social impact and aiming at being part of the initiatives in artificial intelligence for social good .
Therefore , this thesis proposes ways of employing machine learning techniques to process the incident reports more efficiently and effectively . More specifically , we show how context-based word embeddings , vector representations of words , can be used to enhance searching capabilities within the report database and how they can be combined with classifiers to predict labels attributed to the incidents . For these purposes , we subjected several word embedding and classification techniques to a comparative analysis .
Compared to our work these models were trained with Dropout regularization which might improve single model performance . However , ensemble of our models outperforms these models even though they use pre - trained word embeddings . On the CNN dataset our single model with best validation accuracy achieves a test accuracy of 69.5 % .
[ reference ] use a sentencelevel character LSTM encoding for parsing . [ reference ] show that contextual embeddings using character convolutions improve accuracy for number of NLP tasks . [ reference ] is probably the jumping - off point for most current architectures for tagging models with recurrent neural networks .
First , this layer projects each of the one - hot vectors ( of size ) into a - dimensional continuous vector space with a pre - trained weight matrix such as GloVe PenningtonSM14 . Next , it uses contextualized word representations , ELMo PetersNIGCLZ18 , which is a character - level two - layer bidirectional language model pre - trained on a large - scale corpus . ELMo representations allow our model to use morphological clues to form robust representations for out - of - vocabulary words unseen in training .
Our adaptation of AT for use in software engineering has evolved over time as we have used it in a case study and developed limited tools that can support designers both during initial system design and during system evolution . Here we describe how the USE tool was applied to develop the DSL and how we have used this tool to create instances of AT models and analyze them for structural constraint inconsistencies . We identify some of the issues encountered in this process and some of the remaining open issues regarding a USE model as an implementation of our DSL .
We describe the features available in the UML/OCL tool USE and extend its features by introducing a language for defining properties of desired snapshots and by showing how such snapshots are generated . We explain the functionality offered by the USE tool . In particular , we demonstrate how the diverse windows , e.g. , object diagram , class invariant , class extent , or OCL expression evaluation window , support the development process in early phases of software development .
When designing nanomagnetic circuitry , several challenges arise , such as the design of a clocking system able to avoid signal disruption due to the thermal noise effect . In this paper , we compare four NML clocking schemes : BANCS , USE , RES , and 2DDWave by analyzing scalability and area overhead of combinational and sequential circuits .
However , we believe that , in addition to recognising the expressive power of modelling languages , it is crucial to detect the capabilities and the weaknesses of analysis tools that parse and analyse models written in these languages . In order to explore this area , we have chosen four model analysis tools : USE , Alloy Analyzer , ZLive and ProZ and observed how modelling and validation stages of MBT are handled by these tools for the same system . Through this experiment , we not only concretise the tasks that form the modelling and validation stages of MBT process , but also reveal how efficiently these tasks are carried out in different tools .
EIGHTY-THREE REFERENCES ARE PRESENTED ON LAND USE STUDIES . LAND USE METHODOLOGY IS IN A RAPID STATE OF EVOLUTION . THE THREE MAJOR COMPONENTS OF A LAND USE INVENTORY APPEAR TO BE : ( 1 ) PLACE , OR SOME FORM OF GEOGRAPHIC IDENTIFICATION , ( 2 ) A MEASURE OF QUANTITY OR INTENSITY , SUCH AS AREA OF THE PARCEL , AND ( 3 ) CLASSIFICATION OF THE LAND USE OR ACTIVITY THROUGH A CODING SYSTEM .
Abstract This paper presents the User Software Engineering ( USE ) approach for developing specifications for an interactive information system ( IIS ) and shows how the method is applied to the specification of a Perinatal Data Registry system . Two linked views of the system are developed : a user view suitable for computer-naive users , and a design/verification view , suitable for computer-knowledgeable users .
Sponsored by SIG USE , SIG DL , SIG ED . It is increasingly easy to imagine the time in the future when most of the information and some of the support needed by library users is available electronically .
They also pose questions that the existing metrics may not yet answer , which may be critical in solving the toughest problems . System methodologies include the USE method , workload characterization , drill-down analysis , off-CPU analysis , and more . This talk will discuss various system performance issues , and the methodologies , tools , and processes used to solve them .
Now , with the availability of 3-D ultrasound data , a lot of research efforts are being devoted to developing 3-D ultrasound strain elastography ( USE ) systems . Because 3-D motion tracking , a core component in any 3-D USE system , is computationally intensive , a lot of efforts are under way to accelerate 3-D motion tracking . In the literature , the concept of Sum-Table has been used in a serial computing environment to reduce the burden of computing signal correlation , which is the single most computationally intensive component in 3-D motion tracking .
We enable users to control lighting , a TV set , a coffee machine and the shutters of the smart home . We evaluate the performance ( accuracy , interaction time ) , usability and feasibility ( USE questionnaire ) on 12 healthy subjects and 2 disabled subjects . We find that healthy subjects achieve 77 % task accuracy .
Abstract . This paper describes the structure of the LAND USE SCANNER model , a GIS based model developed to generate spatial forecasts for various types of land use for a large number of grids . The model basically allocates land according to bid prices for various types of land use .
Its network and value-added services enable customers to reduce costs , raise productivity and control risks . Security is one of the key benefits that S.W.I.F.T. offers to customers , and the User Security Enhancement ( USE ) programme is a major part of the company 's security strategy .
In this paper , we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation ( MT ) to contextualize word vectors . We show that adding these context vectors ( CoVe ) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks : sentiment analysis ( SST , IMDb ) , question classification ( TREC ) , entailment ( SNLI ) , and question answering ( SQuAD ) . For fine-grained sentiment analysis and entailment , CoVe improves performance of our baseline models to the state of the art .
The addition of ELMo representations alone significantly improves the state of the art in every case , including up to 20 % relative error reductions . For tasks where direct comparisons are possible , ELMo outperforms CoVe McCann2017LearnedIT , which computes contextualized representations using a neural machine translation encoder . Finally , an analysis of both ELMo and CoVe reveals that deep representations outperform those derived from just the top layer of an LSTM . Our trained models and code are publicly available , and we expect that ELMo will provide similar gains for many other NLP problems .
This is competitive with a state - of - the - art WSD - specific supervised model using hand crafted features Iacobacci2016EmbeddingsFW and a task specific biLSTM that is also trained with auxiliary coarse - grained semantic labels and POS tags Raganato2017NeuralSL . The CoVe biLSTM layers follow a similar pattern to those from the biLM ( higher overall performance at the second layer compared to the first ) ; however , our biLM outperforms the CoVe biLSTM , which trails the WordNet first sense baseline . POS tagging To examine whether the biLM captures basic syntax , we used the context representations as input to a linear classifier that predicts POS tags with the Wall Street Journal portion of the Penn Treebank ( PTB ) Marcus1993BuildingAL .
However , unlike WSD , accuracies using the first biLM layer are higher than the top layer , consistent with results from deep biLSTMs in multi - task training Sgaard2016DeepML , joint - many - iclr07 and MT Belinkov2017WhatDN . CoVe POS tagging accuracies follow the same pattern as those from the biLM , and just like for WSD , the biLM achieves higher accuracies than the CoVe encoder . Implications for supervised tasks Taken together , these experiments confirm different layers in the biLM represent different types of information and explain why including all biLM layers is important for the highest performance in downstream tasks .
Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition Zoho at SemEval-2019 Task 9: Semi-supervised Domain Adaptation using Tri-training for Suggestion Mining Identifying Protein Interaction Abstracts with Contextual Bag of Words SensEmBERT: Context-Enhanced Sense Embeddings for Multilingual Word Sense Disambiguation MICRON: Multigranular Interaction for Contextualizing RepresentatiON in Non-factoid Question Answering Bidirectional Attention Flow for Machine Comprehension Contextual Lensing of Universal Sentence Representations Classificação de opinião no Twitter em português utilizando o Multilingual Universal Sentence Encoder para apoiar pesquisas sobre filter bubble Transfer Learning for Arabic Named Entity Recognition With Deep Neural Networks Evaluation of Sentence Representations in Polish XCMRC: Evaluating Cross-lingual Machine Reading Comprehension Learning semantic sentence representations from visually grounded language without lexical knowledge Aggressive Analysis in Twitter using a Combination of Models An Improved Mechanism for Universal Sentence Representations Learnt from Natural Language Inference Data Using Bi-directional Information Empirical Study of Sentence Embeddings for English Sentences Quality Assessment Supervised Learning of Universal Sentence Representations from Natural Language Inference Data Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings A Comparison of Context-sensitive Models for Lexical Substitution Abusive language in Spanish children and young teenager's conversations: data preparation and short text classification with contextual word embeddings Medical Incident Report Classification using Context-based Word Embeddings Text Understanding with the Attention Sum Reader Network Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings Multi-style Generative Reading Comprehension Experiences Developing a Requirements Language Based on the Psychological Framework Activity Theory Tool support for validating UML and OCL models through automatic snapshot generation Evaluating nanomagnetic logic circuit layouts using different clock schemes A Comparison of State-Based Modelling Tools for Model Validation Selected References On Land Use Inventory Methods A Specification Method for Interactive Medical Information Systems The library of the future: Interweaving the virtual and the physical. Sponsored by SIG USE, SIG DL, SIG ED System Methodology—Holistic Performance Analysis on Modern Systems Accelerating 3-D GPU-based Motion Tracking for Ultrasound Strain Elastography Using Sum-Tables: Analysis and Initial Results Feasibility of BCI Control in a Realistic Smart Home Environment LAND USE SCANNER: An integrated GIS based model for long term projections of land use in urban and rural areas User security enhancement: A worldwide security project Learned in Translation: Contextualized Word Vectors Deep contextualized word representations
a5ec883e080d858b5df60f1b5d711d514459b1e4 05ae1917294570cc42e3d39799720bb4fc8d3d93 919805b3ed6ae3b2a4e640146701402ae0b4299c ce424d51bccd2fc3e10d47fd829a6bb925ed9056 8e4c9cbd77f124a1d7e44b6e1166857a926faafb e0a513faef30fc5989e89467449bdcb7200f348c 6a2330617839eadd031ffc27528ae387363f965e 68114171ae0bd7287edff133702e9c637e19265e 05d3023ff47c7374cb128bf47f0e3268777200dc 0405ea2845f0cbb4549e0e41fd09df3e7989151c 1bc1682df67bf28352a1a4d72b740ed972202740 e964a459258e763e798b448f33832a1ed6f80159 ebb407ee120eb31a0cdfb3e1768b2e72981eea8c 4a9a4ffd200414fdbcc694775c8b21b7fcd98c43 a396391b8858753a07b6cecda68fca0d24688005 f8859b09c36f0ad3b6a9fd22b8c3bcc9ea7439eb 7096ae39138d7a9d64c4a182f7098d210da21969 e8290fc7dfd2e9eb64f154da7b285254c362fde3 ed71b233e2e1a7e9aa351e93592575e483c82b20 83c61b223cb8e0f916ad2c1906d14af07e6e7e70 5a5a949cbbf25aa80a373605e1b009748c739505 f532ec1bc0d5b31fd3905b11c51949c9bbc4af59 b9f497c5649947f58651f30816185ce0c56dda53 649c75e4bbfd2e32650e7ac331770e3846b30ff6 bcf53a3019a3ca0f43c3a46fd8271bcdaac34902 03c18a7159fbfebdd7f546fdc575321512354df7 0b2465b08c1dca7b550dce3632ec106f7572bf13 942c1eecf607a23d3311b0a822a9ab7459d7510a badd2613c3d1f4aaf5d80f556a16147a55a30543 bc8fa64625d9189f5801837e7b133e7fe3c581f7
["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Psychology","Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Medicine","Engineering"] ["Medicine","Psychology"] ["Geography","Computer Science"] ["Computer Science"] ["Computer Science"]
2019 2019 2007 2020 2019 2017 2020 2019 2020 2019 2019 2019 2019 2019 2019 2017 2019 2019 2019 2020 2017 2016 2018 2019 2013 2003 2020 2008 1969 1980 2005 2016 2019 2016 1999 1998 2017 2018
AAAI AAAI EMNLP/IJCNLP (1) ICLR IberLEF@SEPLN EMNLP Transactions of the Association for Computational Linguistics ACL IWCS (1) LREC ACL ACL ACL OCL@MoDELS TOOLS (46) ASIST NAACL-HLT
https://api.semanticscholar.org/a5ec883e080d858b5df60f1b5d711d514459b1e4 https://api.semanticscholar.org/05ae1917294570cc42e3d39799720bb4fc8d3d93 https://api.semanticscholar.org/919805b3ed6ae3b2a4e640146701402ae0b4299c https://api.semanticscholar.org/ce424d51bccd2fc3e10d47fd829a6bb925ed9056 https://api.semanticscholar.org/8e4c9cbd77f124a1d7e44b6e1166857a926faafb https://www.semanticscholar.org/paper/3a7b63b50c64f4ec3358477790e84cbd6be2a0b4 https://api.semanticscholar.org/e0a513faef30fc5989e89467449bdcb7200f348c https://api.semanticscholar.org/6a2330617839eadd031ffc27528ae387363f965e https://api.semanticscholar.org/68114171ae0bd7287edff133702e9c637e19265e https://api.semanticscholar.org/05d3023ff47c7374cb128bf47f0e3268777200dc https://api.semanticscholar.org/0405ea2845f0cbb4549e0e41fd09df3e7989151c https://api.semanticscholar.org/1bc1682df67bf28352a1a4d72b740ed972202740 https://api.semanticscholar.org/e964a459258e763e798b448f33832a1ed6f80159 https://api.semanticscholar.org/ebb407ee120eb31a0cdfb3e1768b2e72981eea8c https://api.semanticscholar.org/4a9a4ffd200414fdbcc694775c8b21b7fcd98c43 https://www.semanticscholar.org/paper/ee7b883e35d754ae4f71c21bb71f9f03e4ffbb2c https://www.semanticscholar.org/paper/160563abbd75265b19afc8b4169bab9e1eb33d97 https://www.semanticscholar.org/paper/30b09a853ab72e53078f1feefe6de5a847a2b169 https://api.semanticscholar.org/a396391b8858753a07b6cecda68fca0d24688005 https://api.semanticscholar.org/f8859b09c36f0ad3b6a9fd22b8c3bcc9ea7439eb https://api.semanticscholar.org/7096ae39138d7a9d64c4a182f7098d210da21969 https://www.semanticscholar.org/paper/f2e50e2ee4021f199877c8920f1f984481c723aa https://www.semanticscholar.org/paper/cc5582606aef717855bb37c7aa35639da755cb13 https://www.semanticscholar.org/paper/4c09d30704c0ceb128bb31ee09f957ee58d5032c https://api.semanticscholar.org/e8290fc7dfd2e9eb64f154da7b285254c362fde3 https://api.semanticscholar.org/ed71b233e2e1a7e9aa351e93592575e483c82b20 https://api.semanticscholar.org/83c61b223cb8e0f916ad2c1906d14af07e6e7e70 https://api.semanticscholar.org/5a5a949cbbf25aa80a373605e1b009748c739505 https://api.semanticscholar.org/f532ec1bc0d5b31fd3905b11c51949c9bbc4af59 https://api.semanticscholar.org/b9f497c5649947f58651f30816185ce0c56dda53 https://api.semanticscholar.org/649c75e4bbfd2e32650e7ac331770e3846b30ff6 https://api.semanticscholar.org/bcf53a3019a3ca0f43c3a46fd8271bcdaac34902 https://api.semanticscholar.org/03c18a7159fbfebdd7f546fdc575321512354df7 https://api.semanticscholar.org/0b2465b08c1dca7b550dce3632ec106f7572bf13 https://api.semanticscholar.org/942c1eecf607a23d3311b0a822a9ab7459d7510a https://api.semanticscholar.org/badd2613c3d1f4aaf5d80f556a16147a55a30543 https://api.semanticscholar.org/bc8fa64625d9189f5801837e7b133e7fe3c581f7 https://www.semanticscholar.org/paper/3febb2bed8865945e7fddc99efd791887bb7e14f