Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline. Pretrained word embeddings BID30 BID32 are a staple tool for NLP. These models provide continuous representations for word types, typically learned from cooccurrence statistics on unlabeled data, and improve generalization of downstream models across many domains. Recently, a number of models have been proposed for contextualized word embeddings. Instead of using a single, fixed vector per word type, these models run a pretrained encoder network over the sentence to produce contextual embeddings of each token. The encoder, usually an LSTM BID18 or a Transformer BID47 , can be trained on objectives like machine translation BID29 or language modeling BID33 BID19 , for which large amounts of data are available. The activations of this network-a collection of one vector per token-fit the same interface as conventional word embeddings, and can be used as a drop-in replacement input to any model. Applied to popular models, this technique has yielded significant improvements to the state-of-the-art on several tasks, including constituency parsing BID22 , semantic role labeling BID44 , and coreference , and has outperformed competing techniques BID8 ) that produce fixed-length representations for entire sentences.Our goal in this work is to understand where these contextual representations improve over conventional word embeddings. Recent work has explored many token-level properties of these representations, such as their ability to capture part-of-speech tags BID4 BID3 BID42 , morphology BID2 b) , or word-sense disambiguation BID33 . BID34 extends this to constituent phrases, and present a heuristic for unsuper- Figure 1 : Probing model architecture ( § 3.1). All parameters inside the dashed line are fixed, while we train the span pooling and MLP classifiers to extract information from the contextual vectors. The example shown is for semantic role labeling, where s(1) = [1, 2) corresponds to the predicate ("eat"), while s (2) = [2, 5) is the argument ("strawberry ice cream"), and we predict label A1 as positive and others as negative. For entity and constituent labeling, only a single span is used.vised pronominal coreference. We expand on this even further and introduce a suite of edge probing tasks covering a broad range of syntactic, semantic, local, and long-range phenomena. In particular, we focus on asking what information is encoded at each position, and how well it encodes structural information about that word's role in the sentence. Is this information primarily syntactic in nature, or do the representations also encode higher-level semantic relationships? Is this information local, or do the encoders also capture long-range structure?We approach these questions with a probing model (Figure 1 ) that sees only the contextual embeddings from a fixed, pretrained encoder. The model can access only embeddings within given spans, such as a predicate-argument pair, and must predict properties, such as semantic roles, which typically require whole-sentence context. We use data derived from traditional structured NLP tasks: tagging, parsing, semantic roles, and coreference. Common corpora such as OntoNotes BID49 provide a wealth of annotations for well-studied concepts which are both linguistically motivated and known to be useful intermediates for high-level language understanding. We refer to our technique as "edge probing", as we decompose each structured task into a set of graph edges ( § 2) which we can predict independently using a common classifier architecture ( § 3.1) 2 . We probe four popular contextual representation models ( § 3.2): CoVe BID29 , ELMo BID33 , OpenAI GPT , and BERT .We focus on these models because their pretrained weights and code are available, since these are most likely to be used by researchers. We compare to word-level baselines to separate the contribution of context from lexical priors, and experiment with augmented baselines to better understand the role of pretraining and the ability of encoders to capture long-range dependencies. We introduce a suite of "edge probing" tasks designed to probe the sub-sentential structure of contextualized word embeddings. These tasks are derived from core NLP tasks and encompass a range of syntactic and semantic phenomena. We use these tasks to explore how contextual embeddings improve on their lexical (context-independent) baselines. We focus on four recent models for contextualized word embeddings-CoVe, ELMo, OpenAI GPT, and BERT.Based on our analysis, we find evidence suggesting the following trends. First, in general, contextualized embeddings improve over their non-contextualized counterparts largely on syntactic tasks (e.g. constituent labeling) in comparison to semantic tasks (e.g. coreference), suggesting that these embeddings encode syntax more so than higher-level semantics. Second, the performance of ELMo cannot be fully explained by a model with access to local context, suggesting that the contextualized representations do encode distant linguistic information, which can help disambiguate longer-range dependency relations and higher-level syntactic structures.We release our data processing and model code, and hope that this can be a useful tool to facilitate understanding of, and improvements in, contextualized word embedding models.
We probe for sentence structure in ELMo and related contextual embedding models. We find existing models efficiently encode syntax and show evidence of long-range dependencies, but only offer small improvements on semantic tasks. Proposes the "edge probing" method and focuses on the relationship between spans rather than individual words, enabling the authors to look at syntactic constituency, dependencies, entity labels, and semantic role labeling. Provides new insights on what is captured contextualized word embeddings by compiling a set of “edge probing” tasks. 