A large number of natural language processing tasks exist to analyze syntax, semantics, and information content of human language. These seemingly very different tasks are usually solved by specially designed architectures. In this paper, we provide the simple insight that a great variety of tasks can be represented in a single unified format consisting of labeling spans and relations between spans, thus a single task-independent model can be used across different tasks. We perform extensive experiments to test this insight on 10 disparate tasks as broad as dependency parsing (syntax), semantic role labeling (semantics), relation extraction (information content), aspect based sentiment analysis (sentiment), and many others, achieving comparable performance as state-of-the-art specialized models. We further demonstrate benefits in multi-task learning. We convert these datasets into a unified format to build a benchmark, which provides a holistic testbed for evaluating future models for generalized natural language analysis. A large number of natural language processing (NLP) tasks exist to analyze various aspects of human language, including syntax (e.g., constituency and dependency parsing), semantics (e.g., semantic role labeling), information content (e.g., named entity recognition and relation extraction), or sentiment (e.g. sentiment analysis). At first glance, these tasks are seemingly very different in both the structure of their output and the variety of information that they try to capture. To handle these different characteristics, researchers usually use specially designed neural network architectures. In this paper we ask the simple questions: are the task-specific architectures really necessary? Or with the appropriate representational methodology, can we devise a single model that can perform -and achieve state-of-the-art performance on -a large number of natural language analysis tasks? Interestingly, in the domain of efficient human annotation interfaces, it is already standard to use unified representations for a wide variety of NLP tasks. On the right we show one example of the annotation interface BRAT (Stenetorp et al., 2012) , which has been used for annotating data for tasks as broad as part-of-speech tagging, named entity recognition, relation extraction, and many others. Notably, this interface has a single unified format that consists of spans (e.g. the span of an entity), labels on the spans (e.g. the variety of entity such as "person" or "location"), and labeled relations between the spans (e.g. "born-in"). These labeled relations can form a tree or graph structure (e.g., dependency tree), expressing the linguistic structure of sentences. We detail this BRAT format and how it can be used to represent a wide number of natural language analysis tasks in Section 2. The simple hypothesis behind our paper is: if humans can perform natural language analysis in a single unified format, then perhaps machines can as well. Fortunately, there already exist NLP models that perform span prediction and prediction of relations between pairs of spans, such as the end-to-end neural coreference model of . We extend this model with minor architectural modifications (which are not our core contributions) and pre-trained contextualized (Peters et al., 2018) BERT (Devlin et al., 2019) BERT baseline (Shi & Lin, 2019) SpanBERT (Joshi et al., 2019) Single Table 1 : The unified span-relation model can work on multiple NLP tasks, in contrast to previous works usually designed for a subset of tasks. representations (e.g., BERT; Devlin et al. (2019) 1 ) then demonstrate the applicability and versatility of this single model on 10 tasks, including named entity recognition (NER), relation extraction (RE), coreference resolution (Coref.), open information extraction (OpenIE), part-of-speech tagging (POS), dependency parsing (Dep.), constituency parsing (Consti.), semantic role labeling (SRL), aspect based sentiment analysis (ABSA), and opinion role labeling (ORL). While previous work has used similar formalisms to understand the representations learned by pre-trained embeddings (Tenney et al., 2019a; b) , to the best of our knowledge this is the first work that uses such a unified model to actually perform analysis. Moreover, despite it simplicity we demonstrate that such a model can achieve comparable performance with special-purpose state-of-the-art models on the tasks above (Table 1 ). We also demonstrate that this framework allows us to easily perform multi-task learning among different tasks, leading to improvements when there are related tasks to be learned from or data is sparse. In summary, our contributions are: • We provide the simple insight that a great variety of natural language analysis tasks can be represented and solved in a single unified format, i.e., span-relation representations. This insight may seem obvious in hindsight, but it has not been examined, particularly to this scale, by previous work on model-building for NLP. • We perform extensive experiments to test this insight on 10 disparate tasks, achieving comparable empirical results as the state-of-the-art, using a single task-independent modeling framework. • We further use this framework to perform an analysis of the benefits from multi-task learning across all of the tasks above, gleaning various insights about task relatedness and how multi-task learning performs with different token representations. • Upon acceptance of the paper, we will release our General Language Analysis Datasets (GLAD) benchmark with 8 datasets covering 10 tasks in the BRAT format, and provide a leaderboard to facilitate future work on generalized models for NLP. Compared to the full sentence-level tasks in the GLUE leaderboard (Wang et al., 2019a ;b), we cover a wide variety of natural language analysis tasks that require analyzing of the finer grained text units (e.g., words, phrases, clauses). We provide the simple insight that a large number of natural language analysis tasks can be represented in a single format consisting of spans and relations between spans. As a result, these tasks can be solved in a single modeling framework that first extracts spans and predicts their labels, then predicts relations between extracted spans. We attempted 10 tasks with this SpanRel model under this unified representation and show that this generic task-independent model can achieve competitive performance as state-of-the-art methods tailored for each tasks. We merge 8 datasets into our GLAD benchmark for evaluating future models for natural language analysis. Table 7 : Single-task learning performance of the SpanRel model with different token representations. BERT large requires a large amount of memory so we cannot feed the entire document to the model in coreference resolution.
We use a single model to solve a great variety of natural language analysis tasks by formulating them in a unified span-relation format. This paper generalizes a wide range of natural language processing tasks as a single span-based framework and proposes a general architecture to solve all these problems. This work presents a unified formulation of various phrase and token level NLP tasks.