Summarization of long sequences into a concise statement is a core problem in natural language processing, requiring non-trivial understanding of the input. Based on the promising results of graph neural networks on highly structured data, we develop a framework to extend existing sequence encoders with a graph component that can reason about long-distance relationships in weakly structured data such as text. In an extensive evaluation, we show that the resulting hybrid sequence-graph models outperform both pure sequence models as well as pure graph models on a range of summarization tasks. Summarization, the task of condensing a large and complex input into a smaller representation that retains the core semantics of the input, is a classical task for natural language processing systems. Automatic summarization requires a machine learning component to identify important entities and relationships between them, while ignoring redundancies and common concepts.Current approaches to summarization are based on the sequence-to-sequence paradigm over the words of some text, with a sequence encoder -typically a recurrent neural network, but sometimes a 1D-CNN BID34 or using self-attention BID32 -processing the input and a sequence decoder generating the output. Recent successful implementations of this paradigm have substantially improved performance by focusing on the decoder, extending it with an attention mechanism over the input sequence and copying facilities BID32 . However, while standard encoders (e.g. bidirectional LSTMs) theoretically have the ability to handle arbitrary long-distance relationships, in practice they often fail to correctly handle long texts and are easily distracted by simple noise BID19 .In this work, we focus on an improvement of sequence encoders that is compatible with a wide range of decoder choices. To mitigate the long-distance relationship problem, we draw inspiration from recent work on highly-structured objects BID23 BID20 BID14 BID12 . In this line of work, highly-structured data such as entity relationships, molecules and programs is modelled using graphs. Graph neural networks are then successfully applied to directly learn from these graph representations. Here, we propose to extend this idea to weakly-structured data such as natural language. Using existing tools, we can annotate (accepting some noise) such data with additional relationships (e.g. co-references) to obtain a graph. However , the sequential aspect of the input data is still rich in meaning, and thus we propose a hybrid model in which a standard sequence encoder generates rich input for a graph neural network. In our experiments, the resulting combination outperforms baselines that use pure sequence or pure graph-based representations.Briefly, the contributions of our work are: 1. A framework that extends standard sequence encoder models with a graph component that leverages additional structure in sequence data. 2. Application of this extension to a range of existing sequence models and an extensive evaluation on three summarization tasks from the literature. 3. We release all used code and data at https://github.com/CoderPat/structured-neural-summarization. add a parameter to this dynamic parameter list BILSTM → LSTM: adds a new parameter to the specified parameter BILSTM+GNN → LSTM:creates a new instance of the dynamic type specified BILSTM+GNN → LSTM+POINTER: add a parameter to a list of parameters Figure 1 : An example from the dataset for the METHODDOC source code summarization task along with the outputs of a baseline and our models. In the METHODNAMING dataset, this method appears as a sample requiring to predict the name Add as a subtoken sequence of length 1. We presented a framework for extending sequence encoders with a graph component that can leverage rich additional structure. In an evaluation on three different summarization tasks, we have shown that this augmentation improves the performance of a range of different sequence models across all tasks. We are excited about this initial progress and look forward to deeper integration of mixed sequence-graph modeling in a wide range of tasks across both formal and natural languages. The key insight, which we believe to be widely applicable, is that inductive biases induced by explicit relationship modeling are a simple way to boost the practical performance of existing deep learning systems. We use the datasets and splits of BID4 provided by their website. Upon scanning all methods in the dataset, the size of the corpora can be seen in Table 4 . More information can be found at BID4 .
One simple trick to improve sequence models: Compose them with a graph model This paper presents a structural summarization model with a graph-based encoder extended from RNN. This work combines Graph Neural Networks with a sequential approach to abstractive summarization, effective across all datasets in comparison to external baselines.