* (144) Venot, O., Fray, N., Bénilan, Y., Gazeau, M.C., Hébrard, E., Larcher, G., Schwell, M., Dobrijevic, M., Selsis, F.: High-temperature measurements of VUV-absorption cross sections of CO2 and their application to exoplanets.  Astron. Astrophys. 551, A131 (2013)
  * (145) Venot, O., Hébrard, E., Agúndez, M., Dobrijevic, M., Selsis, F., Hersant, F., Iro, N., Bounaceur, R.: A chemical model for the atmosphere of hot Jupiters.  Astron. Astrophys. 546, A43 (2012)
  * (146) Venot, O., Parmentier, V., Blecic, J., Cubillos, P.E., Waldmann, I.P., Changeat, Q., Moses, J.I., Tremblin, P., Crouzet, N., Gao, P., Powell, D., Lagage, P.O., Dobbs-Dixon, I., Steinrueck, M.E., Kreidberg, L., Batalha, N., Bean, J.L., Stevenson, K.B., Casewell, S., Carone, L.: Global chemistry and thermal structure models for the Hot Jupiter WASP-43b and predictions for JWST.  Astrophys. J. 890, 176 (2020)
  * (147) Venturini, J., Helled, R.: The formation of mini-Neptunes.  Astrophys. J. 848, 95 (2017)
  * (148) Visscher, C., Lodders, K., Fegley Jr., B.: Atmospheric chemistry in giant planets, brown dwarfs, and low-mass dwarf stars. III. Iron, magnesium, and silicon.  Astrophys. J. 716, 1060–1075 (2010a)
  * (149) Visscher, C., Moses, J.I.: Quenching of carbon monoxide and methane in the atmospheres of cool brown dwarfs and hot Jupiters.  Astrophys. J. 738, 72 (2011)
  * (150) Visscher, C., Moses, J.I., Saslow, S.A.: The deep water abundance on Jupiter: New constraints from thermochemical kinetics and diffusion modeling.  Icarus 209, 602–615 (2010b)
  * (151) Wang, D., Miguel, Y., Lunine, J.: Modeling synthetic spectra for transiting extrasolar giant planets: Detectability of H2S and PH3 with the James Webb Space Telescope.  Astrophys. J. 850, 199 (2017)
  * (152) Wang, H., Wordsworth, R.: Extremely long convergence times in a 3D GCM simulation of the sub-Neptune Gliese 1214b.  Astrophys. J. 891, 7 (2020)
  * (153) Yung, Y.L., Allen, M., Pinto, J.P.: Photochemistry of the atmosphere of Titan: Comparison between model and observations.  Astrophys. J. Suppl. Ser.55, 465–506 (1984)
  * (154) Yung, Y.L., DeMore, W.B.: Photochemistry of Planetary Atmospheres.  Oxford University Press (1999)
  * (155) Zahnle, K., Marley, M.S., Morley, C.V., Moses, J.I.: Photolytic hazes in the atmosphere of 51 Eri b.  Astrophys. J. 824, 137 (2016)
  * (156) Zahnle, K.J., Marley, M.S.: Methane, carbon monoxide, and ammonia in brown dwarfs and self-luminous giant planets.  Astrophys. J. 797, 41 (2014)
  * (157) Zhang, X., Showman, A.P.: Effects of bulk composition on the atmospheric dynamics on close-in exoplanets.  Astrophys. J. 836, 73 (2017)
  * (158) Zhang, X., Showman, A.P.: Global-mean vertical tracer mixing in planetary atmospheres. I. Theory and fast-rotating planets.  Astrophys. J. 866, 1 (2018).  DOI 10.3847/1538-4357/aada85
  * (159) Zhang, X., Showman, A.P.: Global-mean vertical tracer mixing in planetary atmospheres. II. Tidally Locked Planets.  Astrophys. J. 866, 2 (2018).  DOI 10.3847/1538-4357/aada7c


# Text-to-Table: A New Way of Information Extraction

Xueqing Wu, 1 111Work done in Bytedance AI Lab Jiacheng Zhang, 2 Hang Li, 2

###### Abstract

We study a new problem setting of information extraction (IE), referred to as
text-to-table, which can be viewed as an inverse problem of the well-studied
table-to-text. In text-to-table, given a text, one creates a table or several
tables expressing the main content of the text, while the model is learned
from text-table pair data. The problem setting differs from those of the
existing methods for IE. First, the extraction can be carried out from long
texts to large tables with complex structures. Second, the extraction is
entirely data-driven, and there is no need to explicitly define the schemas.
As far as we know, there has been no previous work that studies the problem.
In this work, we formalize text-to-table as a sequence-to-sequence (seq2seq)
problem. We first employ a seq2seq model fine-tuned from a pre-trained
language model to perform the task. We also develop a new method within the
seq2seq approach, exploiting two additional techniques in table generation:
table constraint and table relation embeddings. We make use of four existing
table-to-text datasets in our experiments on text-to-table. Experimental
results show that the vanilla seq2seq model can outperform the baseline
methods of using relation extraction and named entity extraction. The results
also show that our method can further boost the performances of the vanilla
seq2seq model. We further discuss the main challenges of the proposed task.
The code and data will be made publicly available.

##  1 Introduction

Information extraction (IE) is a task that aims to extract information of
interest from text data and represent the extracted information in a
structured form. Traditional IE tasks include named entity recognition which
recognizes entities and their types (Huang, Xu, and Yu 2015; Ma and Hovy 2016;
Lample et al. 2016; Devlin et al. 2019), and relation extraction which
identifies the relationships between entities (Zheng et al. 2017; Zeng et al.
2018; Luan et al. 2019; Zhong and Chen 2020). Since the results of IE are
structured, they can be easily used by computer systems in different
applications such as text mining.

Figure 1: A example of text-to-table from the Rotowire dataset. The text is a
report of a basketball game, and the tables are the scores of the teams and
players.

In this work, we study IE in a new setting, referred to as text-to-table.
First, the system receives a training dataset containing text-table pairs.
Each text-table pair contains a text and a table (or tables) representing
information extracted from the text. The system learns a model for information
extraction. Next, the system employs the learned model to conduct information
extraction from a new text and outputs the result in a table (or tables).
Figure 1 gives an example of text-to-table, where the input (above) is a
report of a basketball game, and the output (below) is two tables summarizing
the scores of the teams and players from the input.

Our work is inspired by research on the so-called table-to-text (or data-to-
text) problem, which is the task of generating a description for a given
table. Table-to-text is useful in applications where the content of a table
needs to be described in natural language. Thus, text-to-table can be regarded
as an inverse problem of table-to-text. However, there are also differences.
Most notably, their applications are different. Text-to-table can be applied
to document summarization, text mining, etc.

Text-to-table is unique compared to the traditional IE approaches. First, it
is mainly designed to extract information on complex relations between items
from a long text (e.g., the whole document). Second, the schemas for
extraction are implicitly included in the training data, and there is no need
to explicitly define the schemas. As a result, one can easily build a system
to perform the task.

In this work, we formalize text-to-table as a sequence-to-sequence (seq2seq)
task. More specifically, we translate the text into a sequence representation
of a table (or tables), where the schema of the table is implicitly contained
in the representation. We also assume that the seq2seq model is built on top
of a pre-trained language model, such as BART (Lewis et al. 2019) and T5
(Raffel et al. 2020). Although the approach is a natural application of
existing technologies, as far as we know, there has been no previous study to
investigate to what extent the approach works. We also develop a new method
for text-to-table within the seq2seq approach with two additional techniques,
table constraint and table relation embeddings. Table constraint controls the
creation of rows in a table and table relation embeddings affect the
alignments between cells and their row headers and column headers. Both are to
make the generated table well-formulated.

The approach to IE based on seq2seq has already been proposed. Methods for
conducting individual tasks of relation extraction (Zeng et al. 2018; Nayak
and Ng 2020), named entity recognition (Chen and Moschitti 2018; Yan et al.
2021), and event extraction (Lu et al. 2021) have been developed. Methods for
jointly performing multiple tasks of named entity recognition, relation
extraction, and event extraction have also been devised (Paolini et al. 2021).
Most of the methods exploit suitable pre-trained models such as BERT. However,
all the existing methods rely on pre-defined schemas for extraction. Moreover,
their models are designed to extract information from short texts, rather than
long texts, and extract information with simple structures (such as an entity
and its type), rather than information with complicated structures (such as a
table).

We conduct extensive experiments on four existing table-to-text datasets.
Experimental results show that the vanilla seq2seq model fine-tuned from BART
(Lewis et al. 2019) can outperform the state-of-the-art IE models fine-tuned
from BERT (Devlin et al. 2019; Zhong and Chen 2020). Furthermore, results show
that our proposed approach to text-to-table with the two techniques can
further improve the extraction accuracies. We also summarize the challenging
issues with the seq2seq approach to text-to-table for future research.

We make the following contributions in this work:

  1. 1.

We propose the new task of text-to-table for IE. We derive four new datasets
for the task from existing datasets.

  2. 2.

We formalize the task as a seq2seq problem and propose a new method within the
seq2seq approach using the techniques of table constraint and table relation
embeddings.

  3. 3.

We conduct extensive experiments to verify the effectiveness of the proposed
approach.

##  2 Related Work

Information Extraction (IE) is a task of extracting information (structured
data) from a text (unstructured data). For example, named entity recognition
(NER) recognizes entities appearing in a text. Relation extraction (RE)
identifies the relationships between entities. Another example is event
extraction (EE), which discovers events occurring in a text.

Traditionally, researchers formalize the task as a language understanding
problem. The state-of-the-art methods for NER perform the task on the basis of
the pre-trained language model BERT (Devlin et al. 2019). The pipeline
approach to RE divides the problem into NER and relation classification, and
conducts the two sub-tasks in a sequential manner (Zhong and Chen 2020), while
the end-to-end approach jointly carries out the two sub-tasks (Zheng et al.
2017; Zeng et al. 2018; Luan et al. 2019). The state-of-the-art methods for EE
also employ BERT and usually jointly train the models with other tasks such as
NER and RE (Wadden et al. 2019; Zhang, Ji, and Sil 2019; Lin et al. 2020). All
the methods assume the use of pre-defined schemas (e.g., entity types for NER,
entity and relation types for RE, and event templates for EE). Besides, most
methods are designed for extraction from short texts. Therefore, existing
methods for IE cannot be directly applied to text-to-table.

IE is also conducted at document level, referred to as doc-level IE. For
example, some NER methods directly perform NER on a long document (Strubell et
al. 2017; Luo et al. 2018), and others encode each sentence in a document, use
attention to fuse document-level information, and perform NER on each sentence
(Hu et al. 2020; Xu, Wang, and He 2018). There are also RE methods that
predict the relationships between entities in a document (Yao et al. 2019; Nan
et al. 2020a). However, existing doc-level IE approaches usually do not
consider extraction of complex relations between many items.

Sequence-to-sequence (seq2seq) is the general problem of transforming one text
into another text (Sutskever, Vinyals, and Le 2014; Bahdanau, Cho, and Bengio
2014), which includes machine translation, text summarization, etc as special
cases. The use of the pre-trained language models of BART (Lewis et al. 2019)
and T5 (Raffel et al. 2020) can significantly boost the performances of
seq2seq, such as machine translation (Lewis et al. 2019; Raffel et al. 2020;
Liu et al. 2020) and text summarization (Lewis et al. 2019; Raffel et al.
2020; Huang et al. 2020).

Recently, some researchers also formalize the IE problems as seq2seq, that is,
transforming the input text into an internal representation. One advantage is
that one can employ a single model to extract multiple types of information.
Experimental results show that this approach works better than or equally well
as the traditional approach of language understanding, in RE (Zeng et al.
2018; Nayak and Ng 2020), NER (Chen and Moschitti 2018; Yan et al. 2021) and
EE (Lu et al. 2021).

Data-to-text aims to generate natural language descriptions from the input
structured data such as sport commentaries (Wiseman, Shieber, and Rush 2017).
The structured data is usually represented as tables (Wiseman, Shieber, and
Rush 2017; Thomson, Reiter, and Sripada 2020; Chen et al. 2020), sets of table
cells (Parikh et al. 2020; Bao et al. 2018), semantic representations
(Novikova, Dušek, and Rieser 2017), or sets of relation triples (Gardent et
al. 2017; Nan et al. 2020b). The task requires the model to select the salient
information from the data, organize it in a logical order, and generate an
accurate and fluent natural language description (Wiseman, Shieber, and Rush
2017). Data-to-text models usually adopt the encoder-decoder architecture. The
encoders are specifically designed to model the input data, such as multi-
layer perceptron (Puduppully, Dong, and Lapata 2019a, b), recurrent neural
network (Juraska et al. 2018; Liu et al. 2018; Shen et al. 2020), graph neural
network (Marcheggiani and Perez-Beltrachini 2018; Koncel-Kedziorski et al.
2019), or Transformer (Gong, Crego, and Senellart 2019).

##  3 Problem Formulation

Text-to-table takes a text as input and produces one or several tables to
summarize the content of the text, as shown in Figure 1. This can be
considered as an inverse problem of data-to-text, and each has its
applications.

The input is a text denoted as
$\mathbf{x}=x_{1},x_{2},\cdots,x_{|\mathbf{x}|}$. The output is one table or
multiple tables. For simplicity suppose that there is only one table denoted
as $T$. Further suppose that $T$ has $n_{r}$ rows and $n_{c}$ columns. There
are $n_{r}\times n_{c}$ cells in $T$, where the cell of row $i$ and column $j$
is a sequence of words
$\mathbf{t}_{i,j}=t_{i,j,1},t_{i,j,2},...,t_{i,j,|\mathbf{t}_{i,j}|}$.

There are three types of table: one that has both column headers and row
headers, one that has only column headers, and one that only has row headers.
For example, the player table in Figure 1 has both column headers (“Assists”,
“Points”, etc) and row headers (“Al Horford”, “Isaish Thomas”, etc). We let
$\mathbf{t}_{1,j},\ j=2,3,\cdots,n_{c}$ to denote the column headers, let
$\mathbf{t}_{i,1},\ i=2,3,\cdots,n_{r}$ to denote the row headers, and let
$\mathbf{t}_{1,j},\ i=2,3,\cdots,n_{r},j=2,3,\cdots,n_{c}$ to denote the non-
header cells of the table. For example, in the player table in Figure 1,
$\mathbf{t}_{1,2}=\text{Assists}$, $\mathbf{t}_{2,1}=\text{Al Horford}$, and
$\mathbf{t}_{2,2}=\text{5}$.

We consider using machine learning to perform text-to-table. In learning, a
number of text-table pairs are given as training data, and a model is trained
from the data. In inference, the learned model is utilized to generate a table
or tables given a new text.

Once the information of a text is extracted into tables via text-to-table, it
can be leveraged in many different applications such as document summarization
and text mining. For example, in Figure 1, one can quickly obtain the key
information of the text by simply looking at the tables summarized from the
text.

There are differences between text-to-table and traditional settings of
information extraction. As can be seen from the example in Figure 1,
extraction of information is performed from the entire document. The extracted
information (structured data) is in a complex form, specifically multiple
types of scores of teams and players in a basketball game. Furthermore, the
data-driven approach is taken, and the schemas of the tables do not need to be
explicitly defined.

Figure 2: The sequence representation of the player table in Figure 1. The
blue items are separation tokens $\langle\textmd{s}\rangle$ and the yellow
items are new-line tokens $\langle\textmd{n}\rangle$.

##  4 Our Method

We develop a method for text-to-table using the seq2seq approach and the two
techniques of table constraint and table relation embeddings.

###  4.1 Seq2Seq Framework

We formalize text-to-table as a sequence-to-sequence (seq2seq) problem
(Sutskever, Vinyals, and Le 2014; Bahdanau, Cho, and Bengio 2014).
Specifically, given an input text, we generate a sequence representing the
output table (or tables). We introduce two special tokens, a separation token
denoted as “$\langle\textmd{s}\rangle$” and a new-line token denoted as
“$\langle\textmd{n}\rangle$”. For a table $\mathbf{t}$, we represent each row
$\mathbf{t}_{i}$ with a sequence of cells delimited by separation tokens:

$\displaystyle\mathbf{t}_{i}=\text{$\langle\textmd{s}\rangle$}{},\mathbf{t}_{i,1},\text{$\langle\textmd{s}\rangle$}{},\cdots,\text{$\langle\textmd{s}\rangle$}{},\mathbf{t}_{i,n_{c}},\text{$\langle\textmd{s}\rangle$}{}.$
(1)

We represent the entire table with a sequence of rows delimited by new-line
tokens:

$\displaystyle\mathbf{t}$ $\displaystyle=$
$\displaystyle\text{$\langle\textmd{s}\rangle$}{},\mathbf{t}_{1,1},\text{$\langle\textmd{s}\rangle$}{},\cdots,\text{$\langle\textmd{s}\rangle$}{},\mathbf{t}_{1,n_{c}},\text{$\langle\textmd{s}\rangle$}{},\text{$\langle\textmd{n}\rangle$}{},$
(2)
$\displaystyle\text{$\langle\textmd{s}\rangle$}{},\mathbf{t}_{2,1},\text{$\langle\textmd{s}\rangle$}{},\cdots,\text{$\langle\textmd{s}\rangle$}{},\mathbf{t}_{2,n_{c}},\text{$\langle\textmd{s}\rangle$}{},\text{$\langle\textmd{n}\rangle$}{},$
$\displaystyle\cdots\cdots$
$\displaystyle\text{$\langle\textmd{s}\rangle$}{},\mathbf{t}_{n_{r},1},\text{$\langle\textmd{s}\rangle$}{},\cdots,\text{$\langle\textmd{s}\rangle$}{},\mathbf{t}_{n_{r},n_{c}},\text{$\langle\textmd{s}\rangle$}{}\quad$

Figure 2 shows the sequence of the player table in Figure 1. When there are
multiple tables, we create a sequence of tables using the captions of the
tables as delimiters.

Let $\mathbf{x}=x_{1},\cdots,x_{|\mathbf{x}|}$ and
$\mathbf{y}=y_{1},\cdots,y_{|\mathbf{y}|}$ denote the input and output
sequences respectively. In inference, the model generates the output sequence
based on the input sequence. We conduct decoding in an auto-regressive way,
which generates one token at each step based on the tokens it has generated so
far. Formally, the model calculates the conditional probability:

$\displaystyle
p(\mathbf{y}|\mathbf{x})=\prod_{t=1}^{|\mathbf{y}|}p(y_{i}|\mathbf{x},\mathbf{y}_{<t})$
(3)

We use decoding algorithms such as beam search or greedy search to find an
output sequence that approximately maximizes the conditional probability.

In training, we learn the model based on the text-table pairs
$\\{(\mathbf{x}_{1},\mathbf{y}_{1}),(\mathbf{x}_{2},\mathbf{y}_{2}),\cdots,(\mathbf{x}_{n},\mathbf{y}_{n})\\}$.
The objective of learning is to minimize the cross-entropy loss

$\displaystyle\arg\min_{\mathbf{\theta}}L=\arg\min_{\mathbf{\theta}}-\sum_{i=1}^{n}\log
p_{\mathbf{\theta}}(\mathbf{y}_{i}|\mathbf{x}_{i})$ (4)

where $\mathbf{\theta}$ denotes the parameter of the model.

In our work, we adopt Transformer as the model (Vaswani et al. 2017), which is
the state-of-the-art method for seq2seq. We build the model on top of the pre-
trained language model BART (Lewis et al. 2019) by fine-tuning.

We refer to the method described above as “vanilla seq2seq”. There is no
guarantee, however, that the output sequence of vanilla seq2seq represents a
well-formulated table. We add a post-processing method to ensure that the
output sequence is a table. The post-processing method takes the first row
generated as well-defined, deletes extra cells at the end of the other rows
and inserts empty cells at the end of the other rows.

###  4.2 Techniques

We develop two techniques to improve table generation, called table constraint
and table relation embeddings. We use “our method” to denote the seq2seq
approach with these two techniques.

#### Table Constraint

Our method exploits a constraint in the decoding process to ensure that the
output sequence represents a well-formulated table. Specifically, our method
calculates the number of cells in the first row it generates, and then forces
the following rows to contain the same number of cells. The algorithm of the
decoder is shown in Algorithm 1.

1

2

Input: $\mathbf{x}=[x_{1},x_{2},\cdots,x_{|\mathbf{x}|}]$

Output: $\mathbf{y}=[y_{1},y_{2},\cdots,y_{|\mathbf{y}|}]$

3

4$\mathbf{y}\leftarrow[]$

5

6

7repeat

/* generates the first row: only allows generation of
$\langle\textmd{n}\rangle$ or $\langle\textmd{eos}\rangle$ after
$\langle\textmd{s}\rangle$ */

8 $p(\cdot)\leftarrow$ seq2seq($\mathbf{x}$, $\mathbf{y}$)

9 if  _$\mathbf{y}_{|\mathbf{y}|}\neq\text{$\langle\textmd{s}\rangle$}{}$ _
then

10 $p\left(\text{$\langle\textmd{n}\rangle$}{}\right)\leftarrow 0,\
p\left(\text{$\langle\textmd{eos}\rangle$}{}\right)\leftarrow 0$

11

12

13 $\mathbf{y}$.append$\left(\textnormal{\text{decode}}\left(p\right)\right)$

14

15until  _$\mathbf{y}_{|\mathbf{y}|}=\text{$\langle\textmd{n}\rangle$}{}$ or
$\mathbf{y}_{|\mathbf{y}|}=\text{$\langle\textmd{eos}\rangle$}{}$ _

16if  _$\mathbf{y}_{|\mathbf{y}|}=\text{$\langle\textmd{eos}\rangle$}{}$ _
then

17 return $\mathbf{y}$

18

19 $n_{c}\leftarrow$ number of cells of the first row

20

21

22repeat

/* generates the next rows: each row contains exactly $n_{c}$ cells */

23

24 repeat

/* generates a row */

25 $p(\cdot)\leftarrow$ seq2seq($\mathbf{x}$, $\mathbf{y}$)

26 if  _current row has $n_{c}$ columns _ then

27 $p(t)\leftarrow 0,\ \forall
t\neq\text{$\langle\textmd{eos}\rangle$}{}\textmd{ and
}t\neq\text{$\langle\textmd{n}\rangle$}{}$

28

29 else

30 $p\left(\text{$\langle\textmd{n}\rangle$}{}\right)\leftarrow 0,\
p\left(\text{$\langle\textmd{eos}\rangle$}{}\right)\leftarrow 0$

31

32 $\mathbf{y}$.append$\left(\textnormal{\text{decode}}\left(p\right)\right)$

33

34 until  _$\mathbf{y}_{|\mathbf{y}|}=\text{$\langle\textmd{n}\rangle$}{}$ or
$\mathbf{y}_{|\mathbf{y}|}=\text{$\langle\textmd{eos}\rangle$}{}$ _

35 if  _$\mathbf{y}_{|\mathbf{y}|}=\text{$\langle\textmd{eos}\rangle$}{}$ _
then

36 return $\mathbf{y}$

37

38

39

Algorithm 1 Decoding using table constraint. $\langle\textmd{eos}\rangle$,
$\langle\textmd{s}\rangle$, and $\langle\textmd{n}\rangle$ denote the end of
sentence, separation token, and new-line token respectively. Seq2seq denotes
the seq2seq model. Decode denotes the decoding algorithm such as beam search
and greedy search.

#### Table Relation Embeddings

Our method also incorporates table relation embeddings including row relation
embeddings and column relation embeddings into the self-attention of the
Transformer decoder. Given a token in a non-header cell, the row relation
embeddings $\tau_{r}^{K}$ and $\tau_{r}^{V}$ indicate which row header the
token is aligned to, and the column relation embeddings $\tau_{c}^{K}$ and
$\tau_{c}^{V}$ indicate which column header the token is aligned to.

Let us consider the self-attention function in one block of Transformer
decoder: at each position, self-attention only attends to the previous
positions. For simplicity, let us only consider one head in the self-
attention. At the $t$-th position, the input of self-attention is the sequence
of representations $z=(z_{1},\cdots,z_{t})$ and the output is the sequence of
representations $h=(h_{1},\cdots,h_{t})$, where $z_{i}\in\mathbb{R}^{d}$ and
$h_{i}\in\mathbb{R}^{d}$ are the representations at the $i$-th position
$(i=1,\cdots,t)$.

In a conventional Transformer decoder, self-attention is defined as follows,

$\displaystyle h_{i}=\left(\sum_{j=1}^{i}\alpha_{ij}(z_{j}W^{V})\right)W^{O},$
(5) $\displaystyle\alpha_{ij}=\frac{\exp e_{ij}}{\sum_{j=1}^{i}\exp e_{ij}},\
e_{ij}=\frac{(z_{i}W^{Q})(z_{j}W^{K})^{\text{T}}}{\sqrt{d_{k}}},$ (6)
$\displaystyle i=1,\cdots,t,\ j=1,\cdots,i$

where $W^{Q},W^{K},W^{V}\in\mathbb{R}^{d\times d_{k}}$ are the query, key, and
value weight matrices respectively, and $W^{O}\in\mathbb{R}^{d_{k}\times d}$
is the output weight matrix.

In our method, self-attention is defined as follows:

$\displaystyle
h_{i}=\left(\sum_{j=1}^{i}\alpha_{ij}(z_{j}W^{V}+r_{ij}^{V})\right)W^{O},$ (7)
$\displaystyle\alpha_{ij}=\frac{\exp e_{ij}}{\sum_{j=1}^{i}\exp e_{ij}},\
e_{ij}=\frac{(z_{i}W^{Q})(z_{j}W^{K}+r_{ij}^{K})^{\text{T}}}{\sqrt{d_{k}}},$
(8) $\displaystyle i=1,\cdots,t,\ j=1,\cdots,i$

where $r_{ij}^{K}$ and $r_{ij}^{V}$ are relation vectors representing the
relationship between the $i$-th position and the $j$-th position.

Dataset | Train | Valid | Test | # of tokens | # of rows | # of columns
---|---|---|---|---|---|---
E2E | 42.1k | 4.7k | 4.7k | 24.90 | 4.58 | 2.00
WikiTableText | 10.0k | 1.3k | 2.0k | 19.59 | 4.26 | 2.00
WikiBio | 582.7k | 72.8k | 72.8k | 122.30 | 4.20 | 2.00
Table 1: Statistics of E2E, WikiTableText, and WikiBio datasets, including
number of instances in training, validation, and test sets, number of BPE
tokens per instance and number of rows per instance. Figure 3: Construction of
relation vectors. Red and yellow arrows represent alignments with column
headers and alignments with row headers respectively. The relation vectors
regarding tokens “11” and one $\langle\textmd{s}\rangle$ are illustrated.

The relation vectors $r_{ij}^{K}$ and $r_{ij}^{V}$ are defined as follows. For
the token at the $i$-th position, if the token at the $j$-th position is a
part of its row header, then $r_{ij}^{K}$ and $r_{ij}^{V}$ are set to the row
relation embeddings $\tau_{r}^{K}$ and $\tau_{r}^{V}$. Similarly, for the
token at the $i$-th position, if the token at the $j$-th position is a part of
its column header, then $r_{ij}^{K}$ and $r_{ij}^{V}$ are set to the column
relation embeddings $\tau_{c}^{K}$ and $\tau_{c}^{V}$. Otherwise, $r_{ij}^{K}$
and $r_{ij}^{V}$ are set to $0$. To identify the row header or the column
header of a token, we parse the sequence generated so far to create a partial
table using the new-line tokens and separation tokens in the sequence. Figure
3 illustrates how relation vectors are constructed.

##  5 Experiments

###  5.1 Datasets

We make use of four existing datasets which are traditionally utilized for
data-to-text: Rotowire (Wiseman, Shieber, and Rush 2017), E2E (Novikova,
Dušek, and Rieser 2017), WikiTableText (Bao et al. 2018), and WikiBio (Lebret,
Grangier, and Auli 2016). In each dataset, we filter out the content in the
tables that does not appear in the texts. We plan to make the processed
datasets publicly available for future research. Table 2 gives the statistics
of the Rotowire dataset and Table 1 gives the statistics of the other three
datasets.

Rotowire is from the sports domain. Each instance is composed of a text and
two tables, where the text is a report of a basketball game and the two tables
represent the scores of teams and players respectively (cf., Figure 1). Each
table has column headers describing the types of scores, and row headers
describing the names of teams or players. The texts are long and may contain
irrelevant information such as the performance of players in other games. Thus
this is a challenging dataset for text-to-table.

Train | Valid | Test | # of tokens
---|---|---|---
3.4k | 727 | 728 | 351.05

| # of rows | # of columns | # of cells
---|---|---|---
Team | 2.71 | 4.84 | 6.56 (85.40%)
Player | 7.26 | 8.75 | 22.63 (43.93%)

Table 2: Statistics of Rotowire dataset. The first table shows sizes of
training, validation, and test sets, number of BPE tokens per instance. The
second table shows number of rows, number of columns, and number and ratio of
non-empty cells.

E2E is from the restaurant domain. Each instance is a pair of short text and
automatically constructed table, where the text is a description of a
restaurant, and the table has two columns with column headers summarizing the
characteristics of the restaurant. The tables are automatically constructed,
where the texts in the tables are from a limited set and thus are lack of
diversity.

WikiTableText is an open-domain dataset. Each instance includes a text and a
table, where the text is a description and the table has a row and two columns
with column headers, collected from a Wikipedia infobox. The texts are short
and contain information similar to that in the tables.

WikiBio is extracted from the Wikipedia biography pages. Each instance
consists of a text and a table, where the text is the introduction of
Wikipedia page222The original dataset only uses the first sentence of the
introduction. We use the entire introduction. and the table is from the
infobox of Wikipedia page and has two columns with column headers. The input
texts are usually long and contain more information than the tables.

###  5.2 Procedure

Methods: We conduct experiments with vanilla seq2seq and our method, as well
as baselines.

We know of no existing method that can be directly employed in text-to-table.
For each dataset, we first define the schemas based on the training data, then
use an existing method of relation extraction (RE) or named entity extraction
(NER) to extract information, and finally create tables based on the schemas
and extracted information. We take it as the baseline for the dataset. No
baseline can be applied to all four datasets. For RE, we use PURE, a state-of-
the-art method (Zhong and Chen 2020). For NER, we use BERT model (Devlin et
al. 2019).

l—cccc—cccc 1-4Team 1-4Player
Pre. Rec. F1 Err. Pre. Rec. F1 Err.
Sent-level RE 81.05 75.29 77.17 0.00 86.10 76.00 79.59 0.00
Doc-level RE 78.57 74.74 75.66 0.00 86.19 77.88 80.76 0.00
Vanilla seq2seq 84.05 83.59 82.97 0.49 84.56 81.32 81.96 7.40
Our method 84.40 84.01 83.36 0.00 84.77 82.20 82.53 0.00

Table 3: Experimental results of our method, vanilla seq2seq, and the
baselines of doc-level RE and sent-level RE, on Rotowire.

l—cccc—cccc—cccc 1-4E2E 1-4WikiTableText 1-4WikiBio
Pre. Rec. F1 Err. Pre. Rec. F1 Err. Pre. Rec. F1 Err.
NER 99.39 84.99 90.80 0.00 60.63 47.02 52.23 0.00 73.64 49.87 56.51 0.00
Vanilla seq2seq 97.95 97.87 97.87 0.00 59.84 59.30 59.26 0.41 70.78 70.44
68.98 0.00
Our method 97.97 97.89 97.88 0.00 59.74 59.17 59.14 0.00 70.93 70.38 69.02
0.00

Table 4: Experimental results of our method, vanilla seq2seq, and the baseline
of NER, on WikiTableText and WikiBio.

Training: For vanilla seq2seq and our method, we fine-tune the seq2seq models
from BART-base, which has 12 layers, 768 hidden dimensions, 16 heads, and 139M
parameters. For RE and NER, we fine-tune the models from BERT-base-uncased,
which has 12 layers, 768 hidden dimensions, 12 heads, and 110M parameters.

All models are trained with the Adam optimizer until convergence, and the
hyper-parameters are tuned on the development sets. Appendix A shows the
hyper-parameters. For the small datasets of Rotowire and WikiTableText, we run
experiments five times with different random seeds and take average of results
to reduce variance.

Evaluation: We evaluate the performance of a method based on the number of
correct non-empty cells in the tables (i.e., we ignore empty cells). To judge
whether a cell is correctly generated in the table, we use not only its
content but also its row header and column header to ensure that the cell is
on the right row and right column. Exact match is used to compare the content
of the generated cell and the ground truth. We adopt precision, recall, and F1
score as evaluation measures. We calculate the measures on each generated
table and then take the average on all tables. We also evaluate the percentage
of output sequences that cannot represent well-formulated tables, referred to
as error rate.

###  5.3 Results on Rotowire

Table 5.2 shows the results on the Rotowire dataset. One can see that in terms
of F1 score, our method performs the best followed by vanilla seq2seq, and
both outperform the baselines of doc-level RE and sent-level RE. The RE
baselines perform quite well, but they heavily rely on rules and cannot beat
the seq2seq approach. Among them the doc-level RE performs better than sent-
level RE, because some information in Rotowire can only be extracted when
cross-sentence context is provided.

We implement two baselines of RE, namely doc-level RE and sent-level RE. We
take team names, player names, and numbers of scores as entities and take
types of scores as relations. Sent-level RE predicts the relations between
entities within each sentence. Doc-level RE predicts the relations between
entities within a window (the window size is $12$ entities) and uses the
approximation model proposed by Zhong and Chen (2020) to speed up inference.

###  5.4 Results on E2E, WikiTableText and WikiBio

Table 5.2 shows the results of our method, vanilla seq2seq, and the baseline
of NER on E2E, WikiTableText, and WikiBio. Again, the seq2seq approach
outperforms the baseline. The NER baseline has slightly higher precision, but
the seq2seq approach has significantly higher recall and F1. Our method and
vanilla seq2seq are comparable, because the table structures in the three
datasets are very simple (there are only two columns in the tables), and the
use of the two techniques does not further improve the performances. The NER
baseline has high precision but low recall, mainly because NER can only make
the right decision when it is clear.

We implement the baseline of NER in the following way. We view the non-head
cells in the tables as entities and their row headers as entity types. In
training, we match the non-head cells into the texts and take them as
“entities” in the texts. Only a proportion of the non-header cells can be
matched into the texts ($85\%$ for E2E, $74\%$ for WikiTableText, and $69\%$
for WikiBio).

###  5.5 Additional Study

We carry out ablation study on our method, specifically we exclude pre-trained
language model, table constraint (TC) and table relation embeddings (TRE) from
our method. Note that our method without TC and TRE is equivalent to vanilla
seq2seq. Table 5 gives the results on the four datasets.

Pre | TC | TRE | Rotowire/Team | Rotowire/Player | E2E | WikiTableText | WikiBio
---|---|---|---|---|---|---|---
✗ | ✗ | ✗ | 28.05 | 7.75 | 94.45 | 46.37 | 67.51
✗ | ✓ | ✓ | 30.61 | 10.67 | 95.53 | 47.13 | 67.43
✓ | ✗ | ✗ | 82.97 | 81.96 | 97.87 | 59.26 | 68.98
✓ | ✓ | ✗ | 83.09 | 82.24 | 97.88 | 59.29 | 68.98
✓ | ✗ | ✓ | 83.30 | 82.50 | 97.87 | 59.12 | 69.02
✓ | ✓ | ✓ | 83.36 | 82.53 | 97.88 | 59.14 | 69.02
Table 5: Experimental results of ablation study on our method by excluding pre-trained language model (Pre), table constraint (TC) and table relation embeddings (TRE). Method | Rotowire/Team | Rotowire/Player | E2E | WikiTableText | WikiBio
---|---|---|---|---|---
Vanilla seq2seq (BART base) | 82.97 | 81.96 | 97.87 | 59.26 | 68.98
Our method (BART base) | 83.36 | 82.53 | 97.88 | 59.14 | 69.02
Vanilla seq2seq (BART large) | 86.31 | 86.59 | 97.94 | 62.71 | 69.66
Our method (BART large) | 86.31 | 86.83 | 97.90 | 62.41 | 69.71
Table 6: Experimental results of our method and vanilla seq2seq with base and
large BART models on all four datasets. Figure 4: A bad case generated by
vanilla seq2seq. The assists, points and total rebounds of Rajon Rondo are
$18$, $7$ and $8$ respetively. The model generates one less column between
“Assists” and “Personal fouls”.

It can be seen that the use of both TC and TRE can significantly improve the
performance on Rotowire, which indicates that our method is particularly
effective when the tables are large with many rows and columns. There are not
significant improvements on E2E, WikiTableText, and WikiTableText, apparently
because formulation of tables is easy for the three datasets. Therefore, we
conclude that the two techniques of TC and TRE are helpful when the task is
difficult.

The use of pre-trained language model can boost the performances on all
datasets, especially on Rotowire and WikiTableText. This indicates that pre-
trained language model is particularly helpful when the task is difficult and
the size of training data is small.

We observe that vanilla seq2seq makes more formatting errors than our method,
especially on player tables in Rotowire that have a large number of columns.
It indicates that for vanilla seq2seq, it is difficult to keep track of the
columns in each row and make alignments with the column headers. In contrast,
the two techniques of our method can help effectively cope with the problem.
Figure 4 shows a bad case of vanilla seq2seq, where the model correctly infers
the column of “assists” but fails to infer the columns of “personal fouls”,
“points”, and “total rebounds” for the row of “Rajon Rondo”. In contrast, our
method can successfully handle the case, because TC can eliminate the
incorrectly formatted output, and TRE can make correct alignments with the
column headers.

We also investigate the effect of the scale of pre-trained language model
BART. We use both BART-base and BART-large and conduct fine-tuning on top of
them for vanilla seq2seq and our method. Table 6 gives the results on the four
datasets. The results show that the use of BART-large can further boost the
performances on all four datasets, indicating that it is better to use larger
pre-trained models when computation cost is not an issue.

###  5.6 Discussions

We analyze the experimental results on the four datasets and identify five
challenging issues.

(1) Text Diversity: Extraction of the same content from different expressions
is one challenge. For example, the use of synonyms is very common in Rotowire.
The team of “Knicks” is often referred to as “New York”, its home city.
Identification of the same entities from different expressions is needed in
the task.

(2) Text Redundancy: There are cases such as those in WikiBio, in which the
texts contain much redundant information. This poses a challenge to the text-
to-table model to have a strong ability in summarization. It seems that the
seq2seq approach works well to some extent but further improvement is
undoubtedly necessary.

(3) Large Table: The tables in Rotowire have large numbers of columns, and the
extraction from them is challenging even for our method of using TC and TRE.

(4) Background Knowledge: WikiTableText and WikiBio are from open domain.
Thus, performing text-to-table on such kind of datasets require the use of
much background knowledge. The use of more powerful pre-trained language
models can be further explored in the future.

(5) Reasoning: Sometimes the information is not explicitly presented in the
text, and reasoning is required to conduct correct extraction. For example, an
article in Rotowire reports a game between the two teams “Nets” and “Wizards”.
From the sentence: “The Nets seized control of this game from the very start,
opening up a 31 - 14 lead after the first quarter”, humans can infer that the
point of “Wizards” is $14$, which is still difficult for machines.

##  6 Conclusion

We propose employing text-to-table as a new way of information extraction
(IE), which extracts information of interest from the input text and
summarizes the extracted information in tables. The advantage of the approach
is that one can easily conduct information extraction from either short texts
or long texts to create simple tables or complex tables without explicitly
defining the schemas. Text-to-table can be viewed as an inverse problem of
table-to-text. We formalize text-to-table as a sequence-to-sequence problem on
top of a pre-trained model. We further propose an improved method using a
seq2seq model and table constraint and table relation embeddings techniques.
We conduct experiments on four datasets derived from existing table-to-text
datasets. The experimental results demonstrate that our proposed approach
outperforms existing methods using conventional IE techniques. We further
analyze the challenges of text-to-table for future study. The issues include
diversity of text, redundancy of text, large-table, background knowledge, and
reasoning.

## References

  * Bahdanau, Cho, and Bengio (2014) Bahdanau, D.; Cho, K.; and Bengio, Y. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Bao et al. (2018) Bao, J.; Tang, D.; Duan, N.; Yan, Z.; Lv, Y.; Zhou, M.; and Zhao, T. 2018.  Table-to-text: Describing table region with natural language.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Chen and Moschitti (2018) Chen, L.; and Moschitti, A. 2018.  Learning to progressively recognize new named entities with sequence to sequence models.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , 2181–2191.
  * Chen et al. (2020) Chen, W.; Chen, J.; Su, Y.; Chen, Z.; and Wang, W. Y. 2020.  Logical Natural Language Generation from Open-Domain Tables.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7929–7942.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186.
  * Gardent et al. (2017) Gardent, C.; Shimorina, A.; Narayan, S.; and Perez-Beltrachini, L. 2017.  Creating training corpora for nlg micro-planning.  In _55th annual meeting of the Association for Computational Linguistics (ACL)_.
  * Gong, Crego, and Senellart (2019) Gong, L.; Crego, J. M.; and Senellart, J. 2019.  Enhanced transformer model for data-to-text generation.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , 148–156.
  * Hu et al. (2020) Hu, A.; Dou, Z.; Nie, J.-Y.; and Wen, J.-R. 2020.  Leveraging multi-token entities in document-level named entity recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 7961–7968.
  * Huang et al. (2020) Huang, D.; Cui, L.; Yang, S.; Bao, G.; Wang, K.; Xie, J.; and Zhang, Y. 2020.  What Have We Achieved on Text Summarization?  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 446–469.
  * Huang, Xu, and Yu (2015) Huang, Z.; Xu, W.; and Yu, K. 2015.  Bidirectional LSTM-CRF models for sequence tagging.  _arXiv preprint arXiv:1508.01991_.
  * Juraska et al. (2018) Juraska, J.; Karagiannis, P.; Bowden, K.; and Walker, M. 2018.  A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 152–162.
  * Koncel-Kedziorski et al. (2019) Koncel-Kedziorski, R.; Bekal, D.; Luan, Y.; Lapata, M.; and Hajishirzi, H. 2019\.  Text Generation from Knowledge Graphs with Graph Transformers.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2284–2293.
  * Lample et al. (2016) Lample, G.; Ballesteros, M.; Subramanian, S.; Kawakami, K.; and Dyer, C. 2016.  Neural Architectures for Named Entity Recognition.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 260–270.
  * Lebret, Grangier, and Auli (2016) Lebret, R.; Grangier, D.; and Auli, M. 2016.  Neural Text Generation from Structured Data with Application to the Biography Domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , 1203–1213.
  * Lewis et al. (2019) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lin et al. (2020) Lin, Y.; Ji, H.; Huang, F.; and Wu, L. 2020.  A joint neural model for information extraction with global features.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7999–8009.
  * Liu et al. (2018) Liu, T.; Wang, K.; Sha, L.; Chang, B.; and Sui, Z. 2018.  Table-to-text generation by structure-aware seq2seq learning.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Liu et al. (2020) Liu, Y.; Gu, J.; Goyal, N.; Li, X.; Edunov, S.; Ghazvininejad, M.; Lewis, M.; and Zettlemoyer, L. 2020.  Multilingual Denoising Pre-training for Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 8: 726–742.
  * Lu et al. (2021) Lu, Y.; Lin, H.; Xu, J.; Han, X.; Tang, J.; Li, A.; Sun, L.; Liao, M.; and Chen, S. 2021.  Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction.  _arXiv preprint arXiv:2106.09232_.
  * Luan et al. (2019) Luan, Y.; Wadden, D.; He, L.; Shah, A.; Ostendorf, M.; and Hajishirzi, H. 2019.  A general framework for information extraction using dynamic span graphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 3036–3046.
  * Luo et al. (2018) Luo, L.; Yang, Z.; Yang, P.; Zhang, Y.; Wang, L.; Lin, H.; and Wang, J. 2018.  An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition.  _Bioinformatics_ , 34(8): 1381–1388.
  * Ma and Hovy (2016) Ma, X.; and Hovy, E. 2016.  End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 1064–1074.
  * Marcheggiani and Perez-Beltrachini (2018) Marcheggiani, D.; and Perez-Beltrachini, L. 2018.  Deep Graph Convolutional Encoders for Structured Data to Text Generation.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , 1–9.
  * Nan et al. (2020a) Nan, G.; Guo, Z.; Sekulic, I.; and Lu, W. 2020a.  Reasoning with Latent Structure Refinement for Document-Level Relation Extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 1546–1557.
  * Nan et al. (2020b) Nan, L.; Radev, D.; Zhang, R.; Rau, A.; Sivaprasad, A.; Hsieh, C.; Tang, X.; Vyas, A.; Verma, N.; Krishna, P.; et al. 2020b.  Dart: Open-domain structured data record to text generation.  _arXiv preprint arXiv:2007.02871_.
  * Nayak and Ng (2020) Nayak, T.; and Ng, H. T. 2020.  Effective modeling of encoder-decoder architecture for joint entity and relation extraction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 8528–8535.
  * Novikova, Dušek, and Rieser (2017) Novikova, J.; Dušek, O.; and Rieser, V. 2017.  The E2E Dataset: New Challenges For End-to-End Generation.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , 201–206.
  * Paolini et al. (2021) Paolini, G.; Athiwaratkun, B.; Krone, J.; Ma, J.; Achille, A.; Anubhai, R.; Santos, C. N. d.; Xiang, B.; and Soatto, S. 2021.  Structured prediction as translation between augmented natural languages.  _arXiv preprint arXiv:2101.05779_.
  * Parikh et al. (2020) Parikh, A.; Wang, X.; Gehrmann, S.; Faruqui, M.; Dhingra, B.; Yang, D.; and Das, D. 2020.  ToTTo: A Controlled Table-To-Text Generation Dataset.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 1173–1186.
  * Puduppully, Dong, and Lapata (2019a) Puduppully, R.; Dong, L.; and Lapata, M. 2019a.  Data-to-text generation with content selection and planning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, 6908–6915.
  * Puduppully, Dong, and Lapata (2019b) Puduppully, R.; Dong, L.; and Lapata, M. 2019b.  Data-to-text Generation with Entity Modeling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2023–2035.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21: 1–67.
  * Shen et al. (2020) Shen, X.; Chang, E.; Su, H.; Niu, C.; and Klakow, D. 2020.  Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7155–7165.
  * Strubell et al. (2017) Strubell, E.; Verga, P.; Belanger, D.; and McCallum, A. 2017.  Fast and Accurate Entity Recognition with Iterated Dilated Convolutions.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2670–2680.
  * Sutskever, Vinyals, and Le (2014) Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014.  Sequence to Sequence Learning with Neural Networks.  _Advances in Neural Information Processing Systems_ , 27: 3104–3112.
  * Thomson, Reiter, and Sripada (2020) Thomson, C.; Reiter, E.; and Sripada, S. 2020.  SportSett: Basketball-A robust and maintainable data-set for Natural Language Generation.  In _Proceedings of the Workshop on Intelligent Information Processing and Natural Language Generation_ , 32–40.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is All you Need.  _Advances in Neural Information Processing Systems_ , 30: 5998–6008.
  * Wadden et al. (2019) Wadden, D.; Wennberg, U.; Luan, Y.; and Hajishirzi, H. 2019.  Entity, Relation, and Event Extraction with Contextualized Span Representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 5784–5789.
  * Wiseman, Shieber, and Rush (2017) Wiseman, S.; Shieber, S. M.; and Rush, A. M. 2017.  Challenges in Data-to-Document Generation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2253–2263.
  * Xu, Wang, and He (2018) Xu, G.; Wang, C.; and He, X. 2018.  Improving clinical named entity recognition with global neural attention.  In _Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data_ , 264–279. Springer.
  * Yan et al. (2021) Yan, H.; Gui, T.; Dai, J.; Guo, Q.; Zhang, Z.; and Qiu, X. 2021.  A Unified Generative Framework for Various NER Subtasks.  _arXiv preprint arXiv:2106.01223_.
  * Yao et al. (2019) Yao, Y.; Ye, D.; Li, P.; Han, X.; Lin, Y.; Liu, Z.; Liu, Z.; Huang, L.; Zhou, J.; and Sun, M. 2019.  DocRED: A Large-Scale Document-Level Relation Extraction Dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 764–777.
  * Zeng et al. (2018) Zeng, X.; Zeng, D.; He, S.; Liu, K.; and Zhao, J. 2018.  Extracting relational facts by an end-to-end neural model with copy mechanism.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 506–514.
  * Zhang, Ji, and Sil (2019) Zhang, T.; Ji, H.; and Sil, A. 2019.  Joint entity and event extraction with generative adversarial imitation learning.  _Data Intelligence_ , 1(2): 99–120.
  * Zheng et al. (2017) Zheng, S.; Wang, F.; Bao, H.; Hao, Y.; Zhou, P.; and Xu, B. 2017.  Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 1227–1236.
  * Zhong and Chen (2020) Zhong, Z.; and Chen, D. 2020.  A Frustratingly Easy Approach for Joint Entity and Relation Extraction.  _arXiv preprint arXiv:2010.12812_.