# Cross-Task Instance Representation Interactions and Label Dependencies
for Joint Information Extraction with Graph Convolutional Networks

Minh Van Nguyen, Viet Dac Lai and Thien Huu Nguyen
Department of Computer and Information Science
University of Oregon, Eugene, Oregon, USA
<EMAIL_ADDRESS>
###### Abstract

Existing works on information extraction (IE) have mainly solved the four main
tasks separately (entity mention recognition, relation extraction, event
trigger detection, and argument extraction), thus failing to benefit from
inter-dependencies between tasks. This paper presents a novel deep learning
model to simultaneously solve the four tasks of IE in a single model (called
FourIE). Compared to few prior work on jointly performing four IE tasks,
FourIE features two novel contributions to capture inter-dependencies between
tasks. First, at the representation level, we introduce an interaction graph
between instances of the four tasks that is used to enrich the prediction
representation for one instance with those from related instances of other
tasks. Second, at the label level, we propose a dependency graph for the
information types in the four IE tasks that captures the connections between
the types expressed in an input sentence. A new regularization mechanism is
introduced to enforce the consistency between the golden and predicted type
dependency graphs to improve representation learning. We show that the
proposed model achieves the state-of-the-art performance for joint IE on both
monolingual and multilingual learning settings with three different languages.

##  1 Introduction

Information Extraction (IE) is an important and challenging task in Natural
Language Processing (NLP) that aims to extract structured information from
unstructured texts. Following the terminology for IE in the popular ACE 2005
program Walker et al. (2006), we focus on four major IE tasks in this work:
entity mention extraction (EME), relation extraction (RE), event trigger
detection (ETD), and event argument extraction (EAE).

Given an input sentence, a vast majority of prior work has solved the four
tasks in IE independently at both instance and task levels (called independent
prediction models). First, at the instance level, each IE task often requires
predictions/classifications for multiple instances in a single input sentence.
For instance, in RE, one often needs to predict relations for every pair of
entity mentions (called relation instances) in the sentence while multiple
word spans in the sentence can be viewed as multiple instances where event
type predictions have to be made in ETD (trigger instances). As such, most
prior work on IE has performed predictions for instances in a sentence
separately by treating each instance as one example in the dataset Zhou et al.
(2005); Nguyen and Grishman (2015a); Santos and Guimaraes (2015); Chen et al.
(2015); Nguyen and Grishman (2015b); Lai et al. (2020). Second, at the task
level, prior work on IE tends to perform the four tasks in a pipelined
approach where outputs from one task are used as inputs for other tasks (e.g.,
EAE is followed by EME and ETD) Li et al. (2013); Chen et al. (2015); Veyseh
et al. (2020c).

Figure 1: A sentence example with the annotations for the four IE tasks. Blue
words corresponds to entity mentions while red words are event triggers. Also,
orange edges represent relations while green edges indicate argument roles.
Figure 2: Overall architecture of our proposed model. At the representation
level, $\texttt{GCN}^{inst}$ is used to enrich the representations for
instances of the four tasks. At the label level, $\texttt{GCN}^{type}$ is
responsible for capturing the connections between the types in the dependency
graphs, thus helping the model learn the structural difference between the
gold graph $\textbf{G}^{gold}$ and the predicted graph $\textbf{G}^{pred}$.

Despite its popularity, the main issue of the independent prediction models is
that they suffer from the error propagation between tasks and the failure to
exploit the cross-task and cross-instance inter-dependencies within an input
sentence to improve the performance for IE tasks. For instance, such systems
are unable to benefit from the dependency that the Victim of a Die event has a
high chance to also be the Victim of an Attack event in the same sentence
(i.e., type or label dependencies). To address these issues, some prior work
has explored joint inference models where multiple tasks of IE are performed
simultaneously for all task instances in a sentence, using both feature-based
models Roth and Yih (2004); Li et al. (2013); Miwa and Sasaki (2014); Yang and
Mitchell (2016) and recent deep learning models Miwa and Bansal (2016); Zhang
et al. (2019). However, such prior work has mostly considered joint models for
a subset of the four IE tasks (e.g., EME+RE or ETD+EAE), thus still suffering
from the error propagation issue (with the missing tasks) and failing to fully
exploit potential inter-dependencies between the four tasks. To this end, this
work aims to design a single model to simultaneously solve the four IE tasks
for each input sentence (joint four-task IE) to address the aforementioned
issues of prior joint IE work.

Few recent work has considered joint four-task IE, using deep learning to
produce state-of-the-art (SOTA) performance for the tasks Wadden et al.
(2019); Lin et al. (2020). However, there are still two problems that hinder
further improvement of such models. First, at the instance level, an important
component of deep learning models for joint IE involves the representation
vectors of the instances that are used to perform the corresponding prediction
tasks for IE in an input sentence (called predictive instance
representations). For joint four-task IE, we argue that there are inter-
dependencies between predictive representation vectors of related instances
for the four tasks that should be modeled to improve the performance for IE.
For instance, the entity type information encoded in the predictive
representation vector for an entity mention can constrain the argument role
that the representation vector for a related EAE instance (e.g., involving the
same entity mention and some event trigger in the same sentence) should
capture and vice versa. As such, prior work for joint four-task IE has only
computed predictive representation vectors for instances of the tasks
independently using shared hidden vectors from some deep learning layer Wadden
et al. (2019); Lin et al. (2020). Although this shared mechanism helps capture
the interaction of predictive representation vectors to some extent, it fails
to explicitly present the connections between related instances of different
tasks and encode them into the representation learning process. Consequently,
to overcome this issue, we propose a novel deep learning model for joint four-
task IE (called FourIE) that creates a graph structure to explicitly capture
the interactions between related instances of the four IE tasks in a sentence.
This graph will then be consumed by a graph convolutional network (GCN) Kipf
and Welling (2017); Nguyen and Grishman (2018) to enrich the representation
vector for an instance with those from the related (neighboring) instances for
IE.

Second, at the task level, existing joint four-task models for IE have only
exploited the cross-task type dependencies in the decoding step to constrain
predictions for the input sentence (by manually converting the type dependency
graphs of the input sentence into global feature vectors for scoring the
predictions in the beam search-based decoding) Lin et al. (2020). The
knowledge from cross-task type dependencies thus cannot contribute to the
training process of the IE models. This is unfortunate as we expect that
deeper integration of this knowledge into the training process could provide
useful information to enhance representation learning for IE tasks. To this
end, we propose to use the knowledge from cross-task type dependencies to
obtain an additional training signal for each sentence to directly supervise
our joint four-task IE model. In particular, our motivation is that the types
expressed in a sentence for the four IE tasks can be organized into a
dependency graph between the types (global type dependencies for the
sentence). As such, in order for a joint model to perform well, the type
dependency graph generated by its predictions for a sentence should be similar
to the dependency graph obtained from the golden types (i.e., a global type
constraint on the predictions in the training step). A novel regularization
term is thus introduced into the training loss of our joint model to encode
this constraint, employing another GCN to learn representation vectors for the
predicted and golden dependency graphs to facilitate the graph similarity
promotion. To our knowledge, this is the first work that employs global type
dependencies to regularize joint models for IE.

Finally, our extensive experiments demonstrate the effectiveness of the
proposed model on benchmark datasets in three different languages (e.g.,
English, Chinese, and Spanish), leading to state-of-the-art performance on
different settings.

##  2 Problem Statement and Background

Problem Statement: The joint four-task IE problem in this work takes a
sentence as the input and aims to jointly solve four tasks EAE, ETD, RE, and
EAE using an unified model. As such, the goal of EME is to detect and classify
entity mentions (names, nominals, pronouns) according to a set of predefined
(semantic) entity types (e.g., Person). Similarly, ETD seeks to identify and
classify event triggers (verbs or normalization) that clearly evoke an event
in some predefined set of event types (e.g., Attack). Note that event triggers
can involve multiple words. For RE, its concern is to predict the semantic
relationship between two entity mentions in the sentence. Here, the set of
relations of interest is also predefined and includes a special type of None
to indicate no-relation. Finally, in EAE, given an event trigger, the systems
need to predict the roles (also in a predefined set with a special type None)
that each entity mention plays in the corresponding event. Entity mentions are
thus also called event argument candidates in this work. Figure 1 presents a
sentence example where the expected outputs for each IE task are illustrated.

Graph Convolutional Networks (GCN): As GCNs are used extensively in our model,
we present their computation process in this section to facilitate the
discussion. Given a graph $\textbf{G}=(\textbf{V},\textbf{E})$ where
$\textbf{V}=\\{v_{1},\ldots,v_{u}\\}$ is the node set (with $u$ nodes) and E
is the edge set. In GCN, the edges in G are often captured via the adjacency
matrix $\textbf{A}\in\mathbb{R}^{u\times u}$. Also, each node
$v_{i}\in\textbf{V}$ is associated with an initial hidden vector
$\textbf{v}^{0}_{i}$. As such, a GCN model involves multiple layers of
abstraction in which the hidden vector $\textbf{v}^{l}_{i}$ for the node
$v_{i}\in\textbf{V}$ at the $l$-th layer is computed by ($l\geq 1$):

$\textbf{v}^{l}_{i}=\textrm{ReLU}(\frac{\sum^{u}_{j=1}\textbf{A}_{ij}\textbf{W}^{l}\textbf{v}^{l-1}_{j}+\textbf{b}^{l}}{\sum^{u}_{j=1}\textbf{A}_{ij}})$

where $\textbf{W}^{l}$ and $\textbf{b}^{l}$ are trainable weight and bias at
the $l$-th layer. Assuming $N$ GCN layers, the hidden vectors for the nodes in
V at the last layer $\textbf{v}^{N}_{1},\ldots,\textbf{v}^{N}_{u}$ would
capture richer and more abstract information for the nodes, serving as the
outputs of the GCN model. This process is denoted by:
$\textbf{v}^{N}_{1},\ldots,\textbf{v}^{N}_{u}=\texttt{GCN}(\textbf{A};\textbf{v}^{0}_{1},\ldots,\textbf{v}^{0}_{u};N)$.

##  3 Model

Given an input sentence $\textbf{w}=[w_{1},w_{2},\ldots,w_{n}]$ (with $n$
words), our model for joint four-task IE on w involves three major components:
(i) Span Detection, (ii) Instance Interaction, and (iii) Type Dependency-based
Regularization.

###  3.1 Span Detection

This component aims to identify spans of entity mentions and event triggers in
w that would be used to form the nodes in the interaction graph between
different instances of our four IE tasks for w. As such, we formulate the span
detection problems as sequence labeling tasks where each word $w_{i}$ in w is
associated with two BIO tags to capture the span information for entity
mentions and event triggers in w. Note that we do not predict entity types and
event types at this step, leading to only three possible values (i.e., B, I,
and O) for the tags of the words.

In particular, following Lin et al. (2020), we first feed w into the pre-
trained BERT encoder Devlin et al. (2019) to obtain a sequence of vectors
$\textbf{X}=[\textbf{x}_{1},\textbf{x}_{2},\ldots,\textbf{x}_{n}]$ to
represent w. Here, each vector $\textbf{x}_{i}$ serves as the representation
vector for the word $w_{i}\in\textbf{w}$ that is obtained by averaging the
hidden vectors of the word-pieces of $w_{i}$ returned by BERT. Afterward, X is
fed into two conditional random field (CRF) layers to determine the best BIO
tag sequences for event mentions and event triggers for w, following Chiu and
Nichols (2016). As such, the Viterbi algorithm is used to decode the input
sentence while the negative log-likelihood losses are employed as the training
objectives for the span detection component of the model. For convenience, let
$L^{ent}_{span}$ and $L^{trg}_{span}$ be the negative log-likelihoods of the
gold tag sequences for entity mentions and event triggers (respectively) for
w. These terms will be included in the overall loss function of the model
later.

###  3.2 Instance Interaction

Based on the tag sequences for w from the previous component, we can obtain
two separate span sets for the entity mentions and event triggers in w (the
golden spans are used in the training phase to avoid noise). For the next
computation, we first compute a representation vector for each span $(i,j)$
($1\leq i\leq j\leq n$) in these two sets by averaging the BERT-based
representation vectors for the words in this span (i.e.,
$\textbf{x}_{i},\ldots,\textbf{x}_{j}$). For convenience, let
$\textbf{R}^{ent}=\\{\textbf{e}_{1},\textbf{e}_{2},\ldots,\textbf{e}_{n_{ent}}\\}$
($n_{ent}=|\textbf{R}^{ent}|$) and
$\textbf{R}^{trg}=\\{\textbf{t}_{1},\textbf{t}_{2},\ldots,\textbf{t}_{n_{trg}}\\}$
($n_{trg}=|\textbf{R}^{trg}|$) be the sets of span representation vectors for
the entity mentions and event triggers in w111We will also refer to entity
mentions and event triggers interchangeably with their span representations
$\textbf{e}_{i}$ and $\textbf{t}_{i}$ in this work.. The goal of this
component is to leverage such span representation vectors to form instance
representations and enrich them with instance interactions to perform
necessary predictions in IE.

Instance Representation. Prediction instances in our model amount to the
specific objects that we need to predict a type for one of the four IE tasks.
As such, the prediction instances for EME and ETD, called entity and trigger
instances, correspond directly to the entity mentions and event triggers in
$\textbf{R}^{ent}$ and $\textbf{R}^{trg}$ respectively (as we need to predict
the entity types for $\textbf{e}_{i}\in\textbf{R}^{ent}$ and the event types
for $\textbf{t}_{i}\in\textbf{R}^{trg}$ in this step). Thus, we also use
$\textbf{R}^{ent}$ and $\textbf{R}^{trg}$ as the sets of initial
representation vectors for the entity/event instances for EME and ETD in the
following. Next, for RE, the prediction instances (called relation instances)
involve pairs of entity mentions in $\textbf{R}^{ent}$. To obtain the initial
representation vector for a relation instance, we concatenate the
representation vectors of the two corresponding entity mentions, leading to
the set of representation vectors $\textbf{rel}_{ij}$ for relation instances:
$\textbf{R}^{rel}=\\{\textbf{rel}_{ij}=[\textbf{e}_{i},\textbf{e}_{j}]\mid\textbf{e}_{i},\textbf{e}_{j}\in\textbf{R}^{ent},i<j\\}$
($|\textbf{R}^{rel}|=n_{ent}(n_{ent}-1)/2$). Finally, for EAE, we form the
prediction instances (called argument instances) by pairing each event trigger
in $\textbf{R}^{trg}$ with each entity mention in $\textbf{R}^{ent}$ (for the
argument role predictions of the entity mentions with respect to the event
triggers/mentions). By concatenating the representation vectors of the paired
entity mentions and event triggers, we generate the initial representation
vectors $\textbf{arg}_{ij}$ for the corresponding argument instances:
$\textbf{R}^{arg}=\\{\textbf{arg}_{ij}=[\textbf{t}_{i},\textbf{e}_{j}]\mid\textbf{t}_{i}\in\textbf{R}^{trg},\textbf{e}_{j}\in\textbf{R}^{ent}\\}$
($|\textbf{R}^{arg}|=n_{trg}n_{ent}$)222In our implementation,
$\textbf{R}^{rel}$ and $\textbf{R}^{arg}$ are transformed into vectors of the
same size with those in $\textbf{R}^{ent}$ and $\textbf{R}^{trg}$ (using one-
layer feed forward networks) for future computation.. We also use the
prediction instances and their representation vectors interchangeably in this
work.

Instance Interaction. The initial representation vectors for the instances so
far do not explicitly consider beneficial interactions between related
instances. To address this issue, we explicitly create an interaction graph
between the prediction instances for the four IE tasks to connect related
instances to each other. This graph will be consumed by a GCN model to enrich
instance representations with interaction information afterward. In
particular, the node set $\textbf{N}^{inst}$ in our instance interaction graph
$\textbf{G}^{inst}=\\{\textbf{N}^{inst},\textbf{E}^{inst}\\}$ involves all
prediction instances for the four IE tasks, i.e.,
$\textbf{N}^{inst}=\textbf{R}^{ent}\cup\textbf{R}^{trg}\cup\textbf{R}^{rel}\cup\textbf{R}^{arg}$.
The edge set $\textbf{E}^{inst}$ then captures instance interactions by
connecting the instance nodes in $\textbf{N}^{inst}$ that involve the same
entity mentions or event triggers (i.e., two instances are related if they
concern the same entity mention or event trigger). As such, the edges in
$\textbf{E}^{inst}$ are created as follows:

(i) An entity instance node $\textbf{e}_{i}$ is connected to all relation
instance nodes of the forms
$\textbf{rel}_{ij}=[\textbf{e}_{i},\textbf{e}_{j}]$ and
$\textbf{rel}_{ki}=[\textbf{e}_{k},\textbf{e}_{i}]$ (sharing entity mention
$\textbf{e}_{i}$).

(ii) An entity instance node $\textbf{e}_{j}$ is connected to all argument
instance nodes of the form $\textbf{arg}_{ij}=[\textbf{t}_{i},\textbf{e}_{j}]$
(sharing entity mention $\textbf{e}_{j}$).

(iii) A trigger node $\textbf{t}_{i}$ is connected to all argument instance
nodes of the form $\textbf{arg}_{ij}=[\textbf{t}_{i},\textbf{e}_{j}]$ (i.e.,
sharing event trigger $\textbf{t}_{i}$).

GCN. To enrich the representation vector for an instance in
$\textbf{N}^{inst}$ with the information from the related (neighboring) nodes,
we feed $\textbf{G}^{inst}$ into a GCN model (called $\texttt{GCN}^{inst}$).
For convenience, we rename the initial representation vectors of all the
instance nodes in $\textbf{N}^{inst}$ by:
$\textbf{N}^{inst}=\\{\textbf{r}_{1},\ldots,\textbf{r}_{n_{i}}\\}$
($n_{i}=|\textbf{N}^{inst}|$). Also, let
$\textbf{A}^{inst}\in\\{0,1\\}^{n_{i}\times n_{i}}$ be the adjacency matrix of
the interaction graph $\textbf{G}^{inst}$ where $\textbf{A}^{inst}_{ij}=1$ if
the instance nodes $\textbf{r}_{i}$ and $\textbf{r}_{j}$ are connected in
$\textbf{G}^{inst}$ or $i=j$ (for self-connections). The interaction-enriched
representation vectors for the instances in $\textbf{N}^{inst}$ are then
computed by the $\texttt{GCN}^{inst}$ model:
$\textbf{r}^{inst}_{1},\ldots,\textbf{r}^{inst}_{n_{i}}=\texttt{GCN}^{inst}(\textbf{A}^{inst};\textbf{r}_{1},\ldots,\textbf{r}_{n_{i}};N_{i})$
where $N_{i}$ is the number of layers for the $\texttt{GCN}^{inst}$ model.

Type Embedding and Prediction. Finally, the enriched instance representation
vectors $\textbf{r}^{inst}_{1},\ldots,\textbf{r}^{inst}_{n_{i}}$ will be used
to perform the predictions for the four IE tasks. In particular, let
$t_{k}\in\\{ent,trg,rel,arg\\}$ be the corresponding task index and $y_{k}$ be
the ground-truth type (of the task $t_{k}$) for the prediction instance
$\textbf{r}_{k}$ in $\textbf{N}^{inst}$. Also, let
$\mathcal{T}=\mathcal{T}^{ent}\cup\mathcal{T}^{trg}\cup\mathcal{T}^{rel}\cup\mathcal{T}^{arg}$
be the union of the possible entity types (in $\mathcal{T}^{ent}$ for EME),
event types (in $\mathcal{T}^{trg}$ for ETD), relations (in
$\mathcal{T}^{rel}$ for RE), and argument roles (in $\mathcal{T}^{arg}$ for
EAE) in our problem ($y_{k}\in\mathcal{T}^{t_{k}}$). Note that
$\mathcal{T}^{rel}$ and $\mathcal{T}^{arg}$ contain the special types None. To
prepare for the type predictions and the type dependency modeling in the next
steps, we associate each type in $\mathcal{T}$ with an embedding vector (of
the same size as $\textbf{e}_{i}$ and $\textbf{t}_{i}$) that is initialized
randomly and updated during our training process. For convenience, let
$\mathcal{T}=[\bar{\textbf{t}}_{1},\ldots,\bar{\textbf{t}}_{n_{t}}]$ where
$\bar{\textbf{t}}_{i}$ is used interchangeably for both a type and its
embedding vector in $\mathcal{T}$ ($n_{t}$ is the total number of types). As
such, to perform the prediction for an instance $\textbf{r}_{k}$ in
$\textbf{N}^{inst}$, we compute the dot products between
$\textbf{r}^{inst}_{k}$ and each type embedding vectors in
$\mathcal{T}^{t_{k}}\cap\mathcal{T}$ to estimate the possibilities that
$\textbf{r}_{k}$ has a type in $\mathcal{T}^{t_{k}}$. Afterward, these scores
are normalized by the softmax function to obtain the probability distribution
$\hat{\textbf{y}}_{k}$ over the possible types in $\mathcal{T}^{t_{k}}$ for
$\textbf{r}_{k}$:
$\hat{\textbf{y}}_{k}=softmax(\textbf{r}^{inst}_{k}\bar{\textbf{t}}^{T}|\bar{\textbf{t}}\in\mathcal{T}^{t_{k}}\cap\mathcal{T})$.
In the decoding phase, the predicted type $\hat{y}_{k}$ for $\textbf{r}_{k}$
is obtained via the argmax function (greedy decoding):
$\hat{y}_{k}=\texttt{argmax }\hat{\textbf{y}}_{k}$. The negative log-
likelihood over all the prediction instances is used to train the model:
$L_{type}=-\sum^{n_{i}}_{k=1}\log\hat{\textbf{y}}_{k}[y_{k}]$.

###  3.3 Type Dependency-based Regularization

In this section, we aim to obtain the type dependencies across tasks and use
them to supervise the model in the training process (to improve the
representation vectors for IE). As presented in the introduction, our
motivation is to generate global dependency graphs between types of different
IE tasks for each input sentence whose representations are leveraged to
regularize the model during training. In particular, starting with the golden
types $\textbf{y}=y_{1},y_{2},\ldots,y_{n_{i}}$ and the predicted types
$\hat{\textbf{y}}=\hat{y}_{1},\hat{y}_{2},\ldots,\hat{y}_{n_{i}}$ for the
instance nodes in $\textbf{N}^{inst}$, we build two dependency graphs
$\textbf{G}^{gold}$ and $\textbf{G}^{pred}$ to capture the global type
dependencies for the tasks (called the golden and predicted dependency graphs
respectively). Afterward, to supervise the training process, we seek to
constrain the model so the predicted dependency graph $\textbf{G}^{pred}$ is
similar to the golden graph $\textbf{G}^{gold}$ (i.e., using the dependency
graphs as the bridges to inject the global type dependency knowledge in
$\textbf{G}^{gold}$ into the model).

Dependency Graph Construction. Both $\textbf{G}^{gold}$ and
$\textbf{G}^{pred}$ involve the types of all the four IE tasks in
$\mathcal{T}$ as the nodes. To encode the type dependencies, the
connections/edges in $\textbf{G}^{gold}$ are computed based on the golden
types $\textbf{y}=y_{1},y_{2},\ldots,y_{n_{i}}$ for the instance nodes in
$\textbf{N}^{inst}$ as follows:

(i) For each relation instance node
$\textbf{r}_{k}=[\textbf{e}_{i},\textbf{e}_{j}]\in\textbf{N}^{inst}$ that has
the golden type $y_{k}\neq None$, the relation type node $y_{k}$ is connected
to the nodes of the golden entity types for the corresponding entity mentions
$\textbf{e}_{i}$ and $\textbf{e}_{j}$ (called entity_relation type edges).

(ii) For each argument instance node
$\textbf{r}_{k}=[\textbf{t}_{i},\textbf{e}_{j}]$ that has the role type
$y_{k}\neq None$, the role type node $y_{k}$ is connected to both the node for
the golden event type of $\textbf{t}_{i}$ (called event_argument type edges)
and the node for the golden entity type of $\textbf{e}_{j}$ (called
entity_argument type edges).

The same procedure can be applied to build the predicted dependency graph
$\textbf{G}^{pred}$ based on the predicted types
$\hat{\textbf{y}}=\hat{y}_{1},\hat{y}_{2},\ldots,\hat{y}_{n_{i}}$. Also, for
convenience, let $\textbf{A}^{gold}$ and $\textbf{A}^{pred}$ (of size
$n_{t}\times n_{t}$) be the binary adjacency matrices of $\textbf{G}^{gold}$
and $\textbf{G}^{pred}$ (including the self-loops) respectively.

Regularization. In the next step, we obtain the representation vectors for the
dependency graphs $\textbf{G}^{gold}$ and $\textbf{G}^{pred}$ by feeding them
into a GCN model (called $\texttt{GCN}^{type}$). This GCN model has $N_{t}$
layers and uses the initial type embeddings
$\mathcal{T}=[\bar{\textbf{t}}_{1},\ldots,\bar{\textbf{t}}_{n_{t}}]$ as the
inputs. In particular, the outputs of $\texttt{GCN}^{type}$ for the two graphs
involve
$\bar{\textbf{t}}^{gold}_{1},\ldots,\bar{\textbf{t}}^{gold}_{n_{t}}=\texttt{GCN}^{type}(\textbf{A}^{gold};\bar{\textbf{t}}_{1},\ldots,\bar{\textbf{t}}_{n_{t}};N_{t})$
and
$\bar{\textbf{t}}^{pred}_{1},\ldots,\bar{\textbf{t}}^{pred}_{n_{t}}=\texttt{GCN}^{type}(\textbf{A}^{pred};\bar{\textbf{t}}_{1},\ldots,\bar{\textbf{t}}_{n_{t}};N_{t})$
that encode the underlying information for the type dependencies presented in
$\textbf{G}^{gold}$ and $\textbf{G}^{pred}$. Finally, to promote the
similarity of the type dependencies in $\textbf{G}^{gold}$ and
$\textbf{G}^{pred}$, we introduce the mean square difference between their
$\texttt{GCN}^{type}$-induced representation vectors into the overall loss
function for minimization:
$L_{dep}=\sum^{n_{t}}_{i=1}||\bar{\textbf{t}}^{gold}_{i}-\bar{\textbf{t}}^{pred}_{i}||^{2}_{2}$.

Our final training loss is thus:
$L=L^{ent}_{span}+L^{trg}_{span}+L_{type}+\lambda L_{dep}$ ($\lambda$ is a
trade-off parameter).

Approximating $\textbf{A}^{pred}$. We distinguish two types of parameters in
our model so far, i.e., the parameters used to compute instance
representations, e.g., those in BERT and $\textbf{G}^{inst}$ (called
$\theta^{inst}$), and the parameters for type dependency regularization, i.e.,
those for the type embeddings
$\bar{\textbf{t}}_{1},\ldots,\bar{\textbf{t}}_{n_{t}}$ and $\textbf{G}^{type}$
(called $\theta^{dep}$). As such, the current implementation only enables the
training signal from $L_{dep}$ to back-propagate to the parameters
$\theta^{dep}$ and disallows $L_{dep}$ to influence the instance
representation-related parameters $\theta^{inst}$. To enrich the instance
representation vectors with type dependency information, we expect $L_{dep}$
to be deeper integrated into the model by also contributing to
$\theta^{inst}$. To achieve this goal, we note that the block of back-
propagation between $L_{dep}$ and $\theta^{inst}$ is due to their only
connection in the model via the adjacency matrix $\textbf{A}^{pred}$, whose
values are either one or zero. As such, the values in $\textbf{A}^{pred}$ are
not directly dependent on any parameter in $\theta^{inst}$, making it
impossible for the back-propagation to flow. To this end, we propose to
approximate $\textbf{A}^{pred}$ with a new matrix $\hat{\textbf{A}}^{pred}$
that directly involves $\theta^{inst}$ in its values. In particular, let
$\textbf{I}^{inst}$ be the index set of the non-zero cells in
$\textbf{A}^{pred}$: $\textbf{I}^{inst}=\\{(i,j)|\textbf{A}^{pred}_{ij}=1\\}$.
As the elements in $\textbf{I}^{inst}$ are determined by the indexes
$i_{1},\ldots,i_{n_{i}}$ in $\mathcal{T}$ of the predicted types
$\hat{y}_{1},\hat{y}_{2},\ldots,\hat{y}_{n_{i}}$ (respectively), we also seek
to compute the values for the approximated matrix $\hat{\textbf{A}}^{pred}$
based on such indexes. Accordingly, we first define the matrix
$\textbf{B}=\\{b_{ij}\\}_{i,j=1..n_{t}}$ where the element $b_{ij}$ at the
$i$-th row and $j$-th column is set to $b_{ij}=i*n_{t}+j$. The approximated
matrix $\hat{\textbf{A}}^{pred}$ is then obtained by:

$\hat{\textbf{A}}^{pred}=\sum_{(i,j)\in\textbf{I}^{inst}}\exp(-\beta(\textbf{B}-in_{t}-j)^{2})$
(1)

Here, $\beta>0$ is a large constant. For each element
$(i,j)\in\textbf{I}^{inst}$, all the elements in the matrix
$(\textbf{B}-in_{t}-j)^{2}$ are strictly positive, except for the element at
$(i,j)$, which is zero. Thus, with a large value for $\beta$, the matrix
$\exp(-\beta(\textbf{B}-in_{t}-j)^{2})$ has the value of one at cell $(i,j)$
and nearly zero at other cells. Consequently, the values of
$\hat{\textbf{A}}^{pred}$ at the positions in $\textbf{I}^{inst}$ are close to
one while those at other positions are close to zero, thus approximating our
expected matrix $\textbf{A}^{pred}$ and still directly depending on the
indexes $i_{1},\ldots,i_{n_{t}}$.

Addressing the Discreteness of Indexes. Even with the approximation
$\hat{\textbf{A}}^{pred}$, the back-propagation still cannot flow from
$L_{dep}$ to $\theta^{inst}$ due to the block of the discrete and non-
differentiable index variables $i_{1},\ldots,i_{n_{t}}$. To address this
issue, we propose to apply the Gumbel-Softmax distribution Jang et al. (2017)
that enables the optimization of models with discrete random variables, by
providing a method to approximate one-hot vectors sampled from a categorical
distribution with continuous ones.

In particular, we first rewrite each index $i_{k}$ by:
$i_{k}=\textbf{h}_{k}\textbf{c}^{T}_{k}$, where $\textbf{c}_{k}$ is a vector
whose each dimension contains the index of a type in $\mathcal{T}^{t_{k}}$ in
the joint type set $\mathcal{T}$, and $\textbf{h}_{k}$ is the binary one-hot
vector whose dimensions correspond to the types in $\mathcal{T}^{t_{k}}$.
$\textbf{h}_{k}$ is only turned on at the position corresponding to the
predicted type $\hat{y}_{k}\in\mathcal{T}^{t_{k}}$ (indexed at $i_{k}$ in
$\mathcal{T}$). In our current implementation, $\hat{y}_{k}$ (thus the index
$i_{k}$ and the one-hot vector $\textbf{h}_{k}$) is obtained via the argmax
function: $\hat{y}_{k}=\texttt{argmax }\hat{\textbf{y}}_{k}$, which causes the
discreteness. As such, the Gumbel-Softmax distribution method helps to relax
argmax by approximating $\textbf{h}_{k}$ with a sample
$\hat{\textbf{h}}_{k}=\hat{h}_{k,1},\ldots,\hat{h}_{k,|\mathcal{T}^{t_{k}}|}$
from the Gumbel-Softmax distribution:

$\hat{h}_{k,j}=\frac{\textrm{exp}((\textrm{log}(\pi_{k,j})+g_{j})/\tau)}{\sum^{|\mathcal{T}^{t_{k}}|}_{j^{\prime}=1}\textrm{exp}((\textrm{log}(\pi_{k,j^{\prime}})+g_{j^{\prime}})/\tau)}$
(2)

where
$\pi_{k,j}=\hat{\textbf{y}}_{k,j}=softmax_{j}(\textbf{r}^{inst}_{k}\bar{\textbf{t}}^{T}|\bar{\textbf{t}}\in\mathcal{T}^{t_{k}}\cap\mathcal{T})$,
$g_{1},\ldots,g_{|\mathcal{T}^{t_{k}}|}$ are the i.i.d samples drawn from
Gumbel(0,1) distribution Gumbel (1948):
$g_{j}=-\textrm{log}(-\textrm{log}(u_{j}))$
($u_{j}\sim\textrm{Uniform}(0,1)$), and $\tau$ is the temperature parameter.
As $\tau\rightarrow 0$, the sample $\hat{\textbf{h}}_{k}$ would become close
to our expected one-hot vector $\textbf{h}_{k}$. Finally, we replace
$\textbf{h}_{k}$ with the approximation $\hat{\textbf{h}}_{k}$ in the
computation for $i_{k}$: $i_{k}=\hat{\textbf{h}}_{k}\textbf{c}^{T}_{k}$ that
directly depends on $\textbf{r}^{inst}_{k}$ and is applied in
$\hat{\textbf{A}}^{pred}$. This allows the gradients to flow from $L_{dep}$ to
the parameters $\theta^{inst}$ and completes the description of our model.

##  4 Experiments

Datasets. Following the prior work on joint four-task IE Wadden et al. (2019);
Lin et al. (2020), we evaluate our joint IE model (FourIE) on the ACE 2005
Walker et al. (2006) and ERE datasets that provide annotation for entity
mentions, event triggers, relations, and argument roles. In particular, we use
three different versions of the ACE 2005 dataset that focus on three major
joint inference settings for IE: (i) ACE05-R for joint inference of EME and
RE, (ii) ACE05-E for joint inference of EME, ETD and EAE, and (iii) ACE05-E+
for joint inference of the four tasks EME, ETD, RE, and EAE. ACE05-E+ is our
main evaluation setting as it fits to our model design with the four IE tasks
of interest.

Datasets | Split | sents | ents | rels | events
---|---|---|---|---|---
ACE05-R | Train | 10,051 | 26,473 | 4,788 | -
Dev | 2,424 | 6,362 | 1,131 | -
Test | 2,050 | 5,476 | 1,151 | -
ACE05-E | Train | 17,172 | 29,006 | 4,664 | 4,202
Dev | 923 | 2,451 | 560 | 450
Test | 832 | 3,017 | 636 | 403
ACE05-E+ | Train | 19,240 | 47,525 | 7,152 | 4,419
Dev | 902 | 3,422 | 728 | 468
Test | 676 | 3,673 | 802 | 424
ERE-EN | Train | 14,219 | 38,864 | 5,045 | 6,419
Dev | 1,162 | 3,320 | 424 | 552
Test | 1,129 | 3,291 | 477 | 559
ACE05-CN | Train | 6,841 | 29,657 | 7,934 | 2,926
Dev | 526 | 2,250 | 596 | 217
Test | 547 | 2,388 | 672 | 190
ERE-ES | Train | 7,067 | 11,839 | 1,698 | 3,272
Dev | 556 | 886 | 120 | 210
Test | 546 | 811 | 108 | 269

Table 1: Numbers of sentences (i.e., sents), entity mentions (i.e., ents),
relations (i.e., rels), and events (i.e., events) in the datasets.

For ERE, following Lin et al. (2020), we combine the data from three datasets
for English (i.e., LDC2015E29, LDC2015E68, and LDC2015E78) that are created
under the Deep Exploration and Filtering of Test (DEFT) program (called ERE-
EN). Similar to ACE05-E+, ERE-EN is also used to evaluate the joint models on
four IE tasks.

To demonstrate the portability of our model to other languages, we also apply
FourIE to the joint four-IE datasets on Chinese and Spanish. Following Lin et
al. (2020), we use the ACE 2005 dataset for the evaluation on Chinese (called
ACE05-CN) and the ERE dataset (LDC2015E107) for Spanish (called ERE-ES).

To ensure a fair comparison, we adopt the same data pre-processing and splits
(train/dev/test) in prior work Lin et al. (2020) for all the datasets. As
such, ACE05-R, ACE05-E, ACE05-E+, and AC05-CN involve 7 entity types, 6
relation types, 33 event types, and 22 argument roles while ERE-ES and ERE-EN
include 7 entity types, 5 relation types, 38 event types, and 20 argument
roles. The statistics for the datasets are shown in Table 1.

Hyper-parameters and Evaluation Criteria. We fine-tune the hyper-parameters
for our model using the development data. The suggested values are shown in
the appendix. To achieve a fair comparison with Lin et al. (2020), we employ
the bert-large-cased model for the English datasets and bert-multilingual-
cased model for the Chinese and Spanish datasets. Finally, we follow the same
evaluation script and correctness criteria for entity mentions, event
triggers, relations, and argument as in prior work Lin et al. (2020). The
reported results are the average performance of 5 model runs using different
random seeds.

Performance Comparison. We compare the proposed model FourIE with two prior
models for joint four-task IE: (i) DyGIE++ Wadden et al. (2019): a BERT-based
model with span graph propagation, and (ii) OneIE Lin et al. (2020): the
current state-of-the-art (SOTA) model for joint four-task IE based on BERT and
type dependency constraint at the decoding step. Table 2 presents the
performance (F1 scores) of the models on the test data of the English
datasets. Note that in the tables, the prefixes “Ent”, “Trg”, “Rel”, and “Arg”
represent the extraction tasks for entity mentions, event triggers, relations,
and arguments respectively while the suffixes “-I” and “-C” correspond to the
identification performance (only concerning the offset correctness) and
identification+classification performance (evaluating both offsets and types).

Datasets | Task | DyGIE++ | OneIE | FourIE | $\Delta\%$
---|---|---|---|---|---
ACE05-R | Ent-C | 88.6 | 88.8 | 88.9 | 0.1
Rel-C | 63.4 | 67.5 | 68.9${\dagger}$ | 1.4
ACE05-E | Ent-C | 89.7 | 90.2 | 91.3${\dagger}$ | 1.1
Trg-I | - | 78.2 | 78.3 | 0.1
Trg-C | 69.7 | 74.7 | 75.4${\dagger}$ | 0.7
Arg-I | 53.0 | 59.2 | 60.7${\dagger}$ | 1.5
Arg-C | 48.8 | 56.8 | 58.0${\dagger}$ | 1.2
ACE05-E+ | Ent-C | - | 89.6 | 91.1${\dagger}$ | 1.5
Rel-C | - | 58.6 | 63.6${\dagger}$ | 5.0
Trg-I | - | 75.6 | 76.7${\dagger}$ | 1.1
Trg-C | - | 72.8 | 73.3${\dagger}$ | 0.5
Arg-I | - | 57.3 | 59.5${\dagger}$ | 2.2
Arg-C | - | 54.8 | 57.5${\dagger}$ | 2.7
ERE-EN | Ent-C | - | 87.0 | 87.4 | 0.4
Rel-C | - | 53.2 | 56.1${\dagger}$ | 2.9
Trg-I | - | 68.4 | 69.3${\dagger}$ | 0.9
Trg-C | - | 57.0 | 57.9${\dagger}$ | 0.9
Arg-I | - | 50.1 | 52.2${\dagger}$ | 2.1
Arg-C | - | 46.5 | 48.6${\dagger}$ | 2.1

Table 2: F1 scores of the models on the test data of English datasets.
$\Delta$ indicates the performance difference between FourIE and OneIE. Rows
with ${\dagger}$ designate the significant improvement ($p<0.01$) of FourIE
over OneIE.

As can be seen from the table, FourIE is consistently better than the two
baseline models (DyGIE++ and OneIE) across different datasets and tasks. The
performance improvement is significant for almost all the cases and clearly
demonstrates the effectiveness of the proposed model.

Finally, Table 3 reports the performance of FourIE and OneIE on the Chinese
and Spanish datasets (i.e., ACE05-CN and ERE-ES). In addition to the
monolingual setting (i.e., trained and evaluated on the same languages),
following Lin et al. (2020), we also evaluate the models on the multilingual
training settings where ACE05-CN and ERE-ES are combined with their
corresponding English datasets ACE05-E+ and EAE-EN (respectively) to train the
models (for the four IE tasks), and the performance is then evaluated on the
test sets of the corresponding languages (i.e., ACE05-CN and ERE-ES). It is
clear from the table that FourIE also significantly outperforms OneIE across
nearly all the different setting combinations for languages, datasets and
tasks. This further illustrates the portability of FourIE to different
languages.

| Test Data
---
| Train Data
---
Task | OneIE | FourIE | $\Delta\%$
ACE05-CN | ACE05-CN | Ent-C | 88.5 | 88.7 | 0.2
Rel-C | 62.4 | 65.1${\dagger}$ | 2.7
Trg-C | 65.6 | 66.5${\dagger}$ | 0.9
Arg-C | 52.0 | 54.9${\dagger}$ | 2.9
ACE05-CN ACE05-E+ | Ent-C | 89.8 | 89.1 | -0.7
Rel-C | 62.9 | 65.9${\dagger}$ | 3.0
Trg-C | 67.7 | 70.3${\dagger}$ | 2.6
Arg-C | 53.2 | 56.1${\dagger}$ | 2.9
ERE-ES | ERE-ES | Ent-C | 81.3 | 82.2${\dagger}$ | 0.9
Rel-C | 48.1 | 57.9${\dagger}$ | 9.8
Trg-C | 56.8 | 57.1 | 0.3
Arg-C | 40.3 | 42.3${\dagger}$ | 2.0
ERE-ES ERE-EN | Ent-C | 81.8 | 82.7${\dagger}$ | 0.9
Rel-C | 52.9 | 59.1${\dagger}$ | 6.2
Trg-C | 59.1 | 61.3${\dagger}$ | 2.2
Arg-C | 42.3 | 45.4${\dagger}$ | 3.1

Table 3: F1 scores on Chinese and Spanish test sets. ${\dagger}$ marks the
significant improvement ($p<0.01$) of FourIE over OneIE.

Effects of  GCNinst and GCNtype. This section evaluates the contributions of
the two important components in our proposed model FourIE, i.e., the instance
interaction graph with $\texttt{GCN}^{inst}$ and the type dependency graph
with $\texttt{GCN}^{type}$. In particular, we examine the following
ablated/varied models for FourIE: (i) “FourIE-GCNinst”: this model excludes
the instance interaction graph and the GCN model GCNinst from FourIE so the
initial instance representations $\textbf{r}_{k}$ are directly used to predict
the types for the instances (replacing the enriched vectors
$\textbf{r}^{inst}_{k}$), (ii) “FourIE-GCNtype”: this model eliminates the
type dependency graph and the GCN model GCNtype (thus the loss term $L_{dep}$
as well) from FourIE, (iii) “FourIE-GCNinst-GCNtype”: this model removes both
the instance interaction and type dependency graphs from FourIE, (iv) “FourIE-
GCNtype+TDDecode”: this model also excludes GCNtype; however, it additionally
applies the global type dependencies features to score the joint predictions
for the beam search in the decoding step (the implementation for this beam
search is inherited from Lin et al. (2020) for a fair comparison), and (v)
“FourIE-$\hat{\textbf{A}}^{pred}$”: instead of employing the approximation
matrix $\hat{\textbf{A}}^{pred}$ in FourIE, this model directly uses the
adjacency matrix $\textbf{A}^{pred}$ in the $L_{dep}$ regularizer ($L_{dep}$
thus does not influence the instance representation-related parameters
$\theta^{inst}$). Table 4 shows the performance of the models on the
development dataset of ACE05-E+ for four IE tasks.

Models | Ent-C | Rel-C | Trg-C | Arg-C
---|---|---|---|---
FourIE | 89.6 | 64.3 | 71.0 | 59.0
FourIE-GCNinst | 89.1 | 62.3 | 70.3 | 57.5
FourIE-GCNtype | 88.5 | 61.8 | 69.9 | 56.6
FourIE-GCNinst-GCNtype | 88.2 | 59.3 | 68.9 | 56.1
FourIE-GCNtype+TDDecode | 88.8 | 59.6 | 70.8 | 56.8
FourIE-$\hat{\textbf{A}}^{pred}$ | 89.0 | 62.3 | 70.2 | 57.6

Table 4: F1 scores of the models on the ACE05-E+ dev data.

The most important observation from the table is that both GCNinst and GCNtype
are necessary for FourIE to achieve the highest performance for the four IE
tasks. Importantly, replacing GCNtype in FourIE with the global type
dependency features for decoding (i.e., “FourIE-GCNtype+TDDecode”) as in Lin
et al. (2020) or eliminating the approximation $\hat{\textbf{A}}^{pred}$ for
$L_{dep}$ produces inferior performance, especially for relation and argument
extraction. This clearly demonstrates the benefits for deeply integrating
knowledge from type dependencies to influence representation learning
parameters with $L_{dep}$ for joint four-task IE.

Contributions of Type Dependency Edges. Our type dependency graphs
$\textbf{G}^{gold}$ and $\textbf{G}^{pred}$ involves three categories of
edges, i.e., entity_relation, entity_argument, and event_argument type edges.
Table 5 presents the performance of FourIE (on the development data of
ACE05-E+) when each of these edge categories is excluded from our type
dependency graph construction.

Models | Ent-C | Rel-C | Trg-C | Arg-C
---|---|---|---|---
FourIE | 89.6 | 64.3 | 71.0 | 59.0
FourIE - entity_relation | 88.7 | 61.9 | 71.0 | 57.5
FourIE - entity_argument | 89.3 | 63.2 | 70.0 | 56.9
FourIE - event_argument | 89.5 | 64.1 | 69.8 | 57.7

Table 5: F1 scores of the ablated models for type dependency edges on the
ACE05-E+ dev data.

The table clearly shows the importance of different categories of type
dependency edges for FourIE as the elimination of any category would generally
hurt the performance of the model. In addition, we see that the contribution
level of the type dependency edges intuitively varies according to the tasks
of consideration. For instance, entity_relation type edges are helpful mainly
for entity mention, relation and argument extraction. Finally, an error
analysis is conducted in the appendix to provide insights about the benefits
of the type dependency graphs $\textbf{G}^{gold}$ and $\textbf{G}^{pred}$ for
FourIE (i.e., by comparing the outputs of FourIE and “FourIE-GCNtype”).

##  5 Related Work

The early joint methods for IE have employed feature engineering to capture
the dependencies between IE tasks, including Integer Linear Programming for
Global Constraints Roth and Yih (2004); Li et al. (2011), Markov Logic
Networks Riedel et al. (2009); Venugopal et al. (2014), Structured Perceptron
Li et al. (2013, 2014); Miwa and Sasaki (2014); Judea and Strube (2016), and
Graphical Models Yu and Lam (2010); Yang and Mitchell (2016).

Recently, the application of deep learning has facilitated the joint modeling
for IE via shared parameter mechanisms across tasks. These joint models have
focused on different subsets of the IE tasks, including EME and RE Zheng et
al. (2017); Katiyar and Cardie (2017); Bekoulis et al. (2018); Fu et al.
(2019); Luan et al. (2019); Sun et al. (2019); Veyseh et al. (2020b, a), event
and temporal RE Han et al. (2019), and ETD and EAE Nguyen et al. (2016); Zhang
et al. (2019); Nguyen and Nguyen (2019). However, none of these work has
explored joint inference for four IE tasks EME, ETD, RE, and EAE as we do. The
two most related works to ours include Wadden et al. (2019) that leverages the
BERT-based information propagation via dynamic span graphs, and Lin et al.
(2020) that exploits BERT and global type dependency features to constrain the
decoding step. Our model is different from these works in that we introduce a
novel interaction graph for instance representations for four IE tasks and a
global type dependency graph to directly inject the knowledge into the
training process.

##  6 Conclusion

We present a novel deep learning framework to jointly solve four IE tasks
(EME, ETD, RE, and EAE). Our model attempts to capture the inter-dependencies
between instances of the four tasks and their types based on instance
interaction and type dependency graphs. GCN models are employed to induce
representation vectors to perform type predictions for task instances and
regularize the training process. The experiments demonstrate the effectiveness
of the proposed model, leading to SOTA performance over multiple datasets on
English, Chinese, and Spanish. In the future, we plan to extend the model to
include more IE tasks (e.g., coreference resolution).

## Acknowledgments

This research has been supported by the Army Research Office (ARO) grant
W911NF-21-1-0112. This research is also based upon work supported by the
Office of the Director of National Intelligence (ODNI), Intelligence Advanced
Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006
under the Better Extraction from Text Towards Enhanced Retrieval (BETTER)
Program. The views and conclusions contained herein are those of the authors
and should not be interpreted as necessarily representing the official
policies, either expressed or implied, of ARO, ODNI, IARPA, the Department of
Defense, or the U.S. Government. The U.S. Government is authorized to
reproduce and distribute reprints for governmental purposes notwithstanding
any copyright annotation therein. This document does not contain technology or
technical data controlled under either the U.S. International Traffic in Arms
Regulations or the U.S. Export Administration Regulations.

## References

  * Bekoulis et al. (2018) Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018.  Adversarial training for multi-context joint entity and relation extraction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Chen et al. (2015) Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015.  Event extraction via dynamic multi-pooling convolutional neural networks.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_.
  * Chiu and Nichols (2016) Jason PC Chiu and Eric Nichols. 2016.  Named entity recognition with bidirectional lstm-cnns.  In _Transactions of the Association for Computational Linguistics_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Fu et al. (2019) Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019.  GraphRel: Modeling text as relational graphs for joint entity and relation extraction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Gumbel (1948) Emil Julius Gumbel. 1948.  Statistical theory of extreme values and some practical applications: a series of lectures.  In _US Government Printing Office_.
  * Han et al. (2019) Rujun Han, Qiang Ning, and Nanyun Peng. 2019.  Joint event and temporal relation extraction with shared representations and structured prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparameterization with gumbel-softmax.  In _Proceedings of the 5th International Conference on Learning Representations_.
  * Judea and Strube (2016) Alex Judea and Michael Strube. 2016.  Incremental global event extraction.  In _Proceedings of the 26th International Conference on Computational Linguistics_.
  * Katiyar and Cardie (2017) Arzoo Katiyar and Claire Cardie. 2017.  Going out on a limb: Joint extraction of entity mentions and relations without dependency trees.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _Proceedings of the 5th International Conference on Learning Representations_.
  * Lai et al. (2020) Viet Dac Lai, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020.  Event detection: Gate diversity and syntactic importance scores for graph convolution neural networks.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Li et al. (2011) Qi Li, S. Anzaroot, W. Lin, Xiang Li, and H. Ji. 2011.  Joint inference for cross-document information extraction.  In _Proceedings of the 20th ACM international conference on Information and knowledge management_.
  * Li et al. (2014) Qi Li, Heng Ji, Yu Hong, and Sujian Li. 2014.  Constructing information networks using one single model.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_.
  * Li et al. (2013) Qi Li, Heng Ji, and Liang Huang. 2013.  Joint event extraction via structured prediction with global features.  In _Proceedings of the 51th Annual Meeting of the Association for Computational Linguistics_.
  * Lin et al. (2020) Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020.  A joint neural model for information extraction with global features.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Luan et al. (2019) Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019.  A general framework for information extraction using dynamic span graphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016.  End-to-end relation extraction using LSTMs on sequences and tree structures.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_.
  * Miwa and Sasaki (2014) Makoto Miwa and Yutaka Sasaki. 2014.  Modeling joint entity and relation extraction with table representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_.
  * Nguyen et al. (2016) Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016.  Joint event extraction via recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Nguyen and Grishman (2015a) Thien Huu Nguyen and Ralph Grishman. 2015a.  Event detection and domain adaptation with convolutional neural networks.  In _The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_.
  * Nguyen and Grishman (2015b) Thien Huu Nguyen and Ralph Grishman. 2015b.  Relation extraction: Perspective from convolutional neural networks.  In _Proceedings of the 1st NAACL Workshop on Vector Space Modeling for NLP (VSM)_.
  * Nguyen and Grishman (2018) Thien Huu Nguyen and Ralph Grishman. 2018.  Graph convolutional networks with argument-aware pooling for event detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Nguyen and Nguyen (2019) Trung Minh Nguyen and Thien Huu Nguyen. 2019.  One for all: Neural joint modeling of entities and events.  In _Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI)_.
  * Riedel et al. (2009) Sebastian Riedel, Hong-Woo Chun, Toshihisa Takagi, and Jun’ichi Tsujii. 2009.  A markov logic approach to bio-molecular event extraction.  In _Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task_.
  * Roth and Yih (2004) Dan Roth and Wen-tau Yih. 2004.  A linear programming formulation for global inference in natural language tasks.  In _Proceedings of the Eighth Conference on Computational Natural Language Learning_.
  * Santos and Guimaraes (2015) Cicero dos Santos and Victor Guimaraes. 2015.  Boosting named entity recognition with neural character embeddings.  In _Proceedings of the Fifth Named Entity Workshop_.
  * Sun et al. (2019) Changzhi Sun, Yeyun Gong, Yuanbin Wu, Ming Gong, Daxin Jiang, Man Lan, Shiliang Sun, and Nan Duan. 2019.  Joint type inference on entities and relations via graph convolutional networks.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Venugopal et al. (2014) Deepak Venugopal, Chen Chen, Vibhav Gogate, and Vincent Ng. 2014.  Relieving the computational bottleneck: Joint inference for event extraction with high-dimensional features.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_.
  * Veyseh et al. (2020a) Amir Pouran Ben Veyseh, Franck Dernoncourt, Dejing Dou, and Thien Huu Nguyen. 2020a.  Exploiting the syntax-model consistency for neural relation extraction.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Veyseh et al. (2020b) Amir Pouran Ben Veyseh, Franck Dernoncourt, My Thai, Dejing Dou, and Thien Huu Nguyen. 2020b.  Multi-view consistency for relation extraction via mutual information and structure prediction.  In _Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI)_.
  * Veyseh et al. (2020c) Amir Pouran Ben Veyseh, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020c.  Graph transformer networks with syntactic and semantic structures for event argument extraction.  In _Proceedings of the Findings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP Findings)_.
  * Wadden et al. (2019) David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019.  Entity, relation, and event extraction with contextualized span representations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006.  Ace 2005 multilingual training corpus.  In _Technical report, Linguistic Data Consortium_.
  * Yang and Mitchell (2016) Bishan Yang and Tom M. Mitchell. 2016.  Joint extraction of events and entities within a document context.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Yu and Lam (2010) Xiaofeng Yu and Wai Lam. 2010.  Jointly identifying entities and extracting relations in encyclopedia text via a graphical model approach.  In _Proceedings of the 23th International Conference on Computational Linguistics_.
  * Zhang et al. (2019) Junchi Zhang, Yanxia Qin, Yue Zhang, Mengchi Liu, and Donghong Ji. 2019.  Extracting entities and events as a single task using a transition-based neural model.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_.
  * Zheng et al. (2017) Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017.  Joint extraction of entities and relations based on a novel tagging scheme.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_.
  * Zhou et al. (2005) GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005.  Exploring various knowledge in relation extraction.  In _Proceedings of the 43rd annual meeting of the association for computational linguistics_.

##  Appendix A Hyper-parameters

We fine-tune the hyper-parameters for our model FourIE using the development
data of the ACE05-E+ dataset (our main and largest evaluation dataset). The
selection criteria is based on the average F1 scores of the four IE tasks of
consideration (EME, ETD, RE, and EAE). The following values are suggested by
the fine-tuning: $2e$-$5$ for the learning rate of BertAdam for the optimizer;
10 for the batch size; $N_{i}=2$ and $N_{t}=3$ for the numbers of layers in
the GCN models $\texttt{G}^{inst}$ and $\texttt{G}^{pred}$ respectively; 300
hidden units for all the layers of the feed forward networks GCN models, and
type embeddings; $\beta=100$ for the constant in the approximation
$\hat{\textbf{A}}^{pred}$; $\tau=0.1$ for the temperature parameter; and
$\lambda=0.5$ for the trade-off parameter in the loss function. To achieve a
consistency, we apply the same hyper-parameters from this fine-tuning for
other datasets.

##  Appendix B Analysis

Analysis. To better understand the contribution of the knowledge from the type
dependency graphs $\textbf{G}^{gold}$ and $\textbf{G}^{pred}$ for FourIE on
EAE, we analyze the set of all the argument instances on the ACE05-E+
development set (called $\mathcal{A}$) that FourIE can successfully predict
the argument roles while “FourIE-GCNtype” fails to do so. In particular, we
find three major categories of the instances in $\mathcal{A}$ that highlight
the benefits of the type dependency graphs:

(i) One-edge constraints (accounting for 28.9% of $\mathcal{A}$): The
incorrect argument role predictions of “FourIE-GCNtype” for these instances
violate the constraint on the possible argument roles of event types. As
FourIE does not have this issue, it suggests that FourIE can learn and enforce
those constraints (i.e., from the event_argument edges of $\textbf{G}^{gold}$)
from the training. For instance, in the sentence “… the United States upped
its military presence, deploying more missile-firing warships to the Red Sea”,
both FourIE and “FourIE-GCNtype” can recognize “deploying” as an event trigger
of type Transport. However, regarding the entity mention “Red Sea”, FourIE
correctly assigns the Destination role for the Transport event while “FourIE-
GCNtype” incorrectly considers it as the role Place (an invalid role for the
event type Transport).

(ii) Two-edge constraints (representing 36.5% of $\mathcal{A}$): The
predictions from “FourIE-GCNtype” in this category involve argument roles that
are never assigned to an entity mention of some entity type in an event
mention/trigger of some event type. FourIE can avoid this issue as it can
recognize these constraints from the combinations of two neighboring edges
(i.e., an event_argument and and entity_argument edge). For example, in the
sentence “… the tanks and Bradley fighting vehicles … backed by the Apache
attack helicopters … punched through the Republican Guard defenses …”, both
FourIE and “FourIE-GCNtype” can detect “helicopters” as an entity mention of
type Vehicle which is an argument for the “Attack” event triggered by “punched
through”. However, “FourIE-GCNtype” incorrectly predicts the argument role of
“Attacker” for “helicopters” while FourIE can successfully return the
Instrument in this case. In fact, we cannot find any Vehicle entity that plays
the Attacker role in an Attack event in the training data, providing an useful
information for FourIE to learn and fix the error.

(iii) Four-edge constraints (accounting for 19.2% of $\mathcal{A}$): The
failure of “FourIE-GCNtype” for the instances in this category can be fixed if
the model exploits the co-occurrence of event types and argument roles in the
same sentences. In particular, for two event mentions with related event types
in the same sentences, an entity mention that plays some role in one event
tends to also play some related role in the other event. These co-occurrence
can be captured via two event_argument edges and two entity_argument edges
(sharing the same entity type) in the type dependency graphs of FourIE to
address the issue. Consider an example sentence: “Two 13-year-old children
were among those killed in the Haifa bus bombing, Israeli public radio said
…”. Both FourIE and “FourIE-GCNtype” can identify the Person entity mention
“those” as the argument of role Victim for the Die event triggered by
“killed”. However, regarding the Attack event triggered by “bombing”, only
FourIE can correctly predict “those” as an argument of role Target”. This
success can be attributed to the ability of FourIE to learn the co-occurrence
that an entity mention has a higher chance to play the role Target in an
Attack event if it also has a role of Victim for a Die event mentioned in the
same sentence.

Finally, the instances in the remaining 15.4% of $\mathcal{A}$ tend to involve
more complicated constraints/dependencies that cannot be associated with any
of the three categories above.