# Span-Oriented Information Extraction
A Unifying Perspective on Information Extraction

Yifan Ding<EMAIL_ADDRESS>
University of Notre Dame
Notre Dame, IN, USA Michael Yankoski<EMAIL_ADDRESS>
Colby College
Waterville, ME, USA Tim Weninger<EMAIL_ADDRESS>
University of Notre Dame
Notre Dame, IN, USA

###### Abstract

Information Extraction refers to a collection of tasks within Natural Language
Processing (NLP) that identifies sub-sequences within text and their labels.
These tasks have been used for many years to link extract relevant information
and to link free text to structured data. However, the heterogeneity among
information extraction tasks impedes progress in this area. We therefore offer
a unifying perspective centered on what we define to be spans in text. We then
re-orient these seemingly incongruous tasks into this unified perspective and
then re-present the wide assortment of information extraction tasks as
variants of the same basic Span-Oriented Information Extraction task.

Keywords: propaganda, image forensics, social media, Ukraine, Russia

##  1 Introduction

Most of the unstructured or semi-structured text data available today is
available as a sequence-of-bytes (i.e. a sequence of text characters). From
these sequences-of-bytes, both humans and machines create whole-words and
sentences, and then more-complex paragraphs and then stories that, at their
limit, encompass complex narratives, emotions, and literature. Without extra
effort, these sequences-of-bytes cannot be effectively used to extract
insights or create meaning. Indeed, most natural language processing (NLP)
systems, including large language models (LLMs), do not take as input data in
its native sequence-of-bytes format. Instead, raw characters are first broken
up into individual tokens, which are typically individual words, but can also
be sub-word chunks, that are gleaned from the sequence-of-bytes. These tokens
are what is actually used to train NLP systems, So when an NLP system
translates text or answers a question, it is actually generating sequences-of-
tokens, not individual characters or bytes.

Contrary to the token-by-token view used by most NLP systems, real-world
entities and objects are often represented by multi-token sub-sequences (i.e.
names) where a span of one or more tokens represents a single entity. The NLP
task that aims to identify pertinent sub-sequences and provide them a label or
link them to some external structured knowledge is called information
extraction. Because the surface text representing an entity is often comprised
of more than one token, these tokens are commonly referred to as a span.

Take, for example, the entity Tim Cook, the CEO of Apple Inc., who frequently
appears in news articles and social media posts. If we could map the surface
form (i.e. the sequence-of-tokens) of Tim Cook to some structured knowledge
listing in, say, Wikipedia (wiki/Tim_Cook), then all of the related
information in that knowledge base could be leveraged to inform other
downstream tasks, like mitigating citation hallucination in LLMs [63],
Wikification [29], question answering [80], and an overall better
understanding of the text as illustrated in Fig. 1. But, all of these
potential advances are first predicated on how we represent spans within the
NLP system.

Figure 1: An overview of information extraction tasks. The goal of information
extraction is to identify sub-sequences within unstructured or semi-structured
text information and link them to certain class-labels, entities in a
knowledge base, or other items within some structured database. This
structured information plays a central role in many downstream applications
such as question answering and recommender systems.

Unlike the tokenization task and the related tasks of parsing, chunking,
lemmatization, and stemming, each having an enormous literature surrounding
them, the representation of spans in information extraction tasks is
relatively unprincipled. Mostly by coincidence, it appears that a handful of
span-oriented representations have emerged in the information extraction
literature. Despite their centrality in many NLP systems, these mostly ad hoc
methodologies have not yet been cataloged; yet they constitute a critical task
in NLP.

Most information extraction systems either (1) find relevant sub-sequences and
then use them to predict a label or (2) have a label and then attempts to find
an appropriate sub-sequence that represents that label. But these one-way
transformations can be problematic because they do not consider the duality
that exists between sub-sequences and their labels. This is one reason why,
for example, zero-shot entity linking remains difficult: the one-way entity
linking transformation that links known-entities to their surface forms cannot
be used if the entity has never been seen [83, 48].

In contrast, we view a span as simultaneously possessing (1) a sub-sequence of
tokens representing the entity (i.e. a surface form) and (2) a meaning/label.
This is natural because, when reading, humans simultaneously derive meaning
from text while also using that meaning to understand the surface forms that
encode those concepts. With this view, many challenges in information
extraction could be re-considered with respect to this duality. The rise of
LLMs and prompt tuning also fit nicely into this paradigm because the emergent
ability of LLMs (and their faux reasoning ability) could be further empowered
if the generated tokens could consider the duality inherent in their spans.

By re-orienting the disparate representations used across the enormous variety
of tasks in NLP around a unified definition of a span, we show that various
information extraction tasks are actually not that different after all. We
call this perspective Span-Oriented Information Extraction. As the name
implies, the goal of span-oriented information extraction is to (1) extract
spans from sequences-of-tokens and (2) assign each span to a well-defined
entity. Traditional information extraction systems typically ignore the first
step and focus exclusively on the second, but, as we shall see, the way in
which spans are identified and represented has critical implications on the
overall system.

###  1.1 A Brief History of Spans in Information Extraction

The use of spans in information extraction dates back to the early days of
Computational Linguistics and NLP [19]. In its early forms, spans were the
basic units used in parsing and part of speech chunking tasks. The Linguistic
String Project (LSP) [60] at New York University was one of the earliest
information extraction research and development projects. LSP began with a
parsing program to obtain the syntactic relations among words in a sentence.
The first implementation of the LSP parser and grammar was introduced in 1967
with two key components: a set of formal rewriting rules, and a set of
restrictions. The applications of the LSP system drew upon a so-called
sublanguage grammar, where documents could be parsed with quasi-grammatical
rules and further mapped into an early incarnation of a span, called an
information format. LSP mainly focused on medical data analysis including
medical test reports, hospital discharge records, and patient diagnosis
history, but its impact can still be seen in contemporary NLP tools.

Beginning in the late 1980s, the Message Understanding Conference (MUC)
advanced the development of information extraction and established this task
as one of the key branches of NLP. The Defense Advanced Research Projects
Agency (DARPA) funded MUC also introduced early evaluation processes for
information extraction systems [17, 31]. Each team attending the MUC
conference was required to submit an information extraction system for the
given task. MUC-1 to MUC-4 focused on template filling of naval military
intelligence and terrorist incidents, with the goal to fill-out pre-identified
spans within text-templates. Evaluation metrics used in information retrieval
such as precision and recall were introduced formally in MUC-3. In the next
iteration, MUC-5 expanded beyond military domains to joint ventures in the
financial sector and in microelectronics [17]. MUC-6 introduced the concept of
a named entity and formalized the task of named entity recognition (NER), with
its corresponding evaluation metrics [31]. The goal of NER, as originally
defined, was to identify the spans within text that correspond to people,
organizations, and geographic locations. With the intervening development of
knowledge bases like Wikipedia [90], information extraction tasks were refined
to not just identify NER-spans, but to link them to their corresponding entry
in the knowledge base. The development of digital encyclopedias was coincident
with MUC-7, which introduced the task of relation extraction and paved the way
for knowledge bases and knowledge graphs as we know them today.

The Automatic Content Extraction (ACE) program began shortly after MUC was
concluded. ACE mainly focused on news and conversational data including
broadcast conversations, newspapers, and conversational telephone speech. It’s
goal was entity recognition and labeling on a large natural language corpus.
In pursuit of that goal, ACE also established the use of spans and now-
standard labels used for named entities and relation types.

Since 2007, NIST has organized the Text Analysis Conferences (TAC), which has
been another driving force in NLP research. Like MUC, TAC had a task
submission and evaluation setup, and then proceedings to discuss submitted
methods and models. Many well-known information extraction tasks, including
entity linking, slot filling, fine-grained entity typing and others have been
conceived and advanced by participating teams working on these challenges and
datasets.

These initial competitions paved the way for the myriad of information extract
tasks, which are used to understand, generate, and reason about text-based
natural language. Despite the variety of information extraction tasks and the
innumerable ways that spans are represented in these tasks, there does not
appear to be a coherent view of the definition and role of a span within the
various tasks in information extraction. We argue in the present work that a
span is a foundational building block which, if carefully constructed and
considered, naturally ties together these tasks.

###  1.2 Formal Definition of a Span

Formally, given a document $d$ containing a sequence of bytes (typically
characters) that are grouped into tokens $d=\langle
t_{1},t_{2},\ldots,t_{k=\ell(d)}\rangle$ where $\ell(d)$ is the length of $d$,
a span is a triple $s=(b,e,c)$ with a beginning token-index $b$, ending token-
index $e$, and class label(s) $c$.

Informally, a span represents a duality comprised of: (1) a surface form (i.e.
a subset of the original text), and (2) one or more corresponding span classes
(i.e. the class label(s)). Thus the goal of the span-oriented information
extraction task is to extract the correct spans (if not given) with correct
boundaries and assign correct class labels. The makeup of the class set $C$
defines two varieties of information extraction: (1) closed-world, which seeks
to assign each span to one or more classes selected from a pre-defined class
set $C$, and (2) open-world, which does not contain a pre-defined class set,
but rather may assign labels from any source. This definition is
operationalized by most popular software packages like Spacy [35] and
Stanford’s CoreNLP [58], which both implement a span as a built-in class.

###  1.3 Road Map

With the span definition in hand, the goal of the present work is to re-
imagine the landscape of the varieties of span-oriented information extraction
tasks along three dimensions: (1) information extraction tasks, (2)
information extraction evaluation, and (3) information extraction models.

##### Information Extraction Tasks

Information extraction tasks appear to vary significantly in their inputs and
outputs. For example, named entity recognition seeks to label named entities
(e.g.erson, place, organization, or other proper nouns) within sentences; and
machine reading comprehension tasks seeks to answer questions based on
information in the background context. These two tasks appear to be quite
different; however, if we re-imagine these tasks from the perspective of the
span, different information extraction tasks are naturally aligned. As we
shall see in Section 2, vastly different information extraction tasks can be
re-imagined so as to generate the same span outputs with only slightly
different inputs.

##### Information Extraction Evaluation

Because of the varieties of information extraction tasks, there is no shortage
of ways to evaluate the performance of various algorithms and models. It is
important to encourage broad evaluation of these tasks; however, a span-
oriented consideration of information extraction tasks permits, as we shall
see in Section 3, a natural comparison of the predictions made by different
tasks.

##### Information Extraction Models

If we indeed re-imagine information extraction tasks as having the same span-
based outputs, and similar evaluation regimes, then, as we shall see in
Section 4, it becomes natural to consider the various algorithms and models as
simple variants of one another.

The present work endeavors to present the wide assortment of information
extraction tasks as variants of the same basic span-prediction algorithm. With
this in mind, it is possible that algorithms, models, architectures, and
intuitions developed in the solution to one task might be used to solve
others.

##  2 Types of Span-Oriented Information Extraction Tasks

There are a variety of information extraction tasks that are defined based
upon the needs of the system and the data available. These tasks have been
given various names and descriptions in the literature, but they all share the
same basic definition of a span.

Given a sentence in some document, for example the sentence, "Apple CEO Tim
Cook sold his Texas house", different varieties of information extraction
would seek to label different sub-sequences having one or more classes, e.g.
Apple as a company, Texas a US state. Among these varieties are: (1) Entity
disambiguation (ED) [34], (2) Entity Linking (EL) [39], (3) Entity Typing (ET)
[36], (4) Named Entity Recognition (NER) [94], (5) Attribute Value Extraction
(AVE) [120], (6) Machine Reading Comprehension (MRC) [80].

Table 1: Overview of Span-Oriented Information Extraction Tasks Apple CEO Tim
Cook sold his Texas house.
---
Task Name | Span Req. | Span Class | Span Class Example
Entity Disambiguation (ED) | ✓ | Entity | wiki/Tim_Cook, wiki/Texas
Entity Linking (EL) |  | Entity |  wiki/Tim_Cook, wiki/Texas
Entity Typing (ET) | ✓ | Fine-grained Type |  Businessman, State
Named Entity Recognition (NER) |  | Coarse-grained Type |  PER, LOC
Attribute Value Extraction (AVE) |  | Attributes |  CEO: Tim Cook
Machine Reading Comprehension (MRC) |  | Reading Question | Tim Cook

Table 1 provides a non-exhaustive list of different information extraction
tasks and a specific example of that task.

###  2.1 Entity Disambiguation

In cases where the surface form is given, either through text matching or some
other entity identification task, what remains is to match the surface form
with the appropriate class label. In the following examples we identify the
surface form with a beginning and end token-index of the input where the first
token is indexed at 0.

TASK: |  Entity Disambiguation
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house
INPUT: |  [$(2,3,?)$, $(6,6,?)$]
OUTPUT: |  [$(0,0,$ wiki/Apple_Inc.$)$, $(2,3,$ wiki/Tim_Cook$)$, $(6,6,$ wiki/Texas$)$]

Entity disambiguation is so named because the task is mostly to determine
which specific entity, if there are many similarly-named entities, that
surface form represents. To do this, ED systems rely on the context to make
their decisions. The difficulty of ED is that entities typically have a giant
number of classes. Subsequently, rare and infrequent entities are difficult to
disambiguate [103, 5, 85, 83].

###  2.2 Entity Linking

In cases where the surface form is not pre-defined, the position of the spans
and the class of the span (i.e. the entity) must both be extracted. Compared
to ED, EL is much more difficult. EL requires jointly identifying non-standard
surface forms from the input text and assigning the correct labels. As a
result ED systems are typically more precise, while EL systems observe
redundant performance drops in the same datasets [12, 39, 83, 95].

TASK: |  Entity Linking
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house
OUTPUT: |  [$(0,0,$wiki/Apple_Inc.$)$, $(1,1,$wiki/Chief_executive_officer$)$,
$(2,3,$ wiki/Tim_Cook$)$, $(6,6,$ wiki/Texas$)$]

The EL task essentially performs the surface form identification sub-task and
the ED task simultaneously. This provides greater freedom to the system so
that additional context of any found-span might be used to find more spans. As
a result, EL systems generally have greater coverage, but at the expense of
precision.

###  2.3 Entity Typing

There are also cases where the end-user does not seek a specific entity-entry
in some knowledge base, but rather seeks to know the specific (i.e. fine-
grained) types of entities that are resident in some span [18, 68].

TASK: |  Entity Typing
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house
INPUT: |  [$(0,0,?)$, $(2,3,?)$, $(6,6,?)$]
OUTPUT: |  [$(0,0,$ Company$)$, $(2,3,$ Businessman$)$, $(6,6,$ State$)$]

The ET task is a slightly relaxed form of the ED task, where the number of
classes is not as large, but can still be extensive depending on the type-
granularity that the user seeks. Like in the ED task, because the beginning
and ending indexes of the spans are provided as input, this task typically has
high precision at the expense of coverage. However, evaluation of the ET task
can be difficult because properly aligning the right span-label that matches
the ground-truth granularity can be difficult. Indeed, previous work has found
that reasonable (and sometimes arguably more-correct span-labels) are often
counted as incorrect in ET evaluation [18].

###  2.4 Named Entity Recognition

One of the first information extraction tasks from the MUC workshops described
above was the NER task. This task seeks to identify entities from a sentence
and, almost as a bonus, also labels the entities into one of three or four
broad types; typically PER, ORG, and LOC representing person-, organization-,
location-entities respectively.

TASK: |  Named Entity Recognition
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house
OUTPUT: |  [$(0,0,$ ORG$)$, $(2,3,$ PER$)$, $(6,6,$ LOC$)$]

The primary difficulty of the NER task is the identification of the span’s
starting and ending indexes; once the surface form can be identified, the
class label is rather straightforward because the class label set is typically
very small and contains only coarse-grained entity types. However, because NER
systems are typically trained on a small set of coarse-grained entity types,
many spans that would be easily discovered with EL systems (e.g. animals,
technology, works of art) are not easily binned into one of the coarse types.

###  2.5 Attribute Value Extraction

In instances where many sentences describe a general set of items, like, for
example, descriptions of episodes on a streaming service, item descriptions in
an online game, or groceries available on an e-commerce website, the entities
themselves are not of interest. Instead, the AVE task seeks to extract the
collective values corresponding to attributes of interest from the descriptive
document(s). Nevertheless, AVE can still be views as a span-oriented
information extraction task. Continuing the running example, if we imagine a
collection of sentences discussing technology news, then the AVE task might
extract the corresponding technology leaders as follows:

TASK: |  Attribute Value Extraction
---|---
INPUT: |  Satya Nadella says that Microsoft products will soon connect to OpenAI.
Apple CEO Tim Cook sold his Texas house.
Jensen Huang, head of NVIDIA, announces the launch of DGX GH200.
OUTPUT: |  [$(0,1,$ CEO$)$, $(13,14,$ CEO$),(19,20,$ CEO$)$]

###  2.6 Machine Reading Comprehension

In some cases users seek to extract spans related to some free-text question.
Although the MRC task is unlike many span-oriented information extraction
tasks, it still requests the same fundamental output: a span of tokens and a
class.

TASK: |  Machine Reading Comprehension
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house.
INPUT: |  Q: Who is the CEO of Apple?
OUTPUT: |  [$(2,3,$ Who is the CEO of Apple?$)$]

Like in the AVE task, the MRC task contains a beginning- and ending-index and
as well as a class. In this particular case, the class is a direct restating
of the input question. This is an important consideration. This may be best
explained with an analogy to the NER task. If we re-consider the NER task to
be an MRC task, then the question asked of the NER system is the entity-label:

TASK: |  Named Entity Recognition
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house
INPUT: |  Q: PER
OUTPUT: |  [$(2,3,$ PER$)$]

Here we see that the answer to the MRC task is not the class label, but is
instead denoted by the span indices which reveal the answer to be Tim Cook,
who is in Person.

The above list of span-oriented information extraction tasks is by no means
exhaustive, but these examples are meant to be representative of our
philosophy: by re-imagining information extraction tasks as systems that
output spans, then these systems can be considered natural analogs of one
another. With this in mind, the means by which these various information
extraction systems are evaluated can be viewed from a more coherent
perspective too.

##  3 Evaluation of Span-Oriented Information Extraction

The most common way to evaluate information extraction systems is to use the
standard precision, recall and $F_{1}$ metrics [86]. However, any metric that
evaluates spans deserve a more-thorough consideration. This is because the
groundtruth span may not exactly line up with the predicted span, yet still be
close-enough to warrant a true-positive judgement. Likewise, the class-
label(s) within the predicted spans might not exactly match the groundtruth
span, yet still be close-enough to warrant leniency. Because of these
consideration several metrics and metric-variants have been developed to
handle these difficult cases.

The $F_{\beta}$ metric was originally intended as a way to balance the
precision and recall (i.e. coverage) of machine learning system. When the
$\beta=1$ then the precision and recall values are evenly weighted in the
$F$-score. Lower values for $\beta$ give more weight to the precision metric
and vice versa. Unless otherwise specified, systems typically set $\beta=1$
yielding the well-known $F_{1}$-score.

The $F_{1}$-score uses precision and recall metrics, which themselves require
some notion of binary true and false predictions. As applied to span-oriented
information extraction, this creates a rigid requirement that any token can
belong to at most one span and that a span must exactly match the groundtruth
to be considered a true-positive instance.

In the simple case where there are only labels (e.g. yes/no, on/off), the
$F_{1}$-score provides a meaningful, although rigid evaluation metric.
However, as noted above, most of the class-sets in information extraction are
enormous, having thousands (or hundreds of thousands) of classes. In such
cases, a decision needs to be made on how to calculate certain mean-averages.
The two most common decisions are called (1) the micro $F_{1}$-score and (2)
the macro $F_{1}$-score.

###  3.1 Micro $F_{1}$-Score

The micro $F_{1}$ score is widely recognized as the standard evaluation metric
for rigid NER. The notion of rigidity in this instance denotes that any token
of the input document can belong to at most one span and that a true-positive
instance must match the groundtruth span exactly.

Formally, for each class $c^{*}\in C$ and for a groundtruth set $s=(b,e,c)\in
S$ and for a set of predicted span instances
$\hat{s}=(\hat{b},\hat{e},\hat{c})\in\hat{S}$, the number of true positives
($\textrm{TP}_{c^{*}}$) is:

$\textrm{TP}_{c^{*}}=\sum_{(b,e,c)\in
S}\sum_{(\hat{b},\hat{e},\hat{c})\in\hat{S}}\mathbbm{1}\left(b=\hat{b}\land
e=\hat{e}\land c=\hat{c}=c^{*}\right)$ (1)

Likewise we count the number of false negatives ($\textrm{FN}_{c^{*}}$) and
false positives $\textrm{FP}_{c^{*}}$ as:

$\textrm{FN}_{c^{*}}=\sum_{(b,e,c)\in
S}\left(1-\sum_{(\hat{b},\hat{e},\hat{c})\in\hat{S}}\mathbbm{1}\left(b=\hat{b}\land
e=\hat{e}\land c=\hat{c}=c^{*}\right)\right)$ (2)

$\textrm{FP}_{c^{*}}=\sum_{(\hat{b},\hat{e},\hat{c})\in\hat{S}}\left(1-\sum_{({b},{e},{c})\in{S}}\mathbbm{1}\left(b=\hat{b}\land
e=\hat{e}\land c=\hat{c}=c^{*}\right)\right)$ (3)

Note that some tasks, like MRC and AVE sometimes relax the indicator function
$\mathbbm{1}(\cdot)$ so that the beginning- and ending-indexes need not
exactly match, but that the tokens denoted by these indexes match:
$\mathbbm{1}(\textrm{substr}(b,e)=\textrm{substr}(\hat{b},\hat{e})\land
c=\hat{c}=c^{*})$. For example, in MRC we do not need to know that Tim Cook
begins and ends with tokens 2 and 3, only that the tokens between indexes 2
and 3 match the groundtruth answer.

Then, to obtain the micro $F_{1}$-scores the TPs, FNs, and FPs are summed
across the various classes and substituted into the standard precision and
recall metrics to obtain micro-precision and micro-recall.

$\textrm{micro-Prec}=\frac{\sum_{c^{*}\in
C}\textrm{TP}_{c^{*}}}{\sum_{c^{*}\in
C}(\textrm{TP}_{c^{*}}+\textrm{FP}_{c^{*}})};\textrm{micro-
Rec}=\frac{\sum_{c^{*}\in C}\textrm{TP}_{c^{*}}}{\sum_{c^{*}\in
C}(\textrm{TP}_{c^{*}}+\textrm{FN}_{c^{*}})}$ (4)

$\textrm{micro-}F_{1}=\frac{2\times\textrm{micro-Prec}\times\textrm{micro-
Rec}}{\textrm{micro-Prec}+\textrm{micro-Rec}}$ (5)

###  3.2 Macro $F_{1}$-Score

The micro $F_{1}$-score is a natural way to sum up the successes and errors of
the model predictions. However, this simple solution can be easily swayed in
the likely case that the class labels are unbalanced, i.e. certain labels
occur much more frequently than others. In this situation, a class-based
precision and recall measurement can be calculated as follows:

$\textrm{Prec}_{c^{*}}=\frac{\textrm{TP}_{c^{*}}}{\textrm{TP}_{c^{*}}+\textrm{FP}_{c^{*}}};\textrm{Rec}_{c^{*}}=\frac{\textrm{TP}_{c^{*}}}{\textrm{TP}_{c^{*}}+\textrm{FN}_{c^{*}}};F_{1,c^{*}}=\frac{2\times\textrm{Prec}_{c^{*}}\times\textrm{Rec}_{c^{*}}}{\textrm{Prec}_{c^{*}}+\textrm{Rec}_{c^{*}}}$
(6)

Then, the overall macro $F_{1}$-score is the arithmetic mean of $F_{1}$ scores
across all the individual classes as follows:

$\textrm{macro-}F_{1}=\frac{\sum_{c^{*}\in C}F_{1,c^{*}}}{|C|}$ (7)

###  3.3 Exact Match Evaluation

The macro and micro metrics described above require exact matches of the
beginning-index, ending-index, and the class $\mathbbm{1}(b=\hat{b}\land
e=\hat{e}\land c=\hat{c}=c^{*})$ in order to count towards a true positive
instance.

Also note that the string matching function
$\mathbbm{1}(\textrm{substr}(b,e)=\textrm{substr}(\hat{b},\hat{e})\land
c=\hat{c}=c^{*})$ commonly used in the AVE and MRC tasks do not require exact
matches of the indices, but do require exact matches of the sub-sequences
represented by the indexes.

TASK: |  NER/MRC
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house
Tim Cook announces new M2 chip.
GT: |  [$(2,3,$ PER$)$]
OUTPUT: |  [$(8,9,$ PER$)$]

In the example directly above the extracted span $(8,9,$ PER$)$ does match
ground truth span in the MRC and AVE task, but would not match the ground
truth (GT) span in NER, EL, ET, and ED tasks.

###  3.4 Relaxed Match Evaluation

Exact matching requirements are often criticised for imposing too strict of a
requirement onto the system. It is often the case that a sub-sequence or
super-sequence of the ground truth span is an equally valid match. Likewise,
in fine-grained ET or ED tasks, a close, but still inexact match between the
predicted class $\hat{c}$ and the ground truth class $c^{*}$ could also be
equally valid (and our experience shows that sometimes the predicted match is
arguably better than the ground truth match) [24]. To allay this criticism,
the use of relaxed (i.e. partial) span matching is also used in evaluation
[39, 85, 31, 27]. Under a relaxed span matching regime, if a predicted span
has the same class as one of the ground truth spans and the span indices
intersect, then that prediction is counted as a true positive.

TASK: |  NER
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house
Tim Cook announces new M2 chip.
GT: |  [$(2,3,$ PER$)$]
OUTPUT: |  [$(3,6,$ PER$)$]

In the example directly above, the extracted span representing the subsequence
Cook sold his Texas would count as a true positive for the ground truth span
representing the sequence Tim Cook. This additional tolerance almost always
results in a positive performance shift; however, partial matching may
sometimes be too lenient—as shown in the example above. Therefore, strict span
matching remains the standard regime used to evaluate span-oriented
information extraction tasks.

##  4 Taxonomy of Information Extraction Features

In this section, we begin to consider how information extraction systems use
natural language to create a model from which information can be extracted. To
that end, we will summarize the features commonly gleaned from (1) tokens, (2)
spans, and (3) span classes.

Because natural language is digitally represented as a sequence of bytes in
its most basic form, we consider that to be the lowest-level representation of
written (digital) language. From that form natural language tokenizers turn
bytes into words (or sub-words) from which sentences, Tweets, paragraphs,
posts, articles, stories, and narratives are formed. Information extraction
tasks typically operate at the token-level; by our definition, these systems
output a span, which is a sequence of one or more words and a class. Each of
these levels: the token, the span, and the class all have information that can
be used in the construction of a natural language model. This section will
briefly highlight each.

###  4.1 Token Features

Most natural language is grouped into tokens—typically words. These tokens are
fundamental element in communication; dictionaries, for example, are one
source of understanding for these tokens, as are encyclopedias and thesauri.
The same is true in natural language processing. Because tokens are the basic
elements, crafting the corresponding token features to be flexible and
generalized is an important consideration for most tasks.

####  4.1.1 Linguistic Token Features

Many token features are linguistic in their nature. For example, part-of-
speech tags are one of the earliest token features used to distinguish word
classes (e.g. noun, verb, adjective, and adverb) [40]. The abstract syntax
tree is another linguistic feature that transforms plain text into a self-
referential tree structure [67]. These approaches produce symbolic properties
and are easily interpreted by human beings. However, linguistic features have
three significant limitations. First, they do not directly provide information
of interest in to most practical applications. For example, although knowing
whether a token is a noun or a verb or modifies some other token can be useful
in downstream tasks, this is not directly useful in many applications. Second,
training linguistic models requires an enormous amount of expert human
annotations [14, 59]. In the decades since linguistic token features were
first proposed, many datasets have been created, but these features constantly
require updating. Third, even perfect labels result in performance limitations
on many IE tasks [14, 105, 106]. This is because shallow, token-based
information represents a limited view of the deeper intent and meaning within
natural language [48, 86].

####  4.1.2 Pre-trained Token Features from Language Models

With the development of language models (LMs), pre-trained word embeddings
have become a primary source of token features. The goal of any LM (large or
small) is to model the probability distribution over sequences of tokens. That
is, given a document $d$ composed of a sequence of tokens $d=\langle
t_{1},t_{2},\ldots,t_{c},\ldots,t_{\ell(d)}\rangle$, an LM provides for the
estimation of the probability distribution of any token $t_{c}$ by utilizing
other contextual words in the sequence as follows:

$p\left(t_{c}|t_{1},t_{2},\ldots,t_{c-2},t_{c-1},t_{c+1},t_{c+2},\ldots,t_{\ell(d)}\right)$
(8)

Constructing LMs has been one of the most fundamental and important tasks for
the NLP community. So called large LMs (LLMs) are able to scale because they
are trained in a self-supervised regime without any human annotation at all.
As a result, LLMs have shown the ability to learn token features from a wide
variety of documents from different domains. Early pre-trained word embeddings
were based on the bag-of-word or skip-gram models; which is best represented
by word2vec [63], GloVe [69] and fasttext [9]. With the development of text
Transformers, self-supervised token features has been widely adopted and even
supplanted most alternatives in natural language processing. The two most
representative projects in this category are (1) the GPT series having a left-
to-right language model [74, 75, 11] and (2) the BERT series with masked
language modeling (MLM) [21]. The broad pre-training with self-supervised
labels that these LLMs undergo is typically sufficient for most tokens to
obtain robust features [63, 21]. Therefore these pre-trained LLMs can also be
adapted to tackle information extraction tasks. This is also important because
information extraction tasks typically have limited and biased training data
[94, 27], and it is difficult and even inapplicable to employ self-supervised
training to the information extraction tasks [21, 11, 55].

####  4.1.3 Character Features

Our definition of a span uses tokens as the base type. However, in some cases
the tokens are unable to provide a useful properties of characters. In these
cases, it may be beneficial to consider individual characters as extra
supplemental features. These character features can then be used as extra
learning parameters to improve performance in some specific instances
including Chinese language modeling [62, 92], and in fields that commonly use
acronyms and initialisms like chemistry [26, 100], biology [45], and law [13].

####  4.1.4 Token Sequences

Because natural language is (digitally) expressed in a sequence of bytes or
tokens, there has been a large effort to model these sequences [48, 83].
Arguably the most well known model for token sequences is the transformers
[96, 21], although other architectures like the recurrent neural network (RNN)
[33, 50], convolutional neural network (CNN) [44, 91], and the point network
[115, 97] have been used as well. Alternatives to these neural models tend to
use probability graphic models to model token dependencies with
implementations such as the hidden Markov model (HMM) [65], the maximum
entropy Markov model (MeMMs) [61] and the conditional random field (CRF) [41].

###  4.2 Span Features

####  4.2.1 Span Embeddings

A span defined as sequence of one or more tokens may therefore have a variety
of representations. Typically, span embeddings are built on top of token
features. For example, Chen et al. directly used the word embedding of the
first token to represent the span features [15], and Tan et al. used the
concatenation of word embeddings of the first and last tokens [93]. Otherwise,
single-pass frameworks use an averaging over a pool of token embeddings to
form span embeddings [4, 5]. The PURE model further learns a length embedding
as part of span embeddings [121]. The W2NER model uses a convolutional layer
and an LSTM jointly to fuse token embeddings into span embeddings [47].

In addition to formulating span embeddings based on token embeddings, another
approach is to create span embeddings from scratch as a different pre-training
task. Early work on this trajectory extends the word2vec idea to learn span
embeddings from contextual span-token and span-span contextual correlations
[107, 105]. Later work in this area extended the mask language model of BERT
to generate span embeddings as a co-training task within language modeling
[10, 106, 16, 108].

####  4.2.2 Span Sequences

Just as token features can be modelled as token sequences, span embeddings can
likewise be modelling as a sequence of spans. Understanding span sequences is
critical in many information extraction tasks. For example, in entity
disambiguation one of the primary features used to select span candidates is
the context-tokens and other neighbor spans that surround the span in the same
sentence. For example, the tokens Apple and CEO in the running example in
Section 2 could help to disambiguate Tim Cook as wiki/Tim_Cook from some
another entry with a similar name. Likewise, the identification of
wiki/Tim_Cook may aid in the recognition that Apple refers to wiki/Apple_Inc.
and not some other entity. Formally, most previous frameworks consider the
training objective of a common classification model [30, 39, 70, 71] adapted
in Eq.(9):

$g(\textbf{s, c})=\sum_{i=1}^{n}\Phi(s_{i},c_{i})+\sum_{i<j}\Psi(c_{i},c_{j})$
(9)

where the contextual span scores $\Phi(s_{i},c_{i})$ for each span $s_{i}$ and
class $s_{i}$ and the span-span correlation scores $\Psi(c_{i},c_{j})$ for the
predicted classes $c_{i}$ and $c_{j}$ are both used to train the model.
Modelling span sequences tends to work well when the number of classes is
small. In this scenario, adjacent and correlated tokens are typically
sufficient to determine the span class. However, when the number of classes is
large like in the fine-grained ET task and the EL task, contextual words may
not be enough to determine the corresponding span classes[30, 39, 78, 77].

###  4.3 Span Class Representations

Most information extraction models produce spans where the class element is a
simple id or perhaps a pointer (e.g. PER, businessman, wiki/Texas). However,
many span classes contain metadata such as text descriptions and even
relationship information (e.g. in the case of Wikipedia). These span classes
can be used to add additional context to possibly improve performance. Yet
another option is to use pre-trained methods to obtain representations for
each span’s class. For example, the TagMe system used hyperlinks among pages
to learn a class representation [29]; likewise, wikipedia2vec [105] and deep-
ed [30] extended word2vec to learn token and class correlations. In a similar
way, ERNIE [117] and LUKE [106] extended the BERT masking language model to
entity disambiguation in order to obtain better token and span class
representations.

##  5 Model Transformations among Information Extractors

Having previously identified the various information extraction tasks and
their features, the next piece of the puzzle is to describe how spans are
transformed by different information extraction models. This section presents
a different, yet unifying, perspective on information extraction by
considering the various transformations that a span undergoes for different
information extraction tasks.

In their attempt to tackle different information extraction tasks, different
information extraction models employ various transformations to the spans.
These transformations can be grouped into six types of transformations:
(5.1.1) sequential labeling, (5.1.2) token prototyping, (5.1.3) token-pair
transformation, (5.1.4) span classification, (5.1.5) span locating, and
(5.1.6) span generation.

Table 2: Matrix of Information Extraction Tasks by their Transformation Type Transformation |  NER |  ED |  EL |  ET |  AVE |  MRC
---|---|---|---|---|---|---
Sequential Labeling |  [37, 56, 21, 106] |  |  [95, 15, 5, 39] |  |  [110, 120, 104] |  [22, 89, 38]
Token Prototype |  [36] |  [10, 108] |  [10, 108, 30, 105, 106] |  [57] |  [110, 113] |
Token-pair Classification |  [47] |  |  |  |  |
Span Classification |  [114, 3, 52, 93, 88, 121] |  [103, 1, 105, 30, 78, 77, 15] |  [39, 95, 16] |  [18, 20, 66] |  [25] |  [6, 64, 112]
Span Locating |  [51, 87] |  [7, 32] |  [116] |  |  [98, 25] |  [118, 82]
Span Generation |  [55, 28, 109, 102] |  [12] |  [12] |  [23] |  |  [73, 99, 46]

As we shall see, unifying these different tasks reveals the importance of the
various transformations. For example, the sequential labelling transformation
commonly used in the NER task appears to be vastly different than the two-step
transformation used in the EL task. However, as we shall see, because these
tasks share the same input and output, these transformations do naturally
generalize to other transformations and are actually swappable. Despite their
interchangeability, the taxonomy of different transformations (see Tab. 2 for
details) does come with trade offs for different tasks. For example, the
sequential labelling transformation decomposes span-labels into token-labels
where each token assigned a label. This transformation ignores token locality
features, which could be important in accurately finding span boundaries, but
is nevertheless fast and easy to train. In this section we describe different
model transformations and briefly discuss their trade offs.

###  5.1 Transformation Approaches

####  5.1.1 Sequential labeling

Sequential labeling (i.e. token classification) is the most traditional and
common transformation used in NER [37] as well as other information extraction
tasks [120, 110, 5]. The core idea of sequential labelling is to directly
transform spans into token-wise classes labeled with the Begin, Inside,
Outside, End (BIOE) schema, where each label represents a token that begins,
is inside of, ends, or is outside of some span. This schema has been expanded
to also include other labels, like Left and Right (L/R), to represent tokens
to the left and right of a span [48].

TASK: | NER + Sequential Labelling |  |  |
---|---|---|---|---
INDEX: | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7
INPUT: | Apple | CEO | Tim | Cook | sold | his | Texas | house
TRANSFORM: | B-ORG | O | B-PER | E-PER | O | O | B-LOC | O
OUTPUT: |  [$(0,0,$ ORG$)$, $(2,3,$ PER$)$, $(6,6,$ LOC$)$] |  |  |

Continuing the example above, the sequential labelling transformation uses the
B and E labels to identify the beginning and end of the span encompassing Tim
Cook one token at a time. Those tokens that are outside of the span are
labeled with O. Then, during inference, token classes are first labeled and
then span labels are obtained by concatenating one or more continuous tokens
belonging to the same class. Note that the I and E labels can be missing in a
span when a span has only one or two tokens.

####  5.1.2 Token Prototyping

Token prototyping considers each span to be a sequence of tokens, which is
then mapped to the same class [110, 113, 36]. Unlike sequential labelling,
which labels tokens one at a time, token prototypes (e.g. PER, Businessman)
are computed with locality and clustering based objectives. During inference,
the tokens classes are represented as prototypes and embeddings are computed
for each token. Then tokens that are close in the embedding space are
clustered together and the corresponding classes are obtained by selecting the
closest prototype.

TASK: | NER + Token Prototyping |  |  |
---|---|---|---|---
INDEX: | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7
INPUT: | Apple | CEO | Tim | Cook | sold | his | Texas | house
TRANSFORM: | ORG | O | PER | PER | O | O | LOC | O
OUTPUT: |  [$(0,0,$ ORG$)$, $(2,3,$ PER$)$, $(6,6,$ LOC$)$] |  |  |

Because this is a token-oriented approached, the tokens Tim and Cook, from the
example above, are both individually assigned the PER label. Of course, this
provides some ambiguity: it is unclear whether Tim Cook is a single person or
two persons Tim and Cook. Typically, a post-processing step combines labels of
the same type into a single multi-token span, but this isn’t always desirable.

####  5.1.3 Token-pair Classification

In token-pair classification, and as the name suggests, span labels are
transformed into relationships between two tokens. During inference, each
token-pair is labeled as one of an assortment of classes that describes the
relationship between the two words within the same span.

TASK: | NER + Token-Pair Classification |  |  |  |
---|---|---|---|---|---
INDEX |  |  0 |  1 |  2 |  3 |  4 |  5 |  6 |  7
| INPUT: |  Apple |  CEO |  Tim |  Cook |  sold |  his |  Texas |  house
0 |  Apple | NNW |  |  |  |  |  |  |
1 |  CEO |  |  |  |  |  |  |  |
2 |  Tim |  |  |  | NNW |  |  |  |
3 |  Cook |  |  | THW |  |  |  |  |
4 |  sold |  |  |  |  |  |  |  |
5 |  his |  |  |  |  |  |  |  |
6 |  Texas |  |  |  |  |  |  | NNW |
7 |  house |  |  |  |  |  |  |  |
OUTPUT: |  [$(0,0,$ ORG$)$, $(2,3,$ PER$)$, $(6,6,$ LOC$)$] |  |  |  |

For example, the W2NER model uses labels Next-Neighboring-Word (NNW) to
describe the relationship between pairs of words within a single span [47].
Applying this model to the running example produces the transformation above
where blue cells represent the window size permitted by the model. Here the
token-pair Tim–Cook is labeled with an NNW class describing Cook as the next-
neighboring-word of Tim.

Although this transformation produces a sparse matrix with $\ell^{2}$ possible
labels, it does permit non-contiguous dependency references to be labels
produced in certain circumstances. Typically these additional labels are
labeled as a Tail-Head-Word (THW) in the bottom-diagonal. The above
illustration shows an example THW that, depending on the task, might be
expanded THW-PER to indicate that the span refers to a person.

####  5.1.4 Span Classification

The core idea of the span classification transformation has two parts: (1)
span candidate generation and (2) span label assignment. There are many ways
to generate span candidates. The simplest is to enumerate all the possible
spans up to certain window-lengths (i.e. n-grams) [88, 121]. The window-length
is normally a constant which is typically less than 5 in most tasks. Another
way to generate spans is to learn a specific span generation model. For
example, many of the token-oriented transformation approaches can generate
span candidates with high-probability boundaries [119, 93]. In another vein,
the Ask-and-Verify model uses a machine reading comprehension module to
generate span candidates by finding potential boundaries from tokens with high
predicted probabilities [25]. Span candidates can also be generated utilizing
external data and models. For example, most existing entity disambiguation
methods utilize rule based methods such as string match and frequency
statistics [30, 39, 95, 43, 16, 4, 5]. Another popular way is to use retrieval
models like TF-IDF [3], BM25 [54], phrase-mining [84, 114], or dense retrieval
[103] among many others.

After span candidates are generated, the goal of the following span label
assignment step is to find a mapping function to select spans from the most
promising span candidates and provide a class label. One common method is to
employ a span-oriented classifier to distinguish positive span candidates from
negative span candidates by screening all the span candidates [88, 119, 93,
25]. One particularly compelling model, Locate-and-Label, which was inspired
by two-stage object detection methods in computer vision also considers
partially overlapped span candidates as positive samples as long as the
intersection over union (IoU) is larger than a certain threshold value [87].

TASK: | NER + Span Classification |  |  |  |
---|---|---|---|---|---
INDEX |  |  0 |  1 |  2 |  3 |  4 |  5 |  6 |  7
| INPUT: |  Apple |  CEO |  Tim |  Cook |  sold |  his |  Texas |  house
0 |  Apple | ORG |  |  |  |  |  |  |
1 |  CEO |  |  |  |  |  |  |  |
2 |  Tim |  |  |  | PER |  |  |  |
3 |  Cook |  |  |  |  |  |  |  |
4 |  sold |  |  |  |  |  |  |  |
5 |  his |  |  |  |  |  |  |  |
6 |  Texas |  |  |  |  |  |  | LOC |
7 |  house |  |  |  |  |  |  |  |
OUTPUT: |  [$(0,0,$ ORG$)$, $(2,3,$ PER$)$, $(6,6,$ LOC$)$] |  |  |  |

Continuing the running example, we utilize a upper triangular matrix with blue
color to represent valid span candidates and the positive span is labeled with
corresponding classes.

####  5.1.5 Span Locating

Another transformation approach is called span locating. The goal of this
transformation is to consider an input sentence and relevant classes as a
context-query pair and then find the corresponding span boundaries within the
original input sentence [51]. The pipeline of this transformation is similar
to machine reading comprehension (MRC), which is used to find answers in the
context to the corresponding questions [80, 79]. Unlike in the span
classification transformation where span candidates are generated, selected
and labeled, in span locating the decision making process happens in reverse
order: first the class label is determined and then the proper span holding
that label is found.

TASK: | NER + Span Locating |  |  |
---|---|---|---|---
INPUT: |  PER, LOC |  |  |
INDEX: |  0 | 1 | 2 | 3 | 4 | 5 |  6 | 7
INPUT: |  Apple | CEO | Tim | Cook | sold | his |  Texas | house
|  $\uparrow\uparrow$ |  | $\uparrow$ | $\uparrow$ |  |  |  $\uparrow\uparrow$ |
TRANSFORM: |  Start & End |  | Start | End |  |  |  Start & End |
OUTPUT: |  [$(0,0,$ ORG$)$, $(2,3,$ PER$)$, $(6,6,$ LOC$)$] |  |  |

Given a class label there are two typical ways to identify the span(s). First,
given a input sentence with $\ell$ tokens, one way is to employ two
$\ell$-class classifiers to predict the span boundaries (i.e. the start and
end tokens) [25]. The second way is to employ two binary classifiers, one to
predict whether each token is a starting token or not, and the other to
determine whether each token is the end token or not, with the obvious
restriction that the beginning token must precede the end token [51]. The span
locating transformation illustrated in the example shows that first the PER
and LOC classes are identified either via a input or another model. Then the
transformation seeks to identify the boundary tokens that begin and end the
span that represents the class labels within the sentence.

####  5.1.6 Span Generation

Text generation models have become popular especially with the rise in LLMs.
These generation models provide another possibility in span-oriented
transformations. The core idea of span generation is to transform the original
token sequence into an expanded token sequence with span-tokens [46, 76],
similar to machine translation and other natural language generation tasks. In
order to represent span labels in the token sequence, span generation
typically inserts distinct characteristics to indicate span labels including
both span positions and span classes [12, 55, 28].

TASK: | NER + Span Generation |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
INDEX: |  0 |  1 |  2 |  3 |  4 |  5 |  6 |  7 |  8 |  9 |  10 |  11 |  12 |  13
INPUT: |  Apple |  CEO |  Tim |  Cook |  sold |  his |  Texas |  house |  |  |  |  |  |
TRANSFORM: |  ORG-L |  Apple |  ORG-R |  CEO |  PER-L |  Tim |  Cook |  PER-R |  sold |  his |  LOC-L |  Texas |  LOC-R |  house
OUTPUT: |  [$(0,0,$ ORG$)$, $(2,3,$ PER$)$, $(6,6,$ LOC$)$] |  |  |  |  |  |  |

Continuing the running example illustrated above, span generation might
transform the input token sequence into a token sequence having spans
represented by special span tokens like [PER-L] indicating the start of a
person span or LOC-R indicating the end of a location. Span generation models
are commonly designed as an autoregressive token generation task [12, 55, 28].
They take a token sequence and a generated token as input, and predict the
subsequent token. These predictions essentially represent classifications from
a predefined dictionary. During inference, the generation process continues
iteratively until all the desired spans are generated.

The span generation approach is widely utilized in many information extraction
tasks. For example, GENRE employs an auto-regressive model to transform the
entity disambiguation and entity linking tasks into a joint text/entity-name
generation task [12]. For the entity disambiguation task, target entities are
selected with a conditional generation method based on the provided token
sequence. As for the entity linking task, the span indices and their labels
are together transformed into an augmentation of the original sentence.
Specific to the NER task, the BartNER model transforms the token sequence into
uniform index pointers [109]. And more recently, Universal Information
Extraction (UIE) architectures have also been developed to transform different
information extraction tasks (e.g. NER, EL, ED) into the same format through
generative language modeling [55, 28]. UIE systems can extract shared features
and joint correlations from training labels of different information
extraction tasks. Furthermore, different structural signals across different
information extraction tasks can be encoded into similar text allowing
efficient and effective knowledge transfer from pre-trained models [12, 5].

###  5.2 Transformations Trade Offs

Table 3: Trade-offs of Transformation Approaches | Complexity | Class Distr. | Features | Special Cases |
---|---|---|---|---|---
Transformation | # of inst. | +/- | Token | Span | Span-Class | Nested | Discontinuous |
Sequential Labeling | $O(1)$ | 4/4 | ✓ |  |  |  |  |
Token Prototype | $O(1)$ | 4/4 | ✓ |  |  |  |  |
Token-pair Classification | $O(1)$ | 4/60 | ✓ |  |  | ✓ | ✓ |
Span Classification | $O(1)$ | 3/27 |  | ✓ | ✓ | ✓ |  |
Span Locating | $O(K)$ | 6/0 | ✓ |  | ✓ | ✓ |  |
Span Generation | $O(\ell)$ | 6/8 | ✓ |  | ✓ |  |  |

As we alluded to in the previous section, different transformations have
distinct trade-offs. Following Tab. 3, we present these considerations along
four dimensions: (1) computational complexity, (2) the number of positive and
negative class labels, (3) the features considered, and (4) applicability to
nested and discontinuous spans. It is important to note that our discussion
focuses on the general setup of these transformations without considering any
specific design modifications. Although some models and methods may have
specific designs tailored to address certain special cases, it is not our
intention to delve into specific design strategies in the following
discussion.

####  5.2.1 Complexity

Given an input document with $\ell$ tokens and $K$ different target span
classes, different transformation approaches have different computational
complexities. The main difference in complexity is the number of actual
instances (i.e. # of instances in Tab. 3). This count directly corresponds to
the number of times the feed-forward process needs to be executed in order to
generate an inference output. Most transformations require a single input,
namely, the token sequence, and makes several span predictions. The span
locating transformation needs to consider each provided span class as an
individual instance and is therefore in $O(K)$. Like machine translation, the
span generation transformation considers each token in the input sequence as
an individual instance yielding $O(\ell)$.

####  5.2.2 Positive and Negative Span Distributions

Different transformation approaches produce a different number of targets,
i.e. positive and negative spans, and therefore the choice of transformation
has a significant impact on the label distribution and, as a result, the
performance metrics.

Again consider the running example illustrated above, which includes two named
entities. In the sequence labelling transformation, four tokens including
Apple, Tim, Cook, and Texas are positive instances representing span tokens;
the other four tokens are negative instances representing non-span tokens. In
token-pair classification, there are $\ell^{2}=64$ token pairs in total; of
which two are positive samples. In span classification, with a window-length
of five, there are a total of $30$ possible spans with length less or equal to
$5$ from which only three represent positive spans. Finally, the span
generation transformation produces three positive spans using six special
tokens to annotate positive spans. Therefore the six special tokens are the
positive instances and the original eight tokens are considered negative
tokens. Clearly, different transformation approaches yield substantially
different class distributions, which by definition has a large impact on
performance metrics.

Noisy or incomplete labels are also differently impacted by the choice of
transformation. Understanding these differences is important because many
information extraction datasets have noisy training labels of $5\%$ or more
[101, 122] even for the well-known CONLL03 NER dataset [94]. In an interesting
empirical study on the missing labels for information extraction tasks, Li et
al. considered as a special case where noise is only present in negative
samples [52]. They show that, during training, ignoring positive examples has
small impact, but incorrectly labelling positive spans as being negative
samples has significant impact on the final results. The same idea also
applies for different transformations with different positive and negative
span distributions.

####  5.2.3 Features

Different transformations use various models to produce spans, resulting in
different abilities to encode different types of features. We focus on three
main types of features: (1) token embeddings, (2) span embeddings, and (3) the
span-class representation.

Token-wise transformations such as sequential labeling, token prototype, and
token-pair classification decompose span labels into token-wise classes,
making token features easy to encode but precluding the consideration of span
features. In span classification, the embeddings of span candidates can be
obtained and then span classes can be assigned to the entire span. In
contrast, span locating encodes the span class along with the original
sentence as input, and results are obtained by locating the span boundaries
using the span indices, meaning that the span embedding cannot be considered.
Likewise, span generation cannot use span embeddings, but instead represents
span classes as a sequence of tokens to generate.

####  5.2.4 Nested and Discontinuous Spans

Overlapping spans is an important complication for training, inferences, and
evaluation. This occurs when a single token is made to belong to two or more
different spans. Different transformations handle these cases in different
ways. We categorize them into two distinct cases: nested and discontinuous.

Nested spans are situations where a token can belong to multiple spans
simultaneously. Transformations which allow for nested spans include token-
pair classification, span classification, and span locating. Token-pair
considers nested cases into different start-end token pairs; span
classification considers all the possible span candidates, which includes
overlapping spans; and span locating has no restriction on where the beginning
and ending indices of different classes may be placed. In contrast, sequential
labeling and token prototype transformations decompose span labels into token-
wise labels; as a result, a token can not belong to two classes
simultaneously. Likewise, span generation requires the injection of special
tokens to indicate span classes, and therefore can not provide nested spans
either.

Discontinuous cases, on the other hand, involve the formation of spans using
non-adjacent tokens. In these cases, tokens that are not contiguous in the
text can be grouped together to form a single span. Because the token-pair
matrix provides the flexibility to link non-contiguous tokens, it is the only
method that can effectively handle discontinuous cases.

###  5.3 Training Strategies

After a span transformation approach is chosen, the systems needs to be
properly employed to be effective. In other words, different training
strategies have to be considered for a working framework. From our
perspective, we categorize the various standard training strategies into four
broad classes: (5.3.1) Feature tuning, which selects and engineers the most
relevant features for a particular information extraction task; (5.3.2) Model
tuning, which refers to the process of optimizing the parameters of a machine
learning model; (5.3.3) Prompt tuning, which is the relatively new task of
fine-tuning the prompts fed to LLMs to achieve more accurate results; and
(5.3.4) In-Context learning, which involved training models in specific
context information, for example, on specific datasets or niche tasks.

####  5.3.1 Feature Tuning

One of the outputs of pre-trained (large) language models are informative and
well-trained embeddings. These embeddings almost always represent a
span—sometimes as short as a word and sometimes as long as a whole sentence or
paragraph—and are effective features that can be used for information
extraction tasks. Prior to the rise of LLMs, most features used in information
extraction tasks came from linguistic cues such as part-of-speech (POS) tags
[81], word stems and lemmas [8], and syntactic parsers [72]; as well as
statistical learning approaches like word frequency counts [2], word co-
occurrence analysis [63], and semantic analysis [42]. Although these methods
have their own strengths and weaknesses, they are generally less effective
than LLM-based embeddings in capturing the complex relationships between words
in a language[63, 21, 11, 46, 76]. Feature turning, therefore, refers to the
numerous strategies that have been developed to learn ever-more creative and
interesting features for spans. Equipped with these pre-trained features,
spans can be clustered or classified or labeled to solve any number of
information extraction tasks.

####  5.3.2 Model Tuning

Sometimes, the pre-trained features from an LLM do not align well with the
task that is being asked of the system. This misalignment will degrade the
system’s performance. In these cases it is common for LLMs to undergo a fine-
tuning process, which adapts the pre-trained model parameters, including span
features, to the specific task. Previous studies have shown that fine-tuning
the model outperforms feature engineering with similar settings [21, 46], but
can be prone to catastrophic forgetting [53] and other maladies; see the
survey by Li et al. [49] for details. One major problem is that model tuning
requires that the model be loaded into memory and trained, which, for even
medium-sized LLMs, is a non-trivial task.

####  5.3.3 Prompt Tuning

The rise of ChatGPT and other proprietary LLMs has spawned an entirely new
kind of NLP task called prompt-tuning. In this case, instead of using or
training span embeddings, prompt-tuning is the task of devising clever ways to
query the LLM. The advantages of prompt tuning are clear. Because there is no
need to extract, tune, or train any model or features, it is relatively easy
to use the system. The task instead becomes finding the best prompts to feed
to the LLM so that it returns the answers you seek. Another often overlooked
advantage of prompt tuning is that by simply querying the system, it does not
change. As a result any prompts or other rules that are learned can be
maintained.

TASK: |  NER
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house.
Prompt INPUT: |  Apple CEO Tim Cook sold his Texas house. Tim Cook is a [MASK]
Prompt Output: |  [MASK] $\rightarrow$ technology executive
OUTPUT: |  $[(2,3,$ PER$)$]

There are two types of prompt tuning: hard prompt tuning and soft prompt
tuning. In hard prompt tuning, a handcrafted prompt is used to glean results
from the system. Conversely, in soft prompt tuning, the prompt itself can be
trained. This means that an additional NLP model is trained to predict an
adaptable prompt based on some input and labels. During the inference process,
the prompt model initially generates a prompt, which is then concatenated with
a test sample and fed into the LLM to obtain the final prediction; i.e. soft
prompt tuning is a model generating input to feed to another model.

####  5.3.4 In-context Learning

Finally, In-context learning uses language models directly without any extra
training process. This is accomplished by also injecting a few training
examples along with corresponding labels into the prompts. The idea is that
language models are able to see the mapping function between example input
data and their corresponding labels, and then they can subsequently infer that
same correspondence on unseen input data for label prediction.

An example of in-context learning on the NER task might resemble something
like this:
---

TASK: |  NER
---|---
INPUT: |  Apple CEO Tim Cook sold his Texas house.
Prompt INPUT: |  Satya Nadella says that Microsoft products will soon connect to OpenAI.
|  Satya Nadella is a PER
|  Apple CEO Tim Cook sold his Texas house. Tim Cook is a [MASK]
Prompt Output: |  [MASK] is a PER
OUTPUT: |  [$(2,3,$ PER$)$]

With in-context learning a system can achieve robust capabilities with little
cost. Another instance of this kind of learning is the chain-of-source
approach, wherein a question and its corresponding answer are broken down into
a series of sub-problems[53]. By addressing these sub-questions in a
sequential manner, the system is able to arrive at more-comprehensive and
nuanced solutions.

##  6 Discussion

Over the past millennia text has been made by humans for humans. The recent
and broad digitization of human-generated text has served to propel AI systems
and tasks like information extraction. When humans are tasked to perform
information extraction, they do so—with relative ease—by first understanding
the concepts and definitions of the labels in their context, even on unseen
classes or in unknown languages. In contrast, we often find that AI systems on
few-shot and zero-shot scenarios still perform much worse than humans [23,
77]. This gap in performance is due to the AI system’s inability to reason
about the relationships between the context of the input and the context of
the class label. Ongoing work is in this area has aims to properly encode
these contexts. For example, previous work in encoding entity descriptions as
search query targets has shown some ability to retrieve relevant entity
candidates [103, 54]. However, these relationships are nuanced; this research
gap has not been fully explored and a wide gap remains.

Most modern information extraction systems are built atop token-based pre-
trained models. They work by tying together token features to identify spans.
However, as pre-trained LLMs like GPT-4 and beyond grow ever stronger, they
also become less accessible for specific use in AI tasks like information
extraction. Therefore, we expect the use of in-context learning and further
development of context-based prompts will increase. Ongoing work in retrieve-
to-augment and generate-to-augment strategies have potential to enhance
performance in many information extraction tasks as they have already been
successful in open-domain question answering (ODQA) tasks [111].

In summary, the goal of this survey is to challenge the reader to reconsider
information extraction to be the task of finding spans in text that represent
some class-label. By recasting the myriad traditional information extraction
tasks into this new orientation, they can all be viewed as analogs of one
another. Having a unified view of this critical task is important as AI
undergoes a shift due to the advent of LLMs and other groundbreaking AI
models. To that end, we have summarized the various information extraction
tasks, their evaluations, and their models, and we have endeavored to provide
guidance how when and how to use each. We conclude with a challenge to
integrate information extraction into LLMs so as to provide these sometimes
rambling and hallucinating AI systems a bit of context and evidence; and,
conversely, to better use the machinery of these LLMs to help tackle the
information extraction task directly.

## Acknowledgements

This project was funded by DARPA under contracts HR001121C0168 and
HR00112290106.

## References

  * [1] D. Agarwal, R. Angell, N. Monath, and A. McCallum.  Entity linking via explicit mention-mention coreference modeling.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4644–4658, Seattle, United States, July 2022. Association for Computational Linguistics.
  * [2] A. Aizawa.  An information-theoretic perspective of tf–idf measures.  Information Processing & Management, 39(1):45–65, 2003.
  * [3] R. Angell, N. Monath, S. Mohan, N. Yadav, and A. McCallum.  Clustering-based inference for biomedical entity linking.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2598–2608, Online, June 2021. Association for Computational Linguistics.
  * [4] T. Ayoola, J. Fisher, and A. Pierleoni.  Improving entity disambiguation by reasoning over a knowledge base.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2899–2912, Seattle, United States, July 2022. Association for Computational Linguistics.
  * [5] T. Ayoola, S. Tyagi, J. Fisher, C. Christodoulopoulos, and A. Pierleoni.  ReFinED: An efficient zero-shot-capable approach to end-to-end entity linking.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, pages 209–220, Hybrid: Seattle, Washington + Online, July 2022. Association for Computational Linguistics.
  * [6] S. Back, S. C. Chinthakindi, A. Kedia, H. Lee, and J. Choo.  Neurquri: Neural question requirement inspector for answerability prediction in machine reading comprehension.  In International Conference on Learning Representations, 2020.
  * [7] E. Barba, L. Procopio, and R. Navigli.  ExtEnD: Extractive entity disambiguation.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2478–2488, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [8] S. Bird and E. Loper.  NLTK: The natural language toolkit.  In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  * [9] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov.  Enriching Word Vectors with Subword Information.  Transactions of the Association for Computational Linguistics, 5:135–146, 06 2017.
  * [10] S. Broscheit.  Investigating entity knowledge in BERT with simple neural end-to-end entity linking.  In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 677–685, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.
  * [11] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [12] N. D. Cao, G. Izacard, S. Riedel, and F. Petroni.  Autoregressive entity retrieval.  In International Conference on Learning Representations, 2021.
  * [13] I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos.  LEGAL-BERT: The muppets straight out of law school.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904, Online, Nov. 2020. Association for Computational Linguistics.
  * [14] D. Chen and C. Manning.  A fast and accurate dependency parser using neural networks.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar, Oct. 2014\. Association for Computational Linguistics.
  * [15] H. Chen, X. Li, A. Zukov Gregoric, and S. Wadhwa.  Contextualized end-to-end neural entity linking.  In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 637–642, Suzhou, China, Dec. 2020. Association for Computational Linguistics.
  * [16] S. Chen, J. Wang, F. Jiang, and C.-Y. Lin.  Improving entity linking by modeling latent entity type information.  volume 34, pages 7529–7537, Apr. 2020.
  * [17] N. Chinchor and B. Sundheim.  MUC-5 evaluation metrics.  In Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, Maryland, August 25-27, 1993, 1993.
  * [18] E. Choi, O. Levy, Y. Choi, and L. Zettlemoyer.  Ultra-fine entity typing.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87–96, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [19] J. Cowie and W. Lehnert.  Information extraction.  Communications of the ACM, 39(1):80–91, 1996.
  * [20] L. Del Corro, A. Abujabal, R. Gemulla, and G. Weikum.  FINET: Context-aware fine-grained named entity typing.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 868–878, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics.
  * [21] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [22] B. Dhingra, H. Liu, Z. Yang, W. Cohen, and R. Salakhutdinov.  Gated-attention readers for text comprehension.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1832–1846, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  * [23] N. Ding, Y. Chen, X. Han, G. Xu, P. Xie, H.-T. Zheng, Z. Liu, J. Li, and H.-G. Kim.  Prompt-learning for fine-grained entity typing.  arXiv preprint arXiv:2108.10604, 2021.
  * [24] Y. Ding, N. Botzer, and T. Weninger.  Posthoc verification and the fallibility of the ground truth.  In Proceedings of the First Workshop on Dynamic Adversarial Data Collection, pages 23–29, Seattle, WA, July 2022. Association for Computational Linguistics.
  * [25] Y. Ding, Y. Liang, N. Zalmout, X. Li, C. Grant, and T. Weninger.  Ask-and-verify: Span candidate generation and verification for attribute value extraction.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 110–110, Abu Dhabi, UAE, Dec. 2022. Association for Computational Linguistics.
  * [26] Y. Ding, D. Wang, T. Weninger, and M. Jiang.  Preserving composition and crystal structures of chemical compounds in atomic embedding.  In 2019 IEEE International Conference on Big Data (Big Data), pages 6037–6039, 2019.
  * [27] G. Doddington, A. Mitchell, M. Przybocki, L. Ramshaw, S. Strassel, and R. Weischedel.  The automatic content extraction (ACE) program – tasks, data, and evaluation.  In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal, May 2004. European Language Resources Association (ELRA).
  * [28] H. Fei, S. Wu, J. Li, B. Li, F. Li, L. Qin, M. Zhang, M. Zhang, and T.-S. Chua.  LasUIE: Unifying information extraction with latent adaptive structure-aware generative language model.  In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [29] P. Ferragina and U. Scaiella.  Tagme: On-the-fly annotation of short text fragments (by wikipedia entities).  In Proceedings of the 19th ACM International Conference on Information and Knowledge Management, CIKM ’10, page 1625–1628, New York, NY, USA, 2010. Association for Computing Machinery.
  * [30] O.-E. Ganea and T. Hofmann.  Deep joint entity disambiguation with local neural attention.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619–2629, Copenhagen, Denmark, Sept. 2017\. Association for Computational Linguistics.
  * [31] R. Grishman and B. Sundheim.  Message Understanding Conference- 6: A brief history.  In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, 1996.
  * [32] Y. Gu, X. Qu, Z. Wang, B. Huai, N. J. Yuan, and X. Gui.  Read, retrospect, select: An mrc framework to short text entity linking.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12920–12928, 2021.
  * [33] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [34] J. Hoffart, M. A. Yosef, I. Bordino, H. Fürstenau, M. Pinkal, M. Spaniol, B. Taneva, S. Thater, and G. Weikum.  Robust disambiguation of named entities in text.  In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 782–792, Edinburgh, Scotland, UK., July 2011\. Association for Computational Linguistics.
  * [35] M. Honnibal and I. Montani.  spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing.  To appear, 7(1):411–420, 2017.
  * [36] J. Huang, C. Li, K. Subudhi, D. Jose, S. Balakrishnan, W. Chen, B. Peng, J. Gao, and J. Han.  Few-shot named entity recognition: An empirical baseline study.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10408–10423, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
  * [37] Z. Huang, W. Xu, and K. Yu.  Bidirectional lstm-crf models for sequence tagging.  arXiv:1508.01991, 2015.
  * [38] R. Kadlec, M. Schmid, O. Bajgar, and J. Kleindienst.  Text understanding with the attention sum reader network.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 908–918, Berlin, Germany, Aug. 2016. Association for Computational Linguistics.
  * [39] N. Kolitsas, O.-E. Ganea, and T. Hofmann.  End-to-end neural entity linking.  In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 519–529, Brussels, Belgium, Oct. 2018. Association for Computational Linguistics.
  * [40] P. R. Kroeger.  Analyzing grammar: An introduction.  Cambridge University Press, 2005.
  * [41] J. D. Lafferty, A. McCallum, and F. C. N. Pereira.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, page 282–289, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc.
  * [42] T. K. Landauer, P. W. Foltz, and D. Laham.  An introduction to latent semantic analysis.  Discourse processes, 25(2-3):259–284, 1998.
  * [43] P. Le and I. Titov.  Improving entity linking by modeling latent relations between mentions.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1595–1604, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [44] Y. LeCun, Y. Bengio, et al.  Convolutional networks for images, speech, and time series.  The handbook of brain theory and neural networks, 3361(10):1995, 1995.
  * [45] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics, 36(4):1234–1240, 2020.
  * [46] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online, July 2020. Association for Computational Linguistics.
  * [47] J. Li, H. Fei, J. Liu, S. Wu, M. Zhang, C. Teng, D. Ji, and F. Li.  Unified named entity recognition as word-word relation classification.  Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):10965–10973, Jun. 2022.
  * [48] J. Li, A. Sun, J. Han, and C. Li.  A survey on deep learning for named entity recognition.  IEEE Transactions on Knowledge and Data Engineering, 34(1):50–70, 2022.
  * [49] J. Li, T. Tang, W. X. Zhao, and J.-R. Wen.  Pretrained language models for text generation: A survey.  arXiv preprint arXiv:2105.10311, 2021.
  * [50] P.-H. Li, R.-P. Dong, Y.-S. Wang, J.-C. Chou, and W.-Y. Ma.  Leveraging linguistic structures for named entity recognition with bidirectional recursive neural networks.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2664–2669, Copenhagen, Denmark, Sept. 2017\. Association for Computational Linguistics.
  * [51] X. Li, J. Feng, Y. Meng, Q. Han, F. Wu, and J. Li.  A unified MRC framework for named entity recognition.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849–5859, Online, July 2020. Association for Computational Linguistics.
  * [52] Y. Li, lemao liu, and S. Shi.  Empirical analysis of unlabeled entity problem in named entity recognition.  In International Conference on Learning Representations, 2021.
  * [53] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  ACM Computing Surveys, 55(9):1–35, 2023.
  * [54] L. Logeswaran, M.-W. Chang, K. Lee, K. Toutanova, J. Devlin, and H. Lee.  Zero-shot entity linking by reading entity descriptions.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3449–3460, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [55] Y. Lu, Q. Liu, D. Dai, X. Xiao, H. Lin, X. Han, L. Sun, and H. Wu.  Unified structure generation for universal information extraction.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5755–5772, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [56] X. Ma and E. Hovy.  End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064–1074, Berlin, Germany, Aug. 2016. Association for Computational Linguistics.
  * [57] Y. Ma, E. Cambria, and S. Gao.  Label embedding for zero-shot fine-grained named entity typing.  In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 171–180, Osaka, Japan, Dec. 2016. The COLING 2016 Organizing Committee.
  * [58] C. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. Bethard, and D. McClosky.  The Stanford CoreNLP natural language processing toolkit.  In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland, June 2014. Association for Computational Linguistics.
  * [59] M. Marcus, B. Santorini, and M. A. Marcinkiewicz.  Building a large annotated corpus of english: The penn treebank.  1993\.
  * [60] E. Marsh and C. Friedman.  Transporting the linguistic string project system from a medical to a navy domain.  ACM Transactions on Information Systems (TOIS), 3(2):121–140, 1985\.
  * [61] A. McCallum, D. Freitag, and F. C. Pereira.  Maximum entropy markov models for information extraction and segmentation.  In Proceedings of the Eighteenth International Conference on Machine Learning, volume 17, pages 591–598, 2000.
  * [62] Y. Meng, W. Wu, F. Wang, X. Li, P. Nie, F. Yin, M. Li, Q. Han, X. Sun, and J. Li.  Glyce: Glyph-vectors for chinese character representations.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [63] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean.  Distributed representations of words and phrases and their compositionality.  In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
  * [64] S. Min, V. Zhong, L. Zettlemoyer, and H. Hajishirzi.  Multi-hop reading comprehension through question decomposition and rescoring.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097–6109, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [65] S. Morwal, N. Jahan, and D. Chopra.  Named entity recognition using hidden markov model (hmm).  International Journal on Natural Language Computing (IJNLC) Vol, 1, 2012.
  * [66] S. Murty, P. Verga, L. Vilnis, I. Radovanovic, and A. McCallum.  Hierarchical losses and new resources for fine-grained entity typing and linking.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 97–109, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [67] I. Neamtiu, J. S. Foster, and M. Hicks.  Understanding source code evolution using abstract syntax tree matching.  In Proceedings of the 2005 international workshop on Mining software repositories, pages 1–5, 2005.
  * [68] Y. Onoe and G. Durrett.  Fine-grained entity typing for domain independent entity linking.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8576–8583, 2020.
  * [69] J. Pennington, R. Socher, and C. Manning.  GloVe: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, Oct. 2014\. Association for Computational Linguistics.
  * [70] M. C. Phan, A. Sun, Y. Tay, J. Han, and C. Li.  Neupl: Attention-based semantic matching and pair-linking for entity disambiguation.  In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, page 1667–1676, New York, NY, USA, 2017. Association for Computing Machinery.
  * [71] M. C. Phan, A. Sun, Y. Tay, J. Han, and C. Li.  Pair-linking for collective entity disambiguation: Two could be better than all.  IEEE Transactions on Knowledge and Data Engineering, 31(7):1383–1396, 2018.
  * [72] M. J. Pickering and R. P. Van Gompel.  Syntactic parsing.  In Handbook of psycholinguistics, pages 455–503. Elsevier, 2006\.
  * [73] W. Qi, Y. Yan, Y. Gong, D. Liu, N. Duan, J. Chen, R. Zhang, and M. Zhou.  ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2401–2410, Online, Nov. 2020. Association for Computational Linguistics.
  * [74] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [75] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [76] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21:1–67, 2020.
  * [77] J. Raiman.  Deeptype 2: Superhuman entity linking, all you need is type interactions.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8028–8035, 2022.
  * [78] J. Raiman and O. Raiman.  Deeptype: multilingual entity linking by neural type system evolution.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [79] P. Rajpurkar, R. Jia, and P. Liang.  Know what you don’t know: Unanswerable questions for SQuAD.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [80] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang.  SQuAD: 100,000+ questions for machine comprehension of text.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, Nov. 2016. Association for Computational Linguistics.
  * [81] J. Rijkhoff.  Word classes.  Language and Linguistics Compass, 1(6):709–726, 2007.
  * [82] M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi.  Bidirectional attention flow for machine comprehension.  In International Conference on Learning Representations, 2017.
  * [83] Ö. Sevgili, A. Shelmanov, M. Arkhipov, A. Panchenko, and C. Biemann.  Neural entity linking: A survey of models based on deep learning.  Semant. Web, 13(3):527–570, jan 2022.
  * [84] J. Shang, J. Liu, M. Jiang, X. Ren, C. R. Voss, and J. Han.  Automated phrase mining from massive text corpora.  IEEE Transactions on Knowledge and Data Engineering, 30(10):1825–1837, 2018.
  * [85] W. Shen, Y. Li, Y. Liu, J. Han, J. Wang, and X. Yuan.  Entity linking meets deep learning: Techniques and solutions.  IEEE Transactions on Knowledge and Data Engineering, pages 1–1, 2021.
  * [86] W. Shen, J. Wang, and J. Han.  Entity linking with a knowledge base: Issues, techniques, and solutions.  IEEE Transactions on Knowledge and Data Engineering, 27(2):443–460, 2014.
  * [87] Y. Shen, X. Ma, Z. Tan, S. Zhang, W. Wang, and W. Lu.  Locate and label: A two-stage identifier for nested named entity recognition.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2782–2794, Online, Aug. 2021. Association for Computational Linguistics.
  * [88] M. G. Sohrab and M. Miwa.  Deep exhaustive model for nested named entity recognition.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2843–2849, Brussels, Belgium, Oct.-Nov. 2018\. Association for Computational Linguistics.
  * [89] A. Sordoni, P. Bachman, A. Trischler, and Y. Bengio.  Iterative alternating neural attention for machine reading.  arXiv preprint arXiv:1606.02245, 2016.
  * [90] H. C. Staff.  Wikipedia: Complete guide — history, products, founding, and more, 2022\.
  * [91] E. Strubell, P. Verga, D. Belanger, and A. McCallum.  Fast and accurate entity recognition with iterated dilated convolutions.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2670–2680, Copenhagen, Denmark, Sept. 2017\. Association for Computational Linguistics.
  * [92] Z. Sun, X. Li, X. Sun, Y. Meng, X. Ao, Q. He, F. Wu, and J. Li.  ChineseBERT: Chinese pretraining enhanced by glyph and Pinyin information.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2065–2075, Online, Aug. 2021. Association for Computational Linguistics.
  * [93] C. Tan, W. Qiu, M. Chen, R. Wang, and F. Huang.  Boundary enhanced neural span classification for nested named entity recognition.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9016–9023, Apr. 2020.
  * [94] E. F. Tjong Kim Sang and F. De Meulder.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147, 2003.
  * [95] J. M. Van Hulst, F. Hasibi, K. Dercksen, K. Balog, and A. P. de Vries.  Rel: An entity linker standing on the shoulders of giants.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2197–2200, 2020.
  * [96] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [97] O. Vinyals, M. Fortunato, and N. Jaitly.  Pointer networks.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [98] Q. Wang, L. Yang, B. Kanagal, S. Sanghai, D. Sivakumar, B. Shu, Z. Yu, and J. Elsas.  Learning to extract attribute value from product via question answering: A multi-task approach.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 47–55, New York, NY, USA, 2020\. Association for Computing Machinery.
  * [99] S. Wang and J. Jiang.  Machine comprehension using match-lstm and answer pointer.  2016\.
  * [100] X. Wang, V. Hu, X. Song, S. Garg, J. Xiao, and J. Han.  ChemNER: Fine-grained chemistry named entity recognition with ontology-guided distant supervision.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5227–5240, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
  * [101] Z. Wang, J. Shang, L. Liu, L. Lu, J. Liu, and J. Han.  CrossWeigh: Training named entity tagger from imperfect annotations.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5154–5163, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.
  * [102] Z. Wang, K. Zhao, Z. Wang, and J. Shang.  Formulating few-shot fine-tuning towards language model pre-training: A pilot study on named entity recognition.  arXiv preprint arXiv:2205.11799, 2022.
  * [103] L. Wu, F. Petroni, M. Josifoski, S. Riedel, and L. Zettlemoyer.  Scalable zero-shot entity linking with dense entity retrieval.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397–6407, Online, Nov. 2020. Association for Computational Linguistics.
  * [104] H. Xu, W. Wang, X. Mao, X. Jiang, and M. Lan.  Scaling up open tagging from tens to thousands: Comprehension empowered attribute value extraction from product title.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5214–5223, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [105] I. Yamada, A. Asai, J. Sakuma, H. Shindo, H. Takeda, Y. Takefuji, and Y. Matsumoto.  Wikipedia2Vec: An efficient toolkit for learning and visualizing the embeddings of words and entities from Wikipedia.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 23–30, Online, Oct. 2020. Association for Computational Linguistics.
  * [106] I. Yamada, A. Asai, H. Shindo, H. Takeda, and Y. Matsumoto.  LUKE: Deep contextualized entity representations with entity-aware self-attention.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442–6454, Online, nov 2020. Association for Computational Linguistics.
  * [107] I. Yamada, H. Shindo, H. Takeda, and Y. Takefuji.  Joint learning of the embedding of words and entities for named entity disambiguation.  In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 250–259, Berlin, Germany, Aug. 2016. Association for Computational Linguistics.
  * [108] I. Yamada, K. Washio, H. Shindo, and Y. Matsumoto.  Global entity disambiguation with BERT.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3264–3271, Seattle, United States, July 2022. Association for Computational Linguistics.
  * [109] H. Yan, T. Gui, J. Dai, Q. Guo, Z. Zhang, and X. Qiu.  A unified generative framework for various NER subtasks.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5808–5822, Online, Aug. 2021. Association for Computational Linguistics.
  * [110] J. Yan, N. Zalmout, Y. Liang, C. Grant, X. Ren, and X. L. Dong.  AdaTag: Multi-attribute value extraction from product profiles with adaptive decoding.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4694–4705, Online, Aug. 2021. Association for Computational Linguistics.
  * [111] W. Yu, D. Iter, S. Wang, Y. Xu, M. Ju, S. Sanyal, C. Zhu, M. Zeng, and M. Jiang.  Generate rather than retrieve: Large language models are strong context generators.  arXiv preprint arXiv:2209.10063, 2022.
  * [112] Y. Yu, W. Zhang, K. Hasan, M. Yu, B. Xiang, and B. Zhou.  End-to-end answer chunk extraction and ranking for reading comprehension.  arXiv preprint arXiv:1610.09996, 2016.
  * [113] N. Zalmout and X. Li.  Prototype-representations for training data filtering in weakly-supervised information extraction.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 467–474, Abu Dhabi, UAE, Dec. 2022. Association for Computational Linguistics.
  * [114] Q. Zeng, W. Yu, M. Yu, T. Jiang, T. Weninger, and M. Jiang.  Tri-train: Automatic pre-fine tuning between pre-training and fine-tuning for SciNER.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4778–4787, Online, Nov. 2020. Association for Computational Linguistics.
  * [115] F. Zhai, S. Potdar, B. Xiang, and B. Zhou.  Neural models for sequence chunking.  Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017.
  * [116] W. Zhang, W. Hua, and K. Stratos.  EntQA: Entity linking as question answering.  In International Conference on Learning Representations, 2022.
  * [117] Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu.  ERNIE: Enhanced language representation with informative entities.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441–1451, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [118] Z. Zhang, J. Yang, and H. Zhao.  Retrospective reader for machine reading comprehension.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14506–14514, May 2021.
  * [119] C. Zheng, Y. Cai, J. Xu, H.-f. Leung, and G. Xu.  A boundary-aware neural model for nested named entity recognition.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 357–366, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.
  * [120] G. Zheng, S. Mukherjee, X. L. Dong, and F. Li.  Opentag: Open attribute value extraction from product profiles.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 1049–1058, New York, NY, USA, 2018\. Association for Computing Machinery.
  * [121] Z. Zhong and D. Chen.  A frustratingly easy approach for entity and relation extraction.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 50–61, Online, June 2021. Association for Computational Linguistics.
  * [122] W. Zhou and M. Chen.  Learning from noisy labels for entity-centric information extraction.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5381–5392, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.