# Benchmarking Large Language Models with Augmented Instructions for Fine-
grained Information Extraction

Jun Gao1 Huan Zhao2 Yice Zhang1 Wei Wang3 Changlong Yu4 Ruifeng Xu1
1Harbin Institute of Technology (Shenzhen) 24Paradigm. Inc.
3Tsinghua University 4HKUST, Hong Kong, China
<EMAIL_ADDRESS>Work done when Jun Gao was interning at 4Paradigm.

###### Abstract

Information Extraction (IE) is an essential task in Natural Language
Processing. Traditional methods have relied on coarse-grained extraction with
simple instructions. However, with the emergence of Large Language Models
(LLMs), there is a need to adapt IE techniques to leverage the capabilities of
these models. This paper introduces a fine-grained IE benchmark dataset
tailored for LLMs, employing augmented instructions for each information type,
which includes task descriptions, extraction rules, output formats, and
examples. Through extensive evaluations, we observe that encoder-decoder
models, particularly T5 and FLAN-T5, perform well in generalizing to unseen
information types, while ChatGPT exhibits greater adaptability to new task
forms. Our results also indicate that performance is not solely dictated by
model scale, and highlight the significance of architecture, data diversity,
and learning techniques. This work paves the way for a more refined and
versatile utilization of LLMs in Information Extraction.

##  1 Introduction

In the field of Natural Language Processing (NLP), Information Extraction (IE)
is a pivotal task that aims to identify and extract valuable information from
unstructured text. This task encompasses several sub-tasks, including entity
extraction (Yan et al., 2021; Wang et al., 2021) and event extraction (Lin et
al., 2020), which play a crucial role in industries such as finance,
healthcare, and law, facilitating machines in processing large-scale text.

Figure 1: A comparison of the traditional Coarse-Grained IE Instruction (Lu et
al., 2022; Wang et al., 2023) with our proposed Fine-Grained IE Instruction,
using entity extraction as an illustrative example.

Traditional IE methods predominantly depend on supervised learning (Lin et
al., 2020; Du and Cardie, 2020; Lu et al., 2021), which requires vast labeled
datasets for model training. The labeling process can be both time-consuming
and expensive, thus creating barriers to the adoption of IE technologies. In
contrast, the advent of Large Language Models (LLMs) such as GPT-3 (Brown et
al., 2020) has provided an alternative approach. These LLMs demonstrate
promising in-context learning capabilities, which could potentially alleviate
the need for substantial labeled datasets. This is a remarkable stride
forward, as it represents an opportunity to make IE technologies more
accessible and efficient.

Despite the potential benefits of LLMs’ in-context learning in IE, existing
literature has limitations in evaluating these models’ efficacy in IE.
Previous studies tend to focus on evaluating performance across a single
structure (i.e., either using decoder-only (Li et al., 2023b; Gao et al.,
2023b; Ma et al., 2023; Li et al., 2023a) or encoder-encoder (Lu et al., 2022;
Liu et al., 2023; Wang et al., 2023) models) and on unseen information types
(Lu et al., 2022; Wang et al., 2023). However, there is a dearth of research
addressing the generalization performance on different kinds of extraction
tasks, such as moving from event extraction to entity extraction. Moreover,
the current approach (Lu et al., 2022; Liu et al., 2023; Wang et al., 2023; Li
et al., 2023b) in previous work has involved coarse-grained IE tasks, where a
simple instruction without detailed extraction guidelines is used to extract
multiple information types. This neglects vital aspects such as extraction
rules, output format descriptions, and illustrative examples, which are
crucial for adapting to different information types and tasks.

In this paper, we address these shortcomings by introducing a fine-grained IE
benchmark dataset with augmented instructions. The motivation behind
transitioning from coarse-grained to fine-grained IE stems from the
observation that incorporating detailed extraction guidelines for each
information type within the original instructions would cause the instructions
to vastly exceed the input length limitations of the model. Fine-grained IE
differs from coarse-grained IE in that it treats each information type as a
distinct task. Specifically, instead of using a single instruction to extract
multiple information types, fine-grained IE employs augmented instructions for
each information type, including task descriptions, extraction rules, output
formats, and examples. This is depicted in Figure 1, which visually contrasts
the traditional Coarse-Grained IE with our proposed Fine-Grained IE employing
augmented instructions.

A key objective of this study is to stringently evaluate large language
models’ capabilities in in-context learning for fine-grained IE tasks. We
focus on assessing the models’ generalization to novel information types and
task forms, utilizing a diverse dataset. We evaluate an array of models,
including both encoder-decoder and decoder-only architectures, enabling a
thorough analysis of their impact on performance. Our evaluation encompasses
two critical dimensions of generalization: (1) Generalization Across Unseen
Information Types: Models are trained on the same task form but tested on a
different information type. (2) Generalization Across Unseen Task Forms:
Models are trained on a partial form of the task and are tested on an entirely
different form of the task.

Summary of Insights: The experiment unveils several key insights into the
generalization performance of large language models (LLMs) in IE tasks.
Encoder-decoder architectures, notably T5 and FLAN-T5, excel in generalizing
to unseen information types due to their prowess in capturing input-output
relationships. However, they falter in adapting to novel task forms,
highlighting a trade-off between specialization and flexibility. ChatGPT, with
its decoder-only architecture and in-context learning, demonstrates remarkable
adaptability to unfamiliar task structures. Instruction components play a
significant role, where Extraction Rule and Demonstration Examples emerge as
critical for guiding LLMs effectively, whereas Task Description and Output
Format hold variable importance across models. Additionally, the experiment
reveals that performance scaling is non-linear with respect to both training
data quantity and model size, emphasizing the importance of data diversity and
judicious balancing of model scale.

##  2 Evaluating In-context Learning in Information Extraction

In this work, we aim to rigorously evaluate the ability of large language
models to perform in-context Learning for fine-grained IE tasks, with a
particular emphasis on assessing their generalization capabilities to unseen
information types and task forms.

#### Generalization Over Unseen Information Types.

In this scenario, models are presumed to have been trained on a diverse set of
information types within a particular task structure. They are subsequently
evaluated based on their ability to adapt and perform accurately when
confronted with novel information types within the same task structure.
Formally, let us represent the set of information types that the model is
exposed to during training as $I=\\{i_{1},i_{2},i_{3},...,i_{n}\\}$. When the
model is presented with a novel information type $i_{u}\notin I$, we evaluate
its capacity to extrapolate its learned knowledge to this new type. For an
input text $X$ that contains instances of the new information type $i_{u}$,
the model’s task is to extract these instances. We represent this as
$Y=G(X|i_{u})$, where $G$ is the function that the model has learned for IE,
and $Y$ is the set of extracted information instances.

#### Generalization Over Unseen Task Forms.

While models have traditionally been constrained to tasks that closely
resemble the structure they were trained on, the emergence of large language
models (LLMs) introduces the potential for more adaptable models capable of
understanding and adjusting to new task forms. To formalize this, let’s denote
the set of task structures the model is trained on as
$T=\\{t_{1},t_{2},t_{3},...,t_{n}\\}$. Upon encountering a new task form
$t_{u}\notin T$, we assess the model’s capability to apply its existing
knowledge base to effectively execute the task defined by $t_{u}$. For an
input text $X$, the model is expected to produce an output $Y$ that aligns
with the requirements of the new task form. This can be mathematically
represented as $Y=F(X|t_{u})$, where $F$ represents the function that the
model has internalized to map inputs to outputs across different task
structures.

##  3 Augmented Instructions for Fine-grained Information Extraction

To achieve a more comprehensive assessment, our focus is on discerning how
well these models understand and apply extraction rules and demonstration
examples. Given the importance of fine-grained analysis for unearthing
specific strengths and weaknesses of LLMs, our dataset considers each type of
information as an independent task, requiring meticulous attention to detail.
Specifically, our dataset encompasses an extensive spectrum of information
types, including persons, locations, diverse event types, among others. Each
of these information types corresponds to a distinct extraction task, such as
extracting names of persons or identifying various events described in a text.

#### Augmented Instruction Schema.

What sets our fine-grained instructions apart from prior approaches (Lu et
al., 2022; Li et al., 2023b) is the inclusion of more granular information for
each type of information. Instead of just having a task description and output
options, our augmented instruction schema integrates extraction rules,
specifies output formats, and provides illustrative examples. These additional
components are instrumental in equipping the model with an in-depth
understanding of the extraction tasks and in standardizing the output for
further processing. The instruction schema is composed of the following
elements:

  * •

Task Description: A succinct, overarching summary of the task, articulating
the primary objective without delving into particulars.

  * •

Extraction Rule: Comprehensive and unambiguous guidelines, formulated in
natural language, that outline the specifics of extracting the requisite
information from the input text.

  * •

Output Format: Defines the structural and organizational requirements for the
extracted information, offering a systematic template for the model’s output.
This facilitates uniformity in the presentation of results, which is essential
for efficient handling and use of the extracted data.

  * •

Demonstration Examples: Representative input-output pairs that exemplify the
correct application of the extraction rules across varied input texts. These
examples serve to resolve any potential ambiguities and provide practical
demonstrations to reinforce the model’s understanding of the task.

Figure 2: Data division for generalization to Unseen Information Types and
Unseen Task Forms. For a detailed view of the data splits, please refer to
Figure 9 in the Appendix. Figure 3: Data Statistics.

#### Diverse Information Extraction Tasks.

Building a comprehensive dataset for IE from the ground up can be both
resource-intensive and time-consuming. To optimize resource utilization while
still achieving a broad coverage, we have amalgamated a selection of pre-
existing datasets pertinent to IE. This amalgamation comprises 5 datasets,
encompassing three core facets of IE - entity extraction, event extraction,
and sentiment analysis. A visual representation of the data distribution is
depicted in Figure 3. Details of the data construction can be found in
Appendix B.

##  4 Benchmarking LLMs with Fine-grained IE Tasks

###  4.1 Experimental Setup

We assess the generalization capabilities of IE models across different
facets, namely: Generalization to Unseen Information Types, and Generalization
to Unseen Task Forms. Figure 2 shows the dataset partitioning across these
dimensions.

#### Generalization to Unseen Information Types.

In this scenario, the models are trained on a restricted set of information
types and are evaluated on previously unseen information types. The training
dataset includes 4 out of 7 entity types, 23 out of 33 event types, and all 3
sentiment information types. For evaluation, we randomly sampled 100 examples
for each of the 3 entity types to be tested. Since the number of available
samples for each of the event types to be tested was fewer than 100, we
utilized the entire dataset for those event types. In total, the test set
comprises 700 cases.

#### Generalization to Unseen Task Forms.

Here, we evaluate the model’s capability to generalize across different forms
of IE tasks. Unlike the first setup, where the task form remains the same but
the information types differ, here we change the task form itself. The
training set encompasses all event extraction tasks and two of the three
sentiment IE tasks, namely ATE and UABSA. The test set includes 100 randomly
sampled examples for each of the 7 entity types to be tested and 1,000
randomly sampled examples for the ASTE task, summing up to 1,700 test samples.
ASTE extracts aspect, sentiment polarity, and an additional opinion element,
making it a higher-order task compared to ATE and UABSA.

For each training sample, we supplement it with 5 randomly sampled examples
from the training set, sharing the same type as the demonstration examples.
Notably, different training examples are paired with distinct demonstration
examples. For the test samples, we include 5 randomly selected demonstration
examples in their instructions, ensuring that these demonstration examples are
exclusive from the test samples. These demonstration examples remain constant
across all test samples. For a detailed view of the data splits, please refer
to Figure 9 in the Appendix.

###  4.2 Models and Evaluation Metrics

We conducted a comparison between two categories of large language models that
are built on different architectures. In the encoder-decoder category, we
considered models such as T5 (Raffel et al., 2019) and FLAN-T5(Chung et al.,
2022), both of which are available in sizes of 3 billion (3B) and 11 billion
(11B) parameters. In contrast, in the decoder-only category, we looked at
models like LLaMa (Touvron et al., ) and BLOOM (Scao et al., 2022), in
addition to ChatGPT 111https://chat.openai.com/. LLaMa offers models with 7
billion (7B) and 13 billion (13B) parameters, while BLOOM provides models with
3 billion (3B) and 7.1 billion (7.1B) parameters. Note that the results for
ChatGPT were based on testing performed on June 20, 2023. With the exception
of ChatGPT, which was able to utilize our instructions directly for in-context
learning, the remaining models underwent fine-tuning on our training dataset
with fine-grained instructions before being subjected to in-context learning.
Implementation details can be found in Appendix A.

The performance of all models in this task is evaluated using the F1-score as
the metric for assessing the accuracy of the information extracted.

##  5 Experimental Results

###  5.1 Overall Results

|  | Unseen Information Type | Unseen Task Form |
---|---|---|---|---
Structure | Model | Entity | Trigger | Argument | Entity | ASTE | AVG
Enc-Dec | T5 3B | $82.45$ | $84.80$ | $50.30$ | $0.57$ | $0.08$ | $43.64$
T5 11B | $78.70$ | $79.06$ | $53.41$ | $24.50$ | $0.06$ | $47.15$
FLAN-T5 3B | $74.67$ | $84.84$ | $58.02$ | $19.33$ | $0.00$ | $47.37$
FLAN-T5 11B | $74.87$ | $79.00$ | $50.70$ | $10.97$ | $0.00$ | $43.11$
Dec-only | LLaMA 7B | $46.77$ | $55.54$ | $29.55$ | $2.95$ | $0.00$ | $26.96$
LLaMA 13B | $38.07$ | $59.88$ | $32.51$ | $16.85$ | $0.00$ | $29.46$
BLOOM 3B | $20.76$ | $19.65$ | $10.82$ | $14.53$ | $0.00$ | $13.15$
BLOOM 7.1B | $20.90$ | $34.78$ | $20.15$ | $15.00$ | $0.00$ | $18.17$
| ChatGPT* | $64.25$ | $71.17$ | $34.40$ | $55.33$ | $46.04$ | $54.24$
Table 1: Comparison of Large Language Models’ Performance in Generalizing to
Unseen Information Types and Task Forms. We include the average F1 scores for
each model, computed across all tasks. *: ChatGPT was tested using direct in-
context learning and was not trained on our dataset.

#### Analysis of Generalization to Unseen Information Types.

In the generalization to unseen information types, Table 1 demonstrates that
models with an encoder-decoder architecture tend to outperform those with a
decoder-only structure. Specifically, the T5 models with 3B and 11B parameters
achieved F1 scores of 82.45 and 78.70 respectively in the entity extraction
task. These scores significantly surpass the highest F1 score (64.25) achieved
by ChatGPT in the decoder-only category. For trigger and argument extraction,
T5 and FLAN-T5 models consistently perform well, with FLAN-T5 3B achieving the
highest F1 score (58.02) in argument extraction among all models.

It is noteworthy that ChatGPT, which utilizes in-context learning and wasn’t
trained on our dataset, demonstrates respectable performance, particularly in
entity and trigger extraction. This suggests that pre-trained models with
large-scale knowledge can exhibit reasonable generalization even without
specific fine-tuning.

#### Analysis of Generalization to Unseen Task Forms.

As for the generalization to unseen task forms, the performance of most models
substantially declines. Notably, ChatGPT attains significantly better results
compared to others in this category. With an F1 score of 55.33 in entity
extraction and 46.04 in the ASTE task, ChatGPT exhibits the ability to adapt
more efficiently to unfamiliar task forms. On the contrary, encoder-decoder
models, which performed well in generalization to unseen information types,
struggle considerably, with the T5 11B model obtaining the highest F1 score
among them in entity extraction (24.50), but almost negligible performance in
the ASTE task.

###  5.2 In-depth Discussion

#### Effectiveness of Encoder-Decoder Models in Information Types.

The encoder-decoder models, particularly T5 and FLAN-T5, display commendable
proficiency in generalizing to unseen information types. This can be
attributed to the ability of encoder-decoder models to effectively capture the
relationships between inputs and outputs, which is crucial for IE tasks.
Furthermore, the availability of an encoder component might contribute to
better representation learning, which aids in generalization.

#### Limited Generalization to New Task Forms.

Despite the superior performance in information type generalization, encoder-
decoder models exhibit restricted generalization capabilities when subjected
to unfamiliar task forms. This might be due to the high specialization of
these models to the training task forms, which in turn hampers their ability
to adapt to new structures. ChatGPT, however, with its in-context learning,
appears more flexible and can reasonably adapt to new task forms. This
highlights the importance of model adaptability and flexibility in real-world
applications where task forms might not always be consistent.

#### Performance is Not Always Proportional to Scale.

The results also indicate that an increase in the number of parameters does
not always lead to a proportional improvement in performance. For example, the
T5 3B model outperforms the T5 11B model in entity extraction within unseen
information types. This suggests that model capacity, though important, is not
the sole factor in determining performance. Other factors such as model
architecture, training data diversity, and learning techniques play a crucial
role.

#### Decoder-Only Architectures Struggle More in Information Types.

Decoder-only models such as LLaMa and BLOOM tend to struggle more in
generalization to unseen information types as compared to encoder-decoder
models. This could be due to their lack of an encoder component, which is
important for understanding complex input structures that are common in IE
tasks. However, ChatGPT demonstrates that decoder-only models with extensive
pre-training and in-context learning can still achieve reasonable performance.
This indicates that training methodology and in-context adaptation can play a
significant role in improving the generalization of decoder-only models.

##  6 Further Analysis

Figure 4: Impact of Instruction Components. The figure shows the average F1
scores of models with varying instruction components: Full (all components),
-Desc (without Task Description), -Rule (without Extraction Rule), -Format
(without Output Format), and -Demos (without Demonstration Examples).

###  6.1 Impact of Instruction Components

Figure 4 presents the impact of various instruction components on the
performance of LLMs in IE tasks. The components in consideration are Task
Description, Extraction Rule, Output Format, and Demonstration Examples.

#### Task Description.

The exclusion of the Task Description appears to have a marginal effect on the
performance of the models. For example, T5 3B exhibits a slight increase from
43.64 to 43.79, and ChatGPT experiences a minor drop from 54.24 to 53.47. This
suggests that while Task Description provides an overarching summary, it is
not critical for performance. The Extraction Rule and Demonstration Examples
likely offer the detailed guidance necessary for the models.

#### Extraction Rule.

Omitting the Extraction Rule generally leads to a decrease in performance
across most models. For instance, the T5 3B model drops from 43.64 to 42.49,
and ChatGPT decreases from 54.24 to 52.96. This indicates that Extraction
Rule, with its comprehensive guidelines, is crucial in guiding the models to
extract the relevant information effectively.

Figure 5: Impact of the number of examples.

#### Output Format.

The absence of Output Format leads to varied effects across the models.
Notably, T5 3B shows a dramatic decrease from 43.64 to 25.84. On the other
hand, ChatGPT experiences a minor increase in performance (54.67). This
suggests that while Output Format is significant for structuring the output in
some models, it may not be as crucial for others, especially if they have been
pre-trained to handle diverse output structures.

#### Demonstration Examples.

Removing Demonstration Examples has the most pronounced impact on performance.
For example, T5 3B plummets from 43.64 to 0, and ChatGPT falls sharply from
54.24 to 30.06. This underscores the importance of Demonstration Examples in
clarifying ambiguities and reinforcing the understanding of the task.

###  6.2 Analysis on Demonstration Examples

#### Impact of the number of examples.

The impact of varying the number of demonstration examples on the performance
of LLMs is shown in Figure 5. Across all models, it is evident that the
provision of demonstration examples significantly influences performance,
especially when transitioning from zero to one example. However, the effect of
adding more examples varies among models. ChatGPT and T5 models show a
consistent positive trend, while FLAN-T5, LLaMa, and BLOOM models exhibit
varied patterns. This analysis highlights the importance of demonstration
examples in IE tasks and suggests that the optimal number of examples can
differ based on the model’s architecture and capabilities.

Figure 6: Impact of correctness of examples.

#### Impact of correctness of examples.

The quality of demonstration examples is investigated by analyzing the
performance of LLMs with different proportions of incorrect examples. Table 6
presents the average F1 scores as we vary the number of incorrect
demonstration examples. Across all models, the correctness of demonstration
examples plays a vital role in performance. The sensitivity to incorrect
examples, however, varies among models. ChatGPT is the most sensitive, with a
pronounced decrease in performance as incorrect examples are introduced. T5
and FLAN-T5 models show stability or a gradual decline, while LLaMa and BLOOM
models display minor fluctuations. This analysis underlines the importance of
ensuring the accuracy and correctness of demonstration examples in the
instructions provided to LLMs, especially for models like ChatGPT that exhibit
high sensitivity to example quality.

Figure 7: Impact of input-output pairing.

#### Impact of input-output pairing.

Figure 7 presents the performance of LLMs when they are conditioned on
demonstration examples with varying formats - full examples with both inputs
and outputs, examples without outputs, and examples without inputs. The goal
is to understand which part of the demonstration example is crucial for
performance. Across all models, input-output pairing in demonstration examples
plays a crucial role in performance. Models like FLAN-T5 11B and ChatGPT are
heavily reliant on output information. LLaMa and BLOOM models also lean
towards output information, whereas T5 models show variations. This analysis
highlights the importance of including both inputs and outputs in
demonstration examples for optimal performance. However, if one must be
omitted, it appears that maintaining the outputs is generally more beneficial
than retaining only the inputs. This may be due to the fact that the outputs
often embody the essence of the task that the model needs to perform.

###  6.3 Analysis of Scaling Factors

We analyze the generalization performance of the models with respect to two
critical scaling factors: the number of instances per information type and the
size of the models. The results are shown in Figure 8. As the number of
training instances is increased, note that not all information types are
equally represented. Some event types have fewer than 100 samples. In such
cases, the dataset uses the maximum number of available samples for those
types. This leads to an exacerbation of the imbalance among different
information types as the number of instances increases, which is an important
consideration in the scaling trends.

Figure 8: Impact of the number of training examples per information type.

#### Influence of Training Instance Quantity.

Figure 8 reveals that augmenting the number of training instances is generally
associated with improved performance. However, the models react differently to
this scaling factor. T5 models display a more steady improvement as the number
of instances increases, whereas LLaMa models experience an early peak followed
by a decrease. The decline in LLaMa models’ performance could be linked to the
increasing imbalance in the dataset. As the dataset grows, the imbalance may
cause models to become biased towards information types with more samples.
Additionally, the non-linear scaling indicates that there may be a point of
diminishing returns, after which additional data does not yield significant
performance gains or may even be counterproductive.

#### Effect of Model Size on Performance.

When analyzing the impact of model size, the results suggest that larger
models typically have the edge. T5 models, for instance, exhibit more
consistent improvements as their size increases. However, this comes with the
caveat that larger models are more susceptible to overfitting, especially when
the dataset is small or imbalanced. This risk is pertinent as data scarcity
can make it difficult for larger models to effectively generalize. In the
decoder-only category, the difference in performance between LLaMa 13B and
LLaMa 7B is not pronounced at higher training sizes, highlighting that an
increase in model size does not guarantee proportionate performance
improvements. Consequently, a judicious balance between model size and the
quantity and diversity of training data is essential to maximize
generalization performance.

##  7 Related Work

#### Information Extraction.

Previously, Information Extraction (IE) focused on task-specific models
optimized for narrow objectives like Entity Extraction (Yan et al., 2021; Wang
et al., 2021) and Event Extraction (Yan et al., 2021; Wang et al., 2021; Du
and Cardie, 2020; Lu et al., 2021; Gao et al., 2023a). However, their task-
specific design inhibits knowledge sharing across various IE tasks (Lin et
al., 2020). This shortcoming paved the way for Universal Information
Extraction (UIE), which aims at building versatile models for extracting
diverse structured data from unstructured text (Lin et al., 2020; Lu et al.,
2022; Lou et al., 2023; Liu et al., 2023). Current UIE methods employ coarse-
grained instructions with basic task descriptions, overlooking essential
extraction rules and output format descriptions. To address this, we introduce
a fine-grained benchmark dataset for IE with augmented instructions
encompassing task descriptions, extraction rules, output formats, and
demonstration examples.

#### Large Language Models.

Large Language Models (LLMs) are central to NLP due to their impressive
performance on numerous tasks (Devlin et al., 2018; Radford et al., 2019;
Lewis et al., 2019; Raffel et al., 2020; Brown et al., 2020; Chowdhery et al.,
2022). Pretrained on vast corpora, LLMs can be fine-tuned for specialized
tasks. Recently, instruction tuning has emerged, wherein LLMs are fine-tuned
using task instructions for enhanced zero-shot task generalization (Sanh et
al., 2021; Chung et al., 2022; Ouyang et al., 2022). By scaling training
tasks, prompts, and LLM sizes, performance improves markedly. Combining
instruction tuning with demonstration examples further optimizes results (Min
et al., 2021; Chen et al., 2021; Ye et al., 2023). In this work, we assess
LLMs with different architectures (encoder-decoder and decoder-only) and
ChatGPT for the IE task.

##  8 Conclusion

This paper introduced a fine-grained IE benchmark dataset, tailored for LLMs,
utilizing augmented instructions to address the limitations of traditional
coarse-grained IE. Through extensive evaluation, encoder-decoder models,
notably T5 and FLAN-T5, showed prowess in generalizing to unseen information
types, owing to their capacity for capturing complex input-output
relationships. However, they exhibited limited adaptability to novel task
forms. ChatGPT, a decoder-only model with in-context learning, demonstrated
remarkable flexibility and adaptability. Furthermore, we found that model
scale is not the sole determinant of performance, emphasizing the importance
of architecture, data diversity, and learning techniques. our work contributes
to the evolution of IE by enabling more refined IE through LLMs. Future
endeavors should focus on combining the strengths of different architectures
and devising training methods that optimize both specificity and adaptability
in IE tasks.

## References

  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _ArXiv_ , abs/2005.14165.
  * Chen et al. (2021) Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2021.  Meta-learning via language model in-context tuning.  _arXiv preprint arXiv:2110.07814_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Du and Cardie (2020) X. Du and Claire Cardie. 2020.  Event extraction by answering (almost) natural questions.  _ArXiv_ , abs/2004.13625.
  * Gao et al. (2023a) Jun Gao, Changlong Yu, Wei Wang, Huan Zhao, and Ruifeng Xu. 2023a.  Mask-then-fill: A flexible and effective data augmentation framework for event extraction.  _arXiv preprint arXiv:2301.02427_.
  * Gao et al. (2023b) Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023b.  Exploring the feasibility of chatgpt for event extraction.  _arXiv preprint arXiv:2303.03836_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li et al. (2023a) Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023a.  Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness.  _arXiv preprint arXiv:2304.11633_.
  * Li et al. (2023b) Peng-Hsuan Li, Tianxiang Sun, Qiong Tang, Hang Yan, Yuanbin Wu, Xuanjing Huang, Xipeng Qiu Academy for EngineeringTechnology, Fudan University, School of Materials Science, Technology, and East China Normal University. 2023b.  Codeie: Large code generation models are better few-shot information extractors.  _ArXiv_ , abs/2305.05711.
  * Lin et al. (2020) Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020.  A joint neural model for information extraction with global features.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Liu et al. (2023) Chengyuan Liu, Fubang Zhao, Yangyang Kang, Jingyuan Zhang, Xiang Zhou, Changlong Sun, Fei Wu, and Kun Kuang. 2023.  Rexuie: A recursive method with explicit schema instructor for universal information extraction.  _ArXiv_ , abs/2304.14770.
  * Lou et al. (2023) Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2023.  Universal information extraction as unified semantic matching.  _ArXiv_ , abs/2301.03282.
  * Lu et al. (2021) Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, M. Liao, and Shaoyi Chen. 2021.  Text2event: Controllable sequence-to-structure generation for end-to-end event extraction.  _ArXiv_ , abs/2106.09232.
  * Lu et al. (2022) Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022.  Unified structure generation for universal information extraction.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Ma et al. (2023) Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023.  Large language model is not a good few-shot information extractor, but a good reranker for hard samples!  _arXiv preprint arXiv:2303.08559_.
  * Min et al. (2021) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021.  Metaicl: Learning to learn in context.  _arXiv preprint arXiv:2110.15943_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Raffel et al. (2019) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _ArXiv_ , abs/1910.10683.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * (25) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: open and efficient foundation language models, 2023.  _URL https://arxiv. org/abs/2302.13971_.
  * Wang et al. (2023) Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. 2023.  Instructuie: Multi-task instruction tuning for unified information extraction.  _arXiv preprint arXiv:2304.08085_.
  * Wang et al. (2021) Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021.  Improving named entity recognition by external context retrieving and cooperative learning.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Yan et al. (2021) Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021.  A unified generative framework for various ner subtasks.  _ArXiv_ , abs/2106.01223.
  * Ye et al. (2023) Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo. 2023.  Guess the instruction! flipped learning makes language models stronger zero-shot learners.  In _The Eleventh International Conference on Learning Representations_.

##  Appendix A Implementation Details

###  A.1 BLOOM and LLaMa Models

We utilized the DeepSpeed library for the fine-tuning of both the LLaMa and
BLOOM models, which include variants such as LLaMa-7B/13B and BLOOM-3/7.1B.
DeepSpeed enabled us to perform distributed training with mixed precision
support, effectively combining the benefits of memory efficiency and
computational speed. The training process employed a batch size of 8 and did
not require gradient accumulation steps. For optimization, the AdamW optimizer
was chosen, with a learning rate of 5e-5 and a weight decay parameter of 1e-4.
To gradually reduce the learning rate during training, we employed a cosine
annealing scheduler, while forgoing the use of any warm-up steps.

Additionally, the training made use of 16-bit floating-point precision (FP16),
which is known to reduce memory usage while accelerating the training process.
DeepSpeed’s Zero Redundancy Optimizer (ZeRO) was configured at stage 2,
wherein the optimizer states were offloaded to the CPU memory, resulting in a
further reduction in GPU memory consumption. To ensure consistency and
reproducibility in training results, the random seed was fixed at 1024. The
training was carried out over two epochs, and model checkpoints were saved
upon the completion of each epoch.

###  A.2 T5 and FLAN-T5 Models

For the fine-tuning of the T5 and FLAN-T5 models, including their variants
T5-3B/11B and FLAN-T5-3B/11B, we similarly leveraged the DeepSpeed library to
facilitate distributed training across eight GPUs. In this case, the training
was conducted with a batch size of 1 and gradient accumulation steps set to 8,
effectively simulating a larger batch size while avoiding excessive memory
consumption. The optimization process mirrored that of the BLOOM and LLaMa
models, using the AdamW optimizer with identical learning rate and weight
decay parameters, and employing a cosine annealing learning rate scheduler
without warm-up steps.

Both source and target sequence lengths were limited to 1024 tokens to
maintain computational efficiency. Interestingly, the training process was
executed using the bfloat16 numerical format instead of FP16. This format is
known to strike an optimal balance between training speed and numerical
precision, thus proving advantageous in large-scale model training. Like the
BLOOM and LLaMa models, ZeRO stage 2 was applied to offload optimizer states
to CPU memory, and a random seed of 1024 was utilized to assure the
reproducibility of the training outcomes.

##  Appendix B Details of Benchmark Dataset

In our research, the data for events and entities is sourced from the ACE05
dataset, while the data for sentiment information extraction is derived from
four datasets, namely 14lap, 14res, 15res, and 16res. To extract event and
entity information with higher precision, we designed a set of extraction
rules based on the event annotation guidelines of ACE05. Additionally, to
ensure the accuracy of sentiment information extraction, we consulted an
expert with extensive experience in the field of sentiment analysis to write
the corresponding extraction rules. To provide further insight into our data
processing, Figure 9 details the data partitioning in two different
experimental settings. Figure 10, on the other hand, showcases examples of the
instructions that we wrote for three different tasks.

Figure 9: Detailed depiction of data partitioning across training and test
sets in two distinct experimental configurations. Figure 10: Detailed
instructions for the three tasks. More instructions can be found via this link
https://anonymous.4open.science/r/IE-NAINST-6808.