# 5W1H Extraction With Large Language Models

Yang Cao, Yangsong Lan, Feiyan Zhai, Piji Li∗ *Corresponding author. College
of Computer Science and Technology,
Nanjing University of Aeronautics and Astronautics, China  MIIT Key Laboratory
of Pattern Analysis and Machine Intelligence, Nanjing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The extraction of essential news elements through the 5W1H framework (What,
When, Where, Why, Who, and How) is critical for event extraction and text
summarization. The advent of Large language models (LLMs) such as ChatGPT
presents an opportunity to address language-related tasks through simple
prompts without fine-tuning models with much time. While ChatGPT has
encountered challenges in processing longer news texts and analyzing specific
attributes in context, especially answering questions about What, Why, and
How. The effectiveness of extraction tasks is notably dependent on high-
quality human-annotated datasets. However, the absence of such datasets for
the 5W1H extraction increases the difficulty of fine-tuning strategies based
on open-source LLMs. To address these limitations, first, we annotate a high-
quality 5W1H dataset based on four typical news corpora (CNN/DailyMail, XSum,
NYT, RA-MDS); second, we design several strategies from zero-shot/few-shot
prompting to efficient fine-tuning to conduct 5W1H aspects extraction from the
original news documents. The experimental results demonstrate that the
performance of the fine-tuned models on our labelled dataset is superior to
the performance of ChatGPT. Furthermore, we also explore the domain adaptation
capability by testing the source-domain (e.g. NYT) models on the target domain
corpus (e.g. CNN/DailyMail) for the task of 5W1H extraction.

###### Index Terms:

5W1H Extraction, LLMs, Prompting, Fine-tuning

##  I Introduction

Effective extraction of crucial news aspects using the 5W1H framework (What,
When, Where, Why, Who, and How) is important for news aggregation, event
extraction, and text summarization [1, 2, 3, 4]. These 5W1H aspects aim to
capture the most fundamental details of news that people are interested in
when it comes to a specific event or topic. By absorbing these aspects, one
can obtain a comprehensive understanding of the core idea of the event.
Despite the fact that 5W1H extraction is a fundamental task in news analysis,
there is currently no effective method to ensure both accuracy and
comprehensiveness in carrying out this task. The previous approach focused on
implicitly detecting events, and often employed techniques like topic
modeling. These methods typically specialize in extracting task-specific
properties, such as the number of casualties in an accident. Recent studies
have shown some progress in extracting 5W1H aspects. Hamborg et al. [5]
proposed Giveme5W1H, which is an open-source main event retrieval system for
news articles. They first standardized the news content and then used syntax
and domain-specific rules to extract phrases related to 5W1H. Finally, a
scoring mechanism was implemented for candidate statements to determine the
relevant phrases in the main event. Giveme5W1H provides a rule-based direction
for extracting 5W1H elements.

Figure 1:  Different colors represent different sources of information from
the original text.In the context of the article, the mention of Hillary is not
meant to be a specific name, but the answer provided by ChatGPT mistakenly
identifies it as one of the important people. The names of places mentioned in
the green line may also refer to residential areas, but the given answers do
not correspond to them.

For the answers to How and Why that involve a comprehensive understanding of
the full article, however, they cannot propose a satisfied solution.

LLMs such as ChatGPT [6] have demonstrated remarkable capabilities especially
in zero-shot/few-shot scenarios based on the prompting or in-context leaning
strategies [7, 8]. However, 5W1H extraction is a more intricate task that
presents unique challenges. In contrast to tasks like summarization, 5W1H
extraction necessitates a more detailed and precise description of the
content. To use ChatGPT for 5W1H extraction, it is crucial to provide a
comprehensive and detailed task description and select an appropriate prompt
as the questioning method. Additionally, handling complex special cases is
critical for accurate extraction. Nevertheless, in our investigations we
observe that due to the limitations of context length, their abilities of 5W1H
extraction from full news reports are particularly insufficient. As shown in
Figure 1, we input ChatGPT with a news article and request it to extract
aspect Who from the text. From the results we can find that ChatGPT can
extract named entities from the text, but it cannot analyze their specific
types within the context. “Hilary” is not a commonly mentioned person name in
the article, but rather the name of a hurricane. In addition, in terms of the
comprehensiveness of information, ChatGPT ignores the content described in the
green highlighted section of the text. Therefore, solely relying on the
capabilities of LLMs themselves is far from enough.

Fine-tuning a model with task-specific human-created training data can achieve
performance improvement [9, 10, 11]. Therefore, the quality and quantity of
samples in the annotated dataset are crucial for the task. RA-MDS[12] is a
dataset consisting of manually annotated news event aspects, which contains
six different categories of news and annotates the distinctive aspects for
each category. However, it did not extract 5W1H elements for all categories,
resulting in the loss of certain category attributes. Additionally, the number
of articles is relatively small, which may not be sufficient for effectively
fine-tuning LLMs. In addition, the length of the news articles is typically
quite long, which increases the memory requirement for training. Thus
efficient fine-tuning approaches are required due to the limitations of
computing resources. Therefore the extraction of 5W1H elements becomes
considerably more challenging.

To address the above problems, first, we annotate four publicly accessible
news datasets(CNNDM, XSum, NYT, and RA-MDS) to extract statements from the
text that are used in 5W1H questions. In total, we labeled 3,500 entries
within these datasets. Second, for the fine-tuning techniques, recently, some
efficient methods of fine-tuning LLMs are proposed, such as Prefix-Tuning[13],
Low-Rank Adaptation (LoRA)[14], and QLoRA[15]. They significantly reduce the
resources and time required for fine-tuning the model. Then we fine-tune these
LLMs on the annotated dataset and assess their performance in extracting 5W1H
elements. Additionally, we also conduct an evaluation of the capability of
news across different domains.

Our contributions are as follows: 1) We annotate a high-quality 5W1H elements
extraction dataset for news in several different domains on four different
datasets. 2) We fine-tune some LLMs on the annotated datasets and also conduct
comparisons on zero-shot and few-shot capabilities with ChatGPT. The
experimental results show that the performance of fine-tuned models is better
than ChatGPT. 3) we also explore the domain adaptation capability of models by
testing the source-domain (e.g. NYT) model to the target domain corpus (e.g.
CNN/DailyMail) for the task of 5W1H extraction.

##  II Related Work

###  II-A Event extraction

5W1H elements extraction from news articles is a specific area of event
extraction. However, recent research has not used LLMs for extracting 5W1H
information.

Hamborg et al.[16] proposed the first open-source system (Giveme5W) for
retrieving the main event from news articles. The system uses syntactic and
domain-specific rules to extract and score phrase candidates for each 5W
question. Answering How questions becomes particularly complex due to the
following reasons. First, the answer to the How question usually depends on
the context and may require a comprehensive understanding of the relevant
information in the text. This may require the system to search for information
in a wide range of contexts, rather than relying solely on the current
sentence or paragraph. Second, It may be influenced by ambiguity, as an event
or process may have multiple interpretations. The system needs to be able to
recognize and handle this ambiguity in different contexts. In addition, the
How question is considered less important in many use cases when compared to
other questions[17, 18, 19]. Most systems focus only on the extraction of 5W
questions without How questions. Hamborg et al.[2] proposed the framework
Giveme5W1H based on Giveme5W. They addressed How questions using semantic and
grammatical rules and tested their effectiveness on multiple datasets. The
extraction of content related to How and Why questions, requiring a thorough
understanding of the entire text, is notably insufficient when relying solely
on semantic rules.

Recent studies aim to understand events and capture their correlations by
using LLMs. Gao et al.[9] used ChatGPT and task-specific modeling to compare
performance on the event extraction task and explored the effects of different
prompts on ChatGPT. However, their experimental results indicated that despite
attempting numerous prompts, ChatGPT still did not perform as well as task-
specific models. Srivastava et al.[20]presented the first dataset, MAILEX, for
performing event extraction from conversational email threads. They explored
the impact of ChatGPT on the email event extraction task in a few-shot
scenario. But the experimental results showed that in-context learning in
ChatGPT demonstrated poorer performance compared to the fine-tuned approaches.
Li et al.[21] found that ChatGPT’s performance in the standard information
extraction setting is not as effective as that of models based on the BERT-
based models in most cases. We find that there is currently no research on
5W1H elements in news extraction using LLMs. So we use LLMs to train a high-
quality annotated dataset.

###  II-B Fine-tuing LLMs

With the advent of LLMs, many fine-tuning methods have emerged. Instruction
tuning[22] is a method through which pretrained autoregressive language models
are finetuned to follow natural language instructions and generate responses.
It helps reduce the need for annotated data, as the model can be fine-tuned
through guidance without relying on large amounts of annotated task-specific
data. Additionally, the model can be fine-tuned through different natural
language instructions, providing greater flexibility to adapt to multiple
tasks without the necessity of retraining the entire model. Hu[14] proposed
LoRA, which freezes the weights of the pretrained model, significantly
reducing the number of trainable parameters and computational memory.
Dettmers[15] proposed QLoRA, which can replicate 16-bit full finetuning
performance with a 4-bit base model. QLoRA is based on the LoRA framework,
which preserves model performance while further reducing the model’s memory.
In this work, we use QLoRA to fine-tune LLMs.

Figure 2: Fine-tuning LLMs on 5W1H elements extraction from news articles.
Construct annotated datasets, convert them into the required format for input
into LLMs, fine-tune with different models, and select the best model based on
evaluation results.

##  III METHODOLOGY

In this section, we describe the process of constructing a dataset for
extracting 5W1H elements from news and explain how to use LLMs to fine-tune
the constructed dataset. The architecture is shown in Figure 2, which consists
of three parts. First, we extract 5W1H elements from news articles by
professional annotators. Subsequently, we process the annotated dataset into
Supervised Fine-Tuning (SFT) format. Next, we choose three models to train the
dataset and evaluate their effectiveness. Finally, we select the model with
the best performance based on the evaluation results.

###  III-A Dataset Construction

We annotate the 5W1H dataset based on four typical news corpora
(CNN/DailyMail, XSum, NYT, RA-MDS). In total, we label 3,500 entries within
these datasets. We follow Li’s et al.[12] categorization of news during
dataset construction. There are 6 predefined categories in the labeled news:
(1) Accidents and Natural Disasters, (2) Attacks (Criminal/Terrorist), (3) New
Technology, (4) Health and Safety, (5) Endangered Resources, and (6)
Investigations and Trials (Criminal/Legal/Other). A variety of different
elements are individually defined within each category. In contrast to the
Li’s et al.[12] approach, we have added elements of 5W1H to each category. We
ask the experts to determine which category the article belongs to based on
its content. After determining the category of the article, we analyze its
content and extract all the statements related to 5W1H from the original text.

Annotators are primarily engaged in AI data annotation services, with
particular expertise in English business scenarios. The team is one of the
suppliers that has been cooperating with Tencent for a long time, and due to
the confidentiality agreement, we have not provided specific information about
the team. Approximately 10 individuals are dedicated to this task. Each
article is collaboratively annotated by a team of three annotators. For the
short entities mentioned in the article, such as Who, Where, and When, we can
directly extract information from the original text. However, for long
entities like What, Why, and How, a comprehensive understanding of the
article’s content is required to better extract relevant information. We have
made the following attempts to extract Why from the original text. We ask the
annotators to identify all events in the article, each of which may involve
multiple reasons, and identify each corresponding reason from the text one by
one. Finally, the third annotator will double-check for any omissions. The
same applies to What and How. Before the formal annotation process, we select
50 news articles from each dataset for preliminary annotation by our
annotators. Detailed instructions are provided to address any issues
encountered during this preliminary phase. The average token count for
articles in each dataset is presented in Table I. In order to facilitate
labeling by experts we specifically choose articles with an approximate length
of 550 words.

TABLE I: The average words of articles in each dataset before formal annotation. Dataset | AvgDocWords
---|---
CNN/DailyMail | 579
XSum | 523
NYT | 552
RA-MADS | 568

For the accuracy of the answers, all marked elements are directly extracted
from the original text without any semantic generalization or paraphrasing. In
the labeling process, for each statement (if it belongs to the 5W1H elements),
there must be a uniquely identified element corresponding to it. For instance,
if the statement is categorized as What, it cannot be classified as How or any
other case. This approach helps prevent ambiguity in labeling statements. To
ensure that key information is not omitted, we annotate all relevant 5W1H
elements in each article during the labeling process. For instance, in a news
article, there might be multiple instances of time and location mentioned, and
an event could have various reasons. We conduct comprehensive extraction for
each of these elements to cover all potential key information present in the
article. At the same time, to prevent redundancy in labeling each sentence, we
choose to extract a fragment of the statement (some words). For example, a
sentence with a brief mention of the time and place (only about nouns) could
be extracted without including the entire sentence. During the labeling
process, there are certain types of news where identifying the corresponding
element proves challenging. For the identification of some elements, we adopt
a broader definition based on the contextual meaning of the original text. In
some technology-related articles, there may not be explicit mentions of
location-related information. Instead, scenarios could involve discussing the
transmission of information through the medium of the internet. Therefore, in
such cases, we define the internet as a more abstract location. We try to
label each article with 5W1H, but there might be instances where we are unable
to extract all the elements of 5W1H from certain news articles. Finally, we
save the labeled datasets in JSON format. The annotated content of the 5W1H
elements from the article is shown in Figure 3.

Figure 3: Correspondence between 5W1H aspects and the original text, with
different colors representing different answers to questions.

###  III-B Fine-tuning for 5W1H Extraction

After obtaining the annotated dataset, we apply instruction-tuning to LLMs to
compare the differences between our annotated dataset and the results
extracted using ChatGPT. Our template is shown in Figure 3. The LLMs can
directly comprehend the 5W1H methodology and extract relevant content from the
original text. Therefore, we don’t employ the Chain-of-Thought (CoT)
approach[23], which involves guiding the LLMs through a conversational method
to understand and retrieve content for the extraction task. Instead, we
instruct the LLMs to directly extract content related to 5W1H from the
original text. In this approach, the model is presented with articles and
instructions as input and the 5W1H elements are used as the output of the
model. We input the original text and instructions into the LLMs, and the
model will generate the extracted information of the 5W1H elements from the
article. More specifically, the LLMs are presented with an article _X_
${}_{\mbox{\scriptsize passage}}$ as input. For each 5W1H element _E_
${}_{\mbox{\scriptsize elements}}$ that appears in the output, we convert them
into a natural language query, such as ”Please extract What, When, Where, Why,
Who, and How from the news.” Subsequently, we fine-tune the LLMs to generate a
structured output _y_ ${}_{\mbox{\scriptsize i}}$ in the format of a JSON list
containing all entities _E_ ${}_{\mbox{\scriptsize elements}}$. _y_
${}_{\mbox{\scriptsize i}}$ represents the collection of extracted 5W1H
elements from each article.

We choose LLaMA111The checkpoint is “huggyllama/llama”.[24], Vicuna222The
checkpoint is “eachadea/vicuna”.[25], and Guanaco333The checkpoint is
“TheBloke/guanaco”.[15] as training models and employ QLoRA fine-tuning
approach, following the training schedule of Dettmers et al. To assess the
impact of different model sizes, we evaluate the performance of 7B and 13B
models separately. Before fine-tuning with LLMs, we treat the task of
extracting 5W1H elements from the news as a standard Named Entity Recognition
(NER) task, given its close association with specific terms and NER-related
challenges. News, being a form of lengthy textual content, poses a challenge
when we choose the XLNet[26] pre-trained model for processing. However, the
experimental outcomes are less than satisfactory, especially in the extraction
of long entities, where the results are nearly non-existent. This indicates
the inadequacy of relying solely on pre-trained models for effective 5W1H
element extraction. Meanwhile, this also indicates that the NER task is not
suitable for this specific task. Hence, the appropriate evaluation metrics for
assessing the effectiveness of 5W1H element extraction becomes crucial.

During the inference stage, to enhance the model’s inference speed, we utilize
the CTranslate2[27]framework to convert the combined LORA weights from the
fine-tuned model and the base model into the CTranslate2 format. We reduce the
elapsed time by employing int8 quantization, which results in a slight effect
on the performance of the model. However, this will not have a significant
impact on the inference performance of the model. We select the best-
performing model from the fine-tuned models based on evaluation metrics to
apply to other datasets. Finally, we compare fine-tuned models with ChatGPT
and GPT-4 and save the results in JSON format.

##  IV EXPERIMENTS

In this section, we describe the experiment details including datasets,
implementation details, and the results.

###  IV-A Datasets

CNN/DailyMail[28] serves as a single text summarization corpus, which is
commonly used for extractive summarization tasks. It includes summaries with
multiple sentences and is collected from CNN and DailyMail.

XSum[29]dataset is a highly abstractive dataset and consists of single-
sentence summaries. It is particularly suitable for evaluating models in tasks
requiring concise abstractive summarization.

NYT[30]dataset comprises millions of articles from The New York Times,
spanning 1987-2007. It includes over 650,000 staff-written summaries and 1.5
million annotated articles, suitable for tasks like automatic summarization
and text classification.

RA-MDS[12]is a multi-document news summary dataset that focuses on extracting
news comments. The articles highlight readers’ points of interest in news
comments, providing valuable perspectives for summary systems. This dataset
encompasses 450 news articles from six different domains.

The first three datasets each contain 1000 instances. Each dataset is divided
according to 8:1:1 for training, validation, and testing. Since the RA-MDS
dataset contains 450 news, we decide not to divide it separately. Instead, we
merge this part of the dataset into the first three datasets for training.

TABLE II: Each element in the experimental results has valid responses of over
80; responses with fewer than 80 are not displayed in the results. R-2
represents Rouge-2, R-L represents Rouge-L and B-4 represents BLEU-4

CNN/DailyMail |
---|---
Model | What | Who | Why | When | Where | How
R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-1 | R-2 | B-4
Fine-tuing |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
LLaMa-7B | 39.20 | 47.67 | 55.75 | 3.59 | 19.42 | 58.61 | 17.05 | 25.49 | 34.14 | 17.12 | 31.16 | 62.97 | 9.49 | 19.76 | 54.07 | 14.42 | 24.57 | 28.09
Vicuna-7B | — | — | — | 2.88 | 18.42 | 64.81 | 14.83 | 22.07 | 29.62 | 14.31 | 28.92 | 62.63 | 6.83 | 18.17 | 57.90 | 14.60 | 24.60 | 35.62
Guanaco-7B | 26.51 | 36.19 | 28.78 | 3.39 | 19.90 | 51.70 | 17.60 | 26.16 | 24.26 | 11.98 | 25.77 | 56.17 | 8.30 | 19.53 | 51.51 | 16.22 | 25.73 | 19.36
LLaMa-13B | 32.75 | 42.30 | 32.37 | 5.16 | 21.81 | 52.64 | 13.61 | 23.59 | 28.42 | 15.56 | 32.49 | 49.00 | 11.27 | 23.53 | 45.64 | 21.96 | 31.93 | 31.34
Vicuna-13B | 31.25 | 40.61 | 29.53 | 4.71 | 20.66 | 50.45 | 13.91 | 23.41 | 30.08 | 12.35 | 28.07 | 49.87 | 6.43 | 17.27 | 41.77 | 22.41 | 32.35 | 31.98
Guanaco-13B | 28.85 | 38.38 | 29.20 | 4.06 | 20.81 | 50.69 | 14.54 | 24.69 | 24.04 | 17.98 | 36.18 | 46.50 | 14.61 | 27.34 | 43.23 | 21.14 | 31.48 | 27.29
zero-shot |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GPT-3.5-turbo | 13.25 | 28.15 | 46.19 | 3.75 | 11.75 | 26.84 | 10.80 | 21.42 | 33.71 | 3.76 | 11.91 | 21.44 | 10.04 | 19.18 | 30.23 | 7.86 | 21.29 | 35.67
GPT-4 | 16.44 | 29.53 | 52.04 | 3.36 | 9.87 | 22.91 | 10.50 | 22.07 | 29.76 | 2.88 | 7.26 | 15.79 | 3.98 | 9.99 | 22.70 | 7.88 | 20.45 | 35.31
few-shot |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GPT-3.5-turbo | 12.29 | 22.80 | 48.72 | 3.08 | 10.12 | 27.26 | 8.75 | 17.68 | 30.79 | 6.69 | 15.27 | 23.53 | 7.84 | 16.69 | 28.59 | 5.81 | 15.66 | 30.02
GPT-4 | 35.61 | 45.33 | 52.82 | 5.57 | 19.60 | 46.78 | 16.51 | 25.19 | 34.88 | 17.03 | 31.17 | 45.22 | 15.98 | 27.85 | 46.83 | 12.15 | 21.00 | 35.86
XSum |
Model | What | Who | Why | When | Where | How
R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-1 | R-2 | B-4
Fine-tuing |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
LLaMa-7B | 41.74 | 51.19 | 53.96 | 8.00 | 19.49 | 44.70 | 16.83 | 25.59 | 36.77 | 11.39 | 27.07 | 43.81 | 7.59 | 17.77 | 42.74 | 22.63 | 32.34 | 41.67
Vicuna-7B | — | — | — | 9.56 | 21.53 | 45.37 | 18.18 | 27.97 | 35.92 | 11.28 | 27.45 | 46.97 | 7.48 | 17.65 | 43.30 | 19.55 | 29.76 | 36.41
Guanaco-7B | 42.07 | 51.41 | 48.41 | 10.33 | 21.18 | 42.13 | 16.73 | 25.36 | 32.00 | 10.53 | 26.75 | 43.14 | 7.34 | 17.63 | 40.97 | 20.22 | 30.05 | 35.01
LLaMa-13B | 44.32 | 53.36 | 52.04 | 7.49 | 18.40 | 47.49 | 17.24 | 26.61 | 38.19 | 10.58 | 27.67 | 46.81 | 8.82 | 19.89 | 43.29 | 15.52 | 25.19 | 36.50
Vicuna-13B | 40.92 | 50.62 | 54.83 | 8.18 | 21.34 | 45.18 | 20.22 | 29.45 | 40.57 | 11.35 | 27.44 | 45.27 | 7.60 | 17.25 | 41.81 | 19.38 | 28.65 | 38.51
Guanaco-13B | 43.20 | 52.61 | 49.81 | 9.18 | 21.66 | 45.49 | 14.29 | 23.26 | 32.79 | 11.00 | 28.12 | 45.62 | 8.32 | 20.14 | 43.08 | 17.69 | 28.21 | 38.20
zero-shot |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GPT-3.5-turbo | 10.60 | 22.97 | 43.06 | 7.43 | 16.98 | 30.31 | 8.17 | 18.42 | 35.75 | 6.18 | 13.70 | 20.62 | 7.56 | 16.72 | 29.03 | 9.54 | 19.72 | 37.89
GPT-4 | 13.43 | 26.77 | 44.04 | 8.81 | 20.81 | 35.88 | 10.04 | 20.62 | 34.29 | 7.30 | 17.70 | 25.97 | 8.65 | 19.89 | 38.57 | 10.50 | 21.68 | 38.43
few-shot |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GPT-3.5-turbo | — | — | — | 3.95 | 9.98 | 16.61 | 8.39 | 17.17 | 22.74 | 1.79 | 4.96 | 3.58 | 1.92 | 6.08 | 6.31 | 6.99 | 16.45 | 36.19
GPT-4 | 44.99 | 54.24 | 61.58 | 13.09 | 23.31 | 52.01 | 19.02 | 27.78 | 39.22 | 18.87 | 34.08 | 57.62 | 10.05 | 21.83 | 50.44 | 17.74 | 25.32 | 47.24

###  IV-B Implementation details

To adapt to the task of extracting 5W1H elements, we adjust the training
strategy for the Dettmers et al. aspect. We truncate the length of each news
article to 750 tokens. The maximum input length and output length of the
article are determined based on different datasets. We set source_max_len and
target_max_len based on different datasets and set max_new_tokens to 1024. We
save the checkpoint every 500 steps and conduct model training for 1000 steps.
In the decoding phase, we employ the Top-p sampling strategy. In Top-p
sampling, the model selects the next word based on the probability
distribution but only stops when the cumulative probability reaches a certain
threshold. Top-p sampling can be used to balance the diversity and
controllability of generated text. We set the sampling_topp to 0.95 and
sampling_temperature to 0.7. To better ensure that the text output retains as
much relevant information about the 5W1H elements as possible, we set
max_length to 2000.

We don’t follow the evaluation rules and procedure as described by Hamborg et
al.[16]They asked three assessors to judge the relevance of each answer on a
3-point scale (non-relevant, partially relevant, and relevant) and used mean
average generalized precision (MAgP) as a measure of 5W1H question answering.
We think that people may have varying opinions on the answers to the
questions, and it is possible that this scoring mechanism could influence the
final results. Additionally, the extraction for the 5W1H elements task is
different from the event extraction task. It involves extracting not only
short entities like time, location, and people but also corresponding
sentences for long entities from the original text. However, event extraction
tasks do not evaluate sentences. We use the text-generation evaluation metrics
ROUGE[31] and BLEU[32]. The extracted 5W1H information generated by the LLMs
is compared with the labeled datasets to determine the level of similarity. In
the following experiments, we adopt ROUGE-1, ROUGE-2, and BLEU-4 for
evaluation.

All experiments are conducted on NVIDIA RTX3090 GPUs (24G memory). For the 7B
model size, we train using a single NVIDIA RTX 3090 GPU for approximately 13
hours. On the other hand, for the 13B model size, we use two NVIDIA RTX3090
GPUs and train for about 24 hours. Both of these training sessions are
conducted on a single dataset.

TABLE III: The results on XSum and NYT.R-2 represents Rouge-2,R-L represents
Rouge-L and B-4 represents BLEU-4

Model | What | Who | Why | When | Where | How
---|---|---|---|---|---|---
R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-2 | R-L | B-4 | R-1 | R-2 | B-4
XSum(Vicuna 13B) | 40.08 | 50.22 | 55.21 | 10.64 | 26.55 | 47.86 | 16.21 | 25.42 | 35.78 | 19.87 | 39.29 | 51.54 | 10.86 | 25.15 | 47.79 | 23.00 | 32.66 | 40.00
XSum(CNN fine-tune) | 35.34 | 45.01 | 37.87 | 9.26 | 24.99 | 52.84 | 21.13 | 29.74 | 35.89 | 13.66 | 30.77 | 54.71 | 9.73 | 23.71 | 52.23 | 18.99 | 28.45 | 31.20
NYT(Vicuna 13B) | 41.14 | 49.05 | 46.86 | 8.65 | 24.09 | 45.25 | 20.18 | 28.97 | 34.37 | 8.15 | 19.80 | 48.74 | 8.20 | 19.15 | 45.59 | 20.05 | 29.13 | 31.98
NYT(CNN fine-tune) | 30.02 | 38.24 | 27.72 | 10.43 | 24.48 | 36.39 | 19.11 | 27.71 | 31.14 | 6.01 | 17.67 | 44.30 | 8.94 | 20.23 | 44.74 | 18.36 | 27.12 | 27.17
XSum(Guanaco 13B) | 44.70 | 54.19 | 54.04 | 10.47 | 25.07 | 46.14 | 14.48 | 23.89 | 32.95 | 20.08 | 38.75 | 50.89 | 12.64 | 28.33 | 47.13 | 17.20 | 27.21 | 36.36
XSum(CNN fine-tune) | 36.15 | 45.32 | 37.87 | 12.45 | 27.61 | 52.84 | 19.07 | 28.42 | 35.89 | 18.00 | 36.36 | 54.71 | 13.04 | 29.50 | 52.23 | 17.40 | 27.84 | 31.20
NYT(Guanaco 13B) | 42.57 | 50.64 | 42.29 | 9.16 | 26.24 | 49.82 | 19.82 | 28.51 | 38.32 | 7.94 | 24.10 | 47.89 | 10.91 | 22.12 | 47.89 | 13.18 | 23.40 | 35.49
NYT(CNN fine-tune) | 29.25 | 37.74 | 20.28 | 10.00 | 22.69 | 33.72 | 18.38 | 26.54 | 30.62 | 7.41 | 22.80 | 37.43 | 10.39 | 22.21 | 35.29 | 16.42 | 24.97 | 25.93

###  IV-C Experimental Results

We present the results of the experiments with the labeled dataset after fine-
tuning with different LLMs. Furthermore, we assess the efficacy of 5W1H
extraction from different models by analyzing the count of valid entries.
Finally, we verify the compatibility of the models across diverse datasets.

#### Finetuning on LLMs

Table II shows the effect of fine-tuning different size models on extracting
5W1H elements from the CNN/DailyMail and XSum datasets. All experimental
results are based on valid responses. While the 7B model demonstrates
promising results for certain questions, it falls short in generating a
substantial volume of valid responses, particularly when compared with the 13B
model. Due to the limitations in the model’s performance, the 7B model fails
to extract all the relevant 5W1H content during the inference process,
resulting in a much lower count of valid responses compared to the 13B model.
Our findings suggest that responses to questions related to What, Why, and How
yield significantly higher results compared to those for Who, When, and Where.
Understanding What, Why, and How requires a thorough comprehension of the
original text, leading to more detailed and intricate statements. In other
cases, it only requires extracting nouns from the article, such as time,
location, and individuals, which leads to lower results. The results further
illustrate that augmenting the number of model parameters contributes to
enhanced performance, suggesting a potential improvement in the model’s
ability to comprehend and extract 5W1H elements extraction from articles.

#### Zero-shot and Few-shot on ChatGPT and GPT-4

Figure 4:  Effective numbers of different models answering 5W1H questions on
CNN/DailyMail dataset. All models are based on 13B size. Different colors
represent different questions.

We attempt to use both GPT-3.5 and GPT-4 for extracting 5W1H elements under
two conditions: one without providing examples and the other with five given
examples. The prompts are consistent with the fine-tuned LLMs. The
experimental results indicate that without providing annotated samples, the
extraction performance of both ChatGPT and GPT-4 is not satisfactory. After
providing five samples, the extraction performance of ChatGPT doesn’t improve.
In contrast, the results of GPT-4 show a significant enhancement, especially
in answering Who, Why, and Where questions, where the ROUGE and BLEU were
better than those after fine-tuning the LLMs.

Based on the experimental results, ChatGPT does not perform well in the task
of extracting 5W1H elements. This may be due to the following reasons. News is
usually long text and models may face issues of missing information. To
accurately extract 5W1H elements, the model needs to have a deep understanding
of the context of the text. Extracting 5W1H may involve longer entities, such
as detailed information describing the process of events, which can pose
challenges for the model. In contrast, GPT-4 shows significant improvement in
performance after being provided with five examples. Therefore, we believe
that enhancing the model’s performance in 5W1H elements extraction can be
achieved by increasing the number of parameters and improving the quality of
labeled datasets. Figure 4 illustrates the average number of 5W1H answers
efficiently extracted by different models across the three datasets. Vicuna
and Guanaco have more effective answers in extracting 5W1H elements than
LLaMa, and their performance is similar. Therefore we choose these two models
to extract other datasets to verify their applicability in the news extraction
domain.

TABLE IV: Examples of 5W1H element extraction in different situations. The
5W1H elements represented by different colored statements in the original
document are annotated by professionals Source Document (CNN/DailyMail)
---
There is no doubting the significance of Sunday ’s Tyne-Wear derby now .Debate
has raged in Newcastle as to which their fans would rather see - a place in
the Capital One Cup semi-final or a first win over Sunderland in six attempts
. The majority sided with the former and a shot at a first domestic trophy
since 1955 .That dream , however , is dead . Alan Pardew and his players had
talked of a ‘ massive ’ week in their season . Well , it just got bigger
.Massadio Haidara consoles keeper Jak Alnwick after a night to forget for
Newcastle at Tottenham .Newcastle ’s defence reels as Spurs celebrate the
final goal of their 4-0 defeat of the Toon on Wednesday .For Alan Pardew ’s
side , Sunday ’s Tyne-Wear derby takes on a new dimension now they ’re out of
the cup .The manager had , in fairness , named just about the strongest XI
available to him but it looked awfully weak against a Spurs side who were
superior in every department .The against-the-odds spirit that had inspired
Newcastle ’s 2-0 victory at Manchester City in the last round was missing . It
was , in the end , a rather timid surrender .Apart from Moussa Sissoko , the
French battering ram who tried his best to barge his team forward , the
visitors offered little .Sissoko ’s compatriots Yoan Gouffran , Remy Cabella
and Emmanuel Riviere – a combined 20million worth of ‘ talent ’ – were
passengers on a journey destined to end in defeat from the moment Jak Alnwick
spilled Christian Eriksen ’s corner and Nabil Bentaleb looped home the opening
goal .A Newcastle fan , one of 4,200 supporters to travel down to London ,
gives his appraisal of the match .These shirtless lads do their best to lift
their team having made the journey for the Capital One Cup loss .This has been
a Newcastle team characterised by resistance and resilience , neither of which
was in evidence .It was their second heavy loss in north London in five days
and such drubbings will do little for the team ’s confidence come Sunday
.Pardew is the only manager in Newcastle ’s history to lose three in a row
against their North-East rivals and , with the mitigation of a cup run gone ,
he can ill afford another defeat this weekend .It was after Spurs ’ fourth
goal – Roberto Soldado capitalising on another Alnwick spill – that Pardew
turned his attention to the derby .Sunderland boss Gus Poyet issues
instructions to his players on their way to a 1-1 draw with West Ham .Jordi
Gomez celebrates putting Sunderland 1-0 up from the penalty spot at home
against West Ham .The thought of going into that game without Sissoko is a
frightening one and , when he took what proved to be an innocuous knock to the
knee , Pardew withdrew his star man almost immediately .His effort was
recognised by the 4,200 travelling fans and they sang until the last kick of
an otherwise torrid evening .Pardew acknowledged their support but will be in
doubt about how fast such goodwill would evaporate should they fail to show
against Gus Poyet ’s side .The debate on radio phone-ins and in the local
newspaper had been ‘ which of the two matches would you prefer to win ? ’
Losing both was never an option .Pardew , of course , has done tremendously
well to transform his own and Newcastle ’s fortunes , but the mood can change
quickly on Tyneside .After this humiliation , Pardew will be only too aware of
Sunday ’s significance .
Fine-tune LLaMa on 5W1H extraction
”what”: [”Debate has raged in Newcastle as to which their fans would rather
see - a place in the Capital One Cup semi-final or a first win over Sunderland
in six attempts . The majority sided with the former and a shot at a first
domestic trophy since 1955 .That dream , however , is dead”],
”when”: [”on Wednesday, 1955”],
”where”: [” Newcastle, Tottenham, London”],
”why”: [” The majority sided with the former and a shot at a first domestic
trophy since 1955 .That dream , however , is dead”],
”who”: [”Alan Pardew, Massadio Haidara, keeper Jak Alnwick, Newcastle, Moussa
Sissoko, Yoan Gouffran, Remy Cabella, Emmanuel Riviere, Christian Eriksen,
Nabil Bentaleb, Gus Poyet, Jordi Gomez”],
”how”: [”Well , it just got bigger .Massadio Haidara consoles keeper Jak
Alnwick after a night to forget for Newcastle at Tottenham .Newcastle ’s
defence reels as Spurs celebrate the final goal of their 4-0 defeat of the
Toon on Wednesday”]
Few-shot on ChatGPT
When: Sunday, May 12
Where: Tyne-Wear derby in Newcastle
Why: To determine whether Newcastle would rather advance to the Capital One
Cup semi-final or win over Sunderland
Who: Alan Pardew (Newcastle manager), Moussa Sissoko (Newcastle player), Yoan
Gouffran, Remy Cabella, Emmanuel Riviere (Newcastle players), Jak Alnwick
(Newcastle goalkeeper), Christian Eriksen and Nabil Bentaleb (Spurs players),
Roberto Soldado (Spurs player), Gus Poyet (Sunderland manager)
How: Newcastle lost 4-0 to Spurs in the Capital One Cup, causing them to focus
on their upcoming match against Sunderland
Zero-shot on GPT-4
What: The upcoming Tyne-Wear derby between Newcastle and Sunderland.
When: Sunday (No specific date given)
Where: Newcastle
”Why”: The significance of the Tyne-Wear derby has increased after Newcastle’s
loss in the Capital One Cup semi-final.
textcolororangeWho: Newcastle fans, Manager Alan Pardew and his players,
Sunderland team, Spurs side
How: Newcastle lost against Spurs which resulted in their ousting from the
Capital One Cup. This has increased the pressure on them for their upcoming
match against Sunderland.
Few-shot on GPT-4
”what”: [”Sunday ’s Tyne-Wear derby takes on a new dimension now they ’re out
of the cup”, ”It was , in the end , a rather timid surrender”, ”The thought of
going into that game without Sissoko is a frightening one”],
”when”: [”on Wednesday”, ”in five days”, ”this weekend”, ”Sunday”],
”where”: [”Newcastle”, ”at Tottenham”, ”in north London”],
”why”: [”Alan Pardew and his players had talked of a ‘ massive ’ week in their
season . Well , it just got bigger”],
”who”: [”Massadio Haidara ”, ”Jak Alnwick”,”Alan Pardew’s side ”,”Moussa
Sissoko ”],
”how”:[”The manager had , in fairness , named just about the strongest XI
available to him but it looked awfully weak against a Spurs side who were
superior in every department”]

#### Adaptation Between Different Datasets

Table III shows the extraction results of the model on the other two datasets.
The first line represents the result after fine-tuning with its own dataset,
and the second line represents the model after fine-tuning on CNN/DailyMail in
other datasets. We find that statements about Who, Where, and When, which are
relatively concise in their answers, don’t show significant variations in
their effectiveness across different datasets. On the other hand, the
extraction of What, Why, and How, which requires a more comprehensive
understanding of the article, proves to be less effective in comparison. We
think that increasing the training dataset such as integrating the training of
these four labeled datasets may have an improvement in the model’s extraction
performance.

#### Example Analysis

Table IV shows the comparison between the fine-tuned model, ChatGPT and GPT-4.
For short entities, the fine-tuned LLaMa model can extract some entities from
the original text but may still miss some cases. For long entities, due to the
impact of model performance and the size of the fine-tuning dataset, there are
still some repetitions in the answers to Why and What. ChatGPT performs not
well after being provided with examples and does not learn the output format.
In contrast, The output of GPT-4 is similar to manually annotated.

##  V CONCLUSION

In this work, we annotate 5W1H elements from the text to address the
extraction challenges and fine-tune the datasets using LLMs. The extraction
performance is comparable to GPT-4’s few-shot capability and outperforms
ChatGPT. We also find that our annotated datasets can express their ability to
extract 5W1H elements across different news domains by increasing model
parameters and training data. Therefore, creating a high-quality dataset is
crucial in addressing challenges faced by LLMs in news extraction.

##  VI Acknowledgements

We extend our gratitude to all the reviewers for their professional advice.
This research is supported by the National Natural Science Foundation of China
(No.62106105), the CCF-Baidu Open Fund (No.CCF-Baidu202307), the CCF-Zhipu AI
Large Model Fund (No.CCF-Zhipu202315), the Scientific Research Starting
Foundation of Nanjing University of Aeronautics and Astronautics
(No.YQR21022), and the High Performance Computing Platform of Nanjing
University of Aeronautics and Astronautics.

## References

  * [1] F. Hamborg, N. Meuschke, and B. Gipp, “Bias-aware news analysis using matrix-based news aggregation,” _International Journal on Digital Libraries_ , vol. 21, no. 2, pp. 129–147, 2020.
  * [2] F. Hamborg, C. Breitinger, and B. Gipp, “Giveme5w1h: A universal system for extracting main events from news articles,” _arXiv preprint arXiv:1909.02766_ , 2019.
  * [3] F. Hamborg, N. Meuschke, and B. Gipp, “Matrix-based news aggregation: exploring different news perspectives,” in _2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL)_. IEEE, 2017, pp. 1–10.
  * [4] P. Li, L. Bing, W. Lam, H. Li, and Y. Liao, “Reader-aware multi-document summarization via sparse coding,” _arXiv preprint arXiv:1504.07324_ , 2015\.
  * [5] F. Hamborg, C. Breitinger, M. Schubotz, S. Lachnit, and B. Gipp, “Extraction of main event descriptors from news articles by answering the journalistic five w and one h questions,” in _Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries_ , 2018, pp. 339–340.
  * [6] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 730–27 744, 2022.
  * [7] D. Gao, H. Wang, Y. Li, X. Sun, Y. Qian, B. Ding, and J. Zhou, “Text-to-sql empowered by large language models: A benchmark evaluation,” _arXiv preprint arXiv:2308.15363_ , 2023.
  * [8] Y. Chen, Y. Liu, F. Meng, Y. Chen, J. Xu, and J. Zhou, “Improving translation faithfulness of large language models via augmenting instructions,” _arXiv preprint arXiv:2308.12674_ , 2023.
  * [9] J. Gao, H. Zhao, C. Yu, and R. Xu, “Exploring the feasibility of chatgpt for event extraction. arxiv,” _arXiv preprint arXiv:2303.03836_ , 2023.
  * [10] S. Yang, D. Feng, L. Qiao, Z. Kan, and D. Li, “Exploring pre-trained language models for event extraction and generation,” in _Proceedings of the 57th annual meeting of the association for computational linguistics_ , 2019, pp. 5284–5294.
  * [11] C. Lou, J. Gao, C. Yu, W. Wang, H. Zhao, W. Tu, and R. Xu, “Translation-based implicit annotation projection for zero-shot cross-lingual event argument extraction,” in _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2022, pp. 2076–2081.
  * [12] P. Li, L. Bing, and W. Lam, “Reader-aware multi-document summarization: An enhanced model and the first dataset,” _arXiv preprint arXiv:1708.01065_ , 2017.
  * [13] X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” _arXiv preprint arXiv:2101.00190_ , 2021.
  * [14] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” _arXiv preprint arXiv:2106.09685_ , 2021.
  * [15] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” _arXiv preprint arXiv:2305.14314_ , 2023.
  * [16] F. Hamborg, S. Lachnit, M. Schubotz, T. Hepp, and B. Gipp, “Giveme5w: main event retrieval from news articles by extraction of the five journalistic w questions,” in _International conference on information_. Springer, 2018, pp. 356–366.
  * [17] A. Das, S. Bandyaopadhyay, and B. Gambäck, “The 5w structure for sentiment summarization-visualization-tracking,” in _Computational Linguistics and Intelligent Text Processing: 13th International Conference, CICLing 2012, New Delhi, India, March 11-17, 2012, Proceedings, Part I 13_. Springer, 2012, pp. 540–555.
  * [18] K. Parton, K. McKeown, R. E. Coyne, M. T. Diab, R. Grishman, D. Hakkani-Tür, M. Harper, H. Ji, W. Y. Ma, A. Meyers _et al._ , “Who, what, when, where, why? comparing multiple approaches to the cross-lingual 5w task,” 2009.
  * [19] S. Yaman, D. Hakkani-Tür, G. Tur, R. Grishman, M. Harper, K. R. McKeown, A. Meyers, and K. Sharma, “Classification-based strategies for combining multiple 5-w question answering systems,” in _Tenth Annual Conference of the International Speech Communication Association_. Citeseer, 2009.
  * [20] S. Srivastava, G. Singh, S. Matsumoto, A. Raz, P. Costa, J. Poore, and Z. Yao, “Mailex: Email event and argument extraction,” _arXiv preprint arXiv:2305.13469_ , 2023.
  * [21] B. Li, G. Fang, Y. Yang, Q. Wang, W. Ye, W. Zhao, and S. Zhang, “Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness,” _arXiv preprint arXiv:2304.11633_ , 2023.
  * [22] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” _arXiv preprint arXiv:2109.01652_ , 2021.
  * [23] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou _et al._ , “Chain-of-thought prompting elicits reasoning in large language models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 824–24 837, 2022.
  * [24] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [25] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez _et al._ , “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2023.
  * [26] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [27] G. Klein, F. Hernandez, V. Nguyen, and J. Senellart, “The opennmt neural machine translation toolkit: 2020 edition,” in _Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , 2020, pp. 102–109.
  * [28] R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang _et al._ , “Abstractive text summarization using sequence-to-sequence rnns and beyond,” _arXiv preprint arXiv:1602.06023_ , 2016.
  * [29] S. Narayan, S. B. Cohen, and M. Lapata, “Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization,” _arXiv preprint arXiv:1808.08745_ , 2018.
  * [30] E. Sandhaus, “The new york times annotated corpus,” _Linguistic Data Consortium, Philadelphia_ , vol. 6, no. 12, p. e26752, 2008.
  * [31] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in _Text summarization branches out_ , 2004, pp. 74–81.
  * [32] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.