# SlideVQA: A Dataset for Document Visual Question Answering on Multiple
Images

Anonymous Submission  Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku
Hasegawa, Itsumi Saito, Kuniko Saito

###### Abstract

Visual question answering on document images that contain textual, visual, and
layout information, called document VQA, has received much attention recently.
Although many datasets have been proposed for developing document VQA systems,
most of the existing datasets focus on understanding the content relationships
within a single image and not across multiple images. In this study, we
propose a new multi-image document VQA dataset, SlideVQA, containing 2.6k+
slide decks composed of 52k+ slide images and 14.5k questions about a slide
deck. SlideVQA requires complex reasoning, including single-hop, multi-hop,
and numerical reasoning, and also provides annotated arithmetic expressions of
numerical answers for enhancing the ability of numerical reasoning. Moreover,
we developed a new end-to-end document VQA model that treats evidence
selection and question answering in a unified sequence-to-sequence format.
Experiments on SlideVQA show that our model outperformed existing state-of-
the-art QA models, but that it still has a large gap behind human performance.
We believe that our dataset will facilitate research on document VQA.

## Introduction

Building intelligent agents that can read and comprehend real-world documents,
such as webpages, office documents, lecture slides, etc., has been a long-
standing goal of artificial intelligence. To achieve this goal, machine
reading comprehension (MRC), a central task in natural language understanding,
has been intensively studied. The typical definition of the MRC task is quite
simple, wherein given a short natural language text as a context and a
question about it, a machine reads the text and then answers the question by
extracting a span from the text (Rajpurkar et al. 2016; Rajpurkar, Jia, and
Liang 2018). However, this definition is far from real-world applications,
such as customer service chatbots on e-commerce websites (Cui et al. 2017) and
assistant systems for reading professional literature (Hong et al. 2019), in
that the context is composed entirely of text, with no graphical elements.

To this end, visual question answering on document images (document VQA) has
received much attention. It is a challenging vision and language task that
requires methods to reason about document layout, textual content, and visual
elements (Mathew, Karatzas, and Jawahar 2021; Tanaka, Nishida, and Yoshida
2021; Mathew et al. 2022). When the primary content in a document is text
(e.g., e-mails and forms) and the task is to understand it on the basis of its
layout information, state-of-the-art models have already achieved nearly
human-level performance (Xu et al. 2021; Powalski et al. 2021). On the other
hand, challenges remain when it comes to handling diverse real-world
documents. First and foremost is that current models are not capable of
performing reasoning across multiple images since the existing datasets focus
on testing reasoning ability on a single image. Moreover, compared with
humans, document VQA models still have trouble understanding documents that
contain visual elements and understanding questions that require numerical
reasoning (Mathew et al. 2022).

To address the above challenges, we introduce a new document VQA dataset111Our
dataset and codes are publicly available at https://github.com/nttmdlab-
nlp/SlideVQA, SlideVQA, for tasks wherein given a slide deck composed of
multiple slide images and a corresponding question, a system selects a set of
evidence images and answers the question. Slide decks are one of the most
efficient document types that arrange visual and textual elements for
communication. As shown in Figure 1, SlideVQA requires complex reasoning over
slide images, including single-hop, multi-hop, and numerical reasoning. These
reasoning skills play essential roles in MRC tasks (Yang et al. 2018; Dua et
al. 2019).

Figure 1: Examples from our SlideVQA dataset. Some questions can be answered
through single-hop, multi-hop, and numerical reasoning. The colors of the
words match the image borders with the same colors. ($\cdot$) of the right
example in the answer denotes an annotated arithmetic expression to derive the
final answer. The slide deck can be viewed at
https://www.slideshare.net/mslgroup/mediainsights-evolving-sources-of-news-
for-media.

Our main contributions are summarized as follows:

  * •

We introduce a novel task and dataset, SlideVQA, wherein to answer its
questions, a machine has to read and comprehend a slide deck. It is the
largest multi-image document VQA dataset containing 2.6k+ slide decks (each
consisting of 20 slides) and 14.5k questions. It also provides bounding boxes
around textual and visual elements for understanding document layout and
arithmetic expressions for numerical reasoning.

  * •

We developed a Multi-Modal Multi-image Document VQA model, M3D, to jointly
perform evidence selection and question answering tasks and to enhance
numerical reasoning by generating arithmetic expressions.

  * •

Our model outperformed existing state-of-the-art QA models on SlideVQA, but
its performance is still below that of humans by a large margin.

## Related Work

#### Datasets for VQA on document images.

Document VQA is the task of answering questions about document images, and
some useful datasets have been published, such as DocVQA (Mathew, Karatzas,
and Jawahar 2021), VisualMRC (Tanaka, Nishida, and Yoshida 2021), WebSRC (Chen
et al. 2021), and InfographicVQA (Mathew et al. 2022). The task assumes that
the datasets have a single relevant image, containing all the facts required
to answer.

The work most related to ours is DocCVQA (Tito, Karatzas, and Valveny 2021),
wherein a large collection of document images is used to answer a given
question. Our dataset differs from DocCVQA, as follows. First, SlideVQA
consists of 14.5k questions, wheres DocCVQA provides only 20 questions.
Second, SlideVQA requires multi-hop reasoning over multiple slides to find the
answer, while DocCVQA requires only single-hop reasoning on individual images
to find the answer. Besides these differences, SlideVQA provides questions
that require numerical reasoning and arithmetic expression annotations to
answer numerical questions (e.g., “30 - 28” for the answer “2”): no other VQA
dataset, including InfographicVQA that requires numerical reasoning, provides
such annotations. Furthermore, SlideVQA provides the largest number of
bounding boxes on all of the collected images among the related datasets.

Dataset | Document | Multi-images | Multi-hop | Numerical | Answer | Document images | #QAs | #Images | #BBoxes | #Arithmetic | #Evidence
---|---|---|---|---|---|---|---|---|---|---|---
source | input | reasoning | reasoning | type | modal type | annotations | candidates
DocVQA | industry |  |  |  | SS | TL | 50k | 12k | – | – | 1
VisualMRC | web-pages |  |  |  | Ab | TLV | 30k | 10k | 64k | – | 1
WebSRC | web-pages |  |  |  | SS | TLV | 400k | 6.4k | – | – | 1
InfographicVQA | infographics |  |  | ✓ | SS, MS, NS | TLV | 30k | 5k | – | – | 1
DocCVQA | industry | ✓ |  |  | MS | TL | 0.02k | 14k | – | – | 14k
SlideVQA (Ours) | slide decks | ✓ | ✓ | ✓ | SS, MS, NS | TLV | 14.5k | 52k | 890k | 1.7k | 20

Table 1: Comparison of question answering datasets on document images. Answer
types can be broken down into abstractive (Ab), single-span (SS), multi-span
(MS), and non-span (NS). “T/L/V” denotes the “text/layout/visual” modality of
images.

#### Document VQA Models.

In parallel with the development of datasets, Transformer (Vaswani et al.
2017) has come to be used for understanding unstructured text in document
images. LayoutLM (Xu et al. 2020), LayoutLMv2 (Xu et al. 2021), LayoutT5
(Tanaka, Nishida, and Yoshida 2021), and TILT (Powalski et al. 2021) have
achieved impressive results in single-image document VQA tasks by combining
textual, layout, and visual features. By contrast, we focus on endowing models
with the ability to reason and comprehend multiple images. Moreover, while
Tito, Karatzas, and Valveny (2021) used a pipeline of retrieval and reading
models for DocCVQA, we use multi-task learning that jointly performs evidence
selection and question answering.

#### Multi-modal question answering.

This type takes textual and visual information as input contexts, which is
different from document VQA that takes only a document image as the input
context. TQA (Kembhavi et al. 2017) is comprised of middle-school science
lessons containing diagrams and text. MultiModalQA (Talmor et al. 2021)
requires joint reasoning over text, tables, and images in Wikipedia.

#### VQA on videos or image sets.

VideoQA focuses on answering questions about video frames of TV shows (Lei et
al. 2018, 2020) and movies (Tapaswi et al. 2016). A similar task is VQA on
image sets (ISVQA), which involves handling photos taken from different
viewpoint indoors (Bansal, Zhang, and Chellappa 2020). By contrast, our
dataset also requires a model to understand the text in images.

#### Slide images understanding.

Monica Haurilet and Stiefelhagen (2019); Haurilet et al. (2019) introduced a
benchmark for object segmentation on slide-pages. Sun et al. (2021); Fu et al.
(2022) tackled the task of generating slides from research papers. Our work is
the first to focus on answering questions on sets of slide images.

#### Reasoning over textual documents.

Numerical reasoning plays an important role in NLP tasks (Dua et al. 2019;
Zhang et al. 2020, 2021). Moreover, multi-hop reasoning has taken the
spotlight as it aligns with the multi-hop nature of how humans reason to
acquire knowledge, and has led to a proliferation of benchmarks (Talmor and
Berant 2018; Yang et al. 2018). However, there is as yet no dataset for
developing models to perform both multi-hop and numerical reasoning on
document images.

## The SlideVQA Task and Dataset

### Task Overview and Formulation

The SlideVQA task, requires a system to answer a question about a slide deck,
which is composed of an ordered set of slide images and to select evidence
slide images. We formulate the end-to-end SlideVQA task as follows:
MainTask (SlideVQA). Given a question $q$ and a slide deck
$\mathbf{I}=\\{I_{1},\ldots,I_{K}\\}$ ($K=20$), a model outputs an answer $y$
and selects relevant slides
$\mathbf{\hat{I}}=\\{\hat{I}_{1},\ldots,\hat{I}_{K^{\prime}}\\}$.

The task can be decomposed into two subtasks:

######  Subtask 1 (Evidence Selection).

Given a question $q$ and a slide deck $\mathbf{I}$, a model identifies the
images $\mathbf{\hat{I}}$ from which to derive the answer $y$.

######  Subtask 2 (Question Answering).

Given a question $q$ and the slide images ($\mathbf{I}$ or
$\mathbf{\hat{I}}$), a model outputs an answer $y$.

SlideVQA has three answer types (see the examples in Figure 1). A single-span
answer is a contiguous sequence of tokens in the reading order extracted from
the image, and a multi-span answer is formed from multiple spans from the
image. A non-span answer is not extracted and is composed of numerical values
and visual appearances.

We can also use annotations of bounding boxes around the objects (and their
categories) to understand the semantic structure of images and annotations of
arithmetic expressions to understand numerical reasoning as additional input
at training. These annotations are not given at inference.

### Dataset Collection

In this section, we describe the collection process of the SlideVQA dataset.
To control the annotation quality, we recruited crowd workers located in
English-speaking countries and who had passed a rigorous qualification
procedure. Additionally, we asked other workers to assess the quality of the
annotated samples after each collection step.

#### Slide decks collection.

First, we selected and downloaded 25,327 slide decks composed of more than 20
slides from slideshare222https://www.slideshare.net/ and covering 39 topics.
We kept the first 20 slides and truncated the rest of the pages. Then, the
workers filtered the collected decks that did not meet the following criteria:
(i) the main language is English; (ii) the content is easy for workers to
understand; (iii) the decks must contain one or more graphs, tables, figures,
or numerical data to avoid creating questions requiring only text-level
understanding.

#### Bounding boxes and categories annotation.

Figure 2: Example of collected bounding boxes. Colored boxes and words were
annotated by workers. The image can be viewed at
https://www.slideshare.net/andrybrewok/big-data-analytics-a-social-network-
approach.

To facilitate understanding of the semantic components of images, we annotated
all images with bounding boxes and their categories. The workers indicated
specific objects in each image by annotating bounding boxes around the objects
and classifying them into nine classes that were based on SPaSe (Monica
Haurilet and Stiefelhagen 2019) as follows:

  * •

Title: presentation title, slide title

  * •

Page-text: text in slide, bullet-point text list, text list

  * •

Obj-text: text in a figure, image, diagram or table

  * •

Caption: description of figure, image, diagram, or table

  * •

Other-text: footnote, date, affiliation, code, URL

  * •

Diagram: a graphical representation of data, a process

  * •

Table: data arranged in rows and columns

  * •

Image: drawing, logo, map, screenshot, realistic image

  * •

Figure: graph with data points and coordinates

As shown in Figure 2, SlideVQA provides densely annotated bounding boxes in
images.

(a) Bounding box categories. (b) Reasoning types.

(c) Numerical operation types. (d) Answer types.

Figure 3: Distribution of bounding box categories, reasoning types, numerical
operations, and answer types in the test set.

#### Single-hop QA creation.

We asked the workers to create 12,466 QA pairs by selecting a single slide
image from a slide deck. The selected slide can be used as evidence to tell
whether a system arrived at the right answer for the right reasons. We
encouraged questions that needed numerical reasoning, including operations of
arithmetic expressions with $\\{+,-,/,*\\}$, counting, and comparisons.
Additionally, the workers avoided creating questions that (i) contained
selected page numbers; (ii) required external knowledge; (iii) were common to
all of the slides (e.g., “What is the title?”).

#### Multi-hop questions creation.

We created 2,018 QA pairs for multi-hop reasoning by editing the single-hop
questions created in the previous step. For example at the left of Figure 1,
“North” is replaced by the phrase “the region with 70% of journals”. To this
end, we first identified one or two bridge entities in the created questions,
and the workers selected related slides as evidence that mentioned the
identified ones. Then, the content of the selected slides was utilized to
replace the entities in the created questions. The process of creating multi-
hop questions by editing may produce unnatural questions, as mentioned in the
“Limitations” section, but is easily scalable. A similar approach was taken
with MultiModalQA (Talmor et al. 2021), which requires multi-hop reasoning
over text, tables, and images in Wikipedia.

#### Arithmetic expression annotation.

We provided arithmetic expressions like “30 - 28” in which the final numerical
answer can be arrived at with the four arithmetic operations. The
interpretation of the answer generation process is important for creating
explainable QA models.

### Statistics and Analysis

SlideVQA contains 14,484 QA pairs from 2,619 slide decks, consisting of 52,480
slide images annotated with 890,945 bounding boxes. We split the dataset into
10,617 questions for training, 1,652 (2,215) questions for development (test),
making sure that each deck appears in the same split.

#### Images.

SlideVQA provides the largest number of images covering broad range of topics
among the datasets shown in Table 1. Moreover, SlideVQA provides the largest
number of bounding box annotations, where the number of the annotations in
SlideVQA is 14.7 times that of VisualMRC. Figure 3a shows the distribution of
bounding boxes broken down into nine categories, which cover all classes,
including visually related ones (Image and Figure), unlike DocVQA and DocCVQA.
To analyze the OCR tokens, we extracted the text shown in the images by using
the Google Cloud Vision API333https://cloud.google.com/vision. As a result,
the number of OCR tokens the system should consider simultaneously is larger
(1488.88 tokens) than those of single-image document VQA datasets; the largest
dataset (InfographicVQA) has 217.89 tokens.

Figure 4: Distribution of the first three words of the questions. Figure 5:
(a) Our encoder-decoder model architecture and (b) input representations.
Given a question with a task prefix and a slide deck, the model outputs a
corresponding answer/arithmetic-expression and evidence pages. The calculator
outputs the final answer to calculate the generated arithmetic expression.

#### Questions and answers.

As shown in Table 1, SlideVQA requires complex reasoning including
single/multi-hop, and numerical reasoning. Figure 3b shows the diverse
distribution of questions related to reasoning types. 49.3% of the questions
require multi-hop or numerical reasoning. Moreover, SlideVQA provides
annotations of arithmetic expressions to improve numerical reasoning. Figure
3c shows the distribution of numerical operations. 25.5% of the numerical
questions require arithmetic operations, which current systems have particular
difficulty answering. Figure 3d shows that multi-span and non-span account for
32.4% of the answers, indicating systems also need to generate answers as well
as extract multiple spans.

Figure 4 shows the sunburst pattern of the first three words of the questions.
“In” and “Regarding” are frequent first words because SlideVQA needs to search
for evidence images from a slide deck, which is a special pattern in multi-
text document QA (Yang et al. 2018).

## Our Model

Figure 5 shows an overview of our model, called M3D (Multi-Modal Multi-image
Document VQA model). We use Fusion-in-Decoder (FiD) (Izacard and Grave 2021),
which is a state-of-the-art multi-text encoder-decoder model, as our base
model and initialize FiD with a pre-trained T5 (Raffel et al. 2020). We extend
FiD to perform the end-to-end SlideVQA task (defined in MainTask) by (i)
performing evidence selection and question answering tasks as a unified
sequence-to-sequence format using multi-task learning, (ii) predicting
arithmetic expressions as intermediate reasoning steps instead of generating
answers directly to enhance numerical reasoning, and (iii) modifying the input
sequence to learn the visual layout and content of the image.

### Multi-modal Task-Specific Input

#### Input token sequence.

For each image $I_{k}$, we first use Faster-RCNN (Ren et al. 2015), which was
trained on SlideVQA, to extract $N$ semantic regions (bounding boxes) and
their labels (e.g., Title and Image). We parse the slide image for each
extracted region $r$ by using an OCR engine and apply a sub-word tokenizer to
obtain OCR tokens $\mathbf{W}^{r}_{k}=\\{w^{r}_{k,1},\ldots,w^{r}_{k,n}\\}$
and corresponding OCR bounding boxes. To jointly train the evidence selection
and question answering tasks, we add different task prefixes $t\in$ {Evidence
Selection, Question Answering} to the encoder input. Specifically, the input
sequence is as follows:

$x_{k}=(\texttt{task:}t\texttt{ question:}q\texttt{ page:}e_{k}\texttt{
context:}c_{k}),$

where the sequence concatenates each slide and page number pair ($c_{k}$,
$e_{k}$) with the question $q$ and task prefix $t$. To tell the role of each
region, we insert region labels [R${}^{r_{i}}_{k}$], corresponding to the
region label of the $i$-th region $r_{i}$ in $k$-th page, before the OCR
tokens $\mathbf{W}^{r_{i}}_{k}$ extracted in $r_{i}$:

$c_{k}=([{\rm\texttt{R}}^{r_{1}}_{k}],\mathbf{W}^{r_{1}}_{k},[{\rm\texttt{R}}^{r_{2}}_{k}],\mathbf{W}^{r_{2}}_{k},\dots,[{\rm\texttt{R}}^{r_{N}}_{k}],\mathbf{W}^{r_{N}}_{k})$

#### Input embedding.

Following LayoutT5 (Tanaka, Nishida, and Yoshida 2021), the input embeddings
$\mathbf{z}$ of the encoder are defined by utilizing multi-modal information,
including token $\mathbf{z}^{{\rm token}}$, segment $\mathbf{z}^{{\rm seg}}$,
layout $\mathbf{z}^{{\rm lay}}$, and visual embeddings $\mathbf{z}^{{\rm
vis}}$ as follows:

$\mathbf{z}={\rm LN}(\mathbf{z}^{{\rm token}}+\mathbf{z}^{{\rm
seg}}+\mathbf{z}^{{\rm lay}}+\mathbf{z}^{{\rm vis}})\in\mathbb{R}^{L\times
d},$

where LN is a layer normalization (Ba, Kiros, and Hinton 2016), and $L$ and
$d$ are the length of the input sequence and a hidden vector size,
respectively. The segment embedding indicates which regions are included in
the input sequence. The layout embedding denotes the encoded bounding box
coordinates of the token within the image. We normalize all coordinates by the
size of images and use embedding layers to embed x-axis and y-axis features
separately. The visual embedding is the appearance feature of each region and
the OCR bounding boxes, which were obtained from Faster-RCNN. Note that the
layout and visual embeddings are set to zero vectors for the task prefix,
question, and page number.

### Multi-modal Encoder-Decoder

#### Multi-modal encoder.

Our encoder is a stack of $m$ Transformer blocks, consisting of a self-
attention layer and a fully-connected layer with residual connections.
Following FiD (Izacard and Grave 2021), all $K$ input sequences are encoded
independently and then concatenated to form a unified input representation.
Formally, we transform each input sequence $x_{k}$ into
$\mathbf{x}_{k}\in\mathbb{R}^{L\times d}$ and concatenate them into
$\mathbf{X}\in\mathbb{R}^{K\times L\times d}$.

#### Answer/Arithmetic-expression decoder.

Our decoder is another stack of $m$ Transformer blocks similar to the multi-
modal encoder, where each block has an additional layer of cross-attention
between the output sequence and $\mathbf{X}$. The answer decoder is modeled as
a conditional generation $p_{\theta}(y|\mathbf{X})$, where $\theta$ represents
the set of all model parameters. To allow the model to perform numerical
reasoning, we train the system to predict annotated arithmetic expressions
$y^{\prime}$ (e.g., “$30-28$”) instead of numeric values $y$ (e.g., “$2$”) by
modeling $p_{\theta}(y^{\prime}|\mathbf{X})$. During inference, the model
itself decides whether numerical reasoning is required or not for each
question by predicting an indicator token Answer: or Expression: at the
beginning of the output sequence.

#### Evidence selector.

The selector shares the weights and the architecture of the answer/arithmetic-
expression decoder. Instead of only modeling answer generation, we devise a
simple method to train evidence selection in a unified sequence. Specifically,
we define the output sequence as $\hat{\mathbf{I}}_{\text{pages}}$ $=$
(Evidence pages: $\hat{e}_{1}$, $\ldots$, $\hat{e}_{K^{\prime}}$), where each
$\hat{e}$ is the page number of the selected slide.

#### Training and inference.

Our model is trained by minimizing the weighted sum of two losses
$\mathcal{L}=\mathcal{L}_{\text{dec}}+\mathcal{L}_{\text{sel}}$, where
$\mathcal{L}_{\text{dec}}$ and $\mathcal{L}_{\text{sel}}$ are the negative
log-likelihood between the ground-truth and the prediction regarding the
decoder and selector, respectively. During inference, we obtain the final
prediction to post-process the decoded sequence by removing the task
indicator. If an arithmetic expression is generated (i.e., Expression: is
generated), we use a calculator to obtain the final results.

## Experiments

|  | Dev | Test
---|---|---|---
Model | Modal | JEM | JF1 | JEM | JF1
PreasM | T | 30.2 | 38.2 | 23.4 | 34.7
T5 | T | 30.0 | 38.0 | 22.6 | 34.2
T5 + $\mathbf{z}^{{\rm lay}}$ | TL | 30.9 | 39.5 | 23.6 | 35.7
LayoutT5 | TLV | 31.7 | 39.9 | 24.3 | 36.1
LayoutLMv2† | TLV | 22.8 | 30.8 | 16.5 | 26.5
M3D | TLV | 36.2 | 42.8 | 28.0 | 37.3
M3D${}_{\texttt{GT}}$ | TLV | 44.6 | 50.4 | 35.4 | 44.7
Human | – | – | – | 88.6 | 91.9

(a) Performance of main task.

|  | Dev | Test
---|---|---|---
Model | Modal | EM | F1 | EM | F1
BM25 | T | 40.1 | 46.0 | 35.9 | 47.5
CLIP${}_{\texttt{zero}}$ | V | 33.0 | 34.8 | 30.6 | 34.4
CLIP | V | 40.6 | 43.0 | 39.3 | 43.5
BERT | T | 60.9 | 74.4 | 50.3 | 69.2
BERT + $\mathbf{z}^{{\rm lay}}$ | TL | 61.4 | 75.2 | 52.7 | 71.0
LayoutLM | TL | 51.0 | 63.7 | 42.0 | 59.9
LayoutLMv2 | TLV | 63.3 | 77.1 | 51.7 | 71.5
H-LayoutLMv2 | TLV | 81.1 | 89.5 | 69.8 | 85.6
M3D | TLV | 83.1 | 87.7 | 75.0 | 83.8
Human | – | – | – | 97.7 | 98.0

(b) Performance of evidence selection task.

|  | Dev | Test
---|---|---|---
Model | Modal | EM | F1 | EM | F1
Q-only | – | 9.4 | 11.4 | 10.7 | 13.5
UniVL | V | 8.8 | 12.1 | 10.6 | 14.1
PreasM | T | 36.3 | 41.9 | 30.7 | 38.2
T5 | T | 35.2 | 41.3 | 29.3 | 37.9
T5 + $\mathbf{z}^{{\rm lay}}$ | TL | 36.9 | 43.2 | 31.0 | 39.7
LayoutT5 | TLV | 38.9 | 44.8 | 31.7 | 39.9
LayoutLMv2† | TLV | 26.5 | 33.4 | 21.4 | 29.3
FiD | T | 37.6 | 42.9 | 30.4 | 38.9
FiD + $\mathbf{z}^{{\rm lay}}$ | TL | 38.1 | 43.3 | 30.6 | 38.9
M3D | TLV | 41.3 | 47.1 | 33.5 | 41.7
Human | – | – | – | 89.8 | 93.0

(c) Performance of question answering task.

Table 2: Performance of SlideVQA tasks. “T/L/V” denotes the
“text/layout/visual” modality of images. †denotes the extractive approach. The
pipeline models answer the question based on the top-3 evidences obtained by
H-LayoutLMv2. M3D${}_{\texttt{GT}}$ knows the ground-truth evidence. \+
$\mathbf{z}^{{\rm lay}}$ denotes addition of the layout embedding to the input
embeddings. LayoutLM was not pre-trained in any matching task (e.g., text-
image matching). CLIP${}_{\texttt{zero}}$ denotes CLIP without fine-tuning.

### Experimental Setup

We conducted experiments on the SlideVQA task, evidence selection task, and
question answering task respectively defined in MainTask, Subtasks 1 and 2.

#### Main task baselines.

We mainly evaluated pipeline models as baselines, consisting of evidence
selection that produces top-3 evidences and question answering that takes the
selection results as input. Here, we introduced a hierarchical LayoutLMv2
(H-LayoutLMv2) inspired by (Tu et al. 2020; Xu et al. 2021), which encodes all
slides simultaneously by using another Transformer layer, as the evidence
selector. It achieved 96.0% on Recall@3 on the test set. We used three
generative QA models: a textual model T5 (Raffel et al. 2020), a numerical and
multi-hop model PreasM (Yoran, Talmor, and Berant 2022), and a document VQA
model LayoutT5 (Tanaka, Nishida, and Yoshida 2021). We also used an extractive
document VQA model LayoutLMv2 to predict the single span.

#### Evidence selection baselines.

We also evaluated the evidence selection task alone. BM25 (Robertson, Zaragoza
et al. 2009) is a non-neural retrieval framework to estimate the relevance of
texts to a search query. For the neural models, CLIP (Radford et al. 2021)
encodes the question and each image to predict the highest similar pair. BM25
and CLIP used the top-1 slide as the prediction. BERT (Devlin et al. 2019) is
a pre-trained language model which only uses text information with the
Transformer architecture. LayoutLM (Xu et al. 2020) incorporates layout
information into the input embeddings of BERT. LayoutLMv2 includes image
features produced by a CNN backbone in input embeddings. To model the
interactions between the slides, we used H-LayoutLMv2 described in the
previous section. For neural evidence selection baselines (except for CLIP),
we use a hidden state of [CLS] in the last layer to feed into an MLP
classifier with a sigmoid activation. Evidence is selected if its confidence
of binary classification is above the optimal value on the development set.

To evaluate the effectiveness of our generative evidence selection module, we
introduced BinaryClass as a classification baseline, which uses a two-layer
MLP classifier with a sigmoid activation on top of each encoder representation
at the start-of-sequence. We also introduced a generative baseline, ChainGen,
which generates a sequence of selected slide page numbers before the answer
(Wei et al. 2022).

#### Question answering baselines.

In addition to the pipeline models, we developed Q-only, which takes only the
question into T5. We also used a VideoQA model UniVL (Luo et al. 2020) that
can take all of the slide images as input. Furthermore, we evaluated our base
model FiD (Izacard and Grave 2021).

#### Human performance.

We asked six crowdworkers (not among those recruited to collect our dataset)
to select slide images relevant to the question and answer the question.

#### Evaluation metrics.

Following HotpotQA (Yang et al. 2018), we used exact match (EM) and F1 on each
question answering and evidence selection task and also used Joint EM (JEM)
and Joint F1 (JF1) to evaluate both tasks. These joint metrics penalize models
that perform poorly on either task and assess the accuracy and explainability
of the question answering models.

### Implementation Details

We implemented all of the models in PyTorch and experimented on eight Tesla
V100 32GB GPUs. The size of CLIP was Large and the size of the other models
was Base. We fine-tuned the models using AdamW (Loshchilov and Hutter 2017)
with a learning rate of 5e-5 and a dropout rate of 10%, and we linearly warmed
up the learning rate over 1000 steps. The batch size was set to 32. We
evaluated models every 500 steps and selected the best one on the development
set on the basis of the loss. We used a maximum length of 200 tokens for each
input sequence of M3D, and set the maximum target sequence length to 50. We
trained Faster-RCNN (Ren et al. 2015) with a ResNet-101 (He et al. 2016)
backbone by using stochastic gradient descent (SGD) (Ruder 2016) with a
learning rate of 1e-3 and batch size of one. Standard anchor scales of [8, 16,
32] and anchor ratios of [0.5, 1.0, 2.0] were used. For the VideoQA baseline,
we created a new video at a rate of five frames per second. We used the Google
Cloud Vision API to extract text and bounding boxes from images. When the OCR
word is tokenized into sub-word tokens, the bounding box coordinates of a sub-
word token are the same as those of its whole word.

### Experimental Results and Analysis

#### Does our model outperform the baselines?

Table 2 summarizes the results of the main tasks. As shown in Table 2a, M3D
outperformed the baselines on joint EM/F1, where the metrics evaluate the
consistency between the predicted evidence and answers. For the evidence
selection task, Table 2b shows that H-LayoutLMv2 and M3D performed better than
the baselines. This indicates that modeling the interaction between multiple
slides simultaneously is needed to improve performance. For the QA task, Table
2c shows that M3D outperformed the pipeline methods in all metrics. Our end-
to-end M3D model is better at ignoring the slides irrelevant to the question
than the answer generator in the pipeline methods that strongly depend on the
slides narrowed down by the evidence selector. However, M3D${}_{\texttt{GT}}$
in Table 2a achieved a significant improvement by knowing the ground-truth
slides. There is room for improving the correctness of evidence selection.

Figure 6: Performance of models and humans on the answer types, reasoning
types and numerical operation types in the test set. AE stands for “arithmetic
expression”.

#### What are the characteristics of our dataset?

Table 2 shows that adding modality information tended to improve performance
in all tasks. This demonstrates that SlideVQA requires methods to have the
ability to jointly understand the text, layout, and visual modalities of
documents. As shown in Table 2c, Q-only had the lowest performance, showing
that the systems could not answer the question without reading documents in
the SlideVQA task. Additionally, UniVL has a comparative result to Q-only,
indicating that SlideVQA requires different abilities from VideoQA (Le and Hoi
2020), especially the ability to read texts in images. Tables 2a and 2c show
that LayoutT5, a generative model, significantly outperformed LayoutLMv2, an
extractive approach. This result is inline with observations on the DROP
dataset (Dua et al. 2019), which also has non-span answers (Geva, Gupta, and
Berant 2020). Additionally, all of the models performed all of the tasks
significantly worse than humans. To be specific, Figure 6 illustrates that (i)
better multi-hop reasoning over multiple images is needed and (ii) non-span
answers to questions involving arithmetic operations have to be improved.

| Main | Select | QA
---|---|---|---
Model | JEM | JF1 | EM | F1 | EM | F1
M3D | 36.2 | 42.8 | 83.1 | 87.7 | 41.3 | 47.1
w/o AE generation | 35.7 | 42.3 | 82.9 | 87.7 | 40.5 | 46.3
w/o Evidence selection | – | – | – | – | 40.6 | 46.4
w/o Layout features | 35.1 | 42.0 | 82.4 | 87.1 | 40.3 | 46.3
w/o Visual features | 34.2 | 40.9 | 81.5 | 86.3 | 39.0 | 44.9
w/o Text features | 1.0 | 1.5 | 8.4 | 9.8 | 9.8 | 12.0

Table 3: Ablation study of M3D on dev set.

| Main | Select | QA
---|---|---|---
Model | JEM | JF1 | EM | F1 | EM | F1
M3D backbone | – | – | – | – | 39.0 | 44.8
\+ BinaryClass | 24.7 | 34.8 | 54.5 | 68.5 | 38.8 | 44.8
\+ ChainGen | 34.0 | 40.8 | 81.1 | 86.1 | 39.8 | 45.4
\+ MultiGen (Ours) | 35.7 | 42.3 | 82.9 | 87.7 | 40.5 | 46.3

Table 4: Performance comparison of different evidence selection methods on dev
set.

#### Do our sub-modules improve performance?

Table 3 lists the results of an ablation study. Here, performance consistently
decreased as individual modules were removed from M3D. This indicates that
each of the modules is effective. More precisely, the arithmetic expression
(AE) generation was influential on the QA and Joint performance, meaning that
predicting the arithmetic expression instead of the numerical value enhances
the ability to generate answers with numerical reasoning. As shown in Figure
6, applying AE prediction increased F1 by a large margin (+10.4%) in the
arithmetic type.

#### What are the effective evidence selection methods?

Table 4 shows that our method, which generates the evidence selection and
question answering results separately, obtained the highest performance. It
seems that the generative methods (MultiGen and ChainGen) benefited from the
text-to-text pre-training of T5 more than the classification-based method
(BinaryClass). Our MultiGen decoder that separately trains evidence selection
and question answering had the advantage of being easier to train than the
ChainGen baseline decoder that trains the two tasks as a single sequence
generation task.

#### On which categories does the object detection model not work well?

Class | Dev AP | Test AP
---|---|---
Title | 86.8 | 87.5
Page-text | 76.9 | 76.9
Obj-text | 29.5 | 33.4
Caption | 25.6 | 24.9
Other-text | 40.5 | 39.4
Image | 60.4 | 62.2
Diagram | 65.4 | 64.0
Figure | 74.1 | 68.8
Table | 67.0 | 65.6

Table 5: Object detection performance of Faster-RCNN broken down by bounding
box categories. We set an intersection-over union (IoU) threshold to 0.5.

Table 5 lists the object detection performance of Faster-RCNN broken down by
bounding box categories. These results show that detecting randomly placed and
small boxes, such as Obj-text, is more difficult than mostly fixed and large
boxes, such as Title.

Figure 7: Qualitative example. GT denotes the ground-truth. ($\cdot$) means
the generated arithmetic expression. The slide deck can be viewed at
https://www.slideshare.net/musicbizassoc/nielsen-2015-music-biz-presentation-
final.

#### Qualitative examples.

Figure 7 demonstrates our model’s performance by visualizing a qualitative
example. This example needs multi-hop reasoning and an answer involving an
arithmetic operation. FiD gave an incorrect answer because it did not consider
the visual layout of the slides. Moreover, while LayoutT5 could not understand
the process of getting numerical answers, M3D successfully extracted
information (“11%” and “12%”) and generated the same answer as the ground-
truth.

## Discussion and Limitations

SlideVQA is the largest document VQA benchmark that uses multiple images as
input and requires multi-hop reasoning; its limitation is that the multi-hop
questions created by editing are different from the questions humans might
actually ask the system. We argue that developing models that can reason over
multiple images is an important research direction, and therefore, we employed
an editing method that guarantees multi-hop questions and easily extends the
dataset size. Also, our model uses cross-attention on all evidence candidates,
which may cause a computational problem when there are a lot of input images
(e.g., as in the open-domain QA setting like DocCVQA). To remedy this problem,
we consider that models that train a two-stage selector that roughly narrows
down candidates to a small number of images and then accurately selects
evidence images and an answer generator in an end-to-end manner are promising
(Sachan et al. 2021a, b).

## Conclusion

We introduced a new document VQA dataset, SlideVQA, focused on the task of
understanding slide decks composed of multiple images. We also introduced a
unified end-to-end model, M3D, that can perform evidence selection and
question answering tasks and enhance numerical reasoning by generating
arithmetic expressions. While our evaluation highlighted the promise of this
approach, it also revealed a huge gap compared with human performance, and
several challenges emerge from multi-hop reasoning on multiple images and
generating answers with arithmetic operations. We believe that our dataset
will contribute to the development of intelligent assistant agents that can
comprehend diverse real-world documents.

## References

  * Ba, Kiros, and Hinton (2016) Ba, L. J.; Kiros, R.; and Hinton, G. E. 2016.  Layer Normalization.  _arXiv:1607.06450_.
  * Bansal, Zhang, and Chellappa (2020) Bansal, A.; Zhang, Y.; and Chellappa, R. 2020.  Visual question answering on image sets.  In _ECCV_ , 51–67.
  * Chen et al. (2021) Chen, X.; Zhao, Z.; Chen, L.; Ji, J.; Zhang, D.; Luo, A.; Xiong, Y.; and Yu, K. 2021\.  WebSRC: A Dataset for Web-Based Structural Reading Comprehension.  In _EMNLP_ , 4173–4185.
  * Cui et al. (2017) Cui, L.; Huang, S.; Wei, F.; Tan, C.; Duan, C.; and Zhou, M. 2017.  SuperAgent: A Customer Service Chatbot for E-commerce Websites.  In _ACL_ , 97–102.
  * Devlin et al. (2019) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL-HLT_ , 4171–4186.
  * Dua et al. (2019) Dua, D.; Wang, Y.; Dasigi, P.; Stanovsky, G.; Singh, S.; and Gardner, M. 2019.  DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs.  In _ACL_ , 2368–2378.
  * Fu et al. (2022) Fu, T.; Wang, W. Y.; McDuff, D.; and Song, Y. 2022.  DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents.  In _AAAI_ , 634–642.
  * Geva, Gupta, and Berant (2020) Geva, M.; Gupta, A.; and Berant, J. 2020.  Injecting Numerical Reasoning Skills into Language Models.  In _ACL_ , 946–958.
  * Haurilet et al. (2019) Haurilet, M.; Roitberg, A.; Martinez, M.; and Stiefelhagen, R. 2019.  Wise—slide segmentation in the wild.  In _ICDAR_ , 343–348.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _CVPR_ , 770–778.
  * Hong et al. (2019) Hong, Y.; Wang, J.; Jia, Y.; Zhang, W.; and Wang, X. 2019.  Academic Reader: An Interactive Question Answering System on Academic Literatures.  In _AAAI_ , 9855–9856.
  * Izacard and Grave (2021) Izacard, G.; and Grave, E. 2021.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering.  In _EACL_ , 874–880.
  * Kembhavi et al. (2017) Kembhavi, A.; Seo, M. J.; Schwenk, D.; Choi, J.; Farhadi, A.; and Hajishirzi, H. 2017.  Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension.  In _CVPR_ , 5376–5384.
  * Le and Hoi (2020) Le, H.; and Hoi, S. C. H. 2020.  Video-Grounded Dialogues with Pretrained Generation Language Models.  In _ACL_ , 5842–5848.
  * Lei et al. (2018) Lei, J.; Yu, L.; Bansal, M.; and Berg, T. 2018.  TVQA: Localized, Compositional Video Question Answering.  In _EMNLP_ , 1369–1379.
  * Lei et al. (2020) Lei, J.; Yu, L.; Berg, T.; and Bansal, M. 2020.  TVQA+: Spatio-Temporal Grounding for Video Question Answering.  In _ACL_ , 8211–8225.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv:1711.05101_.
  * Luo et al. (2020) Luo, H.; Ji, L.; Shi, B.; Huang, H.; Duan, N.; Li, T.; Li, J.; Bharti, T.; and Zhou, M. 2020.  UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation.  _arXiv:2002.06353_.
  * Mathew et al. (2022) Mathew, M.; Bagal, V.; Tito, R.; Karatzas, D.; Valveny, E.; and Jawahar, C. 2022\.  InfographicVQA.  In _WACV_ , 1697–1706.
  * Mathew, Karatzas, and Jawahar (2021) Mathew, M.; Karatzas, D.; and Jawahar, C. V. 2021.  DocVQA: A Dataset for VQA on Document Images.  In _WACV_ , 2200–2209.
  * Monica Haurilet and Stiefelhagen (2019) Monica Haurilet, Z. A.-H.; and Stiefelhagen, R. 2019.  SPaSe - Multi-Label Page Segmentation for Presentation Slides.  In _WACV_ , 726–734.
  * Powalski et al. (2021) Powalski, R.; Borchmann, Ł.; Jurkiewicz, D.; Dwojak, T.; Pietruszka, M.; and Pałka, G. 2021.  Going full-tilt boogie on document understanding with text-image-layout transformer.  In _ICDAR_ , 732–747.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_ , 8748–8763.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _JMLR_ , 21(140): 1–67.
  * Rajpurkar, Jia, and Liang (2018) Rajpurkar, P.; Jia, R.; and Liang, P. 2018.  Know What You Don’t Know: Unanswerable Questions for SQuAD.  In _ACL_ , 784–789.
  * Rajpurkar et al. (2016) Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _EMNLP_ , 2383–2392.
  * Ren et al. (2015) Ren; Shaoqing; He; Kaiming; Girshicka; Ross; Sun; and Jian. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _NIPS_ , 91–99.
  * Robertson, Zaragoza et al. (2009) Robertson, S.; Zaragoza, H.; et al. 2009.  The probabilistic relevance framework: BM25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4): 333–389.
  * Ruder (2016) Ruder, S. 2016.  An overview of gradient descent optimization algorithms.  _arXiv:1609.04747_.
  * Sachan et al. (2021a) Sachan, D.; Patwary, M.; Shoeybi, M.; Kant, N.; Ping, W.; Hamilton, W. L.; and Catanzaro, B. 2021a.  End-to-End Training of Neural Retrievers for Open-Domain Question Answering.  In _ACL_ , 6648–6662.
  * Sachan et al. (2021b) Sachan, D. S.; Reddy, S.; Hamilton, W. L.; Dyer, C.; and Yogatama, D. 2021b.  End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering.  In _NeurIPS_ , 25968–25981.
  * Sun et al. (2021) Sun, E.; Hou, Y.; Wang, D.; Zhang, Y.; and Wang, N. X. R. 2021.  D2S: Document-to-Slide Generation Via Query-Based Text Summarization.  In _NAACL-HLT_ , 1405–1418.
  * Talmor and Berant (2018) Talmor, A.; and Berant, J. 2018.  The Web as a Knowledge-Base for Answering Complex Questions.  In _NAACL-HLT_ , 641–651.
  * Talmor et al. (2021) Talmor, A.; Yoran, O.; Catav, A.; Lahav, D.; Wang, Y.; Asai, A.; Ilharco, G.; Hajishirzi, H.; and Berant, J. 2021.  MultiModalQA: complex question answering over text, tables and images.  In _ICLR_.
  * Tanaka, Nishida, and Yoshida (2021) Tanaka, R.; Nishida, K.; and Yoshida, S. 2021.  VisualMRC: Machine Reading Comprehension on Document Images.  In _AAAI_ , 13878–13888.
  * Tapaswi et al. (2016) Tapaswi, M.; Zhu, Y.; Stiefelhagen, R.; Torralba, A.; Urtasun, R.; and Fidler, S. 2016.  Movieqa: Understanding stories in movies through question-answering.  In _CVPR_ , 4631–4640.
  * Tito, Karatzas, and Valveny (2021) Tito, R.; Karatzas, D.; and Valveny, E. 2021.  Document Collection Visual Question Answering.  In _ICADR_ , 778–792.
  * Tu et al. (2020) Tu, M.; Huang, K.; Wang, G.; Huang, J.; He, X.; and Zhou, B. 2020.  Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents.  In _AAAI_ , 9073–9080.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In _NIPS_ , 6000–6010.
  * Wei et al. (2022) Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022\.  Chain of thought prompting elicits reasoning in large language models.  _arXiv:2201.11903_.
  * Xu et al. (2020) Xu, Y.; Li, M.; Cui, L.; Huang, S.; Wei, F.; and Zhou, M. 2020.  LayoutLM: Pre-training of Text and Layout for Document Image Understanding.  In _KDD_ , 1192–1200.
  * Xu et al. (2021) Xu, Y.; Xu, Y.; Lv, T.; Cui, L.; Wei, F.; Wang, G.; Lu, Y.; Florêncio, D. A. F.; Zhang, C.; Che, W.; Zhang, M.; and Zhou, L. 2021.  LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding.  In _ACL/IJCNLP_ , 2579–2591.
  * Yang et al. (2018) Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.; Salakhutdinov, R.; and Manning, C. D. 2018.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.  In _EMNLP_ , 2369–2380.
  * Yoran, Talmor, and Berant (2022) Yoran, O.; Talmor, A.; and Berant, J. 2022.  Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills.  In _ACL_ , 6016–6031.
  * Zhang et al. (2021) Zhang, Q.; Wang, L.; Yu, S.; Wang, S.; Wang, Y.; Jiang, J.; and Lim, E.-P. 2021\.  NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset.  In _Findings of EMNLP_ , 4147–4161.
  * Zhang et al. (2020) Zhang, X.; Ramachandran, D.; Tenney, I.; Elazar, Y.; and Roth, D. 2020.  Do Language Embeddings capture Scales?  In _BlackboxNLP Workshop_.


# A dataset for plain language adaptation of biomedical abstracts

Kush Attal  Lister Hill National Center for Biomedical Communications, U.S.
National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
corresponding author: Kush Attal<EMAIL_ADDRESS> Brian Ondov  Lister
Hill National Center for Biomedical Communications, U.S. National Library of
Medicine, National Institutes of Health, Bethesda, MD, USA  Dina Demner-
Fushman  Lister Hill National Center for Biomedical Communications, U.S.
National Library of Medicine, National Institutes of Health, Bethesda, MD, USA

###### Abstract

Though exponentially growing health-related literature has been made available
to a broad audience online, the language of scientific articles can be
difficult for the general public to understand. Therefore, adapting this
expert-level language into plain language versions is necessary for the public
to reliably comprehend the vast health-related literature. Deep Learning
algorithms for automatic adaptation are a possible solution; however, gold
standard datasets are needed for proper evaluation. Proposed datasets thus far
consist of either pairs of comparable professional- and general public-facing
documents or pairs of semantically similar sentences mined from such
documents. This leads to a trade-off between imperfect alignments and small
test sets. To address this issue, we created the Plain Language Adaptation of
Biomedical Abstracts dataset. This dataset is the first manually adapted
dataset that is both document- and sentence-aligned. The dataset contains 750
adapted abstracts, totaling 7643 sentence pairs. Along with describing the
dataset, we benchmark automatic adaptation on the dataset with state-of-the-
art Deep Learning approaches, setting baselines for future research.

## Background & Summary

While reliable resources for health information conveyed in a plain language
format exist, such as the MedlinePlus website from the National Library of
Medicine (NLM) [1], these resources do not provide all the necessary
information for every health-related situation or rapidly changing state of
knowledge arising from novel scientific investigations or global events like
pandemics. In addition, the language used in other health-related articles can
be too difficult for patients and the general public to comprehend[2], which
has a major impact on health outcomes[3]. While work in simplifying text
exists, the unique language of biomedical text warrants a distinct subtask
similar to machine translation, termed adaptation[4]. Adapting natural
language involves creating a simplified version that maintains the most
important details from a complex source. Adaptations are a common tool for
teachers to use to improve comprehension of content for English language
learners[5].

A standard internet search will return multiple scientific articles that
correspond to a patient’s query; however, without extensive clinical and/or
biological knowledge, the user may not be able to comprehend the scientific
language and content[6]. There are articles with verified, plain language
summaries for health information, such as the articles with corresponding
plain language summaries created by medical health organization Cochrane[7].
However, creating manual summaries and adaptations for every article
addressing every user’s queries is not possible. Thus, an automatic adaptation
generated for material responding to a user’s query is very relevant,
especially for patients without clinical knowledge.

Though plain language thesauri and other knowledge bases have enabled rule-
based systems that substitute difficult terms for more common ones, human
editing is needed to account for grammar, context, and ambiguity[8]. Deep
Learning may offer a solution for fully automated adaptation. Advances in
architectures, hardware, and available data have led neural methods to achieve
state-of-the-art results in many linguistic tasks, including Machine
Translation[9] and Text Simplification[10]. Neural methods, however, require
large numbers of training examples, as well as benchmark datasets to allow
iterative progress[11].

Parallel datasets for Text Simplification have been assembled by searching for
semantically similar sentences across comparable document pairs, for example
articles on the same subject in both Wikipedia and Simple English Wikipedia
(or Vikidia, an encyclopedia for children in several languages)[12, 13, 14,
15]. Since Wikipedia contains some articles on biomedical topics, it has been
proposed to extract subsets of these datasets for use in this domain[16, 17,
18, 19]. However, since these sentence pairs exist in different contexts, they
are often not semantically identical, having undergone sentence-level
operations like splitting or merging. Sentence pairs pulled out of context may
also use anaphora on one side of a pair but not the other. This can confuse
models during training and expect impossible replacements during testing.
Further, Simple English Wikipedia often still contains complex medical terms
on the simple side[16, 20, 21]. Parallel sentences have also been mined from
dedicated biomedical sources. Cao et al. have expert annotators pinpoint
highly similar passages, usually consisting of one or two sentences from each
passage, from Merck Manuals, an online website containing numerous articles on
medical and health topics created for both professional and general public
groups[22]. In addition, Pattisapu et al. have expert annotators identify
highly similar pairs from scientific articles and corresponding health blogs
describing them[23]. Though human filtering makes the pairs in both these
datasets much closer to being semantically identical, at less than 1,000 pairs
each, they are too small for training and even less than ideal for
evaluation[24]. Sakakini et al. manually translate a somewhat larger set
(4,554) of instructions for patients from clinical notes[25]. However, this
corpus covers a very specific case within the clinical domain, which itself
constitutes a separate sublanguage from biomedical literature[26].

Since recent models can handle larger paragraphs, comparable corpora have also
been suggested as training or benchmark datasets for adapting biomedical text.
These corpora consist of pairs of paragraphs or documents that are on the same
topic and make roughly the same points, but are not sentence-aligned. Devaraj
et al. present a paragraph level corpus derived from Cochrane review abstracts
and their Plain Language Summaries, using heuristics to combine subsections
with similar content across the pairs. However, these heuristics do not
guarantee identical content[27]. This dataset is also not sentence-aligned,
which limits the architectures that can take advantage of it and results in
restriction of documents to those with no more than 1024 tokens. Other
datasets include comparable corpora or are created at the paragraph-level and
omit relevant details from the original article[27]. To the best of our
knowledge, no datasets provide manual, sentence-level adaptations of the
scientific abstracts[28]. Thus, there is still a need for a high-quality,
sentence-level gold standard dataset for the adaptation of general biomedical
text.

To address this need, we have developed the Plain Language Adaptation of
Biomedical Abstracts (PLABA) dataset. PLABA contains 750 abstracts from PubMed
(10 on each of 75 topics) and expert-created adaptations at the sentence-
level. Annotators were chosen from the NLM and an external company and given
abstracts within their respective expertise to adapt. Human adaptation allows
us to ensure the parallel nature of the corpus down to sentence-level
granularity, but still while using the surrounding context of the entire
document to guide each translation. We deliberately construct this dataset so
it can serve as a gold standard on several levels:

  1. 1.

Document level simplification. Documents are simplified in total, each by at
least one annotator, who is instructed to carry over all content relevant for
general public understanding of the professional document. This allows the
corpus to be used as a gold standard for systems that operate at the document
level.

  2. 2.

Sentence level simplification. Unlike automatic alignments, these pairings are
ensured to be parallel for the purpose of simplification. Semantically, they
will differ only in (1) content removed from the professional register because
the annotator deemed it unimportant for general public understanding, and (2)
explanation or elaboration added to the general public register to aid
understanding. Since annotators were instructed to keep content within
sentence boundaries (or in split sentences), there are no issues with
fragments of other thoughts spilled over from neighboring sentences on one
side of the pair.

  3. 3.

Sentence-level operations and splitting. Though rare in translation between
languages, sentence-level operations (e.g. merging, deletion, and splitting)
are common in simplification[29]. Splitting is often used to simplify syntax
and reduce sentence length. Occasionally sentences may be dropped from the
general public register altogether (deletion). For consistency and simplicity
of annotation, we do not allow merging, creating a one-to-many relationship at
the sentence level.

The PLABA dataset should further enable the development of systems that
automatically adapt relevant medical texts for patients without prior medical
knowledge. In addition to releasing PLABA, we have evaluated state-of-the-art
deep learning approaches on this dataset to set benchmarks for future
researchers.

## Methods

The PLABA dataset includes 75 health-related questions asked by MedlinePlus
users, 750 PubMed abstracts from relevant scientific articles, and
corresponding human created adaptations of the abstracts. The questions in
PLABA are among the most popular topics from MedlinePlus, ranging from topics
like COVID-19 symptoms to genetic conditions like cystic fibrosis[1].

To gather the PubMed abstracts in PLABA, we first filtered questions from
MedlinePlus logs based on the frequency of general public queries. Then, a
medical informatics expert verified the relevance of and lack of accessible
resources to answer each question and chose 75 questions total. For each
question, the expert coded its focus (COVID-19, cystic fibrosis, compression
devices, etc.) and question type (general information, treatment, prognosis,
etc.) to use as keywords in a PubMed search [30]. Then, the expert selected 10
abstracts from PubMed retrieval results that appropriately addressed the topic
of the question, as seen in Figure 1.

To create the corresponding adaptations for each abstract in PLABA, medical
informatics experts worked with source abstracts separated into individual
sentences to create corresponding adaptations across all 75 questions.
Adaptation guidelines allowed annotators to split long source sentences and
ignore source sentences that were not relevant to the general public. Each
source sentence corresponds to no, one, or multiple sentences in the
adaptation. Creating these adaptations involved syntactic, lexical and
semantic simplifications, which were developed in the context of the entire
abstract. Examples taken from the dataset can be seen in Table 1. Specific
examples of adaptation guidelines are demonstrated in Figure 2 and included:

  * •

Replacing arcane words like "orthosis" with common synonyms like "brace"

  * •

Changing sentence structure from passive voice to active voice

  * •

Omitting or incorporating subheadings at the beginning of sentences (e.g.,
"Aim:", "Purpose:")

  * •

Splitting long, complex sentences into shorter, simpler sentences

  * •

Omitting confidence intervals and other statistical values

  * •

Carrying over understandable sentences from the source with no changes into
the adaptation

  * •

Ignoring sentences that are not relevant to a patient’s understanding of the
text

  * •

Resolving anaphora and pronouns with specific nouns

  * •

Explaining complex terms and abbreviations with explanatory clauses when first
mentioned

## Data Records

We archived the dataset with Open Science Framework (OSF) at
https://osf.io/rnpmf/. The dataset is saved in JSON format and organized or
"keyed" by question ID. Each key is a question ID that contains a
corresponding nested JSON object. This nested object contains the actual
question and contains the abstracts and corresponding human adaptations
grouped by the PubMed ID of the abstract. Table 2 shows statistics of the
abstracts and adaptations. Additional details regarding the data structure can
be found in the README file in the OSF archive.

## Technical Validation

We measured the level of complexity, the ability to train tools and how well
the main points are preserved in the automatic adaptations trained on our
data. We first introduce the metrics we used to measure text complexity
followed by the metrics to measure text similarity and inter-annotator
agreement between manually created adaptations. We use the same text
similarity metrics to also compare automatically created adaptations to both
the source abstracts and manually created adaptations.

### Evaluation metrics

To measure text readability and compare the abstracts and manually created
adaptations, we use the Flesch-Kincaid Grade Level (FKGL) test[31]. FKGL uses
the average number of syllables per word and the average number of words per
sentence to calculate the score. A higher FKGL score for a text indicates a
higher reading comprehension level needed to understand the text.

In addition, we use BLEU[32], ROUGE[33], and SARI[4, 34], commonly used text
similarity and simplification metrics, to measure inter-annotator agreement,
compare abstracts to manually created adaptations, and evaluate the
automatically created adaptations. BLEU and ROUGE look at spans of contiguous
words (referred to as n-grams in Natural Language Processing or NLP) to
evaluate a candidate adaptation against a reference adaptation. For instance,
BLEU-4 measures how many of the contiguous sequences from one to four words in
length in the candidate adaptation appear in the reference adaptation.
However, BLEU is a measure of precision and penalizes candidates for adding
incorrect n-grams. ROUGE is a measure of recall and penalizes candidate
adaptations for missing n-grams. Since neither BLEU nor ROUGE is specifically
designed for simplification, we also use SARI, which also incorporates the
source sentence in order to weight the various operations involved in
simplification. While n-grams are still used, SARI balances (1) addition
operations, in which n-grams of the candidate adaptation are shared with the
reference adaptation but not the source, (2) deletion operations, in which
n-grams appear in the source but neither the reference nor candidate, and (3)
keep operations, in which n-grams are shared by all three. We report BLEU-4,
ROUGE-1, ROUGE-2, ROUGE-L (which measures the longest shared sub-sequence
between a candidate and reference), and SARI. All metrics can account for
multiple possible reference adaptations.

### Text readability

To verify that the human generated adaptations simplify the source abstracts,
we calculated the FKGL readability scores for both the adaptations and
abstracts. FKGL scores were lower for the adaptations compared to the
abstracts (_p_ < 0.0001, Kendall’s tau). It is important to note that FKGL
does not measure similarity or content preservation, so additional metrics
like BLEU, ROUGE, and SARI are needed to address this concern.

### Inter-annotator agreement

To measure inter-annotator agreement, we used adaptions from the most
experienced annotator (who also helped define the guidelines) as reference
adaptations. Agreement was measured for all abstracts that were adapted by
this annotator and another annotator. For the inter-annotator agreement
metrics of ROUGE-1, ROUGE-2, ROUGE-L, and BLEU-4, the values ranged from
0.4025-0.5801, 0.1267-0.2983, 0.2591-0.4689, and 0.0680-0.2410, respectively,
for all adaptations that were done by the reference annotator and another
annotator.As the ROUGE-1 results show, the other annotators included, on
average, about half of the words that the reference annotator used. As
expected, ROUGE-2 values are lower, on average, because as n-grams increase in
n, there will be less similarity between adaptations since individuals may use
different combinations of words when creating new text.

We also calculated the similarity between human adaptations and the source
abstracts. Using the abstracts as candidates and adaptations as references
since BLEU-4 can only match multiple references to a single candidate and not
vice versa, the scores in Table 3 show the adaptations contain over half of
the same words and a third of the same bi-grams as the source abstracts.

While ROUGE and BLEU are metrics for text similarity, they do not necessarily
measure correctness. Even if a pair of adaptations have a low ROUGE or BLEU
score, both could be accurate restatements of the source abstract as seen in
Figure 3. While the BLEU-4 score can be low, both adaptations can relevantly
describe the topic in response to the example question. The differences
between the adaptations can be attributed to synonyms and differences in
explanatory content. While BLEU and ROUGE are useful for measuring lexical
similarity, calculating differences between adaptations like these is more
nuanced. To address this issue, researchers are actively developing new
metrics[35].

### Experimental benchmarking

To benchmark the PLABA dataset and show its use in evaluating automatically
generated adaptations, we used a variety of state-of-the-art deep learning
algorithms listed below:

#### Text-to-Text Transfer Transformer (T5)

T5[36] is a transformer-based[37] encoder-decoder model with a bidirectional
encoder setup similar to BERT[38] and an autoregressive decoder that is
similar to the encoder except with a standard attention mechanism. Instead of
training the model on a single task, T5 is pre-trained on a vast amount of
data and on many unsupervised and supervised objectives, including token and
span masking, classification, reading comprehension, translation, and
summarization. The common feature of every objective is that the task can be
treated as a language-generation task, in which the model learns to generate
the proper textual output in response to the textual prompt included in the
input sequence. As with other models, pre-training has been shown to achieve
state-of-the-art results on many NLP tasks[35][36][39]. In our experiments, we
use the T5-Base model.

#### Pre-training with Extracted Gap-sentences for Abstractive SUmmarization
Sequence-to-sequence (PEGASUS)

PEGASUS[40] is another transformer-based encoder-decoder model; however,
unlike T5, PEGASUS is pre-trained on a unique self-supervised objective. With
this objective, entire sentences are masked from a document and collected as
the output sequence for the remaining sentences of the document. In other
words, PEGASUS is designed for abstractive summarization and similar tasks,
achieving human performance on multiple datasets. In our experiments, we use
the PEGASUS-Large model.

#### Bidirectional autoregressive transformer (BART)

BART[41] is another transformer-based encoder-decoder that is pre-trained with
a different objective. Instead of training the model directly on data with a
text-to-text objective or summarization-specific objective, BART was pre-
trained on tasks such as token deletion and masking, text-infilling, and
sentence permutation. These tasks were developed to improve the model’s
ability to understand the content of text before summarizing or translating
it. After this pre-training, BART can be fine-tuned for downstream tasks of
summarization or translation with a more specific dataset to output higher
quality text. These datasets include the CNN Daily Mail[42] dataset, a large
news article dataset designed for summarization tasks. In our experiments, we
use the BART-Base model and BART-Large model fine-tuned on the CNN Daily Mail
dataset (BART-Large-CNN).

#### Experimental Setup

For our experiments, all deep learning models were trained using the abstracts
and adaptations in the PLABA dataset. Each PubMed abstract is used as the
source document, and the human generated adaptations are used as the
references. The dataset was divided such that 70% was used for training, 15%
for validation, and 15% for testing. We utilized the pre-trained models from
Hugging Face[43], and each model was trained with the AdamW optimizer and the
default learning rate of 5e-5 for 20 epochs using V100X GPUs (32 GB VRAM) on a
shared cluster. Maximum input sequence length was set to 512 tokens except for
the BART models, in which the maximum was set to 1024. Validation loss was
measured every epoch, and the checkpoint model with the lowest validation loss
was used for test set evaluation. An overview of the experiments can be seen
in Figure 4

#### Results

Table 4 shows the FKGL scores between the automatically generated adaptations,
all of which were significantly lower than the abstracts and significantly
higher than the manually crafted adaptations (_p_ < 0.0001, Kendall’s tau).
Table 5 shows the comparison between the automatically generated adaptations
and the human generated adaptations with ROUGE and BLEU and the comparison
between the automatically generated adaptations, human generated adaptations,
and source abstracts with SARI. Table 6 shows the comparison between the
automatically generated adaptations and the source abstracts with ROUGE and
BLEU. It is interesting to note that the automatically generated adaptations
are more readable than the abstracts but less readable than the human
generated adaptations. Thus, the dataset gives the models sufficient training
data to develop outputs that outperform the source abstracts in terms of
readability. Regarding SARI, the models tend to perform comparably in terms of
simplification. In terms of ROUGE and BLEU, the automatically generated
adaptations tend to share more n-grams with the source abstracts rather than
the human generated adaptations. This relationship is potentially because the
abstracts tend to be shorter than the adaptations, as seen in Table 2. This
may make it easier for the automatically generated adaptations to share more
contiguous word sequences with the abstracts relative to the human generated
adaptations. In addition, the choice of metrics used for evaluation will
influence the reported performance of a model.

An example of the automatically generated adaptations from each model in
response to the same abstract is shown in Table 7. These demonstrate that the
PLABA dataset, in addition to being a high-quality test set, is useful for
training generative deep learning models with the objective of text adaptation
of scientific articles. Since there are no existing manually crafted datasets
for this objective, PLABA can be a valuable dataset for benchmarking future
research in this domain.

## Usage Notes

We have added instructions in the README file of our OSF repository that show
how to use the PLABA dataset. Pre-processing the dataset and evaluating
adaptation algorithms on it can be located in the code scripts at our GitHub
repository given below.

## Code availability

Code scripts to pre-process PLABA and reproduce the benchmark results of the
experiments can be found at https://github.com/attal-kush/PLABA.

## References

  * [1] MedlinePlus - Health Information from the National Library of Medicine.
  * [2] Rosenberg, S. A. _et al._ Online patient information from radiation oncology departments is too complex for the general population.  _Practical Radiation Oncology_ 7, 57–62, https://doi.org/10.1016/j.prro.2016.07.008 (2017).
  * [3] Stableford, S. & Mettger, W.  Plain language: a strategic response to the health literacy challenge.  _Journal of public health policy_ 28, 71–93 (2007).
  * [4] Xu, W., Napoles, C., Pavlick, E., Chen, Q. & Callison-Burch, C.  Optimizing Statistical Machine Translation for Text Simplification.  _Transactions of the Association for Computational Linguistics_ 4, 401–415, 10.1162/tacl_a_00107 (2016).
  * [5] Carlo, M. S. _et al._ Closing the gap: Addressing the vocabulary needs of english-language learners in bilingual and mainstream classrooms.  _Reading research quarterly_ 39, 188–215 (2004).
  * [6] White, R. W. & Horvitz, E.  Cyberchondria: Studies of the escalation of medical concerns in Web search.  _ACM Trans. Inf. Syst._ 27, 23:1–23:37, 10.1145/1629096.1629101 (2009).
  * [7] Cochrane Handbook for Systematic Reviews of Interventions.
  * [8] Kauchak, D. & Leroy, G.  A web-based medical text simplification tool.  In _53rd Annual Hawaii International Conference on System Sciences, HICSS 2020_ , 3749–3757 (IEEE Computer Society, 2020).
  * [9] Stahlberg, F.  Neural machine translation: A review.  _Journal of Artificial Intelligence Research_ 69, 343–418 (2020).
  * [10] Al-Thanyyan, S. S. & Azmi, A. M.  Automated text simplification: A survey.  _ACM Computing Surveys (CSUR)_ 54, 1–36 (2021).
  * [11] Savery, M., Abacha, A. B., Gayen, S. & Demner-Fushman, D.  Question-driven summarization of answers to consumer health questions.  _Scientific Data_ 7, 1–9 (2020).
  * [12] Jiang, C., Maddela, M., Lan, W., Zhong, Y. & Xu, W.  Neural CRF Model for Sentence Alignment in Text Simplification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7943–7960 (2020).
  * [13] Coster, W. & Kauchak, D.  Simple English Wikipedia: a new text simplification task.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , 665–669 (2011).
  * [14] Hwang, W., Hajishirzi, H., Ostendorf, M. & Wu, W.  Aligning Sentences from Standard Wikipedia to Simple Wikipedia.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 211–217, 10.3115/v1/N15-1022 (Association for Computational Linguistics, Denver, Colorado, 2015).
  * [15] Zhu, Z., Bernhard, D. & Gurevych, I.  A monolingual tree-based translation model for sentence simplification.  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_ , 1353–1361 (2010).
  * [16] Van, H., Kauchak, D. & Leroy, G.  AutoMeTS: The Autocomplete for Medical Text Simplification.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , 1424–1434, 10.18653/v1/2020.coling-main.122 (International Committee on Computational Linguistics, Barcelona, Spain (Online), 2020).
  * [17] Van den Bercken, L., Sips, R.-J. & Lofi, C.  Evaluating neural text simplification in the medical domain.  In _The World Wide Web Conference_ , 3286–3292 (2019).
  * [18] Adduru, V. _et al._ Towards dataset creation and establishing baselines for sentence-level neural clinical paraphrase generation and simplification.  In _KHD@ IJCAI_ (2018).
  * [19] Cardon, R. & Grabar, N.  Parallel sentence retrieval from comparable corpora for biomedical text simplification.  In _RANLP 2019_ (2019).
  * [20] Xu, W., Callison-Burch, C. & Napoles, C.  Problems in Current Text Simplification Research: New Data Can Help.  _Transactions of the Association for Computational Linguistics_ 3, 283–297, 10.1162/tacl_a_00139 (2015).  Place: Cambridge, MA Publisher: MIT Press.
  * [21] Shardlow, M. & Nawaz, R.  Neural text simplification of clinical letters with a domain specific phrase table.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 380–389, 10.18653/v1/P19-1037 (Association for Computational Linguistics, Florence, Italy, 2019).
  * [22] Cao, Y. _et al._ Expertise Style Transfer: A New Task Towards Better Communication between Experts and Laymen.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 1061–1071 (2020).
  * [23] Pattisapu, N., Prabhu, N., Bhati, S. & Varma, V.  Leveraging Social Media for Medical Text Simplification.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 851–860 (2020).
  * [24] Štajner, S., Sheang, K. C. & Saggion, H.  Sentence simplification capabilities of transfer-based models.  _Proceedings of the AAAI Conference on Artificial Intelligence_ (2022).
  * [25] Sakakini, T. _et al._ Context-Aware Automatic Text Simplification of Health Materials in Low-Resource Domains.  In _Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis_ , 115–126 (2020).
  * [26] Friedman, C., Kra, P. & Rzhetsky, A.  Two biomedical sublanguages: a description based on the theories of Zellig Harris.  _Journal of Biomedical Informatics_ 35, 222–235, https://doi.org/10.1016/S1532-0464(03)00012-1 (2002).
  * [27] Basu, C., Vasu, R., Yasunaga, M., Kim, S. & Yang, Q.  Automatic medical text simplification: Challenges of data quality and curation.  In _HUMAN@ AAAI Fall Symposium_ (2021).
  * [28] Ondov, B., Attal, K. & Demner-Fushman, D.  A survey of automated methods for biomedical text simplification.  _Journal of the American Medical Informatics Association_ 29, 1976–1988 (2022).
  * [29] Frankenberg-Garcia, A.  A corpus study of splitting and joining sentences in translation.  _Corpora_ 14, 1–30 (2019).  Publisher: Edinburgh University Press The Tun-Holyrood Road, 12 (2f) Jackson’s Entry ….
  * [30] Deardorff, A., Masterton, K., Roberts, K., Kilicoglu, H. & Demner-Fushman, D.  A protocol-driven approach to automatically finding authoritative answers to consumer health questions in online resources.  _Journal of the Association for Information Science and Technology_ 68, 1724–1736, https://doi.org/10.1002/asi.23806 (2017).  https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.23806.
  * [31] Flesch, R.  A new readability yardstick.  _Journal of Applied Psychology_ 32, 221–233, 10.1037/h0057532 (1948).  Place: US Publisher: American Psychological Association.
  * [32] Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J.  Bleu: a Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , 311–318, 10.3115/1073083.1073135 (Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002).
  * [33] Lin, C.-Y.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text Summarization Branches Out_ , 74–81 (Association for Computational Linguistics, Barcelona, Spain, 2004).
  * [34] Sun, R., Jin, H. & Wan, X.  Document-Level Text Simplification: Dataset, Criteria and Baseline.  _arXiv:2110.05071 [cs]_ (2021).  ArXiv: 2110.05071.
  * [35] Kryscinski, W., McCann, B., Xiong, C. & Socher, R.  Evaluating the Factual Consistency of Abstractive Text Summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 9332–9346, 10.18653/v1/2020.emnlp-main.750 (Association for Computational Linguistics, Online, 2020).
  * [36] Raffel, C. _et al._ Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _arXiv:1910.10683 [cs, stat]_ (2020).  ArXiv: 1910.10683.
  * [37] Vaswani, A. _et al._ Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * [38] Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv:1810.04805 [cs]_ (2019).  ArXiv: 1810.04805.
  * [39] Radford, A. _et al._ Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 9 (2019).
  * [40] Zhang, J., Zhao, Y., Saleh, M. & Liu, P. J.  PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization.  _arXiv:1912.08777 [cs]_ (2020).  ArXiv: 1912.08777.
  * [41] Lewis, M. _et al._ BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7871–7880, 10.18653/v1/2020.acl-main.703 (Association for Computational Linguistics, Online, 2020).
  * [42] Nallapati, R., Zhou, B., santos, C. N. d., Gulcehre, C. & Xiang, B.  Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond.  _arXiv:1602.06023 [cs]_ (2016).  ArXiv: 1602.06023 version: 5.
  * [43] Wolf, T. _et al._ Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 38–45, 10.18653/v1/2020.emnlp-demos.6 (Association for Computational Linguistics, Online, 2020).

## Acknowledgements

This work was supported by the Intramural Research Program of the National
Library of Medicine (NLM), National Institutes of Health, and utilized the
computational resources of the NIH HPC Biowulf cluster (http://hpc.nih.gov).

## Author contributions statement

K.A. created the code scripts for data pre-processing and deep learning
experiments, contributed to adaptation guidelines, contributed to creating the
manual adaptations, and wrote and edited the manuscript. B.O. contributed to
adaptation guidelines and edited the manuscript. D.D.-F conceived the project,
edited the manuscript, and provided feedback at all stages of the study.

## Competing interests

The authors declare no competing interests.

## Figures & Tables

Figure 1: Overview representing how questions and PubMed abstracts for the dataset were searched and chosen for annotators to adapt. PMID refers to the PubMed ID from which the example originates from. SID refers to the sentence ID or number of the example sentence from the source abstract. Figure 2: Example of the guidelines set for annotators. PMID refers to the PubMed ID from which the example originates from. SID refers to the sentence ID or number of the example sentence from the source abstract. Target refers to the manual adaptation. Figure 3: Example of the low BLEU-4 score between human adaptations from two different annotators created from the same source abstract and answering the same question. PMID refers to the PubMed ID from which the example originates from. SID refers to the sentence ID or number of the example sentence from the source abstract. Colored text in an adaptation represents parts of the adaptation that strongly differ from the other adaptation. Figure 4: Overview representing how PubMed abstracts and human adaptations are split for training and testing models. Data Type |  Text
---|---
Question |  _How is strep throat treated?_
Abstract |  Most patients who seek medical attention for sore throat are concerned about streptococcal tonsillopharyngitis, but fewer than 10% of adults and 30% of children actually have a streptococcal infection. Group A beta-hemolytic streptococci (GAS) are most often responsible for bacterial tonsillopharyngitis, although Neisseria gonorrhea, Arcanobacterium haemolyticum (formerly Corynebacterium haemolyticum), Chlamydia pneumoniae (TWAR agent), and Mycoplasma pneumoniae have also been suggested as possible, infrequent, sporadic […]
Adaptation |  Most people who go to a doctor for sore throat are worried they have a strep throat and tonsil infection, but fewer than 10% of adults and 30% of children actually have a strep infection. Group A strep bacteria are the most common cause of bacterial strep throat and tonsil infection, but other bacteria known to cause sexually-transmitted gonorrhea or chlamydia, or head, neck, and lung infections occasionally might […]
Question |  _How do you assist an unconscious victim who is already vomiting?_
Abstract |  The tongue is the most common cause of upper airway obstruction, a situation seen most often in patients who are comatose or who have suffered cardiopulmonary arrest. Other common causes of upper airway obstruction include edema of the oropharynx and larynx, trauma, foreign body, and […]
Adaptation |  The tongue is the most common cause of blocked upper airways, seen most often in people in comas or cardiac arrest (abrupt heart stop). Other common causes of blocked upper airways include swelling of the middle part of the throat and voice box, injury, objects that shouldn’t be swallowed, and […]
Table 1: Examples of questions, abstracts, and adaptations in PLABA. Data Type | Count | Words | Sentences
---|---|---|---
|  | Average | S.d. | Average | S.d.
Questions | 75 | 10 | 6 | 1 | 0
Abstracts | 750 | 240 | 95 | 10 | 4
Adaptations | 921 | 244 | 95 | 12 | 5
Table 2: Average number of words and sentences per data type. Comparison Type | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-4
---|---|---|---|---
Adaptations vs Abstracts | 0.58 | 0.36 | 0.50 | 0.39
Table 3: ROUGE-1, ROUGE-2, ROUGE-L, and BLEU-4 using human adaptations as references and abstracts as candidates. Data Type | FKGL
---|---
| Average | S.d.
Abstracts | 15.78 | 8.06
Adaptations | 12.04 | 2.39
T5 | 13.83 | 2.93
PEGASUS | 13.84 | 2.80
BART-Base | 13.37 | 2.59
BART-Large-CNN | 13.19 | 2.66
Table 4: FKGL scores for automatically generated adaptations. Algorithm Type | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-4 | SARI
---|---|---|---|---|---
T5 | 0.56 | 0.30 | 0.42 | 0.28 | 0.34
PEGASUS | 0.57 | 0.31 | 0.43 | 0.30 | 0.32
BART-Base | 0.56 | 0.31 | 0.43 | 0.29 | 0.33
BART-Large-CNN | 0.58 | 0.31 | 0.42 | 0.29 | 0.26
Table 5: Automatically generated adaptations compared to human adaptations and (only for SARI) source abstracts. Algorithm Type | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-4
---|---|---|---|---
T5 | 0.77 | 0.62 | 0.74 | 0.55
PEGASUS | 0.79 | 0.64 | 0.74 | 0.60
BART-Base | 0.76 | 0.62 | 0.73 | 0.57
BART-Large-CNN | 0.73 | 0.56 | 0.69 | 0.50
Table 6: Automatically generated adaptations compared to source abstracts. Algorithm Type |  Generated Text
---|---
Question |  _How to treat a bakers cyst?_
PEGASUS |  Osteoarthritis of the knee is a common disease, often associated with a Baker’s cyst. Besides osteoarthritis, also other joint diseases (diseases that cause pain and swelling in the joints) can be […] A decrease in volume more than 25% compared to the baseline could be achieved for 75% of patients. Most patients respond to treatment. Radiation therapy of knee osteoarthritis (damage to cartilage and bone) is effective that decreases the number of affected knees. However, whether radiotherapy is the best treatment is still being studied.
T5 |  Osteoarthritis of the knee (knee arthritis) is a common disease. It is often associated with the Baker’s cyst (a cyst in the back of your knee). Besides arthritis, also other joint diseases of knees […] A decrease of more than 25% compared to the baseline could be achieved for 75% of patients in short term and 79% of people in long term follow up. Conclusion: Radiation therapy is an efficient treatment that decreases the amount of swollen area. Most patients respond to this treatment. Whether radiation is effective for Baker’cyst without associated arthritis must be further examined.
BART-Base |  Osteoarthritis of the knee is a common disease, often associated with a Baker’s cyst (a small, bone-related lump). Also, other joints in the leg can be causative […] A decrease of more than 25% compared to the baseline could be achieved for 75% of patients at the beginning and 79% (long-lasting) after the treatment. This conclusion is that radiation therapy of knee osteosarcoma is an important treatment that decreases the amount and severity of an osteoporosis cysteostasis. Most patients respond to it. However, it is still unclear how effective this treatment is.
BART-Large-CNN |  Osteoarthritis of the knee is a common disease, often associated with a Baker’s cyst, a growth of swollen tissues in the joints. Besides bone pain, other problems with the ligament that runs between the thigh bone and knee joint can cause […] A decrease in volume of more than 25% compared to the baseline could be achieved for 75% of patients in both the short term and the long term. In conclusion, radiation therapy of knee bone arthritis is an helpful treatment that decreases the size of an acchly Cyst. Most patients respond to treatment. Whether radiation is the best treatment to treat a Bakty for a kyst without the associated pain caused by bone damage is further examined.
Question |  _Why is transferrin and iron low in covid patients but ferritin high?_
PEGASUS |  Ferritin, which includes twenty-four light and heavy chains in varying proportions in different tissues, is primarily responsible for maintaining the body’s iron metabolism. […] It can be combined with monoclonal antibodies, antioxidants, corticosteroids (antibodies that fight inflammation) and lactoferrin (a type of protein that protects cells from infection). In this article, we analyze the antiviral and antifibrotic activity of iron-chelator drugs, thereby promoting iron depletion therapy as a potentially innovative treatment strategy for Covid-19 (chronic kidney disease).
T5 |  Ferritin, which includes twenty-four light and heavy chains in varying proportions in different tissues, is primarily responsible for maintaining the body’s iron metabolism. […] This can be combined with monoclonal antibodies, antioxidants, corticosteroids (glucosides) and lactoferrin to make iron therapy effective for covid-19 victims. In this article, we analyze the antiviral and antifibrotic activity of iron agents, thereby promoting iron depletion therapy as potentially innovative treatment strategy for the covd-19.
BART-Base |  Ferritin, which includes twenty-four light and heavy chains in varying proportions in different tissues, is primarily responsible for maintaining the body’s iron metabolism. […] These drugs can be combined with monoclonal antibodies, antioxidants, corticosteroids (antibody drugs that help fight diseases that attack the brain and spinal cord) to make iron-chelation therapy effective for Covid-18 victims. In this article, we analyze the antiviral and antifibrotic (anti-seizure) activity of these drugs, thereby promoting iron depletion therapy as a potentially innovative treatment strategy for COV-17.
BART-Large-CNN |  Ferritin, which includes twenty-four light and heavy chains in varying proportions in different tissues, is mainly responsible for maintaining the body’s iron metabolism. […] It can be combined with antibodies, antioxidants, corticosteroids (drugs that fight inflammation), and lactoferrin to make iron removal therapy effective for patients. In this article, we analyze the antiviral and antifibrotic (redness and swelling from infection-fighting) activity of iron chylides, thereby promoting iron reduction therapy as a potentially new treatment strategy for Covid-9.
Table 7: Examples of adaptations created by PEGASUS, T5, BART-Base, BART-
Large-CNN.