Advance Access Publication Date: Day Month Year Manuscript Category

∗To whom correspondence should be addressed.

Received on XXXXX; revised on XXXXX; accepted on XXXXX

Associate Editor: XXXXXXX

# Scientific Discourse Tagging for Evidence Extraction

Data and Text Mining

Xiangci Li ${}^{\text{\sfb 1}*}$ Gully Burns ${}^{\text{\sfb 2}}$ and Nanyun
Peng ${}^{\text{\sfb 1,3}}$ ${}^{\text{\sf 1}}$Information Sciences Institute,
Viterbi School of Engineering, University of Southern California, Marina del
Rey, CA, 90292, USA
${}^{\text{\sf 2}}$Chan Zuckerberg Initiative, Redwood City, CA, 94063, USA
${}^{\text{\sf 3}}$The University of California, Los Angeles, Los Angeles, CA,
90095, USA

(2015; 2015)

###### Abstract

Motivation: Evidence plays a crucial role in any biomedical research
narrative, providing justification for some claims and refutation for others.
We seek to build models of scientific argument using information extraction
methods from full-text papers. We present the capability of automatically
extracting text fragments from primary research papers that describe the
evidence presented in that paper’s figures, which arguably provides the raw
material of any scientific argument made within the paper.
Results: We apply richly contextualized deep representation learning pre-
trained on biomedical domain corpus to the analysis of scientific discourse
structures and the extraction of ‘‘evidence fragments’’ (i.e., the text in the
results section describing data presented in a specified subfigure) from a set
of biomedical experimental research articles. We first demonstrate our state-
of-the-art scientific discourse tagger on two scientific discourse tagging
datasets and its transferability to new datasets. We then show the benefit of
leveraging scientific discourse tags for downstream tasks such as claim-
extraction and evidence fragment detection. Our work demonstrates the
potential of using evidence fragments derived from figure spans for improving
the quality of scientific claims by cataloging, indexing and reusing evidence
fragments as independent documents.

Availability: Our source code is available at
https://github.com/jacklxc/ScientificDiscourseTagging.
Contact: lixiangci8<EMAIL_ADDRESS>

Supplementary information: Supplementary data are available at Bioinformatics
online.

##  1 Introduction

Primary experimental articles (i.e., papers that describe original
experimental work) provide the crucial raw material for all other subsequent
scientific research. However, the drastically growing number of scientific
literature makes it increasingly difficult for domain experts to efficiently
utilize them. Particularly during the emergency periods such as the outbreak
of COVID-19, it is crucial to promptly and accurately extract information from
the relevant literature in order to discover promising new directions to treat
the disease as quickly as possible. Automatic information extraction from
biomedical literature is a crucial step to help researchers to achieve this
goal.

Extracting important information from biomedical literature to facilitate and
accelerate scientific discovery has been a goal for computational linguistics
for some time Hobbs (2002), with the focus of identifying relevant entities,
relations, and events from text to populate a knowledge base. However, these
methods do not take into account the fact that scientific work involves
attempting to provide explanations for evidence derived from experiments and
is therefore driven principally by authors attempting to convince expert
readers that their claims are the ‘‘correct’’ explanations for the
experimental evidence. Thus, an important aspect of building machines capable
of understanding scientific literature is first recognizing different
rhetorical components of scientific discourse, with which we will then be able
to distinguish the observations made in experiments from their implications
and distinguish between claims supported by evidence and hypotheses put
forward to prompt further research. It is this goal, of being able to
distinguish between the different rhetorical components of scientific
discourses so that we can build AI systems to facilitate more accurate
analysis and understanding of scientific literature, that motivates our work.

Scientific discourse tagging is a task that tags clauses or sentences in a
scientific article with different rhetorical components of scientific
discourses. Figure 1 shows an example of a paragraph with discourse tags.

In this work, we leverage a state-of-the-art contextualized word embedding and
a novel word-to-sentence attention mechanism to develop a model for scientific
discourse tagging that achieves the state-of-the-art performances on two
benchmark datasets SciDT Dasigi et al. (2017) and PubMed-20k-RCT Jin and
Szolovits (2018) by 6.9% and 2.3% absolute F1 respectively. More importantly,
we show the strong transferability of our scientific discourse tagger to new
datasets by beating the baseline Huang et al. (2020) via zero-shot prediction
on CODA-19 dataset Huang et al. (2020). Furthermore, we demonstrate the
effectiveness of scientific discourse tagging on two downstream scientific
literature understanding tasks: claim-extraction and evidence fragment
detection, and and demonstrate the benefit of leveraging scientific discourse
tags information. In particular, we outperform the state-of-the-art claim
extraction model Achakulvisut et al. (2019) by 3.8% F1, and outperform figure
span detection baseline Burns et al. (2017) by 5% F1.

---

Figure 1: An example paragraph tagged with scientific discourse tags on each
clause in SciDT dataset Dasigi et al. (2017). The text is tokenized and
converted to lower case.

##  2 Background and Related Works

Problem Formulation. We define ‘scientific discourse tagging‘ as a task that
labels sentences in a scientific article based on its rhetorical elements of
scientific discourse. Formally, a paragraph can be represented as an ordered
collection of sequences $\textbf{S}=[S_{1},S_{2},...,S_{n}]$, and each element
$S_{i}$ is annotated with a discourse label
$L_{i}\in\\{L_{1},L_{2},...,L_{k}\\}$. Note that $S_{i}$ may be defined
differently in different datasets – e.g., sentences in the PubMed-RCT dataset
Dernoncourt and Lee (2017), clauses in the SciDT dataset composed by Burns et
al. (2016) and Dasigi et al. (2017), and sentence fragments in CODA-19 dataset
Huang et al. (2020). For conciseness, we refer all these variations as
_sentences_. The labels also can be slightly different. For example, in
PubMed-RCT, $\textbf{L}=$ {_objective_ , _background_ , _methods_ , _results_
, _conclusions_}, in CODA-19 Huang et al. (2020), $\textbf{L}=$ {_background_
, _purpose_ , _method_ , _finding/contribution_ , _other_} while in SciDT
dataset Burns et al. (2016); Dasigi et al. (2017), the labels $\textbf{L}=$
{_goal_ , _fact_ , _hypothesis_ , _problem_ , _method_ , _result_ ,
_implication_ , _none_} as defined by De Waard and Maat (2012). Table 1 gives
more details about the definitions of the tags.

Type | Definition
---|---
Goal | Research goal
Fact | A known fact, a statement taken
| to be true by the author
Result | The outcome of an experiment
Hypothesis | A claim proposed by the author
Method | Experimental method
Problem | An unresolved or
| contradictory issue
Implication | An interpretation of the results
None | Anything else
Table 1: Eight label taxonomy defined by De Waard and Maat (2012).

###  2.1 Prior Works on Scientific Discourse Tagging

Feature-based Scientific Discourse Tagging.

There has been a significant amount of work aimed at understanding types of
scientific discourse. Teufel and Moens (1999) and Teufel and Moens (2002)
described argumentative zoning, which groups sentences into a few rhetorical
zones highlighted by important clauses such as ‘‘in this paper we develop a
method for’’. Hirohata et al. (2008) used conditional random field (CRF)
Lafferty et al. (2001) with handcrafted features to classify sentences in
abstracts into 4 categories: _objective, methods, results_ , and
_conclusions_. Liakata (2010) defined ‘‘zone of conceptualization’’ which
classifies sentences into 11 categories in scientific papers and Liakata et
al. (2012) used CRF and LibSVM to identify these ‘‘zone of conceptualization".
Guo et al. (2010) used Naive Bayes and Support Vector Machine (SVM) Cortes and
Vapnik (1995) to compare three schema described above. Burns et al. (2016)
studied the problem of scientific discourse tagging, which identifies the
discourse type of each clause in a biomedical experiment paragraph and
composed a dataset for it. They adopted the discourse type taxonomy for
biomedical papers proposed by De Waard and Maat (2012). The taxonomy contains
eight types including _goal, fact, result, hypothesis, method, problem,
implication_ and _none_ as Table 1 shows. Most recently, Cox et al. (2017)
used the same schema De Waard and Maat (2012) by exploring a variety of
methods for balancing classes before applying classification algorithms.

Deep Learning for Scientific Discourse Tagging.

Due to the prevalence of deep learning, neural sequence labeling approach
using bidirectional LSTM Hochreiter and Schmidhuber (1997) and CRF (BiLSTM-
CRF) Huang et al. (2015) has been prevailing for classic word-level sequence
tagging problems such as named entity recognition (NER) and part of speech
tagging (POS) Huang et al. (2015); Ma and Hovy (2016); Chiu and Nichols
(2016). Since scientific discourse tagging, which is a sentence-level sequence
tagging problem, has one additional dimension of input comparing to word-level
sequence tagging problems, an encoder is required to encode word-level
representations to clause/sentence-level representations. While one simple way
is to pre-compute sentence embeddings from word embeddings Arora et al.
(2016), there are more sophisticated methods to compute sentence-level
embeddings on-the-fly using BiLSTM Jin and Szolovits (2018); Srivastava et al.
(2019) or attention Dasigi et al. (2017), before feeding them into a
clause/sentence-level sequence tagger. Alternatively, as BERT Devlin et al.
(2018) prevails among various natural language processing (NLP) tasks, a
simple baseline method is to directly use a BERT-like model’s (e.g. SciBERT
Beltagy et al. (2019)) prefix token ($[CLS]$) representation of each sentence
as the sentence representation for classification task Huang et al. (2020). In
this work, we combine these methods to present a state-of-the-art scientific
discourse tagger.

###  2.2 Downstream Applications

Claim Extraction. Claim extraction has been extensively studied in various
domains. In addition to scientific articles Stab et al. (2014), previous work
has analyzed social media Dusmanu et al. (2017), news Habernal et al. (2014);
Sardianos et al. (2015) and Wikipedia Thorne et al. (2018); Fréard et al.
(2010) for a task called _Argumentation Mining_ to extract claims and
premises. However, there are less attention and dataset available in the
biomedical domain. Achakulvisut et al. (2019) composed a claim-extraction
dataset derived from MEDLINE 111https://www.nlm.nih.gov/bsd/medline.html paper
abstracts, and proposed a neural network model that significantly outperformed
the rule-based method proposed by Sateli and Witte (2015). Figure 2 shows an
example abstract with the last two sentences annotated as claims.

In this work, we formulate claim extraction Achakulvisut et al. (2019)
similarly as scientific discourse tagging: S contains sentences and
$L_{i}\in\\{0,1\\}$ indicates whether the corresponding sentence is a claim or
not.

Figure 2: An example abstract with claim sentences highlighted in claim-
extraction dataset Achakulvisut et al. (2019).

Evidence Fragment detection. Burns et al. (2017) coined the concept of
‘‘evidence fragments’’ as the text section in narrative surrounding a figure
reference that directly describes the experimental figure. They composed an
evidence fragment detection dataset, and proposed the evidence fragment
detection task that tags each clause with semantically referred subfigure
codes. They further proposed a rule-based method of using these subfigure
codes as anchors to link evidence fragments to European Bioinformatics
Institute’s INTACT Orchard et al. (2013) data records. As a result, INTACT’s
preexisting, manually-curated structured interaction data can serve as a gold
standard for machine reading experiments.

Burns et al. (2017) formulated the problem into a clause-level tagging
problem. Formally, each clause $S_{i}$ in a paragraph
$\textbf{S}=[S_{1},S_{2},...,S_{n}]$ is annotated with a set of subfigure
codes $f^{i}=\\{f_{1}^{i},f_{2}^{i},...,f_{m}^{i}\\}$ that each clause is
semantically referring to, where the length $m$ can be any non-negative
integer. Figure 3 shows an illustration of a paragraph of evidence fragment
detection annotation. Each clause in the paragraph is associated with a set of
semantically relevant subfigures.

Figure 3: An example paragraph of evidence fragment detection. The explicit
mention of subfigure codes are underlined. The red lines indicate the borders
of the evidence fragments. For each clause, the discourse type as well as the
BIO tags indicating ‘‘blocks’’ (see Section 3.2.2) are provided. For
simplicity, trivial clauses are omitted (replaced by ……) and long clauses are
truncated (replaced by …).

##  3 Approaches

In this section, we present our model for scientific discourse tagging tasks.

Figure 4: High-level overview of our scientific discourse tagger. Dashed
arrows indicate we may apply dropout in those connections.

###  3.1 Scientific Discourse Tagger

Model Overview. We formulate scientific tagging as a sentence level sequence
tagging problem. We develop a deep structured model extending Dasigi et al.
(2017), which consists of a contextualized word embedding layer, an attention
layer that summarizes word embeddings into sentence embeddings, and a BiLSTM-
CRF sequence tagger Huang et al. (2015) on top of the sentence embeddings for
discourse type tagging. Figure 4 gives an overview of the architecture. We
detail each component in this section.

Embeddings. Pre-trained word embeddings can transfer the lexical semantics of
large unlabeled datasets to downstream models, which is especially beneficial
for tasks on low resource domains as it reduces the need for training data.
Static embeddings such as GloVe Pennington et al. (2014) and recent
contextualized embedding methods such as BERT Devlin et al. (2018) have been
widely used. Comparing to static embedding methods, contextualized word
embedding produces different word vectors for the same word under different
contexts and thus can provide richer representations to each word. We explore
pre-trained BioGloVe embedding Burns et al. (2019), BioBERT Lee et al. (2019)
and SciBERT Beltagy et al. (2019) embedding, which are GloVe and BERT
embeddings trained on the text in biomedical domain.

Sentence Representations via Attention. We observe that certain keywords are
essential to determine the discourse types. Attention is a mechanism for
emphasizing certain inputs and ignoring others. Dasigi et al. (2017) explored
an attention mechanism to summarize word representations to clause
representations. We propose a new variation of attention mechanism, LSTM-
attention, which leverages an LSTM Hochreiter and Schmidhuber (1997) to
provide richer contextualization of the words in each sentence. Conceptually,
we first encode a sentence using an LSTM to get contextualized hidden vectors
of each word, and use them to compute the attention score on each word. We
then apply the attention to summarize the word embeddings into a sentence
embedding.

The dashed circle in Figure 4 illustrates our LSTM-Attention based word-to-
sentence encoder. We first project each input word vector into a lower
dimensional space $D_{l}$ using a projection matrix $P$ of shape $d\times p$.

$\small D_{l}=tanh(D\cdot P)\in\mathbb{R}^{c\times w\times p}\\\ $

Then we score each word in the $i^{th}$ sentence in the context of other words
in the same sentence using an LSTM. The score for each word is a function of
its $p$ dimensional representation $W_{j}$ and the previous words in the
clause represented by the hidden states $h^{i}_{j-1}$ in the LSTM cell. The
equations are the following:

$\displaystyle D^{i}_{l}=D_{l}[i,:,:]\in\mathbb{R}^{w\times p}$ $\displaystyle
W_{j}=D^{i}_{l}[j,:]\in\mathbb{R}^{p}$ $\displaystyle
h^{i}_{j}=LSTM(W_{j},\>h^{i}_{j-1})\in\mathbb{R}^{h}$ $\displaystyle
h^{i}=[h^{i}_{1}\>h^{i}_{2}\>...\>h^{i}_{w}]\in\mathbb{R}^{w\times h}$
$\displaystyle a^{i}=softmax(h^{i}\cdot s)\in\mathbb{R}^{w}$ $\displaystyle
A=[a^{1}\>a^{2}\>...\>a^{i}\>...\>a^{c}]\in\mathbb{R}^{c\times w}$

where $LSTM$ is an LSTM cell with the unit size of $h$. $s$ is a trainable
vector of length $h$.

Finally, a $c\times d$ shaped weighted sum $D_{summ}$ of the input tensor $D$
is computed, with the weights computed by the attention mechanism, and it is
fed to a sentence-level sequence tagger to tag discourse labels.

$D_{summ}[i,:]=A[i,:]\cdot D[i,:,:]\in\mathbb{R}^{d}$

Sentence-level Sequence Tagging. We observe that the discourse labels have a
clear transition of logic flow (e.g. _result_ usually followed by
_implication_ , and _method_ usually followed by _hypothesis_). Therefore, we
extend LSTM sequence tagger used by Dasigi et al. (2017) to BiLSTM-CRF
sequence tagger Huang et al. (2015) to label discourse types for each sentence
in a paragraph.

Labels in BIO Scheme. We use the BIO scheme Sang and Veenstra (1999) to train
all of our models (Baseline models for SciDT dataset do not use BIO2 scheme).
Specifically, we convert the labels into BIO scheme where _none_ label
represents _O_ and all other labels are converted into _B_label_ when the
previous label type is different from the current label and _I_label_ when the
previous label is the same as the current label.

###  3.2 Downstream Applications

####  3.2.1 Claim Extractor

Due to the similar problem formulation of evidence extraction task
Achakulvisut et al. (2019), we directly employ the discourse tagging model for
claim extraction.

####  3.2.2 Evidence Fragment Detector

##### Problem Reduction.

As Figure 3 shows, since each clause in evidence fragment detection task may
refer to more than one subfigure codes, we cannot directly solve it as a
standard classification task. Instead, we reduce it to a clause-level sequence
tagging problem under a block-based assumption. We treat each paragraph as a
single input. During training, we encode the subfigure code reference
sequences of the clauses in each paragraph into a single BIO Sang and Veenstra
(1999) sequence (where _B_ indicates the clause is the beginning of a block,
_I_ indicates the clause is in the same block as the previous clause, and _O_
indicates that no subfigure code is being referred to) as demonstrated at the
end of each clause in Figure 3. For prediction, we decode the semantic
subfigure code references of all clauses from the BIO sequence for each
paragraph following the same block-based assumption.

Block-Based Assumption. Most subfigure code reference labels are block-based.
We call contiguous clauses that share the same subfigure code reference labels
as a block, which is segmented by red lines in Figure 3. We further observe
that most blocks explicitly mention all of the semantically referred subfigure
codes at least once. Therefore, assuming this property is true for all blocks,
we can reconstruct a sequence of semantic subfigure code references for all
clauses in a paragraph. We use the explicitly mentioned subfigure codes for
each block and a BIO sequence indicating where each block starts and ends for
the reconstruction. Consequently, during encoding, we convert annotated
semantically referenced subfigure code labels into BIO scheme. During
decoding, we first localize the start and end position of each block using BIO
predicted tags, then fill each block with all explicitly mentioned subfigure
codes.

Clause-level Sequence Tagger. The key part of our sequence tagging-based
solution for evidence fragment detection is to determine where a block starts
and ends. We apply a clause-level sequence tagger to tag each clause in a
paragraph. Due to the small size of the evidence fragment detection dataset,
we empirically observe that feature-based CRF sequence taggers outperform
neural-network based sequence taggers, we thus adopt the feature-based model.
In addition to the scientific discourse tags, we use explicitly mentioned
subfigure codes as well as unigram, bigram and trigram words as features. For
each clause, we use all features described previously from the current clause
in addition to the same sets of features from the adjacent previous and next
clauses.

##  4 Experimental Setup

We evaluate the performance of our scientific discourse tagger on PubMed-RCT
dataset Dernoncourt and Lee (2017) and SciDT dataset Burns et al. (2016);
Dasigi et al. (2017) (Section 5.1). We also examine the transferablity of our
scientific discourse tagger to new datasets using CODA-19 dataset Huang et al.
(2020) (Section 5.2). We further study the efficiency of scientific discourse
tags on claim-extraction task via transfer learning as well as evidence
fragment detection task in a pipeline fashion (Section 6).

---
Figure 5: Count of each label in three datasets. The lines correspond to the
mappings from SciDT dataset Burns et al. (2016); Dasigi et al. (2017) and
PubMed 20k RCT dataset Dernoncourt and Lee (2017) to CODA-19 dataset Huang et
al. (2020) for zero-shot predictions (Section 5.2).

###  4.1 Datasets

Figure 5 shows the distribution of the labels in the three datasets introduced
below as well as their mappings used for zero-shot predictions in Section 5.2.

PubMed-RCT Dataset. We use PubMed-RCT Dernoncourt and Lee (2017) as the
standard dataset to evaluate our scientific discourse tagger against other
strong baselines. PubMed-RCT is derived from PubMed for sequential sentence
classification. It has two versions – a smaller PubMed 20k RCT, and a 10 times
larger PubMed 200k RCT. Due to our limited availability of computational
resources, we only consider PubMed 20k RCT in this work. PubMed 20k RCT is a
large dataset that consists of 20k abstracts of randomized controlled trials
(RCTs), with vocabulary of 68k across 240k sentences. Each sentence of an
abstract is labeled with one of the following roles (section heads) in the
abstract: _background_ , _objective_ , _method_ , _result_ or _conclusion_.

SciDT Dataset. Similar to PubMed-RCT Dernoncourt and Lee (2017), SciDT dataset
Burns et al. (2016); Dasigi et al. (2017) is a clause-based dataset with more
fine-grained taxonomy. We further expand the scientific discourse tagging
corpus developed by Burns et al. (2016) and Dasigi et al. (2017) by applying
the same clause parsing and annotation pipeline described by Dasigi et al.
(2017). This dataset is derived from the Pathway Logic Eker et al. (2002) and
INTACT databases Orchard et al. (2013). Texts from all sections of each of
those papers were pre-processed by parsing each sentence to generate a
sequence of main and subordinate clauses using Stanford Parser Socher et al.
(2013). Domain experts were asked to label each of the clauses using the
7-label taxonomy proposed by De Waard and Maat (2012) whose distributions are
shown in Figure 5. We apply sequential methods to sequences of clauses in
individual paragraphs.

Overall, scientific discourse tagging dataset has a total of 634 paragraphs
and 6124 clauses. We randomly split 570 paragraphs as the training and
validation set and the rest as the test set. Each paragraph contains up to 30
clauses and the number of word per clause has a mean of 17.7 and a standard
deviation of 12.5. The total vocabulary size is 8563, which is a small dataset
for an NLP task. However, we note the difficulties of obtaining such dataset.
We further perform a quality assessment of the dataset by re-annotating the
test set. We obtain Cohen’s kappa coefficient $\kappa=0.823$, which indicates
a high quality of the dataset.

CODA-19 Dataset. CODA-19 Huang et al. (2020) is a human-annotated dataset on a
subset of the abstracts of CORD-19 Wang et al. (2020), which is a corpus of
scholarly articles about COVID-19. Wang et al. (2020) segmented each abstract
into sentence fragments by comma (,), semicolon (;), and period (.). Each
sentence fragment is labeled with one of the research aspects: _background_ ,
_purpose_ , _method_ , _finding/contribution_ or _other_ , which is similar to
the label sets of PubMed-RCT Dernoncourt and Lee (2017). There are 10966
abstracts in total. We use this dataset to further examine our scientific
discourse tagger architecture’s applicability to new datasets as well as the
transferability of our trained scientific discourse tagger to new datasets.

###  4.2 Baseline Models

PubMed-RCT Dataset. We compare our discourse tagger against two strong
baselines on the PubMed 20k RCT dataset: 1) a hierarchical sequential labeling
network (HSLN) proposed by Jin and Szolovits (2018) and 2) the state-of-the-
art model Srivastava et al. (2019) on this dataset. HSLN Jin and Szolovits
(2018) used bio-word2vec Moen and Ananiadou (2013), a word2vec embedding
Mikolov et al. (2013) trained on corpora of Wikipedia, PubMed, and PMC, a
convolutional neural network (CNN) LeCun et al. (2015) (HSLN-CNN) or a BiLSTM
(HSLN-RNN) as a sentence encoder, followed by a BiLSTM-CRF architecture Huang
et al. (2015) as a sentence-level sequence tagger. Srivastava et al. (2019)
used a similar architecture: bio-word2vec Moen and Ananiadou (2013) as word
embedding, BiLSTM layer with a special dilation mechanism and a capsule layer
Hinton et al. (2011) as the sentence encoder and BiLSTM-CRF Huang et al.
(2015) as the sentence-level sequence tagger. SciDT Dataset. In addition to
the model of Dasigi et al. (2017)222https://github.com/edvisees/sciDT trained
on our expanded SciDT dataset, we also compare with feature based
CRF333CRFsuite package Okazaki (2007) and SVM444https://scikit-
learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html with unigram,
bigram and trigram words in the previous, current and next clauses as
features.

CODA-19 Dataset. Huang et al. (2020) provided a few baselines for scientific
discourse tagging on CODA-19 dataset Huang et al. (2020). Their best model is
a fine-tuned SciBERT Beltagy et al. (2019).

##  5 Experimental Results

###  5.1 Supervised Learning Results

Table 2 reports the test F1 score of our scientific discourse tagger and its
variations against baseline models on PubMed 20k RCT dataset and SciDT
dataset. Our best scientific discourse tagger outperforms the state-of-the-art
model Srivastava et al. (2019) on PubMed 20k RCT dataset by more than 2 %
absolute F1 score. Given the large size of PubMed 20k RCT, this result
robustly demonstrates the strength of our model. Our model also significantly
outperforms Dasigi et al. (2017) with 5% absolute F1 score (per McNemar’s
test, $p<0.01$). Based on these performance, we claim our scientific discourse
tagger as state-of-the-art. Note that for scientific discourse tagging, the
micro F1 performance is equivalent to accuracy.

Model |  | RCT | SciDT
---|---|---|---
Baselines | CRF |  | 0.679
SVM |  | 0.737
Dasigi et al. (2017) |  | 0.791
HSLN-CNN | 0.922 |
HSLN-RNN | 0.926 |
Srivastava et al. (2019) | 0.928 |
Our Approach | Embedding | Attention |  |
BioGloVe | No Context | 0.901 | 0.745
BioGloVe | RNN | 0.909 | 0.763
BioGloVe | LSTM | 0.913 | 0.794
BioBERT | No Context | 0.909 | 0.794
BioBERT | RNN | 0.915 | 0.775
BioBERT | LSTM | 0.927 | 0.794
SciBERT | No Context | 0.918 | 0.806
SciBERT | RNN | 0.922 | 0.817
SciBERT | LSTM | 0.951 | 0.841
Table 2: Scientific discourse tagging performance measured by test F1 score on PubMed 20k RCT and SciDT dataset. Model |  | Test F1
---|---|---
Baselines | Huang et al. (2020) | 0.749
Ours | 0.885
Transfer Learning | Zero-shot Prediction from RCT | 0.760
Zero-shot Prediction from SciDT | 0.761
PubMed-RCT pre-train | 0.909
Table 3: Transfer Learning Performance on CODA-19 Dataset.

####  5.1.1 Ablation Studies.

We also perform ablation studies to compare the effect of different word
embeddings and attention mechanisms to the performance of our scientific
discourse tagger on PubMed-RCT and SciDT dataset in Table 2. All neural
network based models discussed for scientific discourse tagging tasks,
including ours consist of a word embedding, a word-to-sentence encoder and a
sentence-level sequence tagger. As we introduce in Section 3.1, our best model
has SciBERT Beltagy et al. (2019) as our contextualized word embedding, an
LSTM-attention structure as our word-to-sentence encoder and BiLSTM-CRF Huang
et al. (2015) as our sentence sequence tagger. Comparing to other baseline
models, we improve the model design by adopting the state-of-the-art BERT
Devlin et al. (2018) based language model as our contextualized embedding.
Instead of bidirectional LSTM as word-to-sentence encoder used by Jin and
Szolovits (2018) and Srivastava et al. (2019), we improve the attention
structure proposed by Dasigi et al. (2017). We compare the effect of different
embeddings and attention types used in scientific discourse tagger. As Table 2
indicates, our main improvement comes from SciBERT Beltagy et al. (2019). In
addition to BioBERT Lee et al. (2019) which trains BERT Devlin et al. (2018)
on biomedical domain corpus, SciBERT uses a domain specific vocabulary. BERT
as a contextualized embedding also contributes partially to the performance
improvement as the BioBERT embedding globally outperforms BioGloVe Burns et
al. (2019), which is a static embedding trained on biomedical domain corpus,
on PubMed-RCT dataset. Another source of improvement comes from the attention
structure. Our LSTM-attention outperforms the RNN-attention that Dasigi et al.
(2017) used.

####  5.1.2 Attention Visualization.

Figure 6 shows 3 example indicative visualization of the attention weight of
our best scientific discourse tagger on the corresponding SciBERT tokens
encoded by the LSTM-based attention weight of the tokens. The opaque tokens
correspond to those with high weights. Our model learns to attend on certain
keywords, instead of attending on specialized terminologies. For example in
Figure 6, ‘‘to further test’’, ‘‘we examined’’ and ‘‘expression … inhibited’’
are emphasized for _goal_ , _methods_ and _results_ respectively. However in
most cases the attention distribution is much flatter than the examples in
Figure 6.

####  5.1.3 Error Analysis.

Figure 7 compares the confusion matrices of Dasigi et al. (2017) and our best
scientific discourse tagger on SciDT test set. As suggested by the overall
performance, our model globally predicts the discourse tags more precisely
than Dasigi et al. (2017). Specifically, Dasigi et al. (2017) failed to
predict _problem_ tag, but our model achieved 0.63 accuracy on predicting
_problem_ tag. Figure 7 also indicates the different difficulties of
predicting different discourse labels due to the imbalance of the label
distributions, as Table 5 shows.

###  5.2 Transfer Learning on CODA-19 Dataset

We further demonstrate the strong performance of our scientific discourse
tagger by training it on CODA-19 dataset Huang et al. (2020). As Table 3
shows, our model outperforms the baseline from Huang et al. (2020) by 14.6%
absolute F1 on the test set.

More importantly, we use these results as baselines and CODA-19 dataset as an
example dataset to show the transferability of our model to new datasets. We
first perform zero-shot prediction using our best trained scientific discourse
taggers on PubMed-RCT Dernoncourt and Lee (2017) or SciDT dataset Dasigi et
al. (2017). We map the labels from the original datasets to the target CODA-19
dataset by applying majority vote to the predicted labels on the training set
as the lines in Figure 5 show. Then we perform predictions using the best
trained scientific discourse taggers on CODA-19 test set and convert the
predicted labels from the original label sets to the target CODA-19 label set.
As a result, as Table 3 shows, our zero-shot prediction results are even
higher than the baseline from Huang et al. (2020) which was directly trained
on the CODA-19 dataset. This result indicates the strong transferability of
our trained scientific discourse tagger as a useful tool on new datasets.

Furthermore, we separately perform a standard transfer learning by taking the
scientific discourse tagger pre-trained on PubMed-RCT dataset and fine-tuning
it on CODA-19 dataset. We replace the last CRF layer with a new one to match
the labels of CODA-19 dataset. As a result, we achieved 0.909 test F1, which
is another 2.4% absolute F1 improvement on our model directly trained on
CODA-19 dataset. This is likely due to the similar label structures between
PubMed-RCT and CODA-19 dataset.

---
Figure 6: Visualization of the attention weights of the corresponding SciBERT
tokens encoded by alpha value. The ground truth labels of clauses are goal,
methods, results respectively.

Figure 7: Confusion matrix on SciDT test data. Up: Dasigi et al. (2017). Down:
Our scientific discourse tagger.

##  6 Downstream Applications

Having demonstrated the strong performance of scientific discourse tagger on
PubMed-RCT dataset Dernoncourt and Lee (2017) and SciDT dataset Dasigi et al.
(2017) as well as the strong trasnferability on CODA-19 Huang et al. (2020)
dataset, we show scientific discourse tagging is beneficial for downstream-
tasks that apply scientific discourse tags.

###  6.1 Claim Extraction

Dataset. Achakulvisut et al. (2019) introduced an expertly annotated dataset
for extracting claim sentences from biomedical paper abstracts. They followed
the definitions by Sateli and Witte (2015) to annotate a claim as a statement
that either declares something is better, proposes something new, or describes
a new finding or a causal relationship. Each sentence is tagged with a binary
label indicating it is a claim or not. Each abstract may contain multiple
claims as Figure 2 shows. The dataset contains 1500 abstracts sampled from
MEDLINE database.

Baseline Model. Achakulvisut et al. (2019) constructed claim-extraction
dataset and proposed a model using the sentence classification technique
presented by Arora et al. (2016) as sentence encoding method, and the standard
BiLSTM-CRF Huang et al. (2015) as the sentence-level sequence tagger. Their
best model was pre-trained on PubMed 200k RCT Dernoncourt and Lee (2017) for
transfer learning and used GloVe Pennington et al. (2014) as their word
embedding.

Model Performance. Table 4 compares the test binary F1 performance of
Achakulvisut et al. (2019) with our test performance. We first train our
scientific discourse tagger model directly on the claim-extraction dataset. We
obtain test binary F1 score of 0.791, which is already higher than
Achakulvisut et al. (2019). Then as Achakulvisut et al. (2019) suggested, we
pre-train the scientific discourse tagger on PubMed 20k RCT Dernoncourt and
Lee (2017) and fine-tune it on the claim-extraction dataset. We replace the
last CRF layer with a new one to match the binary label structure of claim-
extraction dataset. As a result, we obtain test binary F1 score of 0.828,
which is another 3.7% absolute F1 improvement on our model without transfer
learning. This result demonstrates the benefit of transfer learning from
scientific discourse tagging task to it’s downstream-tasks.

Model | Test F1
---|---
Achakulvisut et al. (2019) | 0.790
Ours (No pre-train) | 0.791
Ours (PubMed-RCT pre-train) | 0.828
Table 4: Claim extraction performance measured by binary F1 score, which regards $0$ as negative label. Model | BIO F1 | Test F1
---|---|---
Burns et al. (2017) | N/A | 0.75
Ours (W/O Discourse Tags) | 0.750 | 0.742
Ours (W/ Discourse Tags) | 0.821 | 0.807
Table 5: Evidence fragment detection performance measured by micro F1 score.
Our block-based decoding method achieves 0.94 F1 using ground truth BIO
sequences.

###  6.2 Evidence Fragment Detection

Dataset. Burns et al. (2017) introduced evidence fragment detection dataset,
which shares the same format and source of clause-based paragraphs with SciDT
dataset Dasigi et al. (2017). As Figure 3 shows, each clause was annotated
with subfigure codes that it is semantically referring to. Each clause may not
refer to any subfigure code, or simultaneously refer to multiple subfigure
codes. The explicit mentions of the subfigure codes were also annotated. All
paragraphs are from _Results_ section of experimental papers, and most of the
paragraphs are from a subset of SciDT dataset Burns et al. (2016); Dasigi et
al. (2017). We further expand evidence fragment detection training set by
annotating extra _Results_ section paragraphs from SciDT dataset. Overall this
small dataset consists of 191 paragraphs as training data and 19 paragraphs as
test data.

Baseline Model. Burns et al. (2017) proposed a rule-based method for evidence
fragment detection task. The key steps are determining where each evidence
fragment begins and ends based on the discourse tags of each clause. They
treat _hypothesis_ , _problem_ and _fact_ as indicators of beginning of a
evidence fragment, and _result_ and _implication_ as indicators of the end of
a evidence fragment. They also used other features including section headings
and whether the references to subfigures are entirely disjoint. Note that
their document-level rule-based tagging is across multiple paragraphs in the
_Results_ section.

Model Performances. We use a feature-based CRF555CRFsuite package Okazaki
(2007) with block-based encoding-decoding method to solve this task as a
sequence tagging problem. The decoding method described in Section 3.2.2
achieves 0.94 F1 score given the ground truth BIO sequences. Table 5 compares
our feature-based CRF model performance with Burns et al. (2017) 666We fix the
erroneous F1 score reported in Burns et al. (2017) by fixing the computation
of F1 score given precision and recall scores.. We also compare our feature-
based CRF model performances trained with or without scientific discourse tags
from SciDT dataset. Our feature-based CRF model without scientific discourse
tags as inputs does not outperform Burns et al. (2017). However, by adding the
scientific discourse tag as a feature, we obtain 5.7% absolute F1 improvement
over Burns et al. (2017), reaching 0.807 test F1. This improvement comes from
the improvement of the CRF sequence tagger. This result shows the strong
benefit of scientific discourse tags as the upstream task of evidence fragment
detection.

##  7 Discussion

We use the claim-extraction task and the evidence fragment detection task as
two examples to demonstrate the benefit of leveraging pre-trained scientific
discourse taggers and scientific discourse tags to improve the downstream-task
performance via transfer learning or in a pipeline fashion. As Burns et al.
(2017) proposed, given the output of evidence fragment detection system, we
can link subfigure codes with INTACT Orchard et al. (2013) records to obtain
evidence fragments for each experimental figure.

We further suggest that the evidence fragment detection task can help
biocurators delineate evidence fragments as independent documents so they can
be cataloged, indexed, and reused. Traditionally scientists’ arguments are
based on relationships between claims and evidences within the same paper and
possibly a limited number of cited papers. With the help of evidence
fragments, we are able to discard the convention of only linking claims to
evidence from a single paper or of following citations, which are often based
on linking separate claims from different papers. As a future work, we can
surface the evidence fragments combined with figures and captions across
multiple papers. Clark et al. (2014) proposed the ‘‘Micropublications’’
semantic model, which is an abstract framework that integrates scientific
argument and evidence from scientific documents. Our scientific discourse
tagger, claim extractor and evidence fragment detector may serve as the actual
implementation of the modules in such a framework. Ultimately, we hope to
dramatically increase the amount of primary evidence used to generate
individual claims and therefore improve the quality of those claims.

##  8 Conclusions

We develop a state-of-the-art model for scientific discourse tagging and
demonstrate its strong performance on PubMed-RCT dataset Dernoncourt and Lee
(2017) and SciDT dataset Burns et al. (2016); Dasigi et al. (2017) as well as
its strong transferability on new datasets such as CODA-19 dataset Huang et
al. (2020). We then demonstrate the benefit of leveraging the scientific
discourse tags on downstream-tasks by providing claim-extraction task and
evidence fragment detection task as two show cases. We further propose a
future direction that scientific discourse tagging can help delineate evidence
fragments as independent documents so they can be cataloged, indexed, and
reused. As a result, we can dramatically increase the amount of primary
evidence used to generate individual claims and therefore improve the quality
of those claims.

## References

  * Abadi et al. (2015) Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. (2015).  TensorFlow: Large-scale machine learning on heterogeneous systems.  Software available from tensorflow.org.
  * Achakulvisut et al. (2019) Achakulvisut, T., Bhagavatula, C., Acuña, D. E., and Körding, K. P. (2019).  Claim extraction in biomedical publications using deep discourse model and transfer learning.  CoRR, abs/1907.00962.
  * Arora et al. (2016) Arora, S., Liang, Y., and Ma, T. (2016).  A simple but tough-to-beat baseline for sentence embeddings.
  * Beltagy et al. (2019) Beltagy, I., Cohan, A., and Lo, K. (2019).  Scibert: Pretrained contextualized embeddings for scientific text.  arXiv preprint arXiv:1903.10676.
  * Burns et al. (2016) Burns, G. A., Dasigi, P., de Waard, A., and Hovy, E. H. (2016).  Automated detection of discourse segment and experimental types from the text of cancer pathway results sections.  Database, 2016.
  * Burns et al. (2017) Burns, G. A., Dasigi, P., and Hovy, E. H. (2017).  Extracting evidence fragments for distant supervision of molecular interactions.  BioRxiv, page 192856.
  * Burns et al. (2019) Burns, G. A., Li, X., and Peng, N. (2019).  Building deep learning models for evidence classification from the open access biomedical literature.  Database, 2019.
  * Chiu and Nichols (2016) Chiu, J. P. and Nichols, E. (2016).  Named entity recognition with bidirectional lstm-cnns.  Transactions of the Association for Computational Linguistics, 4, 357–370.
  * Chollet et al. (2015) Chollet, F. et al. (2015).  Keras.  https://github.com/fchollet/keras.
  * Clark et al. (2014) Clark, T., Ciccarese, P. N., and Goble, C. A. (2014).  Micropublications: a semantic model for claims, evidence, arguments and annotations in biomedical communications.  Journal of biomedical semantics, 5(1), 1–33.
  * Cortes and Vapnik (1995) Cortes, C. and Vapnik, V. (1995).  Support-vector networks.  Machine learning, 20(3), 273–297.
  * Cox et al. (2017) Cox, J., Harper, C. A., and de Waard, A. (2017).  Optimized machine learning methods predict discourse segment type in biological research articles.  In Semantics, Analytics, Visualization, pages 95–109. Springer.
  * Dasigi et al. (2017) Dasigi, P., Burns, G. A., Hovy, E., and de Waard, A. (2017).  Experiment segmentation in scientific discourse as clause-level structured prediction using recurrent neural networks.  arXiv preprint arXiv:1702.05398.
  * De Waard and Maat (2012) De Waard, A. and Maat, H. P. (2012).  Epistemic modality and knowledge attribution in scientific discourse: A taxonomy of types and overview of features.  In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, pages 47–55. Association for Computational Linguistics.
  * Dernoncourt and Lee (2017) Dernoncourt, F. and Lee, J. Y. (2017).  Pubmed 200k RCT: a dataset for sequential sentence classification in medical abstracts.  CoRR, abs/1710.06071.
  * Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Dusmanu et al. (2017) Dusmanu, M., Cabrio, E., and Villata, S. (2017).  Argument mining on twitter: Arguments, facts and sources.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2317–2322.
  * Eker et al. (2002) Eker, S., Knapp, M., Laderoute, K., Lincoln, P., and Talcott, C. (2002).  Pathway Logic: Executable Models of Biological Networks.  In Fourth International Workshop on Rewriting Logic and Its Applications (WRLA’2002, volume 71 of Electronic Notes in Theoretical Computer Science. Elsevier.
  * Fréard et al. (2010) Fréard, D., Denis, A., Détienne, F., Baker, M., Quignard, M., and Barcellini, F. (2010).  The role of argumentation in online epistemic communities: the anatomy of a conflict in wikipedia.  In ECCE 2010. European Conference on Cognitive Ergonomics, pages pp–91.
  * Guo et al. (2010) Guo, Y., Korhonen, A., Liakata, M., Karolinska, I. S., Sun, L., and Stenius, U. (2010).  Identifying the information structure of scientific abstracts: an investigation of three different schemes.  In Proceedings of the 2010 Workshop on Biomedical Natural Language Processing, pages 99–107. Association for Computational Linguistics.
  * Habernal et al. (2014) Habernal, I., Eckle-Kohler, J., and Gurevych, I. (2014).  Argumentation mining on the web from information seeking perspective.  In ArgNLP.
  * Hinton et al. (2011) Hinton, G. E., Krizhevsky, A., and Wang, S. D. (2011).  Transforming auto-encoders.  In International Conference on Artificial Neural Networks, pages 44–51. Springer.
  * Hirohata et al. (2008) Hirohata, K., Okazaki, N., Ananiadou, S., and Ishizuka, M. (2008).  Identifying sections in scientific abstracts using conditional random fields.  In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I.
  * Hobbs (2002) Hobbs, J. R. (2002).  Information extraction from biomedical text.  Journal of biomedical informatics, 35(4), 260–264.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J. (1997).  Long short-term memory.  Neural computation, 9(8), 1735–1780.
  * Huang et al. (2020) Huang, T.-H., Huang, C.-Y., Ding, C.-K. C., Hsu, Y.-C., and Giles, C. L. (2020).  Coda-19: Reliably annotating research aspects on 10,000+ cord-19 abstracts using non-expert crowd.  arXiv preprint arXiv:2005.02367.
  * Huang et al. (2015) Huang, Z., Xu, W., and Yu, K. (2015).  Bidirectional lstm-crf models for sequence tagging.  arXiv preprint arXiv:1508.01991.
  * Jin and Szolovits (2018) Jin, D. and Szolovits, P. (2018).  Hierarchical neural networks for sequential sentence classification in medical scientific abstracts.  CoRR, abs/1808.06161.
  * Kingma and Ba (2014) Kingma, D. P. and Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Lafferty et al. (2001) Lafferty, J., McCallum, A., and Pereira, F. C. (2001).  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In ICML.
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., and Hinton, G. (2015).  Deep learning.  nature, 521(7553), 436–444.
  * Lee et al. (2019) Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang, J. (2019).  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics.
  * Liakata (2010) Liakata, M. (2010).  Zones of conceptualisation in scientific papers: a window to negative and speculative statements.  In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, pages 1–4. Association for Computational Linguistics.
  * Liakata et al. (2012) Liakata, M., Saha, S., Dobnik, S., Batchelor, C., and Rebholz-Schuhmann, D. (2012).  Automatic recognition of conceptualization zones in scientific articles and two life science applications.  Bioinformatics, 28(7), 991–1000.
  * Ma and Hovy (2016) Ma, X. and Hovy, E. (2016).  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  arXiv preprint arXiv:1603.01354.
  * Mikolov et al. (2013) Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013).  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781.
  * Moen and Ananiadou (2013) Moen, S. and Ananiadou, T. S. S. (2013).  Distributional semantics resources for biomedical text processing.  Proceedings of LBM, pages 39–44.
  * Okazaki (2007) Okazaki, N. (2007).  Crfsuite: a fast implementation of conditional random fields (crfs).
  * Orchard et al. (2013) Orchard, S., Ammari, M., Aranda, B., Breuza, L., Briganti, L., Broackes-Carter, F., Campbell, N. H., Chavali, G., Chen, C., Del-Toro, N., et al. (2013).  The mintact project—intact as a common curation platform for 11 molecular interaction databases.  Nucleic acids research, 42(D1), D358–D363.
  * Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C. (2014).  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  * Sang and Veenstra (1999) Sang, E. F. and Veenstra, J. (1999).  Representing text chunks.  In Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics, pages 173–179. Association for Computational Linguistics.
  * Sardianos et al. (2015) Sardianos, C., Katakis, I. M., Petasis, G., and Karkaletsis, V. (2015).  Argument extraction from news.  In Proceedings of the 2nd Workshop on Argumentation Mining, pages 56–66.
  * Sateli and Witte (2015) Sateli, B. and Witte, R. (2015).  Semantic representation of scientific literature: bringing claims, contributions and named entities onto the linked open data cloud.  PeerJ Computer Science, 1, e37.
  * Socher et al. (2013) Socher, R., Bauer, J., Manning, C. D., et al. (2013).  Parsing with compositional vector grammars.  In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 455–465.
  * Srivastava et al. (2019) Srivastava, S., Agarwal, P., Shroff, G., and Vig, L. (2019).  Hierarchical capsule based neural network architecture for sequence labeling.  In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.
  * Stab et al. (2014) Stab, C., Kirschner, C., Eckle-Kohler, J., and Gurevych, I. (2014).  Argumentation mining in persuasive essays and scientific articles from the discourse structure perspective.  In ArgNLP, pages 21–25.
  * Teufel and Moens (1999) Teufel, S. and Moens, M. (1999).  Discourse-level argumentation in scientific articles: human and automatic annotation.  Towards Standards and Tools for Discourse Tagging.
  * Teufel and Moens (2002) Teufel, S. and Moens, M. (2002).  Summarizing scientific articles: experiments with relevance and rhetorical status.  Computational linguistics, 28(4), 409–445.
  * Thorne et al. (2018) Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., and Mittal, A. (2018).  The fact extraction and verification (fever) shared task.  arXiv preprint arXiv:1811.10971.
  * Wang et al. (2020) Wang, L. L., Lo, K., Chandrasekhar, Y., Reas, R., Yang, J., Eide, D., Funk, K., Kinney, R., Liu, Z., Merrill, W., et al. (2020).  Cord-19: The covid-19 open research dataset.  arXiv preprint arXiv:2004.10706.

Supporting Information
Scientific Discourse Tagging for Evidence Extraction
Xiangci Li, Gully Burns and Nanyun Peng

##  A Implementation and Training Details

The scientific discourse tagging model is implemented using Keras Chollet et
al. (2015) with Tensorflow Abadi et al. (2015) backend. We use early stopping
mechanism with toleration of 2 epochs. We schedule the training by training
the model with a learning rate of $lr$ for 20 epochs. We use Adam Kingma and
Ba (2014) for the optimizer. The optimal hyper-parameters and the attempted
range if applicable are listed in Table SI 1.

Hyper-Parameter | Used
---|---
$d_{BERT}$ | 768
$c$ | 40
$w$ | 60
$d$ | 768
$d_{2}$ | 300
$p$ | 200
$h$ | 75
$H$ | 350
$lr$ | $10^{-3}$
Validation Set Ratio | 0.1
Embedding dropout | 0.4
Dense dropout | 0.4
Attention dropout | 0.6
LSTM dropout | 0.5
Batch size | 10
Table SI 1: Optimal hyper-parameters of scientific discourse tagger model


# FPGAs (Can Get Some) SATisfaction

H. Govindasamy1, B. Esfandiari1, P. Garcia2

1{hari<EMAIL_ADDRESS>Dpt. Systems Computer Engineering, Carleton
University, Canada<EMAIL_ADDRESS>International School of Engineering,
Chulalongkorn University, Thailand

###### Abstract

We present a hardware-accelerated SAT solver suitable for processor/Field
Programmable Gate Arrays (FPGA) hybrid platforms, which have become the norm
in the embedded domain. Our solution addresses a known bottleneck in SAT
solving acceleration: unlike prior state-of-the-art solutions that have
addressed the same bottleneck by limiting the amount of exploited parallelism,
our solver takes advantage of fine-grained parallelization opportunities by
hot-swapping FPGA clause assignments at runtime. It is also the first modern
completely open-source SAT accelerator, and formula size is limited only by
the amount of available external memory, not by on-chip FPGA memory.

Evaluation is performed on a Xilinx Zynq platform: experiments support that
hardware acceleration results in shorter execution time across varying formula
sizes, subject to formula partitioning strategy. We outperform prior state-of-
the-art by 1.7x and 1.1x, respectively, for 2 representative benchmarks, and
boast up to 6x performance increase over software-only implementation.

###### Index Terms:

FPGA, SAT, acceleration, embedded, boolean, satisfiability

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE Figure 1: (a) Formula
partitioning. (b) Formula stored directly on FPGA; single clause retrieved for
processing per partition. (c) Formula stored directly on FPGA; partitions
connected in partial order. Clauses processed sequentially. (d) Formula stored
in external memory (”software” view) and partitions hot-swapped as required.
Clauses are mapped directly to clause engines.

##  I Introduction

Efficient computation of Boolean Satisfiability problem (SAT) is extremely
important across several engineering domains, as SAT can encode all NP-
complete problems [1, 2]. In the power/performance constrained embedded world,
SAT solvers were not feasible [3] until the recent advent of sufficiently
powerful Field Programmable Gate Arrays (FPGA) offering acceleration
opportunities [4].

The state of the art on SAT solver acceleration is comprised of either
incomplete solvers [5] or complete solvers with only coarse-grained
parallelization [6, 7]: we point interested readers to a comprehensive survey
of hardware SAT solvers [8]. Our work directly extends (and is compared with)
works by Davis et al [9] and Thong et al [6], both hardware-accelerations of
Boolean Constraint Propagation (BCP), a sub-component of the Davis-Putnam-
Logemann-Loveland (DPLL) SAT solver algorithm.

Figure 1 summarizes Davis et al, Thong et al and our approach. Davis et al’s
[9] approach groups _clauses_ (see Section II-A below) across parallel
accelerators, such that no two clauses on the same accelerator share
variables. Using a two stage process, the variable assignment’s corresponding
clause is first retrieved (we point readers to [9] for an in-depth explanation
of their look-up technique), and the resulting implication (constraints on
variable values in function of prior clauses) is then calculated.

Thong et al [6] identify that clause look-up in Davis et al’s [9] work is
slower than direct access; thus, they evaluate a clause encoding scheme that
links clauses with shared variables on BRAM. The result is a sequential BCP
processor with multi-threaded software support and a more performant solution
when compared to Davis et al. BCP processors are replicated in hardware to
maximize FPGA usage.

In contrast with Thong et al, we eliminate the need for clause look-up in
Davis et al’s [9] work by allowing clauses to share variables within the same
partition, keeping formula in external memory (i.e., in software) and hot-
swapping hardware assignment of clauses at runtime, as required for
implication calculation. This allows us to maintain the same high degree of
parallelization as Davis et al, unlike Thong et al’s sequential solution, at
the expense of more swapping overhead. Experiments suggest swapping overhead
can be negligible, compared to acceleration gains, subject to formula
partitioning: results show our solution can outperform related work if
partitioning results in little swapping, and formula size is limited only by
the size of on-board memory (external RAM, not on-chip FPGA memory). Our
solution is available
here111https://github.com/harigovind1998/FPGA_BCP_acceleration, deployed on
the Zynq chip, the processor-FPGA platform that has become the de facto
solution across the embedded domain [10]; to the best of the authors’
knowledge, this is the first modern open-source SAT solver accelerator.

Figure 2: SAT Accelerator Processor-FPGA Interface. BCP Engine accelerates
part of the software SAT solving algorithm through fine-grained
parallelization across Clause Processors.

##  II The SAT solver Architecture

###  II-A Background: DPLL and BCP

Given a Boolean formula, a SAT solver determines if a satisfying assignment
exists; that is, does a free variable assignment exist, such that the formula
evaluates to true? Boolean formulas are expressed in Conjunctive Normal Form
(CNF), where the input is a conjunction of clauses, each a disjunction of
literals (an occurrence of a variable or its negation).

Incomplete solvers follow a greedy approach, often employing variations of
Stochastic Local Search (SLS) [8]. These solvers are typically quicker than
complete ones; however, satisfying assignments can be eluded and results are
not guaranteed. Complete solvers, on the other hand, search exhaustively,
evaluating every possible assignment combination until a satisfying assignment
is found. A formula is unsatisfiable if no satisfying assignment is found
after checking all combinations.

Many complete solvers are derivations of the DPLL algorithm [8], proposing
improvements to its decision heuristic and branch pruning techniques. DPLL
consists of two stages: decision and Boolean Constraint Propagation (BCP).
During decision, variables are assigned truth values, which are then
propagated and evaluated during BCP, until a formula is deemed satisfiable or
not. The BCP component is substantially expensive, taking up to 80-90% of
DPLL’s total execution time [6]. Both most relevant prior works ([9],[6]) as
well as our own, are based on accelerating BCP.

###  II-B The BCP accelerator architecture

Our solution accelerates BCP execution on FPGA. The architecture processes all
clauses in a given partition in parallel and should be able to seamlessly
accelerate any DPLL-based algorithm. The BCP engine (Figure 2) consists of a
control unit (1), an array of Clause Processors (2) and an implication
selector (3).

The control unit orchestrates the execution flow of the BCP engine, loading
clauses into corresponding processors, clearing backtracking assignments, and
broadcasting decisions and implications to clause processors. Behavior is
depicted in Figure 3. Clause Processors store clauses as an array of literals;
they also contain a copy of the respective variable assignments, updated when
a decision or implication is received. Clauses mapped to clause processors can
be hot-swapped during runtime, allowing instances to be changed without
needing to rebuild the FPGA design. The implication selector is a multiplexer
that selects a single implication when multiple clause processors generate
implications. Unlike Davis et al [9], our design does not employ an
implication conflict detector; instead, the BCP engine propagates the chosen
implication and identifies conflicts during the evaluation stage.

Figure 3: BCP Engine - Control Unit State Diagram.

###  II-C Formulae partitioning

Instances of SAT problems may be comprised of an arbitrary number of variables
and clauses. Available hardware fabric limits the size of problems solvable on
FPGA; large problems must be partitioned. BCP is performed separately on each
partition, and implications are propagated to all other partitions. The
problem partition currently assigned to the FPGA fabric is controlled by
software.

The partitioning approach utilized for the results in this paper is a simple
greedy approach: a partition is as big as possible, limited by the number of
clauses and variables that fit on hardware. Clauses are chosen sequentially,
as they appear in the formula to evaluate. Results (see Section III) suggest
partitioning is the bottleneck in our approach: some partition assignments
result in performance improvement, whilst others result in performance
degradation, depending on the amount of required partition swapping. Thus, a
more selective partitioning algorithm is a logical next step to improve this
area of research, but outside the scope of this paper, and reserved for future
work.

###  II-D Processor-FPGA interface

The processor communicates with the FPGA through AXI [11], sending commands
and data by directly writing to hardware registers. Once commands are issued,
the processor polls for status updates and new implications. Status updates
informs the processor of conflicts, availability of implications and
completion, and dictates the flow of DPLL. The processor stores a copy of
implications to avoid re-assigning implied variables, and to propagate to
other partitions.

##  III Experiments and Results

We evaluate our accelerator on a Zynq chip, with 14400 LUTs and 28800 FF.
Implementation is clocked at 106.66 MHz, utilizing 13151 LUTs, 11059 FFs, and
647 LUTRAM of on-chip memory.

| Millions of BCP/s
---|---
SAT Instance | Davis et al [9] |  Thong et al [6] | Our Design
bmc-galileo-8 | 40 | 102 | 175
bmc-ibm-12 | 33 | 150 | 169
TABLE I: BCP engine throughput (BCPs/s) comparison with related work. Results
reflect only full availability of data to BCP engines; i.e., maximum
theoretical throughput. Figure 4: Total execution time breakdown across
constituent parts.

###  III-A Comparison to related work

Related literature only reports throughput on BCP calculation, in millions of
BCPs per second, measured solely during periods when data is available to the
BCP engine. This metric is not useful, as it excludes software execution and
data transfer times which account for a significant portion of total execution
time. Nonetheless, equivalent comparison between our design and related work
is depicted in Table I. A detailed breakdown of total execution time into its
constituent parts is depicted in Figure 6, contextualizing results of Table I.

###  III-B Software acceleration

To evaluate acceleration gains, we evaluate combinations of clause and
variable sizes, and measure total execution time across software-only and
hardware-accelerated versions. Relative speedup is depicted in Table II for
meaningful combinations. For each combination, we also depict real throughput,
in the form of BCPs/s averaged over total execution time. Notice that entry
with 63 variables and 224 clauses represents theoretical upper-limit, where
the entire formula fits on the accelerator without partitioning.

|  | Variables
---|---|---
|  | 63 | 126 | 252 | 630
Clauses | 224 |  362M BCP/s 2.2x |  17K BCP/s 0.17x | NA | NA
448 |  702K BCP/s 1.6x |  21K BCP/s 0.21x |  13K BCP/s 0.08x | NA
2240 |  441K BCP/s 1.91x |  22K BCP/s 1.26x |  16K BCP/s 0.61x |  12K BCP/s 0.10x
22400 |  313K BCP/s 6.32x |  20K BCP/s 5.04x |  16K BCP/s 4.86x |  14K BCP/s 3.31x
TABLE II: Relative hardware/software speedup and effective BCP engine
throughput for varying clauses/variables sizes.

To evaluate the different effects of clause/variable sizes on execution, we
fix of them and vary only the other, measuring total execution time
respectively: results are depicted in Figures 5 and 6.

Figure 5: Effect of clauses size increase on total execution time, for 63
variables. Figure 6: Effect of variables size increase on total execution
time, for 22400 clauses.

##  IV Conclusion

A SAT solver accelerator suitable for FPGA-based embedded applications has
been described and evaluated. Our approach built on prior work by Davis et al,
exploiting fine-grained parallelization. Unlike Davis et al, our solution is
not constrained by variable sharing across clauses; instead, our partitioning
approach, hot-swapping FPGA assignments at runtime as required for
computation, ensures formulas are eventually deemed satisfiable or not. Unlike
prior work by Thong at al, which also attempted to eliminate Davis et al’s
bottleneck, we still take full advantage of FPGA parallelization
opportunities: this results in higher theoretical maximum performance (Table
I) and higher practical performance for a range of clause-variable assignments
(Table II) when FPGA swapping time, as a function of partitioning algorithm,
does not exceed benefits of BCP acceleration.

Our solution’s performance is constrained by the partitioning approach: in all
likelihood, minimizing the number of partitions each variable appears in will
result in the least amount of swapping time. This, of course, must be balanced
with the total number of partitions. An algorithm that partitions a formula in
such a way as to achieve pareto-optimality among number of partitions and
variable dispersion is not known at this time: future work will focus on this
area of research, potentially through stochastic solutions (as exact solutions
may well be of the same level of time complexity as SAT).

Our artifacts are available in open-source form, and should provide valuable
help to practitioners that wish to deploy them in their own solutions.

## Acknowledgments

We acknowledge the support of the Natural Sciences and Engineering Research
Council of Canada (NSERC)

## References

  * [1] M. R. Garey and D. S. Johnson, _Computers and Intractability; A Guide to the Theory of NP-Completeness_. USA: W. H. Freeman & Co., 1990.
  * [2] S. A. Cook, “The complexity of theorem-proving procedures,” in _Logic, Automata, and Computational Complexity: The Works of Stephen A. Cook_ , 2023, pp. 143–152.
  * [3] P. Zhong, M. Martonosi, P. Ashar, and S. Malik, “Accelerating boolean satisfiability with configurable hardware,” in _Proceedings. IEEE Symposium on FPGAs for Custom Computing Machines (Cat. No. 98TB100251)_. IEEE, 1998, pp. 186–195.
  * [4] ——, “Using configurable computing to accelerate boolean satisfiability,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 18, no. 6, pp. 861–868, 1999.
  * [5] K. Hara, N. Takeuchi, M. Aono, and Y. Hara-Azumi, “Amoeba-inspired stochastic hardware sat solver,” in _20th International Symposium on Quality Electronic Design (ISQED)_ , 2019, pp. 151–156.
  * [6] J. Thong and N. Nicolici, “Fpga acceleration of enhanced boolean constraint propagation for sat solvers,” in _2013 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_ , 2013, pp. 234–241.
  * [7] K. Gulati, S. Paul, S. P. Khatri, S. Patil, and A. Jas, “Fpga-based hardware acceleration for boolean satisfiability,” _ACM Trans. Des. Autom. Electron. Syst._ , vol. 14, no. 2, apr 2009. [Online]. Available: https://doi.org/10.1145/1497561.1497576
  * [8] A. A. Sohanghpurwala, M. W. Hassan, and P. Athanas, “Hardware accelerated sat solvers: A survey,” _Journal of Parallel and Distributed Computing_ , vol. 106, pp. 170–184, 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0743731516301903
  * [9] J. D. Davis, Z. Tan, F. Yu, and L. Zhang, “A practical reconfigurable hardware accelerator for boolean satisfiability solvers,” in _2008 45th ACM/IEEE Design Automation Conference_ , 2008, pp. 780–785.
  * [10] K. Manev, A. Vaishnav, and D. Koch, “Unexpected diversity: Quantitative memory analysis for zynq ultrascale+ systems,” in _2019 International Conference on Field-Programmable Technology (ICFPT)_. IEEE, 2019, pp. 179–187.
  * [11] F. Restuccia, A. Biondi, M. Marinoni, G. Cicero, and G. Buttazzo, “Axi hyperconnect: A predictable, hypervisor-level interconnect for hardware accelerators in fpga soc,” in _2020 57th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 2020, pp. 1–6.