#  Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Benyou Wang<EMAIL_ADDRESS>Department of Information Engineering,
University of PaduaItaly ,  Qianqian Xie<EMAIL_ADDRESS>Department of Computer Science, University of ManchesterUnited Kingdom ,
Jiahuan Pei<EMAIL_ADDRESS>University of AmsterdamNetherlands ,  Prayag Tiwari
<EMAIL_ADDRESS>Department of Computer Science, Aalto UniversityFinland
,  Zhao Li<EMAIL_ADDRESS>The University of Texas Health
Science Center at HoustonUSA and  Jie Fu<EMAIL_ADDRESS>Mila, University
of MontrealCanada

(2021)

###### Abstract.

Pre-trained language models (PLMs) have been the de facto paradigm for most
natural language processing (NLP) tasks. This also benefits biomedical domain:
researchers from informatics, medicine, and computer science (CS) communities
proposes various PLMs trained on biomedical datasets, e.g., biomedical text,
electronic health records, protein, and DNA sequences for various biomedical
tasks. However, the cross-discipline characteristics of biomedical PLMs hinder
their spreading among communities; some existing works are isolated from each
other without comprehensive comparison and discussions. It expects a survey
that not only systematically reviews recent advances of biomedical PLMs and
their applications but also standardizes terminology and benchmarks. In this
paper, we summarize the recent progress of pre-trained language models in the
biomedical domain and their applications in biomedical downstream tasks.
Particularly, we discuss the motivations and propose a taxonomy of existing
biomedical PLMs. Their applications in biomedical downstream tasks are
exhaustively discussed. At last, we illustrate various limitations and future
trends, which we hope can provide inspiration for the future research of the
research community.

Biomedical domain, pre-trained language models, natural language processing

††copyright: acmcopyright††journalyear: 2021††ccs: Computing methodologies
Natural language processing††ccs: Computing methodologies Natural language
generation††ccs: Computing methodologies Neural networks††ccs: Computing
methodologies Bio-inspired approaches

###### Contents

  1. 1 Introduction
  2. 2 Background:
    1. 2.1 Pre-trained Language Models
    2. 2.2 Backbone Networks in Language Models
    3. 2.3 Pre-training and Fine-tuning Paradigm in PLMs
  3. 3 PLMs in Biomedical Domain: Why and How?
    1. 3.1 Motivation
    2. 3.2 How to tailor PLMs to the Biomedical Domain
    3. 3.3 Taxonomy of Usage
  4. 4 PLMs in Biomedical Domain
    1. 4.1 Unstructured Data for Pre-trained Language Models
    2. 4.2 Biomedical Pre-trained Language Models
    3. 4.3 Beyond Text: Language Models for Proteins/DNA
  5. 5 Fine-tuning PLMs for Biomedical Downstream Tasks
    1. 5.1 Information Extraction
    2. 5.2 Text Classification
    3. 5.3 Sentence Similarity
    4. 5.4 Question Answering
    5. 5.5 Dialogue Systems
    6. 5.6 Text Summarization
    7. 5.7 Natural Language Inference
    8. 5.8 Proteins/DNAs Prediction
    9. 5.9 Competition Venues
  6. 6 Discussion
    1. 6.1 Limitations and Concerns
    2. 6.2 Future trends
  7. 7 Conclusion

##  1\. Introduction

As the principal method of communication, humans usually record information
and knowledge in a format of token sequences, resulting in languages such as
natural language, constructed language, and programming language. For
biomedical information and knowledge, tokens in sequences could be of various
types, including words, disease codes, amino acids, and DNA. Tremendous
biomedical information and knowledge in nature and human history are
implicitly encapsulated in these natural token sequences in nature (a.k.a.,
data).

Based on the abstraction degree of biomedical knowledge, we list some of these
data as a pyramid in Fig. 1. The data in the top-level means it is high-level
information that explicitly conveys the biomedical knowledge, which is usually
small-scaled – see biomedical knowledge bases and EHR data (maybe in multi-
modality). At the low level, data, such as protein and DNA sequences, may not
directly convey the biomedical knowledge, e.g., one can hardly know how a
short sequence really means for humans, which needs more effort for
abstraction. Fortunately, data at low-level are usually tremendous. In the
current stage, existing work paid more attention to data in the top and
medium-level, which are relatively small but easily understood by humans. We
argue that biomedical knowledge in various scales should be paid attention to.
To capture and mine the biomedical information and knowledge from various data
scales, there is recently growing attention in the biomedical NLP community to
adopt pre-trained language models (PLMs); since PLMs could leverage these
plain token sequences without human annotations (sometimes called ‘self-
supervision’).

Figure 1. Existing data that involves biomedical information. The level is
based on to which extent humans abstract biomedical knowledge from data.

The biomedical NLP is a cross-discipline research direction from various
communities such as bioinformatics, medicine, and computer science (especially
a major frontier of artificial intelligence, i.e., natural language processing
a.k.a. NLP). The computational biology community (Kingsbury, 1996) and
biomedical informatics community (Cohen, 2004) have made a substantial effort
to make use of NLP tools for information mining and extraction of widespread-
adopted electronic health records, medical scientific publications, medical
WIKI pages, etc. For many decades, NLP has been investigating various
biomedical tasks (Cohen and Demner-Fushman, 2014; Cohen and Hersh, 2005) such
as classification, information extraction, question answering, and drug
discovery. Meanwhile, the approaches in the NLP community are changing
rapidly, as one can witness exponentially-increasing submitted papers in top
conferences like ACL, EMNLP, and NAACL. Tailoring these NLP approaches that
have been evidenced effectively in the NLP community to a specific biomedical
domain is beneficial.

Unfortunately, there is usually a delay for newly proposed NLP approaches
being applied to the biomedical domain. Especially, since the adoption of
various pre-trained language models (e.g., ELMo (Peters et al., 2018), GPT
(Radford et al., 2018), BERT (Devlin et al., 2018), XLNET (Huang et al.,
2019b), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2020) and ELECTRA
(Clark et al., 2020)) (Qiu et al., 2020) have nearly shifted the paradigm in
NLP, their biomedical variants trained using biomedical data comes sooner or
later. With this hot trend of the biomedical pre-trained language model, this
survey aims to bridge the gap between pre-trained language models and their
applications in the biomedical domain.

Figure 2. Overview of selected released Biomedical pre-trained language
models. One can see a more detailed list in Sec. 4. Note that there is a BERT-
like language model embedded in the overall architecture of AlphaFold2.

##### Motivation of pre-trained language models in biomedical domain

The current NLP paradigm is gradually shifting to a two-stage (pre-training
and fine-tuning) paradigm, thanks to recently proposed pre-trained language
models. Comparing to the previous paradigm with purely supervised learning
that relies on feature engineering or neural network architecture engineering
(Liu et al., 2021), the current two-stage paradigm is more friendly to the
scenario when supervised data is limited while large-scaled unsupervised data
is tremendous. Fortunately, the biomedical domain is a typical case of such a
scenario.

The motivation to use pre-trained language models in the biomedical domain are
pretty straightforward. First, annotated data in the biomedical domain is
usually not large-scaled. Therefore, a well-trained pre-trained language model
is more crucial to provide a richer feature extractor, which may slightly
reduce the dependence on annotated data. Second, the biomedical domain is more
knowledge-intensive than the general domain. At the same time, pre-trained
language models could serve as an easily-used soft knowledge base (Petroni et
al., 2019) that captures implicit knowledge from large-scale plain documents
without human annotations. More recently, GPT3 has been shown to have the
potential to ‘remember’ many complicated common knowledge (Brown et al.,
2020). Lastly, large-scaled biomedical corpora and biomedical sequences
(including proteins and DNAs), which are previously thought as difficult to
model, can be effectively modeled by the backbone network structure in
language models (such as transformers stacked in deep layers), to capture
efficient information via model pre-training in the large scale data.

As shown in Fig. 3, in recent three years, we witness a rapid development of
pre-trained language models (e.g., ELMo (Peters et al., 2018), GPT (Radford et
al., 2018), BERT (Devlin et al., 2018), XLNet (Huang et al., 2019b), RoBERTa
(Liu et al., 2019), T5 (Raffel et al., 2020) and ELECTRA (Clark et al., 2020))
in the general NLP domain. Followed by these progress, there are efforts to
tailor these pre-trained language models to their corresponding biomedical
variants, via in-domain data. For example, BERT, the most typical pre-trained
language, has many variants in the biomedical domain, e.g., Med-BERT (Rasmy et
al., 2020), BioBERT (Lee et al., 2020b), publicly available Clinical BERT
Embeddings (Alsentzer et al., 2019), SciBERT (Beltagy et al., 2019),
ClinicalBERT (Huang et al., 2019a), and COVID-twitter-BERT (Müller et al.,
2020) et al. We draw an overview for these models in Fig. 2. It shows that the
extensions of general domain pre-trained language models to the biomedical
domain attract great attention from researchers in both NLP and bioinformatics
communities. Interestingly, we can observe that once the general NLP community
develops a new variant of PLM, it usually leads to a biomedical counterpart
after some months. This parallel development between general PLMs and
biomedical PLMs shows a strong demand and even necessity to summarize the
existing works, which could help beginners to start their contributions on
this field easily.

Figure 3. Parallel development of general and biomedical pre-trained language
models. The time is determined by the released date of the paper, for example,
in arXiv. General pre-trained language models are shown below in the timeline,
and biomedical pre-trained language models are shown above the timeline (refer
to Tab. 4 for detailed date).

##### Difference with existing surveys

There are a few reviews to summarize the NLP applications in the
biomedical/clinical/bioinformatics domain, such as an early one (Spyns, 1996)
and recent ones (Zeng et al., 2015; Wu et al., 2020b; Percha, 2020). They
cover many general methods and applications of biomedical/clinical NLP.
Specifically, (Spyns, 1996) mainly discuss either based on statistics-based
NLP pipeline (including lexicon, co-occurrence patterns, syntactic/semantic
parsing), or word embeddings based neural network approaches (it was mentioned
that 60.8% of them are based on recurrent neural networks) (Wu et al., 2020b)
for NLP applications (e.g., information extraction, text classification, named
entity recognition, and relation extraction et al). Especially, two reviews
(Khattak et al., 2019; Kalyan and Sangeetha, 2020) discuss the word embeddings
used in biomedical NLP.

All the above reviews made thorough summarization of existing work before the
pre-trained language model era of NLP. The NLP techniques in these reviews are
mainly about feature engineering, or architecture engineering (Liu et al.,
2021). However, the NLP recently has been shifted to a pre-training-finetuning
paradigm with large-scale pre-trained language models (see existing surveys
(Qiu et al., 2020; Han et al., 2021; Liu et al., 2020b, 2021; Bommasani et
al., 2021) for pre-trained language model in the general domain). (Bommasani
et al., 2021) called these pre-trained models as ‘foundation models’ to
underscore their critically central. We believe the biomedical NLP
applications have been benefited and will continually benefit from the
development of pre-trained language models. More recently, (Kalyan et al.,
2021) 111The paper is publicly-available on the arXiv without peer review
reviews biomedical textual pre-training, especially using BERT. The difference
between (Kalyan et al., 2021) and this review is many folds. First, biomedical
pre-trained language models in this review are not limited to texts like
(Kalyan et al., 2021), but also protein, DNA, and even we envision biomedical
multi-modal pre-trained language models based on the multi-modal data (e.g.,
in electronic health records) – in general, any data that involves biomedical
information could be used in biomedical PLMs. Secondly, in contrast to (Kalyan
et al., 2021) which only discusses Transformer-based pre-trained language
models, this review also discusses RNN based language models (like ELMO (Jin
et al., 2019a), which is typically considered as the first pre-trained
language model in NLP). Lastly, we also summarize decoder involved pre-trained
language models (like GPT (Kraljevic et al., 2021) and T5 (Phan et al., 2021))
that could enable generation tasks, while (Kalyan et al., 2021) mainly
discusses encoder-based PLMs (BERT or BERT variants). Therefore, we believe
there is a requirement for a survey paper that can review the recent progress
of pre-trained language models in the biomedical domain from a multi-scale
perspective.

##### How do we collect the papers?

In this survey, we collected over a hundred related papers. We used Google
Scholar as the main search engine, and also adopted MedPub, Web of Science, as
an important tool to discover related papers. In addition, we screened most of
the related conferences such as ACL, EMNLP, NAACL, etc. The major keywords we
used including medical pre-trained language model, clinical pre-trained
language model, biological language model, etc. Plus, we take Med-BERT (Rasmy
et al., 2020), BioBERT (Lee et al., 2020b), SciBERT (Beltagy et al., 2019),
ClinicalBert (Huang et al., 2019a), COVID-twitter-BERT (Müller et al., 2020)
as the seed papers to check papers that cited them.

##### Contribution

The contributions of the paper can be summarized as follows:

  * •

We give a comprehensive review to summarize existing biomedical PLMs models,
including training data source, model variants, shared competitions,
downstream tasks, etc.

  * •

We enumerate related resources of existing PLMs and their detailed
configuration, which would facilitate beginners to use them.

  * •

We propose a taxonomy of biomedical PTMs, which categorizes existing PTMs from
various perspectives. This could help to standardize the technical methods for
biomedical PLMs.

  * •

We discuss the limitation and future trends. This will be beneficial for
beginners from both the computer science and bioinformatics fields.

##### Organization

The paper is organized as below: Sec.2 introduces the general pre-trained
language models, and Sec.3 introduces the basic methodology to apply pre-
trained language models in the biomedical domain and proposes a taxonomy.
Sec.4 discusses the training process of language models in the pre-training
phase, and their applications in the fine-tuning phase is illustrated in
Sec.5. More discussions about limitations and future directions is in Sec. 6.
We conclude in Sec.7.

##  2\. Background: Pre-trained Language Models

Pre-trained language models (PLMs) have been widely used in computer visions,
natural language processing, etc., to effectively capture the linguistic
information and knowledge inherited in natural languages. In this paper, we
mainly discuss pre-trained language in NLP tokens 222Tokens usually refers to
words or subwords in NLP, and also protein in the biomedical domain.. One can
read the review paper of PLMs in (Qiu et al., 2020) for more details.

Previously, there were many typical methods to build token representation
(e.g., word vectors) from plain corpora. For example, (Pennington et al.,
2014; Mikolov et al., 2013) build a one-to-one mapping between words and their
vectors, which is called ‘static word embedding’ since it is static and not
related to word context. However, it is well known that words often express
different meanings in different contexts. Inspired by (Peters et al., 2018),
many pre-trained language models adopt ‘contextualized word embedding’ to
model words in a specific context. For ‘contextualized word embedding’, the
vector for a word depends on its specific usage in a context. For example, the
meanings of ‘bank’ in ‘river bank’ and in ‘money bank’ are supposed to have
some difference. The ‘contextualized word embedding’ largely improves the
quality of word representation in various tasks (Devlin et al., 2018).

In this section, we will introduce three basic ingredients of biomedical pre-
training models: the training objective with self-supervised tasks in Sec. 2.1
and basic neural network models in Sec. 2.2, and training paradigm in Sec. 2.3

###  2.1. Pre-trained Language Models

Language models could be considered as an instance of self-supervision.
Compared to data-hungry supervised learning, which usually needs annotations
from humans, language models could make use of massive amounts and cheap plain
corpora from the internet, books, etc. In language models, the next word is a
natural label for a context sentence as a next word prediction task, or one
can artificially mask a known word and then predict it. The paradigm that uses
the unstructured data itself to generate labels (for example, the next word or
the masked word in language models) and train supervised models (language
models) to predict labels thereof is called ‘self-supervision learning’.
Language models are therefore also referred as “auxiliary task” or “pre-
training task”, in which the learned representations in language models can be
used as an initial model for various downstream supervised tasks.

Before self-supervision was invented, neural language models (Bengio et al.,
2003; Mikolov et al., 2013) had been proposed for decades, and their variants
(Mikolov et al., 2013) later became the backbones of modern NLP to provide
pre-trained word features. The overview of neural language models is shown in
Table 1. (Pennington et al., 2014; Mikolov et al., 2013) (including Skip-Gram,
CBow and Glove) typically use linear architecture to conduct calculation
between word vectors, resulting in efficient training. Recently, BERT (Devlin
et al., 2018), and GPT (Radford et al., 2018) proposed to use multiple layers
of transformers as the basic architecture, but that led to much more
parameters. Thus, subwords (limited set of common sub-word units, also called
‘wordpieces’ (Wu et al., 2016) 333For example, a word acknowledgment is
divided to two subwords acknowledge and ##ment, in which ## means that the
current subword is not an independent subword. ) are used to reduce the size
of word vocabulary and therefore save parameters. It is noticed that this is
also beneficial to handle rare words.

Table 1. Typical ways for word vectors and language models. $X=\\{a,b,c,d,e\\}$ is an example text sequence. ELMO, BERT, and GPT usually work on much longer sequences than neural language models (NLMs), Skip-gram and CBOW. Model | Type | Architecture | Task |  Loss function
---|---|---|---|---
NLM (Bengio et al., 2003) | static | 1-layer MLP | $(a,b)\rightarrow c$ |
predicting the next word |  $-\sum_{i=1}^{T}logp(x_{i}|\\{x_{1},...,x_{i-1}\\})$
Skip-Gram (Mikolov et al., 2013) | static | 1-layer MLP | $b\rightarrow c,\quad b\rightarrow a$ |
predicting neighboring words |  $-\sum_{i=1}^{T}logp(\\{x_{i-o},...,x_{i-1},x_{i+1},...,x_{i+o}\\}|x_{i})$, ($o$ is the window size)
CBow (Mikolov et al., 2013) | static | 1-layer MLP | $(a,c)\rightarrow b$ |
predicting central words |  $-\sum_{i=1}^{T}logp(x_{i}|\\{x_{i-o},...,x_{i-1},x_{i+1},...,x_{i+o}\\})$, ($o$ is the window size)
Glove (Pennington et al., 2014) | static | 1-layer MLP | $\vec{w_{i}}^{T}\vec{w_{j}}\propto logp(\\#(w_{i}w_{j}))$ |
predicting the log co-occurrence count |  $-\sum_{i=1,j=1}^{T}f(x_{ij})(\vec{w_{i}}^{T}\vec{w_{j}}+b_{i}+c_{j}-logx_{ij}),(x_{ij}=p(\\#(w_{i}w_{j})))$
ELMO (Peters et al., 2018) | contextualized | LSTM | $(a,b,c,d)\rightarrow e,\quad(e,d,c,b)\rightarrow a$ |
bi-directional language model |  $-\sum_{i=1}^{T}logp(x_{i}|\\{x_{1},...,x_{i-1}\\})+logp(x_{i}|\\{x_{i+1},...,x_{T}\\})$
BERT (Devlin et al., 2018) | contextualized | Transformers | $(a,[\textrm{\tt mask}],c,[\textrm{\tt mask}],e)\rightarrow(\\_,b,\\_,d,\\_)$ |
predicting masked words |  $-\sum_{x\in mask(x)}logp(x|\hat{X})$, $mask(x)$ are masked words, $\hat{x}$ is the rest words
GPT (Radford et al., 2018) | contextualized | Transformers | $(a,b,c,d)\rightarrow e$ |
predicting the next word |  $-\sum_{i=1}^{T}logp(x_{i}|\\{x_{1},...,x_{i-1}\\})$

The training corpora for pre-trained language models (like BERT and GPT)
including: 1) online text like Wikipedia 444https://dumps.wikimedia.org/, 2)
existing books that have been digitized like BooksCorpus (Zhu et al., 2015),
3) crawled online corpora 555https://commoncrawl.org/. Pre-trained language
models trained by these corpora are usually able to capture the common sense
knowledge in the general domain. Therefore, it needs some efforts to make them
be tailored to a specific domain like the biomedical domain, to capture the
domain knowledge.

In this paper, we mainly discuss the pre-trained language models, e.g., BERT
and GPT. The main difference between them is that BERT is a textual encoder to
encode a given document, while GPT is a textual decoder to decode a new
document. This can also be considered as the difference between the
discriminative model and generative model in machine learning. BERT is mainly
used for the discriminative prediction/inference for a given text, like
information extraction, text classification, named entity recognition,
relation extraction, and non-generative question answering, as shown in Sec.
5. The latter is to generate texts, for example, text summarization, text
completions, generative question answering and translations.

###  2.2. Backbone Networks in Language Models

The success of pre-trained language models is also attributed to the
development of their base backbone network, from LSTM (Hochreiter and
Schmidhuber, 1997) to Transformer. Before Transformer (Vaswani et al., 2017)
was invented, LSTM was widely used and became the first base architecture of
the pre-trained language model (ELMO). However, because of its recurrence
structure, it is computationally expensive to scale up LSTM to being deeper in
layers. To this end, Transformer is proposed and becomes the backbone of
modern NLP. Transformers are better architecture can be attributed to: 1)
efficiency: a recurrent-free architecture which could compute the individual
token in parallel, 2) effectiveness: attention allows spatial interaction
across tokens that dynamically depends on the input itself. In this section,
we briefly introduce the two typical architecture in pre-trained language
models, namely, LSTM and Transformers.

##### LSTM

Long short-term memory (LSTM) is a recurrent neural network (RNN) architecture
for sequential modeling. Unlike standard feed-forward neural networks
processing single data points (such as images), LSTM can deal with entire
sequences of data (such as text, speech, or video). A common LSTM unit is
composed of a cell, an input gate, an output gate and a forget gate. The cell
learns hidden states over arbitrary time intervals and the three gates
regulate the flow of information into and out of the cell. LSTM networks are
well-suited for time series data and were developed to deal with the vanishing
gradient problem that can be encountered when training traditional RNNs.
(Peters et al., 2018) tried to adopt Long and Short term memory network (LSTM)
in pre-trained language, which naturally process token sequentially.

##### Transformer

The backbone of most pre-trained language models (e.g., BERT and GPT) is a
neural network called ‘Transformer ’ building upon self-attention networks
(SANs) and feed-forward networks (FFNs). SAN is used to facilitate interaction
between tokens, while FNN is used to refine the token presentation using non-
linear transformation. Since Transformer has been the de facto backbone to
replace recurrent and convolutional units, BERT and GPT adopt the Transformer
as the backbone network. The transformer is superior in terms of capacity and
scalability thanks to, 1) discard recurrent units and process tokens more
efficiently in parallel with the position embeddings(Wang et al., 2019, 2021),
2) relieve saturation issue of expressive power with large-scale data and very
deep layers due to the well-designed architecture including residual
connections, layer normalization, and etc. Interestingly, AlphaFold2 (Jumper
et al., 2021) also borrows some insights to design the so-called ‘Evoformer’
as the core component in its architecture. In Table 2, we introduce some
typical pre-trained language models in general NLP domains, based on these two
backbone neural networks.

Table 2. Representative pre-trained language models in the general domain. NSP means the next sentence prediction task. Model | Objective | Backbone network | Comments
---|---|---|---
ELMO (Peters et al., 2018) | bidirectional LM | Bi-LSTM | the first contextualized word representation
BERT (Devlin et al., 2018) | masked LM, NSP | Transformer (Encoder) | the most commonly-used pre-trained language model
Roberta (Liu et al., 2019) | masked LM | Transformer (Encoder) | a longer-trained BERT variant using more data
ALBERT (Lan et al., 2019) | masked LM, NSP | Transformer (Encoder) | a BERT variant with shared weights and a factorized word embedding
XLNET (Yang et al., 2020b) | generalized autoregressive pretraining | Transformer (Encoder) | a generalized autoregressive pretraining using bidirectional contexts
Electra (Clark et al., 2020) | replaced token prediction | Transformer (Encoder) | a pre-trained based LM trained by replaced token prediction
GPT (Radford et al., 2018) | autoregressive language model | Transformer (Decoder) | a pre-trained based LM for autoregressive generation
T5 (Raffel et al., 2020) | Seq2Seq | Transformer (En-Decoder) | a pre-trained based LM for seq2seq generation
Figure 4.  The difference between Encoder, Decoder and En-Decoder pre-trained
language models. {A,B,C,D,E,F,G} is an example token sequence, while
{A’,B’,C’,D’,E’,F’,G’} is the corresponding abstract representation or labels.

##### Representative Pre-trained language models

Based on whether the input or output is a textual sequence or the label, pre-
trained language models are mainly divided into three categories: Encoder-
only, Decoder-only, and En-Decoder. Encoder-only pre-trained models are mainly
for text classification and sequential labeling tasks, while pre-trained
models equipped with the decoder could deal with generation-related tasks like
translation, summarization, and language models. See Fig. 4 for the
difference: an Encoder model predicts labels for each input tokens (the
brownish yellow color), a Decoder model generates a sequence of tokens w.r.t.
a probability distribution (the blue color), an En-Decoder model predicts a
new sequence conditioned on a given sequence (a.k.a. Seq2Seq) (the grey
color).

###  2.3. Pre-training and Fine-tuning Paradigm in PLMs

One challenge to use PLMs in downstream tasks is that there are two gaps
between PLMs and downstream tasks, the task gap and domain gap. The task gap
means the meta-task in PLMs (usually masked language model in BERT or causal
language model in GPT) usually can not directly be tailored to most downstream
tasks (e.g. sentimental classification). The domain gap refers to the
difference between the trained corpora in PLMs and the needed domain in a
specific downstream task. The adaptation of both task gap and domain gap is
crucial.

##### Adaption

To use the pre-trained language model in a downstream task, it is suggested to
adopt both the domain and task (Rongali et al., 2021; Zhang et al., 2020e;
Gururangan et al., 2020a; Gu et al., 2021), see Table. 3 for the difference.
The domain adaption suggests continuing training pre-trained models trained
from a general domain, in the target domain, e.g., biomedical domain. The task
adaption refers to fine-tuning on similar downstream tasks. In this paper,
without specifying, we mainly discuss the domain-adapted pre-trained models in
various downstream tasks. The task adaption is not the main concern in this
review. Take BERT as an example, BERT is first trained using next sentence
predictions (NSP) and masked language models in the pre-training phase. Such
pre-trained BERT will be used as the initial feature extractor. BERT with an
additional classifier layer is then be fine-tuned to optimize the objective of
down-stream tasks (like MNLI (Williams et al., 2018), NER (Tjong Kim Sang and
De Meulder, 2003), and SQuAD (Rajpurkar et al., 2016)).

Table 3. Categories of usage of pre-trained models Category | Data | Task
---|---|---
Pre-training | general domain | pre-training task
Domain adaption | target domain | pre-training task
Task adaption | general domain | downstream task
Fine-tuning | target domain | downstream task

##  3\. PLMs in Biomedical Domain: Why and How?

Recently, the pre-trained language models have been widely applied to various
NLP tasks and achieved significant improvement in performance. The pre-trained
language models are widely used because: 1) Pre-training on the huge text
corpus can learn universal language representations and help with the
downstream tasks. 2) Pre-training provides a better model initialization,
which usually leads to a better generalization performance and speeds up
convergence on the target task. 3) Pre-training can be regarded as a kind of
regularization to avoid overfitting on small data (Qiu et al., 2020). Pre-
trained language models are firstly trained in general plain corpora from the
Internet, like Wikipedia or crawled webpages. Except for the general domain,
(Feng et al., 2020) trains CodeBERT in the programming language and (Beltagy
et al., 2019) trains SciBERT on scientific publications and biological
sequence. This paper aims to discuss pre-trained language models in the
biomedical domain.

###  3.1. Motivation

In the biomedical domain, the motivation of using pre-trained language models
are manyfold.

  * •

Firstly, the biomedical domain involves many sequential tokens (like
biomedical corpora and history of electronic health records) that usually lack
annotations. However, these sequential data were previously thought of as
difficult to model. Thanks to pre-trained language models, it has been
empirically demonstrated to train these sequential data in a self-supervised
manner effectively. This would open a new door for biomedical pre-trained
language models.

  * •

Second, annotated data in the biomedical domain is usually limited at scale.
Some extreme cases in machine learning are called ‘zero-shot’ or ‘few-shot’.
More recently, GPT3 shows that language models have the potential for few-shot
learning and even zero-shot learning (Brown et al., 2020). Therefore, a well-
trained pre-trained language model is more crucial to provide a richer feature
extractor, which may slightly reduce the dependence on annotated data.

  * •

Plus, the biomedical domain is more knowledge-intensive than the general
domain, since some tasks may need some domain expert knowledge, while pre-
trained language models could serve as an easily-used soft knowledge base
(Petroni et al., 2019) that captures implicit knowledge from large-scale plain
documents without human annotations. More recently, GPT3 has been shown to
have the potential to ‘remember’ many complicated common knowledge (Brown et
al., 2020).

  * •

Lastly, other than text, various types of biological sequential data exist in
biomedical domains, like protein sequences and DNA sequences. Using these data
to trained language models has shown a great success in traditional biological
tasks like protein structure predictions. Therefore, it is expected that pre-
trained language models to solve more challenging problems in biology.

###  3.2. How to tailor PLMs to the Biomedical Domain

The pre-trained language model (Devlin et al., 2018) is a new two-stage
paradigm for natural language tasks. In the first phase, it trains the
language model with a self-supervised meta-task in task-agnostic corpora
(e.g., masked language model and casual language model), and then in the
second phase, it fine-tunes the pre-trained language model to (usually small-
scaled) specific downstream tasks. The trivial way to use a pre-trained
language model on the biomedical domain is to fine-tune it with the domain
data. However, additional adaption is usually adopted to transfer the learned
domain knowledge and task characteristics to the target domain and task. In
this paper, we group the usage of the pre-trained language model into the
categories in Fig. 5. The adaption is basically two-fold: transfer the domain
or task characteristics. The former refers to how to transfer a general pre-
trained language model to a specific biomedical domain.

####  3.2.1. Pre-training

The main challenge in the biomedical domain is that the medical jargon and
abbreviations consist of many terms that are composed of Latin and/or Greek
parts, leads to a gap between the general and medical domains. Moreover,
clinical notes have different syntax and grammar than books or encyclopedias.
This needs to design different vocabulary, resulting in that existing pre-
trained models with different vocabularies (like general BERT or GPT) probably
cannot be used, and training scratch is necessary.

##### Continue Training or from scratch

One may reuse a pre-trained language model from the general domain (e.g.,
general Wikipedia pages 666https://en.wikipedia.org/ or Google books) and then
continue pre-training a few epochs in the new (target) domain (i.e.,
biomedical domain). In the case when the corpora in the target domain are
large-scale enough, one can also directly train the model from scratch since
there is no need to reuse the general knowledge.

####  3.2.2. Fine-tuning

Based on well-trained models, one has to adapt them to downstream tasks. This
is typically implemented to replace the mask language model prediction head
and next sentence prediction head with a downstream prediction head, e.g.,
classification head, or sequence labeling heads.

Since the downstream tasks usually have much less training data than those
used in pre-training, fine-tuning is an unstable process. (Sun et al., 2019b)
investigate different fine-tuning methods of BERT on the natural text
classification tasks. (Mosbach et al., 2020) argue that the fine-tuning
instability is due to vanishing gradients. (Merchant et al., 2020) observe
that fine-tuning mainly modifies the top layers of BERT. Unfortunately, the
solutions (e.g. hyper-parameters of which layer to fine-tune) proposed in
those papers cannot be easily translated to other settings. To automate this
process, automatic hyper-parameter tuning (e.g. Bayesian optimization (Brochu
et al., 2010; Turner et al., 2021)) can come into help.

###  3.3. Taxonomy of Usage

To better standardise the domain of biomedical pre-trained language models, we
propose a systematical taxonomy from the multiple-scale perspective (e.g.,
data, task, and approaches.) to explain the usage of pre-trained language
models in the biomedical domain.

##### Data

The data resources of PLMs range from electronic health records, scientific
literature, social media, online knowledge bases, and biomedical sequences.
The detail of data resources is in Sec. 4.1. In these resources, token types
could be text, proteins, DNAs, etc. For textual data, most of these data are
in English.

##### Models

The model architecture could be encoder architecture, decoder architecture, or
a combination of the both. The base models Typically are Transformer, while
some earlier works adopt LSTM. Finally, one could either reuse an existing
model (e.g., BERT or BioBERT) and continue training, or train a new model from
scratch; the choice of the two above strategies depends on the specific
scenario.

##### Applications

The downstream tasks of biomedical PLMs include typical NLP applications such
as information extraction, text classification, sentence similarity, question
answering, dialogue system, summarization, and natural language inference. See
Sec. 5 for more details.

We also discuss their future trends and limitations in Sec. 6.2.

Figure 5. Taxonomy of PTMs in the biomedical domain.

##  4\. PLMs in Biomedical Domain

Pre-trained language models are effective partially because they are not data-
hungry to labeled data (sometimes called annotated data), which is essential
for supervised learning. Self-supervised learning, which pre-trained language
models rely on, usually adopts plain unstructured corpora in a format of a
sequence of tokens. It is believed that the pre-trained language model can
always benefit from more training corpora. To achieve better performance in
the domain-specific downstream tasks, it is also intuitive that the in-domain
data pre-training is necessary. In the biomedical domain, the in-domain data
can be text in the electronic health records, scientific literature, and
online social media, or biological sequence (e.g., DNA pieces), which will be
introduced in the Sec. 4.1. Next, in the Sec. 4.2, we will introduce existing
pre-trained models in the biomedical domain, which are pre-trained from the
in-domain data as introduced in the Sec. 4.1. We will give an overview of
these models and explain some differences between them. We expect to help one
from both the bioinformatics and computer science community to get knowledge
of the biomedical domain specific pre-trained language model quickly.

###  4.1. Unstructured Data for Pre-trained Language Models

Unstructured plain data for pre-trained language models mainly include
electronic health records, scientific publications, social media text, or
other biological sequence like protein. An overview of EHR mining can be seen
in (Yadav et al., 2018; Eichelberg et al., 2005), and (Gonzalez-Hernandez et
al., 2017) discussed both health records and social media text. One can also
check (Kalyan and Sangeetha, 2020) for some systematic overview of biomedical
textual corpora.

####  4.1.1. Electronic Health Record

Electronic health record (EHR) is a collection of patient and population
electronically stored health information in a digital format that may include
demographics, medical history, medication and allergies, immunization status,
laboratory test results, radiology images, vital signs, personal statistics
like age and weight, and billing information. One can check (Solares et al.,
2020; Weng and Szolovits, 2019) for details about EHR with deep learning.
Assessing such records may be restricted to limited organizations, which
hinders its widespread to the public. The reason may involve some privacy
issues.

##### MIMIC III

Medical Information Mart for Intensive Care III dataset (Johnson et al., 2016)
777https://mimic.mit.edu/ is one of the most popular EHR datasets that
consists of the electronic health records of 58,976 unique hospital admissions
from 38,597 patients in the intensive care unit of the Beth Israel Deaconess
Medical Center between 2001 and 2012. In addition, there are 2,083,180 de-
identified notes associated with the admissions.

##### CPRD

Clinical Practice Research Datalink (CPRD) (Herrett et al., 2015) is primary
the care database of anonymized medical records from 674 general physicians
(GP) practices in the UK, which involves over 11.3 million patients. It
consists of data on demographics, symptoms, tests, diagnoses, therapies, and
health-related behaviors. It is also linked to secondary care (i.e., hospital
episode statistics, or HES) and other health and administrative databases
(e.g., office for national statistics’ death registration). With 4.4 million
actives (alive, currently registered) patients meeting quality criteria,
approximately 6.9% of the UK population are included, this shows that patients
are broadly representative of the UK general population in terms of age, sex
and ethnicity. As a result, CPRD has been widely used across countries and
spawned a lot of scientific research output.

####  4.1.2. Scientific Publications

Scientific publications are another source for biomedical pre-trained language
models since we expect that biomedical knowledge may be encapsulated in
scientific publications. Moreover, such knowledge may not be limited to
traditional common knowledge, but also involves some state-of-art research
output that may be discovered by recent literature.

##### BREATHE

Biomedical Research Extensive Archive To Help Everyone (BREATHE)
888https://cloud.google.com/blog/products/ai-machine-learning/google-ai-
community-used-cloud-to-help-biomedical-researchers, is a large and diverse
dataset collection of biomedical research articles from leading medical
archives. It contains titles, abstracts, and full-body texts. The dataset
collection process was done with public APIs that were used when available.
The primary advantage of the BREATHE dataset is its source diversity. BREATHE
is from nine sources including BMJ, arXiv, medRxiv, bioRxiv, CORD-19, Springer
Nature, NCBI, JAMA, and BioASQ (Chakraborty et al., 2020). BREATHE v1.0
contains more than 6M articles and about 4 billion words. BREATHE v2.0 is the
most recent version.

##### PubMed

PubMed 999https://pubmed.ncbi.nlm.nih.gov/ is a free search engine accessing
primarily the MEDLINE database of references and abstracts on life sciences
and biomedical topics. PubMed comprises more than 32 million citations for
biomedical literature from MEDLINE, life science journals, and online books.
Citations may include links to full-text content from PubMed Central and
publisher websites. PubMed abstracts (PubMed) have 4.5B words, and PubMed
Central full-text articles (PMC) have 13.5B words.

####  4.1.3.  Social Media

Users post information on social media, which many contain biomedical
information. We mainly introduce Reddit and Tweets as examples.

##### Reddit

Reddit is an American social news aggregation, web content rating, and
discussion website. Registered members submit content to the site, such as
links, text posts, images, and videos, then voted up or down by other members.
Posts are organized by subject into user-created boards called ”communities”
or ”subreddits”, which cover a variety of topics such as news, politics,
religion, science, movies, video games, music, books, sports, fitness,
cooking, pets, and image-sharing. Submissions with more up-votes appear
towards the top of their subreddit and, if they receive enough up-votes,
ultimately on the site’s front page. Despite strict rules prohibiting
harassment, Reddit’s administrators have to moderate the communities and, on
occasion, close them. COMETA corpos(Basaldella et al., 2020) crawled health-
themed forums on Reddit using Pushshift (Baumgartner et al., 2020) and
Reddit’s own APIs.

##### Tweets

Twitter is an American micro-blogging and social networking service on which
users post and interact with messages known as ”tweets”. Registered users can
post, like, and retweet tweets. Tweets were originally restricted to 140
characters, but the limit was doubled to 280 for non-CJK languages in November
2017. Audio and video tweets remain limited to 140 seconds for most accounts.
The COVID-twitter-BERT (Müller et al., 2020) is trained on a corpus of 160M
tweets about the coronavirus collected through the Crowdbreaks platform
(Müller and Salathé, 2019) during the period from January 12 to April 16,
2020.

####  4.1.4.  Online Medical Knowledge Sources

Other than unstructured text, there are some online medical knowledge source
that are well-organized. For example, UMLS provides biomedical concepts that
may benefit biomedical pre-trained language models.

##### UMLS

Unified Medical Language System (UMLS) (Bodenreider, 2004) (http://
umlsks.nlm.nih.gov) is a repository of biomedical vocabularies developed by
the US National Library of Medicine. The UMLS has over 2 million names for
900, 000 concepts from more than 60 families of biomedical vocabularies, as
well as 12 million relations among these concepts. These vocabularies include
the NCBI taxonomy, the Medical Subject Headings (MeSH), Gene Ontology, OMIM
and the Digital Anatomist Symbolic Knowledge Base. The UMLS knowledge sources
are updated every quarter. All vocabularies are freely available for research
purposes within an institution, if a license agreement is signed.

####  4.1.5. Biological Sequences

Other than text, there are various types of biomedical token sequence, for
example, amino acids for proteins. Structure of each protein is fully-
determined by a sequence of amino acids (Anfinsen, 1973). These amino acids
are from a limited-size amino acid vocabulary, of which 20 are commonly
observed. This is similar to that the text is composed of words from a lexicon
vocabulary. In this subsection, we introduce a protein dataset called ‘Pfam’
and DNA sequence dataset from Human Genome Project.

##### Pfam Protein Dataset

The Pfam database 101010http://pfam.xfam.org/ is a large collection of protein
families, in which each protein is represented by multiple sequence alignments
using hidden Markov models. The newest version is Pfam 34.0 which was released
in March 2021 and contains 19,179 families (or called ‘entries’) and 645 clans
111111 Clans are the generated higher-level groupings of related entries in
Pfam. A clan is a collection of entries which are related by sequence
similarity, structure or profile-HMM.. The original purpose of the Pfam
database is for classification of protein families and domains. It creates the
database using a semi-automated method of curating information on known
protein families. Pfam 34.0 contains 47 million sequences, which could be used
to train protein language models.

##### DNA Dataset

The DNA sequence is composed of genomic sequence. The Human Genome Project was
the international research effort to determine the DNA sequence of the entire
human genome. Human Genome Project Results. In 2003, an accurate and complete
human genome sequence was finished two years ahead of schedule and at a cost
less than the original estimated budget. (Ji et al., 2020) uses the reference
human genome GRCh38.p13 primary assembly from GENCODE Release
121212https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.39/. The total
sequence length is about 3 Billion.

###  4.2. Biomedical Pre-trained Language Models

Based on the types of training corpora in the biomedical domain as introduced
in the above section 4.1, we mainly introduce two groups of biomedical pre-
trained language models: biomedical textual language models and protein
language models.

####  4.2.1. Overview of Existing Biomedical Textual Language Models

Since BERT was released, various biomedical pre-trained language models have
been proposed via continue training with in-domain corpora based on the BERT
model or training from scratch. Tab. 4 presents existing pre-trained language
models with used corpora, size, release date, and related web pages.

Table 4. Existing textual biomedical pre-trained models. The base setting is
with 0.1B parameters and the large setting is with 0.3B parameters. Data is
based on the submission in arXiv or publish data of the journal or conference
proceeding.

0  Model Corpora Size Date Link BioBERT (Lee et al., 2020b) PubMed and PMC
base & large 2019.01 https://github.com/dmis-lab/biobert BERT-MIMIC (Si et
al., 2019) MIMIC III base and large 2019.02 - SciBERT (Beltagy et al., 2019)
Semantic Scholar papers base 2019.03 https://github.com/allenai/SciBERT
BioELMo (Jin et al., 2019a) PubMed abstracts 93.6 M 2019.04
https://github.com/Andy-jqa/bioelmo Clinical BERT (Alsentzer et al., 2019) EHR
(MIMIC-III 131313all notes or only discharge summaries in base 2019.04
https://github.com/EmilyAlsentzer/clinicalBERT Clinical BERT (Huang et al.,
2019a) EHR (MIMIC-III) base 2019.05
https://github.com/kexinhuang12345/clinicalBERT BlueBERT (Peng et al., 2019)
PubMed+MIMIC-III base & large 2019.05 https://github.com/ncbi-nlp/bluebert
G-BERT (Shang et al., 2019) MIMIC III 2019.06
https://github.com/jshang123/G-Bert BEHRT (Li et al., 2020c) Clinical Practice
Research Datalink - 2019.07 https://github.com/deepmedicine/BEHRT BioFLAIR
(Sharma and au2, 2019) PubMed abstracts lagre 2019.08
https://github.com/zalandoresearch/flair RadBERT (Meng et al., 2019) RadCore
radiology reports - 2019.12 - EhrBERT (Li et al., 2019) MADE corpus base
2019.12 https://github.com/umassbento/ehrbert Clinical XLNet (Huang et al.,
2019b) EHR (MIMIC-III) base 2019.12
https://github.com/lindvalllab/clinicalXLNet CT-BERT 141414a.k.a, COVID-
twitter-BERT (Müller et al., 2020) Tweets about the coronavirus large 2020.05
https://github.com/digitalepidemiologylab/covid-twitter-bert Med-BERT (Rasmy
et al., 2020) Cerner Health Facts (general EHR) - 2020.05
https://github.com/ZhiGroup/Med-BERT ouBioBERT (Wada et al., 2020) PubMed base
2020.05 https://github.com/sy-wada/blue_benchmark_with_transformers Bio-
ELECTRA (Ozyurt, 2020) PubMed base 2020.05
https://github.com/SciCrunch/bio_electra BERT-XML Anonymous Institution EHR
system small and base 2020.06 PubMedBERT (Gu et al., 2020) PubMed base 2020.07
https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract
MCBERT (Zhang et al., 2020b) Chinese social media, wiki and EHR base 2020.08
https://github.com/alibaba-research/ChineseBLUE BioALBERT (Naseem et al.,
2020) PubMed and PMC base & large 2020.09 https://github.com/usmaann/BioALBERT
151515not working now BRLTM (Meng et al., 2021) private EHR customized 2020.09
https://github.com/lanyexiaosa/brltm BioMegatron (Shin et al., 2020) PubMed
0.3/0.8/1.2B 2020.10 https://ngc.nvidia.com/ ClinicalTransformer
161616including BERT, RoBERTa, ALBERT, ELECTRA, XLNET (Yang et al., 2020a)
MIMIC III base 2020.10 https://github.com/uf-hobi-informatics-
lab/ClinicalTransformerNER Bioreddit-BERT (Basaldella et al., 2020)
healththemed forums on Reddit base 2020.10
https://github.com/cambridgeltl/cometa BioRoBERTa (Lewis et al., 2020)
171717we renamed it PubMed, PMC, and MIMIC-III base & large 2020.11
https://github.com/facebookresearch/bio-lm CODER (Yuan et al., 2020) UMLS
Metathesaurus base 2020.11 https://github.com/GanjinZero/CODER bert-for-
radiology (Bressem et al., 2020) daily clinical reports - 2020.11
https://github.com/rAIdiance/bert-for-radiology BioMedBERT (Chakraborty et
al., 2020) BREATHE large 2020.12 https://github.com/BioMedBERT/biomedbert
LBERT (Warikoo et al., 2021) PubMed base 2020.12
https://github.com/warikoone/LBERT ELECTRAMED (Miolo et al., 2021) PubMed
abstracts base 2021.04 https://github.com/gmpoli/electramed SCIFIVE (Phan et
al., 2021) PubMed Abstract and PMC 220/770M 2021.06
https://github.com/justinphan3110/SciFive MedGPT (Kraljevic et al., 2021)
King’s College Hospital and MIMIC-III customized 2021.07
https://pypi.org/project/medgpt/

We introduce some representative pre-trained language models including
encoder-only pre-trained language models like BioBERT, ClinicalBERT, SciBERT,
and COVID-twitter-BERT, decoder-only pre-trained language model like MedGPT,
and encoder-decoder pre-trained language models like SCIFIVE.

  * •

BioBERT (Lee et al., 2020b) is initialized with the general BERT model and
pre-trained on PubMed abstracts and PMC full-text articles. It is further
fine-tuned for biomedical text mining tasks such as named entity recognition
(NER), question answering, and relation extraction.

  * •

ClinicalBERT (Huang et al., 2019a) is trained on clinical text from
approximately 2M notes in the MIMIC-III database (Johnson et al., 2016), a
publicly available dataset of clinical notes.

  * •

SciBERT (Beltagy et al., 2019) is trained on the large scale of scientific
papers from multi-domain based on the BERT. The training papers are the random
sample of 1.14 M full-text papers from Semantic Scholar, in which $82\%$
articles are from the biomedical domain.

  * •

COVID-twitter-BERT (Müller et al., 2020). is a natural language model to
analyze COVID-19 content on Twitter. The COVID-twitter-BERT model is trained
on a corpus of 160M tweets about the coronavirus collected through the
Crowdbreaks platform during the period from January 12 to April 16, 2020.

  * •

MedGPT (Kraljevic et al., 2021) is a GPT-like language model trained by
patients’ medical history in the format of electronic health records (EHRs).
Given the sequence of past events, MedGPT aims to predict future events like a
diagnosis of a new disorder or complication of an existing disorder.

  * •

SCIFIVE (Phan et al., 2021) is a domain-specific T5 model which is pre-trained
on large biomedical corpora. Like T5, SCIFIVE is a typical Seq2seq paradigm to
transform a input sequence to a output sequence.

####  4.2.2. Discussions on Biomedical Pre-trained Language Models

Here, we will discuss the listed models in various aspects as below:

##### Training corpora: EHR, literature, social media, etc., or the hybrid?

Most pre-trained language models are based on scientific publications e.g.,
PubMed and EHR notes. Note that EHR datasets are usually relatively smaller
than scientific publications datasets or Wikipedia. Hence pre-trained language
models with only EHR datasets are typically trained from the initialization of
well-trained BERT (Alsentzer et al., 2019; Huang et al., 2019a), XLNET(Huang
et al., 2019b), etc. Furthermore, some pre-trained language models (e.g.,
BioRoBERTa (Lewis et al., 2020)) adopt both scientific publications and EHRs.
A few models such as CT-BERT and Bioreddit-BERT (Müller et al., 2020;
Basaldella et al., 2020) adopt social media including Twitter and Reddit.

##### Extra features

Unlike typical text, EHR data usually have some extra features, for example,
disease codes, personal information of patients like age, gender. Such extra
features can be embedded as dense vectors used in some models such as Med-BERT
and BEHRT (Rasmy et al., 2020; Li et al., 2020c) like word embedding, position
embedding, and segment embedding that are used in the input layer of the
Transformer.

##### Training from scratch or continue training

The standard approach to obtain a biomedical pre-trained model is to conduct
continual pre-training from a general-domain pre-trained model like BERT
(Devlin et al., 2018), such as the BioBERT (Yu et al., 2019). Specifically,
this approach would initialize the model with the standard BERT model
including its word vocabulary, which is pre-trained by general Wikipedia and
BookCorpus. Besides, some literature demonstrated training from scratch may
fully make use of in-domain data and reduce the negative effect from out-of-
domain corpora, which may be beneficial for downstream tasks such as
PubMedBERT (Gu et al., 2020).

##### Reusing existing vocabulary or building a new one

To make use of well-trained general pre-trained language models like BERT
(Devlin et al., 2018), one has to reuse its vocabulary. However, Biomedical
NLP is more challenging compared to general NLP because it involves jargon and
abbreviations prevail: clinical notes have different syntax and grammar than
books or encyclopedias. A totally new vocabulary necessarily leads to training
from scratch due to different vocabularies that may be more computationally
expensive.

##### Model size

Typically, big models usually have a bigger capacity that needs more data for
training. However, the biomedical domain usually does have as many corpora as
the general domain. Thus, biomedical pre-trained language models are
relatively smaller than general pre-trained language models. Another reason is
that most of them are based BERT or BERT-like encoder based models, while pre-
trained models with decoder architecture (e.g., GPT, T5) could be bigger than
encoder-based pre-trained models. To the best of our knowledge, the biggest
model is Biomegatron (Shin et al., 2020) with 1.2B parameters. Note that
bigger models take longer for inference and this is unfriendly for those
researchers without enough research computing resources.

##### Being publicly available

Thanks to the open-sourced tradition of computer science, most models have web
pages for downloading and documents for usages. Some of them standardized its
model in huggingface (https://huggingface.co), which will largely be
beneficial for its wide-spreading. However, some models are not available for
the public due to privacy issues even data might have been anonymized (Lehman
et al., 2021).

##### Biomedical pre-trained language models in other languages

Most of the biomedical pre-trained language models are in English. However,
there is an increasing need for biomedical pre-trained language models in
other languages. There are typically two solutions: a multilingual solution or
a purely second-language solution. The former may be beneficial for low-
resource languages the latter are usually used in some rich-resource languages
like Chinese (Zhang et al., 2020b).

###  4.3. Beyond Text: Language Models for Proteins/DNA

There are various biological sequences like proteins and DNA which could be
also treated like linguistic tokens in natural language. Therefore, many
existing work explored training language models for these biological
sequences. The most crucial difference between language models for biological
sequences, and the counterparts for natural language is the tokenization (will
be introduced in Sec. 4.3.1), which leads to different vocabularies like
textual vocabularies. In Sec. 4.3.2, we summarize the existing language models
for these biological sequences.

####  4.3.1. Tokenization for Proteins/DNAs

Like words in text, biological sequences such as proteins and DNA sequences
could also be modeled by language models which typically aims to predict the
next token in a sequence. In contrast to that words are in a relatively-big
vocabulary (typically 10k-100k), the vocabularies for biological sequences are
usually small.

##### Tokenization in Proteins

Since the structure of protein is fully determined by its amino acid sequence
(Anfinsen, 1973), one can represent a protein by its amino acid sequences.
Roughly 500 amino acids have been identified in nature, however only 20 amino
acids are found to make up the proteins in human body. The vocabulary of
protein sequences consists of these 20 typical amino acids.

##### Tokenization in DNAs

The two DNA strands are known as polynucleotides and they are composed of
simpler monomeric units (a.k.a. nucleotides). Each nucleotide contains of one
of four nitrogen-containing nucleobases (i.e., cytosine [C], guanine [G],
adenine [A] or thymine [T]). The two separate polynucleotide are bound
together, according to deterministic base pairing rules ([A] with [T] and [C]
with [G]), with hydrogen bonds. Typically, existing work (Ji et al., 2020)
usually adopts a so-called ‘$K$-mer’ representation for DNA sequences 181818
‘$K$-mer’ is like a $k$-size convolutional window for a sequence. For example,
a DNA sequence ATGGCT will be tokenized to a sequence of $3$-mers {ATG TGC GGC
GCT}  or to a sequence of $5$-mers {ATGGC TGGCT}. for richer contexutual
informaiton for DNAs. By doing so, the vocabulary size will increase to the
$4^{k}+5$ which is exponential to $k$ and additionally pluses five special
tokens ([CLS] , [SEP] , [PAD] , [MASK] , [UNK] ).

####  4.3.2. Language Models for biological sequences

##### Protein language models

Since the commonly-found categories of amino acids are relatively-small,
namely 20. Initially, some work applied character-level language models to
protein to deal with limited-size amino acids. In the beginning, there are
many efforts to training RNN-based language models (Alley et al., 2019; Bepler
and Berger, 2018) for protein sequences. (Heinzinger et al., 2019b, a) trains
a deep bi-directional model ELMo for proteins
191919https://github.com/Rostlab/SeqVec. Other than those protein sequences,
protein language models usually adopt additional features for proteins, e.g.,
global structural similarity between proteins and pairwise residue contact
maps for each protein (Bepler and Berger, 2018). Later, (Rao et al., 2019)
introduces the Tasks Assessing Protein Embeddings (TAPE), a suite of
biologically relevant semi-supervised learning tasks. The authors also train
language models based on LSTM, Transformer, and ResNet on the protein
sequences. Bepler et al (Bepler and Berger, 2019) also proposed a novel
framework based on the LSTM model to learning protein sequence embeddings.
They make their embeddings publicly available at
202020https://github.com/tbepler/protein-sequence-embedding-iclr2019. Rives et
al. (2021) trains a contextual transformer-based language model212121The
trained model and code are available at
https://github.com/facebookresearch/esm. on 250 million protein sequences. The
representations learned by this LM encode multi-level information spanning
from biochemical properties of amino acids to remote homology of proteins.
Different from the above line of approaches, MSA Transformer (Rao et al.,
2021) fits a model separately to each family of proteins. ProtTrans (Elnaggar
et al., 2020) trains a variety of LM models with thousands of GPUs, and also
makes the trained models publicly
available222222https://github.com/agemagician/ProtTrans. ProGen (Madani et
al., 2020) is a generative LM trained on 280M protein sequences conditioned on
taxonomic and keyword tags. ProteinLM (Xiao et al., 2021) was recently
proposed, which trained a large-scale pre-train model for evolutionary-scale
protein sequences, and the trained model is available at232323
https://github.com/THUDM/ProteinLM. More recently, DeepMind develops
Alphafold2 (Jumper et al., 2021) that could predict protein structures with a
high accuracy, in the challenging 14th Critical Assessment of protein
Structure Prediction (CASP14). Most interestingly, there is an embedded
protein language models in Alphafold2, which makes Alphafold2 feasible to make
use of unlabelled protein data. In detail, Alphafold2 adopts an auxiliary
BERT-like loss to predict pre-masked residues in multiple sequence alignments
(MSAs).

##### DNA language models

Proteins are translated from DNA through the genetic code. There are 20
natural amino acids that are used to build the proteins that DNA encodes.
Therefore, amino acids cannot be one-to-one mapped by only four nucleotides.
Some work also explored the potential to build language models on DNA
sequences. DNABERT (Ji et al., 2020) is a bidirectional encoder pre-trained on
genomic DNA sequences with up and downstream nucleotide contexts. Yamada and
Hamada (2021a) pre-trains a BERT on RNA sequences and RNA-binding protein
sequences. All the LMs remain largely the same as those used for human
language data. Designing new architectures and pipelines tailored to
protein/DNA sequences is a promising direction.

##  5\. Fine-tuning PLMs for Biomedical Downstream Tasks

Similar to general domain, pre-trained language models have been widely used
in many biomedical downstream tasks as shown in 5. In this section, we will
introduce recent efforts of applying PLMs in various biomedical downstream
tasks and also introduce some competitions and challenges in related workshops
and conferences, in which pre-trained language models may help.

Table 5. Downstream biomedical tasks based on PLMs Tasks | Papers
---|---
Named entity recognition |  BioBERT (Lee et al., 2020b), BlueBERT (Peng et al., 2019), SciBERT (Beltagy et al., 2019), BioELMo (Jin et al., 2019a), PubMedBERT (Gu et al., 2020), BioMegatron (Shin et al., 2020),Yuan et al (Yuan et al., 2021), Alsentze et al (Alsentzer et al., 2019), Singh et al (Sachan et al., 2018), Zhu et al (Zhu et al., 2018), Si et al (Si et al., 2019), Sheikhshab et al (Sheikhshab et al., 2018), Khan et al (Khan et al., 2020), Giorgi et al (Giorgi et al., 2019), Naseem (Naseem et al., 2020), Gao et al (Gao et al., 2021b), Poerner et al (Poerner et al., 2020), Sun et al (Sun et al., 2020), for Spanish (Akhtyamova, 2020; Hakala and Pyysalo, 2019; Miranda-Escalada et al., 2020), for Chinese (Dai et al., 2019; Jiang et al., 2019; Li et al., 2020e; Xue et al., 2019; Wang and E, 2020; Li et al., 2020a), for French (Copara et al., 2020), for Korean (Kim and Lee, 2020), for Russian (Miftahutdinov et al., 2020), for Arabic (Boudjellal et al., 2021), for Italian (Catelli et al., 2020).
Relation extraction |  BioBERT (Lee et al., 2020b), BlueBERT (Peng et al., 2019), SciBERT (Beltagy et al., 2019), PubMedBERT (Gu et al., 2020), BioMegatron (Shin et al., 2020), Yuan et al (Yuan et al., 2021), Wei et al (Wei et al., 2019), Hafiane et al (Hafiane et al., 2020), Thillaisundaram et al (Thillaisundaram and Togia, 2019), Su et al (Su and Vijay-Shanker, 2020), (Su et al., 2021), Xue et al (Xue et al., 2019), Chen et al (Chen et al., 2019c), Lin et al (Lin et al., 2019b; Lin et al., 2020b), Guan et al (Guan et al., 2020), Liu et al (Liu et al., 2020a), Sui et al (Sui et al., 2020).
Event detection |  Trieu et al (Trieu et al., 2020), Wadden et al (Wadden et al., 2019), Zhang et al (Zhang et al., 2020g), Ramponi et al (Ramponi et al., 2020), Wang et al (Wang et al., 2020d).
Text classification |  BioBERT (Lee et al., 2020b), BlueBERT (Peng et al., 2019), SciBERT (Beltagy et al., 2019), PubMedBERT (Gu et al., 2020), BioMegatron (Shin et al., 2020), Yuan et al (Yuan et al., 2021), Wei et al (Wei et al., 2019), Gao et al (Gao et al., 2021a), Mascio et al (Mascio et al., 2020), Guo et al (Guo et al., 2020a), (Gutierrez et al., 2020), Sont et al (Song et al., 2019).
Sentence similarity |  BioBERT (Lee et al., 2020b), BlueBERT (Peng et al., 2019), SciBERT (Beltagy et al., 2019), PubMedBERT (Gu et al., 2020), BioMegatron (Shin et al., 2020), Yuan et al (Yuan et al., 2021), (Wei et al., 2019), Chen et al (Chen et al., 2019b), (Chen et al., 2019a), (Chen et al., 2020), Yang et al (Yang et al., 2020c), Li et al (Li et al., 2020b).
Question answering |  PubMedBERT (Gu et al., 2020), BioMegatron (Shin et al., 2020), Yoon et al (Yoon et al., 2019), Jeong et al (Jeong et al., 2020), Chakraborty et al (Chakraborty et al., 2020), Kamath et al (Kamath et al., 2019), Du et al (Du et al., 2020b), Yoon et al (Yoon et al., 2021), Zhou et al (Zhou et al., 2019), Akdemir et al (Akdemir and Shibuya, 2020), He et al (He et al., 2020), Amherst et al (Rawat et al., 2020), Kommaraju et al (Kommaraju et al., 2020), for COVID-19 (Lee et al., 2020a; Esteva et al., 2020; Reddy et al., 2020; Ngai et al., 2021), Soni et al (Soni and Roberts, 2020), Mairittha et al (Mairittha et al., 2020).
Dialogue Systems |  Zeng at al. (Zeng et al., 2020), DialoGPT (Zhang et al., 2020f), Whang et al. (Whang et al., 2020), Li at el. (Li et al., 2020f)
Text summarization |  Extractive summarization: Du et al (Du et al., 2020a), Moradi et al (Moradi and Samwald, 2019), Padmakumar et al (Padmakumar and He, 2021), Kanwal et al (Kanwal and Rizzo, 2021), Dan et al (Dan et al., 2020), Song et al (Song et al., 2020). Abstractive summarization: Wallace et al (Wallace et al., 2020), Gharebagh et al (Gharebagh et al., 2020). Both: Deyoug et al (DeYoung et al., 2021), Guo et al (Guo et al., 2020b), Kieuvongngam et al (Kieuvongngam et al., 2020).
Natural language inference |  BioELMo (Jin et al., 2019a), BlueBERT (Peng et al., 2019), Sharma et al (Sharma et al., 2019), He et al (He et al., 2020), Zhu et al(Zhu et al., 2021).
Protein and DNA sequence |  (Alley et al., 2019), (Heinzinger et al., 2019b), (Rao et al., 2019), (Rives et al., 2021), MSA Transformer (Rao et al., 2021), ProtTrans (Elnaggar et al., 2020), ProGen (Madani et al., 2020), DNABERT (Ji et al., 2020), (Yamada and Hamada, 2021a)

We first introduce the downstream applications of PLMs in various classical
tasks of the biomedical domain. Similar to the general domain, to evaluate the
effectiveness and facilitate the research development of biomedical pre-
trained language models, the Biomedical Language Understanding Evaluation
(BLUE) benchmark has been proposed in (Peng et al., 2019). BLUE includes five
text mining tasks in biomedical natural language processing, including
sentence similarity, named entity recognition, relation extraction, text
classification, and inference task. However, BLUE has some limitations. It
does not include some important biomedical application tasks such as question
answering and it mixes the applications of clinical data and biomedical
literature. To improve it, (Gu et al., 2020) proposed a novel benchmark, the
Biomedical Language Understanding & Reasoning Benchmark (BLURB). It includes
named entity recognition (NER), evidence-based medical information extraction
(PICO), relation extraction, sentence similarity, document classification, and
question answering task. Moreover, there are works proposed the benchmark in
other languages, such as chinese (Zhang et al., 2020b). In the following, we
will introduce the recent progress of PLMs on these tasks and other critical
tasks in the biomedical domain.

###  5.1. Information Extraction

Information extraction plays a key role on automatically extracting structure
biomedical information (entities, concepts, relations and events) from
unstructured biomedical text data ranging from biomedical literature,
electronic health record (EHR) to biomedical related social media corpus, etc.
One can check a review in (Wang et al., 2018b). It plays an important role in
the applications of intelligent healthcare such as clinical decision support,
biocuration assistance and health monitoring et al. In the biomedical
community, it generally refers to several important sub-tasks including named
entity or concepts recognition, which aims to identify the common biomedical
concept mentions or entity names (such as genes, drug names, adverse effects,
metabolites and diseases et al) of biomedical texts, relation extraction which
determines the relationships among biomedical entities, concepts and
attributes, and event extraction. Similar to the general domain, learning
methods for biomedical information extraction have been rapidly advanced based
on the pre-trained language models in natural language processing recently. We
will next introduce the methods progress based on the pre-trained language
models in each sub-tasks.

####  5.1.1. Named entity recognition

Biomedical named entity recognition (BioNER) is a fundamental task in the
information extraction of biomedical text data. Previous deep learning methods
are generally based on the sequence labeling framework, in which the word
features of texts are automatically extracted with deep learning methods such
as the bidirectional long short-term memory (BiLSTM) and word embeddings, and
then a classification layer based on the conditional random field (CRF) is
trained to predict the entity tag of each word. Recently, inspired by the
effectiveness of pre-trained language models in the various NLP tasks of the
general domain, much work based on PLMs has been proposed for biomedical NER.
In the latest, several biomedical pre-trained language models: BioBERT (Lee et
al., 2020b), BlueBERT (Peng et al., 2019), BioELMo (Jin et al., 2019a) and
SciBERT (Beltagy et al., 2019), have been proposed and achieved great
performance in various NLP tasks including biomedical NER. They conducted
continual pre-training based on the general domain pre-trained language models
including BERT and ELMo with domain-specific datasets. The BioBert model used
PubMed abstracts and PubMed Central full-text articles for further domain-
specific pre-training, while the BlueBERT utilized the PubMed text and de-
identified clinical notes from MIMIC-III. BioBERT is only trained on the
general text and non-clinical biomedical text. To meet the requirement of the
specialized clinical BERT models for clinical narratives (such as physician
notes), Alsentze et al (Alsentzer et al., 2019) proposed the public available
clinical embedding trained on approximately 2 million clinical notes in the
MIMIC-III v1.4 database. It proved that the clinical-specific contextual
embeddings trained on the clinical notes are beneficial to improve the
clinical NER task. Different from them, Sci-BERT constructed its own
vocabulary and pre-trains from the scratch. It is directly pre-trained on the
large scale of mixed domain datasets, in which $18\%$ texts from the computer
science domain and $82\%$ texts from the biomedical domain. In the latest, Gu
et al (Gu et al., 2020) consisted that domain-specific pre-training from
scratch is more effectiveness than domain-specific fine-tuning and mixed-
domain pre-training like the aforementioned methods. They proposed the
PubMedBERT model, which conducted the domain-specific pre-training from
scratch with the pure biomedical domain dataset: PubMed texts. It has achieved
better performance than aforementioned methods on several biomedical NLP tasks
including the biomedical NER task. Yuan et al (Yuan et al., 2021) proposed a
biomedical pre-trained language model KeBioLM to explicitly incorporate the
knowledge from the Unified Medical Language System (UMLS) knowledge bases,
which outperforms other PLMs in the biomedical NER task. Similar to
PubMedBERT, Shin et al (Shin et al., 2020) further presented a larger
BioMegatron model trained on a larger biomedical domain corpus. It has
achieved the state-of-the-art (SOTA) on standard biomedical NLP benchmarks,
including the named entity recognition. For the BioNER task, these models
generally add an LSTM or conditional random field (CRF) based token
classification layer to predict the tag of each token, on top of the
transformer structures of pre-trained language models.

Besides the biomedical pre-trained language models applied in various
biomedical NLP tasks, there are also many works fine-tuning pre-trained
language models for only the biomedical NER task. Singh et al (Sachan et al.,
2018) firstly investigated the pre-trained language model on improving the
performance of biomedical NER. They proposed to make use of unlabeled data:
the PubMed abstract dataset to pre-train the weights of NER model and then
fine-tuned the model with the supervised NER training data. Zhu et al (Zhu et
al., 2018) trained a domain-specific ELMo model in the mixture data of
clinical reports and relevant Wikipedia pages and then utilized it for the
clinical concept extraction. Si et al (Si et al., 2019) presented an analysis
of advanced word embedding methods (including ELMO and BERT) and investigate
their effects on clinical concept extraction tasks. Similar to (Si et al.,
2019), Sheikhshab et al (Sheikhshab et al., 2018) also investigated the
context embeddings from the language model ELMo on improving the biomedical
named entity recognition. Khan et al (Khan et al., 2020) presented a multi-
task transformer-based neural architecture for slot tagging and applied it to
the Biomedical domain (MT-BioNER). It utilized the BioBert as the transformer
encoder layer and the multiple data sets in the task-specific layers. With the
development of general domain language models, Naseem (Naseem et al., 2020)
proposed an effective domain-specific language model bioALBERT trained on the
biomedical domain corpora (PubMed abstracts and PMC full-text articles) for
biomedical named entity recognition. Giorgi et al (Giorgi et al., 2019)
proposed the end-to-end model for jointly extracting named entities and their
relations using the pre-trained language model BERT. It achieved better
performance in the biomedical NER task than (Si et al., 2019) and BlueBERT.
Gao et al (Gao et al., 2021b) explored the effectiveness of transfer learning
based on the pre-trained language models and semi-supervised self-training to
improve the performance of biomedical NER with very limited labeled data. To
save the memory and time consuming of domain adaption, Poerner et al (Poerner
et al., 2020) proposed to train Word2Vec on biomedical domain text and
incorporated it into the general domain BERT to improve the performance of the
biomedical NER task. Instead of treating the BioNER as the sequence labeling
problem, Sun et al (Sun et al., 2020) proposed to consider the BioNER as the
machine reading comprehension (MRC) problem based on BERT. Besides English,
there is much work exploring the pre-trained language models on the BioNER of
other languages, including Chinese (Dai et al., 2019; Jiang et al., 2019; Li
et al., 2020e; Xue et al., 2019; Wang and E, 2020; Li et al., 2020a), Spanish
(Akhtyamova, 2020; Hakala and Pyysalo, 2019; Miranda-Escalada et al., 2020),
French (Copara et al., 2020), Korean (Kim and Lee, 2020), Russian
(Miftahutdinov et al., 2020), Arabic (Boudjellal et al., 2021), Italian
(Catelli et al., 2020). In Table 6, we summary the commonly used datasets in
the BioNER task.

Table 6. Datasets used in the BioNER task. Dataset | Language | Entity type | Text type | Text Genre | Size
---|---|---|---|---|---
BC5-chem (Li et al., 2016) | English | Chemical | Abstract | PubMed | 1,500
BC4-chem (Krallinger et al., 2015) | English | Chemical | Full text | PubMd | 10,000
BC5-disease (Li et al., 2016) | English | Disease | Abstract | PubMed | 1,500
NCBI-disease (Doğan et al., 2014) | English | Disease | Abstract | PubMed | 793
i2b2 2010 (Uzuner et al., 2011) | English | Disease | Report | Clinical records | 871
BC2GM (Smith et al., 2008) | English | Gene/Protein | Sentence | MEDLINE | 20,000
JNLPBA (Kim et al., 2004) | English | Protein,DNA,RNA,cell line | Abstract | MEDLINE | 2,404
LINNAEUS (Gerner et al., 2010) | English | Species | Full text | PMC | 100
Species-800 (Pafilis et al., 2013) | English | Species | Abstract | MEDLINE | 800
EBM PICO (Nye et al., 2018) | English | Participants,interventions, outcomes | Abstract | PubMed | 4,993
CCKS 2017 242424http://www.ccks2017.com/ | Chinese | Body,disease,symptom,test,treatment | Report | Clinical Records | 400
CCKS 2018 252525http://www.ccks2018.cn/ | Chinese | Anatomy,symptom,independent,drug,operation | Report | Clinical Records | 1,000
PharmaCoNER (Agirre et al., 2019) | Spanish | Protein,chemical | Report | Spanish Clinical Case Corpus | 1,000
CANTEMIST (Miranda-Escalada et al., 2020) | Spanish | Tumor morphology | Report | Spanish Clinical Case Corpus | 1,301
CAS (Grabar et al., 2018) | French | Terms,negation,uncertainty | Clinical cases | PubMed | 100

####  5.1.2. Relation Extraction

Biomedical relation extraction (BioRE) is also one of the fundamental tasks in
the information extraction of biomedical texts. It aims to identify the
relationship (semantic correlation) between biomedical entities mentioned
(such as genes, proteins, and diseases) in texts, which has a variety of
biomedical applications such as clinical outcomes prediction, protein
structure prediction, and clinical diagnosis. It is generally be transformed
into the classification problem, to predict the possible relation type of two
identified entities in a given sentence. In the past few years, deep neural
networks including recurrent neural networks (RNNs), convolutional neural
networks (CNNs), and graph neural networks (GNNs) have shown their
effectiveness in the BioRE task, which are efficient to encode the semantic
features of entities and sentences. They rely on labeled data to train the
model. However, the annotated data is quite limited in the biomedical domain.
In the latest, pre-trained language models have been widely explored in the
BioRE. Compared with deep neural network-based methods, they are trained on a
large scale of data, thus can capture more semantic information in the learned
representations. In Table 7, we summarize the commonly used datasets in the
BioRE task.

Wei et al. (2019) conduct the first study to apply pre-trained language models
on the clinical RE task. They investigated fine-tuned BERT on the clinical RE
and combined additional BIO tag features. Similarly, Hafiane et al. (2020)
investigated two transfer learning strategies: frozen and fine-tuning to adapt
BERT on the biomedical RE. Thillaisundaram et al (Thillaisundaram and Togia,
2019) adapted the SciBERT to the BioRE via fine-tuning the representation of
the classification token (CLS). However, it only utilized partial information
of the last layer due to mainly fine-tuning the classification token. To
further explore the potential of utilizing full information in the last layer
to improve performance, Su et al (Su and Vijay-Shanker, 2020) proposed to
utilize the whole last layer when fine-tuning the BERT model on the BioRe
task. To further learn more general representations of entities to improve the
performance, Su et al. (2021) propose to employ the constractive learning to
improve the BERT model for biomedical relation extraction. Xue et al (Xue et
al., 2019) proposed to fine-tune BERT for joint entity and relation extraction
in Chinese medical text, which used the BERT as a shared encoder and the
focused attention to fuse information of NER and RE task. Chen et al (Chen et
al., 2019c) combined BERT with the one-dimensional convolutional neural
network (1d-CNN) to fine-tune the medical relation extraction. Lin et al (Lin
et al., 2019b; Lin et al., 2020b) investigated the BERT model on the clinical
temporal relation extraction. Guan et al (Guan et al., 2020) investigated
several pre-trained language models in predicting the relationships between
clinical events and temporal expressions, including BERT, RoBERTa, ALBERT,
XLNet, BioBERT, ClinicalBERT. They found that RoBERTa generally has the best
performance. To make the biomedical relation extraction in the document with
long-distance dependencies and complex semantics, Liu et al (Liu et al.,
2020a) proposed to use the pre-trained self-attention structure for biomedical
relation extraction in the document level with the entity replacement method.
In many real application scenarios, training the medical relation extraction
model generally requires collecting and storing privacy-sensitive data, which
may conflict with privacy protection. To solve the problem, Sui et al (Sui et
al., 2020) proposed a privacy-preserving medical relation extraction method
FedED based on BERT and federated learning. Moreover, similar to BioNER,
several advanced biomedical domain pre-trained language models as mentioned in
the last section, such as BioBERT (Lee et al., 2020b), BlueBERT (Peng et al.,
2019), SciBERT (Beltagy et al., 2019), PubMedBERT (Gu et al., 2020),
BioMegatron (Shin et al., 2020) have achieved great performance on the BioRE
task. They typically adapt to the BioRE task with the extra binary
classification layer (linear layer or MLP).

Table 7. Datasets used in the BioRE task. Dataset | Entity type | Text type | Relation Size
---|---|---|---
i2b2 2010 (Uzuner et al., 2011) | Medical problems-tests-treatments | Report | 5,261
GAD (Bravo et al., 2015) | Gene–disease | Abstract | 5,330
EU-ADR (Van Mulligen et al., 2012) | Gene–disease | Abstract | 355
DDI (Herrero-Zazo et al., 2013) | Drug–drug | Abstract,full text | 4,701
CHEMPROT (Krallinger et al., 2017) | Protein–chemical | Abstract | 10,031
BioC VI PM (Dogan et al., 2017) | Protein–protein | Full text | 1,629

####  5.1.3. Event Extraction

Event extraction is another important task for mining structured knowledge
from biomedical data. It aims to extract interactions between biological
components (such as protein, gene, metabolic, drug, disease) and the
consequences or effects of these interactions (Ananiadou et al., 2010). It has
wide applications in biomedical ranging from supporting information retrieval,
knowledge base enrichment, and pathway curation. The event generally consists
of event triggers and their arguments (event participants). The triggers are
signal words (generally verbs or nouns) to indicate the appearance and type of
events. The arguments are biomedical entities. The events will finally be
formulated to the graph structures, in which the triggers are connected with
the appropriate argument along with the path. Commonly used datasets are
summarized in Table 8.

Many efforts have been proposed to explore the application of pre-trained
language models on biomedical event extraction recently. Trieu et al (Trieu et
al., 2020) proposed the neural nested event extraction model called
DeepEventMine with the BERT based encoder, in which the task-specific layers
are added on the top of BERT encoder to detect the nested entities and
triggers, roles and nested events. Wadden et al (Wadden et al., 2019) explored
combining the BERT model and graph propagation on the event extraction of a
variety domain including the biomedical domain, to capture both context
information in the sentence and cross-sentence dependencies. They proved that
the long-range dependencies captured by the graph propagation can improve the
performance of the model-based BERT alone. Zhang et al (Zhang et al., 2020g)
investigated transfer learning with the BERT model for Chinese clinical event
detection. Moreover, there are works considering model the biomedical event
extraction task as other NLP tasks. Ramponi et al (Ramponi et al., 2020)
modeled the biomedical event extraction as the sequence labeling problem. They
proposed the neural event extraction model called BEESL, which converted the
event structures into the format of sequence labeling and utilized the BERT
model as the encoder. Wang et al (Wang et al., 2020d) proposed to formulate
the biomedical event extraction as the multi-turn question answering problem
and utilized the question answering system based on the domain-specific
language model SciBERT to achieve great performance.

Table 8. Datasets used in the Biomedical event extraction. Dataset | No. entities | No. triggers | No.relations | No.events
---|---|---|---|---
Cancer Genetics 2013 262626 http://2013.bionlp- st.org/tasks/cancer- genetics | 21,683 | 9,790 | 13,613 | 17,248
EPI 2011 (Ohta et al., 2011) | 16,675 | 2,035 | 3,416 | 2,453
GENIA 2011 (Kim et al., 2011) | 22,673 | 10,210 | 14,840 | 13,560
GENIA 2013 (Kim et al., 2013) | 12,725 | 4,676 | 7,045 | 6,016
Infectious Diseases 2011 (Pyysalo et al., 2011) | 12,788 | 2,155 | 2,621 | 2,779
Pathway Curation 2013 (Pyysalo et al., 2015) | 15,901 | 6,220 | 10,456 | 8,121
Multi-level event extraction (Pyysalo et al., 2012) | 8,291 | 5,554 | 7,588 | 6,677

###  5.2. Text Classification

Text classification is an essential task in biomedical natural language
processing. It aims to classify the biomedical texts into the pre-defined
categories, which plays an important role in the statistical analysis, data
management, retrieval of biomedical data. Compared with the general domain,
text classification in the biomedical domain has more challenges such as data
unbalancing, semantic ambiguity, and irregular data. Fine-tuning pre-trained
language models on the biomedical text classification has attracted great
attention recently, in which the commonly used biomedical text classification
datasets are summarized in Table 9.

Gao et al (Gao et al., 2021a) investigated several different ways of adapting
BERT model on the clinical single-label and multi-label document
classification. However, they found that the BERT or BioBERT model generally
has equal or worse performance in the text classification of clinical data
such as MIMIC-III clinical notes dataset, than a simple CNN model. They
suggested that this is may because BERT or BioBERT models don’t capture
clinical domain knowledge due to trained on the general domain or biomedical
literature datasets. Moreover, BERT model can not handle too long sentences
which is longer than 512 tokens, which may influence the performance. Mascio
et al (Mascio et al., 2020) made a comprehensive analysis of the performance
of various word representation methods (such as Bag-of-Words, Word2Vec, GLoVe,
FastText, BERT, BioBERT) and classification approaches (Bi-LSTM, RNN, CNN) on
the electronic health records. They found that the contextual embeddings from
BERT and BioBERT generally outperform the traditional embeddings, and the
traditional deep neural networks Bi-LSTM enriched with appropriate entity
information and specific domain embeddings have better performance than BERT
and BioBERT. Guo et al (Guo et al., 2020a) compared the performance of three
transformer-based pre-trained language models RoBERTa-base, BERTweet, and
Clinical BioBERT on 25 social media classification datasets, in which 6
datasets are biomedical related. They found that RoBERTa-base and BERTweet
outperform Clinical BioBERT, in which RoBERTa-base can capture general text
semantic characteristics, while BERTweet captures more domain knowledge.
(Gutierrez et al., 2020) also provided an analysis of traditional deep neural
networks and fine-tuning pre-trained language models including BERT and
BioBERT on the performance of multi-label document classification on the
COVID-19 dataset: LitCovid. They found that BERT and BioBERT models have
better performance than deep neural networks such as RNN, CNN and Bi-LSTM in
the datasets. They also found that BioBERT outperforms BERT model due to
domain-specific pre-training. Sont et al (Song et al., 2019) proposed to use
the character-level Bidirectional Encoder Representations from Transformers
(Bert) model to enhance the text convolutional neural network, for the
classification of traditional Chinese medicine cases.

Table 9. Datasets used in the biomedical text classification task. Dataset | Label type | Text type | Data Size
---|---|---|---
HoC (Hanahan and Weinberg, 2000) | Multi-label | PubMed abstract | 1852
MIMIC-III (Johnson et al., 2016) | Multi-label | Discharge summaries | 49785
LitCovid (Gutierrez et al., 2020) | Multi-label | PubMed articles | 23038
CORD-19 Test (Gutierrez et al., 2020) | Multi-label | PubMed articles | 100

###  5.3. Sentence Similarity

Capturing the semantic similarity of sentences plays an important role in the
information extraction and text mining of biomedical data, which is beneficial
for many biomedical applications and downstream tasks such as biomedical
search, evidence sentence retrieval, classification and question answering. It
is generally be formulated into the regression problem to predict the
similarity score of each sentence pair. Recent works have focused on fine-
tuning various pre-trained language models to this task.

Existing pre-trained language models in the biomedical domain and general
domain such as BERT, RoBERTa, BioBERT, SciBERT, ClinicalBERT, BlueBERT,
PubMedBERT, and BioMegatron have achieved great performance in the task, in
which the BioMegatron yielded the best performance due to domain-specific pre-
training and task-specific fine-tuning. However, these models are pre-trained
for yielding embeddings in the token level with the contextual information. To
better capture the semantic information at the sentence level, Chen et al
(Chen et al., 2019b) proposed the first pre-trained open set sentence
embeddings in the biomedical domain, called BioSentVec, which is trained on
over 30 million documents from both biomedical literatures such as PubMed and
clinical notes such as the MIMIC- III Clinical Database. Compared with
existing word embeddings and sentence encoder-based methods, it yields better
performance on both sentence similarity and text classification tasks, due to
better capture the sentence semantic. In (Chen et al., 2019a), they
empirically compared the performance of traditional deep learning methods such
as random forest, RNN, CNN with the pre-trained models BERT and BioBERT, which
shown that pre-trained language models are more effective in capture the
sentence semantic due to pre-trained in large scale of corpora. Moreover, in
(Chen et al., 2020), they further employed the pre-trained sentence embedding
BioSentVec to improve the traditional deep learning models: random forest and
the encoder network on finding similar sentences. Yang et al (Yang et al.,
2020c) explored three pre-trained models including BERT, XLNet, and RoBERTa
for the clinical semantic textual similarity task, in which the XLNet achieves
the best performance among three models. Li et al (Li et al., 2020b) proposed
to integrate the BERT models and bidirectional recurrent neural network (Bi-
RNN) to capture both contextual semantic and semantic textual similarity. The
commonly used sentence similarity datasets are shown in Table 10.

Table 10. Datasets used in the biomedical sentence similarity task. Dataset | Text type | Data Size
---|---|---
BIOSSES (Soğancıoğlu et al., 2017) | PubMed sentences | 100
MedSTS (Wang et al., 2020a) | Clinical report | 174,629
MedSTS_ann (Wang et al., 2020a) | Clinical report | 1,068
n2c2/OHNLP (Wang et al., 2020b) | Clinical report | 1,642

###  5.4. Question Answering

Biomedical question answering (BioQA) is a very important task for the
information retrieval and knowledge acquisition of the large amount of
unstructured data, which can provide service for professionals in the
biomedical domain such as doctors and also general users. It aims to extract
or generate the natural language answers to the given questions. It is a
challenging task due to the lack of large-scale annotated data. In the
biomedical domain, annotating data generally requires domain expertise, this
is time-consuming and expensive. To facilitate the development of the
biomedical question answering, some competitions and datasets have been
proposed such as BioASQ (Tsatsaronis et al., 2012) and MEDIQA
2019272727https://sites.google.com/view/ mediqa2019. Recently, motivated by
the success of unsupervised pre-training models, the fine-tuning and transfer
learning of pre-trained language models have been widely explored in the task.
Most of them are formulated into the machine reading comprehension approach,
which focuses on predicting the text span of answers with the given questions
and passages containing the answers.

Much effort has investigated the BERT and BioBERT on the question answering
task of biomedical literature datasets. Yoon et al (Yoon et al., 2019) applied
the BioBERT to answer biomedical questions such as factoid, list, and yes-no
type questions. To solve the problem of limited training data in the
biomedical domain, they firstly fine-tune BioBERT on the general domain
question answering datasets SQuAD and SQuAD 2.0, and then further fine-tune it
on the task dataset: BioASQ. Instead of the general domain question answering
dataset, Jeong et al (Jeong et al., 2020) proposed to transfer the knowledge
of natural language inference (NLI) with BioBERT to improve the performance.
Inspired by the BioBERT, Chakraborty et al (Chakraborty et al., 2020) proposed
a biomedical domain pre-trained language model BioMedBERT for question
answering (QA) and information retrieval tasks. BioMedBERT is based on the
BERT model pre-trained on a large-scale biomedical literature dataset BREATHE.
Kamath et al (Kamath et al., 2019) compared the effectiveness of pre-trained
models for machine-reading comprehension and question-answering in the general
domain in fine-tuning the biomedical question answering task. They found that
the question answering model fits better to the task. Du et al (Du et al.,
2020b) utilized the BERT model as the encoder to improve the representations
of the question and passage and then used the scaled dot-product attention
mechanism to capture the interaction between them. However, these models can
only extract a single span of passage as the answer and can not detect
multiple spans of the passage when there are multiple answers for the
question. To solve the problem, Yoon et al (Yoon et al., 2021) reformulated
the task as the sequence tagging problem to detect multiple entity spans
simultaneously. They used the BioBERT as the encoder and concatenated the
Sequence Tagging Layer including the linear layer, Bi-LSTM and Bi-LSTM+CRF to
predicting the tag of each token. Zhou et al (Zhou et al., 2019) utilized the
BioBERT and interactive transformer model to both the recognizing question
entailment and question answering task. Similarly, Akdemir et al (Akdemir and
Shibuya, 2020) also explored multi-task learning to improve the performance of
BioBERT on the BioQA task. They proposed the joint BioBERT based model for
biomedical entity recognition and question answering. Some works tried to
incorporate the domain knowledge such as biomedical named entities, into pre-
trained language models. He et al (He et al., 2020) proposed to infuse the
domain knowledge of disease into a series of pre-trained language models
including BERT, BioBERT, SciBERT, ClinicalBERT, BlueBERT, and ALBERT, to
improve their performance. Amherst et al (Rawat et al., 2020) incorporated the
medical entity information with entity embeddings and the auxiliary task on
predicting the logical form of the question to improve the accuracy and
generalization of the BERT model on answering questions. Kommaraju et al.
(2020) also introduced the extra biomedical named entities prediction task to
support the BioBERT and SciBERT on Biomedical QA. To help the prevention of
COVID-19, there are works (Lee et al., 2020a; Esteva et al., 2020; Reddy et
al., 2020; Ngai et al., 2021) build the question answering and information
retrieval system based on BioBERT and BERT.

Besides methods for biomedical literature corpora, other works have been
devoted to proposing question answering models to acquire knowledge from
unstructured electronic health records (EHR). Soni and Roberts (2020)
investigated the performance of various pre-trained language models including
BERT, BioBERT, ClinicalBERT and XLNet on the clinical question answering. They
explored the fine-tuning methods with different datasets including datasets in
the general domain, biomedical and clinical corpora. Mairittha (Mairittha et
al., 2020) explored fine-tune BERT to construct the personalized EHR question
answering system. The commonly used datasets in BioQA are summarized in the
Table 11.

Table 11. Datasets used in the biomedical question answering task. Dataset | Text type | Data Size
---|---|---
PubMedQA (Jin et al., 2019b) | PubMed abstracts | 1,000
BioASQ (Nentidis et al., 2019) | MEDLINE articles | 885
CliCR (Suster and Daelemans, 2018) | Clinical case reports | 100,000
emrQA (Pampari et al., 2018) | Clinical notes | 400,000
cMedQA (Zhang et al., 2018) | Question-answer pair from online community | 61,343
COVID-19 Questions (Tang et al., 2020) | Literature review | 124

###  5.5. Dialogue Systems

Dialogue Systems (DSs) in the biomedical domain has attracted continuous
attention due to the utility applications (e.g., virtual health consultants
and therapists) (Laranjo et al., 2018). The aim of the dialogue system is to
produce a proper response in either a selective (Zhang et al., 2020c; Whang et
al., 2020) or generative (Liu et al., 2020c; Zeng et al., 2020; Zhang et al.,
2020f) way given a dialogue context for the biomedical goals of a user. The
context includes historical utterances from users and systems, biomedical
knowledge base, electronic health records of users, etc. The format of a
response could be various, e.g., a set of structured user goal data (Wei et
al., 2018), a distribution of biomedical labels for diagnosis (Lin et al.,
2019a; Zhang et al., 2020c) and natural language utterances (Zeng et al.,
2020). For different types of contexts and responses, recent work focuses on
end-to-end Dialogue System (DS) (Xu et al., 2019; Zeng et al., 2020) or parts
of four typical DS modules, i.e., Natural Language Understanding (NLU) (Du et
al., 2020c; Shi et al., 2020), Dialogue State Tracking (DST) (Lin et al.,
2019a; Wei et al., 2018), Dialogue Policy Learning (DPL) (Wei et al., 2018;
Xia et al., 2020) and Natural Language Generation (NLG) (Zeng et al., 2020).

Recently, PLMs are well-known for natural language modeling, but it is
nontrivial to pre-train on task datasets that are biased on a specific domain
(Whang et al., 2020). Pre-training DS models in biomedical can be seen as a
task-specific pre-training problem (Gururangan et al., 2020b). It involves two
essential aspects, i.e., biomedicine-domain adaptation and dialogue-task
adaptation.

To adapt to the medical domain, the dominant solution is to pretrain a
language model on a large-scale general/medical corpus and then fine-tune the
model with a medical dialogue dataset. BERT-WWM and BERT-MED (Yan et al.,
2021) first pre-train a BERT on Chinese Wikipedia and medical corpus,
correspondingly, and then fine-tune on $M^{2}$-MedDialog dataset for
understanding the intents and slots of patients. BioBERT and MIMIC-BERT (Wei
et al., 2018) are pre-trained using MIMIC III dataset (Si et al., 2019) and
PubMed articles, respectively, followed by fine-tuning on MZ dataset for
predicting diagnosis actions. Zeng et al. (2020) pre-train Transformer, BERT-
GPT, and GPT on dialog datasets and other large-scale texts, and then fine-
tune models on the Chinese MedDialog dataset for generating clinically correct
and human-like medical responses.

To adapt to DS tasks, the challenges are complex dialogue context modeling
(Pei et al., 2019) and external knowledge (e.g., knowledge base, user
profiles) enrichment (Pei et al., 2021). Naturally, PLMs are talented for
those challenges because: (i) PLMs are based on transformer architecture,
which can capture longer-term dependency for learning complex dialogues
effectively and efficiently (Henderson et al., 2020; Vaswani et al., 2017).
Yan et al. (2021) unify NLU, DPL, NLG tasks into one context-to-response
generation framework and use pretrained GPT2 and MT5 to model complex context
for generating responses. The empirical study proves the positive impact of
PLMs on $M^{2}$-MedDialog dataset. (ii) PLMs can incorporate external
knowledge by pretraining on large-scared corpora in a general domain (Kalyan
et al., 2021; Li et al., 2020f). Shi et al. (2020) use a pretrained BERT as
fixed features to get a well initialized word embedding from the aspect of
transfer learning. DialoGPT (Zhang et al., 2020f) is pre-trained based on
GPT-2 (Radford et al., 2018) with a large in-domain dialogue dataset, and is
able to generate more relevant, informative and coherent responses. Li et al.
(2020f) learn to pretrain language modeling objectives (e.g., MLM and NSP) on
a large-scale general corpus and then fine-tune on medical datasets with task-
specific training objectives. Although recent studies have deployed PLMs
models in medical DSs tasks, medical DS is still under-explored. So we
summarize all available biomedical dialogue datasets in Table 12 for future
research.

Table 12. Datasets used in the biomedical dialogue system tasks. Dataset | Language | Domain | Evaluated Task | Text type | # dialogues
---|---|---|---|---|---
MZ (Wei et al., 2018) | EN | Pediatrics | DPL | Discharge summaries | 710
DX (Xu et al., 2019) | CN | Pediatrics | DPL | Patient-doctor dialogues & patient reports | 527
RD (Liao et al., 2020) | CN | Pediatrics | DPL | Patient-doctor dialogues & patient reports | 1,490
SD (Liao et al., 2020) | CN | 9 domains | DPL | Patient-doctor dialogues & patient reports | 30,000
CMDD (Lin et al., 2019a) | CN | Pediatrics | NLU | Patient-doctor dialogues | 2,067
SAT (Du et al., 2019) | CN | 14 domains | NLU | Patient-doctor dialogues | 2,950
MSL (Shi et al., 2020) | CN | Pediatrics | NLU | Patient-doctor dialogues | 1,652
MIE (Zhang et al., 2020c) | CN | Cardiology | NLU | Patient-doctor dialogues | 1,120
CovidDialog (Yang et al., 2020d) | CN/EN | COVID-19 | NLG | Patient-doctor dialogues | 1,088/603
MedDG (Liu et al., 2020c) | CN | Gastroenterology | NLG | Patient-doctor dialogues | 17,000
MedDialog (Zeng et al., 2020) | CN/EN | 29 domains | NLG | Patient-doctor dialogues & patient reports | 3,407,494/257,332
Chunyu (Lin et al., 2021) | CN | - | NLG | Patient-doctor dialogues | 12,842
KaMed (Li et al., 2021) | CN | 12 domains | NLG | Patient-doctor dialogues | 63,754
$M^{2}$-MedDialog-base (Yan et al., 2021) | CN | 30 domains |  NLU&DPL&NLG | Patient-doctor dialogues & patient reports | 1,557
$M^{2}$-MedDialog-large (Yan et al., 2021) | CN | 40 domains | NLG | Patient-doctor dialogues & patient reports | 95,408

###  5.6. Text Summarization

Automatic text summarizing is an efficient task to conduct information
extraction and retrieval from an ever-growing amount of biomedical texts
(Mishra et al., 2014). It aims to automatically summarize the key information
of single or multiple documents with shorter and fluent texts, which greatly
decreases the time-consuming of acquiring important information. Similar to
the general domain, existing methods generally can be classified into two
categories: extractive summarization methods and abstractive summarization
methods. The former methods extract correlated sentences from given long
documents and concatenate them into the final summary, while the latter
methods generate new sentences based on the information of given long
documents. Therefore, the extractive summarization generally is formulated
into the binary classification, in which the model aims to predict the
sentences whether be selected into the summary, while the abstractive
summarization can be deemed as the conditional text generation problem.

Pre-trained language models have been well-explored on text summarizing of the
general domain. To explore the advanced pre-trained language models in the
text summarizing of biomedical domain, the domain knowledge is incorporated by
existing methods via domain fine-tuning. For biomedical extractive
summarization, Du et al (Du et al., 2020a) proposed a novel model BioBERTSum,
which used the domain-aware pre-trained language model as the encoder and then
fine-tune it on the biomedical extractive summarization task. Gharebagh et al
(Gharebagh et al., 2020) utilized the domain knowledge-the salient medical
ontological terms to help the content selection of the SciBERT based clinical
abstractive summarization model. Moradi et al (Moradi and Samwald, 2019)
proposed the unsupervised extractive summarization for biomedical extractive
summarization. They proposed to use the hierarchical clustering algorithm to
group the contextual embeddings of sentences based on the BERT encoder and
select the most informative sentences from each group to generate the final
summary. Padmakumar et al (Padmakumar and He, 2021) also proposed an
unsupervised extractive summarization model, which used the GPT-2 to encode
the sentences and the pointwise mutual information (PMI) to calculate the
semantic similarity between sentences and documents. The proposed method has
better performance than other similarity-based models on the medical journal
dataset. Kanwal et al (Kanwal and Rizzo, 2021) proposed to fined-tune the BERT
model on the International Classification of Diseases (ICD-9) labeled MIMIC-
III discharge notes for the extractive summarization of electronic health
records. For abstractive summarization, Wallace et al (Wallace et al., 2020)
utilized the Bidirectional and Auto-Regressive Transformers (BART) as the
encoder for generating biomedical evidence summary of multiple clinical
trials. For advancing multi-document summarization of biomedical literature,
Deyoug et al (DeYoung et al., 2021) released a novel dataset of multi-document
summarization on medical studies called $MS^{2}$, which contains over 470k
documents and 20K summaries derived from the biomedical scientific literature.
They also investigated the BART model on the proposed dataset and achieved
promising results. To facilitate the development of methods for generating
plain summaries toward the general public, Guo et al (Guo et al., 2020b)
proposed a novel task of plain language summarization task on the biomedical
scientific reviews, and construct a novel dataset that contains 7805 high-
quality abstract pairs. They explored the BART model for both extractive and
abstractive summarization of the dataset, which was further pre-trained on
general domain dataset CNN/DM and in-domain dataset PubMed and achieved the
best performance.

For the information acquisition of COVID-19 related scientific literature,
Kieuvongngam et al (Kieuvongngam et al., 2020) proposed the BERT and GPT-2
based model for both extractive and abstractive summarization of COVID-19
research literature. There are also works to build the multi-document
summarization system for the information retrieval of COVID-19 research
literature with the Siamese-BERT (Esteva et al., 2020), BioBERT and XLNet (Dan
et al., 2020). Besides biomedical literature and EHR, there are works that
proposed BERT-based summarization methods (Song et al., 2020) toward the
understanding of the medical conversation between patients and doctors.

Table 13. Datasets used in the biomedical text summarization. Dataset | Text type | Summarization type | Data Size
---|---|---|---
COVID-19 (Wang et al., 2020c) | Biomedical literature | Single document | -
MSˆ2 (DeYoung et al., 2021) | Biomedical literature | Multi-document | 470,402
CDSR (Guo et al., 2020b) | Biomedical literature | Single document | 7,805
RCT (Wallace et al., 2020) | Clinical trials | Multi-document | 4,528
PubMed (Cohan et al., 2018) | Biomedical literature | Single document | 215,000

###  5.7. Natural Language Inference

Natural language inference (NLI, also known as the text entailment) is a basic
task for the natural language understanding of biomedical texts. It aims to
infer the relation such as entailment, neutral and contradiction, between two
sentences, named as the premise and hypothesis, which can further benefit
biomedical downstream tasks such as commonsense comprehension, question
answering and evidence inference. In the task, the common neural network model
is based on sentence pair modeling, which encodes the premise and hypothesis
sentences with various neural networks and then classifies the relation
between them with the softmax classifier layer.

Similar to other tasks, pre-trained language models in the biomedical domain
including BioELMo (Jin et al., 2019a) and BlueBERT (Peng et al., 2019), have
shown their effectiveness in the task via task-guided fine-tuning. To
facilitate the development of methods for text inference and entailment in the
medical domain, the MEDIQA 2019 shared task (Abacha et al., 2019) was
organized, in which many participants investigated the SciBERT, BioBERT, and
ClinicalBERT in the medical NLI task. Moreover, some works make efforts on
incorporating the domain knowledge to improve PLMs on the biomedical NLI.
Sharma et al (Sharma et al., 2019) incorporated the embedding of knowledge
graph (UMLS) in the biomedical domain into the BioELMo to improve its
performance. Yadav et al (Yadav et al., 2020) a novel framework Sem-KGN for
the medical textual entailment task, which infused the medical entity
information from the medical knowledge bases into the BERT model. He et al (He
et al., 2020) proposed to infuse the domain knowledge of disease into a series
of pre-trained language models including BERT, BioBERT, SciBERT, ClinicalBERT,
BlueBERT, and ALBERT, to improve their performance in the question answering,
medical inference, and disease name recognition task. Zhu et al (Zhu et al.,
2021) utilized the neural architecture search (NAS) to automatically find a
better transformer structure of the Chinese BERT-wwm-ext model (Cui et al.,
2019) for better medical query understanding.

Table 14. Datasets used in the biomedical natural language inference. Dataset | Text type | Relation Type | Data Size
---|---|---|---
MedNLI (Romanov and Shivade, 2018) | Clinical notes | Entailment, contradiction, or neutral | 14,049
MEDIQA-RQE (Abacha and Demner-Fushman, 2016) | Consumer health questions | Entailment, contradiction | 9,120
CMFAQ (Zhu et al., 2021) | Consumer health questions | Entailment, contradiction | 53,822

###  5.8. Proteins/DNAs Prediction

In this section, we only list some applications that have been well-
investigated or potential, although there are much bigger spaces in biomedical
domains to make use of PLMs.

####  5.8.1. Protein structure predictions

Proteins are essential to life, and knowing their structure can facilitate our
understanding of their function. However, structure of only a small fraction
of proteins is known (Jumper et al., 2021). Predicting 3D structure of a
protein is based solely on its amino acid sequence, a.k.a, ‘protein folding
problem’ (Anfinsen, 1973). To evaluate protein structure predictions, CASP
(Critical Assessment of Structure Prediction) uses proteins with recently
solved structures that have not been deposited in the PDB or publicly
disclosed; it therefore is a blind test for the participators, which is the
gold-standard assessment for protein structure predictions (Moult et al.,
1995; Kryshtafovych et al., 2019). In CASP14, AlphaFold 2 (Jumper et al.,
2021), a model designed by DeepMind achieves much more better performance than
other participating methods (e.g. template-based methods). The authors claims
that AlphaFold 2 could provide precise estimates and it could be confidently
used for protein structure predictions with high reliability. However,
predictions of existing methods including the AlphaFold 2 are more family-
specific than protein-specific, and rely on the evolutionary information
captured in multiple sequence alignments (MSAs). To solve these issues,
Weißenow (Weißenow et al., 2021) proposed to use the attention head from the
pre-trained protein language model ProtT5 without MSAs. Recently, Sturmfels et
al (Sturmfels et al., 2020) presented a new biologically-informed pre-training
task: predicting protein profiles derived from multiple sequence alignments,
which can improve the downstream protein structure prediction task.

####  5.8.2. DNA related applications

There are few work in DNA pre-training, among which DNABERT (Hou et al., 2020)
is the representative one. DNABERT claims that ‘DNABERT could 1) effectively
predicts proximal and core promoter regions; 2) accurately identifies
transcription factor binding sites; 3) allows visualization of important
regions, contexts and sequence motifs; 4) identifying functional genetic
variants with DNABERT; 5) substantially enhances performance and generalizes
to other organisms.’ Hong et al (Hong et al., 2020) proposed to pre-train DNA
vectors to encode enhancers and promoters, and then Incorporated the attention
mechanism to predicting long-range enhancer–promoter interactions (EPIs).
Yamada et al (Yamada and Hamada, 2021b) proposed a novel method based on the
BERT to predict the interactions between RNA sequences and RNA-binding
proteins (RBPs), in which the BERT model is pre-trained on the human reference
genome. Mock et al (Mock et al., 2021) presented the BERTax based on BERT, for
the taxonomic classification of DNA sequences.

###  5.9. Competition Venues

To facilitate the technological developments on biomedical text mining, many
shared tasks and competitions are organized since several years ago, which
focus on various important tasks in the biomedical domain.

  * •

BioNLP workshop. The BioNLP workshop282828https://aclweb.org/aclwiki/SIGBIOMED
has been organized for 20 years and continually promoted the development of
the biomedical domain, in which the community proposed a series of shared
tasks and benchmark datasets. In BioNLP 2019, the BioNLP Open Shared Tasks
(BioNLP-OST) 2019 (Jin-Dong et al., 2019) and the MEDIQA 2019 Shared Task
(Abacha et al., 2019) were organized. The BioNLP-OST 2019 proposed six tasks,
including the information extraction on the bacterial biotopes and phenotypes,
event extraction of genetic and molecular mechanisms, pharmacological
substances, compounds and proteins named entity recognition, integrated
structure, semantics and coreference task, concept extraction for drug
repurposing, and the information retrieval task for neuroscience. The MEDIQA
2019 aims to explore the method development on the natural language inference
(NLI), recognizing question entailment (RQE), and question answering (QA) in
the medical domain. In bioNLP 2021, the MEDIQA 2021 (Abacha et al., 2021)
shared tasks were organized to address three tasks related to the
summarization of medical documents, including the question summarization task,
the multi-answer summarization task, and the radiology report summarization
task.

  * •

BioNLP-OST. The BioNLP Open Shared Tasks (BioNLP-
OST)292929https://2019.bionlp-ost.org/home has been proposed since 2009 and
was motivated to facilitate the development and sharing of methods on various
tasks of the biomedical text mining. It is organized every two years and
organized at different conferences such as BioNLP and EMNLP. The latest
BioNLP-OST 2019 is organized at the BioNLP 2019 as introduced aforementioned.

  * •

BioASQ. The BioASQ303030http://bioasq.org organizes workshops and challenges
on biomedical semantic indexing and question answering. It is held annually
since 2013. In BioASQ 2019, the large-scale biomedical semantic indexing task,
the biomedical information retrieval and question answering task, and
corresponding benchmark datasets are proposed.

  * •

BioCreAtIvE. The Critical Assessment of Information Extraction systems in
Biology (BioCreAtIvE)313131https://biocreative.bioinformatics.udel.edu
organized challenge evaluations for the text mining and information extraction
method on the biological domain since 2004. In the latest BioCreative VII
Challenge, proposed five tracks, in which two tracks are related to the
COVID-19, including the text mining and multi-label topic classification.

  * •

eHealth-KD. The eHealth-KD323232https://knowledge-
learning.github.io/ehealthkd-2019/ organizes challenges on the structure
knowledge extraction of eHealth documents in the Spanish Language. The
eHealth-KD Challenge 2019 proposed the key phrases identification and
classification task, and the semantic relations detection task.

  * •

#SMM4H. The Social Media Mining for Health Applications
(#SMM4H)333333https://healthlanguageprocessing.org/smm4h-2021/ held workshops
and shared tasks related to natural language processing challenges in social
media data for health research since 2015 annually. The shared tasks in the
#SMM4H ’21 involve the information processing methods on Twitter related to
COVID-19, self-report of breast cancer, adverse effect mentions, medication
regimen, and adverse pregnancy outcomes.

Moreover, there are some challenges proposed recently, such as the COVID-19
Open Research Dataset Challenge (CORD-19)343434https://www.kaggle.com/allen-
institute-for-ai/CORD-19-research-challenge in response to the COVID-19
pandemic, EHR DREAM
Challenge353535https://www.synapse.org/#!Synapse:syn18405991/wiki/589657
proposed in October 2019 and focusing on using electronic health record data
to predict patient mortality, and ICLR 2021
workshop363636https://mlpcp21.github.io/pages/challenge devoting to propose
machine learning methods for preventing and combating pandemics. Since the
continual development of pre-trained language models from 2018, in recently
organized challenges, most participants proposed pre-trained language model-
based methods for different tasks.

##  6\. Discussion

###  6.1. Limitations and Concerns

In this subsection, we will mainly discuss the limitations of biomedical pre-
trained language models and raise some concerns about them.

##### Misinformation

The training corpora consist of EHR and social media may include wrong
information. Thus, pre-trained language models pre-trained on them may convey
some misinformation. Furthermore, the biomedical domain itself may have
misclassified disease definitions during its development process.
Misinformation has become much more serious in the biomedical domain than
general domain, since this may lead to some fatal consequences of biomedical
decision-making. However, researchers must be aware of the complexity of
routinely collected electronic health records, including ways to manage
variable completeness. We believe that the predictions from pre-trained
language models should be artificially calibrated by biomedical experts before
it is used by end-uses like the patience or the public.

##### Interpretation issues

Along with the power of neural networks, there is a growing concern about the
interpretability of deep neural networks (DNNs). While in the biomedical
domain, the consequence of bad decisions/predictions may be deadly, thus a
well-interpreted model is more crucial. The interpretation in the biomedical
domain may come from two aspects: (1) biomedical models should be easily
understood and the predictions could be simulated from the raw input, (2) a
(textual) reason should be provided for each prediction. The basic example of
the former (a.k.a, transparency (Lipton, 2018)) is decision trees that could
clearly illustrate the decision path. However, such a transparency goal is
hardly achieved in modern natural language processing especially with pre-
trained language models. More efforts could be made for the latter, one has to
find some textual explanation for each prediction/decision, based on what
doctors and patients could make their own decisions.

##### Identifying causalities from correlations

Similar to interpretability, causality may provide the underlying explanation
of the model decisions. Causality is crucial in many tasks of biomedical
knowledge, e.g., diagnosis, pathology, or systems biology. Causal associations
between biological entities, events, and processes are central to most claims
of interest, see an early review from (Kleinberg and Hripcsak, 2011). With
automatic causality recognition, it could suggest possible causal connections
that may be beneficial for biomedical decisions, which hence greatly reduces
the human workload (Mihăilă et al., 2013).

##### Trade off between coverage or quality?

There is not large-scaled and high-quality training corpora in the biomedical
domain. This means one has to sacrifice its coverage to obtain a high-quality
vertical application, or train a general model with large-scaled yet low-
quality corpora. Pre-trained language models typically consist of many
transformer layers that have many parameters, which usually requires a massive
amount of plain text. This may lead to a general model with great coverage,
but a smaller proportion of high-quality expert knowledge.

##### Heterogeneous training data

For biomedical understandings, there is heterogeneous information including
tables, figures, graphs (fMRI), etc. For example, tables and numbers are
crucial in scientific literature. But most PLMs are unable to interpret tables
and numbers well. To deeply capture the information in these heterogeneous
data, both in-depth data prepossessing and model adaption may be needed.
Especially, multi-domain pre-trained language models in biomedical should be
paid much more attention.

##### Ethics and bias

With the rapid development of AI systems and applications in industrial
products, it should be aware that they should not introduce any bias for a
special groups or populations (Mehrabi et al., 2021), and some of efforts were
taken in the NLP field (Zhao et al., 2017; Sun et al., 2019a; Blodgett et al.,
2020; Garrido-Muñoz et al., 2021). This become more crucial in these sensitive
environments in the biomedical domain that involves life-changing decisions,
like surgery (Rudzicz and Saqur, 2020). It should ensure that the decisions
cannot reflect discriminatory or biased behavior toward specific groups or
certain populations. In the domain of pre-trained language models, the ethics
and bias issues has been quantified by a few works. (Zhang et al., 2020d)
quantifies biases in clinical contextual word embeddings. The reason behind
may due to the training itself is biased with respect to various attributes
like gender, language, race, age, ethnicity, and marital status. For example,
in the MIMIC-III dataset (Johnson et al., 2016), one can find: 1) gender bias:
males have more heart disease than females, and 2) ethnicity bias: black
patients have fewer clinical studies than other groups (Kalyan et al., 2021).
Considering the complexity of directly reducing biases in training corpora,
existing works explore to identify bias by adversarial training (Zhang et al.,
2020d) or data augmentation (Minot et al., 2021).

##### Privacy

Although most corpora used in biomedical pre-training like scientific
publications and social medical are open-access. Some EHRs are private since
some organizations do not want to expose their data. Clinical records may
contain patient visits and medical history. Such sensitive data should not be
exposed as it may harm the patients physically or mentally (Nakamura et al.,
2020). Although sensitive information in EHR records (like MIMIC III) is de-
identified before sharing for research purposes. It is possible to recover
sensitive patient information from the de-identified medical records. Recent
works showed that there is data leakage from pre-trained models in the general
domain i.e., it is possible to recover personal information present in the
pre-training corpora (Lehman et al., 2021). Due to data leakage, the models
pre-trained on proprietary corpora, cannot be released publicly. Recently,
Nakamura et al (Nakamura et al., 2020) proposed KART framework which can
conduct various attacks to assess the leakage of sensitive information from
pre-trained biomedical language models. Also, the federated learning (Yang et
al., 2019; Li et al., 2020d) framework may help when different organizations
and end users could collaboratively learn a shared prediction model while
keeping all the training data on a private side.

###  6.2. Future trends

We further suggest some future trends in this subsection.

##### Standardized benchmark

In general NLP fields, evaluation criteria and standard benchmarks are a
driving force for the NLP community. For example, BERT (Devlin et al., 2018)
were widely accepted in benchmarks (Wang et al., 2018a; Rajpurkar et al.,
2016) that makes it spread to various tasks in NLP. On the other hand, lacking
effective evaluation criterion is one of the bottlenecks of text generations
(Celikyilmaz et al., 2020). In the biomedical domain, various pre-trained
models and their fine-tuning applications have been proposed (as introduced in
Sec. 4 and Sec. 5). However, they are generally not well-compared. Although a
few efforts have been done to standardize benchmarks for biomedical pre-
trained models, which includes but not limits to (Zhang et al., 2021; Gu et
al., 2021). This becomes much more difficult in the cross-discipline domain
like biomedical domain since papers are usually from different communities
like informatics, medicine,and computer science. An open standardized and
well-categorized benchmark (like in (Lewis et al., 2020)) should be proposed
to make use of the advantages of each work and collaboratively push the