# Investigating the Efficacy of Large Language Models for Code Clone Detection

Mohamad Khajezade  University of British ColumbiaAddress LineKelownaB.C.Zip
CodeCanada<EMAIL_ADDRESS>,  Jie JW Wu  0000-0002-7895-2023 University
of British ColumbiaAddress LineKelownaB.C.Zip CodeCanada<EMAIL_ADDRESS>,
Fatemeh Hendijani Fard  University of British ColumbiaAddress
LineKelownaB.C.Zip CodeCanada<EMAIL_ADDRESS>,  Gema Rodríguez-Pérez
University of British ColumbiaAddress LineKelownaB.C.Zip CodeCanada
<EMAIL_ADDRESS>and  Mohamed Sami Shehata  University of British
ColumbiaAddress LineKelownaB.C.Zip CodeCanada<EMAIL_ADDRESS>
###### Abstract.

Large Language Models (LLMs) have demonstrated remarkable success in various
natural language processing and software engineering tasks, such as code
generation. The LLMs are mainly utilized in the prompt-based zero/few-shot
paradigm to guide the model in accomplishing the task. GPT-based models are
one of the popular ones studied for tasks such as code comment generation or
test generation. These tasks are ‘generative’ tasks. However, there is limited
research on the usage of LLMs for ‘non-generative’ tasks such as
classification using the prompt-based paradigm. In this preliminary
exploratory study, we investigated the applicability of LLMs for Code Clone
Detection (CCD), a non-generative task. By building a mono-lingual and cross-
lingual CCD dataset derived from CodeNet, we first investigated two different
prompts using ChatGPT to detect Type-4 code clones in Java-Java and Java-Ruby
pairs in a zero-shot setting. We then conducted an analysis to understand the
strengths and weaknesses of ChatGPT in CCD. ChatGPT surpasses the baselines in
cross-language CCD attaining an F1-score of 0.877  and achieves comparable
performance to fully fine-tuned models for mono-lingual CCD, with an F1-score
of 0.878. Also, the prompt and the difficulty level of the problems has an
impact on the performance of ChatGPT. Finally, we provide insights and future
directions based on our initial analysis111Our code and data is open-sourced
at https://github.com/mkhfring/largeLanguageModels.

Large Language Models, Code Clone Detection, Zero-shot Learning, Few-shot
Learning.

††copyright: none

##  1\. Introduction

Code clones, the functionally identical code fragments, can cause buggy code
to propagate through the whole project (ain2019systematic, ; lei2022deep, ).
Code Clone Detection (CCD) is necessary to develop and maintain source code,
and is beneficial for various applications, including finding library
candidates, code comprehension, finding harmful software, and plagiarism
detection. Code clone is categorized into four types depending on the
similarity measures: 1) Type-1 clones: Code fragments that are identical in
terms of their code and structure, except for variations in whitespace,
comments, and layout. 2) Type-2 clones: Code fragments that are syntactically
identical except for variations in identifiers, literals, types, whitespace,
layout, and comments. 3) Type-3 clones: Code fragments that are mostly
similar, but can have differences in terms of statements, in addition to the
variations in identifiers, literals, types, whitespace, layout, and comments.
4) Type-4 clones: Code snippets that are completely different syntactically
and structurally but share the same functionality. Among them, Type-4 clones
are the most challenging to detect given their different syntactic variants in
implementation and thus require a high degree of reasoning ability
(ain2019systematic, ).

Different types of approaches (ain2019systematic, ) have been proposed for
CCD, including text-based approaches, lexical approaches, syntactic
approaches, and semantic approaches. In recent years, deep learning approaches
have been a major approach for different software engineering tasks, such as
clone detection. The deep learning-based code clone detections leverage deep
models to learn the inherent similarities of clones (lei2022deep, ). These
code clone detectors can effectively detect different clone types, such as
Type-3 and Type-4. However, these methods have been proven to perform poorly
with unseen problems and out-of-distribution data (9678907, ;
sonnekalb2022generalizability, ). Additionally, these models are not effective
for cross-language code clone detection, where code snippets are implemented
in different languages (tao2022c4, ).

Recently, Large Language Models (LLM) have been a prominent approach due to
their emergent abilities (brown2020language, ; min2021recent, ;
weng2022identification, ; xu2022systematic, ; brown2020language, ). LLMs are
models trained on a large number of parameters (e.g., GPT-3 has 175 billion
parameters (brown2020language, )) in an unsupervised manner and have had a
profound impact on multiple fields, including natural language processing
(min2021recent, ), machine learning (weng2022identification, ), and software
development (xu2022systematic, ). These models can analyze large volumes of
textual data, allowing for accurate predictions and the generation of
different contents in a zero-shot manner, i.e., without the need to employ any
labeled data (brown2020language, ). However, it remains unclear what values
can LLMs provide to the problem of CCD, especially Type-4. To address this
problem, our goal in this paper is to investigate and understand the
performance and limitations of using LLM for CCD. We chose Type-4 clone
because detecting this type of code clone is challenging. Previous models have
found it difficult to address Type-4 code clones, especially in the contexts
of few-shot learning and cross-language clone detection (9678907, ).  We
believe the reasoning capabilities of large language models provide a
promising approach for Type-4 clones.

In this paper, we first investigated two different prompts using ChatGPT to
detect code clones in Java-Java and Java-Ruby pairs in zero-shot setting by
building a mono-lingual and cross-lingual CCD dataset derived from CodeNet.
Furthermore, we conducted an analysis to understand the strengths and
weaknesses of ChatGPT in CCD. Our main finding is that ChatGPT surpasses the
baselines in cross-language CCD and achieves comparable performance to fully
fine-tuned models for mono-lingual CCD.  The F1 performance of ChatGPT for
cross-language CCD is 0.877, in contrast to baselines such as CodeBERT, which
achieves an F1-score of 0.663.  Additionally, the prompt and the difficulty
level of the problems have an impact on the performance of ChatGPT.

Difference between this work and existing studies: Though there exist multiple
studies on using LLMs for various code-related and software engineering tasks
(trummer2022codexdb, ; nashid2023retrieval, ; bareiss2022code, ), the number
of works on using LLMs for code clones is limited. The only related works are
(roy2023unveiling, ; dou2023towards, ). The former explores the application of
GPT-3 in generating semantic clones across languages, which is not detecting
clones. The latter assesses the performance of different LLMs in CCD for mono-
lingual setting. Our work is different from these works, as we focus on mono-
and cross-language CCD. Additionally, we are the first to investigate the
complexity and difficulty level of the problems for CCD using LLMs.

##  2\. Investigations

We investigated the efficacy of LLM for CCD by two RQs:

RQ1: What is the effect of different prompts to encourage ChatGPT to identify
Code Clones? Here, we investigate the effect of our designed prompt on the
performance of ChatGPT for Type-4 CCD, both uni-language and cross-language.
The goal is to investigate to what extent the new prompts can improve
performance.

RQ2: What is the performance of ChatGPT for code clone detection compared to
the baselines? In this RQ, we are interested in comparing the results of
ChatGPT with the baselines to detect code clones. Note that for this
experiment, ChatGPT is tested in a zero-shot setting, while the baselines are
fully fine-tuned.

###  2.1. Investigations Methodology

To answer RQ1 and RQ2, we assess the performance of ChatGPT (GPT-3.5-turbo) in
zero-shot binary code clone detection with different prompts based on
F1-Score, Cyclomatic Complexity and Acceptance Rate. The prompt $P_{c}$, which
directs ChatGPT to determine whether two code snippets are code clones, is
defined as follows:

(1) $P_{c}=\\{C_{i,j},C_{k,l},NL\\}$

Here, $i$ and $k$ represent the problem number in the dataset (CodeNet), while
$j$ and $l$ denote the submission number within problems $i$ and $k$. If
$k=i$, the two code snippets in the prompt are code clones; otherwise, they
are non-clone pairs.

Figure 1. Code1 and code2 are two accepted submissions for problem #5 of
CodeNET dataset (puri2021project, ). The figure displays the result of
querying ChatGPT using a basic prompt asking whether code1 and code2 are code
clones. Though ChatGPT correctly identifies the functionality of both scripts,
it fails to identify whether these two are clones. Figure 2. Code1 and code2
are two accepted submissions for problem #5 of CodeNET dataset
(puri2021project, ) as shown in Figure 1. The figure displays the result of
querying ChatGPT using our designed prompt when it correctly mentions that the
two codes are solving the same problem.

Prompt Design: In our investigations of prompt, we found that directly asking
ChatGPT whether two codes are clones of each other is ineffective. For
example, Figure 1 displays two code implementations (code1 and code2) of the
Greatest Common Divisor (GCD) and ChatGPT’s ability to detect whether the two
codes are clones. Despite identifying the intention behind both codes, ChatGPT
fails to accurately recognize whether they are clones or not, indicating a
misconception in its understanding of code clones. Hence, asking the model
whether two given codes are clones does not result in a correct answer. In
contrast, Figure 2 presents ChatGPT’s results for the same code snippets when
using the prompts developed in this paper. As depicted, ChatGPT accurately
classifies these code segments as code clones.

Thus, our prompt is designed to ask ChatGPT to evaluate whether $C_{i,j}$ and
$C_{k,l}$ intend to solve the same problem. In other words, ChatGPT should
determine if $i=k$. Subsequently, if $i=k$, ChatGPT is tasked with
ascertaining whether $C_{i,j}$ and $C_{k,l}$ possess the same inputs and
outputs. In our experiments, we limited ChatGPT to output either ”Yes” for
clone pairs or ”No” for non-clone pairs. Then, the results mapped to 1 for
code clones and 0 otherwise.  The final prompt will be as follows:

”””
{code1}, {code2},
Do code 1 and code 2 solve identical problems with the same inputs and
outputs? answer with yes or no and no explanation.
”””

Note that code1 and code2 are the requested codes in the above prompt.

###  2.2. Evaluation Metrics

We adopted the following three metrics in the evaluation:

F1-Score The F1-Score, also referred to as the F-Score or F-Measure, is
adopted to evaluate the performance of models for binary CCD, since it’s a
widely used performance metric for assessing the accuracy of a binary
classifier (derczynski2016complementarity, ).
Cyclomatic Complexity (CC) We employed the average Cyclomatic Complexity
(ebert2016cyclomatic, ) to determine the difficulty of each problem.
Cyclomatic Complexity (ebert2016cyclomatic, ) is a software metric widely used
to measure the complexity of a given piece of code, based on the control flow
graph of the code. A higher Cyclomatic Complexity indicates more complex code,
often making it more challenging to maintain, understand, and test
(moreno2016comparing, ).
Acceptance Rate We computed this metric for each selected problem in the
CodeNet dataset to analyze the difficulty level of each problem, based on its
metadata. The metric is calculated using the equation
$\text{acceptance\\_rate}_{p_{i}}=\frac{C{p_{i}}}{A{p_{i}}}$, where $p_{i}$
represents a selected problem from the CodeNet dataset, $A_{i}$ denotes the
total number of submissions for $p_{i}$, and $C_{i}$ signifies the total
number of correct submissions for $p_{i}$. The reason we use this metric is
because it has been used in previous studies to assess the difficulty of
programming problems (intisar2018cluster, ) and to enable advanced deep-
learning systems to categorize problems based on their level of difficulty
(intisar2018cluster, ).

###  2.3. Clone Dataset (Type-4)

For our code clone detection study, we used the Project CodeNet dataset
(puri2021project, ), containing 14 million code samples from 4,000 coding
problems in over 50 languages, primarily C++, C, Python, and Java. These
samples include accepted and rejected solutions, along with metadata like code
size and acceptance status. Tools on CodeNet’s GitHub repository
222https://github.com/IBM/Project_CodeNet enable the transformation of these
samples into token sequences or parse trees. We sampled two datasets: one for
Java-Java (CCDJJ) and anothder for Java-Ruby (XCCDJR) pairs, to explore uni-
language and cross-language code clone detection. Java was chosen for its
widespread use and Ruby for its syntactical difference and lower resource
availability. Each dataset comprises 1,000 samples (500 positive, 500 negative
pairs) randomly selected from 100 problems. Positive pairs were from the same
problem, while negative pairs were from different problems, ensuring a
balanced and representative selection. We used CodeNet’s metadata to filter
accepted solutions. Our selection methodology was designed to provide a valid
comparison of results. We also chose CodeNet over other datasets like
CodeSearchNet, hypothesizing that ChatGPT might be less familiar with CodeNet,
which requires preprocessing, potentially leading to more insightful results
about the model’s capabilities. It is worth noting that the research did not
use the standard datasets for code clone detection for two reasons: First, for
cross-language code clone detection, we need datasets comprising the same
problems implemented in different languages. Second, we require metadata to
compute the difficulty of each problem.

###  2.4. Baselines

For code clone detection, we compare ChatGPT, which uses a zero-shot approach,
with baselines fully fine-tuned on the CodeNET dataset. These models, widely
used in literature (yu2023graph, ; tao2022c4, ), include:
CodeBERT: (feng-etal-2020-codebert, ) A bimodal pre-trained model for program
comprehension and natural language tasks, using transformer architecture and a
hybrid objective function. It excels in tasks like code search and code
comment generation (feng-etal-2020-codebert, ), making it a fitting baseline.
RoBERTa (liu2019roberta, ) This enhanced BERT variant focuses on Masked
Language Model training, with a larger learning rate, dynamic masking, and
training on a broad text corpus. It’s chosen for its proven performance in
code clone detection (wang2021syncobert, ; lu2021codexglue, ) and its presence
on the CodeXGLUE leaderboard333https://microsoft.github.io/CodeXGLUE/.
GraphCodeBert (guo2021graphcodebert, ) A transformer model that emphasizes
data flow over syntactic structures, such as AST, in pre-training. It employs
structure-aware approaches for enhanced code representation and is effective
in tasks like code clone detection, translation, refinement, and search.
It should be noted that we excluded C4 (tao2022c4, ) from our baseline for
cross-language detection, due to data format constraints.

##  3\. Investigation Results

Table 1 presents the results of varying temperatures in the API call for Java-
Java pairs. These results are based on the proposed prompt design as described
earlier. As shown, the F1-Scores for $T=0.1$, $T=0.3$, and $T=0.5$ are
$0.852$, $0.855$, and $0.878$, respectively. The optimal performance is
achieved at $T=0.3$. Consequently, this temperature is used for Java-Ruby
pairs. As displayed in table 1, the F1-score for Java-Ruby is 0.877.

Table 1. The results of using ChatGPT for Java-Java pairs and Java-Ruby pairs
from CodeNet with different temperatures

Experiment | Precision | Recall | F1-score
---|---|---|---
Java Codes with T=0.3 | 0.784 | 0.997 | 0.878
Java and Ruby Codes with T=0.3 | 0.796 | 0.977 | 0.877
Java Codes with T=0.1 | 0.887 | 0.855 | 0.852
Java Codes with T=0.5 | 0.889 | 0.858 | 0.855

Table 2. The results for different prompts. Prompt-1 simply asked if two code
snippets are code clones. Prompt-2 is our proposed prompt

Experiment | Precision | Recall | F1-score
---|---|---|---
Java-Java Prompt-1 | 1.0 | 0.348 | 0.517
Java-Ruby Prompt-1 | 0.948 | 0.745 | 0.834
Java-Java Prompt-2 | 0.784 | 0.997 | 0.878
Java-Ruby Prompt-2 | 0.796 | 0.977 | 0.877

Table 3. The performance of the models for Java-Java pairs

Baseline | Precision | Recall | F1-score
---|---|---|---
CodeBERT | 0.912 | 0.881 | 0.896
RoBERTa | 0.899 | 0.852 | 0.874
GraphCodeBERT | 0.947 | 0.883 | 0.914
ChatGPT | 0.784 | 0.997 | 0.878

Table 4. The performance of the models for Java-Ruby pairs

Baseline | Precision | Recall | F1-score
---|---|---|---
CodeBERT | 0.498 | 0.988 | 0.663
RoBERTa | 0.486 | 0.992 | 0.652
GraphCodeBERT | 0.50 | 0.991 | 0.665
ChatGPT | 0.796 | 0.977 | 0.877

Table 5. The Acceptance Rate and Cyclomatic Complexity metrics are averaged
for misclassified samples shared between Java-Java and Java-Ruby datasets. The
positive samples represent the average of clone pairs, while the negative
samples indicate the average for non-clone pairs

Experiment | Acceptance Rate | Cyclomatic Complexity
---|---|---
Positive Samples | 0.038 | 2.54
Negative Samples | 0.035 | 3.16
Selected Problems | 0.058 | 2.98

Figure 3. a) Comparison of the average Acceptance Rates for positive (blue)
and negative (orange) misclassified samples, and the selected CodeNet problems
(green). The lower Acceptance Rate indicates more sophisticated problems. b)
Comparison of the average Cyclomatic Complexity for positive (blue)
misclassified samples and negative (orange) mislabeled samples, and the
selected programs from CodeNET (green). A higher CC represents more difficult
problems.

RQ1: Effect of different prompts.  Table 2 compares two prompts, Prompt-1,
which simply asks ChatGPT if two codes are code clones, and Prompt-2, which is
the prompt proposed in this research. As illustrated, the lowest performance
is observed with Prompt-1 for Java-Java pairs, yielding an F1-score of
$0.517$, a precision of $1.0$, and a recall of $0.348$. Interestingly,
Prompt-1 achieves an F1-score of $0.834$ for Java-Ruby pairs, with a precision
of $0.948$ and a recall of $0.745$. The prompt designed in this research,
Prompt-2, outperforms Prompt-1 for both Java-Java pairs, with an F1-score of
$0.878$, and Java-Ruby pairs, with an F1-score of $0.877$.
RQ2: Comparing the performances. As Prompt-2 achieved the best scores in our
first experiment, we compare the results of Prompt-2 with the baselines in
this RQ2. For Java-Ruby pairs, as displayed in Table 4, ChatGPT delivers the
best performance with an F1-score of $0.877$. Consequently, the performance of
CodeBERT, RoBERTa, and GraphCodeBERT are $0.663$, $0.652$, and $0.665$,
respectively. In summary, ChatGPT surpasses other baselines in performance for
the Java-Ruby cross-language dataset, achieving an F1-score of 0.877.
Conversely, the other baselines yield lower F1-scores in the range of 0.66 for
Java-Ruby pairs, with precision and recall metrics indicating that these
baselines predominantly generate $0$. However, baselines exhibit improved F1
scores for Java-Java pairs compared to ChatGPT, with scores ranging from 0.896
to 0.914, as opposed to 0.878 for ChatGPT. Notably, ChatGPT accomplishes this
performance in a zero-shot manner, while the baselines are fully fine-tuned
for the downstream task.

##  4\. Discussions

The results presented in Table 1 suggest that the optimal performance is
achieved with a temperature of 0.3. However, as shown, altering the
temperature does not significantly impact performance. Another noteworthy
observation from Table 1 is that the performance of Java-Java pairs is nearly
identical to that of Java-Ruby pairs, even though cross-language clone
detection poses a challenge for other baselines. One plausible explanation for
this outcome is that ChatGPT struggles with specific problems for both Java-
Java and Java-Ruby pairs. Considering Tables 3 and 4, we observe a performance
drop of the baselines for the cross-language setting. For instance, the
F1-score of GraphCodeBERT decreases from 0.914 to 0.665. This result aligns
with a recent work demonstrating GraphCodeBERT’s vulnerability in cross-
language CCD (mehrotra2023improving, ). Though cross-language CCD is more
challenging, ChatGPT performs better than baselines. We hypothesize that this
result could stem from the pre-training and datasets and tasks used to train
the LLMs. Note that LLMs have great performance for code translation. In
Prompt-2, we ask the model to identify the problem first. So it might use some
capabilities for code translation in Java-Ruby pair CCD. However, more
investigations are required to determine the reason.

To further examine ChatGPT’s vulnerability to problem complexity, we
calculated two metrics assessing the difficulty and complexity of each
problem, Acceptance Rate, and Cyclomatic Complexity. The former indicates the
number of accepted submissions compared to all submissions of a specific
problem and the latter is a metric for computing the complexity of a program.
Table 5 presents the average of these metrics for the misclassified problems
shared between Java-Java and Java-Ruby pairs. In Table 5, positive samples
represent the average difficulty metrics for clone pairs related to problems
that ChatGPT erroneously labeled as non-clone, while negative samples show the
average difficulty for misclassified non-clone pairs. The table’s final row
calculates the average for all selected CodeNET problems used to sample the
datasets in this study i.e., the test set minus the misclassified problems.

Table 5 indicates that the complexity of positive problems is $2.54$, which is
less than the average complexity of selected problems in CodeNET. On the other
hand, the average complexity for negative samples is $3.16$, which is
significantly more than the average complexity. Figure 3 provides a clearer
illustration of these metrics by comparing the average Acceptance Rate and
average Cyclomatic Complexity for positive mislabeled samples and negative
misclassified samples, alongside the average values for these metrics in the
test set. This result indicates that based on metrics of Acceptance Rate and
Cyclomatic Complexity, the complexity of problems influences ChatGPT’s
performance on CCD. Our hypothesis is that the more complex the code pairs
are, the more challenging this reasoning becomes for ChatGPT on code clone
detection.

Threats. We utilized a public high quality dataset and used an schema that is
consistent with the benchmarks for dataset creation, and used standard
evaluation metrics, to mitigate the internal and construct threats. While the
approach and methodology may not differ significantly for other programming
languages and models, the results might not be generalizable to other
languages.

##  5\. Conclusions and Future Work

We presented an initial investigation of applying LLMs for Type-4 code clone
detection. The preliminary results are promising as we found that ChatGPT is
effective, and its performance is correlated with the problem difficulty and
prompt design.  In the future, we intend to use other LLMs pre-trained on
code, like Phi-2 and Mistral, and investigate the reasons of our findings, as
well as whether the performances are related to specific programming
languages.

###### Acknowledgements.

This research is supported by a grant from the Natural Sciences and
Engineering Research Council of Canada RGPIN-2019-05175.

## References

  * (1) Qurat Ul Ain, Wasi Haider Butt, Muhammad Waseem Anwar, Farooque Azam, and Bilal Maqbool,  “A systematic review on code clone detection,”  IEEE access, vol. 7, pp. 86121–86144, 2019.
  * (2) Maggie Lei, Hao Li, Ji Li, Namrata Aundhkar, and Dae-Kyoo Kim,  “Deep learning application on code clone detection: A review of current knowledge,”  Journal of Systems and Software, vol. 184, pp. 111141, 2022.
  * (3) Chenyao Liu, Zeqi Lin, Jian-Guang Lou, Lijie Wen, and Dongmei Zhang,  “Can neural clone detection generalize to unseen functionalitiesf,”  in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021, pp. 617–629.
  * (4) Tim Sonnekalb, Bernd Gruner, Clemens-Alexander Brust, and Patrick Mäder,  “Generalizability of code clone detection on codebert,”  in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–3.
  * (5) Chenning Tao, Qi Zhan, Xing Hu, and Xin Xia,  “C4: Contrastive cross-language code clone detection,”  in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, 2022, pp. 413–424.
  * (6) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.,  “Language models are few-shot learners,”  Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  * (7) Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth,  “Recent advances in natural language processing via large pre-trained language models: A survey,”  arXiv preprint arXiv:2111.01243, 2021.
  * (8) Min-Hsien Weng, Shaoqun Wu, and Mark Dyer,  “Identification and visualization of key topics in scientific publications with transformer-based language models and document clustering methods,”  Applied Sciences, vol. 12, no. 21, pp. 11220, 2022.
  * (9) Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn,  “A systematic evaluation of large language models of code,”  in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 1–10.
  * (10) Immanuel Trummer,  “Codexdb: Synthesizing code for query processing from natural language instructions using gpt-3 codex,”  Proceedings of the VLDB Endowment, vol. 15, no. 11, pp. 2921–2928, 2022.
  * (11) Noor Nashid, Mifta Sintaha, and Ali Mesbah,  “Retrieval-based prompt selection for code-related few-shot learning,”  in Proceedings of the 45th International Conference on Software Engineering (ICSE’23), 2023.
  * (12) Patrick Bareiß, Beatriz Souza, Marcelo d’Amorim, and Michael Pradel,  “Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code,”  arXiv preprint arXiv:2206.01335, 2022.
  * (13) Palash R Roy, Ajmain I Alam, Farouq Al-omari, Banani Roy, Chanchal K Roy, and Kevin A Schneider,  “Unveiling the potential of large language models in generating semantic and cross-language clones,”  arXiv preprint arXiv:2309.06424, 2023.
  * (14) Shihan Dou, Junjie Shan, Haoxiang Jia, Wenhao Deng, Zhiheng Xi, Wei He, Yueming Wu, Tao Gui, Yang Liu, and Xuanjing Huang,  “Towards understanding the capability of large language models on code clone detection: a survey,”  arXiv preprint arXiv:2308.01191, 2023.
  * (15) Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al.,  “Project codenet: A large-scale ai for code dataset for learning a diversity of coding tasks,”  arXiv preprint arXiv:2105.12655, vol. 1035, 2021.
  * (16) Leon Derczynski,  “Complementarity, f-score, and nlp evaluation,”  in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), 2016, pp. 261–266.
  * (17) Christof Ebert, James Cain, Giuliano Antoniol, Steve Counsell, and Phillip Laplante,  “Cyclomatic complexity,”  IEEE software, vol. 33, no. 6, pp. 27–29, 2016.
  * (18) Jesús Moreno-León, Gregorio Robles, and Marcos Román-González,  “Comparing computational thinking development assessment scores with software complexity metrics,”  in 2016 IEEE global engineering education conference (EDUCON). IEEE, 2016, pp. 1040–1045.
  * (19) Chowdhury Md Intisar and Yutaka Watanobe,  “Cluster analysis to estimate the difficulty of programming problems,”  in Proceedings of the 3rd International Conference on Applications in Information Technology, 2018, pp. 23–28.
  * (20) Dongjin Yu, Quanxin Yang, Xin Chen, Jie Chen, and Yihang Xu,  “Graph-based code semantics learning for efficient semantic code clone detection,”  Information and Software Technology, vol. 156, pp. 107130, 2023.
  * (21) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou,  “CodeBERT: A pre-trained model for programming and natural languages,”  in Findings of the Association for Computational Linguistics: EMNLP 2020, Online, Nov. 2020, pp. 1536–1547, Association for Computational Linguistics.
  * (22) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov,  “Roberta: A robustly optimized bert pretraining approach,”  arXiv preprint arXiv:1907.11692, 2019.
  * (23) Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, and Xin Jiang,  “Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation,”  arXiv preprint arXiv:2108.04556, 2021.
  * (24) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al.,  “Codexglue: A machine learning benchmark dataset for code understanding and generation,”  arXiv preprint arXiv:2102.04664, 2021.
  * (25) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou,  “Graphcode{bert}: Pre-training code representations with data flow,”  in International Conference on Learning Representations, 2021.
  * (26) Nikita Mehrotra, Akash Sharma, Anmol Jindal, and Rahul Purandare,  “Improving cross-language code clone detection via code representation learning and graph neural networks,”  IEEE Transactions on Software Engineering, 2023.


11institutetext: School of Computer Science, University of Manchester, UK
22institutetext: Health eResearch Centre, Farr Institute, Manchester Academic
Health Science Centre, The University of Manchester, UK
33institutetext: Arthritis Research UK Centre for Epidemiology, The University
of Manchester, UK
44institutetext: Manchester Institute of Innovation Research, Alliance
Manchester Business School, The University of Manchester, UK
44email:
<EMAIL_ADDRESS>
# Extracting adverse drug reactions and their context using sequence labelling
ensembles in TAC2017

Maksim Belousov 11 Nikola Milosevic 1144 William Dixon 2233 Goran Nenadic 1122

###### Abstract

Adverse drug reactions (ADRs) are unwanted or harmful effects experienced
after the administration of a certain drug or a combination of drugs,
presenting a challenge for drug development and drug administration. In this
paper, we present a set of taggers for extracting adverse drug reactions and
related entities, including factors, severity, negations, drug class and
animal. The systems used a mix of rule-based, machine learning (CRF) and deep
learning (BLSTM with word2vec embeddings) methodologies in order to annotate
the data. The systems were submitted to adverse drug reaction shared task,
organised during Text Analytics Conference in 2017 by National Institute for
Standards and Technology, achieving F1-scores of 76.00 and 75.61 respectively.

Keywords: health informatics, text mining, drug labels, adverse drug reactions

##  1 Introduction

Adverse drug reactions (ADRs) are unwanted or harmful effects experienced
after the administration of a certain drug or a combination of drugs [8]. They
present a challenge for drug development and drug administration. During 1994,
it was estimated that 700,000 patients in the United States suffered from
adverse drug reaction, while 100,000 died as a consequence of such reactions
[7]. Roughly half of the people in the UK take prescribed medications. Adverse
drug reactions are serious burden on health care systems. About 7% of all
hospital admissions were accounted to ADRs. Moreover, quality of life and
adherence to treatment is, as well, affected by adverse drug reactions
[pirmohamed2004adverse]. Also, they are important source of human phenotypic
data and can be used to predict drug targets [6].

In the United States, drug product labels are required by law to contain the
information regarding clinically significant adverse drug reactions [15]. All
drug product labels in the United States are freely available through the
National Library of Medicine’s DailyMed
website111https://dailymed.nlm.nih.gov/dailymed/index.cfm in a standard format
called Structured Product Label (SPL).

The task of recognising specific mentions (such as ADRs) in a text is a task
of named entity recognition (NER) or tagging, which can be approached using
sequence labelling techniques. Sequence labelling problems are usually solved
using sequence modelling machine learning techniques, such as hidden Markov
models, conditional random fields or recurrent neural networks.

Within the drug informatics domain, the SPLICER system [3] was successfully
applied to extract adverse drug events from text and tables in the Adverse
Reactions section of SPLs. Other efforts focus on side effects and drug
indications [4, 5, 1]. The SIDER (Side Effect Resource) database uses named
entity recognition to extract side effects and indications from product
labelling, including SPLs [6]. More recently, starting with full-text papers
from the Journal of Oncology, drug side effect relationships were extracted
and compared to the SIDER database [18].

Neural networks with word embeddings have recently showed successes in the
biomedical named entity recognition. Word2vec embeddings with bidirectional
recurrent neural networks combined with a CRF tagger and SVM classifier showed
promising results for disease recognition [16]. Named entity recognition
methodology based on recurrent neural networks and word embeddings (GloVe or
Word2vec) was used for de-identification of electronic health records and gave
the state-of-the-art results, producing slightly better results with GloVe
embeddings [2].

In this paper, we present our approaches to the recognition of adverse drug
reactions and related entities, developed for a shared task organised during
the Text Analytics Conference 2017 (TAC 2017). The task was co-organised by
the US National Institute of Standards and Technology (NIST) and the US Food
and Drug Administration (FDA). The objective of the shared task was to extract
adverse drug reactions from drug labelling text documents using natural
language processing techniques. In the task 1, in which we participated, the
participants were supposed to build a system to extract adverse drug reactions
and related mentions such as severity, drug class, negation, factors, and
whether it was reported on
animals222https://bionlp.nlm.nih.gov/tac2017adversereactions/.

###  1.1 Data

The shared task organisers published a training dataset containing 101
annotated drug labels (documents) and a dataset containing 2,208 unannotated
drug labels. An unseen subset of unannotated documents was used as a testing
data during the task evaluation. The drug label is a multi-section document
that may contain headings, paragraphs, tables and lists. In the provided
dataset each drug label was converted to a text document disregarding the
structure (i.e. representing all elements as an unformatted text, keeping only
the main sections of the document). It is worth noting that the gold-standard
dataset contained some _discontinuous annotations_ (6.8% of all annotations).
Annotation that involves more than one continuous span of characters is
considered discontinuous annotation. For the simplicity of tagging schemes, we
ignored discontinuous annotations during the document parsing.

The class distribution of annotated entities is imbalanced, where the majority
of annotations were adverse drug reactions. On the other hand, some related
entities had only a few annotations. The numbers of annotated mentions (groups
of tokens), tokens and the average number of tokens per mention are presented
in Table 1. Lack of data for certain related entities presented a challenge
for developing named entity recognition systems based on machine learning.

Entity class | #mentions | #tokens | Avg. tk/mention
---|---|---|---
Adverse drug reaction | 12,792 | 21,258 | 1.66
Severity | 863 | 1,306 | 1.51
Factor | 602 | 653 | 1.08
Drug class | 248 | 518 | 2.09
Negation | 95 | 109 | 1.47
Animal | 44 | 44 | 1.00
Table 1: The number of annotated mentions (group of tokens), number of tokens,
and the average number of tokens per mention in the provided training data

##  2 System description

The architecture of the proposed systems consists of three stages: (1)
document parsing, (2) word vectorisation, (3) tagging ADRs and their related
entities. During the document parsing stage we attempt to restore the original
structure of the document and recognise elements such as headings, tables
(with rows and cells), lists (with items) and text paragraphs. The word
vectorisation stage depended on the type of tagging model in the following
stage and aimed to generate word vectors from text sequences using either
hand-crafted features or unsupervised learning. The main task of tagging stage
is to extract mentions of specific type from text by sequence labelling of
extracted word vectors. Since some related entities rely on ADR mentions, they
are performed separately, after the ADR tagging is completed. The pipeline is
presented in Figure 1 and the following subsections provide details about each
processing stage.

Figure 1: Document processing and tagging pipeline.

###  2.1 Document parsing

The aim of this stage is to perform re-engineering the structure of the
document so that later their content can be treated differently. For instance,
it might be beneficial to analyse the content of a table cell individually
rather than the whole chunk of the text that contain multiple rows and cells.
We identified four different element types in the document:

  * •

Headings are numbered titles for sections and sub-sections (e.g. “5.1 Asthma-
Related Death [See Boxed Warning]”)

  * •

Tables have _heading rows_ and _content rows_ , each of them is also having
_cells_. Each row might have different number of cells. In addition, a table
may have _caption_ (which usually starts with “Table NUM.”) and a _footer_
that contain additional notes. We treated all text lines after the
aforementioned caption trigger and before the paragraph separator (multiple
empty lines) as potential table rows. Then, we categorised each row candidate
as part of the caption, header, content and footer, based on the number of
potential columns, numerical cells and words in each cell.

  * •

Lists are groups of multiple bullet-points or _items_. Consequent text lines
that starts with asterisk character (*) are considered as list items. List
should have more than one item.

  * •

Paragraphs are any other chunk of text separated with multiple new-line
characters.

For some tagging models we applied two different document _splitting
strategies_ : (1) take the whole element (i.e. table, list, paragraph) and
represent them as text or (2) take the textual content of sub-elements (such
as table cells and list items) and treat them as individual items.

###  2.2 Tagging models

We utilised various types of tagging methods based on knowledge-driven rules,
conditional random fields (CRF), bidirectional long short-term memory networks
(BLSTM) and two different types of ensemble methods. We generated word vectors
differently depending on the sequence labelling approach by using either hand-
crafted features or obtaining word embeddings from unsupervised learning
models trained on large text corpora.

####  2.2.1 Rule-based models

##### Rule-based

methods are based on a knowledge-driven approach and manually curated
dictionaries. In particular, we applied them for _negation_ and _animal_
classes, since there was not enough labelled data to be modelled by machine
learning algorithms.

  * •

To identify negations, we have developed a rule based tagger using the
modification of DepND333https://github.com/zachguo/DepND that uses GENIA
dependency parser [11] to recognise the scope of the negation and the
dictionary of negation triggers. In particular, we added a list of phrases
that need to be ignored if appeared in a negation phrase or scope (such as
“not available” or “could not be assessed”) and labelled negations only when
an ADR mention is found inside the negation scope. We applied the negation
tagger on the sub-element level (i.e. on table cells and list items).

  * •

For the animal class, we made an assumption that animals are not mentioned in
drug labels unless adverse events are reported on them during the trials.
Also, there is a close set of animal spices that are usually used in medical
experiments [9]. We have developed a dictionary-based tagger that labelled all
mentions of animals from our list. The animal tagger was used on the sub-
element level.

####  2.2.2 CRF models

##### Linear chain conditional random fields (CRF)

is a linear statistical model that encodes conditional distributions $p(y|x)$
between observations (input features) and output variables (labels). Prior to
passing a text input into the model, each sequence item (i.e. word or token)
should be converted into a _feature vector_. In particular, we experimented
with lexical features, part-of-speech tags, grammatical relations
(dependencies), vocabulary and semantic features (such as corresponding
semantic types and named entity tags from various medical systems). In order
to capture the context for a given token, the mentioned features were
extracted from a certain number of surrounding tokens (context window). All
CRF models were used on the whole elements (i.e. tables, lists) represented as
a text.

  * •

For ADR mentions, we extracted word lemmas, part-of-speech tags (retrieved
using the GENIA tagger [14]), UMLS semantic types (obtained using QuickUMLS
[12]) and lexicon match (i.e. whether the current word is exist in the ADR
lexicon444http://diego.asu.edu/downloads/publications/ADRMine/ADR_lexicon.tsv).
We trained word2vec on lemmatised sentences extracted from 2,208 unannotated
drug labels that were provided as a part of this task. In particular, we
extracted 200-dimensional feature vectors from continuous bag-of-words model
with a context window of size 5, trained with negative sampling using five
noise words. Then we performed K-means clustering (n=50) of the word-vector
space. For words that are found in the model, we used their corresponding
cluster number, otherwise we used the lemma of the word as a feature. In order
to capture the context we also extracted features from surrounding words (i.e.
five preceding and five following words).

  * •

For the severity, factor and drug class, we used a similar set of features
with additional lexicon features. In particular, a lexicon for drug class was
obtained from DrugBank555https://www.drugbank.ca/ and Anatomical Therapeutic
Chemical Classification System (ATC)666http://www.atccode.com/, whereas for
other aforementioned classes we experimented with lexicons obtained from the
provided labelled data. We also added an additional binary feature that
indicates whether the ADR is mentioned in the surrounding context.

####  2.2.3 BLSTM models

##### Bidirectional Long Short-Term Memory networks (BLSTM)

are specific type of recurrent neural networks designed to learn long-term
dependencies. In order to increase the amount of input information, the given
sequence is read in both directions (forward and backward). For this tagging
model we obtained word vectors from multiple word2vec models trained on large
text corpora from generic and target domains. The generic 200-dimensional word
embeddings were trained on a combination of PubMed and PMC texts with texts
extracted from a recent English Wikipedia dump [10], whereas the target
200-dimensional word embeddings were trained on 2,208 unannotated drug labels.
The _BLSTM_ model was trained using RMSprop [13] algorithm with the learning
rate of $1\text{\times}{10}^{-5}$. For regularisation, dropout with the rate
of $0.1$ was applied on each LSTM layer with 170 units. We trained BLSTM model
for 50 epochs and used early stopping with patience of 10 epochs. Since this
model does not rely on hand-crafted features, we used the same model
configuration for both adverse reactions and related entities. For all entity
types, we have trained a single BLSTM model on the whole elements (i.e.
tables, lists) represented as a text.

####  2.2.4 Ensemble models

We have created two different ensemble models:

  * •

A voting BLSTM and CRF ensemble was training both CRF and BLSTM classifiers in
parallel and selected the best candidate based on the highest average
predicted probability of each class obtained from each classifier.

  * •

A stacked CRF-BLSTM ensemble is our proposed modification of Wolpert’s stacked
generalisation [17] that firstly trains the CRF classifier, using the
previously described features, and then utilises its predicted probabilities
for each class to build an additional token-level embeddings for the BLSTM
classifier. In this way, the obtained word vector has the dimension of the
number of target classes used in CRF and its values will correspond to
predicted probabilities.

For the voting and stacked ensembles we have utilised an ADR-specific feature
extractor and trained a single ensemble model on all classes.

##  3 Evaluation of the tagging models on the training data

The provided labelled data contained 101 documents. We evaluated the
supervised machine learning models using holdout cross-validation; therefore
the dataset was split into training (56 documents), validation (24 documents)
and testing (21 documents) sets. The rule-based models were evaluated on the
whole dataset. The evaluation results for all developed taggers are presented
in Table 2.

Entity class | Method | Precision | Recall | F1-score
---|---|---|---|---
ADR | CRF | 90 | 82 | 86
| BLSTM | 86 | 84 | 85
| Voting BLSTM+CRF | 91 | 84 | 87
| Stacked CRF+BLSTM | 90 | 85 | 87
Severity | CRF | 67 | 51 | 58
| BLSTM | 55 | 75 | 64
| Voting BLSTM+CRF | 70 | 65 | 67
| Stacked CRF+BLSTM | 58 | 71 | 64
Factor | CRF | 52 | 20 | 29
| BLSTM | 73 | 46 | 56
| Voting BLSTM+CRF | 87 | 36 | 51
| Stacked CRF+BLSTM | 82 | 41 | 55
Drug class | CRF | 41 | 35 | 38
| BLSTM | 57 | 21 | 31
| Voting BLSTM+CRF | 62 | 12 | 20
| Stacked CRF+BLSTM | 57 | 24 | 34
Negation | CRF | 25 | 18 | 21
| BLSTM | 22 | 12 | 15
| Voting BLSTM+CRF | 50 | 06 | 11
| Stacked CRF+BLSTM | 57 | 24 | 33
| Rule-based | 66 | 66 | 66
Animal | CRF | 76 | 100 | 87
| BLSTM | 100 | 46 | 63
| Voting BLSTM+CRF | 100 | 38 | 56
| Stacked CRF+BLSTM | 40 | 31 | 35
| Rule-based | 86 | 100 | 93
Table 2: Token-level evaluation of the taggers by the entity class and method
used on the provided 101 labelled documents using holdout cross-validation.

As it can be seen from Table 2, we calculated precision, recall and F1-score
for labelling tokens in the document. Later, sequential labels are post-
processed and merged into mentions.

Both ensemble models usually outperformed individual models especially in
cases where there was enough training and testing samples. The stacked and
voting ensembles performed relatively similar, although the stacked ensemble
was slightly better in general. The F1-score for labelling adverse drug
reactions ranges between 85%-87%, with the maximum score for the ensemble and
BLSTM tagger. The BLSTM tagger performed better on the severity and factor
classes. Drug class gave the best results on the test set with the CRF tagger,
however, these results were quite unstable. While CRF performed on the test
set with F1-score of 38%, on the validation set the F1-score was only 22%. The
rule based approach gave the best results for the rare classes, such as
negation and animal.

##  4 Runs and system evaluation

Using the evaluation results presented in the previous section, we have
combined the best-performing taggers and created two systems which correspond
to the two runs submitted for the final shared task evaluation (on test data).

  * •

Run #1: We applied the rule-based approaches for the _Negation_ and _Animal_
classes. For _Adverse Drug Reactions_ we utilised the CRF model with the hand-
crafted features. For all other entity types (i.e. _Severity_ , _Factor_ and
_Drug class_) we used the BLSTM tagger. The three related entities used one
BLSTM model.

  * •

Run #2: The rule-based tagger was applied only for the _Negation_ class,
whereas all other classes were handled with the Stacked CRF+BLSTM ensemble
model.

###  4.1 Results

The systems were trained on the whole annotated dataset provided (101
documents) and applied on unannotated dataset for automatic tagging (2,208
documents). Then, sample of the automatically tagged documents were used for
the evaluation. The primary metric for this evaluation was the micro-averaged
F1-score. We have presented the system evaluation results in Table 3.

| Considering entity type | Not considering entity type
---|---|---
Submission | Precision | Recall | F1-score | Precision | Recall | F1-score
Run #1 | 80.19 | 72.23 | 76.00 | 80.19 | 72.23 | 76.00
Run #2 | 76.84 | 74.36 | 75.58 | 76.87 | 74.39 | 75.61
Table 3: Performance of the submitted systems on the test data considering and
not considering types of annotated entities. The primary metric used for the
evaluation is marked in bold.

###  4.2 Discussions

The submitted systems had similar performance, with Run 1 having slightly
better performance on the test data (by less than 0.5%). The achieved results
are similar to the results obtained on the training data using 3-fold cross-
validation (F1-score of 77.26 for the Run 1, and 76.61 for the Run 2).

The classes were not balanced. Some classes, such adverse drug reactions had a
fair number of labelled entities in the training set, and therefore the
machine learning models could be efficiently trained on this class. However,
other classes were relatively small compared to the ADR class. Also, other
classes were related to the ADR class and were only triggered if the ADR class
is labelled in its vicinity. Context of the labels had significant importance
in this task, as the same phrase is labelled depending on whether it is in
vicinity of an ADR and whether it closer describes an ADR. For example word
“serious” will be labelled as severity in context of “serious headache”,
however, it will not be labelled in other contexts, such as for example in
“serious consideration”.

On the other hand, some classes, such as animal and negation had only a small
number of annotations in the training dataset. Therefore, it was impossible to
train a machine learning model and it was necessary to develop a rule based
approach. The rules for the negation class were considering context and
whether in the scope of the negation is present an ADR. On the other hand,
mentions of animals unrelated to an ADR were rare. Therefore, it was safe to
make an assumption that all animal mentions are related to adverse drug
reactions.

###  4.3 Conclusion

In this paper we presented a number of different methodologies for labelling
adverse drug reactions and related factors, severity, drug class, negation and
whether they were reported on animals. We presented two systems made out of
the best performing taggers that were submitted to the ADR track shared task
of the Text Analytics Conference (TAC2017). The systems performed with
F1-scores of 76% and 75.58% on the testing data.

There is still space for improvement of the system and performing additional
experiments. More informative features of the text could help improve the CRF
machine learning taggers, while more representative word embeddings could be
helpful for the BLSTM based taggers. This can be achieved using additional
vocabularies, semantic resources and knowledge bases.

Other potential way to improve the performance of the tagging is to
investigate alternative ensemble methods, e.g. utilise an additional meta-
classifier to combine the CRF and BLSTM results. In addition, performance of
the BLSTM model directly depends on the word embeddings that were used,
therefore alternative word representation models in addition to word2vec might
be utilised (e.g. multi-level word representation or knowledge-infused word
embeddings).

However, there is still challenge of labelling classes that have a low number
of examples. In these cases, it is challenging to create a good performing
machine learning models, because of the lack of examples. However, our rule
based approaches can be further improved with additional samples and looking
at additional data. Also, machine learning performance can be probably
improved by using additional annotated data and external data sets.

## References

  * [1] Boyce, R., Gardner, G., Harkema, H.: Using natural language processing to extract drug-drug interaction information from package inserts. In: BioNLP: Proceedings of the 2012 Workshop on Biomedical Natural Language Processing. pp. 206–213
  * [2] Dernoncourt, F., Lee, J.Y., Uzuner, O., Szolovits, P.: De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association 24(3), 596–606 (2017)
  * [3] Duke, J., Friedlin, J., Li, X.: Consistency in the safety labeling of bioequivalent medications. Pharmacoepidemiology and drug safety 22(3), 294–301 (2013)
  * [4] Fung, K.W., Jao, C.S., Demner-Fushman, D.: Extracting drug indication information from structured product labels using natural language processing. Journal of the American Medical Informatics Association 20(3), 482–488 (2013)
  * [5] Khare, R., Li, J., Lu, Z.: Labeledin: cataloging labeled indications for human drugs. Journal of biomedical informatics 52, 448–456 (2014)
  * [6] Kuhn, M., Letunic, I., Jensen, L.J., Bork, P.: The sider database of drugs and side effects. Nucleic acids research p. gkv1075 (2015)
  * [7] Lazarou, J., Pomeranz, B.H., Corey, P.N.: Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. Jama 279(15), 1200–1205 (1998)
  * [8] Lee, A.: Adverse drug reactions. Pharmaceutical press (2006)
  * [9] Mukerjee, M.: Trends in animal research. Scientific American 276(2), 86–93 (1997)
  * [10] Pyysalo, S., Ginter, F., Moen, H., Salakoski, T., Ananiadou, S.: Distributional semantics resources for biomedical text processing (2013), http://bio.nlplab.org/
  * [11] Sagae, K., Tsujii, J.: Dependency parsing and domain adaptation with lr models and parser ensembles. In: Emnlp-conll. vol. 2007, pp. 1044–1050. Prague, Czech Republic (2007)
  * [12] Soldaini, L., Goharian, N.: Quickumls: a fast, unsupervised approach for medical concept extraction. In: MedIR workshop, sigir (2016)
  * [13] Tieleman, T., Hinton, G.: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning 4(2), 26–31 (2012)
  * [14] Tsuruoka, Y., Tateishi, Y., Kim, J.D., Ohta, T., McNaught, J., Ananiadou, S., Tsujii, J.: Developing a robust part-of-speech tagger for biomedical text. In: Panhellenic Conference on Informatics. pp. 382–392. Springer (2005)
  * [15] US Food and Drug Administration: Cfr-code of federal regulations title 21. Current good manufacturing practice for finished pharmaceuticals Part 211 (2014)
  * [16] Wei, Q., Chen, T., Xu, R., He, Y., Gui, L.: Disease named entity recognition by combining conditional random fields and bidirectional recurrent neural networks. Database 2016 (2016)
  * [17] Wolpert, D.H.: Stacked generalization. Neural networks 5(2), 241–259 (1992)
  * [18] Xu, R., Wang, Q.: Large-scale automatic extraction of side effects associated with targeted anticancer drugs from full-text oncological articles. Journal of biomedical informatics 55, 64–72 (2015)