Knowledge-augmented Graph Neural Networks with Concept-aware Attention
  for Adverse Drug Event Detection
Adverse drug events (ADEs) are an important aspect of drug safety. Various
texts such as biomedical literature, drug reviews, and user posts on social
media and medical forums contain a wealth of information about ADEs. Recent
studies have applied word embedding and deep learning -based natural language
processing to automate ADE detection from text. However, they did not explore
incorporating explicit medical knowledge about drugs and adverse reactions or
the corresponding feature learning. This paper adopts the heterogenous text
graph which describes relationships between documents, words and concepts,
augments it with medical knowledge from the Unified Medical Language System,
and proposes a concept-aware attention mechanism which learns features
differently for the different types of nodes in the graph. We further utilize
contextualized embeddings from pretrained language models and convolutional
graph neural networks for effective feature representation and relational
learning. Experiments on four public datasets show that our model achieves
performance competitive to the recent advances and the concept-aware attention
consistently outperforms other attention mechanisms.
Shaoxiong Ji, Ya Gao, Pekka Marttinen
Knowledge-augmented Graph Neural Networks with Concept-aware Attention for Adverse Drug Event Detection

###### Abstract

Adverse drug events (ADEs) are an important aspect of drug safety. Various texts such as biomedical literature, drug reviews, and user posts on social media and medical forums contain a wealth of information about ADEs. Recent studies have applied word embedding and deep learning -based natural language processing to automate ADE detection from text. However, they did not explore incorporating explicit medical knowledge about drugs and adverse reactions or the corresponding feature learning. This paper adopts the heterogenous text graph which describes relationships between documents, words and concepts, augments it with medical knowledge from the Unified Medical Language System, and proposes a concept-aware attention mechanism which learns features differently for the different types of nodes in the graph. We further utilize contextualized embeddings from pretrained language models and convolutional graph neural networks for effective feature representation and relational learning. Experiments on four public datasets show that our model achieves performance competitive to the recent advances and the concept-aware attention consistently outperforms other attention mechanisms.

_Keywords--_ Adverse Drug Event Detection, Graph Neural Networks, Knowledge Augmentation, Attention Mechanism

## 1 Introduction

Pharmacovigilance, i.e., drug safety monitoring, is a critical step in drug development (Wise et al., 2009). It detects adverse events and safety issues and promotes drug safety through post-market assessment; therefore, it promotes safe drug development and shows significant promise in better healthcare service delivery. A drug-related negative health outcome is referred to as an Adverse Drug Event (ADE) (Donaldson et al., 2000). Given the significant harm caused by ADEs, it is essential to detect them for pharmacovigilance purposes.

Clinical trials are the common way to detect ADEs. However, some ADEs are hard to investigate through clinical trials due to their long latency (Sultana et al., 2013). Additionally, regular trials cannot cover all aspects of drug use. Through the voluntary Post-marketing Drug Safety Surveillance System (Li et al., 2014), users report their experiences on drug usage and related safety issues. Nevertheless, the system suffers several limitations such as incomplete reporting, under-reporting, and delayed reporting.

Recent advances in automated pharmacovigilance are based on collecting large amounts of text about adverse drug events from various platforms, such as medical forums (e.g., AskaPatient), biomedical publications, and social media, and training Natural Language Processing (NLP) models to automatically detect whether a given textual record contains information about adverse drug reactions, which is usually framed as a binary classification task. Text mentions of adverse drug events include a plethora of drug names and adverse reactions. Figure 1 shows an example annotated with concepts from the Unified Medical Language System (UMLS). To understand the drug information and corresponding adverse reactions, the NLP model needs to capture abundant medical knowledge and be able to do relational reasoning.

Early studies used rule-based methods (Xu et al., 2010; Sohn et al., 2014) with manually built rules or applied machine learning algorithms such as conditional random fields (Nikfarjam et al., 2015; Wang et al., 2022), support vector machine (Bollegala et al., 2018), and neural networks (Cocos et al., 2017; Huynh et al., 2016). These approaches can process text with manual feature engineering or enable automated feature learning with
deep learning methods, allowing for automated ADE detection. However, they are limited in capturing rich contextual information and relational reasoning.

Graphs are expressive and can represent various data. For example, nodes in a graph for a collection of texts can represent various entities, such as words, phrases, and documents, while edges represent relationships between them. Such text graphs together with graph neural networks are widely used in NLP applications such as sentiment classification and review rating (Yao et al., 2019; Lin et al., 2021; Zhang et al., 2020). Recently, graphs have been used for text representation with graph boosting (Shen et al., 2020) or contextualized graph embeddings (Gao et al., 2022) for ADE detection. Other works have applied knowledge graph embeddings and link prediction to ADE prediction in drug-effect knowledge-graphs (Kwak et al., 2020; Joshi et al., 2022). However, medical knowledge plays an important role in ADE detection from text, and so far there are no studies that incorporate medical knowledge in a text graph and learn concept-aware representations that inject medical concepts (e.g., the UMLS concepts as illustrated in Figure 1) into the text embeddings.

Our previous model CGEM (Gao et al., 2022) applied a heterogeneous text graph, embodying word, concept, and document relations for ADE corpus, to learn contextualized graph embeddings for ADE detection. Here, we extend this work by showing how the graph can be augmented with medical knowledge from the UMLS metathesaurus (Bodenreider, 2004). In addition, we deploy concept-aware self-attention that applies different feature learning for various types of nodes. We name our model as KnowCAGE (Knowledge-augmented Concept-Aware Graph Embeddings). Our contributions are thus summarized as follows:

* We introduce medical knowledge, i.e., the UMLS metathesaurus, to augment the contextualized graph embedding model for representation learning on drug adverse events.
* A concept-aware self-attention is devised to learn discriminable features for concept (from the medical knowledge), word and document nodes.
* Experimental results evaluated in four public datasets from medical forums, biomedical publications and social media show our approach outperforms recent advanced ADE detection models in most cases.

## 2 Related Work

Recent advances on adverse drug event detection use word embeddings and neural network models to extract text features and capture the drug-effect interaction. Many studies deploy recurrent neural networks to capture the sequential dependency in text. For example, Cocos et al. (2017) utilized a Bidirectional Long Short-Term Memory (BiLSTM) network and Luo (2017) proposed to learn sentence- and segment-level representations based on LSTM. To process entity mentions and relations for ADE detection and extraction, pipeline-based systems (Dandala et al., 2019) and jointly learning methods (Wei et al., 2020) are two typical approaches.

Several recent publications studied graph neural networks for ADE detection. Kwak et al. (2020) built a drug-disease graph to represent clinical data for adverse drug reaction detection. GAR (Shen et al., 2021) uses graph embedding-based methods and adversarial learning. CGEM (Gao et al., 2022) combines contextualized embeddings from pretrained language models with graph convolutional neural networks.

Some other studies also adopted other neural network architectures such as capsule networks and self-attention mechanism. Zhang et al. (2020) proposed the gated iterative capsule network (GICN) using CNN and a capsule network to extract the complete phrase information and deep semantic information. The gated iterative unit in the capsule network enables the clustering of features and captures contextual information. The attention mechanism prioritizes representation learning for the critical parts of a document by assigning

Figure 1: An example of a text mentioning an adverse drug event from Karimi et al. (2015). The recognition of drugs and adverse reactions requires medical knowledge and relational reasoning.
them higher weight scores. Ge et al. (2019) employed multi-head self-attention and Wunnava et al. (2020) developed a dual-attention mechanism with BiLSTM to capture semantic information in the sentence.

Another direction of related work is knowledge augmentation for deep learning models. Many publications adopt knowledge graph to guide the representation learning in various applications (Ji et al., 2022). For example, Ma et al. (2018) injected commonsense knowledge into a long short-term memory network and (Liang et al., 2022) enhanced the graph convolutional network with affective knowledge to improve aspect-based sentiment analysis. Knowledge injection is also used for other applications such as hate speech detection (Pamungkas et al., 2021), mental healthcare (Yang et al., 2022), and personality detection Poria et al. (2013).

## 3 Methods

This section introduces the proposed graph-based model with knowledge augmentation, i.e., Knowledge-augmented Concept-Aware Graph Embeddings (KnowCAGE), as illustrated in Figure 2. The model consists of four components: 1) Knowledge-augmented Graph Construction, 2) Heterogeneous Graph Convolution, 3) Concept-aware Attention, and 4) Ensemble-based ADE classification layers. Following TextGCN (Yao et al., 2019), we construct a heterogeneous text graph, which contains three types of nodes: words, documents and concepts, and we augment it with medical knowledge from the Unified Medical Language System metahesaurus. Heterogeneous graph convolution network is then used to encode the text graph and learn rich representations. We use the contextualized embeddings from pretrained BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) to represent the node features in the heterogenous text graph. The adjacency matrix and feature matrix obtained from the embedding layers are inputs to graph neural network encoders which take into account the relationships and information between and within the nodes. Considering different types of nodes, we use a concept-aware self-attention, inspired by the entity-aware representation learning (Yamada et al., 2020), which treats the different types of nodes differently, allowing the most significant content to have the largest contribution to the final prediction. To boost the prediction of ADE even further, we follow the BertGCN model (Lin et al., 2021) and apply an ensemble classifier with contextualized embeddings on one hand and the graph networks on the other, and learn a weight coefficient to balance these two prediction branches.

### Knowledge-augmented Graph Construction

We firstly build the heterogenous text graph for the whole document collection by using the external knowledge source - UMLS - to augment the word/document graph with concept information. Representing text in a heterogeneous graph can provide different perspectives for text encoding and improve ADE detection. In the UMLS metahesaurus, different words or phrases are assigned different Concept Unique Identifiers (CUI), where each CUI represents one concept class. Every concept class has an attribute "preferred name" which is a short description or a synonym of this concept. Our model uses UMLS to obtain the "preferred name" of each word in the dataset and add words in "preferred name" to the graph as the concept nodes. Therefore, the augmented graph also contains concept nodes in addition to the word and document nodes. The number

Figure 2: An illustration of the model architecture with knowledge-augmented graph embeddings and concept-aware representations
of total nodes \(n=n_{d}+n_{w}+n_{c}\), where \(n_{d}\), \(n_{w}\) and \(n_{c}\) are the numbers of documents, words and concepts, respectively. There are five types of edges, i.e., word-word, word-concept, document-concept, concept-concept and document-word edges. The weights of document-word edges and document-concept edges are calculated as the term frequency-inverse document frequency (TF-IDF), while the weights of the other edges are defined as the positive point-wise mutual information (PMI) of the respective words or concepts. Specifically, the weight between the node \(i\) and the node \(j\) is computed as:

\[\mathbf{A}_{ij}=\left\{\begin{array}{ll}\text{PMI}(i,j),&\text{PMI}>0;i,\text { j: word/concept}\\ \text{TF-IDF}_{\text{ij}},&\text{i: document, j: word/concept}\\ 0,&\text{otherwise}\end{array}\right.\]

We use pretrained contextualized embeddings from language models. Given the dimension of embeddings denoted as \(d\), the pooled output of contextualized document encoding are denoted as \(\mathbf{H}_{doc}\in\mathbb{R}^{n_{d}\times d}\). We initialize word and concept nodes with a zero matrix to get the initial feature matrix which is used as input to the graph neural network:

\[\mathbf{H}^{[0]}=\left(\begin{array}{c}\mathbf{H}_{doc}\\ \mathbf{0}\end{array}\right), \tag{1}\]

where \(\mathbf{H}^{[0]}\in\mathbb{R}^{[n_{d}+n_{w}+n_{c})\times d}\) and \([0]\) denotes the initial layer.

### Heterogeneous Graph Convolution

We adopt graph neural networks over the heterogeneous text graph to learn complex relations between words, concepts, and documents. Specifically, given the initial input features \(\mathbf{H}^{[0]}\) obtained from pretrained language models and the adjacency matrix \(\mathbf{A}\), we update the representations via graph convolution. A forward pass of the \(i\)-th layer of a convolutional graph network can be denoted as:

\[\mathbf{H}^{[i+1]}=f\left(\mathbf{\hat{A}}\mathbf{H}^{[i]}\mathbf{W}^{[i]} \right), \tag{2}\]

where \(\mathbf{\hat{A}}\) is the normalized adjacency matrix, \(\mathbf{H}^{[i]}\) are the hidden representations of \(i\)-th layer, \(\mathbf{W}^{[i]}\) is the weight matrix, and \(f(\cdot)\) is an activation function. The KnowCAGE framework can adopt various types of convolutional graph neural networks. Our experimental study chooses three representative models, i.e., Graph Convolutional Network (GCN) (Kipf and Welling, 2017), Graph Attention Network (GAT) (Velickovic et al., 2018), and Deep Graph Convolutional Neural Network (DGCNN) (Zhang et al., 2018). GCN is a spectral-based model with a fixed number of layers where different weights are assigned to layers and the update of node features incorporates information from the node's neighbors. It employs convolutional architectures to get a localized first-order representation. Graph attention layers in GAT assign different attention scores to one node's distant neighbors and prioritize the importance of different types of nodes. DGCNN concatenates hidden representations of each layers to capture rich substructure information and adopt a SortPooling layer to sort the node features.

### Concept-aware Attention Mechanism

Different types of nodes have various impacts on the prediction of adverse drug events. Inspired by the contextualized entity representation learning from the knowledge supervision of knowledge bases (Yamada et al., 2020), we propose to use a concept-aware attention mechanism that distinguishes the types of nodes, especially the concept nodes, and better captures important information related to the positive or negative ADE classes.

Two types of nodes may not have the same impact on each other. Thus, we use different transformations for different types of nodes in the concept-aware attention mechanism in order to learn concept-aware attentive representations. We obtain key and value matrices \(\mathbf{K}\in\mathbb{R}^{l\times d_{h}}\) and \(\mathbf{V}\in\mathbb{R}^{l\times d_{h}}\) similarly to the key and value in the self-attention of transformer network (Vaswani et al., 2017). Concept-aware attention has nine different query matrices \(\mathbf{Q}\) for concept nodes \(c\), word nodes \(w\) and document nodes \(d\), i.e., \(\mathbf{Q}_{ww}\), \(\mathbf{Q}_{cc}\), \(\mathbf{Q}_{dd}\), \(\mathbf{Q}_{cw}\), \(\mathbf{Q}_{wc}\), \(\mathbf{Q}_{wd}\), \(\mathbf{Q}_{dw}\), \(\mathbf{Q}_{dc}\), and \(\mathbf{Q}_{cd}\in\mathbb{R}^{l\times d_{h}}\). Then, we use \(\mathbf{Q}\), \(\mathbf{K}\) and \(\mathbf{V}\) to compute the attention scores. For example, for \(i\)-th document and \(j\)-th concept nodes, i.e., \(\mathbf{x}_{i}\) and \(\mathbf{x}_{j}\in\mathbb{R}^{d_{h}}\), we calculate the attention score as:

\[\alpha_{ij}=\text{Softmax}\left(\frac{(\mathbf{K}\mathbf{x}_{j})^{\top}\mathbf{ Q}_{cd}\mathbf{x}_{i}}{\sqrt{l}}\right) \tag{3}\]

The concept-aware representation \(\mathbf{h}_{i}\in\mathbb{R}^{l}\) for the \(i\)-th document is obtained as:

\[\mathbf{h}_{i}=\sum_{j=1}^{n_{c}}\alpha_{ij}\mathbf{V}\mathbf{x}_{j} \tag{4}\]

We can obtain the representations of word and concept nodes in the same way. These concept-aware representations are fed to the graph network as the node features in the next iteration of model updating.
### Classification Layers and Model Training

We apply the two linear layers and a softmax function over the concept-aware document embeddings \(\mathbf{h}_{i}\) to compute probability of classifying the document in each class \(\mathbf{p}_{g}\), representing the presence or absence of mentions of ADE in the document. Besides, the interpolation of the prediction probability of two classifiers is adopted to combine the prediction of graph-based modules and pretrained language model-based predictions (Lin et al., 2021). We use a similar classification module to process the contextualized embeddings from the pretrained language model (the upper branch in Fig. 2) and denote the corresponding classification probabilities by \(\mathbf{p}_{c}\). A weight coefficient \(\lambda\in[0,1)\) is introduced to balance the results from graph-based encoding and contextualized models:

\[\mathbf{p}=\lambda\mathbf{p}_{g}+(1-\lambda)\mathbf{p}_{c}. \tag{5}\]

This interpolation strategy can also be viewed as a weighted ensemble of two classifiers.

ADE detection is a binary classification task and the classes are highly imbalanced in most datasets. To complicate the matter further, most datasets contain only a small number of samples, making the downsampling method to balance the classes inappropriate. This study applies the weighted binary cross-entropy loss function to alleviate this problem. The weighted loss function is denoted as:

\[\mathcal{L}=\sum_{i=1}^{N}[-w_{+}y_{i}\log(p_{i})-w_{-}(1-y_{i})\log(1-p_{i})], \tag{6}\]

where \(w_{+}=\frac{N_{1}}{N_{0}+N_{1}}\) and \(w_{-}=\frac{N_{0}}{N_{0}+N_{1}}\) are weights of documents predicted as positive or negative samples respectively, \(N_{0}\) and \(N_{1}\) are the numbers of negative/positive samples in the training set, and \(y_{i}\) is the ground-truth label of a document. The Adam optimizer (Kingma and Ba, 2015) is used for model optimization.

## 4 Experimental Setup

Our goal is to conduct experiments on four ADE datasets and answer the following research questions.

**RQ1:**: How does the proposed model perform in ADE detection on texts from various sources, compared to other methods?
**RQ2:**: How does the heterogeneous graph convolution with knowledge augmentation improve the accuracy of ADE detection?
**RQ3:**: Does the concept-aware attention improve the accuracy of the heterogeneous graph convolution to detect ADE?
**RQ4:**: What is the impact of pretraining domains and contextualized language representation on the performance of the method in ADE election?

In this section we will describe the setup of the experiments and in the next section we will present the results of the experiments.

### Data and Preprocessing

We used four datasets from the medical forum, biomedical publications and social media, as summarized in Table 1, for evaluation. We preprocess data by removing stop words, punctuation, and numbers. For the data collected from Twitter, we use the tweet-preprocessor Python package 1 to remove URLs, emojis, and some reserved words for tweets.

Footnote 1: [https://pypi.org/project/tweet-preprocessor/](https://pypi.org/project/tweet-preprocessor/)

**TwiMed (TwiMed-Twitter and TwiMed-Pub) 2** The TwiMed dataset (Alvaro et al., 2017) includes two sets collected from different domains, i.e., TwiMed-Twitter from social media and TwiMed-Pub for biomedical publications. In each document, people from various backgrounds annotate diseases, symptoms, drugs, and their

\begin{table}
\begin{tabular}{l c c c} \hline \hline Dataset & Documents & ADE & non-ADE \\ \hline SMM4H & 2,418 & 1,209 & 1,209 \\ TwiMed-Pub & 1,000 & 191 & 809 \\ TwiMed-Twitter & 625 & 232 & 393 \\ CADEC & 7,474 & 2,478 & 4,996 \\ \hline \hline \end{tabular}
\end{table}
Table 1: A statistical summary of datasets
relationships. A document annotated as outcome-negative is regarded as an adverse drug event. Models are tested using 10-fold cross-validation.

**SMM4H**3 This dataset from Social Media Mining for Health Applications (#SMM4H) shared tasks (Magge et al., 2021) is collected from Twitter with a description of drugs and diseases. We use the official validation set to evaluate the model performance for a fair comparison with baseline models developed in the SMM4H shared task.

Footnote 3: [https://healthlanguageprocessing.org/smm4h-2021/task-1/](https://healthlanguageprocessing.org/smm4h-2021/task-1/)

**CADEC**4 The CSIRO Adverse Drug Event Corpus contains patient-reported posts from a medical forum called AskaPatient (Karimi et al., 2015). It includes extensive annotations on drugs, side effects, symptoms, and diseases. We use 10-fold cross-validation to evaluate the model's performance.

Footnote 4: [https://data.csiro.au/collection/csiro:10948](https://data.csiro.au/collection/csiro:10948)

### Baselines and Evaluation

We compare the performance of our method with two sets of baseline models: 1) models designed for ADE detection and 2) pretrained contextualized models, and report Precision (P), Recall (R), and F1-score.

Customized models for ADE detection are as follows. **CNN-Transfer**(Li et al., 2020) (CNN-T for short) used a convolutional neural network (CNN) for feature extraction and exploited adversarial transfer learning to boost the performance. **HTR-MSA**(Wu et al., 2018) adopted CNN and Bidirectional Long Short-Term Memory (BiLSTM) networks to learn text representations, and learned hierarchical text representation for tweets. It also employed multi-head self-attention. **ATL**(Li et al., 2020) applied adversarial transfer learning to ADE detection with corpus-shared features exploited. **MSAM**(Zhang et al., 2019) used the BiLSTM network to learn semantic representations of sentences and the multi-hop self-attention mechanism to boost the classification performance. **IAN**(Alimova and Solovyev, 2018) interactively learned attention representations through separate modeling on targets and context. **ANNSA**(Zhang et al., 2021) proposed a sentiment-aware attention mechanism to obtain word-level sentiment features and used adversarial training to improve the generalization ability. **CGEM**(Gao et al., 2022), a predecessor of our work, developed a contextualized graph-based model that utilizes contextualized language models and graph neural networks, and also devised an attention classifier to improve the performance.

The previously mentioned ADE detection baselines did not use the SMM4H dataset in their experiments. Therefore, we compare our model with pretrained language models. We use the base version of pretrained models for a fair comparison. Yaseen and Langer (2021) combined the LSTM network with the BERT text encoder (Devlin et al., 2019) for ADE detection. We denote it as BERT-LSTM. Pimpalkhute et al. (2021) introduced a data augmentation method and adopted the RoBERTa text encoder with additional classification layers (Liu et al., 2019) for ADE detection, denoted as RoBERTa-aug. Kayastha et al. (2021) utilized the domain-specific BERTweet (Nguyen et al., 2020) that is pretrained with English Tweets using the same architecture as BERT-base and classified ADE with a single-layer BiLSTM network, denoted as BERTweet-LSTM.

### Hyper-parameters

Table 2 shows the hyper-parameters we tuned in our experiments, where LR is the learning rate. When the number of iterations exceeds a certain threshold, the learning rate scheduler decays the learning rate by the parameter \(\gamma\). In our experiment, we set \(\gamma\) and the iteration milestone to 0.1 and 30, respectively.

## 5 Results

### Comparison with Baselines in Different Domains (RQ1)

We compare our model's predictive performance with baseline models on the TwiMed (Table 3), SMM4H (Table 4) and CADEC (Table 5) datasets. The results of the baselines are taken directly from the original papers.

\begin{table}
\begin{tabular}{l c} \hline \hline Hyper-parameters & Choices \\ \hline LR for text encoder & \(2e^{-5}\), \(3e^{-5}\), \(1e^{-4}\) \\ LR for classifier & \(1e^{-4}\), \(5e^{-4}\), \(1e^{-3}\) \\ LR for graph-based models & \(1e^{-3}\), \(3e^{-3}\), \(5e^{-3}\) \\ Hidden dimension for GNN & 200, 300, 400 \\ Weight coefficient \(\lambda\) & 0, 0.1 0.3, 0.5, 0.7, 0.9 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Choices of hyper-parameters
However, some of the baselines did not conducted experiments on all of the four datasets. Our proposed model outperforms baseline models in most cases, demonstrating its effectiveness on ADE detection from texts in various domains (RQ1). Our model can better balance precision and recall scores, leading to higher F1 scores. Table 5 shows that our model consistently outperforms the baselines. Our proposed model can capture rich features to identify a document containing ADEs.

### Usefulness of the Knowledge Augmented Graph Convolution (RQ2)

Here we investigate in more detail how the heterogeneous graph convolution with knowledge augmentation can help with ADE detection (RQ2). In Table 3, most models such as HTR-MSA, IAN, CNN-T and ATL perform worse on TwiMed-Twitter dataset, showing that it is difficult to process informal tweets with colloquial language. However, the graph-based encoder in our model helps in effectively encoding information from the informal text, resulting in a better ability to capture the relationships between different entities, improving performance in most cases. Table 4 compares our model with several pretrained BERT-based models. Our model

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Models & \multicolumn{2}{c}{P (\%) R (\%) F1 (\%)} \\ \hline BERTweet-LSTM (Kayastha et al., 2021) & 81.2 & 86.2 & 83.6 \\ RoBERTa-aug (Pimplakhute et al., 2021) & 82.1 & 85.7 & 84.3 \\ BERT-LSTM (Yaseen and Langer, 2021) & 77.0 & 72.0 & 74.0 \\ CGEM (Gao et al., 2022) & **86.7** & 93.4 & 89.9 \\ \hline KnowCAGE (GCN) & 86.6 & 93.1 & 89.7 \\ KnowCAGE (GAT) & 85.2 & **96.8** & 90.6 \\ KnowCAGE (DGCNN) & 86.6 & 95.9 & **91.0** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results of SMM4H dataset. Scores are reported for the best performing results, which follows the setup of baselines. The results of baselines are from the corresponding publications. **Bold** text indicates the best performance.

\begin{table}
\begin{tabular}{l c c c|c c c} \hline \hline \multirow{2}{*}{Models} & \multicolumn{3}{c}{TwiMed-Pub} & \multicolumn{3}{c}{TwiMed-Twitter} \\  & P (\%) & R (\%) & F1 (\%) & P (\%) & R (\%) & F1 (\%) \\ \hline HTR-MSA (Wu et al., 2018) & 75.0 & 66.0 & 70.2 & 60.7 & 61.7 & 61.2 \\ IAN (Alimova and Solovyev, 2018) & 87.8 & 73.8 & 79.2 & 83.6 & 81.3 & 82.4 \\ CNN-T (Li et al., 2020) & 81.3 & 63.9 & 71.6 & 61.8 & 60.0 & 60.9 \\ MSAM (Zhang et al., 2019) & 85.8 & 85.2 & 85.3 & 74.8 & **85.6** & 79.9 \\ ATL (Li et al., 2020) & 81.5 & 67.0 & 73.4 & 63.7 & 63.4 & 63.5 \\ CGEM (Gao et al., 2022) & 88.4 & 85.0 & 86.7 & 84.2 & 83.7 & 83.9 \\ \hline KnowCAGE (GCN) & 88.8 & **85.8** & **87.3** & 84.1 & 84.0 & 84.0 \\ KnowCAGE (GAT) & **89.6** & 83.4 & 86.4 & **84.8** & 84.1 & **84.4** \\ KnowCAGE (DGCNN) & 88.7 & 83.7 & 86.1 & 83.5 & 84.1 & 83.8 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results for two TwiMed datasets, i.e., TwiMed-Pub and TwiMed-Twitter. Scores are reported with the mean of 10-fold cross validation following the setup of baselines. The results of baselines are from the corresponding publications. **Bold** text indicates the best performance.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Models & P (\%) & R (\%) & F1 (\%) \\ \hline HTR-MSA (Wu et al., 2018) & 81.8 & 77.6 & 79.7 \\ CNN-T (Li et al., 2020) & 84.8 & 79.4 & 82.0 \\ ATL (Li et al., 2020) & 84.3 & 81.3 & 82.8 \\ ANNSA (Zhang et al., 2021) & 82.7 & 83.5 & 83.1 \\ \hline KnowCAGE (GCN) & 86.2 & 90.2 & 88.2 \\ KnowCAGE (GAT) & 83.9 & 92.0 & 87.8 \\ KnowCAGE (DGCNN) & **86.1** & **92.9** & **89.4** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results for CADEC dataset. Scores are reported with the mean of 10-fold cross validation following the setup of baselines. The results of baselines are from the corresponding publications. **Bold** text indicates the best performance.
differs from the pretrained models by addition employing the GNN architectures in addition to the pretrained embeddings, and the results suggest the GNN can further improve models' performance on this task. Compared with another graph-based model, CGEM, our method applies knowledge-augmentation to incorporate concept information into the graph learning, which is also seen to improve performance of ADE detection in most cases.

Finally, we examine three graph architectures to study which one is most suitable for the ADE detection task. For datasets containing more training samples (i.e., SMM4H and CADEC datasets), DGCNN performs better. When the number of training samples is small, GCN and GAT achieve better performance. Hence, we conclude that the graph-based encoding method improves the performance. However, we also notice that none of examined graph architectures consistently outperforms the others on all three datasets from different domains.

### Effectiveness of the Concept-Aware Attention (RQ3)

We examine the effectiveness of concept-aware attention by comparing it with other two attention mechanisms, i.e., a simple dot-product attention (Gao et al., 2022) and structured self-attention (Lin et al., 2017). Table 6 shows the concept-aware attention consistently achieves the best F1 score on four datasets. The concept-aware attention distinguishes different types of nodes from the heterogeneous graph and make the overall model better utilize the knowledge augmentation from the ULMS, which answers the third research question (RQ3).

### Effect of Pretraining Domains (RQ4)

We use four pretrained contextualized language models to obtain node embeddings. They use the BERT base model architecture but are pretrained with different strategies or corpora collected from different domains. The pretrained language models include: (1) RoBERTa (Liu et al., 2019), a BERT model optimized with more robust approaches; (2) BioBERT (Lee et al., 2020), a domain-specific model trained with biomedical corpora including PubMed abstracts and PubMed Central full-text articles; (3) ClinicalBERT (Alsentzer et al., 2019), a domain-specific model trained on clinical notes from the MIMIC-III database (Johnson et al., 2016); and (4) PubMedBERT (Gu et al., 2021): a domain specific model trained from scratch using abstracts from PubMed biomedical literature. Figure 3 shows that RoBERTa performs slightly worse than other models on the TwiMedPub dataset. For the other three datasets, RoBERTa performs better than its counterparts. One explanation is the discrepancy between different subdomains. SMM4H, TwiMed-Twitter and CADEC from social media and forum contain more informal social text and non-medical terms, while ClinicalBERT, BioBERT and PubMedBERT are pretrained with clinical notes or biomedical articles. Hence, we conclude that the choice of a specific pretrained model is critical for the accuracy in the ADE detection task. Our finding shows domain-specific pretraining can improve the performance to some extent, and RoBERTa can be the first choice when processing informal text, which is the first answer to RQ4.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{Datasets} & \multicolumn{3}{c}{dot-product attention} & \multicolumn{3}{c}{structured attention} & \multicolumn{3}{c}{concept-aware attention} \\  & P (\%) & R (\%) & F1 (\%) & P (\%) & R (\%) & F1 (\%) & P (\%) & R (\%) & F1 (\%) \\ \hline SMM4H & 83.4 & 97.8 & 90.0 & 84.4 & 94.9 & 89.3 & 86.6 & 95.9 & **91.0** \\ TwiMed-Pub & 87.9 & 84.5 & 86.2 & 88.9 & 82.9 & 85.8 & 88.8 & 85.8 & **87.3** \\ TwiMed-Twitter & 84.5 & 82.2 & 83.4 & 83.0 & 81.8 & 82.4 & 84.8 & 84.1 & **84.4** \\ CADEC & 83.8 & 91.8 & 87.6 & 83.5 & 89.3 & 86.3 & 86.1 & 92.9 & **89.4** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison on the choices of attention mechanisms

Figure 3: The effect of contextualized text embeddings pretrained from different domains and with different pretraining strategies.
### Effect of Weight Coefficient (RQ4)

This section studies the weight coefficient \(\lambda\) that balances the pretrained embedding-based and GNN-based classifiers and answers how does the contextualized language representation affect the performance of ADE detection (RQ4). When \(\lambda\) is zero, only pretrained embedding-based classifier is working. Figure 4 shows that F1 score experiences an increase then drops to some extent. In terms of F1 score, the best choices of \(\lambda\) for four datasets are 0.5 (SMM4H), 0.7 (TwiMed-Pub), 0.7 (TwiMed-Twitter) and 0.3 (CADEC). This study reveals that the combination of pretrained embedding and graph learning on the heterogeneous graph boosts the performance, which is the second answer to RQ4.

## 6 Conclusion

The automated detection of adverse drug events from social media content or biomedical literature requires the model to encode text information and capture the relation between drug and adverse effect efficiently. This paper utilizes knowledge-augmented contextualized graph embeddings to learn contextual information and capture relations for ADE detection. We equip different graph convolutional networks with pretrained language representations over the knowledge-augmented heterogeneous text graph and develop a concept-aware attention to optimally process the different types of nodes in the graph. By comparing our model with other baseline methods, experimental results show that graph-based embeddings incorporating concept information from the UMLS can inject medical knowledge into the model and the concept-aware attention can learn richer concept-aware representations, leading to better detection performance.

## Acknowledgements

We acknowledge the computational resources provided by the Aalto Science-IT project and CSC - IT Center for Science, Finland. This work was supported by the Academy of Finland (Flagship programme: Finnish Center for Artificial Intelligence FCAI, and grants 315896, 336033) and EU H2020 (grant 101016775). We thank Volker Tresp, Zhen Han, Ruotong Liao and Zhiliang Wu for valuable discussions.
