Clinical Note Owns its Hierarchy: Multi-Level Hypergraph Neural Networks
  for Patient-Level Representation Learning
Leveraging knowledge from electronic health records (EHRs) to predict a
patient's condition is essential to the effective delivery of appropriate care.
Clinical notes of patient EHRs contain valuable information from healthcare
professionals, but have been underused due to their difficult contents and
complex hierarchies. Recently, hypergraph-based methods have been proposed for
document classifications. Directly adopting existing hypergraph methods on
clinical notes cannot sufficiently utilize the hierarchy information of the
patient, which can degrade clinical semantic information by (1) frequent
neutral words and (2) hierarchies with imbalanced distribution. Thus, we
propose a taxonomy-aware multi-level hypergraph neural network (TM-HGNN), where
multi-level hypergraphs assemble useful neutral words with rare keywords via
note and taxonomy level hyperedges to retain the clinical semantic information.
The constructed patient hypergraphs are fed into hierarchical message passing
layers for learning more balanced multi-level knowledge at the note and
taxonomy levels. We validate the effectiveness of TM-HGNN by conducting
extensive experiments with MIMIC-III dataset on benchmark in-hospital-mortality
prediction.
Nayeon Kim, Yinhua Piao, Sun Kim
# Clinical Note Owns its Hierarchy: Multi-Level Hypergraph

###### Abstract

Leveraging knowledge from electronic health records (EHRs) to predict a patient's condition is essential to the effective delivery of appropriate care. Clinical notes of patient EHRs contain valuable information from healthcare professionals, but have been underused due to their difficult contents and complex hierarchies. Recently, hypergraph-based methods have been proposed for document classifications. Directly adopting existing hypergraph methods on clinical notes cannot sufficiently utilize the hierarchy information of the patient, which can degrade clinical semantic information by (1) _frequent neutral words_ and (2) _hierarchies with imbalanced distribution_. Thus, we propose a taxonomy-aware multi-level hypergraph neural network (TM-HGNN), where multi-level hypergraphs assemble useful neutral words with rare keywords via note and taxonomy level hyperedges to retain the clinical semantic information. The constructed patient hypergraphs are fed into hierarchical message passing layers for learning more balanced multi-level knowledge at the note and taxonomy levels. We validate the effectiveness of TM-HGNN by conducting extensive experiments with MIMIC-III dataset on benchmark in-hospital-mortality prediction.1

Footnote 1: Our codes and models are publicly available at: [https://github.com/ny1031/TM-HGNN](https://github.com/ny1031/TM-HGNN)

## 1 Introduction

With improvement in healthcare technologies, electronic health records (EHRs) are being used to monitor intensive care units (ICUs) in hospitals. Since it is crucial to schedule appropriate treatments for patients in ICUs, there are many prognostic models that use EHRs to address related tasks, such as in-hospital mortality prediction. EHRs consist of three types of data; structured, semi-structured, and unstructured. Clinical notes, which are unstructured data, contain valuable comments or summary of the patient's condition written by medical professionals (doctors, nurses, etc.). However, compared to structured data, clinical notes have been underutilized in previous studies due to the difficult-to-understand contents and the complex hierarchies (Figure 1(a)). Transformer-based (Vaswani et al., 2017) methods like ClinicalBERT (Alsentzer et al., 2019; Huang et al., 2019, 2020) have been proposed to pre-train on large-scale corpus from similar domains, and fine-tune on the clinical notes through transfer learning. While Transformer-based methods can effectively detect distant words compared to other sequence-based methods like convolutional neural networks (Kim, 2014; Zhang et al., 2015) and recurrent neural networks (Mikolov et al., 2010; Tai et al., 2015; Liu et al., 2016), there are still limitations of increasing computational complexity for long clinical notes (Figure 2).

Recently, with the remarkable success of the graph neural networks (GNNs) (Kipf and Welling,

Figure 1: (a) Examples of patient clinical notes with difficult contents (e.g. jargons and abbreviations) and complex structures. Patient \(p_{1}\) owns notes of radiology taxonomy (pink) and nursing taxonomy (blue). (b) Differences between existing hypergraphs and our proposed multi-level hypergraphs.
2017; Velickovic et al., 2018; Brody et al., 2021), graph-based document classification methods have been proposed (Yao et al., 2019; Huang et al., 2019) that can capture long range word dependencies and can be adapted to documents with different and irregular lengths. Some methods build word co-occurrence graphs by sliding fixed-size windows to model pairwise interactions between words (Zhang et al., 2020; Piao et al., 2022; Wang et al., 2022). However, the density of the graph increases as the document becomes longer. Besides, there are also some methods apply hypergraph for document classification (Ding et al., 2020; Zhang et al., 2022), which can alleviate the high density of the document graphs and extract high-order structural information of the documents.

Adopting hypergraphs can reduce burden for managing long documents with irregular lengths, but additional issues remain when dealing with clinical notes: _(1) Neutral words deteriorate clinical semantic information_. In long clinical notes, there are many frequently written neutral words (e.g. "_rhythm_") that do not directly represent the patient's condition. Most of the previous methods treat all words equally at the learning stage, which may result in dominance of frequent neutral words, and negligence of rare keywords that are directly related to the patient's condition. Meanwhile, the neutral word can occasionally augment information of rare keywords, depending on the intra-taxonomy context. Taxonomy represents the category of the clinical notes, where implicit semantic meaning of the words can differ. For example, "_rhythm_" occurred with "_fibrillation_" in _ECG_ taxonomy can represent serious cardiac disorder of a patient, but when "_rhythm_" is written with "_benadryl_" in _Nursing_ taxonomy, it can hardly represent the serious condition. Therefore, assembling intra-taxonomy related words can leverage "_useful_" neutral words with rare keywords to jointly augment the clinical semantic information, which implies the necessity of introducing taxonomy-level hyperedges. (2) _Imbalanced distribution of multi-level hyperedges_. There are a small number of taxonomies compared to notes for each patient. As a result, when taxonomy-level and note-level information are learned simultaneously, note-level information can obscure taxonomy-level information. To learn more balanced multi-level information of the clinical notes, an effective way for learning the multi-level hypergraphs with imbalanced distributed hyperedges is required.

To address the above issues, we propose TM-HGNN (Taxonomy-aware Multi-level HyperGraph Neural Networks), which can effectively and efficiently utilize the multi-level high-order semantic information for patient representation learning. Specifically, we adopt patient-level hypergraphs to manage highly unstructured and long clinical notes and define multi-level hyperedges, i.e., note-level and taxonomy-level hyperedges. Moreover, we conduct the hierarchical message passing from note-level to taxonomy-level hyperedges using edge-masking. To hierarchically learn word embeddings without mixture of information between note and taxonomy, note and taxonomy hyperedges are disconnected. Note-level word embeddings are learned only with intra-note local information. The following taxonomy-level propagation introduce clinical semantic information by assembling the intra-taxonomy words and separating inter-taxonomy words for better patient-level representation learning. The contributions of this article can be summarized as follows (Figure 2):

* To address issue 1, we construct multi-level hypergraphs for patient-level representation learning, which can assemble "_useful_" neutral word with rare keyword via note and taxonomy level hyperedges to retain the clinical semantic information.

Figure 2: Advantages of the proposed model, compared to sequence, graph and hypergraph based models. \(N\) and \(E\) denote the number of nodes and edges respectively. We address issues of complexity and different lengths by adopting the hypergraph to represent each patient. Our model retains semantic information by constructing multi-level hypergraph (Section 3.2), and hierarchical message passing layers (Section 3.3) are proposed for balancing multi-level knowledge for patient representation learning.
* To address issue 2, we propose hierarchical message passing layers for the constructed graphs with imbalanced hyperedges, which can learn more balanced multi-level knowledge for patient-level representation learning.
* We conduct experiments with MIMIC-III clinical notes on benchmark in-hospital-mortality task. The experimental results demonstrate the effectiveness of our approach.

## 2 Related Work

### Models for Clinical Data

With the promising potential of managing medical data, four benchmark tasks were proposed by Harutyunyan et al. (2019) for MIMIC-III (Medical Information Mart for Intensive Care-III) (Johnson et al., 2016) clinical dataset. Most of the previous works with MIMIC-III dataset focus on the structured data (e.g. vital signals with time-series) for prognostic prediction tasks (Choi et al., 2016; Shang et al., 2019) or utilize clinical notes combined with time-series data (Khadanga et al., 2019; Deznabi et al., 2021). Recently, there are approaches focused on clinical notes, adopting pre-trained models such as BERT-based (Alsentzer et al., 2019; Huang et al., 2019; Golmaei and Luo, 2021; Naik et al., 2022) and XLNet-based (Huang et al., 2020) or utilizing contextualized phenotypic features extracted from clinical notes (Zhang et al., 2022).

### Graph Neural Networks for Document Classification

Graph neural networks (Kipf and Welling, 2017; Velickovic et al., 2018; Brody et al., 2021) have achieved remarkable success in various deep learning tasks, including text classification. Initially, transductive graphs have been applied to documents, such as TextGCN (Yao et al., 2019). Transductive models have to be retrained for every renewal of the data, which is inefficient and hard to generalize (Yao et al., 2019; Huang et al., 2019). For inductive document graph learning, word co-occurrence graphs initialize nodes with word embeddings and exploit pairwise interactions between words. TextING (Zhang et al., 2020) employs the gated graph neural networks for document-level graph learning. Following TextGCN (Yao et al., 2019) which applies graph convolutional networks (GCNs) (Kipf and Welling, 2017) in transductive level corpus graph, InducT-GCN (Wang et al., 2022) applies GCNs in inductive level where unseen documents are allowed to use. TextSSL (Piao et al., 2022) captures both local and global structural information within graphs.

However, the density of word co-occurrence graph increases as the document becomes longer, since the fixed-sized sliding windows are used to capture local pairwise edges. In case of hypergraph neural networks, hyperedges connect multiple number of nodes instead of connecting words to words by edges, which alleviates the high density of the text graphs. HyperGAT (Ding et al., 2020) proposes document-level hypergraphs with hyperedges containing sequential and semantic information. HEGEL (Zhang et al., 2022) applies Transformer-like (Vaswani et al., 2017) multi-head attention to capture high-order cross-sentence relations for effective summarization of long documents. According to the reduced computational complexity for long documents (Figure 2), we adopt hypergraphs to represent patient-level EHRs with clinical notes. Considering issues of existing hypergraph-based methods (Figure 2), we construct multi-level hypergraphs at note-level and taxonomy-level for each patient. The constructed graphs are fed into hierarchical message passing layers to capture rich hierarchical information of the clinical notes, which can augment semantic information for patient representation learning.

## 3 Method

### Problem Definition

Our task is to predict in-hospital-mortality for each patient using a set of clinical notes. Given a patient \(p\in\mathcal{P}\) with in-hospital-mortality label \(y\in\mathcal{Y}\), patient \(p\) owns a list of clinical notes \(\mathcal{N}_{p}=[n_{1}^{t_{1}},...,n_{j}^{t_{k}},...]\), and each clinical note \(n^{t}\in\mathcal{N}_{p}\) with taxonomy \(t\in\mathcal{T}_{p}\) contains a sequence of words \(\mathcal{W}_{n^{t}}=[w_{1}^{n^{t}},...,w_{i}^{n^{t}},...]\), where \(j\), \(k\) and \(i\) denote the index of clinical note \(n\), taxonomy \(t\) and word \(w\) of patient \(p\). The set of taxonomies can be represented by \(\mathcal{T}=\{t_{1},t_{2},...,t_{k},...\}\).

Our goal is to construct individual multi-level hypergraphs \(\mathcal{G}_{p}\) for each patient \(p\) and learn patient-level representation \(\mathcal{G}_{p}\) with the multi-level knowledge by hierarchical message passing layers for in-hospital-mortality prediction tasks. Since our model is trained by inductive learning, patient \(p\) is omitted throughout the paper.
### Multi-Level Hypergraph Construction

We construct multi-level hypergraphs for patient-level representation learning, which can address the issues that are mentioned in introduction 1. A hypergraph \(\mathcal{G}^{*}=(\mathcal{V},\mathcal{E})\) consists of a set of nodes \(\mathcal{V}\) and hyperedges \(\mathcal{E}\) where multiple nodes can be connected to single hyperedge \(e\in\mathcal{E}\). A multi-level hypergraph \(\mathcal{G}=\{\mathcal{V},\{\mathcal{E}_{\mathcal{N}}\cup\mathcal{E}_{ \mathcal{T}}\}\}\) is constructed from patient's clinical notes, where \(\mathcal{E}_{\mathcal{N}}\) and \(\mathcal{E}_{\mathcal{T}}\) denote note-level and taxonomy-level hyperedges, respectively. A word node \(v\) exists in note \(n\) with the taxonomy of \(t\) can be represented by \(\{v\in n,n\in t\}\). A note-level hyperedge is denoted as \(e_{n}\), and a taxonomy-level hyperedge is denoted as \(e_{t}\).

Multi-level Positional EncodingThere are three types of entries in the multi-level hypergraph \(\mathcal{G}\), such as word nodes \(\mathcal{V}\), note-level hyperedges \(\mathcal{E}_{\mathcal{N}}\) and taxonomy-level hyperedges \(\mathcal{E}_{\mathcal{T}}\). To distinguish these entries, we propose multi-level positional encoding to introduce more domain-specific meta-information to the hypergraph \(\mathcal{G}\). The function of multi-level positional encoding \(\texttt{mpe}(\cdot)\) can be defined as:

\[\texttt{MPE}(x)=[\tau(x),\mathcal{I}_{\mathcal{W}}(x),\mathcal{I}_{\mathcal{N }}(x),\mathcal{I}_{\mathcal{T}}(x)] \tag{1}\]

where entry \(x\in\{\mathcal{V},\mathcal{E}_{\mathcal{N}},\mathcal{E}_{\mathcal{T}}\}\), and function \(\tau:x\mapsto\{0,1,2\}\) maps entry \(x\) to a single type among nodes, note-level and taxonomy-level hyperedges. Functions \(\mathcal{I}_{\mathcal{W}}(\cdot),\mathcal{I}_{\mathcal{N}}(\cdot)\), and \(\mathcal{I}_{\mathcal{T}}(\cdot)\) maps entry \(x\) to positions in the word, note and taxonomy-level, respectively. To initialize embedding of node \(v\), we concatenate embedding \(\texttt{mpe}(v)\) from multi-level position encoding and word2vec (Mikolov et al., 2010) pre-trained embedding \(\mathbf{z}_{v}\). Since shallow word embeddings are widely used to initialize node embeddings in graph-based document representation (Grohe, 2020), we use word2vec (Mikolov et al., 2010) embedding. A word node embedding \(\mathbf{h}_{v}^{(0)}\) is constructed as follows:

\[\mathbf{h}_{v}^{(0)}=\texttt{MPE}(v)\oplus\mathbf{z}_{v}, \tag{2}\]

where \(\oplus\) denotes concatenation function.

#### 3.2.1 Hyperedge Construction

To extract multi-level information of patient-level representation using clinical notes, we construct patient hypergraphs with two types of hyperedges, one at the note-level hyperedge \(\mathcal{E}_{\mathcal{N}}\) and the other at the taxonomy-level hyperedge \(\mathcal{E}_{\mathcal{T}}\). A word node \(v\) in note \(n\) with taxonomy \(t\) is assigned to one note-level hyperedge \(e_{n}\) and one taxonomy-level hyperedge \(e_{t}\), which can be defined as:

\[\mathcal{E}(v)=\{e_{n},e_{t}|v\in n,n\in t\} \tag{3}\]

Note-level HyperedgesWe adopt linear embedding function \(f_{n}\) and obtain the index embedding

Figure 3: Overview of the proposed TM-HGNN. Taxonomy-aware multi-level hypergraphs are fed into the model for hierarchical message passing. \(\hat{y}\) denotes the patient-level prediction.
using \(\mathcal{I_{N}}(n)\). To preserve time-dependent sequential information of clinical note \(n\), we simply add time information \(\mathbf{t}(n)\) to the embedding. Then initial embedding of note-level hyperedge \(h_{e_{n}}^{(0)}\) with \(\texttt{MPE}(\cdot)\) can be defined as:

\[\mathbf{h}_{e_{n}}^{(0)}=\texttt{MPE}(n)\oplus f_{n}^{\theta}(\mathcal{I_{N}}(n ),\mathbf{t}(n)), \tag{4}\]

where \(\theta\in\mathbb{R}^{d\times d}\) denotes the parameter matrix of function \(f_{n}\). Notably, we set the value of word index \(\mathcal{I_{W}}(n)\) as -1 since the note \(n\) represents higher level information than word \(v\).

Taxonomy-level HyperedgesTaxonomy-level hyperedges \(e_{t}\) are constructed by taxonomy index \(\mathcal{I_{T}}(t)\) through linear layers \(f_{t}\) concatenated with \(\texttt{mpE}(\cdot)\) function, which can be defined as:

\[\mathbf{h}_{e_{t}}^{(0)}=\texttt{MPE}(t)\oplus f_{t}^{\theta}(\mathcal{I_{T}}( t)), \tag{5}\]

where \(\theta\in\mathbb{R}^{d\times d}\) denotes the parameter matrix of function \(f_{t}\). Like note-level hyperedge, we set \(\mathcal{I_{W}}(t)\) and \(\mathcal{I_{N}}(t)\) as -1 since the level of taxonomy \(t\) is higher than the levels of note and word.

### Hierarchical Message Passing

To leverage the characteristics of two types of hyperedges, we propose a hierarchical hypergraph convolutional networks, composed of three layers that allow message passing from different types of hyperedges. In general, we define message passing functions for nodes and hyperedges as follows:

\[\mathcal{F_{W}}(\mathrm{h},\mathcal{E},\theta)=\sigma\bigg{(}\theta\bigg{(} \sum_{u\in\mathcal{E}(v)}\frac{1}{\sqrt{d_{v}}\sqrt{d_{u}}}\mathrm{h}_{u} \bigg{)}\bigg{)}, \tag{6}\]

\[\mathcal{F_{\tau}}(\mathrm{h},\mathcal{V}^{\tau},\theta)=\sigma\bigg{(}\theta \bigg{(}\sum_{z\in\mathcal{V}^{\tau}(e)}\frac{1}{\sqrt{d_{e}}\sqrt{d_{z}}} \mathrm{h}_{z}\bigg{)}\bigg{)}, \tag{7}\]

where \(\mathcal{F_{W}}\) denotes message passing function for word nodes and \(\mathcal{F_{\tau}}\) denotes message passing function for hyperedges with type \(\tau\in\{1,2\}\), i.e., note-level hyperedges and taxonomy-level hyperedges, respectively. Function \(\mathcal{F_{W}}\) updates word node embedding \(\mathrm{h}_{v}\) by aggregating embeddings of connected hyperedges \(\mathcal{E}(v)\). Function \(\mathcal{F_{\tau}}\) updates hyperedge embedding \(\mathrm{h}_{e}\) by aggregating embeddings of connected word nodes \(\mathcal{V}^{\tau}(e)\). \(\sigma\) is the non-linear activation function such as \(\mathrm{ReLU}\), \(\theta\in\mathbb{R}^{d\times d}\) is the weight matrix with dimension \(d\) which can be differently assinged and learned at multiple levels.

Then we can leverage these defined functions to conduct hierarchical message passing learning at the note level and at the taxonomy level.

Initialization LayerDue to the complex structure of the clinical notes, the initial multi-level hypergraph constructed for each patient has a large variance. To prevent falling into local optima in advance, we first use an initialization layer to pre-train the entries of hypergraphs by learning the entire patient graph structure. In this layer, message passing functions are applied to all word nodes \(v\in\mathcal{V}\) and hyperedges \(e\in\mathcal{E_{I}}=\{\mathcal{E_{N}}\cup\mathcal{E_{T}}\}\). Thus, embeddings of node \(v\), hyperedges \(e_{n}\) and \(e_{t}\) at both levels can be defined as:

\[h_{I}(v)=\mathcal{F_{W}}(h_{v}^{(0)},\mathcal{E_{I}}(v),\theta_{I}), \tag{8}\]

\[h_{I}(e_{n})=\mathcal{F_{\tau}}(h_{e_{n}}^{(0)},\mathcal{V}^{\tau}(e_{n}), \theta_{I}),\tau=1 \tag{9}\]

\[h_{I}(e_{t})=\mathcal{F_{\tau}}(h_{e_{t}}^{(0)},\mathcal{V}^{\tau}(e_{t}), \theta_{I}),\tau=2 \tag{10}\]

Note-level Message Passing LayerThen we apply note-level message passing layer on hypergraphs with only word nodes \(v\in\mathcal{V}\) and note-level hyperedges \(e_{n}\in\mathcal{E_{N}}\), and the taxonomy-level hyperedges are masked during message passing. In this layer, the word nodes can only interact with note-level hyperedges, which can learn the intra-note local information.

\[h_{N}(v)=\mathcal{F_{W}}\big{(}h_{I}(v),\mathcal{E_{N}}(v),\theta_{N}\big{)}, \tag{11}\]

\[h_{N}(e_{n})=\mathcal{F_{\tau}}\big{(}h_{I}(e_{n}),\mathcal{V}^{\tau}(e_{n}), \theta_{N}\big{)},\tau=1, \tag{12}\]

\[h_{N}(e_{t})=h_{I}(e_{t}) \tag{13}\]

Taxonomy-level Message Passing LayerThe last layer is the taxonomy-level message passing layer, where all word nodes \(v\in\mathcal{V}\) and taxonomy-level hyperedges \(e_{t}\in\mathcal{E_{T}}\) can be updated. In this layer, we block the hyperedges at the note level. The node representations with note-level information are fused with taxonomy information

\begin{table}
\begin{tabular}{l l} \hline \hline  & Statistics \\ \hline \# of patients & 17,927 \\ \# of ICU stays & 21,013 \\ \hline \# of in-hospital survival & 18,231 \\ \# of in-hospital mortality & 2,679 \\ \hline \# of notes per ICU stay & 13.29 (7.84) \\ \# of words per ICU stay & 1,385.62 (1,079.57) \\ \# of words per note & 104.25 (66.82) \\ \# of words per taxonomy & 474.75 (531.42) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the MIMIC-III clinical notes. Averaged numbers are reported with standard deviation.
via taxonomy-level hyperedges, which can assemble the intra-taxonomy related words to augment semantic information.

\[h_{T}(v)=\mathcal{F}_{V}\big{(}h_{N}(v),\mathcal{E}_{T}(v),\theta_{T}\big{)}, \tag{14}\]

\[h_{T}(e_{n})=h_{N}(e_{n}), \tag{15}\]

\[h_{T}(e_{t})=\mathcal{F}_{\tau}\big{(}h_{N}(e_{t}),\mathcal{V}^{\tau}(e_{t}), \theta_{T}\big{)},\tau=2 \tag{16}\]

#### 3.3.1 Patient-Level Hypergraph Classification

After all aforementioned hierarchical message passing layers, node and hyperedge embeddings \(h_{T}(v),h_{T}(e_{n}),h_{T}(e_{t})\in\mathbf{H}_{T}\) follow mean-pooling operation which summarizes patient-level embedding \(z\), which is finally fed into \(\mathrm{sigmoid}\) operation as follows:

\[\hat{y}=\mathrm{sigmoid}(z) \tag{17}\]

where \(\hat{y}\) denotes the probability of the predicted label for in-hospital-mortality of the patient. The loss function for patient-level classification is defined as the binary cross-entropy loss:

\[\mathcal{L}=-\left(y\times\log\hat{y}+(1-y)\times\log(1-\hat{y})\right) \tag{18}\]

where \(y\) denotes the true label for in-hospital-mortality. The proposed network, TM-HGNN, can be trained by minimizing the loss function.

## 4 Experimental Settings

### Dataset

We use clinical notes from the Medical Information Mart for Intensive Care III (MIMIC-III) Johnson et al. (2016) dataset, which are written within 48 hours from the ICU admission. For quantitative evaluation, we follow Harutyunyan et al.'s (2019) benchmark setup for data pre-processing and train/test splits, then randomly divide 20% of train set as validation set. All patients without any notes are dropped during the data preparation. To prevent overfitting into exceptionally long clinical notes for a single patient, we set the maximum number of notes per patient into 30 from the first admission. Table 1 shows the statistics of pre-processed MIMIC-III clinical note dataset for our experiments. We select top six taxonomies for experiments, since the number of notes assigned to each taxonomy differs in a wide range (Appendix B Table 3). In addition, we select two chronic diseases, hypertension and diabetes, to compare prediction results for patients with each disease.

### Compared Methods

In our experiments, the compared baseline methods for end-to-end training are as follows:

* Word-based methods: word2vec (Mikolov et al., 2013) with multi-layer perceptron classifier, and FastText (Joulin et al., 2017).
* Sequence-based methods: TextCNN (Kim, 2014), Bi-LSTM (Hochreiter and Schmidhuber, 1997), and Bi-LSTM with additional attention layer (Zhou et al., 2016).
* Graph-based methods: TextING (Zhang et al., 2020), InducT-GCN (Wang et al., 2022), and HyperGAT (Ding et al., 2020). In particular, HyperGAT represents hypergraph-based method, and the other graph-based methods employ word co-occurrence graphs.

### Implementation Details

TM-HGNN is implemented by PyTorch (Paszke et al., 2019) and optimized with Adam (Kingma and Ba, 2015) optimizer with learning rate 0.001 and dropout rate 0.3. We set hidden dimension \(d\) of each layer to 64 and batch size to 32 by searching parameters. We train models for 100 epochs with early-stopping strategy, where the epoch of 30 shows the best results. All experiments are trained on a single NVIDIA GeForce RTX 3080 GPU.

## 5 Results

Since the dataset has imbalanced class labels for in-hospital mortality as shown in Table 1, we use AUPRC (Area Under the Precision-Recall Curve) and AUROC (Area Under the Receiver Operating Characteristic Curve) for precise evaluation. It is suggested by Davis and Goadrich (2006) to use AUPRC for imbalanced class problems.

### Classification Performance

Table 2 shows performance comparisons of TM-HGNN and baseline methods. Sequence-based methods outperform word-based methods, which indicates capturing local dependencies between neighboring words benefits patient document classification. Moreover, all graph-based methods outperform sequence-based and word-based methods. This demonstrates ignoring sequential information of words is not detrimental to clinical notes. Furthermore, hypergraphs are more effective than previous word co-occurrence graphs, indicating that
it is crucial to extract high-order relations within clinical notes. In particular, as TM-HGNN outperforms HyperGAT Ding et al. (2020), exploiting taxonomy-level semantic information which represents the medical context of the notes aids precise prediction in patient-level. Another advantage of our model, which captures multi-level high order relations from note-level and taxonomy-level with hierarchy, can be verified by the results in Table 2 where TM-HGNN outperforms T-HGNN. T-HGNN indicates the variant of TM-HGNN, which considers note-level and taxonomy-level hyperedges homogeneous. Likewise, results from hypertension and diabetes patient groups show similar tendencies in overall.

### Robustness to Lengths

To evaluate the performance dependencies to lengths, we divide clinical notes in patient-level into three groups by lengths, which are short, medium, and long (Appendix B, Figure 8). For test set, the number of patients is 645, 1,707, and 856 for short, medium, and long group each, and the percentage of mortality is 6.98%, 10.72%, and 15.89% for each group, which implies patients in critical condition during ICU stays are more likely to have long clinical notes. Figure 4 shows performance comparisons for three divided groups with TextING Zhang et al. (2020) which utilizes word co-occurrence graph, HyperGAT Ding et al. (2020), a ordinary hypergraph based approach, and our multi-level hypergraph approach (TM-HGNN). All three models were more effective to longer clinical notes, which demonstrates graph based models are robust to long document in general. Among the three models, our proposed TM-HGNN mostly performs the best and HyperGAT Ding et al. (2020) follows, and then TextING Zhang et al. (2020). The results demonstrate that our TM-HGNN, which exploits taxonomy-level semantic information, is most effective for clinical notes regardless of the lengths, compared to other graph-based approaches.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Categories**} & \multirow{2}{*}{**Models**} & \multicolumn{2}{c}{**Whole**} & \multicolumn{2}{c}{**Hypertension**} & \multicolumn{2}{c}{**Diabetes**} \\ \cline{3-8}  & & AUPRC & AUROC & AUPRC & AUROC & AUPRC & AUROC \\ \hline \multirow{2}{*}{Word-based} & Word2vec + MLP & 13.49 \(\pm\) 1.68 & 56.65 \(\pm\) 5.12 & 16.82 \(\pm\) 1.78 & 53.56 \(\pm\) 4.20 & 18.15 \(\pm\) 1.42 & 51.94 \(\pm\) 3.40 \\  & FastText & 17.06 \(\pm\) 0.08 & 62.37 \(\pm\) 0.11 & 25.56 \(\pm\) 0.28 & 62.39 \(\pm\) 0.18 & 31.33 \(\pm\) 0.33 & 67.59 \(\pm\) 0.20 \\ \hline \multirow{2}{*}{Sequence-based} & Bi-LSTM w/ Att. & 17.67 \(\pm\) 4.19 & 58.75 \(\pm\) 5.78 & 21.75 \(\pm\) 5.25 & 57.39 \(\pm\) 6.11 & 27.52 \(\pm\) 7.57 & 61.86 \(\pm\) 8.38 \\  & Bi-LSTM w/ Att. & 17.96 \(\pm\) 0.61 & 62.63 \(\pm\) 1.31 & 26.05 \(\pm\) 1.80 & 63.24 \(\pm\) 1.57 & 33.01 \(\pm\) 3.53 & 68.89 \(\pm\) 1.58 \\  & TextCNN & 20.34 \(\pm\) 0.67 & 68.25 \(\pm\) 0.54 & 27.10 \(\pm\) 1.82 & 66.10 \(\pm\) 1.20 & 36.99 \(\pm\) 2.54 & 71.83 \(\pm\) 1.69 \\ \hline \multirow{2}{*}{Graph-based} & TextING & 34.50 \(\pm\) 7.79 & 78.20 \(\pm\) 4.27 & 36.63 \(\pm\) 8.30 & 80.12 \(\pm\) 4.05 & 36.13 \(\pm\) 8.66 & 80.28 \(\pm\) 3.84 \\  & InducT-GCN & 43.03 \(\pm\) 1.96 & 82.23 \(\pm\) 0.72 & 41.06 \(\pm\) 2.95 & 85.56 \(\pm\) 1.24 & 40.59 \(\pm\) 3.67 & 84.42 \(\pm\) 1.45 \\ \hline \multirow{2}{*}{HyperGraph-based} & HyperGAT & 44.42 \(\pm\) 1.96 & 84.00 \(\pm\) 0.84 & 42.32 \(\pm\) 1.78 & 86.41 \(\pm\) 1.01 & 40.08 \(\pm\) 2.45 & 85.03 \(\pm\) 1.20 \\  & T-HGNN (Ours) & 45.85 \(\pm\) 1.91 & 84.29 \(\pm\) 0.31 & 43.53 \(\pm\) 2.01 & 87.07 \(\pm\) 0.64 & 40.47 \(\pm\) 2.29 & 85.48 \(\pm\) 0.92 \\  & TM-HGNN (Ours) & **48.74 \(\pm\) 0.60** & **84.89 \(\pm\) 0.42** & **47.27 \(\pm\) 1.21** & **87.75 \(\pm\) 0.54** & **42.22 \(\pm\) 1.25** & **85.86 \(\pm\) 0.73** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Classification performance comparison on patient-level clinical tasks, evaluated with AUPRC and AUROC in percentages. We report averaged results with standard deviation over 10 random seeds. Values in boldface denote the best results.

Figure 4: Prediction results of TextING, HyperGAT, and TM-HGNN for three patient-level clinical note groups divided by length (short, medium, and long). AUPRC and AUROC are used for evaluation.

Figure 5: Performance results of ablation studies. The effectiveness of the multi-level hypergraph and hierarchical message passing in the proposed model TM-HGNN are validated respectively.
### Ablation Study

Effect of Multi-level HypergraphIn order to validate the effect of multi-level hypergraphs, we ignore taxonomy-level and note-level hyper-edges respectively. _w/o taxonomy_, which ignores taxonomy-level hyperedges, deteriorates the performance most significantly. _w/o note_ shows degraded performance as well. Thus, effectiveness of multi-level hypergraph construction for patient representation learning can be verified (Figure 5).

Effect of Hierarchical Message PassingFigure 5 demonstrates that hierarchical message passing (note-level to taxonomy-level) for multi-level hypergraphs is effective than learning without hierarchies, since _w/o hierarchy_ shows inferior performance compared to TM-HGNN. _w/o hierarchy_ represents T-HGNN from Table 2, which considers every hyperedge as homogeneous. Degraded performance from _w/o initialization_ shows the effectiveness of the initialization layer before hierarchical message passing, which indicates that pre-training on the entire multi-level hypergraphs first benefits the patient-level representation learning.

### Case Study

Hierarchical Message PassingWe visualize the learned node representations based on principal component analysis (PCA) [11] results, as hierarchical message passing continues in TM-HGNN. In Figure 6(a), "_rhythm_" from ECG and Nursing/other taxonomy are mapped closely for initial word embeddings, since they are literally same words. As the patient-level hypergraphs are fed into a global-level, note-level, and taxonomy-level convolutional layers in order, words in the same taxonomies assemble, which can be found in Figure 6(b), (c), and (d). As a result, "_rhythm_" of ECG represents different semantic meanings from "_rhythm_" of Nursing/other, as it is learned considerably close to "_fibrillation_" from the same taxonomy.

Importance of Taxonomy-level Semantic InformationTo investigate the importance of taxonomy-level semantic information extraction, we visualize PCA results of the learned node embeddings from the baseline method and the proposed TM-HGNN. We select patient with hospital admission id (HADM_ID) 147702 for case study since TM-HGNN successfully predicts the true label for in-hospital-mortality, which is pos

Figure 6: PCA results of learned node representations from each layer of TM-HGNN, for patient case HADM_ID=147702. "_Rhythm_" and "_fibrillation_" from ECG, "_rhythm_" and "_benadryl_" from Nursing/other taxonomy are highlighted. (a) Input word node embeddings. (b) Initialized node embeddings from the first layer. (c) After second layer, note-level message passing. (d) Final node embeddings from TM-HGNN, after taxonomy-level message passing. Word node embeddings are aligned with the same taxonomy words.

Figure 7: PCA results of learned node representations from HyperGAT (a) and TM-HGNN (b). "_Rhythm_" and "_fibrillation_" from ECG, "_rhythm_" and “_benadryl_" from Nursing/other taxonomy are highlighted.
itive, but the other baseline methods show false negative predictions. As in Figure 7, HyperGAT learns "_rhythm_" without taxonomy-level semantic information, since it is not assembled with other words in the same taxonomy. But TM-HGNN separately learns "_rhythm_" from ECG and "_rhythm_" from Nursing/other based on different contexts, which results in same taxonomy words aligned adjacently, such as "_fibrillation_" of ECG and "_behavioryl_" of Nursing/other. Therefore, in case of TM-HGNN, frequently used neutral word "_rhythm_" from ECG with a word "_fibrillation_" means an irregular "_rhythm_" of the heart and is closely related to mortality of the patient, but "_rhythm_" from Nursing/other with another nursing term remains more neutral. This phenomenon demonstrates that contextualizing taxonomy to frequent neutral words enables differentiation and reduces ambiguity of the frequent neutral words (e.g. "_rhythm_"), which is crucial to avoid false negative predictions on patient-level representation learning.

## 6 Conclusion

In this paper, we propose a taxonomy-aware multi-level hypergraph neural networks, TM-HGNN, a novel approach for patient-level clinical note representation learning. We employ hypergraph-based approach and introduce multi-level hyperedges (note and taxonomy-level) to address long and complex information of clinical notes. TM-HGNN aims to extract high-order semantic information from the multi-level patient hypergraphs in hierarchical order, note-level and then taxonomy-level. Clinical note representations can be effectively learned in an end-to-end manner with TM-HGNN, which is validated from extensive experiments.

## Limitations

Since our approach, TM-HGNN, aggregates every note during ICU stays for patient representation learning, it is inappropriate for time-series prediction tasks (e.g. vital signs). We look forward to further study that adopts and applies our approach to time-series prediction tasks.

## Ethics Statement

In MIMIC-III dataset Johnson et al. (2016), every patient is deidentified, according to Health Insurance Portability and Accountability Act (HIPAA) standards. The fields of data which can identify the patient, such as patient name and address, are completely removed based on the identifying data list provided in HIPAA. In addition, the dates for ICU stays are shifted for randomly selected patients, preserving the intervals within data collected from each patient. Therefore, the personal information for the patients used in this study is strictly kept private. More detailed information about deidentification of MIMIC-III can be found in Johnson et al. (2016).

## Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)] and the Bio & Medical Technology Development Program of the National Research Foundation (NRF) funded by the Ministry of Science & ICT (RS-2023-00257479), and the ICT at Seoul National University provides research facilities for this study.
