11institutetext: 1 Nanjing University of Aeronautics and Astronautics, Jiangsu
Nanjing, China
2 School of Informatics, University of Leicester, Leicester LE1 7RH, U.K
11email<EMAIL_ADDRESS>
# Multi-scale Edge-based U-shape Network for Salient Object Detection

Han Sun1(✉) Yetong Bian1 Ningzhong Liu1 Huiyu Zhou2

# Multi-scale Edge-based U-shape Network for Salient Object Detection

Han Sun1(✉) Yetong Bian1 Ningzhong Liu1 Huiyu Zhou2

###### Abstract

Deep-learning based salient object detection methods achieve great
improvements. However, there are still problems existing in the predictions,
such as blurry boundary and inaccurate location, which is mainly caused by
inadequate feature extraction and integration. In this paper, we propose a
Multi-scale Edge-based U-shape Network (MEUN) to integrate various features at
different scales to achieve better performance. To extract more useful
information for boundary prediction, U-shape Edge Network modules are embedded
in each decoder units. Besides, the additional down-sampling module alleviates
the location inaccuracy. Experimental results on four benchmark datasets
demonstrate the validity and reliability of the proposed method. Multi-scale
Edge-based U-shape Network also shows its superiority when compared with 15
state-of-the-art salient object detection methods.

###### Keywords:

Salient object detection Multi-scale feature Edge-based

##  1 Introduction

Saliency Object Detection (SOD) is a significant branch of computer vision. It
is involved in plenty of computer vision tasks, such as video summarization
[15], visual object tracking [1], semantic segmentation [29]. The salient
object is defined as one or more objects that are the most attractive in an
image. Saliency Object Detection aims to segment the object with its boundary
and background accurately. It can be viewed as a binary classification task to
assign foreground pixels to saliency and background pixels to non-saliency.

At the very beginning, SOD models mainly depend on manual features such as
texture, color and global contrast [17]. Until 2015, the sudden rise of neural
network sets off a wave of wind in the SOD and even the whole field of
computer vision. Most scholars turned their attention to neural network models
with a great capacity to extract multi-level and multi-scale information,
especially after the Fully Convolutional Neural Network (FCN) [14] was
proposed.

Although SOD models with convolution neural networks (CNNs) achieved
remarkable achievements in recent years, there are still two main problems to
be solved. First, predictions around the boundary areas are prone to make
mistakes. Besides, the background and foreground have high similarities in
some images, making the models confused about object location. These problems
are virtually triggered by improper multi-scale feature integration and
information loss.

In the past years, most researches are devoted to improving the border region
of salient objects, for example, adding some edge information to the framework
[2] [31], utilizing top local features to refine the saliency map via multiple
cycles [5] or raising the weight of edge pixel error punishment in loss
function [4]. Recently, a few articles start to refocus on the accuracy of the
overall positioning of salient objects [27].

Inspired by these articles, we propose a Multi-scale Edge-based U-shape
Network (MEUN), which improves the object location and boundary prediction by
additional down-sampling and edge complement. As known, the features extracted
from shallow layers usually have high resolution with abundant detailed
information. If these features could be fully utilized in the network, the
prediction performance around the saliency borders will be greatly promoted.
The features drawn from the deep layers contain rich global textual clues.
However, too many straight down-sampling operations will lead the features to
lose detailed information and influence the prediction of boundary areas. If
we simply up-sample the down-sampled features to the size of inputs, the
prediction must be too coarse to meet the requirements of SOD nowadays. So we
design our module with the encoder-decoder structure called “U-shape Network”
[18].The outputs of each unit in the encoder are transmitted to corresponding
unit in the decoder to supply some shallow details for the deep global
semantic features. Such a solution could alleviate the problems mentioned
above. Besides, we designed the U-shape edge network (UEN) block and the
additional down-sampling module (ADM) to further explore the fine details at
the bottom sides and the semantic information at the top sides, separately.

In general, to improve the accuracy of the object location and the details of
the object boundary, our method properly extracts and merges features from the
deep layers and the shallow layers. The main contributions are summarized as
following three points:

  * $\bullet$

We propose the U-shape edge network (UEN) block to fuse the edge information
and features extracted by the backbone. Based on the originally advantageous
U-shape structure, this module could efficiently add boundary information so
that the boundary prediction of saliency maps can be improved.

  * $\bullet$

An additional down-sampling module (ADM) is designed to further extract useful
global structural information. It makes the network proposed going deeper than
other networks, so our method can obtain a more accurate saliency object
position.

  * $\bullet$

We build an efficient framework to fully combine and fuse edge information,
detailed information and semantic clues. Many experiments are conducted to
illustrate the validity of our algorithm and this model could surpass most
models on four large-scale salient object detection datasets.

##  2 Related Work

Because our method designs a novel multi-scale feature integration U-shape
Network based on edge information, this part briefly includes three aspects of
works, U-shape models, edge-fused models and multi-scale feature aggregation
models.

U-Net based. U-shape network is a variant of Fully Convolutional Networks and
is widely used in image segmentation and saliency object detection. This
structure has a strong capacity to make the features from different modules
interact with each other. It was first proposed in [20] for biomedical image
segmentation in 2015. Recently, many saliency models adopt U-structure to
obtain multi-scale features and efficient aggregation. Zhou et al. [33] design
a module based on U-Net, in which an attention mechanism is taken to jointly
enhance the quality of salient masks and reduce the consumption of memory
resources. Qin et al. [18] design a two-level nested U-structure without using
any pre-trained backbones from image classification. Although these models
take advantage of the U-structure, there are still some limitations. For
example, U2Net [18] is extremely complex with a large number of parameters up
to 176.3M.

Edge based. Boundary prediction accuracy is always a problem that most models
cannot deal with well. In recent years, edge information is found to be a
valuable complement to the salient boundary prediction. EGNet [31] is design
to explicitly model complementary salient object information and salient edge
information within the network to preserve the salient object boundaries. To
supplement semantics and make networks focus on object boundaries, Cen et al.
[2] introduce an edge-region complementary strategy and an edge-focused loss
function to predict salient maps with clear boundaries accurately. Li et al.
[11] propose an edge information-guided hierarchical feature fusion network.
Zhou et al. [32] designed a multi-stage and dual-path network to jointly learn
the salient edges and regions, in which the region branch network and edge
branch network can interactively learn from each other. These methods improve
the boundary prediction of the salient object to some extent, but the overall
positioning lacks novel improvement.

Multi-scale feature aggregation based. The problems mentioned above are also
partly caused by inappropriate aggregation and fusion. Thus, many articles
looked for more efficient and effective methods to aggregate low-level
features with detailed information and high-level features with semantic
clues. Feng et al. [8] employ a Dilated Convolutional Pyramid Pooling (DCPP)
module to generate a coarse prediction based on global contextual knowledge.
In MINet [16], the aggregate interaction module can efficiently utilize the
features from adjacent layers through mutual learning, while the self-
interaction module makes the network adaptively extract multi-scale
information from data and better deal with scale variation. Li et al. [10]
present a residual refinement network with semantic context features,
including a global attention module, a multi-receptive block module, and a
recurrent residual module. These methods are mainly innovative in feature
extraction and fusion, but they do not explore features with more
possibilities.

##  3 Methodology

In this section, the proposed model is discussed in detail. The whole method
is further separated into four parts. In the first part, we give an overview
of the network architecture. Then we introduce the U-shape edge network block
and the additional down-sampling module. Finally, the use of loss function is
explained clearly.

###  3.1 Overview

Fig. 1 presents the overall framework, mainly containing the UENs and ADM. The
encoder could adopt any common backbone as its encoder, such as ResNet50 [9].
The backbone is divided into five groups according to the size of output
features. One group’s output is transmitted to the next group and the module
with the corresponding scale in the decoder.

Figure 1: The overview of the architecture.

We suppose that detailed supplement is not enough for promoting edge
prediction, so edge information is introduced as a complement for better
performance. Firstly, it should be clear that what kind of edge is needed. The
background prediction will be disturbed by the background edges marked as
saliency. Because in a saliency map, background pixels should be marked as
non-salient. To avoid the disturbance from background edges, a salient edge
ground truth is used to supervise the edge map generation. The edge features
are down-sampled to different scales of the branches. Then processed edge
feature is inputted into the UENs.

We also design an Additional Down-sampling Module (ADM) to mine more semantic
clues with larger and wider receptive field, which can help locate accurately
and precisely. We build the MEUNetwork based on ResNet-50. The final output of
ResNet-50 is at $7\times 7$ scale, so ADM down-samples the input feature map
by a factor of two. Then, the output of ADM is transmitted to the decoder
along with the features from the backbone. The decoder consists of five
U-shape edge networks (UEN), and the UENs are connected from the bottom up.
The input of the module at the bottom is the element-wise addition of
$Feature_{ADM}$ and $Feature_{b5}$. It can be formulated as

$\operatorname{Input}_{5}=\operatorname{Conv}_{-}2_{3\times
3}\left(\operatorname{Conv}_{-}2_{3\times
3}\left(\text{Feature}_{b5}\right)+\text{Feature}_{ADM}\right),$ (1)

in which $Input_{5}$ denotes the input of the UEN at the bottom.
$Conv_{-}2_{3\times 3}$ is two sets of one $3\times 3$ convolutional layer
followed by the batch normalization and ReLU activation, which is used as the
basic convolutional unit in this network. $Feature_{ADM}$ and $Feature_{b5}$
are the output of fifth group filters in the encoder and ADM, respectively.
The input of other UEN blocks can be written as

$Input_{i}=Conv_{-}2_{3\times 3}\left(Conv_{-}2_{3\times
3}\left(Feature_{b_{i}}\right)+Edge_{sal}\right),\quad i=1,2,3,4$ (2)

$Input_{i}$ is the input of the $i$th UEN block. $Feature_{b_{i}}$ represents
the output of the $i$th block in the encoder and $Edge_{sal}$ denotes the edge
supplement. It needs to mention that the outputs with different channels from
the backbone are squeezed to 128 channels by a $1\times 1$ convolution for the
reduction of parameters. Besides, to achieve the best balance between
efficiency and quality, edge feature and features in ADM and UENs are all set
as 128 channels.

Besides being sent to the next module, the output of each UEN is used to
generate a saliency map through two $3\times 3$ convolution layers. Each
intermediate prediction is up-sampled to the size of the original image for
supervision. In the decoder, the deepest features at $7\times 7$ scale are
restored step by step. Compared with other predictions, the last prediction
with the highest resolution is the closest to what we want. The final output
of the model is the fusion of all the intermediate predictions integrated by
convolutional layers. Of course, this step could be skipped because the last
saliency map of the decoder is good enough to be the final result.

###  3.2 UEN module

(a) UEN_5

(b) UEN_A

Figure 2: The structures of UEN5 and UEN_A.

Fig. 2 shows the overview of different UEN modules, including the $UEN\\_5$
and the $UEN\\_A$. $DilatedConv_{i}$ in Fig. 2 denotes the atrous convolution
layer with dilation rate $i$. The $3\times 3$ convolutional layer is heavily
used inside this module. Except for the first convolutional layer, each
convolutional layer is followed by a down-sampling by a factor of two. In this
way, richer information with a global view can be extracted at different
scales. We determine the down-sampling times in an $UEN\\_i$ $(i=4,5)$
according to the scale of the input feature. If the input is large enough,
such as $112\times 112$, we could down-sample for three times. However, if the
input itself is small, excessive down-sampling is redundant and unreasonable.
In this situation, two down-sampling operations are enough. For smaller
scales, such as $7\times 7$, the module discards the down-sampling. More
specifically, we remove the pooling layers and change the original
convolutional layer into a dilated convolutional layer. The structure shown in
Fig. 2 (a) is the specific settings of $UEN\\_5$. Compared to $UEN\\_5$,
$UEN\\_4$ reduces a down-sampling operation and an up-sampling operation
followed by a convolutional layer, respectively. After the encoder, a dilated
convolutional layer is followed by the batch normalization and ReLU activation
function. Here, the input of each convolutional layer is an aggregation of two
features. One is the output from the last convolution. The other one is the
feature transmitted from the encoder. Here, the two features are aggregated by
concatenating. Unlike $UEN\\_i$, in Fig. 2 (b), there are three dilated
convolution branches in $UEN\\_A$ with different dilation rates to extract
semantic features with different receptive fields. Kernels of the three
branches are all $3\times 3$ and dilation rates are 1, 2, 4. The final
convolution output $Feature_{UEN}$ is conducted as the output of this module.
The input size of each UEN block and the type used in each stage are present
in Table. LABEL:table1.

Table 1: Input size and the UEN version of each stage in backbone. stage | Input size | The type of UEN
---|---|---
Stage1 | $112\times 112$ | $UEN\\_5$
Stage2 | $56\times 56$ | $UEN\\_4$
Stage3 | $28\times 28$ | $UEN\\_4$
Stage4 | $14\times 14$ | $UEN\\_A$
Stage5 | $7\times 7$ | $UEN\\_A$

###  3.3 Additional Down-sampling Module

In some scenarios of SOD, some salient areas are quite similar to the non-
salient areas around them, for example, the salient object and the reflection
of itself in the mirror. This problem leads to that sometimes models will
treat these easily-confused un-salient regions as salient regions. The purpose
of the Additional Down-sampling Module is to explore more global semantic
clues as much as possible to assist the model position accurately and
eliminate misleading regions. The structure of ADM is shown in Fig. 3. The
input is down-sampled after two convolutional layers. It is reasonable to
infer that the more non-zero pixels in a channel, the more information
contained. On the contrary, if a channel has comparatively more zero pixels
than other channels, we suppose the information content is inadequate. The
global average pooling is followed by two fully connected layers to calculate
the amount of information in each channel. Each element in the processed
vector corresponds to a channel of the down-sampled feature. The down-sampled
feature is multiplied by the vector in channel-wise to make a channel
enhancement. As a result, the weights of these channels lacking information
are weakened and the others are highlighted. The strengthened $Feature_{ADM}$
is used as the final output of ADM.

Figure 3: The structure of ADM.

###  3.4 Loss Function

Our algorithm supervises five outputs of the model, among which one output is
the edge saliency map. Since there is no ground truth of the saliency object
edge in the training set, it is necessary to generate the edge labels first.
We calculate the gradient for each pixel of the original label. Then, multiply
the gradient by itself to get the new value of each pixel:

$S_{edge}=d_{x}^{2}+d_{y}^{2}$ (3)

$S_{\text{edge }}$ is the edge ground truth wanted. $d_{x}$ and $d_{y}$ are
the gradient values of the pixel in the x-direction and y-direction,
respectively. Finally, the non-zero pixels are set as one and then multiply by
255. The edge saliency map is supervised by the edge ground truth generated in
this way. The edge loss inspired by [2] is written as:

$Edge\\_Loss=-\sum_{e\in E^{+}}\log Pr\left(e_{n}=1\mid W\right)-\sum_{e\in
E^{-}}\log Pr\left(e_{n}=0\mid W\right)$ (4)

$e_{n}$ represents the pixels in the edge prediction map, and $W$ represents
the parameters in the model. $\operatorname{Pr}\left(e_{n}=1\mid W\right)$
means the probability that the pixel $e_{n}$ is calculated as salient edge.
$E^{+}$ indicates the salient edge pixel set, while $E^{-}$ is a set that
contains all of the pixels except salient edge pixels. Original labels
supervise those intermediate saliency maps. For better performance, this
method introduces two kinds of loss functions: $BCE\\_Loss$ and $IoU\\_Loss$.
BCE Loss is widely used in the training process of binary classifiers. The
function is written as:

$BCE\\_{Loss}=-\sum_{(x,y)}{[g(x,y)\log p(x,y)+(1-g(x,y)\log(1-p(x,y)))]},$
(5)

where $g(x,y)\in[0,1]$ is the ground truth label of the pixel $(x,y)$ and
$p(x,y)\in[0,1]$ is the predicted probability of being salient object. IoU
Loss is joint to highlight the prediction error around the salient object
boundary. As mentioned before, one of the difficulties in saliency object
detection is that the prediction error frequently occurs in the object
boundary area, which makes the prediction edge ambiguous and leads to error
and missed detection. If the saliency map fails to align with the ground
truth, the unaligned part will pay the penalty. The addition of IoU Loss can
help the model to correct errors in this specified area. The function is
denoted as

$sum_{inter}=\sum_{y_{i}\in I}y_{i},\quad I=Sal\\_map\odot Ground\\_Truth,$
(6) $sum_{union}=\sum_{y_{j}\in J}y_{j},\quad J=Sal\\_map+Ground\\_Truth,$ (7)
$IoU\\_Loss={\left(1-\frac{sum_{inter}+1}{sum_{union}-sum_{inter}+1}\right)}\times\frac{1}{H\times
W}$ (8)

in which $H$ is the height of $Sal\\_map$ while $W$ is the width. In the
experiment, we found that the lower resolution of a saliency map, the more
inaccurate the result is. If the losses of all saliency map are given the same
weight, it will lead to instability and influence the model performance.
Saliency maps generated from deep layers should be weakened with lower weight,
so the coefficient is reduced by two. The whole loss function could be written
as

$\displaystyle L_{all}=L_{edge}+BCE_{-}Loss_{\text{united
}}+IoU\\_Loss_{united}$ (9)
$\displaystyle+\sum_{i=1}^{5}\frac{1}{2^{i-1}}({BCE\\_Loss}_{i}+IoU\\_Loss_{i})$

$BCE_{-}Loss_{united}+IoU\\_Loss_{united}$ is the loss of united saliency
maps. $BCE_{-}Loss_{i}+IoU\\_Loss_{i}$ is the loss of the $i$th intermediate
saliency map. The lager $i$ is, the closer the saliency map is to the bottom.

##  4 Experiment

###  4.1 Experimental Setting

Datasets and implementation details. The model is trained on the DUTS-TR with
10553 images. In detail, we trained the model using the SGD optimizer with
initial learning rate 3e-5, 0.9 momentum, 5e-4 weight decay, and batch size
16. Because the ResNet-50 parameters are pre-trained on ImageNet, the learning
rate of this part is a tenth of the randomly initialized parts which is set as
3e-5. Then, the trained model is tested on five datasets, including DUTS-TE
with 5019 images, DUT-OMROM with 5168 images, HKU-IS with 4447 images, ECSSD
with 1000 images and PASCAL-S with 850 images.

Metrics. We comprehensively evaluate the model with four metrics widely used
in SOD, including mean F-measure ($mF$), mean absolute error ($MAE$),
structure-measure ($Sm$) and enhanced-alignment measure ($Em$). Mean F-measure
is the average of F-measures, which is calculated as:

$F_{\beta}=\frac{\left(1+\beta^{2}\right)\times{Precision}\times{Recall}}{\beta^{2}\times{Precision}+{Recall}}$
(10)

$\beta^{2}$ is usually set as 0.3 to put more emphasis on precision [21]. Mean
absolute error is the average of each pixel’s absolute error in predictions.
Before calculation, each pixel is normalized to $[0,1]$, and $MAE$ is
calculated as

$MAE=\frac{1}{H\times W}\sum_{i=1}^{H}\sum_{j=1}^{W}|P(i,j)-G(i,j)|$ (11)

$H$ and $W$ is the height and width of the saliency map. $P(i,j)$ is the
prediction at location $(i,j)$. $G(i,j)$ denotes the value at location $(i,j)$
in the ground truth. S-measure is a metric proposed to evaluate the structural
similarity between the prediction and the corresponding ground truth.
S-measure is formulated as

$S=\alpha S_{o}+(1-\alpha)S_{r},$ (12)

where $S_{o}$ represents the object-oriented structural similarity while
$S_{r}$ represents the region-oriented structural similarity. $\alpha$ is set
to 0.5, the same as in [6]. E-measure [7] is widely used in SOD as well.

$E=\frac{1}{W\times H}\sum_{x=1}^{w}\sum_{y=1}^{h}\phi FM(i,j)$ (13)

where $\phi FM(i,j)$ is the enhanced alignment matrix. E-measure considers the
performance of both the global average of image and local pixel when
evaluating the model.

###  4.2 Ablation Study

In general, there is no need to down-sample the feature at $7\times 7$ scale
because it has a sufficiently large receptive field. However, the experiment
results in Table. 2 is somewhat counterintuitive. It shows that there is still
some different global semantic information waiting to be discovered. The
version for comparison is a U-shaped vanilla model, which is trained with the
initial learning rate $10^{-4}$. Here we use the same learning rate to train
the parameters in the backbone and modules designed in our method. The rise of
$MAE$, $mF$, $Sm$ and $Em$ reflects ADM’s reliability and validity. Fig. 4
shows the comparison of visualized results between the vanilla network with
and without ADM. UENs generate multi-scale features on account of the input
level and integrate the features with edge information, which guides the model
to pay attention to the boundary areas. The experimental data also reveals
that the UENs make greater improvements in various aspects.

Figure 4: Comparison of predictions between the vanilla network with and without ADM. Table 2: Comparison of networks with and without the proposed modules, ADM and UENs. Base: a U-structure network with usual U-shape modules.  Datasets | DUTS-TE | DUT-OMROM
---|---|---
Base | +ADM | +UENs | $mF\uparrow$ | $MAE\downarrow$ | $Sm\uparrow$ | $Em\uparrow$ | $mF\uparrow$ | $MAE\downarrow$ | $Sm\uparrow$ | $Em\uparrow$
✓ |  |  | 0.851 | 0.038 | 0.894 | 0.901 | 0.772 | 0.065 | 0.839 | 0.859
✓ | ✓ |  | 0.860 | 0.035 | 0.898 | 0.907 | 0.778 | 0.059 | 0.843 | 0.870
✓ | ✓ | ✓ | 0.870 | 0.031 | 0.904 | 0.917 | 0.790 | 0.052 | 0.851 | 0.881

###  4.3 Comparison with Other State-of-the-art Methods

A large number of experiments are conducted to convince the validity of our
approach. To make a clear comparison with other state-of-the-art (SOTA)
methods, we list all the results in Table. 3. There are 15 SOTA methods
proposed in the past three years, namely, BMPM [30], RAS [3], PiCANet [13],
R3Net [5], BASNet [19], PoolNet [12], TDBU [22], PAGE [23],F3Net [24], CPD
[28], EGNet [31], MLMSNet [26], U2Net [18], MINet [16], LDF [25]. For a fair
comparison, all of the compared saliency maps are provided or generated from
released models by the authors. The evaluation codes are all the same. As
shown in Table. 3 and Fig. 5, our algorithm surpasses most of the methods.

Table 3: Quantitative comparison with state-of-the-art methods on five datasets. The best results are highlighted in bold. The best and the second best results are highlighted in red and green respectively. Datasets | DUTS-TE | ECSSD | DUT-OMROM | HKU-IS
---|---|---|---|---
Metrics | $mF$ | MAE | $Sm$ | $Em$ | $mF$ | MAE | $Sm$ | $Em$ | $mF$ | MAE | $Sm$ | $Em$ | $mF$ | MAE | $Sm$ | $Em$
BMPM[30] | .745 | .049 | .862 | .860 | .868 | .044 | .911 | .914 | .692 | .064 | .809 | .837 | .871 | .038 | .907 | .937
RAS[3] | .751 | .059 | .839 | .861 | .889 | .059 | .893 | .914 | .713 | .062 | .814 | .846 | .874 | .045 | .888 | .931
R3Net[5] | .785 | .057 | .834 | .867 | .914 | .040 | .910 | .929 | .747 | .063 | .815 | .850 | .893 | .036 | .895 | .939
PiCANet[13] | .749 | .051 | .867 | .852 | .885 | .044 | .917 | .910 | .710 | .065 | .835 | .834 | .870 | .039 | .908 | .934
MLMSNet[26] | .799 | .045 | .856 | .882 | .914 | .038 | .911 | .925 | .735 | .056 | .817 | .846 | .892 | .034 | .901 | .945
PAGE[23] | .777 | .051 | .854 | .869 | .906 | .042 | .912 | .920 | .736 | .066 | .824 | .853 | .882 | .037 | .903 | .940
CPD[28] | .805 | .043 | .869 | .886 | .917 | .037 | .918 | .925 | .747 | .056 | .825 | .866 | .891 | .034 | .905 | .944
BASNet[19] | .756 | .048 | .866 | .884 | .880 | .037 | .916 | .921 | .756 | .056 | .836 | .869 | .895 | .032 | .909 | .946
F3Net[24] | .840 | .035 | .888 | .902 | .925 | .033 | .924 | .927 | .766 | .053 | .838 | .870 | .840 | .062 | .855 | .859
PoolNet[12] | .799 | .040 | .879 | .881 | .910 | .042 | .917 | .921 | .739 | .055 | .832 | .858 | .885 | .032 | - | .941
TDBU[22] | .767 | .048 | .865 | .879 | .880 | .040 | .918 | .922 | .739 | .059 | .837 | .854 | .878 | .038 | .907 | .942
EGNet[31] | .815 | .039 | .875 | .891 | .920 | .041 | .918 | .927 | .755 | .052 | .818 | .867 | .898 | .031 | .918 | .948
U2Net[18] | .792 | .044 | .861 | .886 | .892 | .033 | .928 | .924 | .761 | .054 | .847 | .871 | .896 | .031 | .916 | .948
MINet[16] | .828 | .037 | .884 | .917 | .924 | .033 | .925 | .953 | .756 | .055 | .833 | .873 | .908 | .028 | .920 | .961
LDF[25] | .855 | .034 | .892 | .910 | .930 | .034 | .924 | .925 | .773 | .051 | .838 | .873 | .914 | .027 | .919 | .954
Ours | .870 | .031 | .904 | .917 | .936 | .028 | .934 | .929 | .790 | .052 | .851 | .881 | .917 | .026 | .925 | .956

Quantitative comparison. Table. 3 shows the quantitative evaluation results of
the SOTA methods mentioned above and our model in terms of $mF$, $MAE$, $Sm$,
and $Em$. The proposed method consistently performs better than all the
competitors across four metrics on four datasets. In terms of $Em$, our method
achieves the second best overall performance, which is slightly inferior to
MINet. It is worth noting that MEUNet achieves the best performance in terms
of the mean F-measure and structure quality evaluation $Sm$.

Qualitative comparison. Fig. 5 shows the visual comparison between our model
and other SOTA methods. The first column is the images and the second column
is the corresponding ground truths. Our result is in the third column. It can
be observed that our method could locate the salient object accurately and
segment the foreground and background around the boundary areas precisely.
Predictions in rows 1, 2 and 3 reveal the abilities of the models to deal with
detailed extraction, and other lines indicate the location ability. The method
proposed in this article is good at keeping as much detail as possible with
sharp edges.

Figure 5: Visual comparison of the previous SOTA methods and our method
denoted as ’Ours’.

For example, the legs of the bee in the first picture are clearly present in
its prediction, and the snake is continuous and complete only on our map. We
suppose that the detail retention ability ascribes to the UENs and the
location ability owes to the ADM. In addition to this, the artful structural
design allows the features from different layers to complement each other and
fuse properly. Our model achieves the best in terms of the overall effect.

##  5 Conclusion

In this paper, we present a Multi-scale Edge-based U-shape Network with both
enhanced high-level semantic features and low-level detail information for
SOD. The central architecture of our network is a U-shape encoder-decoder
structure which mainly consists of UENs and ADM. The UEN is an outstanding
evolution of U-Net, and it contains multi-scale features with rich information
and edge information that could assist boundary prediction. ADM can help the
network to discover semantic clues from a more global view. Besides, we employ
some tricks to improve the performance of the model. For example, the edge
saliency map is supervised by salient edges to suppress the edges from the
background. The fusion of the parallel four saliency maps is outputed as the
final result.

## References

  * [1] Borji, A., Frintrop, S., Sihite, D.N., Itti, L.: Adaptive object tracking by learning background context. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. pp. 23–30. IEEE (2012)
  * [2] Cen, J., Sun, H., Chen, X., Liu, N., Liang, D., Zhou, H.: Wfnet: A wider and finer network for salient object detection. IEEE Access 8, 210418–210428 (2020)
  * [3] Chen, S., Tan, X., Wang, B., Hu, X.: Reverse attention for salient object detection. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 234–250 (2018)
  * [4] Chen, Z., Zhou, H., Lai, J., Yang, L., Xie, X.: Contour-aware loss: Boundary-aware learning for salient object segmentation. IEEE Transactions on Image Processing 30, 431–443 (2020)
  * [5] Deng, Z., Hu, X., Zhu, L., Xu, X., Qin, J., Han, G., Heng, P.A.: R³net: Recurrent residual refinement network for saliency detection. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. pp. 684–690 (7 2018)
  * [6] Fan, D.P., Cheng, M.M., Liu, Y., Li, T., Borji, A.: Structure-measure: A new way to evaluate foreground maps. In: Proceedings of the IEEE international conference on computer vision. pp. 4548–4557 (2017)
  * [7] Fan, D.P., Gong, C., Cao, Y., Ren, B., Cheng, M.M., Borji, A.: Enhanced-alignment measure for binary foreground map evaluation. arXiv preprint arXiv:1805.10421 (2018)
  * [8] Feng, M., Lu, H., Yu, Y.: Residual learning for salient object detection. IEEE Transactions on Image Processing 29, 4696–4708 (2020)
  * [9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [10] Li, T., Song, H., Zhang, K., Liu, Q.: Learning residual refinement network with semantic context representation for real-time saliency object detection. Pattern Recognition 105, 107372 (2020)
  * [11] Li, X., Song, D., Dong, Y.: Hierarchical feature fusion network for salient object detection. IEEE Transactions on Image Processing 29, 9165–9175 (2020)
  * [12] Liu, J., Hou, Q., Cheng, M.M., Feng, J., Jiang, J.: A simple pooling-based design for real-time salient object detection. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3912–3921 (2019)
  * [13] Liu, N., Han, J., Yang, M.H.: Picanet: Learning pixel-wise contextual attention for saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3089–3098 (2018)
  * [14] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3431–3440 (2015)
  * [15] Ma, Y.F., Lu, L., Zhang, H.J., Li, M.: A user attention model for video summarization. In: Proceedings of the tenth ACM international conference on Multimedia. pp. 533–542 (2002)
  * [16] Pang, Y., Zhao, X., Zhang, L., Lu, H.: Multi-scale interactive network for salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9413–9422 (2020)
  * [17] Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A.: Saliency filters: Contrast based filtering for salient region detection. In: 2012 IEEE conference on computer vision and pattern recognition. pp. 733–740. IEEE (2012)
  * [18] Qin, X., Zhang, Z., Huang, C., Dehghan, M., Zaiane, O.R., Jagersand, M.: U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recognition 106, 107404 (2020)
  * [19] Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., Jagersand, M.: Basnet: Boundary-aware salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7479–7489 (2019)
  * [20] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [21] Rutishauser, U., Walther, D., Koch, C., Perona, P.: Is bottom-up attention useful for object recognition? In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004\. vol. 2, pp. II–II. IEEE (2004)
  * [22] Wang, W., Shen, J., Cheng, M.M., Shao, L.: An iterative and cooperative top-down and bottom-up inference network for salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5968–5977 (2019)
  * [23] Wang, W., Zhao, S., Shen, J., Hoi, S., Borji, A.: Salient object detection with pyramid attention and salient edges. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1448–1457 (2019)
  * [24] Wei, J., Wang, S., Huang, Q.: F³net: Fusion, feedback and focus for salient object detection. In: AAAI (2020)
  * [25] Wei, J., Wang, S., Wu, Z., Su, C., Huang, Q., Tian, Q.: Label decoupling framework for salient object detection. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 13022–13031 (2020)
  * [26] Wu, R., Feng, M., Guan, W., Wang, D., Lu, H., Ding, E.: A mutual learning method for salient object detection with intertwined multi-supervision. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 8142–8151 (2019)
  * [27] Wu, Y., Liu, Y., Zhang, L., Cheng, M.M.: Edn: Salient object detection via extremely-downsampled network. ArXiv abs/2012.13093 (2020)
  * [28] Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3902–3911 (2019)
  * [29] Zeng, Y., Zhuge, Y., Lu, H., Zhang, L.: Joint learning of saliency detection and weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7223–7233 (2019)
  * [30] Zhang, L., Dai, J., Lu, H., He, Y., Wang, G.: A bi-directional message passing model for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1741–1750 (2018)
  * [31] Zhao, J.X., Liu, J.J., Fan, D.P., Cao, Y., Yang, J., Cheng, M.M.: Egnet: Edge guidance network for salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8779–8788 (2019)
  * [32] Zhou, S., Wang, J., Wang, L., Zhang, J., Wang, F., Huang, D., Zheng, N.: Hierarchical and interactive refinement network for edge-preserving salient object detection. IEEE Transactions on Image Processing 30, 1–14 (2020)
  * [33] Zhou, S., Wang, J., Zhang, J., Wang, L., Huang, D., Du, S., Zheng, N.: Hierarchical u-shape attention network for salient object detection. IEEE Transactions on Image Processing 29, 8417–8428 (2020)


# Recent Advances in Predictive Modeling with Electronic Health Records

Jiaqi Wang1 Junyu Luo1 Muchao Ye1 Xiaochen Wang1 Yuan Zhong1 Aofei Chang1
Guanjie Huang1 Ziyi Yin1 Cao Xiao2 Jimeng Sun3&Fenglong Ma1 1Pennsylvania
State University, 2GE Healthcare, 3University of Illinois Urbana-Champaign
{jqwang, junyu, muchao, xcwang, yuanzhong, aofei, gzh8, ziyiyin,
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The development of electronic health records (EHR) systems has enabled the
collection of a vast amount of digitized patient data. However, utilizing EHR
data for predictive modeling presents several challenges due to its unique
characteristics. With the advancements in machine learning techniques, deep
learning has demonstrated its superiority in various applications, including
healthcare. This survey systematically reviews recent advances in deep
learning-based predictive models using EHR data. Specifically, we begin by
introducing the background of EHR data and providing a mathematical definition
of the predictive modeling task. We then categorize and summarize predictive
deep models from multiple perspectives. Furthermore, we present benchmarks and
toolkits relevant to predictive modeling in healthcare. Finally, we conclude
this survey by discussing open challenges and suggesting promising directions
for future research.

##  1 Introduction

Advancements in healthcare technology have sparked a revolution in patient
information storage. An increasing number of health institutions and providers
are now adopting electronic health record (EHR) systems to digitize patient
information. This enables healthcare professionals in extracting valuable
insights from EHR data, thereby supporting various aspects of
healthcare,ranging from enhancing clinical decision-making to predicting
diseases, enabling personalized medicine, driving quality improvement
initiatives, and contributing to public health surveillance and
epidemiological research.

This survey focuses on predictive modeling in healthcare, which refers to
using machine learning techniques to analyze patients’ historical health data
along with current observations to support diagnosis or make predictions about
future health events. This approach usually leverages large datasets, often
derived from EHR systems, to identify patterns, trends, and relationships that
can be used to forecast specific health-related events. Predictive modeling
has various applications in healthcare, including but not limited to disease
risk assessment, hospital discharge potential or readmission risk, treatment
outcomes prediction, and health resource allocation. In other words,
predictive modeling can enhance clinical decision-making, improve patient
care, and contribute to more efficient and cost-effective healthcare delivery.

However, accurate and reliable predictive modeling in healthcare is
challenging due to the unique characteristics of EHR data, summarized as
follows:

  * •

Temporal Dynamics: The sequentially collected EHR data exhibits temporal
dynamics. Changes in a patient’s health status over time may not follow a
linear pattern, wihle capturing temporal dependencies requires sophisticated
modeling techniques.

  * •

High Dimensionality: Medical codes in EHR data (e.g., diagnosis, medication,
procedure codes) are high-dimensional. When encoded using multihot encoding,
the codes that are related to the prediction task are often sparse.

  * •

Multimodalities and Heterogeneity: Health data comprises of multiple
modalities such as clinical notes, laboratory results, imaging reports, and
administrative records. The data could also be assembled from different
sources. The heterogeneity of the shapes and sources of the data can bring
challenges to data integration, which is a foundational task before predictive
modeling.

  * •

Imbalanced Data: In healthcare, some patient cohorts could be far smaller than
certain majority groups, while certain outcomes or events may be rare, leading
to imbalances in the distribution of classes. This can impact the performance
of predictive models, making them biased toward the majority class.

  * •

Clinical Explainability: Healthcare professionals often require predictive
models to be explainable. Many advanced machine learning algorithms, while
effective, may lack explainability, making it challenging for clinicians to
understand, trust, and utilize the model results.

In this comprehensive survey, we present a systematic overview of recent
advancements in deep learning techniques to enhance the performance of
predictive models by addressing the aforementioned challenges. The
contributions of this paper can be summarized as follows: (1) We introduce a
taxonomy based on the techniques employed in existing deep learning-based
predictive modeling approaches, offering a detailed analysis of current
methods in Section 3. (2) Common benchmark datasets and toolkits are
summarized to provide a valuable resource for researchers and practitioners in
Section 4. (3) We discuss open challenges and propose future research
directions in the field of predictive modeling in Section 5.

##  2 Background

#### EHR Data

Let $\mathcal{D}=\\{\mathcal{P}_{1},\cdots,\mathcal{P}_{N}\\}$ denote the
extracted ataset from an EHR system, where $\mathcal{P}_{n}$ represents the
$n$-th patient information. EHR data consists of a sequence of a patient’s
visit information associated with a time stamp, which can be represented as
$\mathcal{P}_{n}=\\{(\mathcal{V}^{n}_{1},t^{n}_{1}),\cdots,(\mathcal{V}^{n}_{T},t^{n}_{T})\\}$,
where $\mathcal{V}_{i}^{n}$ and $t_{i}^{n}$ denotes the visit information and
the time information, respectively, and $T$ is the number of visits. Due to
the heterogeneity of EHR data, each visit $\mathcal{V}_{i}^{n}$ usually
contains $M$ modalities, denoted as
$\mathcal{V}_{i}^{n}=\\{\mathcal{M}_{i,j}^{n}\\}_{j=1}^{M}$, where each
modality $\mathcal{M}_{i,j}^{n}$ can be either demographical information,
diagnosis codes, medication codes, procedure codes, images, time-series
monitoring data, or even a clinical note.

#### Predictive Modeling

The predictive modeling task is formulated as follows: Given the historical
EHR data of the $n$-th patient as represented below,

$\small\begin{split}\mathcal{P}_{n}&=\\{(\mathcal{V}^{n}_{1},t^{n}_{1}),\cdots,(\mathcal{V}^{n}_{T},t^{n}_{T})\\}\\\
&=\\{([\mathcal{M}_{1,1}^{n},\cdots,\mathcal{M}_{1,M}^{n}],t^{n}_{1}),\cdots,([\mathcal{M}_{T,1}^{n},\cdots,\mathcal{M}_{T,M}^{n}],t^{n}_{T})\\},\end{split}$

the goal is to predict the future health event or outcome $y_{n}$ after a
specified time window (e.g., six months or one year after the last timestamp
$T$). The applications of predictive modeling tasks typically involve binary
classification. The cross-entropy (CE) loss is employed to train the models:

$\small\mathcal{L}=\sum_{n=1}^{N}\text{CE}(f(\mathcal{P}_{n};\Theta),y_{n}),$

where $f(\cdot)$ represents the model, and $\Theta$ denotes the model
parameter set. In the following sections, we will summarize recent
advancements in the development of accurate deep learning-based predictive
models (i.e., $f(\cdot)$) in healthcare.

##  3 Existing Progress

We summarize the recent work on predictive modeling from different
perspectives, as shown in Table 1. In the following subsections, we provide
details of each type of existing work.

Model Name | Venue | Model Focus | Tasks | Datasets
---|---|---|---|---
F1 | F2 | F3 | F4 | F5 | F6 | F7
Retain Choi et al. (2016b) | NeurIPS | ✓ |  |  | ✓ |  |  |  | Heart Failure | Private
Dipole Ma et al. (2017) | KDD | ✓ |  |  | ✓ |  |  |  | Diagnosis Prediction | Private
Timeline Bai et al. (2018) | KDD | ✓ |  |  | ✓ |  |  |  | Admission | SEER
Stage-Net Gao et al. (2020) | WWW | ✓ |  |  |  |  |  |  |  Decompensation Mortality |  MIMIC, ESRD
Health-ATM Ma et al. (2018c) | SDM | ✓ | ✓ |  |  |  |  |  | Congestive Heart Failure | Private, EMRbots
BEHRT Li et al. (2020) |  Scientific Reports | ✓ |  |  | ✓ |  |  |  |  Disease Prediction Diesease Embedding | CPRD
HiTANet Luo et al. (2020) | KDD | ✓ | ✓ |  | ✓ |  |  |  | Disease Prediction | Private
T-LSTM Baytas et al. (2017) | KDD | ✓ |  |  |  |  |  |  |  Diabetes Mellitus Parkinson’s Progression |  EMRBots, PPMI
RetainEX Kwon et al. (2018) | TVCG | ✓ |  |  | ✓ |  |  |  |  Heart Failure Cataract | HIRA
Concare Ma et al. (2020b) | AAAI | ✓ | ✓ |  | ✓ |  |  |  | Mortality | MIMIC, Private
TContextGGAN Xu et al. (2022) | TKDE | ✓ | ✓ |  |  |  |  |  |  Heart Failure Sepsis Risk | MIMIC
GRAM Choi et al. (2017) | KDD | ✓ |  | ✓ | ✓ |  |  |  | Dignosis Prediction | MIMIC, Private
KAME Ma et al. (2018b) | CIKM | ✓ |  | ✓ | ✓ |  |  |  | Dignosis Prediction | MIMIC, Private
PRIME Ma et al. (2018a) | KDD | ✓ |  | ✓ |  |  |  |  |  Heart Failure, COPD kidney Disease | Private
MedPath Ye et al. (2021a) | WWW | ✓ |  | ✓ | ✓ |  |  |  |  Heart Failure, COPD Kidney Disease | Private
KerPrint Yang et al. (2023a) | AAAI | ✓ |  | ✓ | ✓ |  |  |  | Disease Prediction | MIMIC, Private
MedRetriever Ye et al. (2021b) | CIKM | ✓ |  | ✓ | ✓ |  |  |  |  Heart Failure, COPD Kidney Disease | Private
F-LSTM & F-CNN Tang et al. (2020) | JAMIA | ✓ |  |  |  | ✓ |  |  |  Acute Renal Failure Shock, Mortality | MIMIC
DCMN Feng et al. (2019) | ICDM | ✓ |  |  |  | ✓ |  |  |  Mortality, Cost | MIMIC, Private
RAIM Xu et al. (2018) | KDD | ✓ |  |  |  | ✓ |  |  |  Decompensation Length of Stay | MIMIC
LstmBert Yang and Wu (2021) | EMNLP | ✓ |  |  |  | ✓ |  |  |  Acute Renal Failure Diagnoses | MIMIC, Private
MedHMP Wang et al. (2023) | EMNLP | ✓ |  |  |  | ✓ |  |  |  Acute Renal Failure, Shock, Mortality, Readmission, Heart Falire, COPD, Amnesia | MIMIC, Private
FedCovid Wang et al. (2022) | ECMLPKDD | ✓ |  |  |  | ✓ |  | ✓ |  Covid-19 Vaccine Side Effects | Private
MUFASA Xu et al. (2021) | AAAI | ✓ |  |  |  | ✓ | ✓ |  | Diagnoses | MIMIC
AutoMed Cui et al. (2022) | BIBM | ✓ |  |  |  |  | ✓ |  |  COPD, Amnesia Kidney Disease | Private
AutoFM Cui et al. (2024) | SDM | ✓ |  |  |  | ✓ | ✓ |  |  Acute Renal Failure Diagnoses | MIMIC
SASMOTE Kosolwattana et al. (2023) |  BioData Mining |  |  |  |  |  |  | ✓ |  Autism Spectrum Disorder, Congenital Heart Disease | Private
NNR Chen and Cai (2015) |  JMBE |  |  |  |  |  |  | ✓ | Diabetes | Private
K-Boosted C5.0 Zhang et al. (2019) | JHE |  |  |  |  |  |  | ✓ | Breast Cancer | Private
MaskEHR Ma et al. (2020a) | SDM | ✓ |  |  |  |  |  | ✓ |  Pastic Quadriplegia CerebralPalsy, Quadriplegia, Diplegic Cerebral Palsy | Private
medGAN Armanious et al. (2020) | MLHC |  |  |  |  |  |  | ✓ | Heart Failure |  Private, MIMIC
synTEG Zhang et al. (2021) | JAMIA | ✓ |  |  |  |  |  | ✓ |  Type-2 diabetes, Heart Failure, Hypertension, COPD | Private
EVA Biswal et al. (2021) | MLHC | ✓ |  |  |  |  |  | ✓ | Heart Failure | Private
TWIN Das et al. (2023) | KDD | ✓ |  |  |  | ✓ |  | ✓ |  Breast Cancer, Lung Cancer |  Project Data Sphere
MedDiffusion Zhong et al. (2024) | SDM | ✓ |  |  |  |  |  | ✓ |  COPD,Heart Failure, Amnesia,Kidney |  MIMIC, Private
promptEHR Wang and Sun (2022) | EMNLP | ✓ |  |  |  | ✓ |  | ✓ | Heart Failure | MIMIC
LSAN Ye et al. (2020) | CIKM | ✓ |  |  | ✓ |  |  |  |  Heart Failure COPD, Kidney Disease | Private
ConCAD Huang and Ma (2021) | ECMLPKDD | ✓ |  | ✓ |  |  |  |  | Sleep Apnea |  Apnea-ECG, MIT-BIH PSG
AccSleepNet Huang et al. (2022) | BIBM | ✓ |  |  |  |  |  |  | Sleep Stage Scoring |  Sleep-Accel, Newcastle-Accel
TrustSleepNet Huang et al. (2022) | BHI | ✓ |  |  |  | ✓ |  |  | Sleep Stage Scoring |  SHHS

Table 1: Summarization of recent work on predictive modeling in healthcare.
F1: modeling temporality; F2: modeling time irregularity; F3: incorporating
extra knowledge; F4: interpretability; F5: modeling multimodal EHR data; F6:
using automated machine learning (AutoML) techniques; and F7: addressing class
imbalance issues.

###  3.1 Basic Deep Learning-based Predictive Models

#### Recurrent Neural Networks-based Models

Due to the temporality of EHR data, recurrent neural networks (RNN) are the
primary structure choice in deep learning for handling temporality. In
general, predictive models of this type regard EHR data as text-like sequence
data and utilize RNNs such as Long Short-Term Memory (LSTM) and Gated
Recurrent Units (GRUs) to propagate each visit information to obtain a
comprehensive representation. An early attempt at this design named Retain
Choi et al. (2016b) is proposed by Choi et al., which augments RNNs with an
attention mechanism and uses those learned attention scores for
interpretability. RNN-based predictive models can even act as a doctor and
predict the diagnosis and medication for a future visit given the previous
input diagnosis and medication codes Choi et al. (2016a). Later, Dipole Ma et
al. (2017) is proposed based on the bidirectional RNN, which aggregates the
input EHR data and can effectively attend features in the hierarchy of visits
for health risk prediction. These successful early attempts show the
effectiveness of the design of combining RNNs with attention mechanisms, and
other designs in this fashion come after that, including Timeline Bai et al.
(2018), Stage-Net Gao et al. (2020) and Health-ATM Ma et al. (2018c).

#### Transformer-based Models

Recent advancements have highlighted the Transformer architecture Vaswani et
al. (2017), which leverages the self-attention mechanism to model
relationships within sequential data. Pioneering studies using Transformers in
healrhcare such as BEHRT Li et al. (2020) and HiTANet Luo et al. (2020), among
others Rasmy et al. (2021); Li et al. (2022); Yang et al. (2023b); Huang et
al. (2022), exemplify this trend. BEHRT Li et al. (2020) innovatively utilizes
a transformer instead of the conventional RNN module to process sequential EHR
records, though it lacks a hierarchical approach for handling code-visit level
data. In contrast, HiTANet Luo et al. (2020) melds the Transformer
architecture with hierarchical modeling. It includes a local evaluation stage,
employing a time-aware Transformer for embedding temporal aspects into visit-
level data, and a global synthesis stage, leveraging a time-aware key-query
attention mechanism for a comprehensive analysis. Adopting Transformer-based
models in EHR predictive modeling marks a significant paradigm shift from
traditional RNNs. These models offer a more sophisticated approach to managing
the intricate and sequential nature of EHR data, demonstrating their immense
potential in enhancing healthcare predictive analytics.

###  3.2 Time-aware Predictive Modeling

A critical aspect of predictive modeling with EHR data is the integration of
temporal information. Contrasting with textual data, the sequence of EHR data
hinges on time information, and the intervals between recordings often vary.
Accurately modeling this time aspect is essential for evaluating the impact of
each patient visit. Addressing this need, T-LSTM Baytas et al. (2017) was
introduced, incorporating an information decay function that acknowledges
potential decay in patient information over time gaps between visits. This
approach modifies the gates of LSTM to enhance risk prediction accuracy.
Building on this concept, several models have emerged. RetainEX Kwon et al.
(2018), an extension of the Retain model, considers information decay and
employs traditional attention mechanisms to assign weights to visits and
diagnosis codes. TimeLine Bai et al. (2018) implements self-attention
mechanisms to boost performance and integrate the information decay function
into patient representation learning. Further advancing this field, HiTANet
Luo et al. (2020) introduces a non-monotonic time attention mechanism tailored
to disease-specific time preferences using a learnable time preference
embedding. Concare Ma et al. (2020b) enhances time modeling through a multi-
channel time series embedding, capturing complex temporal relationships.
Additionally, T-ContextGGAN Xu et al. (2022) employs graph neural networks to
model intricate time paths on graphs. The evolution of time-aware predictive
models in EHR data has significantly progressed from simple time information
integration to complex, disease-specific time preference modeling. These
developments underscore the importance of temporal dynamics in predictive
healthcare analytics, leading to more accurate and personalized patient care
predictions.

###  3.3 Knowledge-Enhanced Predictive Modeling

A critical avenue for enhancing predictive models in the EHR domain lies in
integrating additional external knowledge. Below we categorize the methods for
leveraging external knowledge based on the type of knowledge being utilized.

#### Structured Knowledge

The GRAM framework Choi et al. (2017) utilizes a graph attention network to
integrate hierarchical medical ontologies into the learning process of code
representations. Building upon GRAM, KAME Ma et al. (2018b) advances this
concept by embedding ontology information throughout the entire prediction
process, thereby enriching the model’s contextual understanding. Beyond merely
serving as an additional feature, external knowledge can also function as a
regularization component. For instance, PRIME Ma et al. (2018a) generates a
posterior regularization term from external knowledge, aligning the model’s
predictions with established medical insights. This strategy simplifies
incorporating external knowledge, making it broadly applicable across various
EHR risk prediction frameworks. Besides graph attention, ConCAD Huang and Ma
(2021) develops a cross attention mechanism to combine the deep
representations with expert knowledge, which automatically fuses the knowledge
by emphasizing the important parts based on each other synergistically.

MedPath Ye et al. (2021a) introduces a novel approach by retrieving
personalized external information from knowledge graphs, thereby enhancing the
relevance and utility of the external knowledge. It constructs a personalized
ontology map based on patient-specific data, which is then encoded via a graph
network for application in predictive tasks. Similarly, KerPrint Yang et al.
(2023a) employs both a personalized local knowledge graph based on temporal
data and a global-level knowledge graph, optimizing the usage of external
information. Subsequent methodologies extend the application of this external
knowledge to diverse scenarios, such as the cold-start setting Tan et al.
(2022) and treatment recommendation systems Yao et al. (2023).

#### Unstructured Knowledge

In contrast, some researchers focus on harnessing unstructured data, such as
textual information, which presents unique challenges due to its unorganized
nature. MedRetriever Ye et al. (2021b) innovatively addresses this by creating
a pool of medical text segments from various online sources. It leverages EHR
data and the target disease document embeddings to dynamically retrieve
relevant text segments, offering insights into the progression of a disease
from symptoms to diagnosis.

###  3.4 Preditive Modeling with Multimodal Data

Compared to conventional methods that rely solely on a single modality,
multimodal approaches leverage diverse clinical modalities as input. This
enables them to gather comprehensive information, leading to more promising
performance outcomes.

#### Centralized Learning

The prior study by Tang et al. Tang et al. (2020) suggests the fusion of
demographics and temporal clinical features through concatenation in the early
stages, followed by conventional machine learning models for the ultimate
prediction. In contrast, DCMN Feng et al. (2019) employs a dual memory network
to concurrently process two input modalities, namely waveform and clinical
sequence. The outputs of both memory networks are then summed to facilitate
further predictions.

RAIM Xu et al. (2018) integrates waveform and vital signs as inputs into a
multi-channel attention mechanism module. The module subsequently aggregates
discrete clinical features into the generated embeddings for subsequent
prediction tasks. Another approach, proposed by Yang et al. Yang and Wu
(2021), focuses on fusing temporal clinical features, time-invariant data, and
clinical notes through summation in the output stage. They explore various
combinations of modality-specific encoders to achieve competitive performance.

A more recent work Wang et al. (2023) leverages an attention mechanism to
obtain a weighted summation of multiple clinical modalities, including
temporal clinical features, diagnosis, medication, clinical notes, and
demographics. This approach aims to generate a meaningful multimodal
Electronic Health Record (EHR) representation for both pretraining and task-
specific fine-tuning.

#### Federated Learning

To protect EHR data privacy, the multimodal EHR data is usually stored in
local servers and not shared with others. Thus, collaborative training of
predictive models without sharing EHR data is a practical and challenging
task. FedCovid Wang et al. (2022), a federated learning framework, is
developed to adaptively fuse heterogeneous EHR data at local client sites for
predicting COVID-19 vaccine side effects. Besides, Sachin et al. (2023)
introduces a multimodal contrastive federated learning framework for digital
healthcare, employing a geometric multimodal contrastive representation method
to enhance the representation of various modalities in a shared space, thus
improving inter-modal relationship capture and overall model performance.

###  3.5 AutoML-based Predictive Modeling

EHR typically encompasses structured and unstructured data with sparse and
irregular longitudinal features. In multimodal fusion, the challenge lies in
determining how to effectively fuse different modalities, a problem that often
relies on manual modeling and intuition. We have encountered difficulties
choosing between early, hybrid, and late fusion approaches. Recently,
automated machine learning (AutoML)-based approaches have been proposed to
search for optimal fusion strategies automatically.

MUFASA Xu et al. (2021) is the first work that applied the neural architecture
search (NAS) technique to medical multimodal fusion. It jointly optimizes
multimodal fusion strategies and modality-specific architectures. MUFASA
delineates two types of blocks: modality-specific block, which only allows
inputs from the same modality, and fusion block, which accepts inputs from
both fusion architecture hidden states and modality-specific states. AutoMed
Cui et al. (2022) employs a general Directed Acyclic Graph (DAG) framework,
opting for a cell-based search where each cell consists of a sequentially
ordered set of computation nodes. It uses the same search space design for
modality encoding cells and fusion cells. AutoFM Cui et al. (2024), also
focuses on modality-specific search and multimodal fusion search. It designs
separate search spaces for static and sequential modalities. It incorporates
modality interaction operations within these modality-specific search spaces
to facilitate early interactions. A specialized set of simple fusion
operations is devised to enable effective feature fusion.

###  3.6 Preditive Modeling with Imbalanced Classes

Addressing the issue of class imbalance is another critical challenge in
healthcare predictive modeling. Such imbalance can significantly degrade the
performance of predictive models. To tackle this issue, a variety of
strategies have been developed and extensively researched. In this section, we
systematically categorize and review recent methodologies, providing a
structured overview of the techniques employed to effectively manage class
imbalance in healthcare datasets.

#### Oversampling and Undersampling Techniques

Oversampling techniques are crucial for balancing datasets, effectively
increasing the size of the minority class. SASMOTE Kosolwattana et al. (2023)
that builds on the Synthetic Minority Oversampling Technique (SMOTE) proves
effective in enhancing model accuracy for imbalanced gene risk and heart
disease datasets. Furthermore, deep learning-based resampling NNR Chen and Cai
(2015) shows great success in imbalanced diabetes prediction. Compared to
oversampling, undersampling works by selecting informative samples near the
boundary of the class. For instance, K-Boosted C5.0 Zhang et al. (2019)
utilizes K-means to segregate the majority and minority classes and select an
equal amount of patients per cluster for Breast Cancer prediction.

#### Generative Techniques

Generative models, primarily developed for enhancing entire datasets, can be
specifically tailored to augment minority data. MaskEHR Ma et al. (2020a),
medGAN Armanious et al. (2020), and synTEG Zhang et al. (2021) employ
Generative Adversarial Networks (GANs) for generating synthetic data. EVA
Biswal et al. (2021) and TWIN Das et al. (2023) use Variational Autoencoders
(VAEs) in their approach. MedDiffusion Zhong et al. (2024) leverages the
capabilities of Denoising Diffusion Probabilistic Models (DDPMs).
Additionally, promptEHR Wang and Sun (2022) utilizes a language model for
similar purposes. These models use sequential learning techniques like RNNs
and Transformers to add historical context, which maintains the temporal
coherence of generated data. By incorporating this augmented minority class
data into the training sets, predictive models benefit significantly,
exhibiting enhanced performance and more balanced outcomes.

Name | Data Type | # of Data |  Modalities | Link
---|---|---|---|---
MIMIC-III | Real | 38,597 patients |  Demographics, vital signs, medications, laboratory measurements, observations and notes charted by care providers, fluid balance, procedure codes, diagnostic codes, imaging reports, hospital length of stay, survival data | https://physionet.org/content/mimiciii/1.4/
MIMIC-IV | Real | 40,000+ patients |  Demographics, vital signs, medications, laboratory measurements, observations and notes charted by care providers, fluid balance, procedure codes, diagnostic codes, imaging reports, hospital length of stay, survival data | https://physionet.org/content/mimiciv/2.2/
MIMIC-CXR | Real |  377,110 images 227,835 reports |  Electronic health record data, images (chest radiographs), and natural language (free-text reports) | https://physionet.org/content/mimic-cxr/2.0.0/
eICU | Real |  200,000+ admissions |  Vital sign measurements, care plan documentation, severity of illness measures, diagnosis information, treatment information, and more | https://physionet.org/content/mimic-cxr/2.0.0/
PPMI | Real | 2,230 patients |  Subject characteristics, biospecimen, images, medical history, etc. | https://www.ppmi-info.org/
ADNI | Real | 2,775 patients |  Subject characteristics, genetic data, images, medical history, neuropathology, etc. | https://adni.loni.usc.edu/
Apnea-ECG | Real | 70 recordings |  Subject characteristics, electrocardiogram | https://physionet.org/content/apnea-ecg/1.0.0/
MIT-BIH PSG | Real | 18 recordings |  Subject characteristics, electrocardiogram, electroencephalography, electrooculography, electromyography, etc. | https://physionet.org/content/slpdb/1.0.0/
SHHS | Real | 6,441 patients |  Subject characteristics, electrocardiogram, electroencephalography, electrooculography, electromyography, airflow, etc. | https://sleepdata.org/datasets/shhs
Newcastle-Accel | Real | 28 patients |  Subject characteristics, acceleration, polysomnography. | https://zenodo.org/records/1160410##.YLqiSC1h1qt
Sleep-Accel | Real | 31 patients |  Acceleration, heart rate, steps. | https://physionet.org/content/sleep-accel/1.0.0/
EMRBOTS | Synthetic | 100,000 patients |  Patients’ admissions, demographics, socioeconomics, labs, medications, etc. | http://www.emrbots.org/
Project Data Sphere | Real | 242 studies |  Data provider, sponsor, study phase, linked data, tumor type, access, etc. | https://www.projectdatasphere.org

Table 2: Commonly used publically available health datasets.

###  3.7 Interpretable Preditive Modeling

Interpretability is an important property of health risk prediction models
because it is related to the life and death of patients when they are applied
in real life. Interpretability helps make sure that the doctors and patients
can understand the reasoning of the predictive model and trust the risk
prediction or treatment recommendation of the predictive model. Existing
interpretability mechanisms designed in the health risk prediction models can
be mainly divided into the following three categories.

#### Attention-based Interpretation

The first type of interpretability method uses attention weights to explain
the importance of different diagnosis codes or visits. This method has been
widely used for RNN-based methods Choi et al. (2016b); Ma et al. (2017); Bai
et al. (2018); Ma et al. (2018c) as we have mentioned above. Attention
mechanisms can also be incorporated into Transformer-based predictive models
such as HiTANet Luo et al. (2020) and LSAN Ye et al. (2020). Take LSAN as an
example for illustration. It introduces a hierarchical attention mechanism
that assigns flexible attention weights to different diagnosis codes by their
relevance to corresponding diseases in the diagnosis code level and pays
greater attention to visits with higher relevance in the visit level. Such
design provides a fine-grained interpretability. The attention weights in the
hierarchy of diagnosis codes and visits can be interpreted into which symptoms
and visits are paid more attention to for health risk prediction.

#### Personalized Knowledge Graph-based Interpretation

A weakness of attention-based interpretation is that they cannot explicitly
express the reasoning path of the health risk prediction models. To tackle
that, later research proposes the idea of using a medical knowledge graph to
provide explicit reasoning for interpretation. A representative method for
interpretability in this type is named MedPath Ye et al. (2021a). MedPath uses
SemMed, a large-scale online medical knowledge graph, to extract personalized
knowledge graphs (PKGs) containing all possible disease progression paths from
observed symptoms to target diseases. The extracted PKGs can show how the
existing observed symptoms from a patient can gradually lead to the target
disease, which is more explicit compared to attention weights. Other health
risk prediction models using knowledge graphs include the ones designed by
Yang et al. (2023a); Lyu et al. (2023); Xu et al. (2022). These methods have
shown that knowledge graphs are good candidates for improving performance and
interpretability.

#### Medical Text-based Explicit Interpretation

Another way to provide explicit interpretability is to use unstructured
medical text, first proposed in MedRetriever Ye et al. (2021b). MedRetriever
creates a pool of candidate medical text segments from online medical text
sources and uses the EHR and target disease document embeddings to dynamically
retrieve the related ones that can explain the disease progression from the
symptoms to the target disease. By reading the retrieved medical text in the
end, both patients and doctors can obtain an easy-to-understand
interpretation.

#### Uncertainty-based Interpretation

Although the aforementioned interpretation methods can provide a decent
explanation about how the prediction models work, they still cannot simply say
“I do not know,” when they are uncertain in their predictions. However, it is
particularly important in clinically relevant tasks. Knowing how confident the
prediction is, the clinical experts can trust the results and interpretations
with high confidence and set the ones with low confidence aside for another
manual inspection. To address this issue, Huang and Ma (2022) proposes to use
evidential deep learning to quantify the uncertainty. Besides predicting the
target class, it also predicts the density of the probability of each class,
which can derive associated uncertainty to express how confident the
prediction is.

##  4 Benchmarks and Toolkits

#### Publicly Available Healthcare Databases

We present a summary of commonly used publicly available health datasets in
Table 2. In addition to these widely utilized datasets, certain country-
specific datasets play a crucial role in predictive modeling within the
healthcare domain. For instance, the Clinical Practice Research Datalink
(CPRD, https://cprd.com/) dataset is a comprehensive, anonymized database of
primary care records from the UK. The Health Insurance Review & Assessment
(HIRA) Database (https://www.hira.or.kr/eng/main.do) is a comprehensive
medical database in South Korea, which holds extensive healthcare information
for the entire South Korean population.

#### Private Healthcare Databases

Most extensive healthcare databases operate on a private basis, accessible
only to select researchers who can extract pertinent data, as outlined in
Table 1. Notably, TriNetX (https://trinetx.com/) stands out as a notable
example, representing a comprehensive dataset that seamlessly integrates real-
time access to longitudinal clinical data with cutting-edge analytics. This
integration serves to optimize various aspects, including protocol design,
feasibility assessments, site selection, patient recruitment, and the
generation of real-world evidence. It is noteworthy that the Diamond Network
within TriNetX encompasses data from 92 global institutions, capturing
information from an expansive cohort of 213,167,071 patients. Similarly, the
IBM MarketScan Research Database mirrors TriNetX in functionality, offering
de-identified, longitudinal patient-level claims, and specialty data. This
resource boasts coverage for over 273 million unique patients, providing a
rich source for diverse healthcare analyses.

#### Toolkits

PyHealth (https://pyhealth.readthedocs.io/en/latest/) is a comprehensive deep-
learning toolkit for predictive modeling, which integrates diverse EHR
datasets, such as MIMC, eICU, and all OMOP-CDM-based databases, and several
deep learning algorithms for multiple health-related tasks, e.g., patient
hospitalization prediction, mortality prediction, and ICU length stay
forecasting. This toolkit allows the users to customize their own pipelines by
following the 5 stages: load dataset, define task function, build deep
learning models, model training, and inference. There are also several other
toolkits or open-source codes for specific uses such as MedCAT
(https://medcat.readthedocs.io/en/latest/), Fasten
(https://github.com/fastenhealth), MONAI (https://monai.io/) and NiftyNet
(https://niftynet.io/).

##  5 Open Challenges and Future Directions

In this section, we delve into various open challenges and propose future
research directions across key aspects, including model trustworthiness, data
characteristics, model training, collaborative learning paradigms.

###  5.1 Trustworthy Predictive Modeling

Contemporary predictive modeling techniques predominantly prioritize enhancing
model performance. While performance is undoubtedly pivotal, equal emphasis
must be placed on trustworthiness, encompassing accuracy, reliability, ethics,
and transparency. Trustworthy predictive models play a crucial role in
supporting healthcare professionals in making informed decisions, improving
patient outcomes, and optimizing healthcare processes. Despite its paramount
importance, the exploration of trustworthy predictive models in healthcare
remains an underexplored domain.

In light of this, we outline the following research directions: (1) _LLM-
driven interpretable model design_. Our prior work Wang and Sun (2022)
demonstrates that LLMs encapsulate medical knowledge and possess the potential
to interpret model outputs in human-understandable lay language, thereby
enhancing model reliability. (2) _Ethical model design_. Given the diversity
within EHR databases, designing fair and robust models while ensuring patient
privacy is imperative. A promising research direction involves the development
of unbiased and robust predictive models. (3) _Human-in-the-loop learning_.
Existing models primarily rely on data, often neglecting the vital input of
healthcare professionals. A human-in-the-loop approach advocates for the
inclusion of domain expertise, enabling clinicians to provide feedback and
contributing to the trustworthiness of the model.

These research directions collectively pave the way for the development of
predictive models that not only excel in performance but also uphold the
principles of interpretability, ethics, and collaboration with healthcare
professionals, fostering trust and reliability in healthcare applications.

###  5.2 Data Scarcity/Sparsity

Training deep learning models in the healthcare domain often demands a
substantial volume of data. However, as indicated in Table 2, publicly
available healthcare data is often limited, especially for rare diseases or
specific patient populations. Compounding this issue, EHR data frequently
features missing values, leading to substantial data loss during
preprocessing. Furthermore, inherent quality challenges in EHR data, such as
inaccuracies, duplications, or inconsistencies, further exacerbate the
difficulties associated with working with sparse and scarce data.
Consequently, acquiring an ample dataset for model training becomes a
formidable challenge. To address this fundamental challenge, one potential
solution involves the generation of synthetic EHR data using innovative
techniques. Despite recent proposals of models utilizing VAE Das et al.
(2023); Biswal et al. (2021), GAN Armanious et al. (2020); Zhang et al.
(2021); Ma et al. (2020a), diffusion-based models Zhong et al. (2024), and LLM
Wang and Sun (2022), these approaches still grapple with the limitation of
producing realistic EHR data, specifically in simulating the unique
characteristics inherent in EHR datasets.

###  5.3 Pre-training Across Multiple Data Sources

Representation learning is a fundamental task in healthcare, particularly in
scenarios where labeled data is scarce. Unsupervised or self-supervised
learning techniques become essential for accurately capturing health features.
While models like GRAM Choi et al. (2017) and KAME Ma et al. (2018b) have been
proposed to enhance code representations, they often overlook the multimodal
and hierarchical characteristics inherent in EHR data. In our ongoing work, we
are engaged in training a multimodal model named MedHMP Wang et al. (2023) on
the MIMIC-III dataset. However, this model currently covers only a subset of
modalities, leaving others unexplored. To address this limitation and enhance
representation learning, a promising avenue for future research involves
collaborative pre-training with multiple healthcare datasets from distinct
sources. By leveraging shared modalities across these datasets, we can amass
sufficient training data to establish a health-specific pre-trained model.
This approach holds the potential to comprehensively address the challenge of
representation learning across diverse healthcare data sources, accounting for
their multimodal and

###  5.4 Federated Training for Foundation Models

While the transfer of knowledge from a large pre-trained foundation model
significantly enhances performance, this approach remains largely unexplored
in the context of training foundation models for healthcare. The challenges
arise due to the sensitive and private nature of healthcare data, making it
infeasible to centrally train such a model. To address these challenges, a
potential solution involves the utilization of advanced federated learning
techniques. This approach allows for the collaborative training of a
foundation model without necessitating the sharing of stakeholders’ private
data. Federated learning thus presents a promising avenue for efficiently
training healthcare-specific foundation models while respecting privacy
constraints.

Nevertheless, the direct application of federated learning in healthcare
encounters challenges, primarily stemming from the significant heterogeneity
in the distribution of healthcare data, as illustrated in our prior work Wang
et al. (2022). Addressing the substantial imbalance and heterogeneity in EHR
data across clients represents an open challenge, thereby establishing a
promising research direction for the application of federated learning in
healthcare.

## References

  * Armanious et al. [2020] Karim Armanious, Chenming Jiang, Marc Fischer, Thomas Küstner, Tobias Hepp, Konstantin Nikolaou, Sergios Gatidis, and Bin Yang.  Medgan: Medical image translation using gans.  Computerized medical imaging and graphics, 79:101684, 2020.
  * Bai et al. [2018] Tian Bai, Shanshan Zhang, Brian L Egleston, and Slobodan Vucetic.  Interpretable representation learning for healthcare via capturing disease progression through time.  In KDD, pages 43–51, 2018.
  * Baytas et al. [2017] Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiayu Zhou.  Patient subtyping via time-aware lstm networks.  In KDD, pages 65–74, 2017.
  * Biswal et al. [2021] Siddharth Biswal, Soumya Ghosh, Jon Duke, Bradley Malin, Walter Stewart, Cao Xiao, and Jimeng Sun.  Eva: Generating longitudinal electronic health records using conditional variational autoencoders.  In MLHC, pages 260–282, 2021.
  * Chen and Cai [2015] Long-Sheng Chen and Sheng-Jhe Cai.  Neural-network-based resampling method for detecting diabetes mellitus.  JMBE, 35:824–832, 2015.
  * Choi et al. [2016a] Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun.  Doctor ai: Predicting clinical events via recurrent neural networks.  In MLHC, pages 301–318, 2016.
  * Choi et al. [2016b] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart.  Retain: An interpretable predictive model for healthcare using reverse time attention mechanism.  In NeurIPS, pages 3504–3512, 2016.
  * Choi et al. [2017] Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F Stewart, and Jimeng Sun.  Gram: graph-based attention model for healthcare representation learning.  In KDD, pages 787–795, 2017.
  * Cui et al. [2022] Suhan Cui, Jiaqi Wang, Xinning Gui, Ting Wang, and Fenglong Ma.  Automed: Automated medical risk predictive modeling on electronic health records.  In BIBM, pages 948–953, 2022.
  * Cui et al. [2024] Suhan Cui, Jiaqi Wang, Yuan Zhong, Han Liu, Ting Wang, and Fenglong Ma.  Automated fusion of multimodal electronic health records for better medical predictions.  In SDM, 2024.
  * Das et al. [2023] Trisha Das, Zifeng Wang, and Jimeng Sun.  Twin: Personalized clinical trial digital twin generation.  In KDD, pages 402–413, 2023.
  * Feng et al. [2019] Yujuan Feng, Zhenxing Xu, Lin Gan, Ning Chen, Bin Yu, Ting Chen, and Fei Wang.  Dcmn: Double core memory network for patient outcome prediction with multimodal data.  In ICDM, pages 200–209, 2019.
  * Gao et al. [2020] Junyi Gao, Cao Xiao, Yasha Wang, Wen Tang, Lucas M Glass, and Jimeng Sun.  Stagenet: Stage-aware neural networks for health risk prediction.  In WWW, pages 530–540, 2020.
  * Huang and Ma [2021] Guanjie Huang and Fenglong Ma.  Concad: Contrastive learning-based cross attention for sleep apnea detection.  In ECML-PKDD, pages 68–84, 2021.
  * Huang and Ma [2022] Guanjie Huang and Fenglong Ma.  Trustsleepnet: A trustable deep multimodal network for sleep stage classification.  In BHI, pages 01–04, 2022.
  * Huang et al. [2022] Guanjie Huang, Ye Yuan, Guohong Cao, and Fenglong Ma.  Accsleepnet: An axis-aware hybrid deep fusion model for sleep stage classification using wrist-worn accelerometer data.  In BIBM, pages 1005–1012, 2022.
  * Kosolwattana et al. [2023] Tanapol Kosolwattana, Chenang Liu, Renjie Hu, Shizhong Han, Hua Chen, and Ying Lin.  A self-inspected adaptive smote algorithm (sasmote) for highly imbalanced data classification in healthcare.  BioData Mining, 16(1):15, 2023.
  * Kwon et al. [2018] Bum Chul Kwon, Min-Je Choi, Joanne Taery Kim, Edward Choi, Young Bin Kim, Soonwook Kwon, Jimeng Sun, and Jaegul Choo.  Retainvis: Visual analytics with interpretable and interactive recurrent neural networks on electronic medical records.  IEEE TVCG, 25(1):299–309, 2018.
  * Li et al. [2020] Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshidi.  Behrt: transformer for electronic health records.  Scientific reports, 10(1):7155, 2020.
  * Li et al. [2022] Yikuan Li, Mohammad Mamouei, Gholamreza Salimi-Khorshidi, Shishir Rao, Abdelaali Hassaine, Dexter Canoy, Thomas Lukasiewicz, and Kazem Rahimi.  Hi-behrt: Hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records.  IEEE JBHI, 27(2):1106–1117, 2022.
  * Luo et al. [2020] Junyu Luo, Muchao Ye, Cao Xiao, and Fenglong Ma.  Hitanet: Hierarchical time-aware attention networks for risk prediction on electronic health records.  In KDD, pages 647–656, 2020.
  * Lyu et al. [2023] Kewei Lyu, Yu Tian, Yong Shang, Tianshu Zhou, Ziyue Yang, Qianghua Liu, Xi Yao, Ping Zhang, Jianghua Chen, and Jingsong Li.  Causal knowledge graph construction and evaluation for clinical decision support of diabetic nephropathy.  JBI, 139:104298, 2023.
  * Ma et al. [2017] Fenglong Ma, Radha Chitta, Jing Zhou, Quanzeng You, Tong Sun, and Jing Gao.  Dipole: Diagnosis prediction in healthcare via attention-based bidirectional recurrent neural networks.  In KDD, pages 1903–1911, 2017.
  * Ma et al. [2018a] Fenglong Ma, Jing Gao, Qiuling Suo, Quanzeng You, Jing Zhou, and Aidong Zhang.  Risk prediction on electronic health records with prior medical knowledge.  In KDD, pages 1910–1919. ACM, 2018.
  * Ma et al. [2018b] Fenglong Ma, Quanzeng You, Houping Xiao, Radha Chitta, Jing Zhou, and Jing Gao.  Kame: Knowledge-based attention model for diagnosis prediction in healthcare.  In CIKM, pages 743–752, 2018.
  * Ma et al. [2018c] Tengfei Ma, Cao Xiao, and Fei Wang.  Health-atm: A deep architecture for multifaceted patient health record representation and risk prediction.  In SDM, pages 261–269, 2018.
  * Ma et al. [2020a] Fenglong Ma, Yaqing Wang, Jing Gao, Houping Xiao, and Jing Zhou.  Rare disease prediction by generating quality-assured electronic health records.  In SDM, pages 514–522, 2020.
  * Ma et al. [2020b] Liantao Ma, Chaohe Zhang, Yasha Wang, Wenjie Ruan, Jiantao Wang, Wen Tang, Xinyu Ma, Xin Gao, and Junyi Gao.  Concare: Personalized clinical feature embedding via capturing the healthcare context.  In AAAI, volume 34, pages 833–840, 2020.
  * Rasmy et al. [2021] Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi.  Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction.  NPJ digital medicine, 4(1):86, 2021.
  * Sachin et al. [2023] DN Sachin, B Annappa, Sateesh Ambasange, and Alan E Tony.  A multimodal contrastive federated learning for digital healthcare.  SN Computer Science, 4(5):674, 2023.
  * Tan et al. [2022] Yanchao Tan, Carl Yang, Xiangyu Wei, Chaochao Chen, Weiming Liu, Longfei Li, Jun Zhou, and Xiaolin Zheng.  Metacare++: Meta-learning with hierarchical subtyping for cold-start diagnosis prediction in healthcare data.  In SIGIR, pages 449–459, 2022.
  * Tang et al. [2020] Shengpu Tang, Parmida Davarmanesh, Yanmeng Song, Danai Koutra, Michael W Sjoding, and Jenna Wiens.  Democratizing ehr analyses with fiddle: a flexible data-driven preprocessing pipeline for structured clinical data.  JAMIA, 27(12):1921–1934, 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * Wang and Sun [2022] Zifeng Wang and Jimeng Sun.  Promptehr: Conditional electronic healthcare records generation with prompt learning.  In EMNLP, 2022.
  * Wang et al. [2022] Jiaqi Wang, Cheng Qian, Suhan Cui, Lucas Glass, and Fenglong Ma.  Towards federated covid-19 vaccine side effect prediction.  In ECML-PKDD, pages 437–452. Springer, 2022.
  * Wang et al. [2023] Xiaochen Wang, Junyu Luo, Jiaqi Wang, Ziyi Yin, Suhan Cui, Yuan Zhong, Yaqing Wang, and Fenglong Ma.  Hierarchical pretraining on multimodal electronic health records.  In EMNLP, pages 2839–2852, 2023.
  * Xu et al. [2018] Yanbo Xu, Siddharth Biswal, Shriprasad R Deshpande, Kevin O Maher, and Jimeng Sun.  Raim: Recurrent attentive and intensive model of multimodal patient monitoring data.  In KDD, pages 2565–2573, 2018.
  * Xu et al. [2021] Zhen Xu, David R So, and Andrew M Dai.  Mufasa: Multimodal fusion architecture search for electronic health records.  In AAAI, volume 35, pages 10532–10540, 2021.
  * Xu et al. [2022] Yuyang Xu, Haochao Ying, Siyi Qian, Fuzhen Zhuang, Xiao Zhang, Deqing Wang, Jian Wu, and Hui Xiong.  Time-aware context-gated graph attention network for clinical risk prediction.  IEEE TKDE, 2022.
  * Yang and Wu [2021] Bo Yang and Lijun Wu.  How to leverage multimodal ehr data for better medical predictions?  In EMNLP, pages 4029–4038, 2021.
  * Yang et al. [2023a] Kai Yang, Yongxin Xu, Peinie Zou, Hongxin Ding, Junfeng Zhao, Yasha Wang, and Bing Xie.  Kerprint: local-global knowledge graph enhanced diagnosis prediction for retrospective and prospective interpretations.  In AAAI, volume 37, pages 5357–5365, 2023.
  * Yang et al. [2023b] Zhichao Yang, Avijit Mitra, Weisong Liu, Dan Berlowitz, and Hong Yu.  Transformehr: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records.  Nature Communications, 2023.
  * Yao et al. [2023] Zijun Yao, Bin Liu, Fei Wang, Daby Sow, and Ying Li.  Ontology-aware prescription recommendation in treatment pathways using multi-evidence healthcare data.  ACM TIS, 41(4):1–29, 2023.
  * Ye et al. [2020] Muchao Ye, Junyu Luo, Cao Xiao, and Fenglong Ma.  Lsan: Modeling long-term dependencies and short-term correlations with hierarchical attention for risk prediction.  In CIKM, pages 1753–1762, 2020.
  * Ye et al. [2021a] Muchao Ye, Suhan Cui, Yaqing Wang, Junyu Luo, Cao Xiao, and Fenglong Ma.  Medpath: Augmenting health risk prediction via medical knowledge paths.  In WWW, pages 1397–1409, 2021.
  * Ye et al. [2021b] Muchao Ye, Suhan Cui, Yaqing Wang, Junyu Luo, Cao Xiao, and Fenglong Ma.  Medretriever: Target-driven interpretable health risk prediction via retrieving unstructured medical text.  In CIKM, pages 2414–2423, 2021.
  * Zhang et al. [2019] Jue Zhang, Li Chen, Fazeel Abid, et al.  Prediction of breast cancer from imbalance respect using cluster-based undersampling method.  Journal of healthcare engineering, 2019, 2019.
  * Zhang et al. [2021] Ziqi Zhang, Chao Yan, Thomas A Lasko, Jimeng Sun, and Bradley A Malin.  Synteg: a framework for temporal structured electronic health data simulation.  JAMIA, 28(3):596–604, 2021.
  * Zhong et al. [2024] Yuan Zhong, Suhan Cui, Jiaqi Wang, Xiaochen Wang, Ziyi Yin, Yaqing Wang, Houping Xiao, Mengdi Huai, Ting Wang, and Fenglong Ma.  Meddiffusion: Boosting health risk prediction via diffusion-based data augmentation.  In SDM, 2024.