# A Weakly Supervised Adaptive DenseNet for Classifying Thoracic Diseases and
Identifying Abnormalities

Bo Zhou
School of Computer Science
Carnegie Mellon University
Pittsburgh, PA, USA
Yuemeng Li
Department of Bioengineering
University of Pennsylvania
Philadelphia, PA, USA
Jiangcong Wang
Department of Bioengineering
University of Pennsylvania
Philadelphia, PA, USA

###### Abstract

We present a weakly supervised deep learning model for classifying thoracic
diseases and identifying abnormalities in chest radiography. In this work,
instead of learning from medical imaging data with region-level annotations,
our model was merely trained on imaging data with image-level labels to
classify diseases, and is able to identify abnormal image regions
simultaneously. Our model consists of a customized pooling structure and an
adaptive DenseNet front-end, which can effectively recognize possible disease
features for classification and localization tasks. Our method has been
validated on the publicly available ChestX-ray14 dataset. Experimental results
have demonstrated that our classification and localization prediction
performance achieved significant improvement over the previous models on the
ChestX-ray14 dataset. In summary, our network can produce accurate disease
classification and localization, which can potentially support clinical
decisions.

##  1 Introduction

Large scale annotated visual datasets have boosted performance of deep
learning methods on many challenging computer vision problems [1, 2, 3]. Tasks
like object detection, classification, tracking, and segmentation have been
successfully tackled by techniques built on top of these large-scale dataset
with annotations [4, 5, 6]. There are increasing numbers of applications
utilizing deep learning methods in medical imaging analysis over the last
decade [7]. In clinical procedures, visual evidence such as segmentation or
spatial localization of abnormal regions that supports the diagnosis results,
is an vital part of clinical diagnosis. This provides a comprehensive
interpretation of diagnosis results and potentially decreases the false
positive rate.

In this work, we focus on the automatic disease diagnosis and localization in
chest radiography released by [8] named ChestX-ray 14, which is one of the
largest public chest radiography dataset with image-level disease labels and
contains a small subset of region-level disease localizations (bounding
boxes). Our goal is to develop a deep learning scheme capable of both
classifying the disease and localizing the associated lesion sites. Divergent
from standard strongly supervised object detection, our model does not require
ground truth localization annotations during training. Firstly, we adopted and
modified a pre-trained classification CNN as our front-end feature extractor
[9]. The pre-trained front-end encodes the information from a large perceptive
field. Then, after passing through a simple bridging structure, the extracted
feature from the front-end are fed into a customized two-stage pooling network
structure, which produces both classification and associated localization
simultaneously.

Both quantitative and qualitative visual evaluations show that our proposed
model obtains significant improvement over the previous published state-of-
the-art results on disease classification and localization. Visual evaluations
indicate a strong alignment and correspondence between the clinical
annotations and the predicted disease candidate regions shown in figure 1.

Figure 1: Examples of the disease localizations generated from our network.
The localization output from our network (red bounding box), that trained only
with image-level annotation, match with the ground truth localization (green
bounding box).

Contributions: The contributions of this work are two-folds. First, we
developed a weakly supervised end-to-end learning structure that learns from
chest radiography images containing multiple common thoracic diseases by
explicitly searching over possible disease features and locations in the
image. Second, we performed extensive experimental analysis of our model on
the large-scale ChestX-ray14 datasets. With the optimal parameter settings
found from our analysis, we observed that our model (i) can predict accurate
thoracic disease classifications (ii) output better approximate disease
locations than previous methods with only image-level labels available.

##  2 Related Work

CAD for chest radiography: The chest radiography is the most ordered and
common radiological examination for chest diseases. It is a low-cost, low
radiation and fast imaging exam. Recent studies also demonstrated the chest
radiography’s application on detection and evaluation of coronary artery
diseases, which are usually evaluated using Computed Tomography (CT) with
expensive cost and high radiation dose [10, 11, 12]. CAD techniques have been
widely applied in chest radiography for task such as automatic diagnosis and
patient image retrieval. Previously, Bar _et al_. adapted the Decaf-Net for 8
thoracic diseases classification on a relatively small chest x-ray dataset
[13]. Lajhani _et al_. proposed the DCNN for tuberculosis classification task.
Their model ensembled both Alex-Net and Google-Net and achieved impressive
results [14]. Anavi _et al_. has worked on image retrieval in medicine,
specifically for chest radiography given the pathology [15]. All the
aforementioned research showed promising results. However, they only conducted
their experiments on relatively small dataset, ranging from 10 to 500 images.

ChestX-ray14: Recently, National Institute of Health (NIH) released one of the
largest public chest radiography dataset, consisting of 108,948 posterior-
anterior view images from 32,797 patients with eight major chest diseases [8].
A small subset of this dataset is provided with hand labeled bounding boxes
for evaluation. Lately, NIH further expanded this dataset to 112,120 frontal-
view images with 6 additional thoracic diseases, named ChestX-ray14. Several
deep learning methods have been addressed the application of CNN on this
dataset for thoracic disease classification and localization [16, 17, 18].
Wang _et al_. [8] applied a pre-trained Res-Net as the backbone to generate
heatmap as localization, and subsequently used a global maxpooling to obtain
classification. However, most of the localizations generated by their network
mismatched with the ground-truth bounding boxes due to only limited extracted
feature is used. Yao _et al_. [16] used DenseNet [9] to extract features. To
harness the correlation between some of the diseases, they used a LSTM module
to repeatedly decode the feature vector from a DenseNet [9] front end and
produced one disease prediction at each step. They achieved improved results
compared to the baseline in [8]. One of the most recent work from Rajpurkar
_et al_. achieved a good multi-label classification results by fine-tuning a
pre-trained DenseNet-121 [9, 18]. Their classification results outperformed
the previous methods [8, 16, 17]. However, there is no localization component
and weakly supervision in their work. Another most recent work from Li _et
al_. [17] used a pre-trained Res-Net to extract features and divided them into
patches. They passed the extracted patches through a fully-convolutional
classification CNN to obtain a disease probability map which is supervised by
both image-level annotations and limited pixel-level annotations. In the mean
time, a classification score was acquired by multiplying all probability
values together. They achieved significantly better localization results as
compared to the baseline [8] for certain diseases. But this approach requires
certain amount of bounding box training data for improving their model
performance.

Figure 2: The architecture of the adaptive densenet-169 is shown above. The
network receives input of size $224\times 224$, passes it through a
convolution and max pool layer, and then $4$ dense blocks interleaved with $3$
transition blocks, and produces $14\times 14$ feature maps. (a) illustrates
the dense connection in dense block, where output from each conv-bn-relu layer
is concatenated to the input before fed to the next layer. (b) shows the
average pool layer is taken out in the modified transition block, so that it
maintains same spatial resolution as previous dense block’s. (c) illustrates
the kernel dilation in dilated dense block. Pre-trained convolution kernels in
the last dense block were dilated to keep the receptive field unchanged, after
an average pool layer is removed in (b).

Weakly supervised learning: The ChestX-ray14 dataset [8] contains mostly
classification labels but few bounding boxes annotations. Therefore, we seek a
model that is capable of localizing the diseased regions given only image wise
labels. This falls into category of weakly supervise learning (WSL), which
often refers as the task of capturing object location through a customized
deep learning model that trained with only image-level labels. Oquab _et al_.
[19] proposed a WSL scheme. They used a pre-trained CNN to generate class
probability map across spatial locations and applied max-pooling across
spatial locations to get a single binary score vector. With their strategy,
they were able to get both decent classification accuracy and localization mAP
on the benchmark dataset [20, 3]. Recently, Durand _et al_. proposed a more
sophisticated WSL approach [21]. They presented the idea of class pooling
based on the assumption that an object inherently can be decomposed into
different sub-maps to capture better semantic segmentation in WSL. Their
experimental results tested on PASCAL 2007, PASCAL 2012, MS COCO, 15 Scene,
MIT67 [22, 20, 3, 23, 24] indicated the advantage of such design and achieve
significant improvement over [19] ’s model.

##  3 Method

In this work, we aim to use chest radiographs as input with only image-wise
labels to train a model that generates classification for thoracic disease
along with the localization heatmaps. The architecture of our proposed network
can be summarized into three parts: 1) using an Adaptive DenseNet to generate
feature maps and 2) using a bridging layer to convert the feature maps into
class-specific sub-maps 3) using a WSL pooling structure to pool the sub-maps
into class-specific heatmaps and single probability score for each disease.

###  3.1 Adaptive DenseNet

The Adaptive DenseNet inherits the basic structure from DenseNet [9]. The
dense block of DenseNet consists of direct connections from preceding layers
to all subsequent layers which pass image information in deep model’s
training. We maintained the DenseNet structures before the fully connected
classification layers as our basic structure. However, the output feature map
size after the final dense block is greatly decreased ($7\times 7$), losing
too much spatial resolution as compared to the input image ($224\times 224$),
which is not suitable for our localization task. To overcome this, firstly, we
took out the average pooling operation at the third transition layer to
achieve a fine resolution. Secondly, in order to maintain the same receptive
field of each convolution kernel, we dilated all kernels in the fourth dense
block. The Adaptive DenseNet outputs a feature map with spatial size of
$14\times 14$. Details of our Adaptive Densenet is shown in Figure 2. In this
work, we chose the DenseNet-169 as our basic structure. Our adaptation can be
applied to other DenseNet structures as well, such as DenseNet-121 and
Densenet-201. With the Adaptive DenseNet, we were able to acquire a fine
resolution feature map.

Figure 3: Bridging layer connects the output of Adaptive DenseNet and WSL
pooling. Given the Adaptive DenseNet output feature, the bridging layer
performs a 1x1 convolution to transform the output feature into sub-maps. The
output channels of sub-maps are $M\times C$, where $M$ being number of sub-
maps per class and $C$ being number of class. The sub-maps fed into the WSL
pooling layers. Figure 4: Illustration of our 2-stage pooling process: class-
wise pooling and spatial pooling. During class-wise pooling (green region),
multiple class-specific sub-maps are averaged and a single heatmap is
generated for each class. Each classes’ heatmap are used for their
localization. The heatmap is then passed to spatial pooling. We employed two
different spatial pooling schemes during training and testing. During
training, single random score from top $k_{+}$ scores in the heatmap is
selected to increase robustness against erroneous top scores. During testing,
top $k_{+}$ scores and bottom $k_{-}$ scores are weighted averaged with weight
on bottom samples to give maximum performance. The final output of spatial
pooling is a single class score for each class.

###  3.2 Bridging layer

To connect the output of the Adaptive DenseNet and the pooling structure
described in section 3.3, a bridging layer is proposed and used here.
Specifically, the output of the Adaptive DenseNet is transformed by a $1\times
1$ convolutional layer, which learns the mapping from the Adaptive DenseNet’s
output feature maps to class specific sub-maps. Through the bridging layer,
the channel size of the feature map is transferred from $1664$ (channel size
of Adaptive DenseNet’s output) to $M\times C$, as shown in Figure 3. This
bridging layer connects the Adaptive DenseNet and the WSL pooling structure
together to form the overall network.

###  3.3 WSL pooling

To obtain accurate disease classification along with disease localization
heatmap, we integrated a customized weakly supervised learning strategy into
our network. Similar to [21], we implemented two stages of global pooling
layers to summarize all information for each class in feature map outputted
from the Adaptive DenseNet, which include: 1) a class-wise pooling within the
same class, and 2) a spatial pooling along different classes. The class-wise
pooling summarizes submaps into final localization heatmap, and the spatial-
wise pooling summarizes heatmap into classification. Details of the pooling
designs are demonstrated in Figure 4.

class-wise pooling: The class-wise pooling combines the $M$ submaps from each
of the $C$ classes from the bridging layer. In specific, our class-wise
pooling combines the $M$ maps for all disease classes independently through
Equation 1.

$\bar{D}_{i,j}^{c}=\frac{1}{M}\sum\limits_{m=1}^{M}D_{i,j}^{c,m}$ (1)

where $M$ is the number of sub-maps with the assumption that each sub-map
contains different features of the corresponding disease. Unlike traditional
WSL with only one map considering only one extracted feature, our multiple
maps strategy is useful for the disease classification/localization tasks by
utilizing multiple extracted features. $H$, $W$, $C$ are feature map’s height,
width, and the number of class, respectively. The input feature $D$ with size
of $H\times W\times MC$ is divided into $C$ branches with each size of
$H\times W\times M$. Then, the sub-maps information are composed into one
final feature map for each class through an average pooling. This results in a
transformed class heatmap of size $H\times W\times C$.

spatial-wise pooling: Given a class heatmap with size of $H\times W\times C$,
we used a spatial-wise pooling to generate our final classification output.
Our spatial-wise pooling extracts a classification vector ($1\times 1\times
C$) from the class heatmap outputted from class-wise pooling shown above. The
most common strategy for spatial-wise pooling is maxpooling, which might
potentially ignore most of the feature information, generate localization
heatmap only in small regions, and cause insufficient pass of gradient for
training.

Therefore, we proposed to use a customized spatial-wise pooling strategy for
both training and testing of our network, as shown in Figure 4. During the
training phase, we firstly selected top $k$ scores from $H\times W$ heatmaps
and randomly chose one among the k numbers for each class. Because the class
heatmap may not be perfect, the highest score may not correctly correspond to
the disease location. Thus, random sampling from top $k$ spatial locations
instead of simply choosing the max gives the network higher probability to
capture the correct spatial location of the disease and generate the correct
gradient. During the testing phase, we looked at regions with the
highest($D_{+}$)/lowest($D_{-}$) activation from the heatmap $D$. The highest
activation $D_{+}$ indicates presence of classes while the lowest activation
$D_{-}$ indicates absence of classes. Thus both are incorporated to the
prediction layer to achieve a robust classification performance. We introduced
a weighting factor $\alpha$ to control the relative importance between these
two terms.

Formally, let $D^{c}$ be the heatmap for class $c$, let $D^{c}_{+}$/
$D^{c}_{-}$ be the set of top $k_{+}$/bottom $k_{-}$ scores in $D^{c}$, and
let $d^{c}_{final}$ be the final class score. During training phase, we have:

$d^{c}_{final}=dr_{+},$ (2)

where $dr_{+}\in D^{c}_{+}$ is a single element uniformly random sampled from
$D^{c}_{+}$. During testing phase, we have:

$d^{c}_{final}=\frac{1}{k_{+}}\sum\limits_{d_{+}\in
D^{c}_{+}}d_{+}+\alpha(\frac{1}{k_{-}}\sum\limits_{d_{-}\in D^{c}_{-}}d_{-})$
(3)

where $d_{+}$/$d_{-}$ are elements in $D^{c}_{+}$/$D^{c}_{-}$. and $\alpha$ is
the weighting factor added on bottom scores.

###  3.4 Network’s training

For network initialization, we transfered the weights from the pre-trained
models on ImageNet for our Adaptive DenseNet. Then, we randomly initialized
parameters for bridging and WSL pooling layers. We used learning rate of 0.002
with weight decay of 0.1 for every 10 epochs during training. Our model were
implemented with Pytorch (https://pytorch.org/) on a Nvidia GTX 1080Ti. We
added batch normalization [25] after each convolution and additional drop out
of rate of 0.1 for each DenseBlock inside Adaptive DenseNet [26] to prevent
overfitting. Since there is a large data imbalance between each class, we
trained our model with weighted binary cross entropy loss:

$\displaystyle L(X,y)=-w_{+}\cdot ylogp(Y=1|X)-$ (4) $\displaystyle
w_{-}\cdot(1-y)logp(Y=0|X)$

where $w_{+}$ represents percent of positive sample and $w_{-}$ represents
percent of negative sample among all dataset.

###  3.5 Disease classification and localization

The heatmap obtained from our class-wise pooling is further used as a
reference map to generate disease bounding boxes. A thresholding followed by
connected component analysis are applied for each disease class to generate
bounding boxes. Empirically, we used a threshold value of 0.8 for
”Cardiomegaly” and threshold values of 0.9 for the rest of classes to obtain
the best results. The disease classification score is acquired from the
spatial-wise pooling on the heatmap. In short, our network structure is
designed for both disease classification and localization tasks, yet only use
chest radiographs along with image-level annotations for our training.

##  4 Experiments and Results

###  4.1 Data for experiments

We performed our experiment using the ChestX-ray14 dataset. The diseases in
the corresponding chest radiograph are diagnosed by radiologists and the
labeled ground truth are obtained through text mining on the patients’
diagnostic reports. We used exactly same published data split as in [18, 8,
16]. The dataset is divided into training (70%), validation (10%) and testing
(20%). The original resolution of image is $1024\times 1024$ and was
downscaled to $256\times 256$. In order to fit dataset into ImageNet [1] pre-
trained models, we normalized the image by mean and standard deviation of the
images from ImageNet. We used random crop of size $224\times 224$ from the
downscaled image as the network input for training and we used center crop of
same size for testing. For localization validation, the ChestX-ray14 dataset
contains 880 images with 983 disease bounding boxes annotated by board-
certified radiologists. We calculated the intersection over union (IoU)
between our prediction and the ground truth bounding boxes.

###  4.2 Disease classification

Pathology |  Wang [8] |  Yao [16] |  Li [17] |  Rajpurkar [18] |  Ours
---|---|---|---|---|---
Atelectasis |  0.7003 |  0.772 |  0.80 |  0.8094 |  0.8121
Cardiomegaly |  0.8100 |  0.904 |  0.87 |  0.9248 |  0.9066
Effusion |  0.7585 |  0.859 |  0.87 |  0.8638 |  0.8786
Infiltration |  0.6614 |  0.695 |  0.70 |  0.7345 |  0.7065
Mass |  0.6933 |  0.792 |  0.83 |  0.8676 |  0.8354
Nodule |  0.6687 |  0.717 |  0.75 |  0.7802 |  0.7852
Pneumonia |  0.6580 |  0.713 |  0.67 |  0.7680 |  0.7810
Pneumothorax |  0.7993 |  0.841 |  0.87 |  0.8887 |  0.8911
Consolidation |  0.7032 |  0.788 |  0.80 |  0.7901 |  0.8115
Edema |  0.8052 |  0.882 |  0.88 |  0.8878 |  0.8953
Emphysema |  0.8330 |  0.829 |  0.91 |  0.9371 |  0.9373
Fibrosis |  0.7859 |  0.767 |  0.78 |  0.8047 |  0.8187
Pleural Thickening |  0.6835 |  0.765 |  0.79 |  0.8062 |  0.7792
Hernia |  0.8717 |  0.914 |  0.77 |  0.9164 |  0.9487
Table 1: AUROC is used to evaluate model classification performance for 14
diseases. The best results are shown in bold text. In order to obtain a
comprehensive comparison, performance of all previous networks to our
knowledge are included. All results are obtained from their latest update.

We calculated the Area under Receiver Operating Characteristic curve (AUROC)
for each class to evaluate the classification performance of our model. Our
results are selected based on the best performance of classifier for each
class throughout our training iterations. Our classification performance is
compared with other state-of-the-art results [8, 16, 17, 18]. The experiment
is tested with the same dataset splitting as demonstrated in [8, 16, 18].

Figure 5: Examples of localization results on 8 diseases along with ground
truth bounding box annotations. In each pair, a chest radiograph (left) inputs
into our model and the corresponding heatmap (right) is generated. The
heatmaps produced from our model match with the ground-truth bounding boxes
(green) annotated by radiologists and indicated good IoU results (yellow).

In all 14 diseases, our model achieved better classification AUROC than [8,
16] as shown in Table 1. In comparison to [17] with supervision from both
image-level labels and bounding box annotations, our model achieved
significant better classification AUROC in 13 out of 14 diseases. In
particular, our model obtained more than 3% higher AUROC in ”Cardiomegaly”,
”Nodule”, ”Pneumonia”, ”Fibrosis” and ”Hernia”. Comparing our results with
[18]’s work which only focus on classification, our model outperformed theirs
in 10 out of 14 diseases.

In Table 1, our network performed better on diseases classification with large
lesions than one with smaller lesions. ”Cardiomegaly” and ”Emphysema” are the
representative classes for disease with large lesion and our model achieved
classification score over 90% on both. Although our network performed less
accurate on identifying smaller lesions such as mass and nodule, our model is
improved from [8, 16, 17, 18] on these classes as well. Unlike the traditional
maxpooling which only output the maximal response from a network output, our
class-wise pooling can pool multiple response from the submaps which contain
different feature responses of the target lesion. In this case, the
classification of the large lesions that potentially contains more appearance
feature than small lesions can be better improved with our customized pooling
methods.

In addition, we performed hyper-parameter searching for best classification
performance (details shown in Supplemental Materials). In class-wise pooling,
we tested number of sub-maps ($M$) from number 2 to 18 and found 14 sub-maps
yielded best result. We then kept this setting for the rest of experiment. In
spatial pooling, we tested hyper-parameters on two different strategies
separately. During training, we tested our model using top $k_{+}$ scores,
$k_{+}$ ranging from 1 to 20. We found that $k_{+}$ at 10 yielded best
network. During testing, we tested $k_{+}$ from 1 to 20 and $k_{-}$ from 1 to
25. We found that both $k_{+}$ and $k_{-}$ at 15 yielded best classification
result. We tested $\alpha$ ranging from 0.25 to 1 and $\alpha$ equals at 1
gave the best importance balance between $k_{+}$ and $k_{-}$.

###  4.3 Disease localization

T(IoU) |  Model |  Atelectasis |  Cardiomegaly |  Effusion |  Infiltration |  Mass |  Nodule |  Pneumonia |  Pneumothorax
---|---|---|---|---|---|---|---|---|---
0.1 |  Wang |  0.6888 |  0.9383 |  0.6601 |  0.7073 |  0.4000 |  0.1392 |  0.6333 |  0.3775
|  Ours |  0.4113 |  0.9659 |  0.5950 |  0.8134 |  0.5292 |  0.0758 |  0.7421 |  0.3267
|  Ours* |  0.4283 |  0.9657 |  0.6926 |  0.8211 |  0.5539 |  0.2156 |  0.7752 |  0.3263
0.2 |  Wang |  0.4722 |  0.6849 |  0.4509 |  0.4796 |  0.2588 |  0.0506 |  0.35 |  0.2346
|  Ours |  0.2388 |  0.9453 |  0.3341 |  0.6745 |  0.3058 |  0.0125 |  0.5752 |  0.2043
|  Ours* |  0.2557 |  0.9453 |  0.5036 |  0.6833 |  0.3292 |  0.1522 |  0.6331 |  0.2052
0.3 |  Wang |  0.2444 |  0.4589 |  0.3006 |  0.2764 |  0.1529 |  0.0379 |  0.1666 |  0.1326
|  Ours |  0.1458 |  0.9316 |  0.1448 |  0.5223 |  0.1531 |  0.0031 |  0.4420 |  0.1432
|  Ours* |  0.1613 |  0.9246 |  0.3334 |  0.5368 |  0.1768 |  0.1393 |  0.5010 |  0.1432
0.4 |  Wang |  0.0944 |  0.2808 |  0.2026 |  0.1219 |  0.0705 |  0.0126 |  0.075 |  0.0714
|  Ours |  0.0887 |  0.8772 |  0.0590 |  0.3575 |  0.1174 |  0.0025 |  0.2582 |  0.1124
|  Ours* |  0.1062 |  0.8769 |  0.2678 |  0.3911 |  0.1430 |  0.1388 |  0.3252 |  0.1123
0.5 |  Wang |  0.0500 |  0.1780 |  0.1111 |  0.0650 |  0.0117 |  0.0126 |  0.0333 |  0.0306
|  Ours |  0.0448 |  0.7806 |  0.0133 |  0.2523 |  0.0592 |  0.00022 |  0.1419 |  0.0514
|  Ours* |  0.0612 |  0.7812 |  0.2286 |  0.2843 |  0.0827 |  0.1394 |  0.2086 |  0.0522
0.6 |  Wang |  0.0222 |  0.0753 |  0.0457 |  0.0243 |  0.0000 |  0.0126 |  0.0166 |  0.0306
|  Ours |  0.0116 |  0.5211 |  0.0000 |  0.1552 |  0.0238 |  0.0000 |  0.0422 |  0.0413
|  Ours* |  0.0280 |  0.5208 |  0.2159 |  0.1873 |  0.0475 |  0.1393 |  0.1081 |  0.0409
0.7 |  Wang |  0.0055 |  0.0273 |  0.0196 |  0.0000 |  0.0000 |  0.0000 |  0.0083 |  0.0204
|  Ours |  0.0063 |  0.2342 |  0.0000 |  0.0742 |  0.0123 |  0.0000 |  0.0165 |  0.0108
|  Ours* |  0.0223 |  0.2332 |  0.2161 |  0.1062 |  0.0358 |  0.1395 |  0.0836 |  0.0104
IoU |  Wang |  0.2254 |  0.3776 |  0.2558 |  0.2392 |  0.1277 |  0.0379 |  0.1833 |  0.1282
|  Ours |  0.1353 |  0.7508 |  0.1637 |  0.4071 |  0.1715 |  0.0134 |  0.3169 |  0.1272
|  Ours* |  0.1519 |  0.7497 |  0.3511 |  0.4300 |  0.1956 |  0.1520 |  0.3764 |  0.1272
Table 2: Comparison of disease localization using IoU, where threshold of IoU
are set to 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7 for evaluation, respectively.
Best results between Wang _et al_.’s and ours are shown in bold. Last row
shows average IoU for all diseases. * means results obtained from same dataset
with expanded annotation from certified radiologist.

We generated our predicted bounding boxes by applying a naive thresholding
($T=0.1\sim 0.7$) on the normalized heatmap with value ranged from 0 to 1 that
obtained from our class-wise pooling layer. Examples are shown in Figure 5. No
localization annotation was used during training process. The heatmaps were
generated by our network with training only from image-level labels. All 880
images with bounding boxes ground truth were used for evaluation.

In Table 2, we compared our results with [8]. Our model achieved significantly
higher IoU score over [8] on diseases with large lesion, such as
”Cardiomegaly” (abnormally large heart). The heatmap is well-fitted with the
ground truth bounding box. Disease such as ”Infiltration” (substance such as
blood infiltrates through vessel into lung) and ”Pneumonia” (inflammation in
lung) that affect large area of lungs manifest as visually more prominent
patterns, which are learned by our network. Moreover, the localization for
small disease region such as ”Atelectasis” (partially collapsed lung) and Mass
(an abnormal lump $>$ 3cm) can also be well captured by our network. Noted
that for ”Effusion” (liquid occupying lung space), our network detects the
disease that covers whole lesion while the ground truth includes some extra
parts such as the shoulder.

We presents 5 ambiguous cases on 5 diseases in Figure 6, which represents bias
in localization annotations that leads to lower IoU score. In the case of
”Effusion”, the annotation outlined the liquid-lung boundary. Instead, our
network includes the full liquid rinsed area. In the cases of ”Infiltration”
and ”Pneumonia” that spread both lungs, the ground truth annotations only
includes single lung, whereas our network captures both lungs. For cases like
”Mass” and ”Nodule”, the ground truth bounding box only highlights one of many
instances of ”Mass” and ”Nodule”, but our localization highlights all
instances.

Observing annotation ambiguity, we attempted to quantify the effect. We
expanded the annotations by having a certified radiologist manually re-labeled
bounding boxes on lesion on 80 ambiguous cases. We evaluated our model with
the expanded annotations and the updated localization IoU is shown in Table 2.
The updated IoU score on expanded annotations is significantly improved for
certain classes, such as ”Effusion”, ”Nodule”, ”Atelectasis”, ”Infiltration”,
”Mass”, and ”Pneumonia”.

Figure 6: This shows 5 ambiguous cases on 5 diseases where bounding
annotations are bias. The input image is shown on the right, with two bounding
boxes labels: green is the provided ground truth and red is our localized
bounding boxes. The class-pooling output heatmap is shown on the right. The
red bounding boxes are generated by applying a naive thresholding.

##  5 Discussion

In clinical procedures, visual evidence such as segmentation or spatial
localization of disease lesions, in support of disease classification results,
is a vital part of clinical diagnosis. It provides a comprehensive insight
into the disease and potentially decreases the false positive of diagnosis. In
this work, we proposed a weakly supervised adaptive DenseNet architecture. It
only trains on image-level annotation, yet is able to provide both disease
classification and corresponding visual evidence, making it potentially
valuable in clinical setting.

In our experiment, we specifically looked at 14 different thoracic diseases.
Our network demonstrated its ability to precisely identify disease patterns
which generated accurate disease classification and corresponding heatmaps for
disease localization. In our classification experiments, our network
outperformed the current state-of-the-art method on 10 out of 14 diseases.
Specifically, our network has shown significant classification improvements on
diseases with large lesion such as ”Pneumonia” and ”Emphysema”. Our model also
gave robust classification results on diseases with small lesions such as
”Nodule”. In our localization experiments, our network achieved significant
better localization performance on 5 out of 8 diseases with mean T(IoU)=$0.7$,
as compared to the NIH baseline [8]. Compared to previous methods, the
possible reasons for our network to achieve higher classification and
localization accuracy is two-fold. First of all, the use of a two-stage
pooling (class-wise & spatial-wise) allows the network to capture complex
intra-class variations and highlight different useful lesion features on
multiple sub-maps for one disease class. Secondly, the removal of average
pooling helps to improve spatial resolution and maintain more image details
for the lesion features capturing in the WSL pooling.

We also evaluated the effect of localization label ambiguity. We collaborated
with a certified radiologist to expand the ground truth bounding box
annotations on 80 cases. We found that our localization has significantly
better overlap with the bounding box in the expanded annotation than the one
provided in ChestX-ray14 dataset on certain cases. Examples of the ambiguity
cases are illustrated in our Supplemental Materials with detailed
explanations.

For future work, it is worth noting that the annotations in the ChestX-ray14
dataset is generated by text mining instead of manual annotation, which
guarantees 90% correctness of labeling. In this case, the labels can
potentially be further improved. The model should be able to achieve enhanced
performance if such training data is available. Furthermore, a dataset with
more disease categories included will be favorable as it is closer to real
life clinic settings. Future works include expanding the current ChestX-ray14
dataset with more manual label and more disease categories such as coronary
artery disease [10, 11] for evaluation. Applications of our model on different
medical imaging dataset that targeting multiple diseases should also be
evaluated in the future.

##  6 Conclusion

In summary, we presented a weakly supervised adaptive DenseNet to classify and
localize 14 thoracic diseases by using only image-level annotation during
training. Extensive experiments demonstrated the effectiveness of our network
which achieved both the best classification and localization results among the
previous published state-of-the-art approaches on the ChestX-ray14 dataset.

## References

  * [1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [2] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115(3):211–252, 2015\.
  * [3] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [4] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2980–2988. IEEE, 2017.
  * [5] Ross Girshick.  Fast r-cnn.  arXiv preprint arXiv:1504.08083, 2015.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [7] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen AWM van der Laak, Bram van Ginneken, and Clara I Sánchez.  A survey on deep learning in medical image analysis.  Medical image analysis, 42:60–88, 2017.
  * [8] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3462–3471. IEEE, 2017.
  * [9] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, volume 1, page 3, 2017.
  * [10] Bo Zhou, Di Wen, Katelyn Nye, Robert C Gilkeson, Brendan Eck, David Jordan, and David L Wilson.  Detection and quantification of coronary calcium from dual energy chest x-rays: Phantom feasibility study.  Medical physics, 44(10):5106–5119, 2017.
  * [11] Bo Zhou, Yi Jiang, Di Wen, Robert C Gilkeson, Jun Hou, and David L Wilson.  Visualization of coronary artery calcium in dual energy chest radiography using automatic rib suppression.  In Medical Imaging 2018: Image Processing, volume 10574, page 105740E. International Society for Optics and Photonics, 2018.
  * [12] Di Wen, Katelyn Nye, Bo Zhou, Robert C Gilkeson, Amit Gupta, Shiraz Ranim, Spencer Couturier, and David L Wilson.  Enhanced coronary calcium visualization and detection from dual energy chest x-rays with sliding organ registration.  Computerized Medical Imaging and Graphics, 2018.
  * [13] Yaniv Bar, Idit Diamant, Lior Wolf, Sivan Lieberman, Eli Konen, and Hayit Greenspan.  Chest pathology detection using deep learning with non-medical training.  In Biomedical Imaging (ISBI), 2015 IEEE 12th International Symposium on, pages 294–297. IEEE, 2015.
  * [14] Paras Lakhani and Baskaran Sundaram.  Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks.  Radiology, 284(2):574–582, 2017.
  * [15] Yaron Anavi, Ilya Kogan, Elad Gelbart, Ofer Geva, and Hayit Greenspan.  A comparative study for chest radiograph image retrieval using binary texture and deep learning classification.  In Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE, pages 2940–2943. IEEE, 2015.
  * [16] Li Yao, Eric Poblenz, Dmitry Dagunts, Ben Covington, Devon Bernard, and Kevin Lyman.  Learning to diagnose from scratch by exploiting dependencies among labels.  arXiv preprint arXiv:1710.10501, 2017.
  * [17] Zhe Li, Chong Wang, Mei Han, Yuan Xue, Wei Wei, Li-Jia Li, and Fei-Fei Li.  Thoracic disease identification and localization with limited supervision.  arXiv preprint arXiv:1711.06373, 2017.
  * [18] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  arXiv preprint arXiv:1711.05225, 2017.
  * [19] Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic.  Is object localization for free?-weakly-supervised learning with convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 685–694, 2015.
  * [20] M Everingham, L Van Gool, CKI Williams, J Winn, and A Zisserman.  The pascal visual object classes challenge 2012 (voc2012) results (2012).  In URL http://www. pascal-network. org/challenges/VOC/voc2011/workshop/index. html, 2011.
  * [21] Thibaut Durand, Taylor Mordan, Nicolas Thome, and Matthieu Cord.  Wildcat: Weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), 2017.
  * [22] Mark Everingham, L Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge 2007 (voc 2007) results (2007), 2008.
  * [23] Ariadna Quattoni and Antonio Torralba.  Recognizing indoor scenes.  In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 413–420. IEEE, 2009.
  * [24] Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce.  Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories.  In Computer vision and pattern recognition, 2006 IEEE computer society conference on, volume 2, pages 2169–2178. IEEE, 2006.
  * [25] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [26] Vincent Tinto.  Dropout from higher education: A theoretical synthesis of recent research.  Review of educational research, 45(1):89–125, 1975.


# Representation Learning for Recommender Systems with Application to the
Scientific Literature

Robin Brochier  Université de Lyon, Lyon 2, ERIC EA3083
Digital Scientific Research TechnologyLyonFrance<EMAIL_ADDRESS>
(2019)

###### Abstract.

The scientific literature is a large information network linking various
actors (laboratories, companies, institutions, etc.). The vast amount of data
generated by this network constitutes a dynamic heterogeneous attributed
network (HAN), in which new information is constantly produced and from which
it is increasingly difficult to extract content of interest. In this article,
I present my first thesis works in partnership with an industrial company,
Digital Scientific Research Technology. This later offers a scientific watch
tool, Peerus, addressing various issues, such as the real time recommendation
of newly published papers or the search for active experts to start new
collaborations. To tackle this diversity of applications, a common approach
consists in learning representations of the nodes and attributes of this HAN
and use them as features for a variety of recommendation tasks. However, most
works on attributed network embedding pay too little attention to textual
attributes and do not fully take advantage of recent natural language
processing techniques. Moreover, proposed methods that jointly learn node and
document representations do not provide a way to effectively infer
representations for new documents for which network information is missing,
which happens to be crucial in real time recommender systems. Finally, the
interplay between textual and graph data in text-attributed heterogeneous
networks remains an open research direction.

representation learning; recommender systems; network embedding; scientific
literature

††journalyear: 2019††copyright: iw3c2w3††conference: Companion Proceedings of
the 2019 World Wide Web Conference; May 13–17, 2019; San Francisco, CA,
USA††booktitle: Companion Proceedings of the 2019 World Wide Web Conference
(WWW ’19 Companion), May 13–17, 2019, San Francisco, CA, USA††doi:
10.1145/3308560.3314195††isbn: 978-1-4503-6675-5/19/05††copyright:
rightsretained††ccs: Computing methodologies Unsupervised learning††ccs:
Computing methodologies Artificial intelligence††ccs: Information systems Data
mining

##  1\. Problem

Many applications, which have become everyday tools, offer to search and
filter the vast data sources available on the Web. In particular, there is a
multitude of platforms dealing with scientific literature. From the simple
search engine for scientific articles to the social network for researchers,
all use, as data, the daily publications produced around the world. For the
researcher facing this deluge of information, it has become difficult, if not
impossible, to conduct a regular and exhaustive monitoring of his areas of
expertise. The ongoing research presented in this paper, done in partnership
with an industrial player 111Digital Scientific Research Technology (DSRT) and
its web application Peerus: https://peer.us., deals with the problem of
learning representations in heterogeneous networks of documents applied to the
recommendation of scientific literature in real time.

If the scientific information of a publication is mainly contained in the text
that composes it, rich supplementary information is nested in its metadata.
Thus, networks of co-authors, citations and places of publication of an
article contain important information for the realization of a scientific
recommender system. As such, the scientific literature constitutes a
heterogeneous attributed network (HAN) and since new papers are constantly
published, this HAN grows in real time (Figure 1 shows an hypothetical
example). However, limited information might be observed from the new
aggregated nodes. For example, a newly published paper hasn’t incoming
citation links and a PhD student has a limited number of past co-authorship
links. To face this lack of information, the approach considered in the
proposed research is to focus on learning strong representations of the
attributes, particularly the textual contents of the articles, that can
reflect the partially observable underlying network structure.

123123645PapersAuthors

$\displaystyle A$ $\displaystyle=\begin{pmatrix}0&0&1&1&1&0\\\ 1&1&1&0&0&1\\\
0&0&0&0&0&1\\\ \end{pmatrix}$ $\displaystyle D$
$\displaystyle=\begin{bmatrix}\text{"Learning Framework..."}\\\
\text{"Supervised Machine..."}\\\ \text{"Scholarly Literature..."}\\\
\end{bmatrix}$

Figure 1. Hypothetical example of scientific literature data, here constituted
of 6 authors and 3 papers. The HAN can be noted $G=(V,E,D)$ with $V$ the
nodes, $E$ the edges and $D$ the textual content of the papers. We note A the
biadjacency matrix of the graph $G$.

##  2\. State of the art

The quality and informativeness of data representation greatly influence the
performance of machine learning algorithms. For this reason, a lot of efforts
are devoted to devise new ways of learning representations (Bengio et al.,
2013). In Section 2.1, I describe how the task of learning representations of
nodes, i.e. network embedding, is tightly connected to word embedding. Then,
in Section 2.2, I focus on the interplay between natural language processing
and network embedding. Finally, in Section 2.3, I present recent works that
extend network embedding techniques to HAN.

###  2.1. From Word Embedding to Network Embedding

The distributional hypothesis (Sahlgren, 2008) forms the basis of word
embedding algorithms. This assumes that the distributional similarity of words
correlates strongly with their semantic similarity. In other words, if we
learn a representation of a word that allows us to predict the other words
that occurred in its context, we obtain a representation of its meaning.

Skip-Gram (Mikolov et al., 2013) is an algorithm that builds representations
of words by maximizing the log-likelihood of a multi-set of co-occurring word
pairs. Skip-gram with Negative Sampling is a variation proposed in (Mikolov et
al., 2013) to effectively approach that log-likelihood. This is achieved by
reducing the task to a classification which consists in distinguishing pairs
of words that co-occur with false pairs that do not co-occur. An alternative
approach, GloVe (Pennington et al., 2014), learns representations of words by
factoring a matrix of counts of co-occurrences of the words of a corpus. Its
objective is to minimize the error of reconstruction of the matrix,
considering only the non-zero values of co-occurrence counts.

Even though the distributional hypothesis originated in linguistics and is
naturally leveraged for word embedding, Perozzi et al. establish the
connection with network embedding. To do so, they show that the frequency at
which nodes appear in short random walks follows a power-law distribution,
like the frequency of words in language (Perozzi et al., 2014). They propose
DeepWalk, that consists in applying skip-gram with hierarchical softmax on a
corpus of node sequences, deemed equivalent to sentences, generated with
truncated random walks (Perozzi et al., 2014). For some specific tasks, the
representations learned with DeepWalk offer large performance improvements.
Thus, many subsequent works focus on modifying or extending DeepWalk. Node2vec
replaces random walks with biased random walks, in order to better balance the
exploration-exploitation trade-off, arguing that the added flexibility in
exploring neighborhoods helps learning richer representations (Grover and
Leskovec, 2016). VERSE (Tsitsulin et al., 2018) provides a scalable graph
embedding algorithm by defining a versatile similarity matrix of the nodes and
a learning algorithm using noise-contrastive estimation (Gutmann and
Hyvärinen, 2010), which provably converges to its objective in contrast to
negative sampling.

###  2.2. Natural Language Processing in Networks of Documents

As a special case of attributed networks, graphs of documents bring together
the fields of natural language processing (NLP) and network embedding. A wide
variety of unsupervised learning algorithms to represent words and documents
have been proposed, from the well-known bag-of-word model (Harris, 1954) to
the recently introduced attention-based Transformer (Vaswani et al., 2017)
adapted for unsupervised pre-training (Devlin et al., 2018). However, few
works have fully explored the interplay between NLP techniques and network
embedding.

NetPLSA (Mei et al., 2008) adapts a topic modelling algorithm by regularizing
a statistical topic model with a harmonic regularizer based on a graph
structure. It generates topics that reflect the underlying communities of the
network, providing cleaner topics than regular statistical models.

In (Yang et al., 2015), Yang et al. prove that skip-gram with hierarchical
softmax can be equivalently formulated as a matrix factorization problem. They
then propose Text-Associated DeepWalk (TADW), to deal with networks of
documents. TADW consists in constraining the factorization problem, with a
pre-computed representation of documents via LSA (Deerwester et al., 1990). As
such, each node can be represented as the concatenation of a network
embedding, and a projected textual embedding. CANE (Tu et al., 2017) aims to
improve node representations in a structured corpus by applying a mutual
attention mechanism over the textual contents associated with the vertices of
a graph. Given a connected pair of nodes, the model produces textual
representations for each node contextually to the other node. In this manner,
there are as many representations for a single node as it has neighbors. This
model has the advantage to produce interpretable weights for the words of a
pair of documents, highlighting those that explain the network structure.

###  2.3. Heterogeneous Attributed Networks

Real-world networks are often composed of several types of links and nodes
which are associated with attributes. For example, the scientific literature
is made of articles and authors, with directed citations links between papers,
co-authorship links between authors and articles are associated with their
textual content and their journal of publication. Many works have extended
network embedding to handle heterogeneity and attributes in graphs.

With Metapath2vec (Dong et al., 2017), the authors propose to operate meta-
path based random walks to handle heterogeneous nodes and links. These meta-
path are hand-crafted schemes that guide a random walker over the network to
generate nodes co-occurrences. Using a similar Skip-Gram based objective as
DeepWalk, Metapath2vec achieves significant improvements on multi-class node
classification and node clustering over traditional network embedding
algorithms. (Huang et al., 2017) introduces a Label informed Attributed
Network Embedding (LANE) framework which jointly projects an attributed
network and its labels into a unified embedding space by extracting their
correlations. The mapping of the structural proximities in the attributed
network and labels into an identical embedding space via correlation
projections produces a significant improvement of the embeddings. Some works
go beyond the factorization-based embedding approaches, introducing models
that learn a function to generates embeddings by sampling and aggregating
features from a node’s local neighborhood. GraphSAGE (Hamilton et al., 2017)
makes use of learnable aggregator functions that allow to infer
representations for unseen nodes, given their attributes and links.
(Veličković et al., 2018) adapt recent work on attention mechanism to compute
a node representation by attending to its neighbors.

##  3\. Proposed Approach

The proposed approach is intended to produce a novel model for learning
representations of nodes and documents in a dynamic heterogeneous network with
the goal to compute meaningful recommendations in real-time. The novelty
results in the capacity of the model to infer representations of unseen
documents for which no network information is available, in the same embedding
space as the previously observed nodes. The approach is divided in three
steps:

  1. (1)

design of a first model to learn node representations that can handle textual
attributes. By opposition to TADW, which rely on previously learned LSA
representations, this model should learn word and document embeddings from
scratch to ease the inference for unseen documents. This step should validate
the possibility to learn meaningful text representations from graph
information only.

  2. (2)

improvement of the model by focusing on natural language processing. The model
would be able to predict the similarity of documents in a network, based on
their textual content only. It would deal with more advanced NLP techniques
and further take advantage of the interplay between word and document
representations and the network topology. Compared to CANE, the
representations should be produced only from text information (still using the
network as training supervision) and a strong emphasis on link prediction for
unseen documents should be put.

  3. (3)

integration of the heterogeneity to handle different types of nodes and links.
This would allow to apply the model to a wider variety of tasks, such as user-
item recommendation and expert finding. Handling the diversity of node and
link types shouldn’t rely on hand-crafted meta-path, as proposed in
metapath2vec, but should be learned during the process similarly to GraphSAGE.
At this point, the data provided by DSRT from Peerus would serve as a strong
online evaluation of the proposed model.

Step (1) has been achieved and is detailed in Section 4.2. The results are
presented in Section 5.1. More work on its theoretical background will be done
in a near future. Step (2) is ongoing research, that I briefly present in
Section 4.3 and for which I provide some preliminary results in Section 5.3
motivating the research direction. For all evaluations, I detail the datasets
used and the experimental setups in Section 4.1.

##  4\. Methodology

In this Section, I first provide an overview of the evaluations used for my
research, then I detail a contribution corresponding to the first step of my
thesis and I finally briefly address the planned methodology for the next
steps.

###  4.1. Evaluation

I first detail some datasets commonly used in the literature. Then I briefly
present traditional experiments conducted for evaluating network embedding.

####  4.1.1. Datasets

I present below two small datasets, Cora and CiteSeer, according to the
treatments applied in (Sen et al., 2008) as well as a larger dataset, DBLP,
widely used by the scientific community:

  * •

Cora (McCallum et al., 2000) is a network of scientific articles in the field
of artificial learning, grouping 7 classes (scientific subdomains) with 2708
documents, 1433 distinct words in the vocabulary and 5429 citations links.

  * •

CiteSeer (Giles et al., 1998) is a network of scientific articles grouping 6
classes on 3312 documents, 3703 distinct words in the vocabulary and 4732
citations links.

  * •

DBLP (Ley, 2002) is a database of several million scientific articles in the
field of computer science, started in 1993. A powerful disambiguation system
makes it possible to identify the authors (Ley, 2009).

Many other non-scientific datasets that present similar data structures exist.
Among others, Q&A websites and online encyclopedia provide rich sources of
data for which we can tackle similar challenges as with the scientific
literature. Moreover, the industrial player supporting these research provides
a large dataset of scientific literature with user log activities that allows
to apply and evaluate the proposed models to online recommendation tasks.

####  4.1.2. Experiments

To evaluate network representation learning models, it is common to use the
nodes embeddings as input space for a linear algorithm to classify the nodes.
For each set of representations produced by a particular algorithm, the
proportion of learning representations is varied from 10% to 50% and the
average prediction accuracy of the classifier is computed over the rest of the
node representations, given a set of ground truth labels. This evaluation was
used for the results in Section 5.1.

An extension of this evaluation consists in observing only from 30% to 70% of
the nodes when learning the representations. Then, the accuracies of
classification are computed on the unobserved nodes, using only their
attributes for prediction. As such, we evaluate the algorithm on its capacity
to infer representation for new unseen documents. This evaluation was used in
Section 5.3. Note that this is different from the inductive prediction
performed by GraphSAGE, which also infers representations for unseen nodes,
but with the knowledge of the actual new links of these new nodes.

Finally, to evaluate the model of step (2), link prediction constitutes a good
evaluation task. Several ways to generate a pair of training/test set exist
(random, temporal). The goal is then to distinguish unseen links from non-
existing ones. The most suited metric for this is the ROC AUC. The same way as
with the previous classification task, it is possible to extend this
evaluation for unobserved new documents by hiding a proportion of the nodes
(and not of the links) during learning.

###  4.2. Document Network Embedding

In this section, I present the first contribution of my thesis (Brochier et
al., 2019), GVNR (Global Vectors for Node Representation), a model to learn
node representations with an extension, GVNR-t, to handle text-associated
nodes. We seek to learn two sets of representations of the nodes
$U\in\mathbb{R}^{n\times d}$ and $V\in\mathbb{R}^{n\times d}$, $n$ being the
number of nodes in the network and $d$ the dimension of the learned
embeddings.

####  4.2.1. Factorization Problem

We formulate a factorization problem on a random-walk based co-occurrence
counts matrix $X$ generated from an input network, measuring the error of
reconstruction only for positive coefficients and a fraction of randomly
sampled zero coefficients:

(1)
$\underset{U,V,b^{U},b^{V}}{\mathrm{argmin}}\sum_{i=1}^{n}\sum_{j=1}^{n}s(x_{ij})\big{(}u_{i}\cdot
v_{j}+b^{U}_{i}+b^{V}_{j}-\log(x_{ij})\big{)}^{2}$

$b^{U}_{i}$ and $b^{V}_{j}$ are two learned biases for the pair of node
embeddings. The function $s$ effectively selects the coefficients considered
for measuring the reconstruction error:

(2) $s(x_{ij})=\begin{cases}1&\text{if }x_{ij}>0,\\\ m_{i}&\text{else, with
}m_{i}\sim\text{Bernoulli}\Big{(}\frac{k\times
n_{i}}{n-n_{i}}\Big{)}.\end{cases}$

It takes the value 1 for all positive coefficients of $X$, while for zero
coefficients, its value is given by a Bernoulli random variable, $m_{i}$. It
depends on the number of distinct nodes with which node $i$ co-occurs,
$n_{i}$. We introduce a global hyper-parameter $k\in\mathbb{N^{+}_{*}}$, to
control the proportion of zero coefficients incorporated into the
reconstruction error.

####  4.2.2. Extended Model for Networks of Documents

In this brief section, we show how to extend GVNR to deal with networks where
nodes are short text documents.

Assuming word order is negligible for short documents (such as a scientific
abstract), we can model them as bag-of-words and thus represent a document $j$
with a vector $\text{doc}_{j}\in\mathbb{N^{+}}^{m}$, $m$ being the size of the
vocabulary. We can further assume that the meaning of a short text can be
captured by averaging the representations of its words (Le and Mikolov, 2014).

Therefore, with $W\in\mathbb{R}^{m\times d}$ a word embedding matrix, we
define the context-vector representation of a node in the following way:
$v_{j}=\frac{\text{doc}_{j}~{}W}{|\text{doc}_{j}|_{1}}$

###  4.3. Improving Document Network Embedding

GVNR-t is able to jointly learn word, document and node embeddings in a
network of documents. However, the textual information could highly benefit
from more recent works in the field of NLP. In this direction, the recently
introduced Transformer as shown great promise in learning dependencies between
words for text representation. Besides its low computational complexity and
its strong results achieved on neural machine translation and unsupervised
pre-training for language understanding, its core unit, the Scaled Dot-Product
Attention, provides a good basis for extending GVNR-t.

The Scaled Dot-Product Attention takes as input, a set of keys $K$ and values
$V$, corresponding to projected representations of words in a documents, and a
query $q$, possibly any kind of vector lying in the same space as the keys. As
output, it generates a weighted sum of the values, whose weights are produced
by confronting the keys with the query, following the formula:
$\text{Attention}(q,K,V)=\text{softmax}(\frac{qK^{T}}{\sqrt{d_{k}}})V$,
$d_{k}$ being the dimension of the query and the keys.

My current research focus on exploring the use of this attention mechanism for
mutual attention between pairs of documents in a network. Using pre-trained
word embeddings, I try to find a suitable variation of this unit for
generating sparse weights (hence using other functions than the softmax) and I
explore several ways to build an efficient query $q$ for mutual attention.

##  5\. Results

I first present the results obtained on multi-class classification by GVNR and
its extension with text and then I show some preliminary results indicating
that more emphasis should be set on the representations of the textual content
of nodes in a network. Finally, I show an example of a visualization of the
weights learned by a preliminary model adapting the Scaled Dot-Product
Attention for networks of documents.

###  5.1. Results for GVNR and it Extension With Text

The results presented in Table 1 show the average accuracies for multi-class
classification obtained on Cora. First, we observe that GVNR produces
competitive representations with DeepWalk. Its extension GVNR-t, with the
integration of the textual content of the documents, significantly improves
the quality of the embeddings, achieving even better performances than TADW
which relies however on textual representations more time-consuming, obtained
with latent semantic analysis.

Table 1. Results of multi-class classification on Cora. % of training data | 10% | 20% | 30% | 40% | 50%
---|---|---|---|---|---
DeepWalk | 67.8 | 71.6 | 74.5 | 75.8 | 79.2
GVNR ($x_{\text{min}}=1$) | 69.5 | 72.6 | 75.9 | 78.1 | 80.2
DeepWalk+LSA | 73.8 | 77.9 | 78.4 | 78.1 | 78.1
TADW | 77.1 | 78.8 | 78.2 | 78.8 | 78.6
GVNR-t | 79.3 | 80.7 | 80.8 | 81.4 | 81.1
TADW (text only) | 60.5 | 69.3 | 72.7 | 73.6 | 74.5
GVNR-t (text only) | 74.5 | 76.5 | 78.5 | 78.6 | 79.8

###  5.2. Motivation for a Stronger NLP Component

To gain more insight into the textual representations that are learned, Table
1 shows the accuracies obtained by TADW and GVNR-t with their respective text
components only. We see that GVNR-t has significantly higher accuracies than
TADW, but it is unclear if this is due to an underlying better natural
language understanding. Table 2 shows the results of classification when
predicting on unseen documents. We observe that if GVNR-t is capable of
generalizing on the text attributes of the nodes, TADW relatively fails.
However, the results achieved by GVNR-t are still lower than expected and
motivates the use of more advanced NLP techniques to achieve better
generalization.

Table 2. Unseen documents classification accuracies on Cora. % of training data | 30% | 40% | 50% | 60% | 70%
---|---|---|---|---|---
TADW | 39.7 | 48.9 | 50.2 | 51.5 | 52.2
GVNR-t | 64.3 | 67.7 | 71.2 | 73.6 | 73.8

###  5.3. Mutual Attention for Networks of Documents

My ongoing research is meant to adapt the Scaled Dot-product attention
mechanism for network of documents. The hope is to find a way to effectively
infer weights for the words of the documents that strongly support (i.e
highlight evidence for) the links in the network. Figure 2 shows an example of
generated weights with a first draft of such a model. Interestingly, the model
highlights words related to the field of reinforcement learning in both texts.

Figure 2.  Mutual attention weights for a pair of documents extracted from
Cora.

##  6\. Conclusion and Future Work

Text data and network data are the two most represented information types on
the World Wide Web. Building meaningful representations for both is a crucial
step for the design of efficient recommender systems. Particularly, the ever
growing scientific literature constitute a dynamic heterogeneous text-
attributed network. The interplay between the textual content of scientific
publications and the networks dynamics of the actors of the research brings
strong challenges.

The proposed research aims at discovering an efficient model to represent the
variety of nodes and links in the scientific HAN in a unique representation
space in order to tackle a wide variety of recommendation tasks. The first
works achieved during that research allowed to validate the complementarity of
the two sources of information, text and graph, to learn meaningful
representations. Ongoing research now aims at improving the natural language
understanding component of the model, to truly be able to generate
representations for streams of new documents. A last step will focus on
extending the coverage of the types of nodes and links that the model can
handle and intensively evaluating it on a wide variety of real-world
recommendation tasks.

## References

  * (1)
  * Bengio et al. (2013) Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013\.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ 35, 8 (2013), 1798–1828.
  * Brochier et al. (2019) Robin Brochier, Adrien Guille, and Julien Velcin. 2019\.  Global Vectors for Node Representations. In _Proceedings of the 2019 World Wide Web Conference (WWW ’19)_. International World Wide Web Conferences Steering Committee.
  * Deerwester et al. (1990) Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990\.  Indexing by latent semantic analysis.  _JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE_ 41, 6 (1990), 391–407.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dong et al. (2017) Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. 2017\.  metapath2vec: Scalable representation learning for heterogeneous networks. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, 135–144.
  * Giles et al. (1998) C Lee Giles, Kurt D Bollacker, and Steve Lawrence. 1998\.  CiteSeer: An automatic citation indexing system. In _Proceedings of the third ACM conference on Digital libraries_. ACM, 89–98.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  node2vec: Scalable feature learning for networks. In _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 855–864.
  * Gutmann and Hyvärinen (2010) Michael Gutmann and Aapo Hyvärinen. 2010.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_. 297–304.
  * Hamilton et al. (2017) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017\.  Inductive representation learning on large graphs. In _Advances in Neural Information Processing Systems_. 1024–1034.
  * Harris (1954) Zellig S Harris. 1954\.  Distributional structure.  _Word_ 10, 2-3 (1954), 146–162.
  * Huang et al. (2017) Xiao Huang, Jundong Li, and Xia Hu. 2017.  Label informed attributed network embedding. In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_. ACM, 731–739.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed Representations of Sentences and Documents. In _Proceedings of the 31st International Conference on International Conference on Machine Learning_. 1188–1196.
  * Ley (2002) Michael Ley. 2002\.  The DBLP computer science bibliography: Evolution, research issues, perspectives. In _International symposium on string processing and information retrieval_. Springer, 1–10.
  * Ley (2009) Michael Ley. 2009\.  DBLP: some lessons learned.  _Proceedings of the VLDB Endowment_ 2, 2 (2009), 1493–1500.
  * McCallum et al. (2000) Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. 2000.  Automating the construction of internet portals with machine learning.  _Information Retrieval_ 3, 2 (2000), 127–163.
  * Mei et al. (2008) Qiaozhu Mei, Deng Cai, Duo Zhang, and ChengXiang Zhai. 2008\.  Topic modeling with network regularization. In _Proceedings of the 17th international conference on World Wide Web_. ACM, 101–110.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_. 3111–3119.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014\.  Deepwalk: Online learning of social representations. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 701–710.
  * Sahlgren (2008) Magnus Sahlgren. 2008\.  The distributional hypothesis.  _Italian journal of linguistics_ (2008), 23–53.
  * Sen et al. (2008) Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008\.  Collective classification in network data.  _AI magazine_ 29, 3 (2008), 93.
  * Tsitsulin et al. (2018) Anton Tsitsulin, Davide Mottin, Panagiotis Karras, and Emmanuel Müller. 2018.  VERSE: Versatile Graph Embeddings from Similarity Measures. In _Proceedings of the 2018 World Wide Web Conference on World Wide Web_. International World Wide Web Conferences Steering Committee, 539–548.
  * Tu et al. (2017) Cunchao Tu, Han Liu, Zhiyuan Liu, and Maosong Sun. 2017\.  Cane: Context-aware network embedding for relation modeling. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Vol. 1. 1722–1731.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in Neural Information Processing Systems_. 5998–6008.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks.  _International Conference on Learning Representations_ (2018).  https://openreview.net/forum?id=rJXMpikCZ accepted as poster.
  * Yang et al. (2015) Cheng Yang, Zhiyuan Liu, Deli Zhao, Maosong Sun, and Edward Y Chang. 2015.  Network representation learning with rich text information.. In _IJCAI_. 2111–2117.