###### Abstract

Accurate and fast assessment of resection margins is an essential part of a
dermatopathologist’s clinical routine. In this work, we successfully develop a
deep learning method to assist the pathologists by marking critical regions
that have a high probability of exhibiting pathological features in Whole
Slide Images (WSI). We focus on detecting Basal Cell Carcinoma (BCC) through
semantic segmentation using several models based on the UNet architecture. The
study includes $650$ WSI with $3443$ tissue sections in total. Two clinical
dermatopathologists annotated the data, marking tumor tissues’ exact location
on $100$ WSI. The rest of the data, with ground-truth section-wise labels, is
used to further validate and test the models. We analyze two different
encoders for the first part of the UNet network and two additional training
strategies: a) deep supervision, b) linear combination of decoder outputs, and
obtain some interpretations about what the network’s decoder does in each
case. The best model achieves over $96\text{\,}\%$, accuracy, sensitivity, and
specificity on the test set.

###### keywords:

Digital Pathology; Whole Slide Image; Basal Cell Carcinoma; Skin Cancer; Deep
Learning; UNet

xx 1 5 Received: date; Accepted: date; Published: date Deeply supervised UNet
for semantic segmentation to assist dermatopathological assessment of Basal
Cell Carcinoma (BCC)  Jean Le’Clerc Arrastia 1, Nick Heilenkötter1, Daniel
Otero Baguer 1, Lena Hauberg-Lotte1, Tobias Boskamp2, Sonja Hetzer3, Nicole
Duschner3, Jörg Schaller3, and Peter Maaß 1 J. Le’Clerc Arrastia, N.
Heilenkötter, D. Otero Baguer, L. Hauberg-Lotte, T. Boskamp, S. Hetzer, N.
Duschner, J. Schaller, and P. Maass Correspondence<EMAIL_ADDRESS>
##  1 Introduction

Basal Cell Carcinoma (BCC) is the most common malignant skin cancer with an
increasing incidence by up to $10\text{\,}\%$ a year Wong et al. (2003). It
can be locally destructive and is an essential source of morbidity for
patients, mainly when located on the face. Thus, it must be adequately
treated, despite its slow growth Sahl (1995). Although BCC can be effectively
managed through surgical excision, determining the most suitable surgical
margins is often not trivial. Complete removal of the pathological tissue is
the key to a successful surgical treatment. Initially, the tumor is removed
with a safety margin of surrounding tissue, and it is sent to a laboratory for
analysis. If remaining tumor parts are detected in the margins, further
surgery may be performed. For BCC, approximately $10\text{\,}\%$ of the tumors
recur after the usual removal surgeries Crowson (2006). The resection margins’
microscopic control can reduce the recurrence rate to $1\text{\,}\%$ Miller
(1991); Liersch and Schaller (2014).

As a part of the laboratory routine, each tissue biopsy is cut into several
slices in the center of the probe and close to its edges. The pathologist can
only be sure that the surgery was successful if no tumor is present in the
latter ones. However, this task is time-consuming and error-prone, even for
skilled pathologists. Examination under the microscope is currently the most
common practice. Still, pathology laboratories can use scanners to digitalize
the samples and obtain Whole Slide Images (WSI) to benefit from a Computer-
Aided Diagnostic system. We aim to develop an artificial intelligence method
that can support pathologists in providing fast, reliable, and reproducible
decisions in this context.

The first and most crucial step we do is to automatically highlight where the
tumors are located with the highest probability. That also allows the
pathologist to interpret the computer-based decisions better. To this end, we
obtain a deep learning-based semantic segmentation of a WSI into two classes:
Tumor and Normal, i.e., each pixel of tissue in the image is assigned to one
of these labels. Following, we decide for each section, whether it contains
tumors or not, based on the segmentation, and account for possible model
errors.

###  1.1 Related work

Deep learning has shown great potential to address several problems in
understanding, reconstructing, and reasoning about images. In particular,
convolutional neural network approaches have been actively used for
classification and segmentation tasks in a wide field of applications Litjens
et al. (2017); Chen et al. (2018); Minaee et al. (2020), ranging from robot
vision and understanding to the support of critical medical tasks Ronneberger
et al. (2015); Oktay et al. (2018); Etmann et al. (2019); Behrmann et al.
(2017). The nearly human-expert performance achieved in some medical imaging
applications Esteva et al. (2017); Gulshan et al. (2016) has come to show the
capabilities and potential of these algorithms.

Besides the success of deep learning methods in various histological imaging
tasks Janowczyk and Madabhushi (2016); Iizuka et al. (2020); Kriegsmann et al.
(2020), to the best of our knowledge, there are only a few works Olsen et al.
(2018) on tumor recognition in dermatopathological microscope images. We argue
that the automated diagnostic of microscopic images of the human skin can be
incredibly challenging due to the large variety of relevant data.
Dermatological features like the structure of the extracted tissue depend on
several aspects, e.g., the body part or the patient’s skin type. Therefore,
the pursued deep learning model has to be more robust towards such changes
than models that segment histological images of inner organs. An extensive
database, such as the one we use for this study, is essential for reliable
digital dermatopathology solutions.

Most recent semantic image segmentation methods are based on fully
convolutional networks (FCN) Long et al. (2015). In this work, we use the UNet
Ronneberger et al. (2015) as base architecture, which has been successfully
applied in a large variety of medical image analysis applications Milletari et
al. (2016); Li et al. (2018); Oktay et al. (2018). Recent work has also used a
UNet to segment the epidermis from the skin slice Oskal et al. (2019). While a
previous study Olsen et al. (2018) obtained promising results in diagnosing a
BCC subtype, we focus on the automatic detection of BCC in general, including
several subtypes such as sclerodermiform, nodular, and superficial.

##  2 Methods

###  2.1 Data collection and data parts

The tissue biopsy samples were prepared in the “Dermatopathologie Duisburg”
laboratory using standard protocols. After fixation in formalin and the usual
paraffin embedding, the probe was cut into slices of $\approx$
$3\text{\,}\mathrm{\SIUnitSymbolMicro m}$ thickness, and standard hematoxylin
and eosin (H&E) staining was applied. The samples were digitized using a
“Hamamatsu NanoZoomer S360” scanner with a $20$x lens. In total, this study
includes $650$ WSI annotated in different manners and used for training,
validation, and testing. Two clinical dermatopathologists annotated $100$ WSI,
which were included in the Train and Validation I parts of the data, see Table
1. The annotations contain different interest regions of the tissue:
Tumornest, indicates where tumor islands are; Stroma, highlights supporting
tissue around the tumor islands; and Normal, contains parts of the tissue that
look similar to tumors, such as hair follicles, actinic keratosis, or cysts.
However, only the Tumornest annotations were done exhaustively, whereas the
others were annotated only in some cases. For that reason, we only use
Tumornest annotations for training, and leave the other types for data
balancing. Some annotation examples are shown in Figure 1.

Tumornest Tumornest Normal Stroma Tumornest Normal Tumornest Normal Tumornest Figure 1: Detailed annotations used for training, validation and data balancing. Part | Slides | Detailed annotations | Tumor sections | Normal sections
---|---|---|---|---
Training | $85$ | ✓ | $188$ | $209$
Validation I | $15$ | ✓ | $53$ | $31$
Validation II | $229$ |  | $392$ | $608$
Test | $321$ |  | $1119$ | $843$
Table 1: Distribution of the slides, sections and type of annotations. All
slides contain section-wise annotations with Tumor or Normal labels.

Due to the huge number of WSI, it was not feasible to create detailed
annotations for all the slides. Two parts of the data, namely Validation II
and Test only contain section-wise labels. That means that for each section of
tissue in the WSI, there is a label that indicates whether there is some tumor
inside or not, see Figure 2. To speed up the section-wise annotation process,
we first generated bounding boxes and assigned the label Normal or Tumor
resulting from one of our models. The pathologists then efficiently corrected
these labels by just changing those which were wrong, see Section 3.2. Table 1
contains the amount of WSI for each part of the data and the number of
sections.

Figure 2: Example ground-truth labels (Tumor in red and Normal in green) for
two WSI from the Test data.

###  2.2 Model architecture

The neural network designs we use in this work follow the UNet architecture
Ronneberger et al. (2015), a fully convolutional encoder-decoder network with
skip-connections between the encoder blocks and their symmetric decoder
blocks. We use two different encoders and a standard decoder, similar to the
original one Ronneberger et al. (2015) with only minor changes. The input to
the network is a patch $x\in\mathbb{R}^{512\times 512\times 3}$, and the
output is a segmentation map $\Phi(x)\in[0,1]^{512\times 512\times 2}$. The
segmentation map contains two matrices of size $512\times 512$, which
correspond to the predicted pixel-wise probabilities for each of the two
classes: Tumor and Normal. Since we use the Softmax activation to compute the
segmentation map, this can be seen as only one matrix of size $512\times 512$
with the Tumor probabilities.

####  2.2.1 Encoder

The first encoder follows nearly the same design as the original one
Ronneberger et al. (2015) with $5$ blocks, which successively down-sample the
spatial resolution to increasingly catch higher-level features. Each of the
blocks doubles the number of feature channels and halves the spatial
resolution. It uses two $3\times 3$ convolutions, where the first one uses a
stride of $2$. The second encoder is exactly the convolutional backbone of a
ResNet34 He et al. (2016), as can be observed in Figure 3, and also contains
$5$ blocks.

Both encoders contain an initial block with a $7\times 7$ convolution with 64
filters and a stride of $2$, which decreases the input resolution by half.
That allowed us to effectively enlarge the patches’ size ($512\times 512$) to
incorporate a larger context without a significant increase in computation and
memory consumption.

####  2.2.2 Decoder

The decoder contains an expanding path that seeks to build a segmentation map
from the encoded features, see Figure 3. It has the same number of blocks as
the encoder. Each decoding block duplicates the spatial resolution while
halving the number of feature channels. It performs a bilinear up-sampling and
concatenates the result with the output of its symmetric block in the encoder
(skip-connection). Next, it applies two $3\times 3$ convolutions with the same
number of filters. Unlike the original decoder proposed in Ronneberger et al.
(2015), it has one extra block that does not obtain any skip-connection. The
output of the last decoder block is passed to a $1\times 1$ convolution that
computes the final score-map.

Additionally, at each block of the decoder, we added an extra $1\times 1$
convolution, i.e., a linear combination of the block’s final feature maps, to
produce an intermediate score-map. We used these maps in two settings: a) deep
supervision and b) to merge them through a linear combination to produce the
final output. Except for the $1\times 1$ convolutions, we always use a batch-
normalization layer Ioffe and Szegedy (2015) and a ReLU activation function
right after each convolution in the whole network. Furthermore, we always use
the corresponding padding to keep the same spatial size unchanged by the
convolution’s kernel.

256x25664128x128646464x6412812864x6412812825632x3225625625625625651216x165125126432x3276825638412819264128x128256x256512x51212832Residual
Basic Block with downsampling(3x3 conv → Batch Norm → ReLU) →(3x3 conv, /2 →
Batch Norm →$\displaystyle\oplus$Identity → ReLU)Residual Basic Block(3x3 conv
→ Batch Norm → ReLU) →(3x3 conv → Batch Norm →$\displaystyle\oplus$Identity →
ReLU)Bilinear upsampling(3x3 conv → Batch Norm → ReLU) x27x7 conv, /2 → Batch
Norm → ReLU1x1 conv → softmaxSkip
connectionCopy512x512$\psi_{0}(x)$$\psi_{1}(x)$$\psi_{2}(x)$$\psi_{3}(x)$$\psi_{4}(x)$3216

Figure 3: UNet architecture with a ResNet-34 encoder. The output of the
additional $1\times 1$ convolution after Softmax is shown next to each decoder
block.

###  2.3 Model training

For training the models, we used the Train set, which contains $85$ annotated
WSI. We extracted small patches of size $512\times 512$ at $10$x level of
magnification, on which we performed the semantic segmentation. In almost all
slides, the tumor-free tissue is dominant; therefore, it was necessary to
balance the training data to avoid biases and improve the performance. We did
this based on the amounts of pixels belonging to the $3$ types of annotations:
Tumornest ($T$), Stroma ($S$), and Normal ($N$). The re-sampling was done as
depicted in Table 2. That means that some of the patches were used several
times during a training epoch. That is the case for patches that contain
tumor, are close to tumor areas (Stroma), or contain normal regions which look
similar to tumors (Normal). Additionally, we did extra over-sampling for
patches with high tumor density.

During training, all patches were extensively augmented using: random
rotations, scaling, smoothing, color variations, and elastic deformations to
increase the variety of the data effectively. Additionally, we used the Focal-
Loss with $\gamma=2.0$ Lin et al. (2018), which has a similar effect to down-
weighting the easy examples, to make their contribution to the total loss
smaller. We did not include any re-sampling or augmentation for the validation
dataset.

| Before | After
---|---|---
$T<$0.05\text{\,}\%$$ | 175,771 | 175,771
$T\geq$0.05\text{\,}\%$$ | 9,537 | 30,000
$T\geq$10\text{\,}\%$$ | 5,528 | 10,000
$S\geq$0.05\text{\,}\%$$ | 9,096 | 20,000
$N\geq$0.05\text{\,}\%$$ | 9,458 | 20,000
Total patches | 209,390 | 255,771
Pixel unbalance | 78.48 | 16.81
Table 2: Patches distribution and pixel-wise unbalance before and after re-
sampling the data. The pixel unbalance is computed as the ratio between the
amounts of tumor-free and tumor pixels.

We trained the models using a maximum of $40$ epochs, a batch-size of $64$,
$lr=5\cdot 10^{-4}$, and a scheduler to multiply the learning rate by $0.8$
every $5$ epochs. Optimization was performed with the Adam method Kingma and
Ba (2015) and all computations were done on $4$ NVIDIA GeForce GTX 1080 Ti.

####  2.3.1 Deep supervision

The deep supervision strategy consists of forcing the decoder blocks’ outputs
to yield a meaningful segmentation map. We compare each of the decoder blocks’
output with the corresponding down-sampled version of the target segmentation
map and add the discrepancy to the total loss. This technique was originally
introduced in Lee et al. (2015), for obtaining transparency and robustness of
the features extracted in the middle of the network and helping address the
vanishing gradient problem. In this case, it allows gradient information to
flow back directly from the loss to every block of the decoder. Some recent
works Zhu et al. (2017); Zhou et al. (2018) used a similar idea for training a
UNet.

Let $x\in\mathbb{R}^{512\times 512\times 3}$ be an input patch and its
corresponding target segmentation map ${y\in\\{0,1\\}^{512\times 512\times
2}}$. We define $\psi_{\ell}(x)$ as the output of the $\ell$-block of the
decoder (after Softmax), see Figure 3. The contribution to the loss function
from this single data point $(x,\,y)$ is then defined as

$loss(x,\,y)=\sum_{\ell=0}^{k-1}fl\left(\psi_{\ell}(x),\,\Pi_{\ell}(y)\right),$
(1)

where $k=5$ is the number of decoder blocks,
$\Pi_{\ell}:\,\\{0,1\\}^{512\times 512\times 2}\to\\{0,1\\}^{32\cdot
2^{\ell}\times 32\cdot 2^{\ell}\times 2}$ is a down-sampling operator, and
$fl$ is the Focal-Loss. We also tried using different weights for each of the
scales, but we did not observe any improvement. This strategy does not involve
any change on the previously described architecture; it only needs the
$1\times 1$ convolution and the Softmax at the end of each block of the
decoder. The final output is given by $\Phi(x)=\psi_{k-1}(x)$.

####  2.3.2 Linear merge

The second strategy merges the decoder outputs through a linear combination.
We add a linear layer with weights $w\in\mathbb{R}^{k}$ to the architecture
that computes the final output, i.e.,

$\Phi(x)=\text{Softmax}\left(\sum_{\ell=0}^{k-1}w_{\ell}\cdot\Gamma_{\ell}(\hat{\psi}_{\ell}(x))\right),$
(2)

where $\hat{\psi}_{\ell}(x)$ is the score-map computed at the $\ell$-block of
the decoder, i.e., the same as $\psi_{\ell}(x)$ but without the Softmax
activation, and $\Gamma_{l}:\,\mathbb{R}^{32\cdot 2^{\ell}\times 32\cdot
2^{\ell}\times 2}\to\mathbb{R}^{512\times 512\times 2}$ is a bilinear up-
sampling operator. In this case, the Softmax is only applied after the linear
combination. The weights $w$ are trained together with the other parameters of
the model. Note that the standard model without this strategy is a special
case, since the model could learn to assign $w_{k-1}=1.0$ and $w_{\ell}=0$ for
$0\leq\ell<k$.

###  2.4 Section-wise classification

The final classification task for each section of the WSI (see Figure 2) is to
decide whether it contains tumor or not. First of all, we use the trained
model to generate a heatmap by combining several patches’ predictions and
highlighting the tissue’s parts with the highest probabilities to contain
tumor. We extract the patches so that they cover the whole tissue area and
have at least $256$ pixels ($50\text{\,}\%$) overlapping at every border.
Following, we apply a prediction threshold to obtain a binary mask and find
connected regions. To account for some possible model errors and reduce the
false positive rate, we filter out regions below a certain area threshold. If
any predicted tumor area is left in the section, it is classified as Tumor;
otherwise, it is classified as Normal. The prediction and area thresholds are
selected for each model independently as part of the model selection.

###  2.5 Model selection

We trained our implementation of the original UNet and three other variants
using a ResNet34 encoder. The first alternative does not employ any decoder
strategy (ResNet34-UNet), the second one uses the deep supervision strategy
(ResNet34-UNet + DS), and the third one uses the linear combination of the
decoder outputs (ResNet34-UNet + Linear). During training, we evaluated the
models on the Validation I part of the data. We used the Intersection over
Union (IoU) metric to select the models from the best five training epochs for
each setting. Afterward, we evaluated these models on the section-wise
classification task in the Validation I and Validation II parts of the data to
select the best model together with the prediction and area thresholds. To
this end, we used a grid-search and the $F_{\beta}$ score

$F_{\beta}=\frac{(1+\beta^{2})\cdot\text{precision}\cdot\text{recall}}{\beta^{2}\cdot\text{precision}+\text{recall}},$
(3)

with $\beta=1.5$, to give higher importance to the recall/sensitivity.

The performance of the best models and the selected thresholds are shown in
Table 3. Additionally, Figure 4 presents the outputs of the models and the
resulting masks after applying the corresponding thresholds for some patches
from the Validation I data.

Setting |  | Prediction
---
Threshold
| Tumor-Area
---
Threshold ($\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}^{2}$)
Accuracy | Sensitivity | Specificity | $F_{\beta}$
UNet | 0.45 | 8960 | 0.985 | 1.000 | 0.975 | 0.989
ResNet34-UNet | 0.60 | 3840 | 0.993 | 0.998 | 0.989 | 0.994
ResNet34-UNet + DS | 0.60 | 5120 | 0.994 | 0.996 | 0.992 | 0.993
ResNet34-UNet + Linear | 0.65 | 2560 | 0.996 | 1.000 | 0.994 | 0.997
Table 3: Results on the section-wise classification task for all sections from
the Validation I and Validation II parts of the data. The values correspond to
the best models and selected thresholds.

$x$$y$$\Phi(x)$$\Phi(x)\geq\delta$ ResNet34-UNet \+ DS ResNet34-UNet \+ Linear
ResNet34-UNet UNet
$x$$y$$\Phi(x)$$\Phi(x)\geq\delta$$x$$y$$\Phi(x)$$\Phi(x)\geq\delta$$x$$y$$\Phi(x)$$\Phi(x)\geq\delta$
ResNet34-UNet \+ DS ResNet34-UNet \+ Linear ResNet34-UNet UNet
$x$$y$$\Phi(x)$$\Phi(x)\geq\delta$$x$$y$$\Phi(x)$$\Phi(x)\geq\delta$

Figure 4: Example of inference using the analyzed settings on patches from the
Validation I part of the data.

##  3 Results

Finally, we evaluated the selected models from the validation phase on the
Test data. The results are depicted in Table 4 and Figure 5 shows the number
of wrongly classified sections by each model. The ResNet34-UNet + DS achieved
the best results, obtaining more than $96\text{\,}\%$ overall accuracy,
sensitivity, and specificity. It wrongly classified only $74$ ($30$ FP and
$44$ FN) out of $1962$ sections. Figure 6 shows some correctly classified
sections for different BCC subtypes and the corresponding heatmaps generated
with this model.

UNet ResNet34-UNet ResNet34-UNet + DS ResNet34-UNet + Linear
$0$$50$$100$$150$$133$$33$$30$$25$$33$$51$$44$$59$Misclassified sectionsFalse
PositivesFalse Negatives

Figure 5: Number of wrongly classified sections on the Test part of the data. There are $1962$ sections in total. Setting | Accuracy | Sensitivity | Specificity | $F_{\beta}$
---|---|---|---|---
UNet | 0.915 | 0.971 | 0.842 | 0.945
ResNet34-UNet | 0.957 | 0.954 | 0.961 | 0.959
ResNet34-UNet + DS | 0.962 | 0.961 | 0.964 | 0.964
ResNet34-UNet + Linear | 0.957 | 0.947 | 0.970 | 0.956
Table 4: Results on the section-wise classification task for all sections from
the Test part of the data.

In most false negative cases, the model detects the tumor but without enough
confidence. Due to the selected prediction and area thresholds these sections
are then classified as Normal. On the other hand, false positives are often
due to a hair follicle or other skin structure that is identified as a tumor,
see for example Figure 7.

The ResNet34-UNet + DS was the second-best in the model selection phase. We
argue that the slightly different results on the test set are due to the fact
that the validation data is biased. Initially, the Validation II part was
larger, but we selected challenging slides, based on the pathologists’
feedback, which were then fully annotated and included in the Training part.
All in all, the classification error was reduced from $8.5\text{\,}\%$
(baseline UNet) to $3.8\text{\,}\%$. The baseline UNet, which uses the
standard encoder, has excellent sensitivity but quite a low specificity. In
contrast, the models with the ResNet34 encoder are more balanced and exhibit
better performance.

(a)(b)(c)(d)(g)(h)(e)(f)

Figure 6: Generated heatmaps (ResNet34-UNet + DS) for sections from the Test
part of the data. The images show a variety of BCC subtypes that were part of
the dataset a) sclerodermiform BCC, c) nodular BCC, e) superficial BCC, g) no
tumor. As heatmap b) suggests, the exact segmentation of sclerodermiform BCC
can be quite challenging.

(a)(b)

Figure 7: Generated heatmap (ResNet34-UNet + DS) for part of a section from
the Test data which was wrongly classified as a section with tumor (false
positive). In this case, the model identified a hair-follicle as tumor, as it
can be seen on the highlighted area in (b).

###  3.1 Discussion and interpretability

We now analyze in more detail the models trained with the two additional
strategies, namely ResNet34-UNet + DS and ResNet34-UNet + Linear. In Figure 8,
we show the decoder outputs for some patches from the Validation I data using
these models.

In the deep supervision case, one can observe that, indeed, it is possible to
guide the UNet to produce a meaningful segmentation already from the first
block ($\psi_{0}$) of the decoder, see Figure 8. The difference between
$\psi_{0}(x)$ and $\psi_{4}(x)$ is only the resolution and the number of
details on the borders. We observed that already in $\psi_{0}(x)$, the model
knows where the tumor is if there is any. Therefore, we separately evaluated
the performance of $\psi_{0}$ and the other decoder blocks on the final
classification task, using the same prediction and area threshold selected for
the original model. The results are nearly the same as when using the whole
model, see Table 5; in the case of $\psi_{0}$, the difference is only $5$
sections, whereas in $\psi_{1}$ it is only $1$. Those sections were wrongly
classified because the predicted tumor area was right at the limit.

Moreover, performing inference using the encoder followed by only the
decoder’s first block $\psi_{0}$ is much faster and boosts the heatmap
generation process’s speed. The original model takes approximately
$30\text{\,}\mathrm{s}$ per WSI (several sections) on average on an NVIDIA
GeForce GTX 1080 Ti. In contrast, the reduced one needs approximately only
$20\text{\,}\mathrm{s}$, which represents a reduction of $33\text{\,}\%$ of
the time.

Block | Accuracy | Sensitivity | Specificity | $F_{\beta}$
---|---|---|---|---
$\psi_{0}$ | 0.9597 | 0.959 | 0.961 | 0.962
$\psi_{1}$ | 0.9617 | 0.961 | 0.963 | 0.964
$\psi_{2}$ | 0.9617 | 0.961 | 0.963 | 0.964
$\psi_{3}$ | 0.9622 | 0.961 | 0.964 | 0.964
$\psi_{4}$ | 0.9622 | 0.961 | 0.964 | 0.964
Table 5: Results on the test data for the ResNet34-UNet + DS, where the
heatmaps are generated using the output of each decoder block.

On the other hand, the setting that uses a linear merge of the decoder blocks’
outputs has a fascinating behavior. In this case, the model has more freedom
since we do not guide the decoder blocks to output meaningful segmentations.
We only include the final linear combination in the loss function. In Figure
8, one can observe that in $\psi_{0}(x)$, the model identifies a rough
approximation of the tumor’s location, which is much larger than the final
result. Following, in the subsequent blocks, it creates more details. The
learned weights assigned to each block were
$w=[0.1590,\,-0.1645,\,0.1603,\,-0.2963,\,0.0036]$. Even though the observed
behavior makes sense and offers some interpretability, it seems that, at least
for the final classification task, it does not bring any advantage. As we have
seen, only one block of the decoder seems to be good enough. The linear merge
strategy might be more effective if the aim is to obtain very exact borders.

ResNet34-UNet \+ DS ResNet34-UNet \+ Linear
$\psi_{0}(x)$$\psi_{1}(x)$$\psi_{2}(x)$$\psi_{3}(x)$$\psi_{4}(x)$$\Phi(x)$$x$$y$
ResNet34-UNet \+ DS ResNet34-UNet \+ Linear
$\psi_{0}(x)$$\psi_{1}(x)$$\psi_{2}(x)$$\psi_{3}(x)$$\psi_{4}(x)$$\Phi(x)$$x$$y$
ResNet34-UNet \+ DS ResNet34-UNet \+ Linear
$\psi_{0}(x)$$\psi_{1}(x)$$\psi_{2}(x)$$\psi_{3}(x)$$\psi_{4}(x)$$\Phi(x)$$x$$y$

Figure 8: Decoder outputs for each block of the decoder in the deep
supervision and linear merge settings. The patches belong to the Validation I
part of the data. For the linear merge strategy, the segmentation maps shown
in the figure are after applying the Softmax operation, which we do in this
case only for visualization purposes.

###  3.2 Blind study

The results presented in this work were obtained as part of a blind study. At
the moment of training the section-wise annotations for the Test data did not
exist yet. We created an initial classification based on the ResNet34-UNet
model, and the pathologists checked them and corrected the wrong ones. That
allowed us to obtain the section-wise annotations of the Test data in a very
time-efficient manner. The process was done using our experimental online
platform 111https://digipath-viewer.math.uni-bremen.de/, which was created for
reviewing and visualizing WSI, annotations, and model predictions 222This
platform was also used to obtain some of the detailed annotations for the
Training and Validation I data.. Finally, after the wrong labels were
corrected, we computed the final scores for all our models.

##  4 Conclusions

In this work, we used a UNet architecture with two different encoders and
several training strategies for performing automatic detection of BCC on skin
histological images. Training the network with deep supervision was the
decisive factor for improving the final performance. After trained with this
strategy, each decoder’s block focused on obtaining a segmentation with more
details than the previous one but did not add or remove any tumor content. We
found out that the decoder’s first block is enough for obtaining nearly the
best results on the classification task. That implies a substantial speed
improvement on the forward pass of the network and, therefore, on the heatmap
generation process ($33\text{\,}\%$ time reduction). We performed the final
evaluation on a rather large Test dataset compared to the Training part of the
data. Still, the best model obtained a $96.2\text{\,}\%$ accuracy and similar
sensitivity and specificity on the section-wise classification task. These
results are very promising and show the potential of deep learning methods to
assist dermapatopathological assessment of BCC.

###### Acknowledgements.

Jean Le’Clerc Arrastia and Daniel Otero Baguer acknowledge the support by the
Deutsche Forschungsgemeinschaft (DFG) within the framework of GRK 2224/1
“$\pi^{3}$: Parameter Identification – Analysis, Algorithms, Applications”.
The authors declare no conflict of interest.The funders had no role in the
design of the study; in the collection, analyses, or interpretation of data;
in the writing of the manuscript, or in the decision to publish the results.
References yes

## References

  * Wong et al. (2003) Wong, C.S.M.; Strange, R.C.; Lear, J.T.  Basal cell carcinoma.  BMJ 2003, 327, 794–798.  doi:black10.1136/bmj.327.7418.794.
  * Sahl (1995) Sahl, W.J.  Basal Cell Carcinoma: Influence of tumor size on mortality and morbidity.  International Journal of Dermatology 1995, 34, 319–321.  doi:blackhttps://doi.org/10.1111/j.1365-4362.1995.tb03610.x.
  * Crowson (2006) Crowson, A.N.  Basal cell carcinoma: biology, morphology and clinical implications.  Modern Pathology 2006, 19, S127–S147.  doi:black10.1038/modpathol.3800512.
  * Miller (1991) Miller, S.J.  Biology of basal cell carcinoma (Part I).  Journal of the American Academy of Dermatology 1991, 24, 1–13.  doi:black10.1016/0190-9622(91)70001-i.
  * Liersch and Schaller (2014) Liersch, J.; Schaller, J.  Das Basalzellkarzinom und seine seltenen Formvarianten.  Der Pathologe 2014, 35, 433–442.  doi:black10.1007/s00292-014-1930-2.
  * Litjens et al. (2017) Litjens, G.; Kooi, T.; Bejnordi, B.E.; Setio, A.A.A.; Ciompi, F.; Ghafoorian, M.; van der Laak, J.A.; van Ginneken, B.; Sánchez, C.I.  A survey on deep learning in medical image analysis.  Medical Image Analysis 2017, 42, 60 – 88.  doi:blackhttps://doi.org/10.1016/j.media.2017.07.005.
  * Chen et al. (2018) Chen, L.C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs.  IEEE transactions on pattern analysis and machine intelligence 2018, 40, 834—848.  doi:black10.1109/tpami.2017.2699184.
  * Minaee et al. (2020) Minaee, S.; Boykov, Y.; Porikli, F.; Plaza, A.; Kehtarnavaz, N.; Terzopoulos, D.  Image Segmentation Using Deep Learning: A Survey, 2020, [arXiv:cs.CV/2001.05566].
  * Ronneberger et al. (2015) Ronneberger, O.; Fischer, P.; Brox, T.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015; Navab, N.; Hornegger, J.; Wells, W.M.; Frangi, A.F., Eds.; Springer International Publishing: Cham, 2015; pp. 234–241.
  * Oktay et al. (2018) Oktay, O.; Schlemper, J.; Folgoc, L.L.; Lee, M.; Heinrich, M.; Misawa, K.; Mori, K.; McDonagh, S.; Hammerla, N.Y.; Kainz, B.; others.  Attention u-net: Learning where to look for the pancreas.  arXiv preprint arXiv:1804.03999 2018.
  * Etmann et al. (2019) Etmann, C.; Schmidt, M.; Behrmann, J.; Boskamp, T.; Hauberg-Lotte, L.; Peter, A.; Casadonte, R.; Kriegsmann, J.; Maass, P.  Deep Relevance Regularization: Interpretable and Robust Tumor Typing of Imaging Mass Spectrometry Data, 2019, [arXiv:cs.LG/1912.05459].
  * Behrmann et al. (2017) Behrmann, J.; Etmann, C.; Boskamp, T.; Casadonte, R.; Kriegsmann, J.; Maass, P.  Deep learning for tumor classification in imaging mass spectrometry.  Bioinformatics 2017, 34, 1215–1223, [https://academic.oup.com/bioinformatics/article-pdf/34/7/1215/25119530/btx724.pdf].  doi:black10.1093/bioinformatics/btx724.
  * Esteva et al. (2017) Esteva, A.; Kuprel, B.; Novoa, R.A.; Ko, J.; Swetter, S.M.; Blau, H.M.; Thrun, S.  Dermatologist-level classification of skin cancer with deep neural networks.  nature 2017, 542, 115–118.
  * Gulshan et al. (2016) Gulshan, V.; Peng, L.; Coram, M.; Stumpe, M.C.; Wu, D.; Narayanaswamy, A.; Venugopalan, S.; Widner, K.; Madams, T.; Cuadros, J.; others.  Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.  Jama 2016, 316, 2402–2410.
  * Janowczyk and Madabhushi (2016) Janowczyk, A.; Madabhushi, A.  Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases.  Journal of Pathology Informatics 2016, 7, 29.  doi:black10.4103/2153-3539.186902.
  * Iizuka et al. (2020) Iizuka, O.; Kanavati, F.; Kanavati, F.; Kato, K.; Rambeau, M.; Arihiro, K.; Tsuneki, M.  Deep Learning Models for Histopathological Classification of Gastric and Colonic Epithelial Tumours.  Scientific Reports 2020, 10.  doi:blackhttps://doi.org/10.1038/s41598-020-58467-9.
  * Kriegsmann et al. (2020) Kriegsmann, M.; Haag, C.; Weis, C.A.; Steinbuss, G.; Warth, A.; Zgorzelski, C.; Muley, T.; Winter, H.; Eichhorn, M.E.; Eichhorn, F.; Kriegsmann, J.; Christopoulos, P.; Thomas, M.; Witzens-Harig, M.; Sinn, P.; von Winterfeld, M.; Heussel, C.P.; Herth, F.J.F.; Klauschen, F.; Stenzinger, A.; Kriegsmann, K.  Deep Learning for the Classification of Small-Cell and Non-Small-Cell Lung Cancer.  Cancers 2020, 12.  doi:black10.3390/cancers12061604.
  * Olsen et al. (2018) Olsen, T.G.; Jackson, B.H.; Feeser, T.A.; Kent, M.N.; Moad, J.C.; Krishnamurthy, S.; Lunsford, D.D.; Soans, R.E.  Diagnostic performance of deep learning algorithms applied to three common diagnoses in dermatopathology.  Journal of pathology informatics 2018, 9.
  * Long et al. (2015) Long, J.; Shelhamer, E.; Darrell, T.  Fully convolutional networks for semantic segmentation.  Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
  * Milletari et al. (2016) Milletari, F.; Navab, N.; Ahmadi, S.A.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  2016 fourth international conference on 3D vision (3DV). IEEE, 2016, pp. 565–571.
  * Li et al. (2018) Li, X.; Chen, H.; Qi, X.; Dou, Q.; Fu, C.W.; Heng, P.A.  H-DenseUNet: hybrid densely connected UNet for liver and tumor segmentation from CT volumes.  IEEE transactions on medical imaging 2018, 37, 2663–2674.
  * Oskal et al. (2019) Oskal, K.R.J.; Risdal, M.; Janssen, E.A.M.; Undersrud, E.S.; Gulsrud, T.O.  A U-net based approach to epidermal tissue segmentation in whole slide histopathological images.  SN Applied Sciences 2019, 1, 2523–3971.  doi:blackhttps://doi.org/10.1007/s42452-019-0694-y.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; Sun, J.  Deep residual learning for image recognition.  Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * Ioffe and Szegedy (2015) Ioffe, S.; Szegedy, C.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  International conference on machine learning. PMLR, 2015, pp. 448–456.
  * Lin et al. (2018) Lin, T.Y.; Goyal, P.; Girshick, R.; He, K.; Dollar, P.  Focal Loss for Dense Object Detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence 2018, PP, 1–1.  doi:black10.1109/TPAMI.2018.2858826.
  * Kingma and Ba (2015) Kingma, D.P.; Ba, J.  Adam: A Method for Stochastic Optimization.  3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings; Bengio, Y.; LeCun, Y., Eds., 2015.
  * Lee et al. (2015) Lee, C.Y.; Xie, S.; Gallagher, P.; Zhang, Z.; Tu, Z.  Deeply-Supervised Nets.  Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics; Lebanon, G.; Vishwanathan, S.V.N., Eds.; PMLR: San Diego, California, USA, 2015; Vol. 38, Proceedings of Machine Learning Research, pp. 562–570.
  * Zhu et al. (2017) Zhu, Q.; Du, B.; Turkbey, B.; Choyke, P.L.; Yan, P.  Deeply-supervised CNN for prostate segmentation.  2017 International Joint Conference on Neural Networks (IJCNN), 2017, pp. 178–184.  doi:black10.1109/IJCNN.2017.7965852.
  * Zhou et al. (2018) Zhou, Z.; Rahman Siddiquee, M.M.; Tajbakhsh, N.; Liang, J.  UNet++: A Nested U-Net Architecture for Medical Image Segmentation.  Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support; Stoyanov, D.; Taylor, Z.; Carneiro, G.; Syeda-Mahmood, T.; Martel, A.; Maier-Hein, L.; Tavares, J.M.R.; Bradley, A.; Papa, J.P.; Belagiannis, V.; Nascimento, J.C.; Lu, Z.; Conjeti, S.; Moradi, M.; Greenspan, H.; Madabhushi, A., Eds.; Springer International Publishing: Cham, 2018; pp. 3–11.


Benedek Rozemberczki
The project started when the author was a doctoral student of the Center for Doctoral Training in Data Science at The University of Edinburgh.

United Kingdom

Paul Scherer

University of Cambridge
United Kingdom

Yixuan He

University of Oxford
United Kingdom

George Panagopoulos

École Polytechnique

Alexander Riedel

Ernst-Abbe University for Applied Sciences

Maria Astefanoaei

IT University of Copenhagen

Oliver Kiss

Central European University

Ferenc Beres

ELKH SZTAKI

Guzmán López

Nicolas Collignon

Pedal Me
United Kingdom

Rik Sarkar

The University of Edinburgh
United Kingdom

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Abadi, Barham, Chen, Chen, Davis, Dean, Devin,
Ghemawat, Irving, Isard, et alAbadi et al2016]
authorpersonMartín Abadi, personPaul
Barham, personJianmin Chen, personZhifeng Chen,
personAndy Davis, personJeffrey Dean,
personMatthieu Devin, personSanjay Ghemawat,
personGeoffrey Irving, personMichael Isard,
et al year2016.
Tensorflow: A System for Large-Scale Machine
Learning. In booktitle12th $\{$USENIX$\}$ symposium on
operating systems design and implementation ($\{$OSDI$\}$ 16).

[Anderson, Ward, Barnett, and EnglishAnderson
et al2018]
authorpersonSean Anderson, personEric Ward,
personLewis Barnett, and personPhilippina English.
titlesdmTMB: Spatial and spatiotemporal GLMMs with TMB.

[Bahdanau, Cho, and BengioBahdanau
et al2015]
authorpersonDzmitry Bahdanau,
personKyung Hyun Cho, and personYoshua Bengio.
Neural Machine Translation by Jointly Learning to
Align and Translate. In booktitle3rd International
Conference on Learning Representations, ICLR 2015.

[Bai, Yao, Li, Wang, and WangBai
et al2020]
authorpersonLei Bai, personLina Yao,
personCan Li, personXianzhi Wang, and
personCan Wang. year2020.
Adaptive Graph Convolutional Recurrent Network for
Traffic Forecasting.
journalAdvances in Neural Information Processing
Systems  volume33 (year2020).

[Béres, Kelen, Pálovics, and
BenczúrBéres et al2019]
authorpersonFerenc Béres,
personDomokos M. Kelen, personRóbert
Pálovics, and personAndrás A. Benczúr.
Node Embeddings in Dynamic Graphs.
journalApplied Network Science
volume4, number64 (year2019),

[Béres, Pálovics, Oláh, and
BenczúrBéres et al2018]
authorpersonFerenc Béres,
personRóbert Pálovics, personAnna Oláh,
and personAndrás A. Benczúr.
Temporal Walk Based Centrality Metric for Graph
journalApplied Network Science
volume3, number32 (year2018),

[Bojchevski, Klicpera, Perozzi, Kapoor, Blais,
Rózemberczki, Lukasik, and GünnemannBojchevski
et al2020]
authorpersonAleksandar Bojchevski,
personJohannes Klicpera, personBryan Perozzi,
personAmol Kapoor, personMartin Blais,
personBenedek Rózemberczki, personMichal Lukasik,
and personStephan Günnemann.
Scaling Graph Neural Networks with Approximate
Pagerank. In booktitleProceedings of the 26th ACM SIGKDD
International Conference on Knowledge Discovery & Data Mining.

authorpersonGeorg Brandl.
Sphinx Documentation.
journalURL http://sphinx-doc. org/sphinx. pdf

[Cen, Hou, Wang, Chen, Luo, Yao, Zeng, Guo,
Zhang, Dai, et alCen et al2021]
authorpersonYukuo Cen, personZhenyu Hou,
personYan Wang, personQibin Chen,
personYizhen Luo, personXingcheng Yao,
personAohan Zeng, personShiguang Guo,
personPeng Zhang, personGuohao Dai, et al
CogDL: An Extensive Toolkit for Deep Learning on
journalarXiv preprint arXiv:2103.00959

[Chen, Xu, Wu, and ZhengChen
et al2018]
authorpersonJinyin Chen, personXuanheng Xu,
personYangyang Wu, and personHaibin Zheng.
GC-LSTM: Graph Convolution Embedded LSTM for
Dynamic Link Prediction.
journalarXiv preprint arXiv:1812.04206

[Chen, Li, Li, Lin, Wang, Wang, Xiao, Xu,
Zhang, and ZhangChen et al2015]
authorpersonTianqi Chen, personMu Li,
personYutian Li, personMin Lin,
personNaiyan Wang, personMinjie Wang,
personTianjun Xiao, personBing Xu,
personChiyuan Zhang, and personZheng Zhang.
MXNet: A Flexible and Efficient Machine Learning
Library for Heterogeneous Distributed Systems.
journalarXiv preprint arXiv:1512.01274

[Cho, van Merriënboer, Gulcehre, Bahdanau,
Bougares, Schwenk, and BengioCho et al2014]
authorpersonKyunghyun Cho, personBart van
Merriënboer, personCaglar Gulcehre, personDzmitry
Bahdanau, personFethi Bougares, personHolger
Schwenk, and personYoshua Bengio.
Learning Phrase Representations Using RNN
Encoder–Decoder for Statistical Machine Translation. In
booktitleProceedings of the 2014 Conference on Empirical
Methods in Natural Language Processing (EMNLP).
publisherAssociation for Computational Linguistics,

authorpersonCSIRO's Data61.
titleStellarGraph Machine Learning Library.

[Defferrard, Bresson, and
VandergheynstDefferrard et al2016]
authorpersonMichaël Defferrard,
personXavier Bresson, and personPierre
Vandergheynst. year2016.
Convolutional Neural Networks on Graphs with Fast
Localized Spectral Filtering. In booktitleAdvances in
Neural Information Processing Systems. pages3844–3852.

[Fey and LenssenFey and Lenssen2019]
authorpersonMatthias Fey and personJan E.
Lenssen. year2019.
Fast Graph Representation Learning with PyTorch
Geometric. In booktitleICLR Workshop on Representation
Learning on Graphs and Manifolds.

[Frostig, Johnson, and LearyFrostig
et al2018]
authorpersonRoy Frostig, personMatthew James
Johnson, and personChris Leary.
Compiling Machine Learning Programs via High-Level
journalSystems for Machine Learning

[Gilmer, Schoenholz, Riley, Vinyals, and
DahlGilmer et al2017]
authorpersonJustin Gilmer, personSamuel S
Schoenholz, personPatrick F Riley, personOriol
Vinyals, and personGeorge E Dahl.
Neural Message Passing for Quantum Chemistry. In
booktitleInternational Conference on Machine Learning.
PMLR, pages1263–1272.

[Godwin*, Keck*, Battaglia, Bapst, Kipf, Li,
Stachenfeld, Veličković, and Sanchez-GonzalezGodwin*
et al2020]
authorpersonJonathan Godwin*, personThomas
Keck*, personPeter Battaglia, personVictor Bapst,
personThomas Kipf, personYujia Li,
personKimberly Stachenfeld, personPetar
Veličković, and personAlvaro Sanchez-Gonzalez.
booktitleJraph: A Library for Graph Neural
Networks in Jax.

[Goyal, Chhetri, Mehrabi, Ferrara, and
CanedoGoyal et al2018]
authorpersonPalash Goyal, personSujit Rokka
Chhetri, personNinareh Mehrabi, personEmilio
Ferrara, and personArquimedes Canedo.
DynamicGEM: A Library for Dynamic Graph Embedding
journalarXiv preprint arXiv:1811.10734

[Goyal and FerraraGoyal and
authorpersonPalash Goyal and personEmilio
Ferrara. year[n.d.].
GEM: A Python Package for Graph Embedding
journalJournal of Open Source Software
volume3, number29 (year[n. d.]),

[Grattarola and AlippiGrattarola and
authorpersonDaniele Grattarola and
personCesare Alippi. year2020.
Graph Neural Networks in TensorFlow and Keras with
journalarXiv preprint arXiv:2006.12138

[Guo, Lin, Feng, Song, and WanGuo
et al2019]
authorpersonShengnan Guo, personYoufang
Lin, personNing Feng, personChao Song, and
personHuaiyu Wan. year2019.
Attention Based Spatial-Temporal Graph
Convolutional Networks for Traffic Flow Forecasting. In
booktitleProceedings of the AAAI Conference on Artificial
Intelligence, Vol. volume33. pages922–929.

authorpersonMatthew Hanson.
The Open-Source Software Ecosystem for Leveraging
Public Datasets in Spatio-Temporal Asset Catalogs (STAC). In
booktitleAGU Fall Meeting Abstracts,
Vol. volume2019. pagesIN23B–07.

[Hochreiter and SchmidhuberHochreiter and
authorpersonSepp Hochreiter and
personJürgen Schmidhuber. year1997.
Long Short-Term Memory.
journalNeural computation volume9,
number8 (year1997), pages1735–1780.

authorpersonPetter Holme.
Modern Temporal Network Theory: A Colloquium.
journalThe European Physical Journal B
volume88, number9 (year2015),

[Holme and SaramäkiHolme and
authorpersonPetter Holme and personJari
Saramäki. year2012.
Temporal Networks.
journalPhysics reports volume519,
number3 (year2012), pages97–125.

[Hu, Qian, Fang, Wang, Zhao, Zhang, and XuHu
et al2021]
authorpersonJun Hu, personShengsheng Qian,
personQuan Fang, personYouze Wang,
personQuan Zhao, personHuaiwen Zhang, and
personChangsheng Xu. year2021.
Efficient Graph Deep Learning in TensorFlow with
TF Geometric.
journalarXiv preprint arXiv:2101.11552

[James Bradbury and Roy Frostig and Peter
Hawkins and Matthew James Johnson and Chris Leary and Dougal Maclaurin and
George Necula and Adam Paszke and Jake VanderPlas and Skye Wanderman-Milne
and Qiao ZhangJames Bradbury and Roy Frostig and Peter Hawkins and
Matthew James Johnson and Chris Leary and Dougal Maclaurin and George Necula
and Adam Paszke and Jake VanderPlas and Skye Wanderman-Milne and Qiao
authorpersonJames Bradbury and Roy Frostig and Peter
Hawkins and Matthew James Johnson and Chris Leary and Dougal Maclaurin and
George Necula and Adam Paszke and Jake VanderPlas and Skye Wanderman-Milne
and Qiao Zhang. year2018.
booktitleJAX: Composable Transformations of
Python+NumPy Programs.

[Kingma and BaKingma and Ba2015]
authorpersonDiederik Kingma and
personJimmy Ba. year2015.
Adam: A Method for Stochastic Optimization. In
booktitleProceedings of the 3rd International Conference on
Learning Representations.

[Kipf and WellingKipf and Welling2017]
authorpersonThomas N. Kipf and personMax
Welling. year2017.
Semi-Supervised Classification with Graph
Convolutional Networks. In booktitleInternational
Conference on Learning Representations (ICLR).

[Li, Han, Cheng, Su, Wang, Zhang, and PanLi
et al2019]
authorpersonJia Li, personZhichao Han,
personHong Cheng, personJiao Su,
personPengyun Wang, personJianfeng Zhang, and
personLujia Pan. year2019.
Predicting Path Failure in Time-Evolving Graphs.
In booktitleProceedings of the 25th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining.

[Li, Yu, Shahabi, and LiuLi
et al2018]
authorpersonYaguang Li, personRose Yu,
personCyrus Shahabi, and personYan Liu.
Diffusion Convolutional Recurrent Neural Network:
Data-Driven Traffic Forecasting. In booktitleInternational
Conference on Learning Representations.

[Li, Zemel, Brockschmidt, and TarlowLi
et al2016]
authorpersonYujia Li, personRichard Zemel,
personMarc Brockschmidt, and personDaniel Tarlow.
Gated Graph Sequence Neural Networks. In
booktitleInternational Conference on Learning
Representations (ICLR).

[Liu, Luo, Wang, Xie, Yuan, Gui, Xu, Yu, Zhang,
Liu, Yan, Oztekin, Liu, Zhang, Fu, and JiLiu et al2021]
authorpersonMeng Liu, personYouzhi Luo,
personLimei Wang, personYaochen Xie,
personHao Yuan, personShurui Gui,
personZhao Xu, personHaiyang Yu,
personJingtun Zhang, personYi Liu,
personKeqiang Yan, personBora Oztekin,
personHaoran Liu, personXuan Zhang,
personCong Fu, and personShuiwang Ji.
DIG: A Turnkey Library for Diving into Graph
Deep Learning Research.
journalarXiv preprint arXiv:2103.12608

[Luong, Pham, and ManningLuong
et al2015]
authorpersonMinh-Thang Luong, personHieu
Pham, and personChristopher D Manning.
Effective Approaches to Attention-based Neural
Machine Translation. In booktitleProceedings of the 2015
Conference on Empirical Methods in Natural Language Processing.

[Maynard, Stegemerten, and SchwabMaynard
et al1948]
authorpersonHarold B Maynard, personG J
Stegemerten, and personJohn L Schwab.
booktitleMethods-Time Measurement.

[Nair and HintonNair and Hinton2010]
authorpersonVinod Nair and
personGeoffrey E Hinton. year2010.
Rectified Linear Units Improve Restricted
Boltzmann Machines. In booktitleProceedings of the 27th
International Conference on International Conference on Machine Learning.

[Panagopoulos, Nikolentzos, and
VazirgiannisPanagopoulos et al2021]
authorpersonGeorge Panagopoulos,
personGiannis Nikolentzos, and personMichalis
Vazirgiannis. year2021.
Transfer Graph Neural Networks for Pandemic
Forecasting. In booktitleProceedings of the 35th AAAI
Conference on Artificial Intelligence.

[Pareja, Domeniconi, Chen, Ma, Suzumura,
Kanezashi, Kaler, Schardl, and LeisersonPareja et al2020]
authorpersonAldo Pareja, personGiacomo
Domeniconi, personJie Chen, personTengfei Ma,
personToyotaro Suzumura, personHiroki Kanezashi,
personTim Kaler, personTao B Schardl, and
personCharles E Leiserson. year2020.
EvolveGCN: Evolving Graph Convolutional Networks
for Dynamic Graphs. In booktitleAAAI.

[Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
Killeen, Lin, Gimelshein, Antiga, et alPaszke
et al2019a]
authorpersonAdam Paszke, personSam Gross,
personFrancisco Massa, personAdam Lerer,
personJames Bradbury, personGregory Chanan,
personTrevor Killeen, personZeming Lin,
personNatalia Gimelshein, personLuca Antiga,
et al year2019a.
PyTorch: An Imperative Style, High-Performance
Deep Learning Library. In booktitleAdvances in Neural
Information Processing Systems. pages8024–8035.

[Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
Killeen, Lin, Gimelshein, Antiga, et alPaszke
et al2019b]
authorpersonAdam Paszke, personSam Gross,
personFrancisco Massa, personAdam Lerer,
personJames Bradbury, personGregory Chanan,
personTrevor Killeen, personZeming Lin,
personNatalia Gimelshein, personLuca Antiga,
et al year2019b.
PyTorch: An Imperative Style, High-Performance
Deep Learning Library.
journalAdvances in Neural Information Processing
Systems  volume32 (year2019),

authorpersonEdzer Pebesma.
titlestaRs: Spatiotemporal Arrays: Raster and Vector

[Rey and AnselinRey and Anselin2010]
authorpersonSergio J Rey and personLuc
Anselin. year2010.
PySAL: A Python Library of Spatial Analytical
In booktitleHandbook of Applied Spatial Analysis.
publisherSpringer, pages175–193.

authorpersonEmanuele Rob.
PySTAC: Python library for working with any
SpatioTemporal Asset Catalog (STAC).
journalGitHub repository.

[Rozemberczki, Englert, Kapoor, Blais, and
PerozziRozemberczki et al2020a]
authorpersonBenedek Rozemberczki,
personPeter Englert, personAmol Kapoor,
personMartin Blais, and personBryan Perozzi.
Pathfinder Discovery Networks for Neural Message
journalarXiv preprint arXiv:2010.12878

[Rozemberczki, Kiss, and SarkarRozemberczki
et al2020b]
authorpersonBenedek Rozemberczki,
personOliver Kiss, and personRik Sarkar.
Karate Club: An API Oriented Open-source Python
Framework for Unsupervised Learning on Graphs. In
booktitleProceedings of the 29th ACM International
Conference on Information & Knowledge Management.

[Rozemberczki, Scherer, Kiss, Sarkar, and
FerenciRozemberczki et al2021]
authorpersonBenedek Rozemberczki, personPaul
Scherer, personOliver Kiss, personRik Sarkar, and
personTamas Ferenci. year2021.
titleChickenpox Cases in Hungary: a Benchmark Dataset for
Spatiotemporal Signal Processing with Graph Neural Networks.
[arxiv]2102.08100 [cs.LG]

[Sanders and KandrotSanders and
authorpersonJason Sanders and personEdward
Kandrot. year2010.
booktitleCUDA by Example: An Introduction to
General-Purpose GPU Programming.
publisherAddison-Wesley Professional.

[Scherer and LioScherer and Lio2020]
authorpersonPaul Scherer and personPietro
Lio. year2020.
Learning Distributed Representations of Graphs
with Geo2DR. In booktitleICML Workshop on Graph
Representation Learning and Beyond.

[Schlichtkrull, Kipf, Bloem, Van Den Berg,
Titov, and WellingSchlichtkrull et al2018]
authorpersonMichael Schlichtkrull,
personThomas N Kipf, personPeter Bloem,
personRianne Van Den Berg, personIvan Titov, and
personMax Welling. year2018.
Modeling Relational Data with Graph Convolutional
Networks. In booktitleEuropean Semantic Web
Conference. Springer, pages593–607.

[Seo, Defferrard, Vandergheynst, and
BressonSeo et al2018]
authorpersonYoungjoo Seo, personMichaël
Defferrard, personPierre Vandergheynst, and
personXavier Bresson. year2018.
Structured Sequence Modeling with Graph
Convolutional Recurrent Networks. In
booktitleInternational Conference on Neural Information
Processing. Springer, pages362–373.

[Shi, Zhang, Cheng, and LuShi
et al2019]
authorpersonLei Shi, personYifan Zhang,
personJian Cheng, and personHanqing Lu.
Two-Stream Adaptive Graph Convolutional Networks
for Skeleton-Based Action Recognition. In
booktitleProceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition. pages12026–12035.

[Srivastava, Hinton, Krizhevsky, Sutskever, and
SalakhutdinovSrivastava et al2014]
authorpersonNitish Srivastava,
personGeoffrey Hinton, personAlex Krizhevsky,
personIlya Sutskever, and personRuslan
Salakhutdinov. year2014.
Dropout: A Simple Way to Prevent Neural Networks
from Overfitting.
journalThe Journal of Machine Learning Research
volume15, number1 (year2014),

[Taheri and Berger-WolfTaheri and
authorpersonAynaz Taheri and personTanya
Berger-Wolf. year2019.
Predictive Temporal Embedding of Dynamic Graphs.
In booktitleProceedings of the 2019 IEEE/ACM International
Conference on Advances in Social Networks Analysis and Mining.

[Taheri, Gimpel, and Berger-WolfTaheri
et al2019]
authorpersonAynaz Taheri, personKevin
Gimpel, and personTanya Berger-Wolf.
Learning to Represent the Evolution of Dynamic
Graphs with Recurrent Models. In booktitleCompanion
Proceedings of The 2019 World Wide Web Conference
(seriesWWW ’19). pages301–307.

[Taylor, Harris, Comer, and HarrisTaylor
et al2019]
authorpersonPaul Taylor, personChristopher
Harris, personThompson Comer, and personMark
Harris. year2019.
titleCUDA-Accelerated GIS and Spatiotemporal

[Tu, Yao, Zhang, Cui, Wang, Tian, Zhou, and
YangTu et al2018]
authorpersonCunchao Tu, personYuan Yao,
personZhengyan Zhang, personGanqu Cui,
personHao Wang, personChangxin Tian,
personJie Zhou, and personCheng Yang.
titleOpenNE: An Open Source Toolkit for Network

[Van Der Walt, Colbert, and VaroquauxVan
Der Walt et al2011]
authorpersonStefan Van Der Walt,
personS Chris Colbert, and personGael Varoquaux.
The NumPy Array: a Structure for Efficient
Numerical Computation.
journalComputing in science & engineering
volume13, number2 (year2011),

[Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and PolosukhinVaswani et al2017]
authorpersonAshish Vaswani, personNoam
Shazeer, personNiki Parmar, personJakob Uszkoreit,
personLlion Jones, personAidan N Gomez,
personŁukasz Kaiser, and personIllia
Polosukhin. year2017.
Attention is All You Need. In
booktitleProceedings of the 31st International Conference on
Neural Information Processing Systems. pages6000–6010.

[Whitby, Fecher, and BennightWhitby
et al2017]
authorpersonMichael A Whitby, personRich
Fecher, and personChris Bennight.
GeoWave: Utilizing Distributed Key-Value Stores
for Multidimensional Data. In booktitleInternational
Symposium on Spatial and Temporal Databases. Springer,

[Wu, Pan, Long, Jiang, Chang, and ZhangWu
et al2020]
authorpersonZonghan Wu, personShirui Pan,
personGuodong Long, personJing Jiang,
personXiaojun Chang, and personChengqi Zhang.
Connecting the Dots: Multivariate Time Series
Forecasting with Graph Neural Networks. In
booktitleProceedings of the 26th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining.

authorpersonHongxia Yang.
AliGraph: A Comprehensive Graph Neural Network
Platform. In booktitleProceedings of the 25th ACM SIGKDD
International Conference on Knowledge Discovery & Data Mining.

[Yu, Yin, and ZhuYu et al2018]
authorpersonBing Yu, personHaoteng Yin,
and personZhanxing Zhu. year2018.
Spatio-Temporal Graph Convolutional Networks: a
Deep Learning Framework for Traffic Forecasting. In
booktitleProceedings of the 27th International Joint
Conference on Artificial Intelligence. pages3634–3640.

[Zhang, Bazarevsky, Vakunov, Tkachenka, Sung,
Chang, and GrundmannZhang et al2020]
authorpersonFan Zhang, personValentin
Bazarevsky, personAndrey Vakunov, personAndrei
Tkachenka, personGeorge Sung, personChuo-Ling Chang,
and personMatthias Grundmann. year2020.
titleMediaPipe Hands: On-device Real-time Hand
[arxiv]2006.10214 [cs.CV]

[Zhao, Song, Zhang, Liu, Wang, Lin, Deng, and
LiZhao et al2019]
authorpersonLing Zhao, personYujiao Song,
personChao Zhang, personYu Liu, personPu
Wang, personTao Lin, personMin Deng, and
personHaifeng Li. year2019.
T-GCN: A Temporal Graph Convolutional Network for
Traffic Prediction.
journalIEEE Transactions on Intelligent
Transportation Systems volume21, number9
(year2019), pages3848–3858.

[Zheng, Fan, Wang, and QiZheng
et al2020a]
authorpersonChuanpan Zheng, personXiaoliang
Fan, personCheng Wang, and personJianzhong Qi.
GMAN: A Graph Multi-Attention Network for Traffic
Prediction. In booktitleProceedings of the AAAI Conference
on Artificial Intelligence, Vol. volume34.

[Zheng, Wang, Gan, Zhang, and KarypisZheng
et al2020b]
authorpersonDa Zheng, personMinjie Wang,
personQuan Gan, personZheng Zhang, and
personGeorge Karypis. year2020b.
Learning Graph Neural Networks with Deep Graph
Library. In booktitleCompanion Proceedings of the Web
Conference 2020 (seriesWWW '20).

[Zhu, Song, Zhao, and LiZhu
et al2020]
authorpersonJiawei Zhu, personYujiao Song,
personLing Zhao, and personHaifeng Li.
A3T-GCN: Attention Temporal Graph Convolutional
Network for Traffic Forecasting.
journalarXiv preprint arXiv:2006.11583

[Zimányi, Sakr, and LesuisseZimányi
et al2020]
authorpersonEsteban Zimányi,
personMahmoud Sakr, and personArthur Lesuisse.
MobilityDB: A Mobility Database Based on
PostgreSQL and PostGIS.
journalACM Transactions on Database Systems (TODS)
volume45, number4 (year2020),