# MCUa: Multi-level Context and Uncertainty aware Dynamic Deep Ensemble for
Breast Cancer Histology Image Classification

Zakaria Senousy, Mohammed M. Abdelsamea, Mohamed Medhat Gaber, Moloud Abdar, U
Rajendra Acharya, Abbas Khosravi, and Saeid Nahavandi  Z. Senousy, M.
Abdelsamea, and M. Gaber are with the School of Computing and Digital
Technology, Birmingham City University, Birmingham, UK. M. Abdelsamea is also
with Assiut University, Egypt. M. Gaber is also with the Faculty of Computer
Science and Engineering, Galala University, Egypt (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>mohamed.gaber@bcu.ac.uk).M. Abdar, A. Khosravi, and S. Nahavandi are with
Institute for Intelligent Systems Research and Innovations (IISRI), Deakin
University, Geelong, Australia (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>saeid.nahavandi@deakin.edu.au).U R. Acharya is
with Department of Electronics and Computer Engineering, Ngee Ann Polytechnic,
Singapore & Department of Biomedical Engineering, School of Science and
Technology, SUSS University, Singapore & Department of Biomedical Informatics
and Medical Engineering, Asia University, Taichung, Taiwan (e-mail:
aru@np.edu.sg).

###### Abstract

Breast histology image classification is a crucial step in the early diagnosis
of breast cancer. In breast pathological diagnosis, Convolutional Neural
Networks (CNNs) have demonstrated great success using digitized histology
slides. However, tissue classification is still challenging due to the high
visual variability of the large-sized digitized samples and the lack of
contextual information. In this paper, we propose a novel CNN, called Multi-
level Context and Uncertainty aware (_MCUa_) dynamic deep learning ensemble
model. _MCUa_ model consists of several multi-level context-aware models to
learn the spatial dependency between image patches in a layer-wise fashion. It
exploits the high sensitivity to the multi-level contextual information using
an uncertainty quantification component to accomplish a novel dynamic ensemble
model. _MCUa_ model has achieved a high accuracy of 98.11% on a breast cancer
histology image dataset. Experimental results show the superior effectiveness
of the proposed solution compared to the state-of-the-art histology
classification models.

###### Index Terms:

breast cancer, histology images, convolutional neural networks, context-
awareness, uncertainty quantification.

##  I Introduction

Breast cancer is the driving sort of cancer in women, coming about in 1.68
million modern cases and 522,000 passings in 2012 around the world. It has
been accounted for 25.16% of all cancer cases and 14.71% of cancer-related
passing [1]. Precise determination of breast cancer is pivotal for suitable
treatment and prevention of further progression. A few symptomatic tests have
been utilized, counting physical examination, mammography, magnetic resonance
imaging (MRI), ultrasound, and biopsy. Histology image examination resulted
from biopsy considered as a crucial step for breast cancer diagnosis. In the
diagnosis process, pathologists evaluate the cellular areas of hematoxylin-
eosin (H&E) stained histology images to decide the predominant type of breast
tissues, including normal tissue, benign lesion, in situ carcinoma, and
invasive carcinoma. Histology images are large in size with a complex
morphological structure. Therefore, identifying carcinoma regions based on the
manual investigation conducted by medical professionals is a challenging and
time-consuming process.

Traditionally, histology imaging in clinical practice is focused primarily on
pathologists’ manual qualitative analysis. However, there are three main
issues with such practice. One, there is shortage of pathologists around the
world, especially in developing countries and small hospitals. This scarcity
of resources and unequal allocation is a pressing issue which need to be
addressed. Second, the pathologist’s extensive scientific expertise and long-
term diagnostic experience determine whether the histopathological diagnosis
is accurate or not. This subjectivity may cause in a slew of diagnostic
errors. Finally, pathologists are vulnerable to fatigue and inattention while
reading the complex histology images. In order to address these issues, it is
crucial to establish automated and precise histological image classification
tasks. Thanks to the advancement of computer aid diagnosis (CAD) frameworks
that have made the difference in reducing the workload and improved the
detection accuracy [2].

There are two challenging perspectives in the classification of H&E stained
breast histology images. First, there are colossal intra-class fluctuations
and inter-class likenesses in microscopy images, _e.g._ , the difficult mimics
from benign which has a comparative morphological appearance with carcinoma.
Fig. 1(a) shows benign and carcinoma microscopy images with a similar
morphological structure, in terms of the nuclei distribution. Second, in
histology image analysis, structural and contextual information is usually
lost due to the high resolution of the images. This is due to the fact that
histological image is divided into sections and dealing with only local
representation of image patches makes it difficult to preserve the spatial
dependencies of different image patches. Therefore, learning contextual
information is crucial by integrating important information from different
image parts and hence improving the classification performance. Fig. 1(b)
depicts the shape of image patches used as an input to patch-based deep
convolutional neural network (DCNN) models. Several different feature
engineering [3, 4] and feature learning [5, 6, 7] models have been previously
developed to classify digitized breast histology tissues. Feature Learning
showed great success in addressing numerous issues within the field of digital
pathology, including the above-mentioned challenging problems. As of lately,
deep convolutional neural networks (DCNNs) have been broadly recognized as one
of the foremost capable tools for histology tissue classification. In spite of
their predominance, a single DCNN model has constrained capacity to extract
discriminative features and results in lower classification accuracy [8, 5].
Hence, an ensemble of DCNN models has been developed to memorize the
representation of histology images from distinctive view-points for more
precise classification [9]. However, accommodating contextual information in
the architecture of CNNs is a requirement to cope with the huge size of
histology images [8, 10]. Consequently, ensemble CNNs should allow for the
contextual representation to be learned. Moreover, despite the prevalence of
DCNN models in providing high classification performance and alleviating the
workload encountered by pathologists, a number of histological images might
need assistance in diagnosis by professional medical expertise due to their
complexity. Such images have to be excluded from automated image
classification and to be presented for pathologists for manual investigation.
Consequently, we introduce an uncertainty quantification method which measures
the level of image prediction’s randomness using DCNN models. This approach
aids in the identification of various ambiguous regions which can be
clinically useful. It also helps pathologists and medical practitioners to
prioritize images for annotations.

Figure 1: (a) An example of similar morphological structures between benign
and carcinoma sections. (b) Patches of a high section, which are used by DCNN
models to learn the spatial dependencies information.

In this paper, we propose a novel dynamic ensemble CNN with terming Multi-
level Context and Uncertainty aware (_MCUa_) model111The code is available at
https://github.com/zakariaSenousy/MCUa-Model. for the automated classification
of H&E stained breast histology images. First, we resize input images into two
different scales to capture multi-scale local information. Then we designed
patch feature extractor networks by extracting patches and feed them to pre-
trained fine-tuned DCNNs (_i.e._ , DenseNet-161 and ResNet-152). Unlike the
work conducted in [9], the extracted feature maps are then used by our
context-aware networks to extract multi-level contextual information from
different pattern levels. Finally, a novel uncertainty-aware model ensembling
stage is developed to dynamically select the most certain context-aware models
for the final prediction. To the best of our knowledge, _MCUa_ is the first
attempt to employ uncertainty quantification for ensemble modeling for
histology image classification. We evaluated the performance of our model on
BreAst Cancer Histology Images (BACH) challenge dataset [11], which consists
of 400 high-resolution H&E stained breast histology images and divided into
four categories, namely normal, benign, in situ carcinoma, and invasive
carcinoma. _MCUa_ model alleviates the bias that might be caused during the
traditional workload of histological image analysis by introducing an
automated image classification model which captures the spatial dependencies
among patches of high resolution images. Additionally, it presents a measure
of uncertainty which helps in providing a more robust predictions using a
dynamic ensemble mechanism that improves the diversity of the model by coping
with different network architectures and multi-level contextual information.
This can be achieved by (1) introducing effective pre-trained and fine-tuned
DCNN models which learn to explore hierarchical discriminative features and
differentiate between different class categories and (2) learn spatial
dependencies among image patches to preserve contextual information between
feature maps.

The contributions of this paper are summarized below:

  * •

introduced a multi-scale input and multi-architecture stage for feature
extraction which exploits the granularity in encoding multi-level features and
increase the diversity of the extracted features. Multi-scale and multi-
architecture mechanism helps in capturing different sizes and scales for
nuclei and tissue structures;

  * •

proposed a novel context-aware model to learn multi-scale and multi-level
contextual information by encoding the spatial dependencies among patches in
histology images;

  * •

developed a novel dynamic ensemble strategy by selecting the most certain
models for each particular image based on an uncertainty-aware mechanism. The
proposed mechanism has been designed by measuring the level of randomness of
all models in the ensemble architecture, and consequently a dynamic number of
accurate models is chosen and combined to obtain the final prediction; and

  * •

conducted a thorough experimental study on the BACH image dataset, and
obtained better performance than state-of-the-art computational pathology
models.

The paper is organized as follows. In Section II, we review the related
context-aware methods applied to large-scale and medical image classification
problems, and uncertainty quantification in digital pathology. Section III
discusses in details the architecture of our proposed model. Section IV
describes our experimental results obtained. Finally, Section V discusses our
findings by presenting the summary of our work and introducing few potential
future research directions.

##  II Related Work

In this section, we review the related context-aware classification models
which have been previously developed to cope with large-scale, histology, and
other medical images. It also provides a brief overview of uncertainty
quantification in digital pathology.

###  II-A Context-aware models for large-scale image classification

A few studies have been conducted on building context-aware classification
models for large-scale images [12, 13, 14, 15, 16, 17]. For instance, Dangwei
et al. [12] proposed a multi-scale context-aware network for person re-
identification (reID) to learn salient features over the full body and body
parts. The model has the capability to capture the local contextual
information by stacking multi-scale convolutions in each layer of their
architecture. In [13], an integration-aggregation-update (IAU) block has been
proposed for improving person reID performance. It introduces a spatial-
temporal IAU by combining two different types of contextual information into a
CNN model for target feature learning: a) spatial interactions, to capture
contextual dependencies between different body parts in a single frame, and b)
temporal interactions, to capture contextual dependencies between the same
body parts across all frames. Xingjian et al. [18] proposed a convolutional
long short-term memory (LSTM) for spatial-temporal sequence forecasting to
anticipate long run precipitation escalated in a local region over a
moderately brief period of time. Their model is utilized to construct an end-
to-end trainable model for the precipitation now-casting problem.

A model inspired by Geo-statistics [14] to model spatial uncertainties has
been introduced in a way to compute the labels of mosaic images with context
label agreement using a transition probability model to enforce spatial
properties such as class size and proportions. Zheng et al. [16] introduced a
depth representation for RGB-Depth scene classification based on CNN. Their
CNN framework has dilated convolutions to extend the receptive field and a
spatial pooling to aggregate multi-scale contextual information. A diverse
region-based CNN model [15] has been introduced for hyperspectral image
classification which encoded context-aware representation. This is by merging
the diverse set of special feature representations which led to the CNN
framework yielding spatial-spectral context sensitivity for pixel
classification. Makantasis et al. [19] introduced a CNN framework based on
randomized principal component analysis (PCA) to capture spectral and spatial
information. A contextual deep CNN [20] has been proposed to explore the
contextual interactions by mutually taking advantage of local spatial-spectral
relationships of neighboring pixel vectors within a square window.

###  II-B Context-aware models for histology image classification

In histology image analysis, the importance of learning contextual information
using CNNs has been recently introduced in [8, 10] for the classification of
histology images. These architectures are based on two stacked CNNs. The first
CNN extracts salient features from patches of high-resolution images. The
second CNN, which is stacked on top of the first one extends the context of a
single patch to cover a large tissue region. The results shown from these
studies indicate that contextual information plays a vital part in reducing
anomalies in heterogeneous tissue structures.

Likewise, Huang et al. [21] used a deep fusion network to capture the spatial
relationship among high-resolution histology image patches. This is by
adopting a residual network to learn visual features from cellular-level to
large tissue organization. Consequently, a deep fusion network has been
developed to model the inconsistent construction of distinctive features over
patches and rectify the predictions of the residual network. Also, several
context-aware models [22, 23, 24] have adopted an image down-sampling
mechanism for capturing context information from larger histology images.
Other models used adaptive patch sampling [25] and special patch picking [26]
to incorporate sparse spatial information. However, these strategies are not
competent to capture small regions in high resolution such as tumor cells and
their nearby relevant context. A few researches [27, 28] utilized multi-
resolution approach and developed multi-resolution based classifiers to
extract contextual information. The only problem in these multi-resolution
methods is that they focused on small regions of high-resolution histology
image while the remaining regions are at low resolution to produce the final
inference. In this manner, these methods limit the contextual information of
cellular architecture at a high-resolution level in a histology image. Yan et
al. [29] proposed a hybrid model by integrating convolutional and recurrent
deep neural networks for breast cancer histology image classification. It
considers the short-term and long-term spatial correlations between image
patches using a Bidirectional LSTM network. This is by extracting the feature
representations from image patches of histology image, then feeding the
extracted features into the bidirectional LSTM to preserve the spatial
correlations among feature representations.

###  II-C Context-aware models for other medical images

Fang et al. [30] introduced a lesion-aware CNN for optical coherence
tomography (OCT) image classification by developing a lesion detection network
to produce a delicate attention map. The attention map is then fed to a
classification network to utilize the information from local lesion-related
regions to attain more productive and accurate OCT classification In [31], a
deep learning method has been proposed to detect the intracranial aneurysm in
3D Rotational Angiography (3D-RA) based on a spatial information fusion. They
used 2D image sequences and relied on the morphological differences between
image frames and concatenated consecutive frames of every imaging time series
in a way so as to preserve spatial contextual information. Haarburger et al.
[32] proposed a 3D CNN and a multi-scale curriculum learning technique to
classify malignancy globally using MRI images by generating feature maps that
represent the whole spatial context of the breast.

###  II-D Uncertainty quantification for histology images

As an important initial step to explainable classification and segmentation
models, it is required to measure the uncertainty of the predictions obtained
by machine learning and deep learning methods [33]. A few recently proposed
image segmentation and classification approaches have adopted an uncertainty
quantification component for histology image analysis. For instance, Simon et
al. [34] used a measure of uncertainty in a CNN-based model using an
instability map to highlight zones of equivocalness. Fraz et al. [35] proposed
a framework for micro-vessel segmentation with an uncertainty quantification
component for H&E stained histology images. A calibration approach [36] has
been designed in a way to preserve the overall classification accuracy as well
as improving model calibration. It provides an Expected Calibration Error
(ECE), which is a common metric for quantifying miscalibration. Their approach
can be easily attached to any classification task and showed the ability to
reduce calibration error across different neural network architectures and
datasets.

Mobiny and Singh [37] proposed a Bayesian DenseNet-169 model, which can
activate dropout layers during the testing phase to generate a measure of
uncertainty for skin-lesion images. They investigated how Bayesian deep
learning can help the machine–physician partnership perform better in skin
lesion classification. In another research, Raczkowski et al. [38] proposed an
accurate, reliable and active Bayesian network (ARA-CNN) image classification
framework for classifying histopathological images of colorectal cancer. The
network was designed based on residual network and variational dropout. More
recently, an entropy-based elastic ensemble of DCNN (3E-Net) [39] has been
proposed by introducing an ensemble of image-wise models along with Shannon
entropy as an uncertainty quantification component. Unlike 3E-Net, MCUa has
the ability to (1) capture different size/scale variations of nuclei objects
in histopathological images by introducing multi-scale input and multi-
architecture usage for feature extraction, (2) provide an uncertainty-aware
component based on Monte-Carlo (MC) dropout [40], which generates a predictive
probability distributions instead of a single scalar. This is to integrate
learned information from different versions of a single context-aware model
based on activating dropout layers during multiple forward test passes, which
consequently produces multiple probability distributions of a single input
image. The set of distributions is then used to calculate a high level of
uncertainty measure.

##  III _MCUa_ Model

In this section, we describe our proposed Multi-level Context and uncertainty
aware (_MCUa_) dynamic deep learning ensemble model in details. As illustrated
in Fig. 2, the _MCUa_ model consists of an arbitrary number of multi-level
context-aware models, where each model consists of two components: a) a patch-
wise feature extractor component, to extract the most prominent features from
image patches; and b) a context-aware component, aims at capturing the spatial
dependencies among the extracted patches. _MCUa_ starts by taking the original
image and then resizing the image to $m$ scales to get various and integral
visual features from the multi-scale image feature. A number of patches are
extracted from each image scale to be inserted into a pre-trained feature
extractor. Several salient feature maps are extracted from the pre-trained
feature extractor, which are then inserted to multi-level context-aware
models. Each context-aware model has a certain contextual information level
that can be learned from a group of feature maps. As a final stage, MC-dropout
is applied to each context-aware model to produce a measure of uncertainty.
This is done by applying a number of test passes for each input image through
the context-aware network. Each test pass produces a class probability for the
image, using this information, we calculate the mean and standard deviation to
provide image class label and uncertainty measure, respectively. A dynamic
process of model selection, based on an uncertainty measure value and a pre-
defined threshold, is utilized to pick up the most certain models and then
produce the final class label.

###  III-A Multi-scale feature extraction

Multi-scale image feature extraction is pivotal for having diverse and
complementary visual features in H&E stained breast histopathological
microscopy. To extract multi-scale features, we first resize the original
image to different scales. Then, image patches are extracted from each scale
using a sliding window of size $p_{w}\times p_{h}$ and a stride $s$.
Therefore, the total number of patches extracted from the resized image can be
represented by

$a=\left(1+\left\lfloor\frac{I_{W}-p_{w}}{s}\right\rfloor\right)\times\left(1+\left\lfloor\frac{I_{H}-p_{h}}{s}\right\rfloor\right),$
(1)

where $I_{W}$ and $I_{H}$ are width and height dimensions of the resized
image, respectively.

The images at the different scales are then divided into partially overlapped
patches using different stride values for training and testing data
extraction. This increases the level of locality information and the number of
training patches. Moreover, to increase the diversity of training data and, at
the same time, alleviate the overfitting of DCNN models, several data
augmentation methods have been applied. For instance, each patch has been
transformed using a rotation operation and with/without vertical reflections.
Also, random color perturbations recommended by [28] has been applied to each
patch to alleviate the high visual variability of the patches. The data
augmentation process makes our model learn rotation invariant, reflection
invariant, color invariant features and make pre-processing color
normalization [41].

###  III-B Fine-tuning the backbone networks

The pre-trained DCNN models (namely, ResNet-152 [42] and DenseNet-161 [43])
are fine-tuned to be used as the backbone feature extractors of _MCUa_ model.
We adapted the pre-trained DCNN models to a four-category image classification
problem, by modifying the number of neurons in the last fully-connected layer
from 1000 neurons (where ResNet-152 and DenseNet-161 are pre-trained models on
ImageNet [44]) to only 4 neurons. Consequently, the fine-tuned DCNN models can
take input of microscopy image patches (_i.e._ , augmented versions of the
patches extracted from resized versions of microscopy images) and produce an
output of softmax probabilities belonging to the 4 cases (Normal, Benign, In
situ carcinoma, and Invasive carcinoma) of the BACH dataset.

During the fine-tuning process, we use Adam optimizer [45] to minimize the
categorical cross-entropy loss function, which is defined as

$CE(y,\hat{y})=-\sum_{i}^{C}y_{i}log(\hat{y}_{i}),$ (2)

where $y_{i}$ represents the ground-truth class labels, while $\hat{y}_{i}$
represents the softmax probability for each class $i$ in $C$ (the total number
of classes). Softmax activation is applied to the DCNN model’s predictions of
the last fully connected layer. The activation function $f$ applied to
prediction $q_{i}$ is defined as

$f(q_{i})=\frac{e^{q_{i}}}{\sum_{j}^{C}e^{q_{j}}},$ (3)

where $f(q_{i})$ is the $\hat{y}_{i}$.

Once the training of the pre-trained DCNN models is accomplished, the last
convolutional layer is used to construct our feature space or to extract a
number of feature maps (equivalent to the number of patches in each image).

Figure 2: Overview of _MCUa_ model. Our model takes the original image and
resizes it into multiple scales. For each scale, several patches are extracted
which are then inserted into patch-wise networks (_i.e._ , pre-trained DCNNs)
to extract salient features. The extracted features are then inserted into
multi-level context-aware models to learn spatial dependencies between image
feature maps. Context-aware models work on extracting contextual information
between feature maps based on different levels ($L{1}$ to $L{n}$). $L_{1}$ is
considered as low level context which builds contextual information among two
original feature maps, while $L{n}$ is considered as high level context which
builds contextual information among all the original feature maps extracted
from the image. Finally, a dynamic model selection is applied to select the
most certain models based on uncertainty quantification and a combination of
selected models is applied to produce the final prediction. For each image, a
number of test passes is applied using MC-dropout to produce a list of
probability distributions which are then used to generate mean and standard
deviation. The mean is used for identifying the class label of a single model,
while standard deviation is utilized for measuring the level of randomness and
uncertainty. In a dynamic way, each image in the dataset has a number of
accurate models which are chosen based on low value of uncertainty determined
using a pre-defined threshold. These selected models’ mean predictions are
aggregated for final class prediction.

###  III-C Multi-level context-aware models

To capture the spatial dependencies among image patches, _MCUa_ has been
designed in a context-aware fashion to learn different possible multi-level
contextual information. Here, we used the output of feature extractor of each
pre-trained DCNN model and fed it into several multi-level context-aware
models. The level of contextual information learned by _MCUa_ is determined by
a pattern of neighborhood criteria. More precisely, we encode the spatial
relationship information among patches based on the neighborhood of patches
that form some random shape. In other words, our context-aware models have
been designed based on a pattern tuple $P_{g,S_{i}}=(g,S_{i})$, where $g$ is
the number of patches used in the context-aware process and $S_{i}$ is the set
of shape indices (where each index $i$ is associated to a unique set of shape
indices). To identify a shape index, the starting patch and $g-1$ directions
should be specified. Fig. 3 clarifies an example of how different pattern
levels work to extract contextual information. For instance, $P_{2,S_{1}}$ has
a value of $g=2$ and two shapes. Moreover, $P_{4,S_{2}}$ has a value of $g=4$
and a set of shapes where the shapes are built using a number of feature maps
(_e.g._ , 3, 6, 5, and 4). More precisely, the process of building contextual
information for the shape index represented in $P_{4,S_{2}}$ works by
identifying the starting feature map location (_i.e._ , feature map number 3),
then all the possible directions in the matrix of the feature maps has to be
defined, where direction 1 is for the $down$ direction to pick feature map
number 6, then directions 2 and 3 are for the $left$ directions to pick
feature maps 5 and 4, respectively, (Please see Fig. 3). Each feature map
utilizes the pattern tuple mechanism to bring the spatial dependencies
information with other neighboring feature maps.

The feature patterns have been designed by taking into account the image-level
labels for the final classification during the minimization of loss function.
Context-aware networks are mainly image-wise networks which take concatenated
feature maps generated from the original neighbor feature maps (extracted from
the input image). These concatenated feature maps are fed into context-aware
networks to classify the images based on local and contextual features
extracted from images. Context-aware networks are trained against image-level
labels. More precisely, we minimize the loss function of different patterns of
feature maps inserted as an input to the final class label associated to the
image as an output.

Each Context-aware CNN consists of a sequence of 3 $\times$ 3 convolutional
layers followed by a 2 $\times$ 2 convolution with stride of 2 for down-
sampling. Batch normalization and ReLU activation function were used at the
end of each layer. To obtain the spatial average of feature maps, a 1 $\times$
1 convolutional layer is used before the classifier. The network ends with 3
fully connected layers and a log softmax classifier.

Figure 3: The extraction process of the contextual information (_i.e._ ,
context modeling) with different pattern levels using six feature maps. The
original feature maps (highlighted in orange) are used to encode different
levels of contextual information. For instance, $P_{2,S_{1}}$ represents the
contextual information of a pattern that is composed of 2 neighbor feature
maps, while $P_{4,S_{1}}$ and $P_{4,S_{2}}$ represent the process of building
contextual information for four neighbor feature maps with different set of
shapes, respectively. The blue highlighted feature maps represent the maps
chosen to build contextual information.

During the training of _MCUa_ , a partly overlapped patches are extracted from
the image by using different stride values. The stride value for each scale is
chosen to increase the number of patches and hence improve the contextual
representation of _MCUa_. We found in our experiments that using high stride
decreases the accuracy for a single context-aware model on a validation set.

The context-aware CNN has been trained using categorical cross-entropy loss
and learns to classify images based on the local features of image patches and
spatial dependencies among the different patches. Like pre-trained DCNN, data
augmentation has been applied.

In algorithm 1, we describe the implementation flow of a single context-aware
model. We start by resizing the image to multiple scales to extract smaller
patches. We then pass the extracted patches to a pre-trained DCNN model to
extract feature maps. After that, we iterate over each feature map and get the
indices of all possible feature maps that can build possible pattern of
neighborhood relationships. The related feature maps are then concatenated and
inserted in a set which holds all the concatenated feature maps. Finally, we
pass the concatenated feature maps set to the context-aware CNN. This is to
learn spatial dependencies among the related feature maps and produce the
network output. As a consequence, the feature maps will be fed into the log
softmax function to produce the probability distribution of the image.

Input: Original image $X$ to be classified

Output: class label $\hat{y}$

$X^{{}^{\prime}}$ $\leftarrow$ $X$ // resize original image to $m$ scales

// extractPatches is a function which takes image and patch dimensions as an
input and outputs $n$ patches $\\{{x^{(1)},x^{(2)},...,x^{(n)}}\\}$

$\\{{x^{(1)},x^{(2)},...,x^{(n)}}\\}$ $\leftarrow$
extractPatches($X^{{}^{\prime}}$, $p_{w}$, $p_{h}$)

// featureExtractor network takes $n$ patches as an input and outputs $n$
feature maps $\\{{fm^{(1)},fm^{(2)},...,fm^{(n)}}\\}$

$\\{{fm^{(1)},fm^{(2)},...,fm^{(n)}}\\}$ $\leftarrow$
featureExtractor($\\{{x^{(1)},x^{(2)},...,x^{(n)}}\\}$)

$F$ $\leftarrow$ $\\{{fm^{(1)},fm^{(2)},...,fm^{(n)}}\\}$ // store all the
extracted feature maps to set $F$

$T$, $Y$ $\leftarrow$ {} // define and initialize two empty sets

for  _$fm\in F$_ do

// getPatternIndices is a function which takes a feature map $fm$ and returns
the indices of all possible neighbor feature maps which form a pattern

$\\{{i_{1},i_{2},...,i_{n}}\\}$ $\leftarrow$ getPatternIndices($fm$)

// concatenate $fm$ with possible neighbor feature maps which form a pattern

$Y$ $\leftarrow$ $fm\parallel k_{i_{1}}\parallel
k_{i_{2}}\parallel...\parallel k_{i_{n}}$; where
$k_{i_{1}},k_{i_{2}},...,k_{i_{n}}\in F\wedge
k_{i_{1}},k_{i_{2}},...,k_{i_{n}}\neq fm$

// append the newly concatenated feature maps $Y$ to $T$

$T$ $\leftarrow$ $T$ $\parallel$ $Y$

end for

// contextAwareCNN is the network responsible for learning the spatial
dependencies of all feature maps and their formed patterns

$O$ $\leftarrow$ contextAwareCNN($T$)

// logSoftmax function is used to map the output of contextAwareCNN $O$ to
probability distribution $V$

$V$ $\leftarrow$ logSoftmax($O$)

$\hat{y}$ $\leftarrow$ $\arg\max V$

Algorithm 1 Single context-aware model

###  III-D Dynamic model selection and combination

The final stage of _MCUa_ model is to dynamically ensemble the most certain
models for each image. To this end, we adapted an ensemble-based uncertainty
quantification component to allow for a dynamic selection of context-aware
models to produce the final prediction for an input image. To measure the
uncertainty of the individual context-aware models in _MCUa_ , we adopted MC
dropout [40] for each model, in the test phase, to produce a list of
probability predictions for each class of the input image. Then, we calculated
the mean and standard deviation for each class. The mean is used to produce
the final class label ($\hat{y}$) of the image, while standard deviation is
considered as a measure of uncertainty for the context-aware model. Based on
such uncertainty measures, a dynamic number of context-aware models will be
selected (based on uncertainty threshold value ($\delta$)) for each particular
image.

More precisely, each input image will be sensitive to a certain number of
context-aware models to form the final model ensemble. A context-aware model
will be selected if its uncertainty measure value is less than a pre-defined
$\delta$, as described in our experimental study. More importantly, images
with zero chosen models during this dynamic selection process can be provided
to medical professionals (pathologists) for reviewing and annotating. Once the
context-aware models are selected, the mean class predictions is aggregated to
produce the final class prediction distribution. Here, we formulate the mean
prediction and standard deviation as

$\mu=\frac{1}{z}\sum_{i=1}^{z}\beta\left(\Phi_{i}(X);W\right),$ (4)

$\sigma=\frac{1}{z}\sum_{i=1}^{z}\left(\beta\left(\Phi_{i}(X);W\right)-\mu\right)^{2},$
(5)

where $\mu$ represents the mean prediction, $\sigma$ defines the uncertainty
and $z$ defines the number of MC dropout test passes. The function $\beta$
represents the context-aware CNN with input $X$ and $W$ denotes the network
weights, while $\Phi_{i}$ defines a MC dropout test pass $i$ to the input
image $X$.

Algorithm 2 provides a detailed description of the ensemble process of _MCUa_
model. Each model produces a single probability distribution. We applied some
MC dropout test passes to generate a list of probability distributions for
each model. Then, to get the final class prediction and the measure of
uncertainty for each model, we computed the mean and standard deviation of the
generated list of probability distributions, respectively. Finally, using a
$\delta$ value, we include only the most certain models and we aggregate the
mean of probability distributions for these models to produce $\hat{y}$.

Input: Original image $X$ to be classified

Output: Class label $\hat{y}$

$X_{scale{1}},X_{scale2},...,X_{scalem}$ $\leftarrow$ $X$ // resize original
image to $m$ scales

$\\{{x_{m}^{(1)},x_{m}^{(2)},...,x_{m}^{(n)}}\\}$ $\leftarrow$
extractPatches($X_{scalem}$, $p_{w}$, $p_{h}$)

$F_{FeatureExtractor_{m}}$ $\leftarrow$
FeatureExtractor($\\{{x_{m}^{(1)},x_{m}^{(2)},...,x_{m}^{(n)}}\\}$)

for  _$fm\in F_{FeatureExtractor_{m}}$_ do

${i_{1},i_{2},...,i_{n}}$ $\leftarrow$ getPatternIndices($fm$)

$Y$ $\leftarrow$ $fm\parallel k_{i_{1}}\parallel
k_{i_{2}}\parallel...\parallel k_{i_{n}}$

$T$ $\leftarrow$ $T$ $\parallel$ $Y$

end for

$T_{all}$ $\leftarrow$ $\\{{T_{M1},T_{M2},...,T_{Mn}}\\}$ // output $T_{all}$
of context-aware stage from $n$ context-aware models
$\\{M_{1},M_{2},...,M_{n}\\}$

for  _$j\in\mathrm{MC_{dropout}TestPasses}$_ do

$\\{{O_{M1},O_{M2},...,O_{Mn}}\\}$ $\leftarrow$
contextAwareCNN($\\{{T_{M1},T_{M2},...,T_{Mn}}\\}$)

// probability distribution $V$ from $n$ context-aware models

$\\{{V_{M1},V_{M2},...,V_{Mn}}\\}$ $\leftarrow$
logSoftmax($\\{{O_{M1},O_{M2},...,O_{Mn}}\\}$)

$V_{total}$.append($\\{{V_{M1},V_{M2},...,V_{Mn}}\\}$)

end for

// get model-wise mean and uncertainty of probability distributions

$\\{\mu_{1},\mu_{2},...,\mu_{n}\\}$ $\leftarrow$ mean($V_{total}$)

$\\{\sigma_{1},\sigma_{2},...,\sigma_{n}\\}$ $\leftarrow$
standardDeviation($V_{total}$)

chosenModels $\leftarrow$ {}

for  _$j\in contextAwareModels$_ do

if  _$\sigma_{j}$ $<$ $\delta$_ then

chosenModels.append($\mu_{j}$)

end if

end for

// aggregate the mean probability distributions of chosen models

$B$ $\leftarrow$ aggregate(chosenModels)

$\hat{y}$ $\leftarrow$ $\arg\max B$

Algorithm 2 _MCUa_ Model

##  IV Experimental Study

###  IV-A Dataset

In this experimental study, we used BACH dataset which is part of ICIAR 2018
challenge for classification of H&E stained breast cancer histology images.
The dataset is composed of two parts (namely Part A and Part B). Part A of the
dataset is composed of 400 sections of microscopy images that are equally
distributed among four classes (normal, benign, in situ, and invasive). On the
other hand, Part B is composed of 10 high resolution whole slide images, where
the annotations are provided for a semantic segmentation task. In this work,
we focused on Part A of the dataset to evaluate the performance of the
classification models. The dataset was annotated by two medical experts and
all microscopy images are relevant to different patients. The total number of
patients involved in the production of the dataset was 39. The anonymization
process of the dataset does not allow to retrieve the origin of all images.
All the microscopy images have the same size of 2048 $\times$ 1536 pixels at
20X magnification level (where, the pixel resolution of the images is 0.42
$\mu m$).

We evaluated the performance of _MCUa_ model using 400 training images with
stratified five-fold cross validation. To train and fine-tune patch-wise
networks (_i.e._ , pre-trained DCNNs), we used microscopy patches extracted
from training images which are augmented using different rotations and
reflections. We evaluated the performance of the ensemble of patch-wise
networks using the validation set before stacking context-aware networks on
the top of patch-wise networks. Likewise, for context-aware models, which are
stacked on the top of patch-wise networks, we followed the same training
process conducted in patch-wise networks.

###  IV-B Hyperparameter Settings

For multi-scale image features, we managed to try different images scales
including the scale of the original image. Based on a comprehensive
experimentation as well as the recommendation of the work conducted in [9], we
decided to resize the original image (of the size 2048 x 1536) to 448 x 336
(scale 1), and 296 x 224 (scale 2). To extract image patches from the multi-
scale resized images, we utilized sliding window technique of size
$p_{w}=p_{h}=224$. Also, we set the stride (at scale 1) to 28 and 56 for
training data extraction and testing data extraction, respectively. For scale
2, we set the stride to 9 and 18 for training data extraction and testing data
extraction, respectively. In this work, for a fair comparison, we followed the
same hyperparameter settings as pointed out in [9], where the same backbone
networks were used.

The overlapped extracted patches are then fed into the pre-trained DCNN
models. We used DenseNet-161 for scale 1 and 2, while ResNet-152 is utilized
for scale 1 only. This gives three ensemble pre-trained feature extractors.
The choice of these three pre-trained DCNNs with the associated image scales
was aligned with the conclusion that has been drawn from the work conducted in
[9]. An ablation study was conducted in [9] using several different ImageNet
pre-trained networks. The study has included different image scales (2048 x
1536, 1024 x 768, 448 x 336, and 296 x 224) for the BACH dataset and different
pre-trained networks (DenseNet-161, ResNet-101, and ResNet-152). They also
considered different combinations of the fine-tuned DCNN models (with
different image scales) for the ensemble modeling. Our work utilized the
optimal combination recommended by their study, which is using DenseNet-161
for scale 1 and 2, while using ResNet-152 for scale 1. In the training
process, we applied data augmentation for each patch by applying rotation
operation of 90 degrees with/without vertical rotation. This results in eight
versions of a single patch. We set the learning rate to 0.0001 for 5 training
epochs with a batch size of 32.

The feature maps extracted from each pre-trained DCNN are then inserted into
multi-level context-aware models which present different levels of contextual
information. We utilized six multi-level context-aware models for each pre-
trained DCNN giving us a total number of 18 context-aware models. Based on
initial experimentation, we designed our _MCUa_ model in a constructive way by
experimenting a group of 3 context-aware models until reaching the total
number of context-aware models represented in this work. In our experiments,
we considered the amount of GPU memory available and, at the same time,
covering different prominent levels of spatial dependencies, different pre-
trained DCNN models, and different image scales when choosing the total number
of context-aware models.

For context-aware networks, we utilized stride s = 112 for scale 1 and s = 9
for scale 2. The stride values are chosen after comprehensive experimentation
to pick up the suitable values which give higher accuracy as well as improving
the contextual assumption for _MCUa_ model. The settings for a context-aware
network are exactly like the pre-trained DCNN settings except that we used 10
training epochs and batch size equals to 8. For data augmentation, we used
same transformations applied for pre-trained DCNN models, but using rotation
operation of 180 degrees. Moreover, as overfitting is a major problem in this
network, dropout was used with 0.7 rate.

As a final stage, for each image, the most certain models have been selected
and combined, in a dynamic way. To implement this, we utilize MC-dropout with
a total number of 50 test passes (which is sufficient to generate a
statistically valid distribution) for each image. Based on the mean and
standard deviation of the 50 distributions, we used the mean to produce the
final prediction, while standard deviation was used as a measure of
uncertainty. The dynamic picking of context-aware models is performed using
$\delta$ threshold which ranges from 0.001 to 1.75.

###  IV-C Performance Evaluation

We adopt accuracy, precision, recall and F1-score. Precision is intuitively
the ability of the classifier not to label as positive a sample that is
negative, recall is the ability of the classifier to find all the positive
samples and F1-score can be interpreted as a weighted average of the precision
and recall. We computed the accuracy, precision, recall and F1-score:

$Accuracy=\frac{TP+TN}{TP+TN+FP+FN},$ (6)

$Precision=\frac{TP}{TP+FP},$ (7)

$Recall=\frac{TP}{TP+FN},$ (8)

$F1-score=2\cdot\frac{Precision\times Recall}{Precision+Recall},$ (9)

where $TP$ and $TN$ represent correct predictions by our ensemble architecture
for the occurrence of certain class or not, respectively. $FP$ and $FN$ are
the incorrect architecture predictions for all cases.

####  IV-C1 Performance of a single context-aware model

Table IV-C3 presents the classification accuracy for our individual context-
aware models that have been designed on the top of three pre-trained DCNNs
(_e.g._ , DenseNet-161 using two image scales 448 $\times$ 336 (scale 1) and
296 $\times$ 224 (scale 2) and ResNet-152 using image scale 1). The context-
aware models are implemented based on different pattern levels and shape
indices ($P_{2,S_{1}}$, $P_{3,S_{1}}$, $P_{4,S_{1}}$, $P_{4,S_{2}}$,
$P_{5,S_{1}}$, $P_{6,S_{1}}$ and $P_{8,S_{1}}$). Based on trial and error
experiments, we excluded $P_{7,S_{1}}$ as it gives lower accuracy compared to
the other pattern levels. Also, to demonstrate the idea of using different
shapes within the same pattern level, we experimented $P_{4,S_{1}}$ and
$P_{4,S_{2}}$, where each one of them has a unique set of shape indices. As
illustrated by Table IV-C3, the highest classification accuracies are obtained
by $P_{2,S_{1}}$, $P_{4,S_{2}}$ and $P_{5,S_{1}}$ with the three pre-trained
DCNNs. Moreover, most of the context-aware models for image scale 1 achieved a
classification accuracy which varied between 93% and 94.75%, while the
context-aware models for image scale 2 achieved less accuracy ranging between
88.75% and 90.25%.

####  IV-C2 Static _MCUa_ Model

We have presented the accuracy, precision, recall, and AUC of the proposed
static ensemble context-aware architecture (_i.e._ , ensemble of the total 18
models) to distinguish each category of images and overall classification
accuracy in Table II. As illustrated by the table, invasive carcinoma tissues
and benign tissues can be differentiated clearly from other classes. We
achieved an average precision of 95.90% $\pm$ 2.40% and an overall
classification accuracy of 95.75% $\pm$ 2.44%, which illustrates the viability
of our proposed architecture in classifying breast histology images.

####  IV-C3 Static vs. Dynamic _MCUa_ Model

To demonstrate the sensitivity of _MCUa_ to the uncertainty quantification
component, we studied the performance of the static ensemble of context-aware
models and our dynamic ensemble mechanism. For a fair comparison, we utilized
two other metrics: (1) Weighted Average Accuracy ($WA_{ACC}$), which computes
accuracy for each fold of the 5 folds weighted by the number of included
images in that fold and after that it averages the weighted accuracies of 5
folds over the total number of included images all over the dataset; and (2)
abstain percentage ($Abs$), which calculates the percentage of excluded images
in the dataset through different $\delta$ values. We formulated WA-ACC and Abs
as:

$WA_{ACC}=\frac{1}{\sum_{i=1}^{r}w_{i}}\sum_{i=1}^{r}Accuracy_{i}\times
w_{i},$ (10)

$Abs=\left(\frac{\sum_{i=1}^{r}\sum_{j=1}^{h}X_{ij}^{{}^{\prime\prime}}}{D_{all}}\right)\times
100,$ (11)

where $Accuracy_{i}$ represents classification accuracy $i$ over $r$ folds,
$w_{i}$ is the weight of the included images in fold $i$,
$X_{ij}^{{}^{\prime\prime}}$ represents excluded image $j$ over $h$ excluded
images in fold $i$, and $D_{all}$ is the total number of images in BACH
dataset.

Table III illustrates the effectiveness of _MCUa_ by improving the
classification accuracy obtained by static ensemble mechanism. As demonstrated
by Table III, _MCUa_ has achieved $WA_{ACC}$ of 98.11% with $\delta$ of 0.001
and around 97.70% with $\delta$ values of 0.002, 0.003, 0.006 and 0.02.

Fig. 4(a), 4(b), and 4(c) depict $WA_{ACC}$ curve for included images, $Abs$,
and $WA_{ACC}$ curve for excluded images on BACH dataset, respectively, over
$\delta$ ranges from 0.001 to 1.75. The $WA_{ACC}$ curve for the included
images shows that the best $WA_{ACC}$ is achieved when the $\delta$ value is
low and the accuracy starts to decrease with increasing $\delta$ values until
it reaches 0.1. Moreover, as shown by the same figure, the accuracy increases
until settling at 95% with $\delta$ value of 0.5 to 1.75. On the other hand,
$Abs$ curve shows that the percentage of abstained images decreases when we
use higher $\delta$ values, and starting from 0.25, the number of excluded
images dropped to zero. Finally, the $WA_{ACC}$ curve for excluded images
shows the performance of _MCUa_ model using static ensemble, where the
accuracy was around 80% for small $\delta$ and then the accuracy starts to
drop until reaching 50% at $\delta$ of 0.1. The accuracy is zero when the
number of excluded images equals to zero. This demonstrates that excluded
images are typically harder to classify, and may well require a pathologist to
make an expert judgment.

TABLE I: Classification Accuracy for context-aware models based on different
pattern levels using stratified five-fold cross validation on BACH dataset
(%).

Pre-trained DCNN |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
(Image Scale) | Context-aware Pattern Levels - Accuracy
|  $P_{2,S_{1}}$ |  $P_{3,S_{1}}$ |  $P_{4,S_{1}}$ |  $P_{4,S_{2}}$ |  $P_{5,S_{1}}$ |  $P_{6,S_{1}}$ |  $P_{8,S_{1}}$
DenseNet-161 (Scale 1) |  93.75 |  93.00 |  93.50 |  93.25 |  93.50 |  93.25 |  –
DenseNet-161 (Scale 2) |  89.00 |  89.75 |  – |  90.25 |  89.75 |  88.75 |  90.25
ResNet-152 (Scale 1) |  94.00 |  93.25 |  93.50 |  94.75 |  94.75 |  93.75 |  –

TABLE II: Performance (mean $\pm$ standard deviation) of _MCUa_ (static ensemble) on BACH Dataset with stratified five-fold cross validation (%). Category |  Precision |  Recall |  F1-score |  Accuracy
---|---|---|---|---
Normal |  93.32 $\pm$ 5.34 |  95.00 $\pm$ 5 |  94.07 $\pm$ 4.10 |  97.00 $\pm$ 2.09
Benign |  96.00 $\pm$ 4 |  95.00 $\pm$ 5 |  95.45 $\pm$ 4.15 |  97.75 $\pm$ 2.05
InSitu |  95.28 $\pm$ 4.68 |  96 $\pm$ 2.24 |  95.56 $\pm$ 1.98 |  97.75 $\pm$ 1.04
Invasive |  99.00 $\pm$ 1 |  97 $\pm$ 2.74 |  97.97 $\pm$ 2.10 |  99.00 $\pm$ 1
Total |  95.90 $\pm$ 2.40 |  95.75 $\pm$ 2.44 |  95.77 $\pm$ 2.42 |  95.75 $\pm$ 2.44

TABLE III: Accuracy (%) of _MCUa_ model with both static and dynamic ensemble on BACH dataset. Method |  $\delta$ |  Accuracy
---|---|---
_MCUa_ (Static Ensemble) |  NA |  95.75
_MCUa_ (Dynamic Ensemble) |  0.001 |  98.11
0.002 |  97.93
0.003 |  97.60
0.006 |  97.65
0.01 |  97.53

####  IV-C4 Comparison with Recent Methods

In Table IV, we compare the performance of our proposed model with the
following state-of-the-art recent methods: (1) a two-stage CNN proposed by
Nazeri et al. [5], which consists of patch-wise network for feature extraction
and image-wise network for image level classification, (2) a context-aware
learning strategy using transferable features, which is based on a pre-trained
DCNN and SVM for classification [8], (3) Bayesian DenseNet-169 proposed by
Mobiny and Singh [37], which generates uncertainty measure for input images,
(4) deep spatial fusion CNN introduced by Huang and Chung [21], which uses
patch-wise residual network for feature extraction and deep spatial fusion
network that has been designed to capture the spatial relationship among image
patches using the spatial feature maps, (5) ARA-CNN introduced by Raczkowski
et al. [38], which uses variational dropout during the testing phase to
measure the uncertainty of input images, (6) ScanNet with feature aggregation
method of [26], which applies feature extraction and concatenation towards the
final classification, (7) Hybrid DNN introduced by Yan et al. [29] which uses
inception network for feature extraction of image patches along with bi-
directional LSTM network which learns contextual information among feature
maps generated from inception network, (8) EMS-Net proposed by Zhanbo et al.
[9], which applies an ensemble of pre-trained DCNNs, and (9) 3E-Net [39] which
builds an ensemble of image-wise networks with a measure of uncertainty using
Shannon entropy to pick the most certain image-wise models for the final image
classification. As demonstrated by Table IV, our model outperformed other
models when both static and dynamic ensemble mechanisms are used. Moreover,
Fig. 4(f) illustrates ROC curves for our proposed _MCUa_ (with both dynamic
and static ensemble) to confirm the superiority of our proposed solution.
Consequently, the importance of integrating multi-level contextual information
into DCNNs, to alleviating the high visual variability in breast histology
images, has been emphasized and experimentally proofed.

TABLE IV: Performance (mean $\pm$ standard deviation) comparison of the proposed _MCUa_ model and recent methods on BACH Dataset (%). Method |  Precision |  Recall |  F1-score |  Accuracy
---|---|---|---|---
Two-Stage CNN [5] |  86.35 $\pm$ 2.70 |  85.50 $\pm$ 3.38 |  85.49 $\pm$ 3.25 |  85.50 $\pm$ 3.38
DCNN + SVM [8] |  86.88 $\pm$ 1.52 |  85.75 $\pm$ 1.90 |  85.58 $\pm$ 1.92 |  85.75 $\pm$ 1.90
Bayesian DenseNet-169 [37] |  89.28 $\pm$ 4.71 |  88.50 $\pm$ 5.03 |  88.45 $\pm$ 5.05 |  88.50 $\pm$ 5.03
Deep Spatial Fusion CNN [21] |  89.93 $\pm$ 4.11 |  89.00 $\pm$ 3.89 |  88.93 $\pm$ 4.02 |  89.00 $\pm$ 3.89
Variational Dropout ARA-CNN [38] |  90.25 $\pm$ 2.87 |  89.50 $\pm$ 3.14 |  89.48 $\pm$ 3.13 |  89.50 $\pm$ 3.14
ScanNet + Feature Aggregation [26] |  90.90 $\pm$ 3.87 |  90.50 $\pm$ 3.81 |  90.46 $\pm$ 3.86 |  90.50 $\pm$ 3.81
Hybrid DNN [29] |  91.79 $\pm$ 3.50 |  91.00 $\pm$ 3.46 |  90.98 $\pm$ 3.45 |  91.00 $\pm$ 3.46
EMS-Net [9] |  95.23 $\pm$ 2.13 |  95.00 $\pm$ 2.17 |  94.98 $\pm$ 2.13 |  95.00 $\pm$ 2.17
3E-Net [39] |  95.68 $\pm$ 3.15 |  95.46 $\pm$ 3.22 |  95.45 $\pm$ 3.24 |  95.46 $\pm$ 3.22
_MCUa_ (Static Ensemble) |  95.90 $\pm$ 2.40 |  95.75 $\pm$ 2.44 |  95.77 $\pm$ 2.42 |  95.75 $\pm$ 2.44
_MCUa_ (Dynamic Ensemble ($\delta=0.001$)) |  98.25 $\pm$ 1.58 |  98.11 $\pm$ 1.77 |  98.10 $\pm$ 1.78 |  98.11 $\pm$ 1.77

####  IV-C5 Performance of _MCUa_ on BreakHis Dataset

To confirm the effectiveness of our solution, we applied _MCUa_ model on the
Breast Cancer Histopathological Database (BreakHis) [46]. BreakHis has 7909
breast cancer histology images collected from 82 patients, obtained with
different magnification levels (40X, 100X, 200X, and 400X). The dataset
consists of 2480 benign and 5429 malignant microscopic images with resolution
of 700 $\times$ 460 pixels each. We used images with a magnification level of
40X, which included 625 benign and 1370 malignant samples (1995 microscopic
samples in total).

In this study, we used the same hyperparameter settings that we used for the
BACH dataset. For example, we down-sampled the original input image (700
$\times$ 460) to two image scales (scale 1: 448 $\times$ 336 and scale 2: 296
$\times$ 224). These image scales are fed as input to the pre-trained DCNN
models (DenseNet-161 and ResNet-152) for extraction of features from image
patches. The extracted features are then inserted into 18 context-aware models
to learn the spatial relationships among the image patches. We used the same
patch stride and data augmentation settings (that has been applied to the BACH
dataset) for both feature extraction and context-aware modeling networks. As
BreakHis dataset has 2 classes (Benign and Malignant), we fine-tuned the pre-
trained DCNN models by modifying the number of neurons of the last fully
connected layer to only two neurons. As shown in Table V, _MCUa_ demonstrated
to be effective in both static and dynamic techniques. Using 5-fold cross
validation, we achieved a classification accuracy of 99.80% using the static
ensemble technique. The model has achieved exceptional classification
accuracies of 100%, 99.95%, and 99.90% using dynamic ensemble on $\delta$
values of 0.001, 0.003, and 0.03, respectively. Fig. 4(d) and 4(e) depict the
$WAA$ and $Abs$ curves for _MCUa_ using BreakHis dataset.

Figure 4: (a) Weighted average accuracy ($WA_{ACC}$) for the included images
on BACH dataset, (b) abstain percentage ($Abs$) of BACH dataset, (c)
$WA_{ACC}$ for the excluded images on BACH dataset, (d) $WA_{ACC}$ for
BreakHis’ included images, (e) abstain percentage ($Abs$) for BreakHis’
excluded images, and (f) Receiver Operating Characteristic (ROC) curves for
the static and dynamic methods of _MCUa_ Model using 5-fold cross validation
on BACH dataset.

####  IV-C6 Ablation Study

In this work, we describe the ablation study that we conducted to reach the
final version of the building components of our _MCUa_ model. All the
conducted experiments in this ablation study are validated with BACH dataset.
As an initial step towards our final version of _MCUa_ , we implemented a
single DCNN with a target to learn multi-scale and multi-level feature
patterns. This is accomplished by using multiple patch scales (224 x 224, 112
x 112, 56 x 56, and 28 x 28) to identify different nuclei sizes in histology
images. Then, we utilized all the feature maps extracted from the
aforementioned patch scales by applying fusion for the multi-scale, multi-
level feature maps for final classification. The single DCNN was built using a
sequence of 3 x 3 filters in the convolutional layers, followed by a pooling
layer, with the number of channels doubled after each down-sampling. We used 2
x 2 filters in the convolutional layers with stride of 2 for down-sampling the
feature maps. Batch normalization and ReLU activation were used after all
convolutional layers. Finally, a fully connected layer followed by softmax
layer are used to produce the final image classification. We applied
stratified 5-fold cross validation and achieved classification accuracy of
87.50%.

TABLE V: Accuracy (%) of _MCUa_ model with both static and dynamic ensemble on BreakHis dataset. Method |  $\delta$ |  Accuracy
---|---|---
_MCUa_ (Static Ensemble) |  NA |  99.80
_MCUa_ (Dynamic Ensemble) |  0.001 |  100
0.003 |  99.95
0.03 |  99.90
0.04 |  99.85

In another trial, we implemented single DCNNs to extract feature maps from
image patches, learn spatial dependencies among image patches arranged in a
certain pattern, and generated the final image classification. We used
DenseNet-161 with image scales (scale 1: 448 $\times$ 336 and scale 2: 296
$\times$ 224) and ResNet-152 with image scale (scale 1: 448 $\times$ 336) as
the single DCNNs in this study. We applied stratified 5-fold cross validation,
and we achieved a classification accuracy of 93.00% and 88.50% for
DenseNet-161 with scales 1 and 2, respectively, while, ResNet-152 using scale
1 yielded a classification accuracy of 94.50%. Although the aforementioned
methods are straightforward and easy to implement, we argue that single DCNNs
lack diversity in generating discriminative features which is vital in the
usage of ensemble strategy. This helps to generate features from multi-scale
and multi-architecture perspectives to help in representing multi-level
haematological objects (such as nuclei and glands) within the histology
images.

Consequently, we applied an ensemble of three pre-trained single DCNNs with
two image scales and achieved a classification accuracy of 95.00%.
Furthermore, instead of using the pre-trained DCNNs for classification task,
we used them for feature extraction of image patches, then we stacked 18
context-aware models over the three pre-trained DCNNs. The ensemble process of
18 context-ware models yielded a classification accuracy of 95.75% (_MCUa_
static ensemble).

In the final stage of _MCUa_ , we evaluated the contribution of uncertainty-
aware component, which is stacked over 18 context-aware models. This strategy
introduces the machine-confidence in the automated prediction of histology
images. The full version of _MCUa_ (based on the uncertainty-aware component)
yielded a classification accuracy of 98.11%. This justifies the effectiveness
of using multi-scale input, multi-architecture feature extraction, multi-level
context-aware modeling, and uncertainty quantification for the dynamic
ensemble mechanism.

##  V Conclusion and Future Work

In this paper, we proposed a novel dynamic ensemble of context-aware models,
we called Multi-level Context and Uncertainty aware (_MCUa_) model, to
classify H&E stained breast histology images into four classes including
normal tissue, benign lesion, in situ carcinoma, and invasive carcinoma.
_MCUa_ model has been designed in a way to learn the spatial dependencies
among the patches in a histology image by integrating multi-level contextual
information into the learning framework of deep convolutional neural networks.
Capturing the spatial relationships among the patches has been accomplished
using a pattern of neighborhood criteria through multiple context-aware
models. _MCUa_ model has also an uncertainty quantification component that
allows for a dynamic ensemble of the context-aware model to not only improve
the performance (by improving the learning diversity of the model) but also
quantify the difficulties in classifying images. _MCUa_ has achieved high
accuracy of 95.75% and 98.11% with both static ensemble and dynamic ensemble
mechanisms, respectively, on the BACH dataset, and outperformed other related
state-of-the-art models. In the future, we aim to extend our _MCUa_ to cope
with semantic segmentation problem of whole-slide images and study the effect
of multi-level contextual information on the robustness of the segmentation.
Another research direction is to add an explainability component to the _MCUa_
model to understand the decision and internal working mechanism of the model.
Moreover, one can extend _MCUa_ by using Bayesian-based dynamic ensemble
method and compare the performance with current settings.

## References

  * [1] B. Stewart and C. Wild, _World Cancer Report 2014_. The International Agency for Research on Cancer, 2014.
  * [2] A. Ibrahim, P. Gamble, R. Jaroensri, M. M. Abdelsamea, C. H. Mermel, P.-H. C. Chen, and E. A. Rakha, “Artificial intelligence in digital breast pathology: Techniques and applications,” _The Breast_ , vol. 49, pp. 267–273, 2020\.
  * [3] J. Barker, A. Hoogi, A. Depeursinge, and D. Rubin, “Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles,” _Medical Image Analysis_ , vol. 30, 12 2015.
  * [4] J. P. Vink, M. Leeuwen, C. Deurzen, and G. Haan, “Efficient nucleus detector in histopathology images,” _Journal of microscopy_ , vol. 249, 12 2012.
  * [5] K. Nazeri, A. Aminpour, and M. Ebrahimi, “Two-stage convolutional neural network for breast cancer histology image classification,” in _Image Analysis and Recognition_. Springer International Publishing, 2018, pp. 717–726.
  * [6] S. Mehta, E. Mercan, J. Bartlett, D. Weaver, J. Elmore, and L. Shapiro, “Y-Net: Joint Segmentation and Classification for Diagnosis of Breast Biopsy Images,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2018.
  * [7] T. Araújo, G. Aresta, E. Castro, J. Rouco, P. Aguiar, C. Eloy, A. Polónia, and A. Campilho, “Classification of breast cancer histology images using convolutional neural networks,” _PLOS ONE_ , vol. 12, p. e0177544, 06 2017\.
  * [8] R. Awan, N. A. Koohbanani, M. Shaban, A. Lisowska, and N. Rajpoot, “Context-aware learning using transferable features for classification of breast cancer histology images,” in _Image Analysis and Recognition_. Springer International Publishing, 2018, pp. 788–795.
  * [9] Z. Yang, L. Ran, S. Zhang, Y. Xia, and Y. Zhang, “Ems-net: Ensemble of multiscale convolutional neural networks for classification of breast cancer histology images,” _Neurocomputing_ , vol. 366, 07 2019.
  * [10] B. Ehteshami Bejnordi, G. Zuidhof, M. Balkenhol, M. Hermsen, P. Bult, B. Ginneken, N. Karssemeijer, G. Litjens, and J. van der Laak, “Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images,” _Journal of Medical Imaging_ , vol. 4, 05 2017.
  * [11] G. Aresta, T. Araújo, S. Kwok, S. S. Chennamsetty, M. Safwan, V. Alex, B. Marami, M. Prastawa, M. Chan, M. Donovan, G. Fernandez, J. Zeineh, M. Kohl, C. Walz, F. Ludwig, S. Braunewell, M. Baust, Q. D. Vu, M. N. N. To, E. Kim, J. T. Kwak, S. Galal, V. Sanchez-Freire, N. Brancati, M. Frucci, D. Riccio, Y. Wang, L. Sun, K. Ma, J. Fang, I. Kone, L. Boulmane, A. Campilho, C. Eloy, A. Polónia, and P. Aguiar, “Bach: Grand challenge on breast cancer histology images,” _Medical Image Analysis_ , vol. 56, pp. 122–139, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841518307941
  * [12] D. Li, X. Chen, Z. Zhang, and K. Huang, “Learning deep context-aware features over body and latent parts for person re-identification,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 7398–7407.
  * [13] R. Hou, B. Ma, H. Chang, X. Gu, S. Shan, and X. Chen, “Iaunet: Global context-aware feature learning for person reidentification,” _IEEE transactions on neural networks and learning systems_ , vol. PP, 09 2020.
  * [14] F. Codevilla, S. S. C. Botelho, N. Duarte, S. Purkis, A. S. M. Shihavuddin, R. Garcia, and N. Gracias, “Geostatistics for context-aware image classification,” in _Computer Vision Systems_. Springer International Publishing, 2015, pp. 228–239.
  * [15] M. Zhang, W. Li, and Q. Du, “Diverse region-based cnn for hyperspectral image classification,” _IEEE Transactions on Image Processing_ , vol. PP, pp. 1–1, 02 2018.
  * [16] Y. Zheng, H. Ye, L. Wang, and J. Pu, “Learning multiviewpoint context-aware representation for rgb-d scene classification,” _IEEE Signal Processing Letters_ , vol. 25, no. 1, pp. 30–34, Jan 2018.
  * [17] S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 3859–3869.
  * [18] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-k. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , ser. NIPS’15. Cambridge, MA, USA: MIT Press, 2015, p. 802–810.
  * [19] K. Makantasis, K. Karantzalos, A. Doulamis, and N. Doulamis, “Deep supervised learning for hyperspectral data classification through convolutional neural networks,” in _2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS)_ , 2015, pp. 4959–4962.
  * [20] H. Lee and H. Kwon, “Going deeper with contextual cnn for hyperspectral image classification,” _IEEE Transactions on Image Processing_ , vol. 26, no. 10, pp. 4843–4855, 2017.
  * [21] Y. Huang and A. C.-S. Chung, “Improving high resolution histology image classification with deep spatial fusion network,” in _Computational Pathology and Ophthalmic Medical Image Analysis_. Springer International Publishing, 2018, pp. 19–26.
  * [22] I. Koné and L. Boulmane, “Hierarchical resnext models for breast cancer histology image classification,” in _Image Analysis and Recognition_. Springer International Publishing, 2018, pp. 796–803.
  * [23] M. Kohl, C. Walz, F. Ludwig, S. Braunewell, and M. Baust, “Assessment of breast cancer histology using densely connected convolutional networks,” in _Image Analysis and Recognition_. Springer International Publishing, 2018, pp. 903–913.
  * [24] S. S. Chennamsetty, M. Safwan, and V. Alex, “Classification of breast cancer histology image using ensemble of pre-trained neural networks,” in _Image Analysis and Recognition_. Springer International Publishing, 2018, pp. 804–811.
  * [25] A. Cruz-Roa, H. Gilmore, A. Basavanhally, M. Feldman, S. Ganesan, N. Shih, J. Tomaszewski, A. Madabhushi, and F. González, “High-throughput adaptive sampling for whole-slide histopathology image analysis (hashi) via convolutional neural networks: Application to invasive breast cancer detection,” _PLOS ONE_ , vol. 13, p. e0196828, 05 2018.
  * [26] X. Wang, H. Chen, C. Gan, H. Lin, Q. Dou, E. Tsougenis, Q. Huang, M. Cai, and P.-A. Heng, “Weakly supervised deep learning for whole slide lung cancer image analysis,” _IEEE Transactions on Cybernetics_ , vol. 50, no. 9, pp. 3950–3962, 2020.
  * [27] A. BenTaieb, H. Li-Chang, D. Huntsman, and G. Hamarneh, “A structured latent model for ovarian carcinoma subtyping from histopathology slides,” _Medical image analysis_ , vol. 39, p. 194—205, July 2017. [Online]. Available: https://doi.org/10.1016/j.media.2017.04.008
  * [28] Y. Liu, K. Gadepalli, M. Norouzi, G. E. Dahl, T. Kohlberger, A. Boyko, S. Venugopalan, A. Timofeev, P. Q. Nelson, G. S. Corrado, and et al., “Detecting cancer metastases on gigapixel pathology images,” _CoRR_ , vol. abs/1703.02442, 2017. [Online]. Available: http://arxiv.org/abs/1703.02442
  * [29] R. Yan, F. Ren, W. Zihao, L. Wang, T. Zhang, Y. Liu, X. Rao, C. Zheng, and F. Zhang, “Breast cancer histopathological image classification using a hybrid deep neural network,” _Methods_ , vol. 173, 02 2020.
  * [30] L. Fang, C. Wang, S. Li, H. Rabbani, X. Chen, and Z. Liu, “Attention to lesion: Lesion-aware convolutional neural network for retinal optical coherence tomography image classification,” _IEEE Transactions on Medical Imaging_ , vol. PP, pp. 1–1, 02 2019.
  * [31] Y. Zeng, X. Liu, N. Xiao, Y. Li, Y. Jiang, J. Feng, and S. Guo, “Automatic diagnosis based on spatial information fusion feature for intracranial aneurysm,” _IEEE Transactions on Medical Imaging_ , vol. PP, pp. 1–1, 11 2019.
  * [32] C. Haarburger, M. Baumgartner, D. Truhn, M. Broeckmann, H. Schneider, S. Schrading, C. Kuhl, and D. Merhof, “Multi scale curriculum cnn for context-aware breast mri malignancy classification,” in _Medical Image Computing and Computer Assisted Intervention (MICCAI)_ , 2019.
  * [33] M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya _et al._ , “A review of uncertainty quantification in deep learning: Techniques, applications and challenges,” _Information Fusion_ , 2021.
  * [34] S. Graham, H. Chen, J. Gamper, Q. Dou, P.-A. Heng, D. Snead, Y. W. Tsang, and N. Rajpoot, “Mild-net: Minimal information loss dilated network for gland instance segmentation in colon histology images,” _Medical Image Analysis_ , vol. 52, pp. 199–211, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841518306030
  * [35] M. Fraz, S. A. Khurram, S. Graham, M. Shaban, A. Loya, and N. Rajpoot, “Fabnet: feature attention-based network for simultaneous segmentation of microvessels and nerves in routine histology images of oral cancer,” _Neural Computing and Applications_ , vol. 32, 07 2020.
  * [36] G. Liang, Y. Zhang, X. Wang, and N. Jacobs, “Improved trainable calibration method for neural networks on medical imaging classification,” in _British Machine Vision Conference (BMVC)_ , 2020.
  * [37] A. Mobiny and A. Singh, “Risk-aware machine learning classifier for skin lesion diagnosis,” _Journal of Clinical Medicine_ , vol. 8, p. 1241, 08 2019\.
  * [38] L. Raczkowski, M. Mozejko, J. Zambonelli, and E. Szczurek, “Ara: accurate, reliable and active histopathological image classification framework with bayesian deep learning,” _Scientific Reports_ , vol. 9, 10 2019.
  * [39] Z. Senousy, M. M. Abdelsamea, M. M. Mohamed, and M. M. Gaber, “3e-net: Entropy-based elastic ensemble of deep convolutional neural networks for grading of invasive breast carcinoma histopathological microscopic images,” _Entropy_ , vol. 23, no. 5, 2021. [Online]. Available: https://www.mdpi.com/1099-4300/23/5/620
  * [40] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_ , 2016, pp. 1050–1059.
  * [41] M. Macenko, M. Niethammer, J. S. Marron, D. Borland, J. T. Woosley, X. Guan, C. Schmitt, and N. E. Thomas, “A method for normalizing histology slides for quantitative analysis,” in _2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro_ , vol. 9, 2009, pp. 1107–1110.
  * [42] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [43] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2261–2269.
  * [44] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” _International Journal of Computer Vision_ , vol. 115, no. 3, pp. 211–252, Dec 2015. [Online]. Available: https://doi.org/10.1007/s11263-015-0816-y
  * [45] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _International Conference on Learning Representations_ , 12 2014.
  * [46] F. A. Spanhol, L. S. Oliveira, C. Petitjean, and L. Heutte, “A dataset for breast cancer histopathological image classification,” _IEEE Transactions on Biomedical Engineering_ , vol. 63, no. 7, pp. 1455–1462, 2016\.