# Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer
Histology Images

Muhammad Shaban, Ruqayya Awan, Muhammad Moazam Fraz, ,
Ayesha Azam, David Snead, *Nasir M. Rajpoot Manuscript received April 18,
2019. Asterisk indicates corresponding author.Muhammad Shaban, Ruqayya Awan,
Muhammad Moazam Fraz, Ayesha Azam, and Nasir M. Rajpoot are with the
Department of Computer Science, University of Warwick, CV4 7AL Coventry, U.K.
(e-mail: m.shaban, r.awan.1, moazam.fraz, ayesha.azam,
n.m.rajpoot@warwick.ac.uk)Muhammad Moazam Fraz is also with School of
Electrical Engineering and Computer Science, National University of Sciences
and Technology, H-12, Islamabad, PakistanAyesha Azam, David Snead, and Nasir
M. Rajpoot are with the Department of Pathology, University Hospitals
Coventry, Warwickshire, UKNasir M. Rajpoot is also with the The Alan Turing
Institute, London, UK

###### Abstract

Digital histology images are amenable to the application of convolutional
neural network (CNN) for analysis due to the sheer size of pixel data present
in them. CNNs are generally used for representation learning from small image
patches (e.g. $224\times 224$) extracted from digital histology images due to
computational and memory constraints. However, this approach does not
incorporate high-resolution contextual information in histology images. We
propose a novel way to incorporate larger context by a context-aware neural
network based on images with a dimension of $1,792\times 1,792$ pixels. The
proposed framework first encodes the local representation of a histology image
into high dimensional features then aggregates the features by considering
their spatial organization to make a final prediction. The proposed method is
evaluated for colorectal cancer grading and breast cancer classification. A
comprehensive analysis of some variants of the proposed method is presented.
Our method outperformed the traditional patch-based approaches, problem-
specific methods, and existing context-based methods quantitatively by a
margin of $3.61\%$. Code and dataset related information is available at this
link: https://tia-lab.github.io/Context-Aware-CNN

###### Index Terms:

Computational pathology, Deep learning, Context-Aware convolutional networks,
Cancer grading.

This work has been submitted to the IEEE for possible publication. Copyright
may be transferred without notice, after which this version may no longer be
accessible.

##  I Introduction

Histology slides are used by pathologists to analyze the micro-anatomy of
cells and tissues through a microscope. However, recent technological
developments in digital imaging solutions [1] have digitized the histology
slides (histology images) which enable the pathologists to do the same
analysis over the computer screen. These histology images are way larger than
natural images, where one cell nucleus usually takes around $50\times 50$
square pixels at the highest magnification level (e.g. $40\times$) and each
image contains tens of thousands of cells. The digitization process results in
an explosion of data which leads to new avenues of research for machine
learning and deep learning communities.

Convolutional neural networks (CNNs) have been widely used to achieve the
state-of-the-art results for different histology image analysis tasks such as
nuclei detection and classification [2, 3, 4], metastasis detection [5, 6, 7],
tumor segmentation [8] and cancer grading [9, 10, 11]. Each task requires a
different amount of contextual information, for instance, cell classification
needs only high-resolution cell appearance along with little neighboring
tissue whereas tumour detection and segmentation rely on a larger context
covering multiple cells simultaneously. Due to tumour heterogeneity, cancer
grading requires high-resolution cell information as well as the contextual
spatial organization of cells in the tumour microenvironment (TME). Most
existing CNN based methods applied to histology images follow a patch based
approach to training different models tends to ignore contextual information
due to memory constraints. Although these models are often trained on a large
number of image patches extracted from histology images, often spatial
relationships between neighbouring patches are ignored. Due to the lack of
large contextual information, the inference is independent of underlying
tissue architecture and it is performed based on the limited context captured
by individual patches. This approach works well for problems where contextual
information is relatively less important for prediction. However, contextual
information becomes vital in problems where diagnostic decisions are made on
the basis of underlying tissue architecture such as cancer grading.

In this paper, we consider colorectal cancer (CRC) grading to demonstrate the
significance of context-aware CNNs in cancer histology image analysis. CRC is
the fourth most common cause of cancer-related deaths worldwide [12]. The
grade of CRC is determined by pathologists by collective analysis of
individual cancer cells’ abnormality and their spatial organization as
distorted glandular structure in the histology image. Several studies on the
prognostic significance of CRC adopted a two-tiered grading system to reduce
the inter-observer variability [13, 14], merging the well and moderately
differentiated glands into a low-grade tumor and classifying tissue with
poorly and undifferentiated glands as a high-grade tumor. In this work, we
consider diagnostic regions captured from CRC histology images containing
enough context to reliably predict the cancer grade (see Figure 1). We refer
to them as visual fields in this paper as selected by an expert pathologist. A
CNN based method for CRC grading requires a high-resolution view of cancer
cells in the visual field along with the large contextual information to
capture cell organization for accurate grading.

Figure 1: Three visual field regions of colorectal tissue which highlight the
importance of larger context for correct grading. Each cell of the overlaid
grid shows the $224\times 224$ pixel context captured by a standard patch
classifier at $20\times$ magnification.

We propose a novel framework for context-aware learning of histology images.
The proposed framework first learns the local representation by a CNN (LR-CNN)
and then aggregate the contextual information through a representation
aggregation CNN (RA-CNN), as shown in Figure 2. The proposed framework takes
$1,792\times 1,792$ input image which is 64 times larger than the standard
patch classifier’s input. The LR-CNN divides the input image into patches and
converts them into high-dimensional feature vectors. These feature vectors are
arranged in the form of a feature-cube using the same spatial arrangement in
which the corresponding patches were extracted. This feature-cube is then fed
into the RA-CNN to make predictions based on both high-resolution feature
representation and spatial context. The proposed context-aware framework is
flexible enough to incorporate any state-of-the-art image classifier as LR-CNN
for local representation learning with the RA-CNN. We present detailed results
and that our proposed framework achieves superior performance over traditional
patch-based approaches and existing context-aware methods. Moreover, the
proposed framework also outperforms the problem specific methods for CRC
grading. Our main contributions in this paper are as follows:

  * •

We propose a novel framework for context-aware learning from large high-
resolution input images e.g. $1,792\times 1,792$ at $20\times$ resolution.

  * •

The proposed framework is highly flexible since it can leverage any state-of-
the-art network design for local representation learning.

  * •

We explore different context-aware learning and training strategies to enhance
the framework’s ability to learn the contextual information.

  * •

We report comprehensive experiments (with 100+ network models) and comparisons
to demonstrate the superiority of the proposed context-aware learning
framework over traditional patch-based methods and existing context-aware
learning methods.

Figure 2: Flow diagram of the proposed context-aware framework for CRC
grading. The top row shows the local representation learning. The bottom row
illustrates the network architecture for representation aggregation learning
which consists of multiple context blocks and other standard layers. Dashed
lines represent the blocks of a specific network design whereas solid lines
represent the common blocks (see Table I for notations).

##  II Related Work

Related work is divided into two subsections: methods related to context-aware
learning and some problem specific methods on cancer grading.

###  II-A Context-Aware Learning

In literature, various different approaches have been presented to incorporate
the contextual information for the classification of histology images. Some
researchers [15, 16, 17] used image down-sampling, a common practice followed
in natural image classification, to capture the context from larger histology
image. However, this approach is not suitable for problems where cell
information is as important as the context. Adaptive patch sampling [18] and
discriminative patch selection [19] from histology images is another way to
integrate the sparse context. These methods are not capable of capturing small
regions of interest at high resolution e.g. tumor cells and their local
contextual arrangement. Some methods [20, 21, 22, 6] leverage the multi-
resolution nature of histology images and use multi-resolution based
classifiers to capture context. These multi-resolution approaches only
consider a small part of an image at high resolution and the remaining part at
low resolutions to make a prediction. Therefore, these approaches lack the
contextual information of cellular architecture at high resolution in a
histology image.

Recently, some works [23, 24, 25, 26] have used larger high-resolution patches
to improve the segmentation of histology images. Zanjani et al. [25] and Li et
al. [26] used a CNN based feature extractor followed by a Conditional Random
Field (CRF) for context learning. Latter is end-to-end trainable with a patch
size of $672\times 672$, considerably smaller than the patch size used in our
proposed method. Agarawalla et al.[23] and Kong et al. [24] used a 2D-LSTM
instead of CRF to improve tumor segmentation. Some works [27, 28] used larger
patches at high resolution for the task of context-based classification.
Koohbanani et al.[27] proposed a context-aware network for breast cancer
classification. They used standard SVM to learn the context from the CNN based
features of the patches extracted from a high-resolution image. Due to the
nature of the final classifier, this work is only capable of capturing a
limited context. Bejnordi et al. [28] proposed a similar approach for breast
tissue classification. They trained their network in two steps. In the first
step, they used a small patch size and in the second step, they fixed the
weights of half of the network to feed a larger patch for training the
remaining half of the network. Their network also suffers from a limited
context problem as they managed to train a network with the largest patch size
of $1,024\times 1,024$ pixels with small batch size ($10$ patches).
Sirinukunwattana et al. [29] presented a systematic comparison of different
context-aware methods to highlight the importance of context-aware learning.

As opposed to the aforementioned methods, our proposed method is different in
a network design such that it is flexible enough to accommodate any state-of-
the-art CNN based image classifiers for representation learning and a custom
CNN based architecture for representation aggregation. The representation
learning and aggregation are stacked together for context-aware learning with
larger image size, $1,792\times 1,792$ and a typical batch size of 64 images.

###  II-B Problem Specific Method

A number of automated methods for objective grading of breast, prostate, and
colorectal cancer [30, 31, 32, 11] have been proposed in the literature. For
instance, in [31, 30, 32], a linear classifier is trained with handcrafted
features based on the glandular morphology for prostate cancer grading.

Awan et al. [11] presented a method for two-tier CRC grading based on the
extent of deviation of the gland from its normal shape (circular/elliptical).
They proposed a novel Best Alignment Metric (BAM) for this purpose. As a pre-
processing step, CNN based gland segmentation was performed, followed by the
calculation of BAM for each gland. For every image, average BAM was considered
as a feature along with two more features inspired by BAM values. In the end,
an SVM classifier was trained using this feature set for CRC grading. Our
proposed method differs from these existing methods in two ways. First, it
does not depend on the intermediate step of gland segmentation making it
independent of segmentation inaccuracies. Second, the proposed method is
entirely based on a deep neural network which makes this framework independent
of cancer type. Therefore, the proposed framework could be used for other
context-based histology image analysis problems. In this regard, besides CRC
grading, we have demonstrated the application of the proposed method for
breast cancer classification.

##  III The Proposed Method

The proposed framework for context-aware grading consists of two stacked CNNs
as shown in Fig 2. The first network, LR-CNN, converts the high-resolution
information of an image into high dimensional feature-cube through patch based
feature extraction. The second network, RA-CNN aggregates the learned
representation in order to learn the spatial context from the feature-cube to
make a prediction. We leverage the power of traditional patch classifiers to
learn local representation from individual patches. However, we explore
different network architectures for context block in RA-CNN for context-aware
learning. Moreover, different training strategies are explored to build a
powerful context-aware grading model. The following section explains each
building block of the proposed framework in detail. The notation used to
describe each building block is summarized in Table I.

###  III-A Network Input

The input to our framework is an image ($X^{k}$) from a dataset,
$D=\\{X^{k},Y^{k},S^{k};k=1,\dots K\\}$, of large high resolution images which
consists of $K$ images with corresponding labels $Y^{k}\in\\{1,\dots,C\\}$ for
classification into $C$ classes and coarse patch level segmentation masks
$S^{k}\in\\{1,\dots,C\\}$ for multi-task learning. Each image is divided into
$M\times N$ patches of same size where $x^{k}_{ij}$ and $y^{k}_{ij}$ represent
the ${ij}^{th}$ patch of $k^{th}$ image and its corresponding label,
respectively. We used a patch dataset,
$d=\\{(x^{k}_{ij},y^{k}_{ij}),|~{}x^{k}_{ij}\in X^{k},y^{k}_{ij}\in Y^{k}\\}$,
which consists of patches and their corresponding labels for pre-training of
LR-CNN.

TABLE I: Enumeration of symbols used in the paper

Symbol | Description | Symbol | Description
---|---|---|---
$D$ | Image dataset | $X^{k}$ |  $k^{th}$ image
$K$ | Number of images | $Y^{k}$ | Label of $k^{th}$ image
$C$ | Number of Classes | $S^{k}$ | Mask of $k^{th}$ image
X | Set of all images | Y | Labels of X
S | Masks of X | $d$ | Patch dataset
$M$ | Patches in an image column | $i$ | $1,\dots,M$
$N$ | Patches in an image row | $j$ | $1,\dots,N$
$x^{k}_{ij}$ |  $ij^{th}$ patch of $X^{k}$ | $y^{k}_{ij}$ | Label of $x^{k}_{ij}$ patch
$\mathcal{F}(\cdot)$ | Feature extractor | $f^{k}_{ij}$ | Features of $x^{k}_{ij}$
$L_{f}$ | Fully connected layer | $L_{p}^{g}$ | Global pooling layer
$L_{c}^{a\times a}$ |  $a\times a$ convolution layer | $L_{s}$ | Softmax layer
$\rightarrow$ | Transition between layers | $\bullet$ | Preceding layer’s output
$\otimes$ | Hadamard product | $\oplus$ | Feature Concatenation
$\mathcal{B}(\cdot)$ | Context-block | $\mathcal{C}(\cdot)$ | Context-Net
F | Feature of X | $\textbf{F}^{\prime}$ | Weighted Feature of X
$\textbf{Y}^{\prime}$ | Predicted labels of X | $Y^{\prime k}$ | Predicted label of $X^{k}$
$\textbf{S}^{\prime}$ | Predicted Masks of X | $S^{\prime k}$ | Predicted Mask of $X^{k}$
$W^{k}$ |  $k^{th}$ image weight | $\theta$ | Learnable Parameters
$\mathcal{L}_{cls}$ | Classification cost function | $\mathcal{L}_{wgt}$ | Weighted cost function
$\mathcal{L}_{seg}$ | Segmentation cost function | $\mathcal{L}_{joint}$ | Joint cost function

###  III-B Local Representation Learning

First part of the proposed framework encodes an input image $X^{k}$ into a
feature-cube $F^{k}$. All the input images are processed through the LR-CNN in
a patch based manner. The proposed framework is flexible enough to use any
state-of-the-art image classifier as a LR-CNN such as ResNet50 [33], MobileNet
[34], Inception [35], or Xception [36]. This flexibility also enables it to
use pre-trained weights in case of a limited dataset. Moreover, it is possible
to train the LR-CNN independently before plugging it into the proposed
framework, enabling it to learn meaningful representation [37] which leads to
early convergence of the context-aware learning part of the framework.

###  III-C Feature Pooling

The spatial dimensions of the output feature $f^{k}_{ij}$ of a patch
$x^{k}_{ij}$ may vary depending on the input patch dimensions and the network
architecture for feature extraction. A global feature pooling layer is
employed to get a similar dimensional feature vector for all variations of the
proposed framework. Both average and max global pooling strategies are
explored. After global pooling, features of all patches are rearranged in the
same spatial order ($M\times N$) as extracted patches to construct the
feature-cube $F^{k}$ for context-aware learning. The depth of the feature-cube
depends on the choice of LR-CNN. For the sake of generality, we will represent
the output of our LR-CNN as follows,

$\textbf{F}=\mathcal{F}(\textbf{X},\theta_{\mathcal{F}})\rightarrow
L_{p}^{g}(\bullet)$ (1)

where $\mathcal{F}$ represents the fully convolutional part of the LR-CNN and
acts as a feature extractor whereas X is the batch of images and F is the
local feature representation of X after pooling $L_{p}^{g}$, which could be a
global average or max pooling layer. The operator ($\rightarrow$) provides the
output of the preceding layer to the following layer and operator ($\bullet$)
represents the output of the preceding layer.

###  III-D Feature Attention

As the input to the proposed framework has a relatively large spatial
dimension, there may be some part of the image that may not have any
significance for the prediction of image label. We introduce an attention
block to give less weight to insignificant features and vice-versa. This
attention block takes feature-cube as input and learns the weight of
usefulness for each value in the feature-cube. Hadamard product is taken
between the weights and input feature-cube to increase the impact of more
important areas of an image in label prediction and vice-versa. The weighted
feature-cube $\textbf{F}^{\prime}$ is defined as:

$\textbf{F}^{\prime}=L_{c}^{1\times 1}(\textbf{F},\theta_{c})\rightarrow
L_{s}(\bullet)\otimes\textbf{F},$ (2)

where $L_{c}^{1\times 1}$ and $\theta_{c}$ represent the $1\times 1$
convolution layer and its parameters, respectively. $L_{s}$ denotes the
softmax layer and the operator $\otimes$ is used to represent Hadamard
product.

###  III-E Context Blocks

Since the LR-CNN is used to encode the important patch-based image
representation into a feature-cube, therefore the main aim of the context
block (CB) is to learn the spatial context within the feature cube. The CB
learns the relation between the features of the image patches considering
their spatial location. We propose three different CB architectures, each with
different complexity and capability to capture the context information. First
CB, $\mathcal{B}_{1}(\cdot)$, is comprised of a $3\times 3$ convolution layer
followed by ReLU activation and batch normalization. Second CB,
$\mathcal{B}_{2}(\cdot)$, uses residual block [33] architecture with two
different filter sizes. It consists of three convolution layers each followed
by batch normalization and ReLU activation. The first and last layers are with
$1\times 1$ convolution filter to squeeze and expand the feature depth. The
output feature-maps of the last layer are concatenated with the input
features-maps which makes its final output. The $\mathcal{B}_{2}(\cdot)$ is
defined as:

$\displaystyle\mathcal{B}_{2}(\textbf{F}^{\prime},\theta_{\mathcal{B}_{2}})$
$\displaystyle=[L_{c}^{1\times
1}(\textbf{F}^{\prime},\theta_{\mathcal{B}^{1}_{2}})\rightarrow L_{c}^{3\times
3}(\bullet,\theta_{\mathcal{B}^{2}_{2}})$ (3) $\displaystyle\rightarrow
L_{c}^{1\times
1}(\bullet,\theta_{\mathcal{B}^{3}_{2}})]\oplus\textbf{F}^{\prime},$

where $L_{c}^{1\times 1}$ and $L_{c}^{3\times 3}$ denote the convolution
layers with $1\times 1$ and $3\times 3$ filter sizes;
$\theta_{\mathcal{B}^{1}_{2}}$, $\theta_{\mathcal{B}^{2}_{2}}$, and
$\theta_{\mathcal{B}^{3}_{2}}$ are the parameters of different convolution
layers and $\theta_{\mathcal{B}_{2}}$ represents parameter of the whole
context block for brevity. The operator $\oplus$ represents the concatenation
of feature-maps.

Unlike the previous two context blocks, our third CB processes the input
feature-maps in parallel with different filter sizes to capture context from
varying receptive fields. Similar to the blocks in [35], it consists of
multiple $1\times 1$ and $3\times 3$ convolution layers each followed by batch
normalization and ReLU activation. A $3\times 3$ average pooling layer
$L_{p}^{3\times 3}$ is also used to average the local context information. The
CB, $\mathcal{B}_{3}$, is defined as:

$\displaystyle\mathcal{B}_{3}(\textbf{F}^{\prime},\theta_{\mathcal{B}_{3}})$
$\displaystyle=[L_{c}^{1\times
1}(\textbf{F}^{\prime},\theta_{\mathcal{B}^{1}_{3}})\rightarrow L_{c}^{3\times
3}(\bullet,\theta_{\mathcal{B}^{2}_{3}})$ (4) $\displaystyle\rightarrow
L_{c}^{3\times 3}(\bullet,\theta_{\mathcal{B}^{3}_{3}})]\oplus[L_{c}^{1\times
1}(\textbf{F}^{\prime},\theta_{\mathcal{B}^{4}_{3}})]$
$\displaystyle\oplus[L_{c}^{1\times
1}(\textbf{F}^{\prime},\theta_{\mathcal{B}^{5}_{3}})\rightarrow L_{c}^{3\times
3}(\bullet,\theta_{\mathcal{B}^{6}_{3}})]$ $\displaystyle\oplus[L_{p}^{3\times
3}(\textbf{F}^{\prime})\rightarrow L_{c}^{1\times
1}(\bullet,\theta_{\mathcal{B}^{7}_{3}})],$

where $\theta_{\mathcal{B}^{1}_{3}}$ to $\theta_{\mathcal{B}^{7}_{3}}$ are the
parameters of different convolution layers and $\theta_{\mathcal{B}_{3}}$
represents parameter of the whole context block for the sake of notational
simplicity.

###  III-F Representation Aggregation for Context Learning

The local representation tissue has been learned by the LR-CNN. Therefore, the
task of spatial context learning from feature-cube is relatively less
challenging as compared to context learning from the raw image. A cascaded set
of three context blocks ($\mathcal{C}(\cdot)$) of the same type
($\mathcal{B}_{1}$,$\mathcal{B}_{2}$, or $\mathcal{B}_{3}$) is used in RA-CNN.
These context blocks are explained in section III-E. The output of
$\mathcal{C}(\cdot)$ is followed by a global average pooling, a fully
connected, and a softmax layer to make the final prediction in the required
number of classes. The final prediction $\textbf{Y}^{\prime}$ from the
features of input images X is computed as:

$\textbf{Y}^{\prime}=\mathcal{C}(\textbf{F}^{\prime},\theta_{\mathcal{C}})\rightarrow
L_{p}^{g}(\bullet)\rightarrow L_{f}(\bullet,\theta_{f^{\prime}})\rightarrow
L_{s}(\bullet),$ (5)

where $\theta_{\mathcal{C}}$ and $\theta_{f^{\prime}}$ represent the
parameters of all context blocks and the fully connected layer in RA-CNN,
respectively. The proposed framework is trained end-to-end with categorical
cross-entropy loss based cost function $\mathcal{L}_{cls}(\cdot)$ which is
defined as:

$\mathcal{L}_{cls}(\textbf{Y},\textbf{Y}^{\prime})=-\frac{1}{K}\sum_{k=1}^{K}\sum_{c=1}^{C}{Y^{k}_{c}\log_{2}(Y^{{}^{\prime}k}_{c})},$
(6)

where $Y^{k}_{c}$ and $Y^{{}^{\prime}k}_{c}$ are the ground truth and
predicted probabilities of $k^{th}$ image for $c^{th}$ class.

###  III-G Auxiliary Block

The proposed framework is designed for the classification of large input
images. Therefore, the label of an input image may depend on a set of
different primitive structures (such as glands) and their spatial
organization. To exploit these primitive structures for better classification
we proposed an auxiliary block which acts as patch based segmentation of the
primitive structures in the input image. This will improve the convergence of
proposed networks and also output the coarse patch based segmentation mask
($S_{s}^{\prime}$) along with image label ($Y^{\prime}$). The segmentation
masks ($\textbf{S}^{\prime}$) of input images X from their features
$\textbf{F}^{\prime}$ is defined as:

$\textbf{S}^{\prime}=\mathcal{C}(\textbf{F}^{\prime},\theta_{\mathcal{C}})\rightarrow
L_{c}^{1\times 1}(\bullet,\theta_{c^{\prime}})\rightarrow L_{s}(\bullet),$ (7)

where $L_{c}^{1\times 1}$ is a convolution layer with $\theta_{c^{\prime}}$
parameters. The addition of auxiliary block enables the proposes framework to
learn in a multi-task setting, where the coarse segmentation-map guides the
network to improve the individual patch based feature classification in
addition to the prediction of the input image. This leads to a network with
improved classification performance since it is minimizing both segmentation
and classification loss simultaneously. The segmentation-map based loss
function ($\mathcal{L}_{seg}$) and joint loss function ($\mathcal{L}_{joint}$)
are defined as:

$\mathcal{L}_{seg}(\textbf{S},\textbf{S}^{\prime})=-\frac{1}{K}\sum_{k=1}^{K}\sum_{c=1}^{C}{S^{k}_{c}\log_{2}(S^{{}^{\prime}k}_{c})},$
(8)

$\displaystyle\mathcal{L}_{joint}(\textbf{Y},\textbf{Y}^{\prime},\textbf{S},\textbf{S}^{\prime})=$
$\displaystyle\alpha\times\mathcal{L}_{cls}(\textbf{Y},\textbf{Y}^{\prime})+$
(9)
$\displaystyle(1-\alpha)\times\mathcal{L}_{seg}(\textbf{S},\textbf{S}^{\prime}),$

where $\alpha$ is a hyper-parameter which defines the contribution of both
loss functions in the final loss. Similar to patch classifier, the loss
function ($\mathcal{L}_{joint}$) is minimized with RMSprop optimizer [38].

###  III-H Training Strategies

We trained the proposed framework in four different ways with varying ability
to capture the spatial context. First, the proposed framework is trained
without attention block and by minimizing the $\mathcal{L}_{cls}(\cdot)$ loss
only. This configuration is represented by solid line blocks in Fig 2. Second,
the same configuration as first but trained with a sample-based weighted loss
function, $\mathcal{L}_{wgt}(\cdot)$, which give more weight to the image
patches with relatively less region of interest (glandular region) as compared
to the background. The weight of an image and $\mathcal{L}_{wgt}(\cdot)$ are
defined as follow,

$W^{k}=\begin{cases}\frac{1}{R^{k}_{roi}},&\text{if }R^{k}_{roi}>\alpha\\\
\frac{1}{\alpha},&\text{otherwise}\end{cases}$ (10)
$\mathcal{L}_{wgt}(\textbf{Y},\textbf{Y}^{\prime})=-\frac{1}{K}\sum_{k=1}^{K}\sum_{c=1}^{C}{W^{k}Y^{k}_{c}\log_{2}(Y^{{}^{\prime}k}_{c})},$
(11)

where $R^{k}_{roi}$ and $W^{k}$ represent the ratio of the region of interest
and weight of the $k^{th}$ image. The $\alpha$ is the ratio threshold,
selected empirically as 0.10, sets the upper limit of an image weight. Third,
multi-task learning based training with the help of an auxiliary block by
using joint classification and segmentation loss, $\mathcal{L}_{joint}$. Last,
training using the same joint loss but with attention-based feature-cube to
amplify the contribution of more important features in the feature-cube. The
network configuration of this strategy is represented by both solid and dotted
lines blocks in Fig 2. We termed these strategies as standard, weighted,
auxiliary, and attention, respectively.

##  IV Datasets & Performance Measures

In this section, we explain the dataset details used for training and
evaluation of the proposed framework and metrics for performance evaluation.

###  IV-A Datasets

The proposed framework is evaluated on two different tissue types for two
different tasks in order to demonstrate its capabilities. Our colorectal
cancer dataset [11] is comprised of visual fields (refer as images for
simplicity) extracted from colorectal histology images based on a two-tier
[13, 14] grading system. The CRC dataset consists of 139 images with an
average size of $4,548\times 7,520$ pixels obtained at $20\times$
magnification. These images are classified into three different classes
(normal, low grade, and high grade) based on the organization of glands in the
images by the expert pathologist. We follow 3-fold cross validation for a fair
comparison of the proposed method with the method presented in [11]. We
extracted patches of two different sizes for the training of traditional patch
classifiers ($224\times 224$ pixels) and our proposed framework ($1,792\times
1,792$ pixels). A detailed distribution of the patches is presented in table
II. We introduced background class to handle the patches with no or little
glandular regions. In breast cancer classification [39], our goal is to
classify the images into four different tissue sub-types namely normal,
benign, in-situ, and invasive. We use exactly the same training, validation,
and test splits as in [29] for a fair comparison of our proposed framework
with the existing context-aware methods.

TABLE II: Number of patches in each class and fold of training dataset.
Patches of size 224$\times$224
---
Folds | Background | Normal | Low Grade | High Grade
1 | 25,000 | 25,000 | 25,000 | 25,000
2 | 25,000 | 25,000 | 25,000 | 25,000
3 | 25,000 | 25,000 | 25,000 | 25,000
Total | 75,000 | 75,000 | 75,000 | 75,000
Patches of size 1,792$\times$1,792
1 | 1,750 | 3,500 | 3,500 | 3,500
2 | 1,750 | 3,500 | 3,500 | 3,500
3 | 1,750 | 3,500 | 3,500 | 3,500
Total | 5,250 | 10,500 | 10,500 | 10,500

###  IV-B Performance Measures

We have used three metrics, the average accuracy, F1 score and Rank-sum
measure for performance evaluation. The average accuracy refers to the
percentage of images classified correctly, across the three folds. Rank-sum
based evaluation metric is used to summarize the accuracy of different models
trained using a specific setting in order to compare models trained with
different context-blocks and LR-CNNs. Different colors are used to represent
different rank for better illustrative visualization as shown in Table IV, V
and VIII. The orange color indicates the best performing method whereas green,
blue, yellow, and red colours indicate that the results are within 97.5%, 95%,
90%, and 85% of the best performing method, respectively. The rank for these
colors are: orange = 1, green = 2, blue = 3, yellow = 4, red = 5, and no
colour = 6. The lowest rank-sum shows the best performance.

##  V Experiments & Results

The proposed framework is extensively evaluated and compared with existing
approaches in three different categories, i.e. traditional patch based
classifiers, existing context-aware approaches and domain oriented methods for
CRC grading. The details of experimental evaluation are given in following
subsections.

###  V-A Experimental Setup

The CRC images are divided into patches of size $1,792\times 1,792$, and the
label of each patch is predicted using the proposed framework with a stride of
$224\times 224$. To avoid redundant processing of the same region, the input
images are processed with LR-CNN to get representation features of each local
region. Afterwards, RA-CNN is applied in a sliding window manner to aggregate
local representation for context-aware predictions. Through this approach, we
process the input image with a 64 times bigger context as compared to standard
patch classifier, with only 10% additional processing time. The majority
voting among predicted grades (normal, low grade and high grade) is used to
obtain the final grade of an image. The performance of the different variants
of the proposed framework is evaluated for the task of CRC grading to show its
stability and superior performance over other methods. These variations
include the use of four different state-of-the-art classifiers for local
representation learning in LR-CNN; spatial dimensionality reduction through
average and max-pooling; the usage of three different context-blocks in RA-
CNN; and four different training strategies. By employing different
combinations of above-mentioned variations, we trained around 100 models in
total for each fold of CRC training dataset. Note that, all reported results
for CRC grading are image based not patch based. The details of these
combinations and their results are given in the following subsection.

###  V-B LR-CNN based Classifiers

Four different LR-CNNs are trained using (ResNet50 [33], Inception [35],
MobileNet [34], and Xception [36]) with patch size of $224\times 224$ to get
the baseline CRC grading results. The ResNet-50 [33] and Inception network are
the winner of Image-Net [40] challenge in 2015 and 2016, respectively.
MobileNet is a lightweight network with just 3 million parameters whereas
Xception network uses separable convolutions which results in a significant
reduction in computational complexity. The performance of these classifiers
for CRC grading is reported in Table III. Although, the performance of all
classifiers is comparable, MobileNet shows superior performance with highest
mean accuracy. On the other hand, Xception classifier shows consistent
performance across three folds with the lowest standard deviation (Std.).

TABLE III: Accuracy comparison of four patch classifiers. Network | Fold-1 | Fold-2 | Fold-3 | Mean | Std.
---|---|---|---|---|---
ResNet50 | 93.48 | 93.62 | 89.13 | 92.08 | 2.08
MobileNet | 93.48 | 95.74 | 89.13 | 92.78 | 2.74
Inception-v3 | 95.65 | 91.49 | 86.96 | 91.37 | 3.55
Xception | 93.48 | 91.49 | 91.30 | 92.09 | 0.98

###  V-C RA-CNN based Context-Aware Learning

We experimented with three context-blocks, $\mathcal{B}_{1}$,
$\mathcal{B}_{2}$, and $\mathcal{B}_{3}$, to train three different variation
of RA-CNN, which we termed as RA-CNN 1, RA-CNN 2, and RA-CNN 3. These three
RA-CNN classifiers are trained separately with all four LR-CNNs as explained
in section III-F, hence giving 12 different combinations of the context-aware
network. The rank-sum method is used to compare the CRC grading performance of
these networks with each other and also with the LR-CNNs. The results in table
IV, shows that context-aware networks achieve superior performance as compare
to standard patch based classifiers (LR-CNNs). The RA-CNN 3 achieve the best
Rank-sum (lowest) which shows its robustness across different representation
learning networks. Other two context-aware networks also show comparable
performance by remaining in the 97.5% of the best performer.

TABLE IV: Accuracy comparison of three different context-aware networks with
standard patch classifiers.

LR-CNN (Avg) | Baseline | RA-CNN 1 | RA-CNN 2 | RA-CNN 3
---|---|---|---|---
ResNet50 | 92.08$\pm$2.08 | 94.25$\pm$2.70 | 92.08$\pm$2.08 | 93.51$\pm$3.10
MobileNet | 92.78$\pm$2.74 | 93.52$\pm$3.55 | 93.52$\pm$1.78 | 94.25$\pm$2.70
InceptionV3 | 91.37$\pm$3.55 | 94.23$\pm$3.71 | 94.96$\pm$2.72 | 95.68$\pm$1.78
Xception | 92.09$\pm$0.98 | 94.96$\pm$2.72 | 94.96$\pm$2.72 | 95.68$\pm$3.55
Rank-sum | 10 | 7 | 8 | 5

###  V-D Local Representation Robustness

We also conducted different experiments to analyze the robustness of local
representation learned by different LR-CNNs. These LR-CNNs are used in
combination with different RA-CNNs for context learning along with different
feature pooling strategies. Each LR-CNN is used to training three RA-CNNs with
both global average and global max pooled feature-cubes. The table V compares
the results using Rank-sum method. It can be observed that the Xception model
turns-out as most robust feature extractor in LR-CNNs with the best rank-sum
score of 8. Inception model shows comparable results to the best performer as
its network design has significant overlap with Xception architecture.

TABLE V: Robustness analysis of feature extractors across different methods.

Methods | ResNet50 (%) | MobileNet(%) | InceptionV3(%) | Xception(%)
---|---|---|---|---
RA-CNN 1 (Avg) | 94.25$\pm$2.70 | 93.52$\pm$3.55 | 94.23$\pm$3.71 | 94.96$\pm$2.72
RA-CNN 1 (Max) | 93.52$\pm$1.87 | 93.51$\pm$3.10 | 94.23$\pm$2.07 | 93.54$\pm$3.03
RA-CNN 2 (Avg) | 92.08$\pm$2.08 | 93.52$\pm$1.78 | 94.96$\pm$2.72 | 94.96$\pm$2.72
RA-CNN 2 (Max) | 95.68$\pm$3.55 | 93.52$\pm$3.55 | 92.80$\pm$2.72 | 93.54$\pm$3.03
RA-CNN 3 (Avg) | 93.51$\pm$3.10 | 94.25$\pm$2.70 | 95.68$\pm$1.78 | 95.68$\pm$3.55
RA-CNN 3 (Max) | 94.23$\pm$2.07 | 92.82$\pm$2.01 | 94.25$\pm$2.70 | 94.96$\pm$2.72
Rank-sum | 12 | 12 | 10 | 8

###  V-E Training Strategies

We experimented with four different context related training strategies
(Standard, Weighted, Auxiliary and Attention) to explored their impact on
overall performance. The details of each training strategy are given in
Section III-H. Table VI shows the comparison of these training strategies for
Xception based LR-CNN. Each entry in the table contains the average accuracy
across three RA-CNNs for particular feature pooling (shown in rows) and the
training strategies (in columns). Attention-based training shows the superior
results for max-pooled features whereas standard training strategy achieves
comparable performance for average-pooled features. However, auxiliary loss
based training remains robust for both pooling types and achieves the best
overall accuracy. More importantly, each model shows superior performance than
the baseline LR-CNN classifier as shown in Fig. 3. The graphical illustration
of 24 experiments using the best performing LR-CNN is shown in Fig. 3. The
results obtained with different combinations of feature pooling type, the
context blocks in RA-CNN and the training strategies used for the experiments
are illustrated in the pie-chart format for better visual comparison. The
accuracy obtained by Xception based LR-CNN is considered as the baseline for
comparative analysis. Pie-charts for results with other LR-CNNs are given in
supplementary material.

TABLE VI: Comparison for different training strategies with Xception as feature extractor. Feature | Standard | Weighted | Auxiliary | Attention
---|---|---|---|---
Xception - Max | 94.01 | 94.49 | 94.73 | 95.21
Xception - Avg | 95.20 | 94.72 | 94.72 | 94.00
Mean | 94.61 | 94.60 | 94.72 | 94.61
Figure 3: (Left) Results of 24 experiments using best performing local
representation features (Xception). (Right) Legend represents the feature
pooling type, context-aware net and training strategies used for the
experiments. Baseline accuracy is the accuracy of standard patch based
Xception classifier.

###  V-F Comparative Evaluation

The proposed method is compared with both domain-based and other context-based
methods. Awan et al. [11] presented a two-step problem specific method for CRC
grading where the first step is deep learning based gland segmentation and in
the second step the domain knowledge is exploited by computing the Best
Alignment Metric (BAM) to capture the shape difference of CRC glands from
their expected shape (shape of normal gland). They experimented with two
different feature sets which we refer as BAM-1 and BAM-2 in this paper. BAM-1
comprises of average BAM and BAM entropy while BAM-2 comprises of an
additional feature known as regularity index. Their method achieved good
accuracy for binary classification, normal vs cancer, however, it lakes the
robustness required for multi-class grading of CRC images (see Table VII). Our
best performing context-aware network, RA-CNN 3, with Xception based LR-CNN
and attention based training method achieved superior performance as compared
to the BAM based methods.

We also compared our method with a set of context-aware approaches explored in
a systemic study on context-aware learning by Sirinukunwattana et al. [29].
Ten different approaches (A-J) were considered to capture contextual
information. For a detailed description of these methods please see Figure 2
of [29]. It can be observed that these approaches have significant overlap
with each other. some of these approaches differ only in term of input patch
resolution, as some of these used shared weights for different networks. First
four (A-D) approaches try to capture the context by down-sampling the high-
resolution images at different magnification levels e.g.
$20\times,10\times,5\times,$ and $2.5\times$. Channel-wise concatenated four
multi-resolution images as an input of a CNN are consider in approach E.
However, approaches F and I concatenate the CNN features of four multi-
resolution input images in vector form before prediction. The full image at
$20\times$ magnification is used as an input in approach H. This approach is
not feasible in case of an image with very large spatial dimensions due to
memory constraints. Approaches G and J use LSTM to capture the context from
the CNN features of four multi-resolution input images. The code of method G
is publicly available by the authors of [29] and we use that code to retrain
the LSTM based method on CRC dataset for a fair comparison. Our method
outperformed their context-aware method [29] as well for both binary and
three-class CRC grading (Table VII).

Visual comparison of best performing patch classifier, Sirinukunwattana et al.
(Context-G) and the proposed method on three different images with normal, low
and high grades are shown in Figure 4. Patch classifier’s prediction is quite
irregular for any given image due to the lake of contextual information. The
predictions of Context-G are relatively smooth but it predicts the wrong label
for the low-grade image which might be due to the use of a low-resolution
image for context learning. However, the proposed method predictions are
smooth and consistent with the ground truth labels.

We retrained our method on breast cancer dataset to make a direct comparison
to all the context-aware approaches presented in [29]. Sirinukunwattana et al.
used a customized network as feature extractor which contains 5 convolution
layers with $4\times 4$ filter and each layer followed by batch-norm and
leaky-ReLU activation. To highlight the significance of RA-CNN, we adopted
their network for LR-CNN and trained our RA-CNN 3 with standard training
strategy in end-to-end training manner. Moreover, the same experimental setup
(as in [29]) has been used for retraining of our method such as data-splits
and F1-score based Rank-sum evaluation metric. Table VIII presents the
comparison of our proposed method with ten other context-aware approaches. Our
method outperformed its counterpart methods with significant margin and
achieved best rank-sum scored for breast cancer classification task as well.

TABLE VII: Comparison with state-of-the-art on Colorectal Dataset. Method | Binary (%) | Three-class (%)
---|---|---
BAM - 1 [11] | 95.70 - 2.10 | 87.79 - 2.32
BAM - 2 [11] | 97.12 - 1.27 | 90.66 - 2.45
Context - G [29] | 96.44 - 3.61 | 89.96 - 3.54
ResNet50 [33] | 98.57 - 1.01 | 92.08 - 2.08
MobileNet [34] | 97.83 - 1.77 | 92.78 - 2.74
InceptionV3 [35] | 98.57 - 1.01 | 91.37 - 3.55
Xception [36] | 98.58 - 2.01 | 92.09 - 0.98
Proposed | 99.28 - 1.25 | 95.70 - 3.04
Figure 4: Visual results of CRC grading are shown for patch classifier, existing context, and the proposed method on an image of size $1,792\times 1,792$. The stride size for context networks is equal to the size of patch ($224\times 224$) used for patch classifier. Green, blue and red colors of overlaid rectangular boxes show the normal, low and high-grade predictions respectively, whereas empty box areas represent non-glandular/background regions. TABLE VIII: Rank-sum based comparison as measured by the F1-measure on Breast Dataset. | Methods
---|---
Classes | A | B | C | D | E | F | G | H | I | J | Proposed
Normal | 0.501 | 0.468 | 0.523 | 0.513 | 0.509 | 0.603 | 0.573 | 0.252 | 0.241 | 0.323 | 0.643
Benign | 0.453 | 0.468 | 0.482 | 0.444 | 0.410 | 0.369 | 0.423 | 0.489 | 0.333 | 0.437 | 0.511
InSitu | 0.468 | 0.476 | 0.486 | 0.533 | 0.615 | 0.614 | 0.581 | 0.286 | 0.311 | 0.452 | 0.362
Invasive | 0.401 | 0.477 | 0.430 | 0.54 | 0.557 | 0.548 | 0.576 | 0.520 | 0.446 | 0.580 | 0.576
Rank-sum | 23 | 22 | 22 | 20 | 16 | 16 | 17 | 20 | 24 | 18 | 10

##  VI Conclusion

In this paper, we present a novel context-aware deep neural network for cancer
grading, which is able to incorporate 64 times larger context than standard
CNN based patch classifiers. The proposed network is well-suited for CRC
grading task which relies on recognizing abnormalities in glandular
structures. These clinically significant structures vary in size and shape
that cannot be captured efficiently with standard patch classifiers due to
computational and memory constraints. The proposed context-aware network is
comprised of two stacked CNNs. The first LR-CNN is used for learning the local
representation of the histology image. The learned local representation is
then aggregated considering its spatial pattern by RA-CNN. The proposed
context-aware model is evaluated for CRC grading and breast cancer
classification. A comprehensive analysis of different variations of the
proposed model is presented and compared with existing approaches in the same
evaluation setting. The qualitative and quantitative results demonstrate that
our method outperformed the patch based classification methodologies, the
domain-oriented techniques and existing context-based methods. This approached
is suitable for cancer analysis which requires large contextual information in
the histology images. This includes Gleason grading in prostate cancer and
tumor growth pattern classification in lung cancer. Moreover, this approach
can further be extended to perform downstream analysis at digital whole slide
image level for patient survival analysis.

## References

  * [1] N. Farahani, A. V. Parwani, and L. Pantanowitz, “Whole slide imaging in pathology: advantages, limitations, and emerging perspectives,” _Pathol Lab Med Int_ , vol. 7, pp. 23–33, 2015.
  * [2] N. A. Koohababni, M. Jahanifar, A. Gooya, and N. Rajpoot, “Nuclei detection using mixture density networks,” in _International Workshop on Machine Learning in Medical Imaging_. Springer, 2018, pp. 241–248.
  * [3] K. Sirinukunwattana, S. E. A. Raza, Y.-W. Tsang, D. R. Snead, I. A. Cree, and N. M. Rajpoot, “Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images,” _IEEE transactions on medical imaging_ , vol. 35, no. 5, pp. 1196–1206, 2016\.
  * [4] T.-H. Song, V. Sanchez, H. ElDaly, and N. Rajpoot, “Simultaneous cell detection and classification in bone marrow histology images,” _IEEE journal of biomedical and health informatics_ , 2018.
  * [5] B. E. Bejnordi, M. Veta, P. J. Van Diest, B. Van Ginneken, N. Karssemeijer, G. Litjens, J. A. Van Der Laak, M. Hermsen, Q. F. Manson, M. Balkenhol _et al._ , “Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer,” _Jama_ , vol. 318, no. 22, pp. 2199–2210, 2017.
  * [6] N. A. Koohbanani, T. Qaisar, M. Shaban, J. Gamper, and N. Rajpoot, “Significance of hyperparameter optimization for metastasis detection in breast histology images,” in _Computational Pathology and Ophthalmic Medical Image Analysis_. Springer, 2018, pp. 139–147.
  * [7] H. Lin, H. Chen, S. Graham, Q. Dou, N. Rajpoot, and P.-A. Heng, “Fast scannet: Fast and dense analysis of multi-gigapixel whole-slide images for cancer metastasis detection,” _IEEE transactions on medical imaging_ , 2019.
  * [8] T. Qaiser, K. Sirinukunwattana, K. Nakane, Y.-W. Tsang, D. Epstein, and N. Rajpoot, “Persistent homology for fast tumor segmentation in whole slide histology images,” _Procedia Computer Science_ , vol. 90, pp. 119–124, 2016\.
  * [9] E. Arvaniti, K. S. Fricker, M. Moret, N. J. Rupp, T. Hermanns, C. Fankhauser, N. Wey, P. J. Wild, J. H. Rueschoff, and M. Claassen, “Automated gleason grading of prostate cancer tissue microarrays via deep learning,” _bioRxiv_ , p. 280024, 2018.
  * [10] N. Ing, Z. Ma, J. Li, H. Salemi, C. Arnold, B. S. Knudsen, and A. Gertych, “Semantic segmentation for prostate cancer grading by convolutional neural networks,” in _Medical Imaging 2018: Digital Pathology_ , vol. 10581\. International Society for Optics and Photonics, 2018, p. 105811B.
  * [11] R. Awan, K. Sirinukunwattana, D. Epstein, S. Jefferyes, U. Qidwai, Z. Aftab, I. Mujeeb, D. Snead, and N. Rajpoot, “Glandular morphometrics for objective grading of colorectal adenocarcinoma histology images,” _Scientific reports_ , vol. 7, no. 1, p. 16852, 2017.
  * [12] B. Stewart, C. P. Wild _et al._ , “World cancer report 2014,” _Health_ , 2017.
  * [13] W. Blenkinsopp, S. Stewart-Brown, L. Blesovsky, G. Kearney, and L. Fielding, “Histopathology reporting in large bowel cancer.” _Journal of clinical pathology_ , vol. 34, no. 5, pp. 509–513, 1981.
  * [14] J. Jass, W. Atkin, J. Cuzick, H. Bussey, B. Morson, J. Northover, and I. Todd, “The grading of rectal cancer: historical perspectives and a multivariate analysis of 447 cases,” _Histopathology_ , vol. 10, no. 5, pp. 437–459, 1986\.
  * [15] S. S. Chennamsetty, M. Safwan, and V. Alex, “Classification of breast cancer histology image using ensemble of pre-trained neural networks,” in _International Conference Image Analysis and Recognition_. Springer, 2018, pp. 804–811.
  * [16] M. Kohl, C. Walz, F. Ludwig, S. Braunewell, and M. Baust, “Assessment of breast cancer histology using densely connected convolutional networks,” in _International Conference Image Analysis and Recognition_. Springer, 2018, pp. 903–913.
  * [17] I. Koné and L. Boulmane, “Hierarchical resnext models for breast cancer histology image classification,” in _International Conference Image Analysis and Recognition_. Springer, 2018, pp. 796–803.
  * [18] X. Wang, H. Chen, C. Gan, H. Lin, Q. Dou, Q. Huang, M. Cai, and P.-A. Heng, “Weakly supervised learning for whole slide lung cancer image classification,” 2018.
  * [19] A. Cruz-Roa, H. Gilmore, A. Basavanhally, M. Feldman, S. Ganesan, N. Shih, J. Tomaszewski, A. Madabhushi, and F. González, “High-throughput adaptive sampling for whole-slide histopathology image analysis (hashi) via convolutional neural networks: Application to invasive breast cancer detection,” _PloS one_ , vol. 13, no. 5, p. e0196828, 2018.
  * [20] N. Alsubaie, M. Shaban, D. Snead, A. Khurram, and N. Rajpoot, “A multi-resolution deep learning framework for lung adenocarcinoma growth pattern classification,” in _Annual Conference on Medical Image Understanding and Analysis_. Springer, 2018, pp. 3–11.
  * [21] A. BenTaieb, H. Li-Chang, D. Huntsman, and G. Hamarneh, “A structured latent model for ovarian carcinoma subtyping from histopathology slides,” _Medical image analysis_ , vol. 39, pp. 194–205, 2017.
  * [22] Y. Liu, K. Gadepalli, M. Norouzi, G. E. Dahl, T. Kohlberger, A. Boyko, S. Venugopalan, A. Timofeev, P. Q. Nelson, G. S. Corrado _et al._ , “Detecting cancer metastases on gigapixel pathology images,” _arXiv preprint arXiv:1703.02442_ , 2017.
  * [23] A. Agarwalla, M. Shaban, and N. M. Rajpoot, “Representation-aggregation networks for segmentation of multi-gigapixel histology images,” _arXiv preprint arXiv:1707.08814_ , 2017.
  * [24] B. Kong, X. Wang, Z. Li, Q. Song, and S. Zhang, “Cancer metastasis detection via spatially structured deep network,” in _International Conference on Information Processing in Medical Imaging_. Springer, 2017, pp. 236–248.
  * [25] F. G. Zanjani, S. Zinger _et al._ , “Cancer detection in histopathology whole-slide images using conditional random fields on deep embedded spaces,” in _Medical Imaging 2018: Digital Pathology_ , vol. 10581. International Society for Optics and Photonics, 2018, p. 105810I.
  * [26] Y. Li and W. Ping, “Cancer metastasis detection with neural conditional random field,” _arXiv preprint arXiv:1806.07064_ , 2018.
  * [27] R. Awan, N. A. Koohbanani, M. Shaban, A. Lisowska, and N. Rajpoot, “Context-aware learning using transferable features for classification of breast cancer histology images,” _arXiv preprint arXiv:1803.00386_ , 2018\.
  * [28] B. E. Bejnordi, G. Zuidhof, M. Balkenhol, M. Hermsen, P. Bult, B. van Ginneken, N. Karssemeijer, G. Litjens, and J. van der Laak, “Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images,” _Journal of Medical Imaging_ , vol. 4, no. 4, p. 044504, 2017.
  * [29] K. Sirinukunwattana, N. K. Alham, C. Verrill, and J. Rittscher, “Improving whole slide segmentation through visual context-a systematic study,” _arXiv preprint arXiv:1806.04259_ , 2018.
  * [30] S. Naik, S. Doyle, S. Agner, A. Madabhushi, M. Feldman, and J. Tomaszewski, “Automated gland and nuclei segmentation for grading of prostate and breast cancer histopathology,” in _Biomedical Imaging: From Nano to Macro, 2008\. ISBI 2008. 5th IEEE International Symposium on_. IEEE, 2008, pp. 284–287.
  * [31] R. Farjam, H. Soltanian-Zadeh, K. Jafari-Khouzani, and R. A. Zoroofi, “An image analysis approach for automatic malignancy determination of prostate pathological images,” _Cytometry Part B: Clinical Cytometry_ , vol. 72, no. 4, pp. 227–240, 2007.
  * [32] K. Nguyen, B. Sabata, and A. K. Jain, “Prostate cancer grading: Gland segmentation and structural features,” _Pattern Recognition Letters_ , vol. 33, no. 7, pp. 951–961, 2012.
  * [33] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [34] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [35] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich _et al._ , “Going deeper with convolutions.” Cvpr, 2015.
  * [36] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” _arXiv preprint_ , pp. 1610–02 357, 2017.
  * [37] Z. Zhang, Y. Xie, F. Xing, M. McGough, and L. Yang, “Mdnet: A semantically and visually interpretable medical image diagnosis network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6428–6436.
  * [38] G. Hinton, N. Srivastava, and K. Swersky, “Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.”
  * [39] T. Araújo, G. Aresta, E. Castro, J. Rouco, P. Aguiar, C. Eloy, A. Polónia, and A. Campilho, “Classification of breast cancer histology images using convolutional neural networks,” _PloS one_ , vol. 12, no. 6, p. e0177544, 2017.
  * [40] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.


# Towards Automatic Model Completion: from Requirements to SysML State
Machines

Maria Stella de Biase  Dip. di Matematica e Fisica
Università della Campania “L. Vanvitelli”
Caserta, Italy
<EMAIL_ADDRESS>Stefano Marrone  Dip. di Matematica e
Fisica
Università della Campania “L. Vanvitelli”
Caserta, Italy
<EMAIL_ADDRESS>Angelo Palladino  Aerospace Business Unit
Kineton srl
Napoli, Italy
<EMAIL_ADDRESS>
###### Abstract

Even if model-driven techniques have been enabled the centrality of the models
in automated development processes, the majority of the industrial settings
does not embrace such a paradigm due to the procedural complexity of managing
model life cycle. This paper proposes a semi-automatic approach for the
completion of high-level models of critical systems. The proposal suggests a
specification guidelines that starts from a partial SysML (Systems Modeling
Language) model of a system and on a set of requirements, expressed in the
well-known Behaviour-Driven Design paradigm. On the base of such requirements,
the approach enables the automatic generation of SysML state machines
fragments. Once completed, the approach also enables the modeller to check the
results improving the quality of the model and avoiding errors both coming
from the mis-interpretation of the tool and from the modeller himself/herself.
An example taken from the railway domain shows the approach.

###### Index Terms:

Behavior Driven Development, Requirement Engineering, Natural Language
Processing, SysML, Critical System Design

##  I Introduction

The increasing complexity of critical systems requires a greater and greater
level of software dependability while market pressure pushes down products
time-to-market. To this aim, it is crucial to shorten software development
periods without sacrificing quality. The best way to reduce fatal errors is to
take preventive measures. Since the definition of requirements is the first
fundamental step in the development process, according to the software
engineering V-Model [3] (mainly used in safety-critical systems development
[11]), it is necessary to sift through the clarity, correctness and
consistency of the requirements in order to eliminate certain types of errors
in advance.

In this research, we follow the well-known Model-Based Development (MBD) [10],
which explored different aspects of the software development and of the above-
mentioned V-model. This notwithstanding, the MBD approaches mainly focused in
the past years on the “lower-level” activities (e.g., code generation,
testing). On the other hand, industries do not prefer big changes in their
assessed processes due to the necessity of managing multi-years ongoing
projects; instead, they prefer small improvements that easily adapt to
existing practices.

To this aim, this paper describes the first idea of an ongoing research whose
primary aim is mixing the contribution of both humans and automatic tools in a
requirement engineering process. The goal is to build a tool that, starting
from a partial SysML model of a system and from a set of textual requirements,
is able to complete the model by creating some SysML model fragments to
integrate in the global one. We are aware that this goal is very ambitious and
then there are some working hypotheses for the research:

  * •

The requirements are defined according the well-known Behavioural Driven
Development (BDD) [2] principle and using the Given-When-Then paradigm;

  * •

The requirements are also structured according to guidelines considering
different ways (requirement templates) of writing a BDD requirement;

  * •

SysML model is itself built according to a specific guideline where components
and their main states are partially defined.

Under these preliminary conditions, the first goal is to build an automatable
process able to: (1) read requirements and choose the most suitable model
fragments for them; (2) generate state machines’ transitions capturing the
behaviour described by the requirement; (3) annotate the requirement with a
set of SysML annotations to improve traceability and to ease the modeller in
verifying the final result of the model.

The fulfillment of this concrete objective will be possible by enhancing
traditional writing processes with formal grammars and using Artificial
Intelligence (AI) and Natural Processing Language (NLP) algorithms. SysML
profiling methods will also be followed, too.

The industrial impact of this research is high since one of the most critical
obstacles in the adoption of a full model-driven process in industrial
settings is constituted by the physiological disalignment between models and
other “runnable” artefacts. In this way, the model would be at the center of
the system development process since its early phases; automation would enable
the model as a “living object” of the process, and the traceability
information enables explainability and interpretability that are crucial for
critical systems, mitigating the traditional disinclination of international
standards as ISO 26262, DO-178B, and DO-178C to AI and NLP adoption. One of
the ever-changing critical systems is the railway domain.

Railway is one of the safest modes of transport111https://international-
railway-safety-council.com/safety-statistics/ and the safety is achieved by
properly managing its infrastructure. To this aim, the European Train Control
System (ETCS) is constantly improving performance and safety of the standard:
currently, the ETCS Level 3 (ETCS-L3) [13] is under development and different
research projects are still assessing the preliminary set of the standard
requirements. A concrete example taken from the railway signalling systems
will show the presented approach.

This paper is structured as follows: Section II presents an overview of the
state of the art, issues and solutions related to requirements definition and
domain modelling. Section III describe the steps to be followed, supported by
a small example. Section IV discusses technical concerns as well as reporting
the status of the research.

##  II Related Works

Requirement definition is a crucial and delicate stage. The requirement
writing process generates a detailed description of the system, indeed
requirements express what and how much well system functions should perform
[3]. According to the expressed features of the system, it is possible to
categorize the requirements (e.g., Functional, Safety, etc.). Requirements are
usually defined in Natural Language (NL), although intuitive may generate
omissions or ambiguities [4]. Over the years, many works developed solutions
and techniques to improve the definition of system requirements. All the
revised works consider the attributes of acceptability as starting point (see
Table I). They propose different approaches aimed at improving the requirement
definition, e.g., grammar-based [5], pattern-based [1, 4], formal models-based
[7, 8, 9].

Attribute |  Description
---|---
Clear and Consistent |  Readily understandable
Consistency |  Terms used are consistent
Correct |  Does not contain error of fact
Feasible |  Can be satisfied by technologies, etc.
Singular |  One actor-verb-object requirement
Verify |  Provable at the level of the architecture
Without ambiguity |  Only one interpretation makes sense
For pairs and sets of Requirement Statements
Absence of conflicts |  Not in conflict with other requirements
Absence of redundancy |  Each requirement specified only once
TABLE I: Attributes of acceptability [3]

The survey in [6] defines two categories of requirements formulation
approaches: direct and statistical machine translation. The first category is
based on the usage of a context free grammar to translate NL text in a
temporal model. The second one deals with the writing requirement process,
considering them as a result of a language translation subject to
probabilistic rules. Under the direct approaches umbrella, which uses patterns
to collect requirements, it is possible to find the BDD, a set of methods
developed by Dan North in the agile development framework [2]. Cucumber [1] is
the software that supports BDD technique providing
Gherkin222https://cucumber.io/docs/gherkin/. The latter is a language that
allows the definition of system requirements in NL by means of a patterns. The
authors in [8] analysed different temporal logic formulae extraction
techniques and then derived the common steps. The extraction of syntactic
information is one of them, and it is applied in order to highlight entities
in the texts with appropriate tags. This stage of NLP is called Named Entity
Recognition (NER) and it is used to label each word of the text with its
grammatical category (e.g., noun, verb, etc.). This step is carried out in [9]
for sentence classification purpose.

The integration of modelling and AI is a field of growing interest, as it is
witnessed by some emerging workshops on the topic333https://mde-
intelligence.github.io/ and by a recent special issue on Software and System
Modeling [17]. Other relevant works include: [12], where an NLP-based
assistant is proposed to facilitate the domain modelling providing completion
suggestion for the partial model under construction; [14], where an
interactive bot proposes to the modeller configurations for the model under
construction. Other works develop NLP-based techniques to extract domain
models from text [15, 16].

##  III Methodology

The goal of this research work is to provide a semi-automatic support to the
software development process focusing on a first fundamental step: the
Requirement Engineering phase. Looking at Fig. 1 describes the proposed
approach at-a-glance by representing a conceptual architecture of the
components to develop. Three layers are presented to separate and organize the
elements of the approach.

Figure 1: A Conceptual Architecture

#### User layer

it considers the parts of the approach that are in charge of the System
Analyst. The underlying hypothesis is that the requirement engineer has to
provide both a set of requirements expressed according to the BDD paradigm and
a partial system model (i.e., Architecture in the figure). The discussion on
the size of such initial model is in Section IV. Here, we make the hypothesis
that such an initial model considers a complete Class Diagram (CD)/Block
Diagram (BD) containing the complete classes/blocks, their relationships,
methods, attributes, ports, flows, etc. The initial model also considers a
partial State Machine Diagram (SMD) — the Behaviour in Fig. 1 — where only
states are defined without any indication of the transitions444The part of
such SMD is in white in the figure..

#### Knowledge layer

the approach is based on the detect-and-translate idea, i.e., the detection of
requirement patterns inside the specification, with the consequent definition
of a model fragment that best represents the patterns in SysML. To formalise
such idea, the Knowledge layer mainly contains the Translation Rules knowledge
base mapping meta-requirements (MetaReq) — the requirement patterns — and the
resulting SysML fragment (MetaFragment). From a user perspective, this
knowledge base “generates” two guidelines, one for writing requirements and
one for modelling behaviours in SysML, respectively Requirement and Modelling
Guidelines. The schemes of the MetaReqs are discussed in Section IV.

#### Tool layer

to support the automation of the approach, a toolchain will be developed,
considering the following steps. First, a ClauseExtractor is in charge of
getting the parts of a BDD requirement. To this aim, a simple parser based on
a formal grammar will be developed considering the main keywords of the
Gherkin language (i.e., given, when, then, and, or). Then the clauses are
separated and a simple Abstract Syntax Tree (AST) is generated from the
requirement. Three different tools — namely givenCmp, whenCmp, thenCmp —
receive as input: the AST of the requirement, the initial SysML model, the
knowledge base, improving as output the SMD by creating transitions and
annotating them with triggers, conditions and actions.

###  III-A A Railway Signalling example

Let us consider the example of the partial system model, in the railways
domain depicted in Fig. 2 to show the final aim of the automation to define. A
trackside controller communicates with the train by sending the
EmergencyStop() message in case of immediate braking. A related safety
requirement is:

Given a Train in running, When the Braking Supervision receives an Emergency
Stop Message, Then the Braking Supervision activates the Emergency Brake and
goes in braking.

By also considering the partial SMD of the train reported in Fig. 3(a) where
only two states are present (i.e., Running and Braking), the proposed approach
is aimed at enrich such SMD then obtaining the one of Fig. 3(b). This is done
by adding the transition from the Running to Braking states. Such a transition
has as trigger, the EmergencyStop() signal, and an action Activate() which
causes a signal to be sent to the Brake blocks.

Figure 2: Train System Block Diagram

(a) Incomplete SMD

(b) Complete SMD

Figure 3: Train SMDs

##  IV Technical Discussion

Some technical details on how to design and implement the approach are
presented in the following.

#### Structure of the Knowledge Base

Even if the structures of the knowledge base, of the MetaReqs and of the
MetaFragments are not defined yet, here a preliminary discussion about their
formalisation is reported. Table II reports on the upper part, the requirement
considered in the railway example (Requirement) and MetaReq structure
(MetaReq).

MetaReq |  Requirement
---|---
Given $<<$Block as context1$>>$ in $<<$State as starting$>>$, When $<<$Block as context2$>>$ receives $<<$Signal as event$>>$, Then $<<$Block as context3$>>$ $<<$Signal as operation$>>$ to the $<<$Block as context4$>>$ and goes in $<<$State as final$>>$. |  Given a Train in running, When the Braking Supervision receives an Emergency Stop Message, Then the Braking Supervision activates the Emergency Brake and goes in braking.
Elements Matching
Role |  Value
context1 (Block) |  Train
starting (State) |  Running
context2 (Block) |  Braking Supervision
event (Signal) |  EmergencyStop()
context3 (Block) |  Braking Supervision
operation (Signal) |  Activate()
context4 (Block) |  Brake
final (State) |  Braking
TABLE II: Pattern, text and elements matching

The MetaReq includes some of the keyword of the BDD requirement structure and
a list of matching element patterns, whose syntax is given by the notation
$<<$metclass as role$>>$. A concrete requirement fits a MetaRequirement if for
each element pattern $<<$metclass as role$>>$ there exists a word that
corresponds to a model element, instance of the $<$metclass$>$, that plays the
given $<role>$ in the requirement. For example, in “Given a Train in running”,
the word Train corresponds to the element pattern $<<$Block as context1$>>$
because Train is a Block in the SysML model. So, Train plays the role of
context1. Roles are then used to fill the corresponding MetaFragment, that is
depicted in Fig. 4.

Figure 4: Clauses-SMD Mapping

The lower part of the Table II reports the matching of the elements on the
considered railway domain example.

Another consideration is related to the matching mechanism. Two “extreme”
criteria are possible: 1) to define a rigid Backus-Naur Form (BNF) for the
MetaReq, forcing the System Analyst to use a fixed schema, 2) to fully use NLP
techniques to match the concrete requirement with the possible MetaReq. The
flexibility/effectiveness trade-offs between these two solutions is an ongoing
part of the research.

#### On the Initial Model

The work presented so far starts from relaxed initial conditions. The
automatic approach presented in this paper starts from a System Under Test
(SUT) model comprising blocks, detailed with attributes, methods and ports if
necessary, furthermore states of the main blocks are also considered. The idea
is to develop this research through incremental steps starting from a basic
but incomplete SMD model until the construction of such a SMD model from
scratch. In this way, it will be possible to assess the results obtained at
each step. Future works would also consider the possibility to built
automatically the rest of the SUT model.

#### On the Traceability of the Results

as we are aware that AI techniques are hard to be accepted in the development
and certification processes of critical systems, we explicitly consider
traceability as a prime citizen in our approach. The idea is to generate
automatically also some traceability information to allow the System Analyst
to not lose control of the model. To this aim, each time a requirement is
matched with a MetaReq, the tool also generates a Requirement Diagram (RD).
Fig. 5 reports the RD of the requirement considered in our example. Model
elements are reported in the diagram as well, connected to the requirement by
means of $<<$satisfy$>>$ stereotyped dependencies; further information (for
simplicity in a comment) could be used to specify the role(s) that the model
element plays with respect to the requirement.

Figure 5: Requirement Traceability

#### Status of the Research

Currently, the work is focused on clarifying the method and to define a
sufficient number of MetaReqs and related MetaFragments. First, we will devise
methods to manipulate requirements considering rigid BNF grammars then we will
focus on the extraction of lexical information from texts. In particular,
NLTK555https://www.nltk.org and spaCy666https://spacy.io libraries will be
used and compared in order to develop the core algorithm of the
ClauseExtractor. The definition of a SysML Profile that supports the
traceability is also on the top of the working agenda.

## Acknowledgment

The work of Maria Stella de Biase is granted by PON Ricerca e Innovazione
$2014/2020$ MUR — Ministero dell’Università e della Ricerca (Italy) — with the
PhD program XXXVI cycle.

## References

  * [1] Wynne M. and Hellesoy A., “The Cucumber Book: Behaviour-Driven Development for Testers and Developers”, Pragmatic Bookshelf, 2012.
  * [2] North D., “Introducing BDD,” in Better Software Magazine, 2006.
  * [3] Kapurch, S.J., “NASA Systems Engineering Handbook”, Eds. DIANE Publishing Company, 2010.
  * [4] Campanile L., de Biase M.S., Marrone S., Raimondo M., Verde L., “On the Evaluation of BDD Requirements with Text-based Metrics: the ETCS-L3 Case Study.”, 2022, to apper in Proc. of 14th Int.l KES Conference - Intelligent Decision Technologies, Springer.
  * [5] Fantechi A., Gnesi S., Ristori G., Carenini M., Vanocchi M., Moreschini P., “Assisting requirement formalization by means of natural language translation” in Formal Methods in System Design, 1994.
  * [6] Buzhinsky I., “Formalization of natural language requirements into temporal logics: a survey”, In IEEE 17th International Conference on Industrial Informatics (INDIN), vol. 1, 2019.
  * [7] Sascha K., Cheng B.H.C., “Automated Analysis of Natural Language Properties for UML Models”, In International Conference on Model Driven Engineering Languages and Systems, 2006.
  * [8] Brunello A., Montanari A., Reynolds M., “Synthesis of LTL formulas from natural language texts: State of the art and research directions”, Leibniz International Proceedings in Informatics, Vol. 147, 2019.
  * [9] Frederiksen S.J., Aromando J., Hsiao M.S., “Automated Assertion Generation from Natural Language Specifications” In 2020 IEEE International Test Conference (ITC), 2020.
  * [10] Brambilla M., Cabot J., Wimmer M., “Model-driven software engineering in practice.” Synthesis lectures on software engineering, 2017.
  * [11] Kharchenko V., Kovalenko A., Siora O., Andrashov A., “V-models of safety critical system life cycle: Classification and application,” IEEE 9th International Conference on Dependable Systems, Services and Technologies (DESSERT), 2018.
  * [12] Burgueño L., Clarisó R., Gérard S., Li S., Cabot J., “An NLP-based architecture for the autocompletion of partial domain models.”, International Conference on Advanced Information Systems Engineering, 2021.
  * [13] Furness N., van Houten H., Arenas L., Maarten B., “ERTMS Level 3: the Game-Changer”, 2017.
  * [14] Saini R., Mussbacher G., Guo J. L., Kienzle J., “Automated, interactive, and traceable domain modelling empowered by artificial intelligence”, Software and Systems Modeling, 2022.
  * [15] Ibrahim M., Ahmad R., “Class diagram extraction from textual requirements using natural language processing (NLP) techniques”, International Conference on Computer Research and Development, 2010.
  * [16] Jyothilakshmi M.S., Samuel P., “Domain ontology based class diagram generation from functional requirements”, International Conference on Intelligent Systems Design and Applications (ISDA), 2012.
  * [17] L. Burgueño, J. Cabot, M. Wimmer, S. Zschaler; Guest editorial to the theme section on AI-enhanced model-driven engineering (2022) Software and Systems Modeling, 21 (3), pp. 963 - 965. DOI: 10.1007/s10270-022-00988-0