# Data augmentation and multimodal learning for predicting drug response in
patient-derived xenografts from gene expressions and histology images

Alexander Partin  Computing, Environment and Life Sciences, Argonne National
Laboratory, Lemont, IL, USA  Correspondence<EMAIL_ADDRESS>Computing,
Environment and Life Sciences, Argonne National Laboratory, Lemont, IL, USA
Thomas Brettin  Computing, Environment and Life Sciences, Argonne National
Laboratory, Lemont, IL, USA  Yitan Zhu  Computing, Environment and Life
Sciences, Argonne National Laboratory, Lemont, IL, USA  James M. Dolezal
Department of Medicine, Section of Hematology/Oncology, University of Chicago
Medical Center, Chicago, IL, USA  Sara Kochanny  Department of Medicine,
Section of Hematology/Oncology, University of Chicago Medical Center, Chicago,
IL, USA  Alexander T. Pearson  Department of Medicine, Section of
Hematology/Oncology, University of Chicago Medical Center, Chicago, IL, USA
Maulik Shukla  Computing, Environment and Life Sciences, Argonne National
Laboratory, Lemont, IL, USA  Yvonne A. Evrard  Frederick National Laboratory
for Cancer Research, Leidos Biomedical Research, Inc. Frederick, MD, USA
James H. Doroshow  Division of Cancer Therapeutics and Diagnosis, National
Cancer Institute, Bethesda, MD, USA  Rick L. Stevens  Computing, Environment
and Life Sciences, Argonne National Laboratory, Lemont, IL, USA  Department of
Computer Science, The University of Chicago, Chicago, IL, USA

###### Abstract

Background. Prediction of drug response is a critical research area in
precision oncology and has been previously explored with large drug screening
studies of cancer cell lines (CCLs). Patient-derived xenografts (PDXs) are an
appealing platform for preclinical drug studies because the in vivo
environment of PDXs helps preserve tumor heterogeneity and usually better
mimics drug response of patients with cancer compared to CCLs.

Methods. We investigate multimodal neural network (MM-Net) and data
augmentation for drug response prediction in PDXs. The MM-Net learns to
predict response using drug descriptors, gene expressions (GE), and histology
whole-slide images (WSIs) where the multi-modality refers to tumor features
only. The MM-Net uses late integration where separate subnetworks are used to
encode the different input feature types before concatenation and prediction
layers. Median tumor volume per treatment group is assessed relative to the
control group to create a binary variable representing response. We explore
whether the integration of WSIs with GE improves predictions as compared with
models that use GE alone. We use two methods to address the limited number of
response values in the dataset: 1) homogenize drug representations which
allows to combine single-drug and drug-pairs treatments into a single dataset,
2) augment drug-pair samples by switching the order of drug features which
doubles the sample size of all drug-pair samples. These methods enable us to
combine single-drug and drug-pair treatments, allowing us to train multimodal
and unimodal neural networks (NNs) without changing architectures or the
dataset.

Results. Prediction performance of three unimodal NNs which use GE are
compared to assess the contribution of data augmentation methods. NN that uses
the full dataset which includes the original and the augmented drug-pair
treatments as well as single-drug treatments significantly outperforms NNs
(p-values < 0.01) that ignore either the augmented drug-pairs or the single-
drug treatments. In assessing the contribution of multimodal learning based on
the MCC metric, MM-Net statistically significantly outperforms all the
baselines (p-values < 0.05).

Conclusion. Our results show that data augmentation and integration of
histology images with GE can improve prediction performance of drug response
in PDXs.

_K_ eywords Drug response prediction $\cdot$ Patient-derived xenograft $\cdot$
PDX $\cdot$ Histology whole-slide images $\cdot$ Multimodal deep learning
$\cdot$ Precision oncology

##  1 Introduction

With recent advancements in applications of artificial intelligence in
medicine and biology, predictive modeling has gradually become one of the
primary directions in cancer research for analytically predicting the response
of tumors to anticancer treatments [1, 2]. In particular, conventional machine
learning (ML) and deep learning (DL) methods have been widely investigated for
building computational drug response prediction models for cancer cell lines
(CCLs) with large datasets of omics profiles [3]. The complex heterogeneities
of cancer that occur within and between tumors present a major obstacle to
successful discovery of robust biomarkers and therapies [4, 5]. Patient-
derived tumor xenografts (PDXs) are a contemporary biological model that is
created by grafting cancerous tissue, obtained from human tumor specimens,
into immunodeficient mice. The in vivo environment of PDXs helps preserve
tumor heterogeneity as compared to in vitro CCLs, and therefore, is presumed
to better mimic the response of human patients with certain cancer types. PDXs
continue gain reputation for studying cancer and investigating drug response
in preclinical drug studies [6, 7, 8].

Predicting the response of tumors to drug treatments with accurate and robust
computational models provides a modern approach for identifying top candidates
for preclinical drug screening experiments or personalized cancer treatments.
A variety of ML and DL approaches have been explored with high-throughput drug
screens with CCLs [9, 10]. Alternatively, our literature search retrieved only
two publications that have used only PDX data to train prediction models for
drug response [11, 12]. Both studies used the Novartis PDX data (NIBR PDXE),
which were generated using a 1x1x1 experimental design [13, 14], where each
drug was tested against each patient PDX model using only one entumored mouse
per model. Nguyen et al. used an optimal model complexity (OMC) strategy with
random forests to build drug response models for 26 treatment-cancer type
combinations [11]. They considered three genomic feature types in their
analyses, including gene expressions (GE), copy-number alterations (CNAs), and
single-nucleotide variants (SNVs). While considering a single feature type at
a time, they used OMC to determine an optimal subset of features to obtain the
best performing model for each treatment-cancer type pair. They showed that
for the majority of cases, models developed with OMC outperform models that
used all the available features. In another study, Kim et al. proposed PDXGEM,
a pipeline that identifies biomarkers predictive of drug response in PDX and
then uses the identified markers to train prediction models [12]. To identify
predictive genes based on GE and drug response, the pipeline utilizes a
strategy similar to co-expression extrapolation (COXEN) [15, 16], and
consequently selects the genes whose co-expression patterns are best preserved
between PDXs and patient tumors. They trained prediction models using random
forests for six treatment-tumor type combinations and then predicted response
in patients.

A primary challenge in modeling drug response with PDXs is the limited
availability of drug response data. The sample size of PDXs is usually orders
of magnitude smaller than the analogous CCL datasets. It has been shown that
increasing the amount of training samples improves generalization performance
of supervised learning models in vision and text applications [17, 18], as
well as drug response models in CCLs [19, 20]. Collecting PDX response data,
either through experiments or integration of multiple datasets, carries
considerable technical and financial challenges. Alternatively, instead of
directly increasing the sample size, the volume of data can be expanded by
representing each sample with multiple feature types. Multimodal architectures
that integrate genomic and histology images have been shown to improve
prognosis prediction of patients with cancer as compared with unimodal
architectures that learn only from a single data modality (i.e., feature type)
[21, 22, 23]. Another possible direction to address the limited sample size is
data augmentation. Augmentation techniques have been extensively explored with
image and text data, but not much with drug response. While augmenting images
has become a common practice, tabular datasets such as omics profiles lack
standardized augmentation methods.

In this study, we investigate two approaches for predicting drug response in
PDX, including multimodal learning and data augmentation. We explore a
multimodal neural network (MM-Net) that learns to predict drug response in
PDXs using GE and histology whole-slide images (WSIs), two feature types
representing cancer tissue, and molecular descriptors that represent drugs.
The multimodal architecture is designed to take four feature sets as inputs:
(1) GE, (2) histology images, and (3,4) molecular descriptors of a pair of
drugs. We benchmark the prediction performance of MM-Net against three
baselines: (1) NN trained with drug descriptors and GE, (2) NN trained with
drug descriptors and WSIs, and (3) LightGBM model [24] trained with drug
descriptors and GE. With multimodal learning, our goal is to explore whether
the integration of histology images with GE improves the prediction
performance as compared with models that use GE features alone. For data
augmentation, we homogenize the drug representation of single-drug and drug-
pair treatments in order to combine them into a single dataset. Moreover, we
introduce an augmentation method that doubles the sample size of all drug-pair
treatments. The proposed augmentations allow us to combine single-drug and
drug-pair treatments to train MM-Net and the baselines without changing the
architectures and the dataset. We explore the contribution of augmented data
for improving the prediction of drug response.

This paper provides unique contributions compared with existing works that
train drug response models with PDX data [11, 12]. First, we build general
drug response models for PDXs across multiple cancer types and drug
treatments. Alternatively, prediction models in [11, 12] are built for
specific combinations of cancer type and drug treatment. Thus, our study
targets a more challenging task. Second, we utilize PDX histology images with
multimodal architecture which has not yet been studied for drug response
prediction in PDX. Our study presents a framework for integrating image data
with genomic measurements and drug chemical structure for predicting treatment
effect. Third, we combine multiple treatments into a single dataset by
homogenizing single-drug and drug-pair treatments and utilize drug features
for training models. This provides an advantage over existing works which
built prediction models for unique combinations of drugs and cancer types, and
therefore, disregard drug features when model training. Furthermore, we
propose an augmentation method for drug-pairs that doubles the sample size of
the drug-pair treatments in the dataset. Fourth, existing studies built
prediction models using the PDXE drug screening data which were generated
using a 1x1x1 experimental design, i.e. one mouse per model per treatment. In
contrast, we utilized the PDMR dataset where treatment response is measured by
comparing a group of treated mice to a group of untreated mice. The group
approach allows to assess the variability of response across mice and might be
considered as more reliable in capturing tumor heterogeneity.

##  2 Methods

###  2.1 Data

####  2.1.1 Experimental Design of Drug Efficacy in PDX

We used unpublished PDX drug response data from the NCI Patient-Derived Models
Repository (PDMR; http://pdmr.cancer.gov). The NCI PDMR performs
histopathology assessment, whole-exome sequencing and RNA-Seq analysis of a
subset of tumors from each PDX model to establish baseline histology and omic
characterization for each model. To date, over six hundred unique PDX models
have been characterized and data made available through the public website.
Baseline pathology and omic characterization from 487 models were used for
this analysis. The efficacy of drug treatments in PDMR is measured through
controlled group experiments. Fig. 1 illustrates the process of obtaining
primary tumor specimen from a patient, engrafting tumor tissue into PDX
models, performing baseline characterization, expanding tumor tissue over
multiple passages within a lineage, and then using the expanded tumors in drug
treatment experiments. A total of 96 PDX models from 89 unique patients were
used for the experiments. The tumors are grown subcutaneously in NOD.Cg-
Prkdcscid Il2rgtm1Wjl/SzJ (NSG) host mice and staged to an approximate tumor
weight of 200 mm3 for the drug studies. The control group is treated with a
vehicle only (vehicle is a solution that delivers the drug in the treated
groups). The preclinical dataset includes twelve single-drug and 36 drug-pair
treatment arms. Median tumor volume over time for each vehicle or treatment
group is used for response assessment. The GE profiles and WSIs were
aggregated and preprocessed for the downstream ML and DL analysis.

Figure 1: Expansion of tumor tissue from the source specimen (ORIGINATOR) to
mice across multiple passages. Mice originated from the same specimen are
divided into a control group and multiple treatment groups. Tumors from
certain mice were histologically and molecularly profiled resulting in whole-
slide images and omics profiles.

####  2.1.2 Drug Response in PDX

The growth of tumor volume over time represents the response of PDX tumors to
drug treatments, as shown in Fig. 2. The group approach intends to capture the
variability of PDX drug response across mice of the same lineage [25]. Median
tumor volume per treatment group is assessed relative to the control group to
create a binary variable representing response. Specifically, for each drug
treatment experiment, a single experienced preclinical study analyst assessed
the curves of median tumor volume over time for each treatment arm, and
assigned "1" for response (regression of at least 10% from staging for more
than one consecutive time point at any point during the study) and "0" for
non-response. In essence, a modified RECIST score for regression versus no
regression was used to label the response. Thus, a single best response value
was assigned for each treatment arm.

Figure 2: Representative tumor growth curves for vehicle control and drugs
three drugs. Label of response assigned to Drug B as tumor achieved regression
and non-response the remaining drugs.

####  2.1.3 Data Generation

Three feature types were used for model training, including drug descriptors,
GE, and histology image tiles.

Gene Expressions. The transcriptomic data of PDXs were generated using RNA-
Seq. We converted the values into TPM (transcripts per kilobase million),
transformed with log2, and finally standardized each gene to have a zero mean
and a unit standard deviation. For building drug response models, we selected
942 landmark genes discovered by the Library of Integrated Network-Based
Cellular Signatures (LINCS) project [26], which have been shown to well-
represent transcriptomic changes.

Drug Descriptors. We used the Dragon software package (version 7.0) to
calculate numerical descriptors of drug molecular structure. The software
calculates various types of molecular descriptors, such as atom types,
estimations of molecular properties, topological and geometrical descriptors,
functional groups and fragment counts, and drug-like indices. A total of 1,993
descriptors were used for the analysis after removing descriptors with missing
values. We standardized the descriptor values across drugs to have a zero mean
and a unit standard deviation.

Histology Images. During PDX model expansion, entumored mice were sacrificed
between 1000-2000 mm3 for collection of tumors for representative model
characterization including histopathological examination. Hematoxylin and
Eosin (H&E) stained pathology slides were digitized into WSIs using an Aperio
AT2 digital whole slide scanner (Leica Biosystems) at 20x magnification. A
board-certified pathologist from Frederick National Laboratory of Cancer
Research reviewed the slides to ensure the PDX models were consistent with the
original patient diagnosis. Tumor regions of interest (ROIs) were annotated
within the image slides using QuPath [27] by a single University of Chicago
pathologist.

Whole slide images were processed into individual tiles using the Slideflow
software package [28, 29], as shown in Fig. 3. Image tiles were extracted from
within annotated ROIs in a grid pattern at 302 $\mu$m by 302 $\mu$m with no
overlap, then downsampled to 299 pixels by 299 pixels, resulting in an
effective optical magnification of 10x. Background tiles were removed with
grayscale filtering, where each tile is converted to the HSV color space and
removed if more than 60% of its pixels have a hue value of less than 0.05.
Image tiles then underwent digital stain normalization using the Reinhard
method [30] and were subsequently standardized to give each image a mean of
zero with a variance of one.

Figure 3: Whole-slide histology image processing. A) Whole-slide images were
annotated with region of interest (ROI) outlines, and image tiles were
extracted from within ROIs in a grid-wise fashion. B) Extracted non-background
tiles underwent digital stain normalization using the Reinhard method [30].

####  2.1.4 Constructing PDX Drug Response Dataset

In constructing the drug response dataset, we populated samples from each
group experiment with the corresponding response label. Each sample that was
molecularly and histologically profiled consists of three feature types and a
binary response value. The feature types include drug descriptors, GE, and
histology tiles, as illustrated in Fig. 4. Table 1 lists the summary
statistics of the dataset.

The PDMR preclinical dataset contains experiments of single-drug treatments
and drug pairs. In order to include both single-drug and drug-pair treatments
in the dataset, and ensure consistent dimensionality of drug features, we
homogenized single-drug treatments by duplicating drug descriptors to form a
pseudo drug-pair that includes two identical drug feature vectors. In this
case, the samples of single-drug and drug-pair treatments will follow the same
input dimensionality for all ML models. Moreover, because switching the
position of drug features in drug-pair treatments should not change the drug
response, we augmented all drug-pair samples by switching the position of the
two drugs while keeping the drug response value unchanged. Such data
augmentation doubles the number of drug-pair samples in the dataset.

Following the integration of group samples into the dataset and the
augmentation of drug-pair treatments, the drug response dataset contains 6,962
samples. The total number of treatment groups in the dataset is 959 with 917
non-response and 42 response groups. The dataset contains three feature types
(modalities): two vectors of drug descriptors (a vector for each drug), GE
profile, and histology tiles. Each sample consists of a unique combination of
drug descriptors and GE profiles. However, each such sample contains multiple
image tiles from a corresponding histology slide. Concretely, each sample
consists of a GE profile, vector of descriptors for drug 1 and drug 2, and
multiple image tiles as shown in Fig. 4. We store the data in TFRecords
(TensorFlow file format) which enables efficient data prefetching and loading,
and therefore, considerably decreases the training and inference time.

Figure 4: Data arrangement of the PDX drug response dataset. The dataset contains 959 treatment groups after homogenizing and augmenting the drug experiments as described in 2.1.4. For example, Sample 1 is a single-drug treatment that is structured as a pseudo drug-pair treatment where Drug 1 and Drug 2 features are the same feature vectors; Sample N is an augmented version of Sample N-1, in which the positions of drug feature vectors are switched. Note that each sample contains multiple histology image tiles that were extracted from a large WSI. Table 1: Summary of the PDX drug response dataset used for building prediction models. Patients | 89
---|---
Primary tumor specimens | 96
Single-drug treatments | 12
Drug-pair treatments (excludes augmented samples) | 36
Treatment groups | 959
Gene expression profiles | 487
Histology whole-slide images (WSIs) | 487
Histology image tiles (extracted from WSIs) | 177,468
Single-drug response values | 2,556
Drug-pair response values (includes augmented values) | 4,406
Drug response values | 6,962

####  2.1.5 Data Splits

Data leakage can lead to overly optimistic predictions [31]. Two primary
characteristics of our dataset may lead to leakage if random splitting is used
to generate training, validation, and test sets. First, a drug response label
is assigned to all the samples in the entire treatment group. To prevent
leakage, we make sure that samples from the same treatment group always appear
together in one of the training, validation, or test sets. Second, the
augmented drug-pair samples represent, in reality, the same experiment, and
therefore, are also placed together when generating the splits. With this
strategy, we generated 100 data splits for the analysis (10-fold cross-
validation repeated ten times with different random seeds), where the tissue
features (GE profiles and WSIs) of the same treatment group are kept together
and not shared across training, validation, and test sets of each data split.

###  2.2 Prediction Models

We explore the performance of MM-Net, shown in Fig. 5, in predicting the drug
response in PDXs. The model takes preprocessed feature sets as inputs,
including drug descriptors, GE, and histology tiles, and passes them through
subnetworks of layers. The encoded features from the subnetworks are merged
via a concatenation layer and propagated to the output for predicting a binary
drug response.

Figure 5: Multimodal neural network (MM-Net).

Since the dataset is highly redundant in terms of GE and drug features as
shown in Fig 4 and Table 1 (there are 48 unique drug treatments and 487 unique
expression sets), we use a single layer of trainable weights to encode these
features with the goal to mitigate overfitting. The image tiles are passed
through a subnetwork of convolutional layers of the Xception model [32] with
weights pre-trained on ImageNet [33]. The output from the convolutional neural
network (CNN) is passed through a series of dense layers before being
concatenated with the encoded GE and drug descriptor representations.

We compare the performance of MM-Net with three unimodal baselines that use
either GE or WSI as tumor features: (1) UME-Net, NN that uses GE, (2) UMH-Net,
NN that uses histology tiles, and (3) LGBM, LightGBM that uses GE. Note that
all models use drug descriptors.

###  2.3 Training and Evaluation

We used a randomized search to obtain a set of hyperparameters (HPs) for UME-
Net, including optimizer, learning rate, and layer dimensions that encode GE
and drug descriptors. The values of these HPs were also used for MM-Net. A few
remaining HPs that are unique to MM-Net were determined in a separate search
using the MM-Net architecture. Another randomized search was performed to
obtain the HPs for LGBM such as the number of leaves in the decision tree and
the number of trees.

To mitigate overfitting, we used the early stopping mechanism in TensorFlow
and LightBGM where model trainings terminate automatically if the predictions
on a validation set have not been improved for a predefined number of training
iterations. The early stopping parameter was set to 10 epochs for all NNs and
100 boosting rounds for the LGBM. All NNs were trained for 400 training epochs
which triggered early stopping and ensured model convergence.

Since the dataset is highly imbalanced in terms of drug response distribution,
we used a weighted loss function that penalizes more heavily incorrect
predictions of the response samples as opposed to the non-response samples.
For training MM-Net, we used only 10% of the image tiles that were available
in each WSI, because our preliminary experiments revealed that the prediction
performance does not improve if additional tiles are used. The 10% of the
tiles have been drawn at random from each WSI.

For model evaluation, we used three performance metrics for binary
classification tasks, including Matthews correlation coefficient (MCC), area
under the receiver operating characteristic curve (AUROC), and area under the
precision-recall curve (AUPRC). We calculated the metrics based on a test set
of each one of the 100 splits. To compute each metric for a given test set, we
aggregated all the sample predictions in the test set. In the case of baseline
models where each tumor sample is represented by a GE vector, the prediction
model generates in a single probability value for each sample. However, in the
case of MM-Net where each tumor sample is represented by multiple image tiles
in addition to the GE profile, the prediction model generates a single
probability value for each image tile which results in multiple predicted
probability values for a single sample. To conform with the output of the
baseline models, the tile predictions from MM-Net were aggregated via mean to
provide a single probability value for each sample. Note that while only 10%
of the available tiles were used for training MM-Net, all tiles in the test
set were used to compute predictions and subsequently obtain the performance
metrics.

##  3 Results

A total of six prediction models were analyzed as summarized in Table 2. The
models differ in terms of the feature sets and the samples that were used for
training and validation (binary columns in Table 2). All models were evaluated
across the same 100 data splits. Fig. 6 shows the performance metrics,
including MCC, AUPRC, and AUROC where each data point is a metric value
calculated for a given split. The average score of each model was aggregated
via mean across the splits for each metric (listed in Table 2).

Table 2: Performance metrics including MCC, AUPRC, and AUROC are listed for drug response prediction models (UME-Net, UME-Netorg, UME-Netpairs, UMH-Net, MM-Net, and LGBT). As summarized in the binary columns, the differences between the models are in the feature sets (GE, WSI, or both) and the samples. To compute the average score for each metric and model, the predictions were aggregated via mean across the data splits. Model | WSI | GE | Single-drug | Drug-pair | Augmented drug-pairs | MCC | AUPRC | AUROC
---|---|---|---|---|---|---|---|---
UME-Net | – | v | v | v | v | 0.2958 | 0.2996 | 0.8047
UME-Netorg | – | v | v | v | – | 0.2391 | 0.2610 | 0.7766
UME-Netpairs | – | v | – | v | v | 0.2039 | 0.2355 | 0.7423
UMH-Net | v | – | v | v | v | 0.2124 | 0.2303 | 0.7977
MM-Net | v | v | v | v | v | 0.3102 | 0.2974 | 0.7978
LGBM | – | v | v | v | v | 0.2594 | 0.2784 | 0.8065
Figure 6: Boxplots showing the distribution of scores for the investigated
drug response prediction models. The differences between the models are
summarized in Table 2.

In constructing the drug response dataset, we used two approaches to increase
the number of response values, as described in section 2.1.4. We analyzed the
effect of these two methods on the prediction performance by comparing three
unimodal NNs that were trained with GE and drug descriptors on subsets of the
dataset: (1) UME-Net, trained with the full dataset that includes the original
and the augmented drug-pair treatments as well as single-drug treatments, (2)
UME-Netpairs, trained with only drug-pair samples which include the original
and the augmented samples, and (3) UME-Netorg, trained with the original
single-drug and drug-pair samples that exclude the augmented drug-pair
samples. Fig. 6 shows the prediction performance of using the different
training subsets across the data splits where each data point is a metric
value calculated for a given split. While the distribution of scores across
the splits is quite substantial, removing either subset of samples (single-
drug samples or augmented drug-pairs) results in a significant decline in
prediction performance (p-values < 0.05). In other words, augmentation methods
lead to a significant improvement in performance of the NNs when trained with
GE and descriptors. Hence, we used the full set of the available training
samples for the analyses of multimodal learning.

The MM-Net architecture, shown in Fig. 5, was compared against three baseline
models, including UME-Net, UMH-Net, and LGBM. The performance metrics
represent the ability of the models to generalize on a test set of unseen
observations. In addition, we computed a paired t-test of the scores across
the 100 splits to compare whether there is statistically significant
difference between the values. Based on the aggregated MCC, MM-Net
statistically significantly outperforms all the baselines (p-values < 0.05)
except for UME-Net. When considering the AUPRC and AUROC, MM-Net outperforms
UMH-Net but there is no significant difference when comparing MM-Net with UME-
Net or LGBM.

We can observe a large spread of scores for all models and metrics. This
indicates that for certain data splits, the models exhibit very high
generalization performance, while for other splits, the models almost entirely
fail to learn a meaningful mapping function for predicting drug response. In
practical scenarios, where the goal is to design a highly generalizable model,
a careful analysis should be conducted to determine the training and
validation sets that adequately represent the test set. In our analysis,
however, the objective was to conduct large-scale trainings across multiple
data splits and examine the overall capacity of MM-Net in predicting drug
response across multiple cancer types and treatments. We observe that for
certain dataset splits, MM-Net outperforms the baselines but for other splits
it underperforms, as shown in Fig. 7. An in-depth investigation is further
required to understand in which cases MM-Net trained with WSI significantly
improves prediction generalization.

(a)

(b)

Figure 7: Receiver operating characteristic (ROC) curves for two different
data splits illustrating the MM-Net outperforming the baselines on the left
plot, and underperforming on the right.

##  4 Discussion

In this study, we investigated data augmentation methods and a multimodal
architecture for predicting drug response in PDXs. We utilized the PDMR drug
response dataset of single-drug and drug-pair treatments that were generated
in controlled group experiments with PDX models of multiple cancer types. To
assess the utility of the proposed methods, we conducted a large-scale
analysis by training MM-Net and three baseline models over 100 data splits
that contain GE profiles, histology image tiles, and molecular drug
descriptors. We demonstrated that data augmentation methods lead to a
significant improvement in drug response predictions across all performance
metrics (MCC, AUPRC, AUROC). Alternatively, the MM-Net model exhibits
statistically significant improvement in prediction performance only when
measured by the MCC.

The data splitting strategy and the choice of performance metrics play an
important role in the downstream analysis when evaluating the utility of
prediction models for practical applications. The dataset size and its
diversity in terms of PDX models and treatments allowed us to generate
multiple data splits while mitigating data leakage between training,
validation, and test sets. Since each split comprises unique GE and histology
images, we face a challenging prediction problem as opposed to a situation in
which the samples are randomly split. Alternative splitting strategies may
involve a careful choice of a single test set with the goal to reduce the
distributional shift between training and test sets [34]. Instead of carefully
assembling the most representative test set, we chose to conduct a large-scale
analysis to assess the empirical range of prediction performance with NNs and
LGBM. The results show a large spread of scores across the splits, indicating
that for certain data splits the models exhibit high prediction performance,
while for other splits, the learning of models fails. When we specifically
focus on the performance of MM-Net as compared with the baselines, we discover
that in 46 out of the 100 splits, the MM-Net outperformed the UME-Net
baseline. This observation implies that for certain data splits, the histology
images boost the generalization performance of the prediction model, and
therefore, its potential utility in preclinical and clinical settings. A
further investigation is required to better understand the cases and data
characteristics in which histology images improve response prediction.

Technological progress in digital pathology and high-throughput omic profiling
have led researchers to generate big data repositories of histology images and
omics data, as well as algorithms to jointly analyze these diverse data types.
Several papers have explored multimodal architectures that combine histology
images with omics data for predicting survival outcomes of cancer patients.
Mobadersany et al. demonstrated that a CNN-based supervised learning model
combined with cox regression accurately predicts survival outcomes of glioma
patients from histology and mutation data [21]. Cheerla et al. proposed an
unsupervised learning method to learn a low-dimensional representation for
each feature type and, consequently, concatenated the learned representations
to predict survival outcome of cancer patients [22]. They have also
demonstrated on 20 cancer types that a custom dropout layer that randomly
drops an entire feature vector improves predictions. Building upon existing
works, Chen et al. introduced a supervised architecture for multimodal fusion
of histology and omics data to predict patient survival and applied their
method to glioma and clear cell renal cell carcinoma patients [23]. The model
uses graph convolutional network (GCN) and CNN to encode histology image data
and feed-forward network for mutation data. Each encoded feature vector is
passed through an attention mechanism and subsequently fused via a Kronecker
product. The cox regression is finally used to predict patient survival. While
these papers do not consider drug treatments in their analysis, they exploit
modern approaches for enhancing predictions of multimodal NNs with histology
and omics data and can be further explored for drug response prediction.

Considering the scale of existing PDX datasets, data fusion and augmentation
provide promising research directions for enhancing predictive capabilities
with PDXs. However, special care should be taken because high-dimensional
features sets can often lead to severe overfitting and poor generalization.
Presumably to mitigate overfitting, Nguyen et al. [11] and Kim et al. [12]
used feature selection methods to reduce the dimensionality of PDX data while
considering a single omics feature type at a time. Therefore, multimodal
learning exhibits tradeoff between enriching the feature space via multimodal
fusion and overfitting. To alleviate this tradeoff, alternative methods can be
explored to reduce the feature space while incorporating multiple feature
types [35]. With the methods proposed in this study and ongoing research into
novel augmentation and fusion techniques, PDX pharmaco-omic datasets may
become more suitable for modern deep learning techniques and further increase
interest for building prediction models to advance precision oncology.

## Funding

These activities were part of the IMPROVE project under the NCI-DOE
Collaboration Program that has been funded in whole or in part with Federal
funds from the National Cancer Institute, National Institutes of Health, Task
Order No. 75N91019F00134 and from the Department of Energy under Award Number
ACO22002-001-00000.

## References

  * [1] P. J. Ballester, R. Stevens, B. Haibe-Kains, R. S. Huang, and T. Aittokallio, “Artificial intelligence for drug response prediction in disease models,” Briefings in Bioinformatics, Oct. 2021.
  * [2] G. Adam, L. Rampášek, Z. Safikhani, P. Smirnov, B. Haibe-Kains, and A. Goldenberg, “Machine learning approaches to drug response prediction: challenges and recent progress,” npj Precision Oncology, vol. 4, no. 1, 2020.
  * [3] H. Sharifi-Noghabi, S. Jahangiri-Tazehkand, P. Smirnov, C. Hon, A. Mammoliti, S. K. Nair, A. S. Mer, M. Ester, and B. Haibe-Kains, “Drug sensitivity prediction from cell line-based pharmacogenomics data: guidelines for developing machine learning models,” Briefings in Bioinformatics, vol. 22, Nov. 2021.
  * [4] R. Fisher, L. Pusztai, and C. Swanton, “Cancer heterogeneity: implications for targeted therapeutics,” British Journal of Cancer, vol. 108, no. 3, pp. 479–485, 2013.
  * [5] J. Seoane and L. D. Mattos-Arruda, “The challenge of intratumour heterogeneity in precision medicine,” Journal of Internal Medicine, vol. 276, no. 1, pp. 41–51, 2014.
  * [6] Y. A. Evrard, A. Srivastava, J. Randjelovic, J. H. Doroshow, D. A. Dean, J. S. Morris, and J. H. Chuang, “Systematic Establishment of Robustness and Standards in Patient-Derived Xenograft Experiments and Analysis,” Cancer research, vol. 80, no. 11, pp. 2286–2297, 2020.
  * [7] T. Goto, “Patient-Derived Tumor Xenograft Models: Toward the Establishment of Precision Cancer Medicine,” Journal of Personalized Medicine, vol. 10, no. 3, 2020.
  * [8] G. J. Yoshida, “Applications of patient-derived tumor xenograft models and tumor organoids,” Journal of Hematology & Oncology, vol. 13, no. 1, 2020\.
  * [9] Y.-C. Chiu, H.-I. H. Chen, A. Gorthi, M. Mostavi, S. Zheng, Y. Huang, and Y. Chen, “Deep learning of pharmacogenomics resources: moving towards precision oncology,” Briefings in Bioinformatics, vol. 21, no. 6, pp. 2066–2083, 2020.
  * [10] Y. Zhu, T. Brettin, Y. A. Evrard, A. Partin, F. Xia, M. Shukla, H. Yoo, J. H. Doroshow, and R. L. Stevens, “Ensemble transfer learning for the prediction of anti-cancer drug response,” Scientific Reports, vol. 10, no. 1, pp. 1–11, 2020.
  * [11] L. C. Nguyen, S. Naulaerts, A. Bruna, G. Ghislat, and P. J. Ballester, “Predicting Cancer Drug Response In Vivo by Learning an Optimal Feature Selection of Tumour Molecular Profiles,” Biomedicines, vol. 9, Oct. 2021.
  * [12] Y. Kim, D. Kim, B. Cao, R. Carvajal, and M. Kim, “PDXGEM: patient-derived tumor xenograft-based gene expression model for predicting clinical response to anticancer therapy in cancer patients,” BMC Bioinformatics, vol. 21, no. 1, 2020.
  * [13] G. Migliardi, F. Sassi, D. Torti, F. Galimi, E. R. Zanella, M. Buscarino, D. Ribero, A. Muratore, P. Massucco, A. Pisacane, M. Risio, L. Capussotti, S. Marsoni, F. D. Nicolantonio, A. Bardelli, P. M. Comoglio, L. Trusolino, and A. Bertotti, “Inhibition of MEK and PI3K/mTOR Suppresses Tumor Growth but Does Not Cause Tumor Regression in Patient-Derived Xenografts of RAS-Mutant Colorectal Carcinomas,” Clinical Cancer Research, vol. 18, pp. 2515–2525, May 2012.
  * [14] H. Gao, J. M. Korn, S. Ferretti, J. E. Monahan, Y. Wang, M. Singh, C. Zhang, C. Schnell, G. Yang, Y. Zhang, O. A. Balbin, S. Barbe, H. Cai, F. Casey, S. Chatterjee, D. Y. Chiang, S. Chuai, S. M. Cogan, S. D. Collins, E. Dammassa, N. Ebel, M. Embry, J. Green, A. Kauffmann, C. Kowal, R. J. Leary, J. Lehar, Y. Liang, A. Loo, E. Lorenzana, E. Robert McDonald, M. E. McLaughlin, J. Merkin, R. Meyer, T. L. Naylor, M. Patawaran, A. Reddy, C. Röelli, D. A. Ruddy, F. Salangsang, F. Santacroce, A. P. Singh, Y. Tang, W. Tinetto, S. Tobler, R. Velazquez, K. Venkatesan, F. Von Arx, H. Q. Wang, Z. Wang, M. Wiesmann, D. Wyss, F. Xu, H. Bitter, P. Atadja, E. Lees, F. Hofmann, E. Li, N. Keen, R. Cozens, M. R. Jensen, N. K. Pryer, J. A. Williams, and W. R. Sellers, “High-throughput screening using patient-derived tumor xenografts to predict clinical trial drug response,” Nature Medicine, vol. 21, pp. 1318–1325, Nov. 2015.
  * [15] S. C. Smith, A. S. Baras, J. K. Lee, and D. Theodorescu, “The COXEN principle: Translating signatures of in vitro chemosensitivity into tools for clinical outcome prediction and drug discovery in cancer,” Cancer Research, vol. 70, no. 5, pp. 1753–1758, 2010.
  * [16] Y. Zhu, “Enhanced Co-Expression Extrapolation (COXEN) Gene Selection Method for Building Anti-Cancer Drug Response Prediction Models,” Genes, vol. 11, no. 1070, 2020.
  * [17] J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou, “Deep Learning Scaling is Predictable, Empirically,” arXiv:1712.00409 [cs, stat], Dec. 2017.
  * [18] J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit, “A Constructive Prediction of the Generalization Error Across Scales,” in ICLR, 2020.
  * [19] F. Xia, J. Allen, P. Balaprakash, T. Brettin, C. Garcia-Cardona, A. Clyde, J. Cohn, J. Doroshow, X. Duan, V. Dubinkina, Y. Evrard, Y. J. Fan, J. Gans, S. He, P. Lu, S. Maslov, A. Partin, M. Shukla, E. Stahlberg, J. M. Wozniak, H. Yoo, G. Zaki, Y. Zhu, and R. Stevens, “A cross-study analysis of drug response prediction in cancer cell lines,” Briefings in Bioinformatics, Sept. 2021.
  * [20] A. Partin, T. Brettin, Y. A. Evrard, Y. Zhu, H. Yoo, F. Xia, S. Jiang, A. Clyde, M. Shukla, M. Fonstein, J. H. Doroshow, and R. L. Stevens, “Learning curves for drug response prediction in cancer cell lines,” BMC Bioinformatics, vol. 22, May 2021.
  * [21] P. Mobadersany, S. Yousefi, M. Amgad, D. A. Gutman, J. S. Barnholtz-Sloan, J. E. Velázquez Vega, D. J. Brat, L. A. D. Cooper, L. A. D. C. Designed, and L. A. D. C. Performed, “Predicting cancer outcomes from histology and genomics using convolutional networks,” PNAS, vol. 115, no. 13, 2018.
  * [22] A. Cheerla and O. Gevaert, “Deep learning with multimodal representation for pancancer prognosis prediction,” Bioinformatics, vol. 35, pp. 446–454, 2019.
  * [23] R. J. Chen, M. Y. Lu, J. Wang, D. F. K. Williamson, S. J. Rodig, N. I. Lindeman, and F. Mahmood, “Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis,” IEEE Transactions on Medical Imaging, 2019\.
  * [24] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” in Advances in Neural Information Processing Systems, vol. 30, Curran Associates, Inc., 2017.
  * [25] A. S. Mer, W. Ba-Alawi, P. Smirnov, Y. X. Wang, B. Brew, J. Ortmann, M.-S. Tsao, D. W. Cescon, A. Goldenberg, and B. Haibe-Kains, “Integrative Pharmacogenomics Analysis of Patient-Derived Xenografts,” Cancer Research, vol. 79, no. 17, pp. 4539–4550, 2019.
  * [26] A. Subramanian, R. Narayan, S. M. Corsello, D. D. Peck, T. E. Natoli, X. Lu, J. Gould, J. F. Davis, A. A. Tubelli, J. K. Asiedu, D. L. Lahr, J. E. Hirschman, Z. Liu, M. Donahue, B. Julian, M. Khan, D. Wadden, I. C. Smith, D. Lam, A. Liberzon, C. Toder, M. Bagul, M. Orzechowski, O. M. Enache, F. Piccioni, S. A. Johnson, N. J. Lyons, A. H. Berger, A. F. Shamji, A. N. Brooks, A. Vrcic, C. Flynn, J. Rosains, D. Y. Takeda, R. Hu, D. Davison, J. Lamb, K. Ardlie, L. Hogstrom, P. Greenside, N. S. Gray, P. A. Clemons, S. Silver, X. Wu, W. N. Zhao, W. Read-Button, X. Wu, S. J. Haggarty, L. V. Ronco, J. S. Boehm, S. L. Schreiber, J. G. Doench, J. A. Bittker, D. E. Root, B. Wong, and T. R. Golub, “A Next Generation Connectivity Map: L1000 Platform and the First 1,000,000 Profiles,” Cell, vol. 171, pp. 1437–1452, 2017.
  * [27] P. Bankhead, M. B. Loughrey, J. A. Fernández, Y. Dombrowski, D. G. McArt, P. D. Dunne, S. McQuaid, R. T. Gray, L. J. Murray, H. G. Coleman, J. A. James, M. Salto-Tellez, and P. W. Hamilton, “QuPath: Open source software for digital pathology image analysis,” Scientific Reports, vol. 7, Dec. 2017.
  * [28] J. M. Dolezal, A. Trzcinska, C.-Y. Liao, S. Kochanny, E. Blair, N. Agrawal, X. M. Keutgen, P. Angelos, N. A. Cipriani, and A. T. Pearson, “Deep learning prediction of BRAF-RAS gene expression signature identifies noninvasive follicular thyroid neoplasms with papillary-like nuclear features,” Modern Pathology, vol. 34, pp. 862–874, May 2021.
  * [29] J. Dolezal, skochanny, and fmhoward, “jamesdolezal/slideflow: Slideflow 1.0 \- Official Public Release,” Nov. 2021.
  * [30] E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, “Color transfer between images,” IEEE Computer Graphics and Applications, vol. 21, pp. 34–41, July 2001.
  * [31] S. Kaufman, S. Rosset, and C. Perlich, “Leakage in Data Mining: Formulation, Detection, and Avoidance,” ACM Transactions on Knowledge Discovery from Data, vol. 6, no. 4, 2012.
  * [32] F. Chollet, “Xception: Deep Learning with Depthwise Separable Convolutions,” arXiv:1610.02357 [cs], Apr. 2017.
  * [33] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, June 2009.
  * [34] K. Stacke, G. Eilertsen, J. Unger, and C. Lundstrom, “Measuring Domain Shift for Deep Learning in Histopathology,” IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 2, pp. 325–336, 2021.
  * [35] Y. Zhu, T. Brettin, F. Xia, A. Partin, M. Shukla, H. Yoo, Y. A. Evrard, J. H. Doroshow, and R. L. Stevens, “Converting tabular data into images for deep learning with convolutional neural networks,” Scientific Reports, vol. 11, no. 1, 2021.


# How Flexible is Your Computing System?

Shihua Huang<EMAIL_ADDRESS>Prodrive Technologies ,  Luc Waeijen
GrAI Matter Labs<EMAIL_ADDRESS>and  Henk Corporaal  Eindhoven
University of Technology<EMAIL_ADDRESS>
###### Abstract.

In literature computer architectures are frequently claimed to be highly
flexible, typically implying there exist trade-offs between flexibility and
performance or energy efficiency. Processor flexibility, however, is not very
sharply defined, and as such these claims can not be validated, nor can such
hypothetical relations be fully understood and exploited in the design of
computing systems. This paper is an attempt to introduce scientific rigour to
the notion of flexibility in computing systems. A survey is conducted to
compile an overview of references to flexibility in literature both in the
computer architecture domain as well as related fields. A classification is
introduced to categorize different views on flexibility, which ultimately form
the foundation for a qualitative definition of flexibility. Departing from the
qualitative definition of flexibility, a generic quantifiable metric is
proposed, enabling valid quantitative comparison of the flexibility of various
architectures. To validate the proposed method, and evaluate the relation
between the proposed metric and the general notion of flexibility, the
flexibility metric is measured for 25 computing systems, including CPUs, GPUs,
DSPs, and FPGAs, and 40 ASIPs taken from literature. The obtained results
provide insights into some of the speculative trade-offs between flexibility
and properties such as energy efficiency and area efficiency. Overall the
proposed quantitative flexibility metric shows to be commensurate with some
generally accepted qualitative notions of flexibility collected in the survey,
although some surprising discrepancies can also be observed. The proposed
metric and the obtained results are placed into context of the state of the
art on compute flexibility, and extensive reflection provides not only a
complete overview of the field, but also discusses possible alternative
approaches and open issues. Note that this work does not aim to provide a
final answer to the definition of flexibility, but rather provides a framework
to initiate a broader discussion in the computer architecture society on
defining, understanding, and ultimately taking advantage of flexibility.

flexibility, versatility, metric

††copyright: none††ccs: General and reference Metrics††ccs: Theory of
computation Abstract machines††ccs: General and reference Surveys and
overviews

##  1\. Introduction

Arguably one of the most famous books in the field is “Computer Architecture —
A Quantitative approach” by John L. Hennessy and David A. Patterson (Hennessy
and Patterson, 2011). The title itself is concise and apt, so it is
interesting the authors opted to add this particular subtitle: a quantitative
approach. It implies the belief that quantifying design choices ultimately
leads to better computer architectures, a message that certainly could be
directed towards those who make claims about flexible architectures without
means of quantifying these claims, or even without as much as a commonly
accepted qualitative definition of flexibility. With Moore’s law seemingly
coming to an end, new advancements in computing will have to be made on the
architectural side. To advance the state of the art, fundamental understanding
of various trade-offs in computer design is vital. The way forward therefore,
is a quantitative one.

Many key system properties such as performance, power dissipation, and energy
efficiency are all well defined in a quantitative manner. With these metrics
in place, quantitative, objective comparisons can be conducted between
different machines. For flexibility however, such a quantitative (and even
qualitative!) definition is lacking, despite it’s increasing importance in
system design. In product research and development, computing platforms are
required to sufficiently support new or updated algorithms, as algorithms are
changing at a striking speed. Exemplary are the current developments in
artificial intelligence, which result in new compute-intensive algorithms at a
high cadence. Such rapidly developing markets require systems that can deal
with changing applications, which is the property flexibility typically seems
to refer to. However, in absence of a proper definition, it is impossible to
make solid statements, and compare designs on flexibility.

Despite the lack of a formal definition of flexibility, there appear to be
some commonly accepted notions surrounding flexibility. In particular,
flexibility seems mainly used to refer to the adaptability of processors to
different applications. This leads to the common idea that a programmable
processor which can be reused across applications is ‘flexible’. On the other
hand, a processor with fixed logic such as an ASIC cannot adapt, exposing its
inflexibility (Null and Lobur, 2014). As can be seen in Figure 1 the authors
of these figures appear to agree with this sentiment. However there are also
some contradictions to this view on flexibility however. For example in Figure
1(a), among programmable processors the field-programmable devices are claimed
by the authors to be less flexible than software programmable processors due
to their inadequate programmability (Karuri and Leupers, 2014). Unfortunately
the term “programmability” is also ill-defined here. Perhaps the best
definition of programmability in existence is to check Turing completeness of
a programmable device, but this would leave only two classes of
programmability making it a measure with low practical value. Another
perspective on flexibility refers to how well a processor supports different
applications, in which case FPGAs could be seen as the most flexible, since
any hardware, including DSPs, GPUs, and CPUs, can be instantiated on FPGAs.
Apart from this debate on how to rank the flexibility of architecture classes,
perhaps even more worrisome are the contradicting claims on relations between
flexibility and other metrics. In Figure 1(d) Ahmed Osman El-Rayis equates
flexibility to area, whereas Tobias Noll sees it as directly related to power
dissipation in Figure 1(a). While this is definitely not an exhaustive list of
views on flexibility, it painfully exposes how the lack of a formal definition
leads to a wild-west of claims and conflicting visions, none of which can be
backed up with objective measurements.

[]Graph with on the horizontal axis performance, left vertical axis
flexibility, and right vertical axis power dissipation. Architectures are
grouped, roughly following a parabolic function with it’s top
flexibility/power dissipation at the lowest performance point, sloping down
towards zero as performance increases. For left to right the architecture
groups are: General purpose processors, domain specific processors, ASIPs,
field programmable devices, application specific ICs, physically optimized ICs

(a) Flexibility related to performance and power according to Tobias Noll
(Kappen and Noll, 2006; Karuri and Leupers, 2014). Note that according to this
figure flexibility is directly related to power dissipation.  (b) Flexibility
versus performance plot by Geoffrey Ndu (Ndu, 2012).The ordering of the
flexibility of architecture classes aligns quite well with the ordering given
in Figure 1(a), although curve is quite different.  (c) ‘Application
Flexibility‘ versus ‘Performance Efficiency‘ by Markus Willems from Synopsys
(Willems, 2019). Unfortunately no further definition of both these metrics is
provided in the accompanying text.  (d) An interesting graph by Ahmed Osman
El-Rayis (El-Rayis, 2014) which relates many metrics including flexibility.In
contrast to Figure 1(a) flexibility is here claimed to be directly related to
area.

Figure 1. Collection of published figures with claims about the flexibility of
architecture classes, and relations between flexibility and other metrics such
as performance and power. Note that none of the axis in these figures are
labelled with units.

Despite the greatly varying interpretations of flexibility, many seem to agree
that there may be interesting relations and trade-offs between flexibility and
other properties, such as performance and energy efficiency. As illustrated in
Figure 1, processing platform alternatives have been evaluated and ranked in
terms of flexibility, performance, power dissipation, and area (Kappen and
Noll, 2006; Karuri and Leupers, 2014; Ndu, 2012; Willems, 2019; El-Rayis,
2014). A variety of architectures have been developed which claim to balance
energy efficiency and flexibility. The development of domain-specific
functional units and a transition to heterogeneous multi-core systems are a
testament to this notion (Kumar et al., 2003; Hameed, 2013; Fasthuber et al.,
2013). These hypothetical relations suggest that understanding flexibility is
beneficial when designing a system, enabling informed trade-offs.

To overcome the lack of understanding of flexibility, this work sets out to
provide both a qualitative and quantitative definition of flexibility. It
should be noted though that, with such a fragmented landscape of
interpretations of flexibility, the authors are under no illusion that it is
possible to unify the field and reach consensus without a wider discussion.
Instead, this work is to be seen as a first attempt, which does not so much
aspire to provide a definitive answer, as it hopes to be thought provoking and
spark a discussion within the community.

To arrive at a quantitative measure for flexibility, a qualitative definition
is established by exploring uses of the term flexibility in literature and
then examining various options. Based on this qualitative definition, a
quantitative measure is derived. In the translation of flexibility, from a
qualitative term to a quantitative definition, there exist several degrees of
freedom. The final choices made in this translation are motivated extensively.
However, more importantly, the alternatives are discussed systematically in
similar detail. The intention is that this systematic approach can provide an
initial framework for a broader discussion in the community on how flexibility
should be defined, such that eventually a standard accepted metric can be
determined.

To validate the metric proposed in this work, in total 14 applications are
benchmarked on 25 different commercial off the shelf (COTS) platforms. It is
shown that results align with several common concepts of flexibility found in
the literature. For example, GPUs deliver the highest performance in general
for the used parallel benchmarks but sacrifice in terms of flexibility,
compared to general purpose processors (GPPs) in Figure 1(a). Furthermore the
flexibility of 40 application specific architectures from literature is
determined, to evaluate the relation between specialisation and flexibility.

The main contributions of this work can be summarized as follows:

  1. (1)

Survey of flexibility in computing and other engineering fields;

  2. (2)

A qualitative and quantitative definition of flexibility;

  3. (3)

Definition of intrinsic workload to normalize performance, and accompanying
open-source tool (Huang and Waeijen, 2021) for automated extraction;

  4. (4)

Evaluation of the proposed metric on 25 COTS platforms for 14 benchmarks, and
40 application specific processors taken from literature;

  5. (5)

In-depth Comparison with alternative definitions of flexibility/versatility;

  6. (6)

Extensive discussion on the proposed metric, and various alternatives, placing
it in context of the current state of the art.

NOTE: This document is a partial preprint. Full text will be made available
after publication elsewhere.

## References

  * (1)
  * El-Rayis (2014) Ahmed Osman El-Rayis. 2014\.  _Reconfigurable architectures for the next generation of mobile device telecommunications systems_.  Ph.D. Dissertation.
  * Fasthuber et al. (2013) Robert Fasthuber, Francky Catthoor, Praveen Raghavan, and Frederik Naessens. 2013.  _Energy-Efficient Communication Processors: Design and Implementation for Emerging Wireless Systems_.  Springer Publishing Company, Incorporated.
  * Hameed (2013) Rehan Hameed. 2013\.  _Balancing Efficiency and Flexibility In Specialized Computing_.  Ph.D. Dissertation.
  * Hennessy and Patterson (2011) John L. Hennessy and David A. Patterson. 2011.  _Computer Architecture, Fifth Edition: A Quantitative Approach_ (5th ed.).  Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
  * Huang and Waeijen (2021) Shihua Huang and Luc Waeijen. 2021.  Intrinsic WorkloadEstimator.  https://github.com/lwaeijen/WorkloadEstimator.
  * Kappen and Noll (2006) Götz Kappen and Tobias Noll. 2006.  Application specific instruction processor based implementation of a GNSS receiver on an FPGA. 58–63.
  * Karuri and Leupers (2014) Kingshuk Karuri and Rainer Leupers. 2014.  _Application Analysis Tools for ASIP Design: Application Profiling and Instruction-set Customization_.  Springer Publishing Company, Incorporated.
  * Kumar et al. (2003) R. Kumar, K. I. Farkas, N. P. Jouppi, P. Ranganathan, and D. M. Tullsen. 2003.  Single-ISA heterogeneous multi-core architectures: the potential for processor power reduction. In _Proceedings. 36th Annual IEEE/ACM International Symposium on Microarchitecture, 2003. MICRO-36._ 81–92.
  * Ndu (2012) Geoffrey Ndu. 2012\.  _Boosting Single Thread Performance in Mobile Processors using Reconfigurable Acceleration_.  Ph.D. Dissertation.
  * Null and Lobur (2014) Linda Null and Julia Lobur. 2014.  _The Essentials of Computer Organization and Architecture_ (4th ed.).  USA.
  * Willems (2019) Markus Willems. 2019\.  _Application-Specific Processors for High Throughput, Low Latency, and Flexible 5G Communication SoCs_.  Synopsis.  https://www.synopsys.com/designware-ip/technical-bulletin/5g-asips-communication-socs.html